Instabilities and melting in a two-dimensional magnetic dipolar system