Vortex lattice instability at the nanoscale in a parallel magnetic field