Rayleigh instability of confined vortex droplets in critical superconductors