Theory of the vortex-clustering transition in a confined two-dimensional quantum fluid