Towards a finite-N hologram