A random matrix model for random approximate $t$-designs