A strengthened asymptotic freeness result for random matrices with applications to free entropy