Breakdown of effective-medium theory beyond the critical angle