Patterned random matrices: deviations from universality