Variable Selection in Linear Mixed Models Using an Extended Class of Penalties