Model selection in irregular problems: Applications to mapping quantitative trait loci