Significance tests and sample homogeneity loophole

In their recent comment, published in Nature, Jeffrey T.Leek and Roger D.Peng discuss how P-values are widely abused in null hypothesis significance testing . We agree completely with them and in this short comment we discuss the importance of sample homogeneity tests. No matter with how much scrutiny data are gathered if homogeneity tests are not performed the significance tests suffer from sample homogeneity loophole and the results may not be trusted. For example sample homogeneity loophole was not closed in the experiment testing local realism in which a significant violation of Eberhard inequality was found. We are not surprised that Bell type inequalities are violated since if the contextual character of quantum observables is properly taken into account these inequalities cannot be proven. However in order to trust the significance of the violation sample homogeneity loophole must be closed. Therefore we repeat after Jeffrey T.Leek and Roger D.Peng that sample homogeneity loophole is probably just the tip of the iceberg.