Using and understanding cross-validation strategies. Perspectives on Saeb et al.

Abstract This three-part review takes a detailed look at the complexities of cross-validation, fostered by the peer review of Saeb et al.’s paper entitled “The need to approximate the use-case in clinical machine learning.” It contains perspectives by reviewers and by the original authors that touch upon cross-validation: the suitability of different strategies and their interpretation.

[1]  Konrad P. Kording,et al.  The need to approximate the use-case in clinical machine learning , 2017, GigaScience.

[2]  David C. Mohr,et al.  Making Activity Recognition Robust against Deceptive Behavior , 2015, PloS one.

[3]  Koby Crammer,et al.  Learning Bounds for Domain Adaptation , 2007, NIPS.

[4]  Dimitris Samaras,et al.  Deriving reproducible biomarkers from multi-site resting-state data: An Autism-based example , 2016, NeuroImage.

[5]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[6]  Andres Hoyos Idrobo,et al.  Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines , 2016, NeuroImage.

[7]  Daniel S. Margulies,et al.  Predicting brain-age from multimodal imaging data captures cognitive impairment , 2016, NeuroImage.

[8]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[9]  Klaus-Robert Müller,et al.  Covariate Shift Adaptation by Importance Weighted Cross Validation , 2007, J. Mach. Learn. Res..

[10]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[11]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[12]  Marti J. Anderson,et al.  Permutation Tests for Linear Models , 2001 .

[13]  Saharon Rosset,et al.  Leakage in data mining: formulation, detection, and avoidance , 2011, TKDD.

[14]  Jianhua Z. Huang,et al.  Asymptotic optimality and efficient computation of the leave-subject-out cross-validation , 2012, 1302.4607.

[15]  Max A. Little,et al.  Accurate telemonitoring of Parkinson’s disease progression by non-invasive speech tests , 2009 .

[16]  Max A. Little,et al.  Accurate Telemonitoring of Parkinson's Disease Progression by Noninvasive Speech Tests , 2009, IEEE Transactions on Biomedical Engineering.

[17]  Aakash Gupta,et al.  Activity recognition in patients with lower limb impairments: Do we need training data from each patient? , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[18]  David J. Hand,et al.  Classifier Technology and the Illusion of Progress , 2006, math/0606441.