Permutation Tests for Classification: Towards Statistical Significance in Image-Based Studies

Estimating statistical significance of detected differences between two groups of medical scans is a challenging problem due to the high dimensionality of the data and the relatively small number of training examples. In this paper, we demonstrate a non-parametric technique for estimation of statistical significance in the context of discriminative analysis (i.e., training a classifier function to label new examples into one of two groups). Our approach adopts permutation tests, first developed in classical statistics for hypothesis testing, to estimate how likely we are to obtain the observed classification performance, as measured by testing on a hold-out set or cross-validation, by chance. We demonstrate the method on examples of both structural and functional neuroimaging studies.

[1]  Wiro J. Niessen,et al.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2001 , 2001, Lecture Notes in Computer Science.

[2]  J. Schmee.  Applied Statistics—A Handbook of Techniques , 1984 .

[3]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[4]  Anders M. Dale,et al.  Automated manifold surgery: constructing geometrically accurate and topologically correct models of the human cerebral cortex , 2001, IEEE Transactions on Medical Imaging.

[5]  A. Dale,et al.  High‐resolution intersubject averaging and a coordinate system for the cortical surface , 1999, Human brain mapping.

[6]  W. Eric L. Grimson,et al.  Discriminative Analysis for Image-Based Studies , 2002, MICCAI.

[7]  Martin Styner,et al.  Shape versus Size: Improved Understanding of the Morphology of Brain Structures , 2001, MICCAI.

[8]  Thomas E. Nichols,et al.  Nonparametric permutation tests for functional neuroimaging: A primer with examples , 2002, Human brain mapping.

[9]  A M Dale,et al.  Measuring the thickness of the human cerebral cortex from magnetic resonance images. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Alex Pentland,et al.  Shape analysis of brain structures using physical and experimental modes , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[12]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[13]  Isabelle Guyon,et al.  What Size Test Set Gives Good Error Rate Estimates? , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Ron Kikinis,et al.  Medical Image Computing and Computer-Assisted Intervention — MICCAI 2002 , 2002, Lecture Notes in Computer Science.

[15]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[16]  John Suckling,et al.  Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain , 1999, IEEE Transactions on Medical Imaging.

[17]  N. Kanwisher,et al.  How Distributed Is Visual Category Information in Human Occipito-Temporal Cortex? An fMRI Study , 2002, Neuron.

[18]  U. Grenander,et al.  Hippocampal morphometry in schizophrenia by high dimensional brain mapping. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[19]  P. Good,et al.  Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses , 1995 .

[20]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[21]  Kiralee M. Hayashi,et al.  Dynamics of Gray Matter Loss in Alzheimer's Disease , 2003, The Journal of Neuroscience.