Automatic sleep spindle detection: benchmarking with fine temporal resolution using open science tools

Sleep spindle properties index cognitive faculties such as memory consolidation and diseases such as major depression. For this reason, scoring sleep spindle properties in polysomnographic recordings has become an important activity in both research and clinical settings. The tediousness of this manual task has motivated efforts for its automation. Although some progress has been made, increasing the temporal accuracy of spindle scoring and improving the performance assessment methodology are two aspects needing more attention. In this paper, four open-access automated spindle detectors with fine temporal resolution are proposed and tested against expert scoring of two proprietary and two open-access databases. Results highlight several findings: (1) that expert scoring and polysomnographic databases are important confounders when comparing the performance of spindle detectors tested using different databases or scorings; (2) because spindles are sparse events, specificity estimates are potentially misleading for assessing automated detector performance; (3) reporting the performance of spindle detectors exclusively with sensitivity and specificity estimates, as is often seen in the literature, is insufficient; including sensitivity, precision and a more comprehensive statistic such as Matthew's correlation coefficient, F1-score, or Cohen's κ is necessary for adequate evaluation; (4) reporting statistics for some reasonable range of decision thresholds provides a much more complete and useful benchmarking; (5) performance differences between tested automated detectors were found to be similar to those between available expert scorings; (6) much more development is needed to effectively compare the performance of spindle detectors developed by different research teams. Finally, this work clarifies a long-standing but only seldomly posed question regarding whether expert scoring truly is a reliable gold standard for sleep spindle assessment.

[1]  Manuel Schabus,et al.  Hemodynamic cerebral correlates of sleep spindles during human non-rapid eye movement sleep , 2007, Proceedings of the National Academy of Sciences.

[2]  Nurettin Acir,et al.  Automatic recognition of sleep spindles in EEG by using artificial neural networks , 2004, Expert Syst. Appl..

[3]  Giulio Tononi,et al.  Thalamic dysfunction in schizophrenia suggested by whole-night deficits in slow and fast spindles. , 2010, The American journal of psychiatry.

[4]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[5]  Thierry Dutoit,et al.  Automatic sleep spindles detection — Overview and development of a standard proposal assessment method , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[6]  Christian O'Reilly,et al.  Combining time-frequency and spatial information for the detection of sleep spindles , 2015, Front. Hum. Neurosci..

[7]  Nancy Chinchor,et al.  MUC-4 evaluation metrics , 1992, MUC.

[8]  Beena Ahmed,et al.  An automatic sleep spindle detector based on wavelets and the teager energy operator , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  Christian O’Reilly,et al.  Assessing EEG sleep spindle propagation. Part 2: Experimental characterization , 2014, Journal of Neuroscience Methods.

[10]  A. Borbély,et al.  All‐night dynamics of the human sleep EEG , 1993, Journal of sleep research.

[11]  Ann K. Shinn,et al.  Reduced Sleep Spindles and Spindle Coherence in Schizophrenia: Mechanisms of Impaired Memory Consolidation? , 2012, Biological Psychiatry.

[12]  Darrel C. Ince,et al.  The case for open computer programs , 2012, Nature.

[13]  Lalu Mansinha,et al.  Localization of the complex spectrum: the S transform , 1996, IEEE Trans. Signal Process..

[14]  Laura B Ray,et al.  Validating an automated sleep spindle detection algorithm using an individualized approach , 2010, Journal of sleep research.

[15]  Ninad Gujar,et al.  Sleep-Dependent Facilitation of Episodic Memory Details , 2011, PloS one.

[16]  Pierre Baldi,et al.  Assessing the accuracy of prediction algorithms for classification: an overview , 2000, Bioinform..

[17]  D. Fabó,et al.  Overnight verbal memory retention correlates with the number of sleep spindles , 2005, Neuroscience.

[18]  David J. Hand,et al.  ROC Curves for Continuous Data , 2009 .

[19]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.

[20]  Elizaveta Solomonova,et al.  REM sleep characteristics of nightmare sufferers before and after REM sleep deprivation. , 2010, Sleep medicine.

[21]  Tore Nielsen,et al.  Revisiting the ROC curve for diagnostic applications with an unbalanced class distribution , 2013, 2013 8th International Workshop on Systems, Signal Processing and their Applications (WoSSPA).

[22]  B Saletu,et al.  Automatic Sleep-Spindle Detection Procedure: Aspects of Reliability and Validity , 1994, Clinical EEG.

[23]  I Feinberg,et al.  Systematic trends across the night in human sleep cycles. , 1979, Psychophysiology.

[24]  Pietro Perona,et al.  Sleep spindle detection: crowdsourcing and evaluating performance of experts, non-experts, and automated methods , 2014, Nature Methods.

[25]  Athanasios Tsanas,et al.  Stage-independent, single lead EEG sleep spindle detection using the continuous wavelet transform and local weighted smoothing , 2015, Front. Hum. Neurosci..

[26]  Carlyle T. Smith,et al.  The function of the sleep spindle: A physiological index of intelligence and a mechanism for sleep-dependent memory consolidation , 2011, Neuroscience & Biobehavioral Reviews.

[27]  Poul Jennum,et al.  Inter-expert and intra-expert reliability in sleep spindle scoring , 2015, Clinical Neurophysiology.

[28]  Vahid Tarokh,et al.  DiBa: A Data-Driven Bayesian Algorithm for Sleep Spindle Detection , 2012, IEEE Transactions on Biomedical Engineering.

[29]  G Dirlich,et al.  The REM-NREM sleep cycle: renewal process or periodically driven process? , 1980, Sleep.

[30]  M. L. Lauzon,et al.  A General Description of Linear Time-Frequency Transforms and Formulation of a Fast, Invertible Transform That Samples the Continuous S-Transform Spectrum Nonredundantly , 2010, IEEE Transactions on Signal Processing.

[31]  Christian O’Reilly,et al.  Assessing EEG sleep spindle propagation. Part 1: Theory and proposed methodology , 2014, Journal of Neuroscience Methods.

[32]  S. Chokroverty,et al.  The visual scoring of sleep in adults. , 2007, Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine.

[33]  K. Crowley,et al.  The effects of normal aging on sleep spindle and K-complex production , 2002, Clinical Neurophysiology.

[34]  Nikolaos K. Uzunoglu,et al.  Sleep spindle detection using artificial neural networks trained with filtered time-domain EEG: A feasibility study , 2005, Comput. Methods Programs Biomed..

[35]  Lois F. Lunin,et al.  The Neurological Information Network , 1987, J. Am. Soc. Inf. Sci..

[36]  A. Chesson,et al.  The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology, and Techinical Specifications , 2007 .

[37]  J. Born,et al.  Grouping of Spindle Activity during Slow Oscillations in Human Non-Rapid Eye Movement Sleep , 2002, The Journal of Neuroscience.

[38]  C. O’Reilly,et al.  Montreal Archive of Sleep Studies: an open‐access resource for instrument benchmarking and exploratory research , 2014, Journal of sleep research.

[39]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[40]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[41]  Antti Saastamoinen,et al.  Development and comparison of four sleep spindle detection methods , 2007, Artif. Intell. Medicine.

[42]  M. Ferrara,et al.  Sleep spindles: an overview. , 2003, Sleep medicine reviews.

[43]  Christian O'Reilly,et al.  Spindles in Svarog: framework and software for parametrization of EEG transients , 2015, Front. Hum. Neurosci..

[44]  E. Wolpert.  A Manual of Standardized Terminology, Techniques and Scoring System for Sleep Stages of Human Subjects. , 1969 .

[45]  Günther J. L. Gerhardt,et al.  Benchmarking matching pursuit to find sleep spindles , 2006, Journal of Neuroscience Methods.

[46]  Masako Tamaki,et al.  Fast sleep spindle (13-15 hz) activity correlates with sleep-dependent improvement in visuomotor performance. , 2008, Sleep.

[47]  Osman Erogul,et al.  Efficient sleep spindle detection algorithm with decision tree , 2009, Expert Syst. Appl..

[48]  E G Silver.  The rem. , 1981, Science.