Sleep spindle detection based on non-experts: A validation study

Accurate and efficient detection of sleep spindles is a methodological challenge. The present study describes a method of using non-experts for manual detection of sleep spindles. We recruited five experts and 168 non-experts to manually identify spindles in stage N2 and stage N3 sleep data using a MATLAB interface. Scorers classified each spindle into definite and indefinite spindle (with weights of 1 and 0.5, respectively). First, a method of optimizing the thresholds of the expert/non-expert group consensus according to the results of experts and non-experts themselves is described. Using this method, we established expert and non-expert group standards from expert and non-expert scorers, respectively, and evaluated the performance of the non-expert group standards by compared with the expert group standard (termed EGS). The results indicated that the highest performance was the non-expert group standard when definite spindles were only considered (termed nEGS-1; F1 score = 0.78 for N2; 0.68 for N3). Second, four automatic spindle detection methods were compared with the EGS. We found that the performance of nEGS-1 versus EGS was higher than that of the four automated methods. Our results also showed positive correlation between the mean F1 score of individual expert in EGS and the F1 score of nEGS-1 versus EGS across 30 segments of stage N2 data (r = 0.61, P < 0.001). Further, we found that six and nine non-experts were needed to manually identify spindles in stages N2 and N3, respectively, while maintaining acceptable performance of nEGS-1 versus EGS (F1 score = 0.79 for N2; 0.64 for N3). In conclusion, this study establishes a detailed process for detection of sleep spindles by non-experts in a crowdsourcing scheme.

[1]  G. Tononi,et al.  Reduced sleep spindle activity in schizophrenia patients. , 2007, The American journal of psychiatry.

[2]  Pietro Perona,et al.  Sleep spindle detection: crowdsourcing and evaluating performance of experts, non-experts, and automated methods , 2014, Nature Methods.

[3]  Laura B Ray,et al.  Validating an automated sleep spindle detection algorithm using an individualized approach , 2010, Journal of sleep research.

[4]  M. Bertini,et al.  Topographical distribution of spindles: variations between and within nrem sleep cycles. , 2000, Sleep research online : SRO.

[5]  Lucas C. Parra,et al.  Subject position affects EEG magnitudes , 2013, NeuroImage.

[6]  T. Dutoit,et al.  Automatic Sleep Spindle Detection in Patients with Sleep Disorders , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[7]  János Körmendi,et al.  The individual adjustment method of sleep spindle analysis: Methodological improvements and roots in the fingerprint paradigm , 2009, Journal of Neuroscience Methods.

[8]  A Kumar,et al.  Human and automatic validation of a phase-locked loop spindle detection system. , 1980, Electroencephalography and clinical neurophysiology.

[9]  M. Ptito,et al.  Topographical distribution of spindles and K-complexes in normal subjects. , 1997, Sleep.

[10]  Laurent Mottron,et al.  Atypical sleep architecture and the autism phenotype. , 2005, Brain : a journal of neurology.

[11]  K. Crowley,et al.  The effects of normal aging on sleep spindle and K-complex production , 2002, Clinical Neurophysiology.

[12]  Mark C. Jones.  PRINCIPLES AND PRACTICE OF SLEEP MEDICINE , 1990 .

[13]  Jean-François Gagnon,et al.  Sleep spindles in Parkinson's disease may predict the development of dementia , 2015, Neurobiology of Aging.

[14]  Fabrizio Vecchio,et al.  An electroencephalographic fingerprint of human sleep , 2005, NeuroImage.

[15]  Nurettin Acir,et al.  Automatic recognition of sleep spindles in EEG by using artificial neural networks , 2004, Expert Syst. Appl..

[16]  Pietro Cortelli,et al.  Familial and sporadic fatal insomnia , 2003, The Lancet Neurology.

[17]  J. Born,et al.  The memory function of sleep , 2010, Nature Reviews Neuroscience.

[18]  G Tononi,et al.  Topographic and sex-related differences in sleep spindles in major depressive disorder: a high-density EEG investigation. , 2013, Journal of affective disorders.

[19]  Joseph E. Burns,et al.  Note: This Copy Is for Your Personal Non-commercial Use Only. to Order Presentation-ready Copies for Distribution to Your Colleagues or Clients, Contact Us at Www.rsna.org/rsnarights. Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for Ct Colonography 1 , 2022 .

[20]  Leslie G. Ungerleider,et al.  Motor sequence learning increases sleep spindles and fast frequencies in post-training sleep. , 2008, Sleep.

[21]  Manuel Schabus,et al.  Hemodynamic cerebral correlates of sleep spindles during human non-rapid eye movement sleep , 2007, Proceedings of the National Academy of Sciences.

[22]  P. Montagna.  Fatal familial insomnia: a model disease in sleep physiopathology. , 2005, Sleep medicine reviews.

[23]  A. Chesson,et al.  The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology, and Techinical Specifications , 2007 .

[24]  J. Gaillard,et al.  Spindle density in sleep of normal subjects. , 1981, Sleep.

[25]  Marcello Massimini,et al.  Reduced sleep spindle activity in schizophrenia patients , 2007 .

[26]  K. Blinowska,et al.  High resolution study of sleep spindles , 1999, Clinical Neurophysiology.

[27]  Adrián Pótári,et al.  Sleep Spindles and Intelligence: Evidence for a Sexual Dimorphism , 2014, The Journal of Neuroscience.

[28]  Max Hirshkowitz,et al.  Normal human sleep: an overview. , 2004, The Medical clinics of North America.

[29]  Antti Saastamoinen,et al.  Development and comparison of four sleep spindle detection methods , 2007, Artif. Intell. Medicine.

[30]  P. Anderer,et al.  Topographic distribution of sleep spindles in young healthy subjects , 1997, Journal of sleep research.

[31]  Rhodri Cusack,et al.  Expert and crowd-sourced validation of an individualized sleep spindle detection method employing complex demodulation and individualized normalization , 2015, Front. Hum. Neurosci..

[32]  J. Born,et al.  Learning-Dependent Increases in Sleep Spindle Density , 2002, The Journal of Neuroscience.

[33]  B Saletu,et al.  Automatic Sleep-Spindle Detection Procedure: Aspects of Reliability and Validity , 1994, Clinical EEG.

[34]  Stefanie Nowak,et al.  How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation , 2010, MIR '10.

[35]  Alexander E. Hramov,et al.  Sleep spindles and spike–wave discharges in EEG: Their generic features, similarities and distinctions disclosed with Fourier transform and continuous wavelet analysis , 2009, Journal of Neuroscience Methods.

[36]  Carlyle T. Smith,et al.  The function of the sleep spindle: A physiological index of intelligence and a mechanism for sleep-dependent memory consolidation , 2011, Neuroscience & Biobehavioral Reviews.

[37]  J. Born,et al.  Grouping of Spindle Activity during Slow Oscillations in Human Non-Rapid Eye Movement Sleep , 2002, The Journal of Neuroscience.

[38]  Thierry Dutoit,et al.  Automatic sleep spindles detection — Overview and development of a standard proposal assessment method , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[39]  Z. Clemens,et al.  Prediction of general mental ability based on neural oscillation measures of sleep , 2005, Journal of sleep research.

[40]  A. Chesson,et al.  The American Academy of Sleep Medicine (AASM) Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications , 2007 .

[41]  Johannes Hewig,et al.  Skull thickness and magnitude of EEG alpha activity , 2008, Clinical Neurophysiology.

[42]  Nikolaos K. Uzunoglu,et al.  Sleep spindle detection using artificial neural networks trained with filtered time-domain EEG: A feasibility study , 2005, Comput. Methods Programs Biomed..

[43]  Ann K. Shinn,et al.  Reduced Sleep Spindles and Spindle Coherence in Schizophrenia: Mechanisms of Impaired Memory Consolidation? , 2012, Biological Psychiatry.

[44]  Poul Jennum,et al.  Inter-expert and intra-expert reliability in sleep spindle scoring , 2015, Clinical Neurophysiology.

[45]  Timothy M. Kowalewski,et al.  Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performance. , 2014, The Journal of surgical research.

[46]  J. Montplaisir,et al.  Sleep spindle characteristics in healthy subjects of different age groups , 2001, Clinical Neurophysiology.

[47]  A. Goldbarg,et al.  I. Normal Subjects , 2015 .

[48]  Tunde Peto,et al.  Crowdsourcing as a Screening Tool to Detect Clinical Features of Glaucomatous Optic Neuropathy from Digital Photography , 2015, PloS one.

[49]  P. Achermann,et al.  Spindle frequency activity in the sleep EEG: individual differences and topographic distribution. , 1997, Electroencephalography and clinical neurophysiology.

[50]  Poul Jennum,et al.  Decreased sleep spindle density in patients with idiopathic REM sleep behavior disorder and patients with Parkinson’s disease , 2014, Clinical Neurophysiology.

[51]  Julie Carrier,et al.  Topography of age-related changes in sleep spindles , 2013, Neurobiology of Aging.

[52]  Francis Eustache,et al.  Is there a link between sleep changes and memory in Alzheimer's disease? , 2008, Neuroreport.

[53]  C. Smith,et al.  Sleep spindles and learning potential. , 2007, Behavioral neuroscience.

[54]  M. Walker.  The Role of Sleep in Cognition and Emotion , 2009, Annals of the New York Academy of Sciences.