Inter-rater reliability of sleep cyclic alternating pattern (CAP) scoring and validation of a new computer-assisted CAP scoring method

OBJECTIVE To assess inter-rater reliability between different scorers, from different qualified sleep research groups, in scoring visually the Cyclic Alternating Pattern (CAP), to evaluate the performances of a new tool for the computer-assisted detection of CAP, and to compare its output with the data from the different scorers. METHODS CAP was scored in 11 normal sleep recordings by four different raters, coming from three sleep laboratories. CAP was also scored in the same recordings by means of a new computer-assisted method, implemented in the Hypnolab 1.2 (SWS Soft, Italy) software. Data analysis was performed according to the following steps: (a) the inter-rater reliability of CAP parameters between the four different scorers was carried out by means of the Kendall W coefficient of concordance; (b) the analysis of the agreement between the results of the visual and computer-assisted analysis of CAP parameters was also carried out by means of the Kendall W coefficient; (c) a 'consensus' scoring was obtained, for each recording, from the four scorings provided by the different raters, based on the score of the majority of scorers; (d) the degree of agreement between each scorer and the consensus score and between the computer-assisted analysis and the consensus score was quantified by means of the Cohen's k coefficient; (e) the differences between the number of false positive and false negative detections obtained in the visual and in the computer-assisted analysis were also evaluated by means of the non-parametric Wilcoxon test. RESULTS The inter-rater reliability of CAP parameters quantified by the Kendall W coefficient of concordance between the four different scorers was high for all the parameters considered and showed values above 0.9 for total CAP time, CAP time in sleep stage 2 and percentage of A phases in sequence; also CAP rate showed a high value (0.829). The most important global parameters of CAP, including total CAP rate and CAP time, scored by the computer-assisted analysis showed a significant concordance with those obtained by the raters. The agreement between the computer-assisted analysis and the consensus scoring for the assignment of the CAP A phase subtype was not distinguishable from that expected from a human scorer. However, the computer-assisted analysis provided a number of false positives and false negatives significantly higher than that of the visual scoring of CAP. CONCLUSIONS CAP scoring shows good inter-rater reliability and might be compared in different laboratories the results of which might also be pooled together; however, caution should always be taken because of the variability which can be expected in the classical sleep staging. The computer-assisted detection of CAP can be used with some supervision and correction in large studies when only general parameters such as CAP rate are considered; more editing is necessary for the correct use of the other results. SIGNIFICANCE This article describes the first attempt in the literature to evaluate in a detailed way the inter-rater reliability in scoring CAP parameters of normal sleep and the performances of a human-supervised computerized automatic detection system.

[1]  H. Kuwahara,et al.  Automatic real-time analysis of human sleep stages by an interval histogram method. , 1988, Electroencephalography and clinical neurophysiology.

[2]  Umberto Barcaro,et al.  An automatic method for the recognition and classification of the A-phases of the cyclic alternating pattern , 2002, Clinical Neurophysiology.

[3]  R. Thomas.  Cyclic alternating pattern and positive airway pressure titration. , 2002, Sleep medicine.

[4]  S Smirne,et al.  The macrostructure and microstructure of sleep in patients with autosomal dominant nocturnal frontal lobe epilepsy. , 2000, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[5]  L. Ferini-Strambi,et al.  Arousal Fluctuations in Non‐Rapid Eye Movement Parasomnias: The Role of Cyclic Alternating Pattern as a Measure of Sleep Instability , 1995, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[6]  A. C Rosa,et al.  Automatic detection of cyclic alternating pattern (CAP) sequences in sleep: preliminary results , 1999, Clinical Neurophysiology.

[7]  M. Terzano,et al.  Atlas, rules, and recording techniques for the scoring of cyclic alternating pattern (CAP) in human sleep. , 2002, Sleep medicine.

[8]  A Kumar,et al.  An automatic spindle analysis and detection system based on the evaluation of human ratings of the spindle quality. , 1979, Waking and sleeping.

[9]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[10]  M. Terzano,et al.  Cyclic alternating pattern and spectral analysis of heart rate variability during normal sleep , 2000, Journal of sleep research.

[11]  C. Guilleminault,et al.  EEG arousals: scoring rules and examples: a preliminary report from the Sleep Disorders Atlas Task Force of the American Sleep Disorders Association. , 1992, Sleep.

[12]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[13]  A. Muzet,et al.  Sleep stage scoring using the neural network model: comparison between visual and automatic analysis in normal subjects and patients. , 1996, Sleep.

[14]  M. Terzano,et al.  The cyclic alternating pattern as a physiologic component of normal NREM sleep. , 1985, Sleep.

[15]  S. Ancoli-Israel,et al.  Night-to-night arousal variability and interscorer reliability of arousal measurements. , 1999, Sleep.

[16]  Daniel J Buysse,et al.  Microstructure of sleep in depressed patients according to the cyclic alternating pattern. , 2003, Journal of affective disorders.

[17]  M. Scharf,et al.  Evaluation of sleep architecture and cyclic alternating pattern rates in depressed insomniac patients treated with nefazodone hydrochloride. , 1999, American journal of therapeutics.

[18]  M. Marciani,et al.  Baroreflex Buffering of Sympathetic Activation During Sleep: Evidence From Autonomic Assessment of Sleep Macroarchitecture and Microarchitecture , 2004, Hypertension.

[19]  P. Halász.  Hierarchy of micro-arousals and the microstructure of sleep , 1998, Neurophysiologie Clinique/Clinical Neurophysiology.

[20]  L Parrino,et al.  The cyclic alternating pattern sequences in the dynamic organization of sleep. , 1988, Electroencephalography and clinical neurophysiology.

[21]  R. M. Colognola,et al.  A simple electronic and computer system for automatic spindle detection , 1989, Neurophysiologie Clinique/Clinical Neurophysiology.

[22]  M. Bertini,et al.  The Cyclic Alternating Pattern Decreases as a Consequence of Total Sleep Deprivation and Correlates with EEG Arousals , 2002, Neuropsychobiology.

[23]  O. Bruni,et al.  Sleep cyclic alternating pattern in normal school-age children , 2002, Clinical Neurophysiology.

[24]  J. R. Smith,et al.  Automatic detection of the K-complex in sleep electroencephalograms. , 1970, IEEE transactions on bio-medical engineering.

[25]  L Parrino,et al.  Polysomnographic analysis of arousal responses in obstructive sleep apnea syndrome by means of the cyclic alternating pattern. , 1996, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[26]  Salvatore Smirne,et al.  The impact of cyclic alternating pattern on heart rate variability during sleep in healthy young adults , 2000, Clinical Neurophysiology.

[27]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[28]  A. Schlögl,et al.  Interrater reliability between scorers from eight European sleep laboratories in subjects with different sleep disorders , 2004, Journal of sleep research.

[29]  R. D. Joseph,et al.  Pattern recognition of EEG-EOG as a technique for all-night sleep stage scoring. , 1972, Electroencephalography and clinical neurophysiology.

[30]  W. Herrmann,et al.  The future of computer-assisted investigation of the polysomnogram: sleep microstructure. , 1996, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[31]  F Ferrillo,et al.  A method for the automatic detection of arousals during sleep. , 1999, Sleep.

[32]  Liborio Parrino,et al.  Clinical applications of cyclic alternating pattern , 1993, Physiology & Behavior.

[33]  S. Kubicki,et al.  Sleep EEG evaluation: a comparison of results obtained by visual scoring and automatic analysis with the Oxford sleep stager. , 1989, Sleep.

[34]  D. Rapoport,et al.  Interobserver agreement among sleep scorers from different centers in a large dataset. , 2000, Sleep.

[35]  M Kurachi,et al.  Agreement of Visual Scoring of Sleep Stages among Many Laboratories in Japan: Effect of a Supplementary Definition of Slow Wave on Scoring of Slow Wave Sleep , 1993, The Japanese journal of psychiatry and neurology.

[36]  A Värri,et al.  A simple format for exchange of digitized polygraphic recordings. , 1992, Electroencephalography and clinical neurophysiology.

[37]  N. Collop.  Scoring variability between polysomnography technologists in different sleep laboratories. , 2002, Sleep medicine.

[38]  J. Fleiss.  Statistical methods for rates and proportions , 1974 .

[39]  J P Macher,et al.  Neural network model: application to automatic analysis of human sleep. , 1993, Computers and biomedical research, an international journal.

[40]  M Hirshkowitz,et al.  Atlas, rules, and recording techniques for the scoring of cyclic alternating pattern (CAP) in human sleep. , 2001, Sleep medicine.

[41]  G. Jamal,et al.  Periodicity of a noninvasive measure of cardiac vagal tone during non-rapid eye movement sleep in non-sleep deprived and sleep-deprived normal subjects. , 1999, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[42]  A. Rechtschaffen,et al.  A manual of standardized terminology, technique and scoring system for sleep stages of human subjects , 1968 .

[43]  L Parrino,et al.  Relationship of slow and rapid EEG components of CAP to ASDA arousals in normal sleep. , 2001, Sleep.

[44]  S. Redline,et al.  Reliability of scoring respiratory disturbance indices and sleep staging. , 1998, Sleep.

[45]  R Ferri,et al.  Comparison between the results of an automatic and a visual scoring of sleep EEG recordings. , 1989, Sleep.

[46]  C J Griffiths,et al.  Interobserver variability in recognizing arousal in respiratory sleep disorders. , 1998, American journal of respiratory and critical care medicine.