STUDY OBJECTIVES
The American Academy of Sleep Medicine (AASM) Inter-scorer Reliability program provides a unique opportunity to compare a large number of scorers with varied levels of experience to determine agreement in the scoring of respiratory events. The objective of this paper is to examine areas of disagreement to inform future revisions of the AASM Manual for the Scoring of Sleep and Associated Events.
METHODS
The sample included 15 monthly records, 200 epochs each. The number of scorers increased steadily during the period of data collection, reaching more than 3,600 scorers by the final record. Scorers were asked to identify whether an obstructive, mixed, or central apnea; a hypopnea; or no event was seen in each of the 200 epochs. The "correct" respiratory event score was defined as the score endorsed by the most scorers. Percentage agreement with the majority score was determined for each epoch and the mean agreement determined.
RESULTS
The overall agreement for scoring of respiratory events was 93.9% (κ = 0.92). There was very high agreement on epochs without respiratory events (97.4%), and the majority score for most of the epochs (87.8%) was no event. For the 364 epochs scored as having a respiratory event, overall agreement that some type of respiratory event occurred was 88.4% (κ = 0.77). The agreement for epochs scored as obstructive apnea by the majority was 77.1% (κ = 0.71), and the most common disagreement was hypopnea rather than obstructive apnea (14.4%). The agreement for hypopnea was 65.4% (κ = 0.57), with 16.4% scoring no event and 14.8% scoring obstructive apnea. The agreement for central apnea was 52.4% (κ = 0.41). A single epoch was scored as a mixed apnea by a plurality of scorers.
CONCLUSIONS
The study demonstrated excellent agreement among a large sample of scorers for epochs with no respiratory events. Agreement for some type of event was good, but disagreements in scoring of apnea vs. hypopnea and type of apnea were common. A limitation of the analysis is that most of the records had normal breathing. A review of controversial events yielded no consistent bias that might be resolved by a change of scoring rules.
[1]
A. Chesson,et al.
The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology, and Techinical Specifications
,
2007
.
[2]
David Gozal,et al.
The scoring of respiratory events in sleep: reliability and validity.
,
2007,
Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine.
[3]
D. Rapoport,et al.
Choice of oximeter affects apnea-hypopnea index.
,
2005,
Chest.
[4]
R. Rosenberg,et al.
The American Academy of Sleep Medicine inter-scorer reliability program: sleep stage scoring.
,
2013,
Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine.
[5]
Koby Todros,et al.
Assessment of automated scoring of polysomnographic recordings in a population with suspected sleep-disordered breathing.
,
2004,
Sleep.
[6]
A. Chesson,et al.
The American Academy of Sleep Medicine (AASM) Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications
,
2007
.
[7]
S. Quan,et al.
Rules for scoring respiratory events in sleep: update of the 2007 AASM Manual for the Scoring of Sleep and Associated Events. Deliberations of the Sleep Apnea Definitions Task Force of the American Academy of Sleep Medicine.
,
2012,
Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine.
[8]
Thomas Penzel,et al.
Agreement in the scoring of respiratory events and sleep among international sleep centers.
,
2013,
Sleep.
[9]
D. Greenblatt,et al.
The International Classification of Sleep Disorders
,
1992
.
[10]
S. Redline,et al.
Reliability of scoring respiratory disturbance indices and sleep staging.
,
1998,
Sleep.