Crowdsourcing as a Novel Method to Evaluate Aesthetic Outcomes of Treatment for Unilateral Cleft Lip

Background: Lack of convenient and reliable methods to grade aesthetic outcomes limits the ability to study results and optimize treatment of unilateral cleft lip. Crowdsourcing methods solicit contributions from a large group to achieve a greater task. The authors hypothesized that crowdsourcing could be used to reliably grade aesthetic outcomes of unilateral cleft lip. Methods: Fifty deidentified photographs of 8- to 10-year-old subjects (46 with unilateral cleft lip and four controls) were assembled. Outcomes were assessed using multiple pairwise comparisons that produced a rank order (Elo rank) of nasal appearance and, on a separate survey, by Asher-McDade ratings. Both surveys were repeated to assess reliability. A group of expert surgeons repeated the same tasks on a smaller subset of photographs. Results: The authors obtained 2500 and 1900 anonymous, layperson evaluations by means of crowdsourcing on each Elo rank and Asher-McDade survey, respectively. Elo rank and Asher-McDade scores were highly reproducible (correlation coefficients, 0.87 and 0.98), and crowd evaluations agreed with those by expert surgeons (0.980 and 0.96 for Elo rank and Asher-McDade score, respectively). Crowdsourcing surveys were completed within 9 hours, whereas the expert surgeons required 3 months. On further analysis of their cleft subject sample set, the authors found that greater initial cleft severity was associated with worse aesthetic outcome. Conclusions: Outcomes assessed by crowds were reliable and correlated well with expert assessments. Crowdsourcing allows acquisition of massive numbers of layperson assessments on an unprecedented scale, and is a convenient, rapid, and reliable means of assessing aesthetic outcome of treatment for unilateral cleft lip. CLINICAL QUESTION/LEVEL OF EVIDENCE: Diagnostic, IV.