Conducting Online Expert panels: a feasibility and experimental replicability study

BackgroundThis paper has two goals. First, we explore the feasibility of conducting online expert panels to facilitate consensus finding among a large number of geographically distributed stakeholders. Second, we test the replicability of panel findings across four panels of different size.MethodWe engaged 119 panelists in an iterative process to identify definitional features of Continuous Quality Improvement (CQI). We conducted four parallel online panels of different size through three one-week phases by using the RAND's ExpertLens process. In Phase I, participants rated potentially definitional CQI features. In Phase II, they discussed rating results online, using asynchronous, anonymous discussion boards. In Phase III, panelists re-rated Phase I features and reported on their experiences as participants.Results66% of invited experts participated in all three phases. 62% of Phase I participants contributed to Phase II discussions and 87% of them completed Phase III. Panel disagreement, measured by the mean absolute deviation from the median (MAD-M), decreased after group feedback and discussion in 36 out of 43 judgments about CQI features. Agreement between the four panels after Phase III was fair (four-way kappa = 0.36); they agreed on the status of five out of eleven CQI features. Results of the post-completion survey suggest that participants were generally satisfied with the online process. Compared to participants in smaller panels, those in larger panels were more likely to agree that they had debated each others' view points.ConclusionIt is feasible to conduct online expert panels intended to facilitate consensus finding among geographically distributed participants. The online approach may be practical for engaging large and diverse groups of stakeholders around a range of health services research topics and can help conduct multiple parallel panels to test for the reproducibility of panel conclusions.

[1]  F. Hasson,et al.  A critical review of the Delphi technique as a research methodology for nursing. , 2001, International journal of nursing studies.

[2]  A. J. Conger.  Integration and generalization of kappas for multiple raters. , 1980 .

[3]  M Eccles,et al.  The potential influence of small group processes on guideline development. , 2001, Journal of evaluation in clinical practice.

[4]  Susan G. Straus,et al.  ExpertLens: A System for Eliciting Opinions from a Large Pool of Non-Collocated Experts with Diverse Knowledge , 2011 .

[5]  P. Shekelle,et al.  Identifying continuous quality improvement publications: what makes an improvement intervention ‘CQI’? , 2011, BMJ quality & safety.

[6]  Selma Vonderwell,et al.  An examination of asynchronous communication experiences and perspectives of students in an online course: a case study , 2003, Internet High. Educ..

[7]  Gilad Ravid,et al.  Information overload and the message dynamics of online interaction spaces: a theoretical model and empirical exploration , 2004, IEEE Engineering Management Review.

[8]  Lynne Wainfan,et al.  Challenges in Virtual Collaboration: Videoconferencing Audioconferencing and Computer--Mediated Communications , 2005 .

[9]  A. Rogers,et al.  How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings? , 2004, Quality and Safety in Health Care.

[10]  R. Brook,et al.  Consensus methods: characteristics and guidelines for use. , 1984, American journal of public health.

[11]  Cheryl Bagley Thompson,et al.  Comparison of mailed vs. Internet applications of the Delphi technique in clinical informatics research , 2000, AMIA.

[12]  P. Shekelle,et al.  Identifying quality improvement intervention evaluations: is consensus achievable? , 2010, Quality and Safety in Health Care.

[13]  Kathryn H. Bowles,et al.  Expert Consensus for Discharge Referral Decisions Using Online Delphi , 2003, AMIA.

[14]  Sara B. Kiesler,et al.  The Equalization Phenomenon: Status Effects in Computer-Mediated and Face-to-Face Decision-Making Groups , 1991, Hum. Comput. Interact..

[15]  Barry Smith,et al.  Proceedings of the AMIA Symposium , 2005 .

[16]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[17]  J. Dent,et al.  The Montreal Definition and Classification of Gastroesophageal Reflux Disease: A Global Evidence-Based Consensus , 2006, The American Journal of Gastroenterology.

[18]  Aileen Clarke,et al.  Developing a quality criteria framework for patient decision aids: online international Delphi consensus process , 2006, BMJ : British Medical Journal.

[19]  D. Spiegelhalter,et al.  Consensus development methods, and their use in clinical guideline development. , 1998, Health technology assessment.

[20]  Rupert Brown,et al.  Group Processes: Dynamics Within and Between Groups , 1988 .

[21]  H. McKenna.  The Delphi technique: a worthwhile research approach for nursing? , 1994, Journal of advanced nursing.

[22]  J P Kahan,et al.  The reproducibility of a method to identify the overuse and underuse of medical procedures. , 1998, The New England journal of medicine.

[23]  M. Adler,et al.  Gazing into the oracle : the Delphi method and its application to social policy and public health , 1996 .

[24]  Rosalind Raine,et al.  Developing clinical guidelines: a challenge to current methods , 2005, BMJ : British Medical Journal.

[25]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[26]  N. Black,et al.  Consensus Development Methods: A Review of Best Practice in Creating Clinical Guidelines , 1999, Journal of health services research & policy.

[27]  N. Black,et al.  An experimental study of determinants of the extent of disagreement within clinical guideline development groups , 2005, Quality and Safety in Health Care.

[28]  D. Hunter,et al.  Qualitative Research: Consensus methods for medical and health services research , 1995 .

[29]  S Hempel,et al.  Finding order in heterogeneity: types of quality-improvement intervention publications , 2008, Quality & Safety in Health Care.