Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities

The policy Delphi is a method that uses iterative stages of data collection to reveal positions on an issue within a panel of people with relevant knowledge. Policy Delphi surveys have become popular in a variety of disciplines since the method was first proposed in this journal in 1970. In this paper, we benchmark the state-of-the-art in policy Delphi methods, focusing on strengths and limitations, and on innovative ways of addressing key shortcomings. We report findings from a systematic review of 63 empirical studies conducted between 1971 and the end of 2014 that used the policy Delphi method. We found little consistency in how studies have been designed and executed. The inherent flexibility of the method is a strength, but a lack of consistency in how it is used undermines the ability of analysts to generate accessible insights. Specifically, our analysis reveals limited use of validity and reliability tests, a blurring of conventional and policy Delphi rationales, diverse data collection and analysis techniques, and mixed quality when reporting the approach, format, and results for individual studies. Indeed, potential new users of the method will struggle to understand what a policy Delphi survey actually is. We conclude with advice for addressing key shortcomings in current policy Delphi practice.

[1]  R. D. Needham,et al.  THE POLICY DELPHI: PURPOSE, STRUCTURE, AND APPLICATION , 1990 .

[2]  G. Hickey,et al.  Improving the social robustness of research networks for sustainable natural resource management: Results of a Delphi study in Canada , 2012 .

[3]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[4]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[5]  Nathan W. Moon,et al.  Wireless Technologies and Accessibility for People With Disabilities: Findings From a Policy Research Instrument , 2008, Assistive technology : the official journal of RESNA.

[6]  Barry Wellar,et al.  Using the Delphi Technique in Normative Planning Research: Methodological Design Considerations , 2008 .

[7]  D. Scott,et al.  Changing Climate, Challenging Choices: Identifying and Evaluating Climate Change Adaptation Options for Protected Areas Management in Ontario, Canada , 2011, Environmental management.

[8]  Pasi Rikkonen,et al.  Future prospects of alternative agro-based bioenergy use in Finland—Constructing scenarios with quantitative and qualitative Delphi data , 2009 .

[9]  B. Andrews.  Curriculum Renewal Through Policy Development In Arts Education , 2004 .

[10]  Kathy K. Franklin,et al.  Idea Generation and Exploration: Benefits and Limitations of the Policy Delphi Research Method , 2006 .

[11]  J. Smit,et al.  A Policy Delphi study in the socialist Middle East , 1990 .

[12]  J. Landeta.  Current validity of the Delphi method in social sciences , 2006 .

[13]  K. Hill,et al.  The methodological worth of the Delphi forecasting technique , 1975 .

[14]  Christopher Groves,et al.  Four scenarios for nanotechnologies in the UK, 2011–2020 , 2013, Technol. Anal. Strateg. Manag..

[15]  Muhammad Imran Yousuf,et al.  Using Experts` Opinions Through Delphi Technique , 2007 .

[16]  Monica G. Turner,et al.  Diversity in Current Ecological Thinking: Implications for Environmental Management , 2009, Environmental management.

[17]  Erik Taal,et al.  Giving rheumatology patients online home access to their electronic medical record (EMR): advantages, drawbacks and preconditions according to care providers , 2012, Rheumatology International.

[18]  Simon Hakim,et al.  Using the Delphi process to analyze social policy implementation: A post hoc case from vocational rehabilitation , 1993 .

[19]  S. Gennaro.  Where Should You Share , 2015 .

[20]  Petri Tapio,et al.  Disaggregative Policy Delphi: Using cluster analysis as a tool for systematic scenario formation , 2003 .

[21]  A. Strauss,et al.  Grounded theory , 2017 .

[22]  Heiko A. von der Gracht,et al.  Surface- and deep-level diversity in panel selection — Exploring diversity effects on response behaviour in foresight , 2014 .

[23]  O. L. Ervin,et al.  The Policy Delphi and Public Involvement Programs , 1982 .

[24]  Gordon M. Hickey,et al.  A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi , 2011 .

[25]  Ian Miles,et al.  Foresight tools for participative policy-making in inter-governmental processes in developing countries: Lessons learned from the eLAC Policy Priorities Delphi , 2009 .

[26]  M K Rayens,et al.  Kentucky legislators' views on tobacco policy. , 1999, American journal of preventive medicine.

[27]  Seth Bullock,et al.  Complexity at the social science interface , 2014, Complex..

[28]  Yi-Fang Lee,et al.  Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. , 2008, Evaluation and program planning.

[29]  P. Swain,et al.  Androgenic anabolic steroid policy and high school sports: results from a policy Delphi study , 2014 .

[30]  R. N. Nwanko,et al.  The Context of Television Broadcasting in Cameroon , 1990 .

[31]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[32]  Chu Hyun Kim,et al.  Scientific Framework for Research on Disaster and Mass Casualty Incident in Korea: Building Consensus Using Delphi Method , 2013, Journal of Korean medical science.

[33]  J. Kola,et al.  Policies and Measures for Multifunctional Agriculture: Experts' Insight * , 2005 .

[34]  Simone J. van Zolingen,et al.  Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education , 2003 .

[35]  B. Berg.  Qualitative Research Methods for the Social Sciences , 1989 .

[36]  F. Hasson,et al.  Enhancing rigour in the Delphi technique research , 2011 .

[37]  Cary A. Brown,et al.  Evaluating Students' Use of Web-Based Communication during Practice Placements , 2010 .

[38]  Alexandros Paraskevas,et al.  Beyond consensus: an alternative use of Delphi enquiry in hospitality research , 2012 .

[39]  Mary Kay Rayens,et al.  Building Consensus Using the Policy Delphi Method , 2000 .

[40]  Ralph S. Brower,et al.  Three traditions of network research: What the public management research agenda can learn from other research communities , 2004 .

[41]  Murray Turoff,et al.  The design of a policy Delphi , 1970 .

[42]  Jerry B. Schneider,et al.  The policy Delphi: A regional planning application , 1971 .

[43]  P. Rikkonen.  Scenarios for future agriculture in Finland : a Delphi study among agri-food sector stakeholders , 2008 .

[44]  S. More,et al.  Policy Delphi with vignette methodology as a tool to evaluate the perception of equine welfare. , 2009, Veterinary journal.

[45]  Michael R. Reich,et al.  Introducing a first AIDS vaccine in Switzerland: A Policy Delphi analysis , 2005, Sozial- und Präventivmedizin.

[47]  Murray Turoff,et al.  Delphi: A brief look backward and forward , 2011 .

[48]  S. More,et al.  Evaluation of current equine welfare issues in Ireland: causes, desirability, feasibility and means of raising standards. , 2010, Equine veterinary journal.

[49]  M. Sandelowski,et al.  Qualitative Metasynthesis: Reflections on Methodological Orientation and Ideological Agenda , 2004, Qualitative health research.

[50]  Rob C. de Loë,et al.  Exploring complex policy questions using the policy Delphi: A multi-round, interactive survey method , 1995 .

[51]  W. Gartner.  What are we talking about when we talk about entrepreneurship , 1990 .

[52]  S. Padmanabhan,et al.  A national energy-conservation policy Delphi: report of the findings , 1987 .

[53]  James. Longhurst,et al.  Exploring a city's potential low carbon futures using Delphi methods: some preliminary findings , 2012 .

[54]  P. Meskell,et al.  Insights into the use and complexities of the Policy Delphi technique. , 2014, Nurse researcher.

[55]  Adam Michael Edwards,et al.  Urban security in Europe: Translating a concept in public criminology , 2013 .

[56]  R. A. Manley,et al.  The Policy Delphi: A Method for Identifying Intended and Unintended Consequences of Educational Policy , 2013 .

[57]  Alan Kelly,et al.  Equity in resource allocation in the Irish health service. A policy Delphi study. , 2004, Health policy.

[58]  Kathy K. Franklin,et al.  Influence of Web-based Distance Education on the Academic Department Chair Role , 2006, J. Educ. Technol. Soc..

[59]  R. D. Needham,et al.  Moving best practice forward: delphi characteristics, advantages, potential problems, and solutions. , 2009 .