Expert Judgment Quantification

Expert judgments are used when there are no objective data is available. It is critical to solicit these judgments accurately for decision makers. This chapter reviews methods and issues around the expert judgment quantification.

[1]  Dundar F. Kocaoglu,et al.  A participative approach to program evaluation , 1983, IEEE Transactions on Engineering Management.

[2]  A. K. Chakravarti,et al.  Modified Delphi Methodology for Technology Forecasting Case Study of Electronics and Information Technology in India , 1998 .

[3]  Daniel Weisser,et al.  Costing electricity supply scenarios: A case study of promoting renewable energy technologies on Rodriguez, Mauritius , 2004 .

[4]  L. Suganthi,et al.  Renewable energy in India — a modelling study for 2020–2021 , 2000 .

[5]  D. Altman,et al.  Statistics notes: Cronbach's alpha , 1997 .

[6]  Muhammad Amer,et al.  Development of fuzzy cognitive map (FCM) based scenarios , 2011, 2011 Proceedings of PICMET '11: Technology Management in the Energy Smart World (PICMET).

[7]  A. Carr,et al.  Primary total hip replacement surgery: a systematic review of outcomes and modelling of cost-effectiveness associated with different prostheses. , 1998, Health technology assessment.

[8]  Stephen M. Millett,et al.  A manager's guide to technology forecasting and strategy analysis methods , 1991 .

[9]  Tugrul U. Daim,et al.  Technology assessment for clean energy technologies: The case of the Pacific Northwest , 2009 .

[10]  Martin Steinert,et al.  A dissensus based online Delphi approach: An explorative research tool , 2009 .

[11]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[12]  Vivek Utgikar,et al.  Energy forecasting: Predictions, reality and analysis of causes of error , 2006 .

[13]  J. Landeta.  Current validity of the Delphi method in social sciences , 2006 .

[14]  F. Hasson,et al.  A critical review of the Delphi technique as a research methodology for nursing. , 2001, International journal of nursing studies.

[15]  Tim C. E. Engels,et al.  Enhancing a Delphi study on family-focused prevention , 2007 .

[16]  Jiangjiang Wang,et al.  Review on multi-criteria decision analysis aid in sustainable energy decision-making , 2009 .

[17]  B. Brown,et al.  THE DELPHI METHOD, IV: EFFECT OF PERCENTILE FEEDBACK AND FEED-IN OF RELEVANT FACTS, , 1970 .

[18]  W. Huss,et al.  Scenario planning—What style should you use? , 1987 .

[19]  Wayne K. Talley,et al.  Stability and agreement criteria for the termination of Delphi studies , 1979 .

[20]  Hans-Horst Schröder,et al.  Integrating market and technological knowledge in the fuzzy front end: an FCM-based action support system , 2003, Int. J. Technol. Manag..

[21]  Robert E. Linneman,et al.  The use of multiple scenarios by U.S. industrial companies , 1979 .

[22]  Vincent‐Wayne Mitchell,et al.  The delphi technique: an exposition and application , 1991 .

[23]  Nan Zhou,et al.  The reality and future scenarios of commercial building energy consumption in China , 2008 .

[24]  Pisek Gerdsri,et al.  A systematic approach to developing national technology policy and strategy for emerging technologies: A case study of nanotechnology for Thailand's agriculture industry , 2009, PICMET '09 - 2009 Portland International Conference on Management of Engineering & Technology.

[25]  N Jairath,et al.  The Delphi methodology (Part one): A useful administrative approach. , 1994, Canadian journal of nursing administration.

[26]  Joseph F. Coates,et al.  In defense of Delphi: : A review of Delphi assessment, expert opinion, forecasting, and group process by H. Sackman☆ , 1975 .

[27]  E. Hiltunen.  Scenarios: Process and Outcome , 2009 .

[28]  T. Daim,et al.  Selection of Renewable Energy Technologies for a Developing County: A Case of Pakistan , 2011 .

[29]  James J. Winebrake,et al.  The future of hydrogen fueling systems for transportation: An application of perspective-based scenario analysis using the analytic hierarchy process , 2003 .

[30]  Pedro Pérez-Higueras,et al.  Proposal for a combined methodology for renewable energy planning. Application to a Spanish region , 2009 .

[31]  Reza Moinpour,et al.  Managerial Judgment in Marketing: The Concept of Expertise , 1983 .

[32]  R. R. Clark.  Validation and legitimation of an analytic hierarchy approach to integrated resource planning for electric utilities , 1997, IECEC-97 Proceedings of the Thirty-Second Intersociety Energy Conversion Engineering Conference (Cat. No.97CH6203).

[33]  Thomas L. Saaty,et al.  Multicriteria Decision Making: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation , 1990 .

[34]  Wayne K. Talley,et al.  Individual stability in Delphi studies , 1980 .

[35]  Rod Taber,et al.  Quantization effects on the equilibrium behavior of combined fuzzy cognitive maps , 2007, Int. J. Intell. Syst..

[36]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[37]  Patrick Ronde,et al.  Delphi analysis of national specificities in selected innovative areas in Germany and France , 2003 .

[38]  Theodore J. Gordon,et al.  RT Delphi: An efficient, “round-less” almost real time Delphi method ☆ , 2006 .

[39]  Rod Taber,et al.  Knowledge processing with Fuzzy Cognitive Maps , 1991 .

[40]  A. V. D. Ven,et al.  Group Techniques for Program Planning , 1975 .

[41]  D. Spiegelhalter,et al.  Consensus development methods, and their use in clinical guideline development. , 1998, Health technology assessment.

[42]  Marcel R. Kastein,et al.  Delphi, the issue of reliability , 1993 .

[43]  Kerstin Cuhls,et al.  Outlook for Japanese and German Future Technology , 1994 .

[44]  Thomas L. Saaty,et al.  Decision-making with the AHP: Why is the principal eigenvector necessary , 2003, Eur. J. Oper. Res..

[45]  Jeryl L. Mumpower,et al.  Expert Judgement and Expert Disagreement , 1996 .

[46]  Melih Soner Celiktas,et al.  From potential forecast to foresight of Turkey's renewable energy with Delphi approach , 2010 .

[47]  R H Naomi,et al.  MANAGING MODERATOR STRESS: TAKE A DEEP BREATH. YOU CAN DO THIS , 2009 .

[48]  Harold Sackman,et al.  Delphi critique : expert opinion, forecasting, and group process , 1975 .

[49]  Earl R. Babbie,et al.  The practice of social research , 1969 .

[50]  Pasi Rikkonen,et al.  Future prospects of alternative agro-based bioenergy use in Finland—Constructing scenarios with quantitative and qualitative Delphi data , 2009 .

[51]  Glenn Regehr,et al.  Delphi as a method to establish consensus for diagnostic criteria. , 2003, Journal of clinical epidemiology.

[52]  T. Daim,et al.  Technology and Science Policies in Transitional Economy: A Case of Turkey , 2012 .

[53]  R. Erffmeyer,et al.  The Delphi Technique: An Empirical Evaluation of the Optimal Number of Rounds , 1986 .

[54]  H. Kahn,et al.  The Year 2000: A Framework for Speculation on the Next Thirty-Three Years, , 1967 .

[55]  N. Gerdsri.  An analytical approach to building a technology development envelope (TDE) for roadmapping of emerging technologies , 2005, A Unifying Discipline for Melting the Boundaries Technology Management:.

[56]  Monica R Geist,et al.  Using the Delphi method to engage stakeholders: a comparison of two studies. , 2010, Evaluation and program planning.

[57]  Dong-Sheng Jeng,et al.  Application of fuzzy analytic hierarchy process to assess the potential of offshore wind energy in Taiwan , 2008 .

[58]  Wolf Fichtner,et al.  The development of the German energy market until 2030—A critical survey of selected scenarios , 2011 .

[59]  J. Shanteau.  The Psychology of Experts An Alternative View , 1992 .

[60]  Naim Afgan,et al.  MULTI-CRITERIA ASSESSMENT OF NEW AND RENEWABLE ENERGY POWER PLANTS , 2002 .

[61]  Suzanne D. Pawlowski,et al.  The Delphi method as a research tool: an example, design considerations and applications , 2004, Inf. Manag..

[62]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[63]  Kerstin Cuhls,et al.  Personal attitudes in the assessment of the future of science and technology: A factor analysis approach , 2001 .

[64]  Rafikul Islam.  Modification of the nominal Group Technique by using the Analytic Hierarchy Process , 2001 .

[65]  F. Elkarmi,et al.  Increasing the utilization of solar energy technologies (SET) in Jordan: Analytic hierarchy process , 1993 .

[66]  S. Dolinsek,et al.  Scenario planning for climate strategies development by integrating group Delphi, AHP and dynamic fuzzy cognitive maps , 2008, PICMET '08 - 2008 Portland International Conference on Management of Engineering & Technology.

[67]  S. Iniyan,et al.  The application of a Delphi technique in the linear programming optimization of future renewable energy options for India , 2003 .

[68]  H. Grupp,et al.  Technology foresight using a Delphi approach: a Japanese‐German co‐operation , 1994 .

[69]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[70]  Niki Frantzeskaki,et al.  Sustainable energy planning by using multi-criteria analysis application in the island of Crete , 2009 .

[71]  Murray Turoff,et al.  The design of a policy Delphi , 1970 .

[72]  Florentin Smarandache,et al.  FUZZY COGNITIVE MAPS AND NEUTROSOPHIC COGNITIVE MAPS , 2003, math/0311063.

[73]  S. Iniyan,et al.  An optimal renewable energy model for various end-uses , 2000 .

[74]  Priyantha D.C. Wijayatunga,et al.  Strategies to overcome barriers for cleaner generation technologies in small developing power systems: Sri Lanka case study , 2006 .

[75]  Ozcan Saritas,et al.  Systemic analysis of UK foresight results Joint application of integrated management model and roadmapping , 2004 .

[76]  Lutz E. Schlange.  Scenarios: The art of strategic conversation , 1997 .

[77]  Theodore T. Herbert,et al.  A COMPARISON OF DECISION QUALITY UNDER NOMINAL AND INTERACTING CONSENSUS GROUP FORMATS: THE CASE OF THE STRUCTURED PROBLEM* , 1979 .

[78]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[79]  George Wright,et al.  The origins and evolution of scenario techniques in long range business planning , 2005 .

[80]  Antonie J. Jetter,et al.  Educating the guess: strategies, concepts and tools for the fuzzy front end of product development , 2003, PICMET '03: Portland International Conference on Management of Engineering and Technology Technology Management for Reshaping the World, 2003..

[81]  Haydar Aras,et al.  Multi-criteria selection for a wind observation station location using analytic hierarchy process , 2004 .

[82]  Ian Miles,et al.  Practical Guide to Regional Foresight in the United Kingdom , 2002 .

[83]  Uma G. Gupta,et al.  Theory and applications of the Delphi technique: A bibliography (1975–1994) , 1996 .

[84]  Petros A. Pilavachi,et al.  Multicriteria evaluation of power plants impact on the living standard using the analytic hierarchy process , 2008 .

[85]  Peter A. Shulman,et al.  Analysis of US Energy Scenarios , 2003 .

[86]  Martin Wietschel,et al.  Development of European hydrogen infrastructure scenarios--CO2 reduction potential and infrastructure investment , 2006 .

[87]  K. Czaplicka-Kolarz,et al.  Technology foresight for a vision of energy sector development in Poland till 2030. Delphi survey as an element of technology foresighting , 2009 .

[88]  David H. Gustafson,et al.  Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes , 1976 .

[89]  G. Finnveden,et al.  Scenario types and techniques: Towards a user's guide , 2006 .

[90]  K. Hill,et al.  The methodological worth of the Delphi forecasting technique , 1975 .

[91]  Michael Morrison,et al.  El modelo exacto del alto ejecutivo moderno , 2007 .

[92]  John Rohrbaugh,et al.  Improving the quality of group judgment: Social judgment analysis and the Delphi technique. , 1979 .

[93]  Muhammad Amer,et al.  A review of scenario planning , 2013 .

[94]  K. A. Ericsson,et al.  Toward a general theory of expertise : prospects and limits , 1991 .

[95]  Bent Sørensen,et al.  Hydrogen as an energy carrier: Scenarios for future use of hydrogen in the Danish energy system , 2004 .

[96]  Andrew B. Hargadon,et al.  Brainstorming groups in context: Effectiveness in a product design firm , 1996 .

[97]  Celeste Varum,et al.  Directions in scenario planning literature – A review of the past decades , 2010 .

[98]  Wolf D. Rauch,et al.  The decision Delphi , 1979 .

[99]  Daniela Fogli,et al.  Benchmark exercise on expert judgment techniques in PSA Level 2 , 2001 .

[100]  Nie Rui,et al.  Scenario Analysis of China's Primary Energy Demand and CO2 Emissions Based on IPAT Model , 2011 .

[101]  F. Gracceva,et al.  Hydrogen perspectives in Italy: Analysis of possible deployment scenarios , 2008 .

[102]  Antonio J. Conejo,et al.  A methodology to generate statistically dependent wind speed scenarios , 2010 .

[103]  Karen L. McGraw,et al.  Knowledge Acquisition: Principles and Guidelines , 1989 .

[104]  Satu Pätäri,et al.  Industry- and company-level factors influencing the development of the forest energy business — insights from a Delphi Study , 2010 .

[105]  Kerstin Cuhls,et al.  Outlook for Japanese and German Future Technology: Comparing Technology Forecast Surveys , 1994 .

[106]  Jane M. Booker,et al.  Eliciting and analyzing expert judgement - a practical guide , 2001, ASA-SIAM series on statistics and applied probability.

[107]  W. E. Riggs,et al.  The Delphi technique: An experimental evaluation , 1983 .

[108]  R. Balasubramanian,et al.  Analytical search of problems and prospects of power sector through Delphi study: case study of Kerala State, India , 2003 .

[109]  Dundar F. Kocaoglu,et al.  Applying the Analytic Hierarchy Process (AHP) to build a strategic framework for technology roadmapping , 2007, Math. Comput. Model..

[110]  Ramakrishnan Ramanathan,et al.  Multicriteria Methods for Technology Foresight , 2003 .

[111]  Jon Landeta,et al.  Results of a Delphi survey in drawing up the input–output tables for Catalonia , 2008 .

[112]  Juri Pill,et al.  The Delphi method: Substance, context, a critique and an annotated bibliography , 1971 .

[113]  Saul I. Gass,et al.  The Analytic Hierarchy Process - An Exposition , 2001, Oper. Res..

[114]  M. M. Kablan,et al.  Decision support for energy conservation promotion:: an analytic hierarchy process approach , 2004 .

[115]  Fang-Ming Hsu,et al.  Renewable energy technology portfolio planning with scenario analysis: A case study for Taiwan , 2009 .

[116]  Olaf Helmer-Hirschberg,et al.  Improving the Reliability of Estimates Obtained from a Consensus of Experts , 1964 .

[117]  M. K. Ahsan,et al.  Monitoring healthcare performance by analytic hierarchy process: a developing‐country perspective , 2004 .

[118]  Bilal Akash,et al.  Multi-criteria selection of electric power plants using analytical hierarchy process , 1999 .

[119]  L. A. Goodman,et al.  Measures of association for cross classifications , 1979 .

[120]  Ralph L. Keeney,et al.  On the uses of expert judgment on complex technical problems , 1989 .

[121]  Gento Mogi,et al.  The competitiveness of Korea as a developer of hydrogen energy technology: The AHP approach , 2008 .

[122]  Murat Bengisu,et al.  Forecasting emerging technologies with the aid of science and technology databases , 2006 .

[123]  L. A. Goodman,et al.  Measures of Association for Cross Classifications. II: Further Discussion and References , 1959 .

[124]  Alan L. Porter,et al.  Forecasting and Management of Technology , 1991 .

[125]  Yicheng Peng,et al.  Portland General Electric (PGE): Clean Power Generation Wind Project in Biglow Canyon & Boardman Coal Plant , 2007, PICMET '07 - 2007 Portland International Conference on Management of Engineering & Technology.

[126]  Cojazzi Giacomo giovanni mariano,et al.  Benchmark Exercise on Expert Judgement Techniques in PSA Level 2. Extended Final Report. , 2000 .

[127]  Joseph Paul Martino,et al.  Technological Forecasting for Decisionmaking , 1975 .

[128]  Tugrul U. Daim,et al.  Application of technology roadmaps for renewable energy sector , 2010 .

[129]  Robert E. Linneman,et al.  The use of multiple scenarios by U.S. industrial companies: A comparison study, 1977–1981☆ , 1983 .

[130]  Taeyoung Shin,et al.  Using Delphi for a Long-Range Technology Forecasting, and Assessing Directions of Future R&D Activities The Korean Exercise , 1998 .

[131]  Vilja Varho,et al.  Wind power in Finland up to the year 2025 , 2005 .

[132]  F. Woudenberg.  An Evaluation of Delphi , 1991 .

[133]  Gento Mogi,et al.  A fuzzy analytic hierarchy process approach for assessing national competitiveness in the hydrogen technology sector , 2008 .

[134]  Paul J. H. Schoemaker,et al.  Integrating scenarios into strategic planning at Royal Dutch/Shell , 1992 .

[135]  C A Lindeman,et al.  DELPHI SURVEY OF PRIORITIES IN CLINICAL NURSING RESEARCH , 1975, Nursing research.

[136]  George Wright,et al.  Delphi: A reevaluation of research and theory , 1991 .

[137]  Philippe Durance,et al.  Scenario building: Uses and abuses , 2010 .

[138]  E. Paul Torrance,et al.  Group Decision-Making and Disagreement , 1957 .

[139]  Joseph P. Martino,et al.  A review of selected recent advances in technological forecasting , 2003 .

[140]  Guido Reger,et al.  Advantages and Disadvantages of Scenario Approaches for Strategic Foresight , 2005 .

[141]  C. Powell.  The Delphi technique: myths and realities. , 2003, Journal of advanced nursing.

[142]  P. A. Pilavachi,et al.  Evaluation of hydrogen production methods using the Analytic Hierarchy Process , 2009 .

[143]  John Rohrbaugh,et al.  Improving the quality of group judgment: Social judgment analysis and the nominal group technique , 1981 .

[144]  Marvin A. Jolson,et al.  http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless , 2022 .

[145]  Amy H. I. Lee,et al.  Multi-criteria decision making on strategic selection of wind farms , 2009 .

[146]  Busuttil Antoine,et al.  Energy scenarios for Malta , 2008 .

[147]  Jamal O. Jaber,et al.  Evaluation of conventional and renewable energy sources for space heating in the household sector , 2008 .

[148]  G. Tichy.  Over-Optimism Among Experts in Assessment and Foresight , 2002 .

[149]  Jane M. Booker,et al.  6. Selecting and Motivating the Experts , 2001 .

[150]  Fredrik Lagergren,et al.  Getting back to scenario planning: Strategic action in the future of energy Europe , 2011, 2011 8th International Conference on the European Energy Market (EEM).

[151]  Petros A. Pilavachi,et al.  Technological, economic and sustainability evaluation of power plants using the Analytic Hierarchy Process , 2009 .

[152]  Murray Turoff,et al.  Delphi: A brief look backward and forward , 2011 .

[153]  Antonie J. Jetter,et al.  Elicitation — Extracting Knowledge from Experts , 2006 .

[154]  Thomas J. Chermack,et al.  Exploring the relationship between scenario planning and perceptions of learning organization characteristics , 2006 .

[155]  R. Ono,et al.  Assessing the validity of the Delphi technique , 1994 .

[156]  Simone J. van Zolingen,et al.  Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education , 2003 .

[157]  S. Lee,et al.  A study on making a long-term improvement in the national energy efficiency and GHG control plans by the AHP approach , 2007 .

[158]  Reinhard Madlener,et al.  New ways for the integrated appraisal of national energy scenarios: The case of renewable energy use in Austria , 2007 .

[159]  Peter Schwartz,et al.  The Art of the Long View: Planning for the Future in an Uncertain World , 1996 .