A Quality Scorecard for the Administration of Online Education Programs: A Delphi Study

As the demands for public accountability increase for the higher education, institutions are seeking methods for continuous improvement in order to demonstrate quality within programs and processes, including those provided through online education. A six round Delphi study was undertaken with 43 seasoned administrators of online education programs who agreed upon 70 quality indicators that administrators should examine within their programs to evaluate quality. A method for scoring was also developed. The original set of quality indicators from the Institute for Higher Education Policy study, Quality on the Line: Benchmarks for Success in Internet-Based Distance Education (2000) were used as a starting point and determined still valid in 2010, with modifications. The study resulted in a quality scorecard for the administration of online education programs.

[1]  Roger Lindsay,et al.  Total quality management in education , 2006, Br. J. Educ. Technol..

[2]  C. Webb,et al.  The Delphi technique: a methodological discussion. , 1994, Journal of advanced nursing.

[3]  Carl B. Montano,et al.  Total quality management in higher education , 1999 .

[4]  Badrul H. Khan,et al.  A framework for web-based learning , 2000 .

[5]  Richard L. Webb.  The online game modding community: A connectivist instructional design for online learning? , 2009 .

[6]  C. Powell.  The Delphi technique: myths and realities. , 2003, Journal of advanced nursing.

[7]  J. Jackson Barnette,et al.  Delphi Methodology: An Empirical Investigation. , 1978 .

[8]  Amy S. C. Leh,et al.  Striving for Quality Control in Distance Education , 2002 .

[9]  K. Meyer.  Quality in Distance Education: Focus on On-Line Learning. ASHE-ERIC Higher Education Report. Jossey-Bass Higher and Adult Education Series. , 2002 .

[10]  Charles D. Dziuban,et al.  Using Quality Assurance Strategies for Online Programs , 2002 .

[11]  Suzanne C. Flores.  A Delphi method case study of how one university's exemplary instructors are providing quality learning experiences in online education , 2007 .

[12]  N. Dalkey,et al.  An Experimental Application of the Delphi Method to the Use of Experts , 1963 .

[13]  Brent D. Ruben,et al.  Evaluating the impact of organizational self‐assessment in higher education , 2007 .

[14]  Alan T. Seagren.  The Baldrige Review: Using the Baldrige Criteria for Review of Business Operations. , 1995 .

[15]  Demetrius Karathanos,et al.  Applying the Balanced Scorecard to Education , 2005 .

[16]  Ernest T. Pascarella,et al.  What Does Institutional Selectivity Tell Us About Educational Quality? , 2004 .

[17]  C Duffield,et al.  The Delphi technique. , 1988, The Australian journal of advanced nursing : a quarterly publication of the Royal Australian Nursing Federation.

[18]  Robert C. Judd.  Delphi Decision Methods in Higher Education Administration. , 1972 .

[19]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[20]  J. Daniel,et al.  Breaking Higher Education's Iron Triangle: Access, Cost, and Quality , 2009 .

[21]  Jon F. Wergin.  Higher Education: Waking up to the Importance of Accreditation-- Ramifications of Proposed 2005 Extension of the Higher Education Act. , 2005 .

[22]  Kathy K. Franklin,et al.  Idea Generation and Exploration: Benefits and Limitations of the Policy Delphi Research Method , 2006 .

[23]  A. Benson.  Dimensions of Quality in Online Degree Programs , 2003 .

[24]  Paul Joseph Green,et al.  The content of a college-level outdoor leadership course for land-based outdoor pursuits in the pacific northwest : a Delphi consensus , 1981 .

[25]  Allan R. Cohen,et al.  Transformational Change at Babson College Notes From the Firing Line , 2003 .

[26]  Allan R. Cohen,et al.  Major Change at Babson College: Curricular and Administrative, Planned and Otherwise , 2005 .

[27]  R. Kaplan,et al.  The Balanced Scorecard: Translating Strategy into Action , 1996 .

[28]  Karina Lovell,et al.  How expert are the experts? An exploration of the concept of 'expert' within Delphi panel techniques. , 2006, Nurse researcher.

[29]  F. Hasson,et al.  Research guidelines for the Delphi survey technique. , 2000, Journal of advanced nursing.

[30]  Ronald A. Phipps,et al.  Assuring quality in distance learning a preliminary review , 1998 .

[31]  I. E. Allen,et al.  Staying the course: online education in the United States, 2008 , 2008 .

[32]  Namir Shammas,et al.  Using Delphi , 1995 .

[33]  Linda C. Smith,et al.  Online program Assessment: A Case Study of the University of Illinois at Urbana-Champaign Experience , 2005 .

[34]  Richard N. Foster,et al.  The Delphi technique: Survey and comment: Essentials for corporate use , 1971 .

[35]  Gregory J. Skulmoski,et al.  Journal of Information Technology Education the Delphi Method for Graduate Research , 2022 .

[36]  Marilyn M. Helms,et al.  A Retention Assessment Process: Utilizing Total Quality Management Principles and Focus Groups. , 2005 .

[37]  J. Day,et al.  A Generic Toolkit for the Successful Management of Delphi Studies , 2005 .

[38]  Richard G. Fuller.  Faculty practices in successful asynchronous online distance education: A study within health education programs , 2006 .

[39]  T. Sumsion,et al.  The Delphi Technique: An Adaptive Research Tool , 1998 .

[40]  A. V. D. Ven,et al.  Group Techniques for Program Planning , 1975 .

[41]  Kenneth M. Zeichner.  Myths and Realities , 1980 .

[42]  Stacey Clawson.  DOES QUALITY MATTER? MEASURING WHETHER ONLINE COURSE QUALITY STANDARDS ARE PREDICTIVE OF STUDENT SATISFACTION IN HIGHER EDUCATION , 2007 .

[43]  Cam Scholey,et al.  Hands-on Scorecarding in the Higher Education Sector. , 2006 .

[44]  A. W. Bates,et al.  Effective Teaching with Technology in Higher Education: Foundations for Success , 2003 .

[45]  Kit. O'Toole.  Toward a tri-level model and comprehensive theory for online writing laboratory (OWL) research design. , 2006 .

[46]  A. V. D. Ven,et al.  The Effectiveness of Nominal, Delphi, and Interacting Group Decision Making Processes , 1974 .

[47]  J. Eaton,et al.  Institutions, Accreditors, and the Federal Government: Redefining Their "Appropriate Relationship" , 2007 .

[48]  Elizabeth Reed Osika.  The Concentric Support Model: A Model for the Planning and Evaluation of Distance Learning Programs. , 2006 .

[49]  Jack D. Dilbeck.  Perceptions of academic administrators towards quality indicators in Internet based distance education , 2008 .

[50]  Badrul Khan,et al.  Managing E-Learning Strategies: Design, Delivery, Implementation and Evaluation , 2005 .

[51]  Alicia Maria,et al.  Pedagogy in online graduate business learning environments , 2007 .

[52]  Katrina A. Meyer.  The Impact of Competition on Program Quality. , 2004 .

[53]  George Wright,et al.  Expert Opinions in Forecasting: The Role of the Delphi Technique , 2001 .

[54]  B. Brown,et al.  THE DELPHI METHOD, II: STRUCTURE OF EXPERIMENTS, , 1969 .

[55]  Chia-Chien Hsu,et al.  Minimizing Non-Response in The Delphi Process: How to Respond to Non-Response , 2007 .

[56]  P. Harroff,et al.  Dimensions of Program Quality in Web-Based Adult Education , 2006 .

[57]  William J. Nunez,et al.  Strategic Planning Synergy. , 2001 .

[58]  Robert C. Wolpert,et al.  A Review of the , 1985 .

[59]  MaryBeth Walpole,et al.  The Promise of Baldrige for K-12 Education. Act Policy Report. , 2002 .

[60]  W. Rossiter,et al.  Progress report , 1954, Research newsletter. College of General Practitioners.

[61]  Zhiming Xue.  Effective practices of Continuous Quality Improvement in United States colleges and universities , 1998 .

[62]  A. Stella,et al.  Quality assurance in distance education: The challenges to be addressed , 2004 .

[63]  André L. Delbecq,et al.  A Group Process Model for Problem Identification and Program Planning , 1971 .

[64]  D. Casey.  A Journey to Legitimacy: The Historical Development of Distance Education through Technology , 2008 .

[65]  G. Moses,et al.  Effective Teaching with Technology , 2006, Proceedings. Frontiers in Education. 36th Annual Conference.

[66]  Roy A. Bauer,et al.  Application of the Baldrige Model for Innovation in Higher Education , 2007 .

[67]  J. Murry,et al.  Delphi: A Versatile Methodology for Conducting Qualitative Research , 2017 .

[68]  Susan M. Fritz,et al.  A quality assessment using the Baldrige criteria: Non-academic service units in a large university , 1993 .

[69]  Donna Redman Nasmyth.  A Delphi study of online graduate courses in the United States , 2007 .

[70]  Ike Iheanacho,et al.  Myths and realities , 2011 .

[71]  Richard Pollard,et al.  Using the Delphi Method for E-Research , 2008 .

[72]  Arthur Wizenried,et al.  Delphi Studies: The Value of Expert Opinion Bridging the Gap - Data to Knowledge , 2021, IASL Annual Conference Proceedings.

[73]  Marilyn Lockhart,et al.  An assessment model and methods for evaluating distance education programmes , 2002 .

[74]  Mark Clayton.  Delphi: a technique to harness expert opinion for critical decision‐making tasks in education , 1997 .

[75]  Shuki Dror,et al.  The Balanced Scorecard versus quality award models as strategic frameworks , 2008 .

[76]  Farhad Saba,et al.  Evaluating Distance Education Programs. , 2000 .

[77]  S. Thangaratinam,et al.  The Delphi technique , 2005 .

[78]  Mark H. Rossman,et al.  Needed Functions, Knowledge, and Skills for Hospital Education Directors in the 1980's: A Delphi Study. , 1982 .

[79]  F. Hasson,et al.  Consulting the oracle: ten lessons from using the Delphi technique in nursing research. , 2006, Journal of advanced nursing.

[80]  T. Waits.  Distance Education at Degree-Granting Postsecondary Institutions: 2000-2001. E.D. Tabs. , 2003 .

[81]  Dc Washington,et al.  Council for Higher Education Accreditation. , 2002 .

[82]  Gary R. Pike,et al.  Measuring Quality: A Comparison of U.S. News Rankings and NSSE Benchmarks , 2004 .

[83]  Mike Moore,et al.  Distance Education: A Systems View , 1995 .

[84]  K. Cameron,et al.  ORGANIZATIONAL QUALITY: An Examination of the Malcolm Baldrige National Quality Framework , 1998 .

[85]  Daniel Taylor,et al.  Continuous-improvement strategies in higher education: a progress report , 2003 .

[86]  R. Ackoff.  Continuous improvement-I , 1991 .

[87]  John W. Creswell,et al.  Research Design: Qualitative and Quantitative Approaches , 1997 .

[88]  Phillip Choi,et al.  A Continuous Improvement Process , 2005 .

[89]  Juha Kettunen,et al.  Strategic planning and quality assurance in the Bologna Process , 2007 .

[90]  Kaye Shelton,et al.  An Administrator's Guide to Online Education , 2005 .

[91]  S. Dorman,et al.  A Primer on Quality Indicators of Distance Education , 2009, Health promotion practice.

[92]  Wallace K. Pond,et al.  Distributed Education in the 21st Century: Implications for Quality Assurance. , 2002 .

[93]  Brian Satterlee.  Continuous Improvement and Quality: Implications for Higher Education , 1996 .

[94]  Catherine C. Giapponi,et al.  Strategy revitalization in academe: a balanced scorecard approach , 2008 .

[95]  Mary W. Hendrix.  Quality assurance in online doctoral programs and courses: A Delphi study to determine specific indicators , 2005 .

[96]  Charles A. Schwartz,et al.  Managing technological change: strategies for college and university leaders , 2000 .

[97]  Cathy Cavanaugh.  Distance education quality: success factors for resources, practices and results , 2002 .

[98]  Jeffrey W. Alstete.  College Accreditation: Managing Internal Revitalization and Public Respect , 2007 .

[99]  Etter Zuniga,et al.  Quality Assurance in Distance Education. , 1996 .

[100]  MURRAY TUROFF,et al.  Philosophical and Methodological Foundations of Delphi , 2002 .

[101]  Vincent‐Wayne Mitchell,et al.  The delphi technique: an exposition and application , 1991 .

[102]  Chia-Chien Hsu,et al.  The Delphi Technique: Making Sense of Consensus , 2007 .

[103]  Muhammad Imran Yousuf,et al.  Using Experts` Opinions Through Delphi Technique , 2007 .

[104]  Brent D. Ruben,et al.  Developing More Adaptive, Innovative, and Interactive Organizations. , 2002 .