A Pilot Systematic Review and Meta-Analysis on the Effectiveness of Problem Based Learning.

This paper reports on the development and piloting of a systematic review and meta analysis of research on the effectiveness of problem based learning (PBL). The systematic review protocol was pilot tested with a sample of studies cited as providing "evidence" about the effectiveness of PBL. From the 5 studies mentioned in the sample of reviews, 91 citations were identified, and 15 of these were adjudged to meet the review inclusion criteria. Of the 15, only 12 reported extractable data. All of these studies used PBL in higher education programs for health professional education. The review has established that the limited high quality evidence available from existing reviews does not provide robust evidence about the effectiveness of different kinds of PBL in different contexts with different student groups. It is apparent that there is scope for a systematic review of PBL that is specific in terms of the intervention being evaluated, comprehensive in terms of strategy used, and methodologically rigorous. One appendix contains the coding sheet used, and the other is a bibliography of reviewed papers. The second part of the paper is the review protocol. (Contains 1 figure, 10 tables, and 91 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the original document. PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY M. Newman TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) 1 U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) dThis document has been reproduced as received from the person or organization originating it. 0 Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. A pilot systematic review and meta-analysis on the effectiveness of Problem Based Learning Mark Newman Middlesex University On behalf of the Campbell Collaboration Systematic Review Group on the effectiveness of Problem Based Learning MIDDLESEX UNIVERSITY BEST COPY AVAILABLE

[1]  William Shadish David Myers.  Campbell Collaboration Research Design Policy Brief , 2004 .

[2]  Ann Oakley,et al.  Research evidence, knowledge management and educational practice: early lessons from a systematic approach , 2003 .

[3]  J. Sterne,et al.  How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. , 2003, Health technology assessment.

[4]  D. Gough,et al.  Systematic Research Synthesis to Inform Policy, Practice and Democratic Debate , 2002, Social Policy and Society.

[5]  J. Verbeek,et al.  Problem based learning in continuing medical education: a review of controlled evaluation studies , 2002, BMJ : British Medical Journal.

[6]  R. Boruch,et al.  The Campbell Collaboration , 2001, BMJ : British Medical Journal.

[7]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Journal of the American Podiatric Medical Association.

[8]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Annals of internal medicine.

[9]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials , 2001, The Lancet.

[10]  Mark Petticrew,et al.  Systematic reviews from astronomy to zoology: myths and misconceptions , 2001, BMJ : British Medical Journal.

[11]  R. Rosenthal,et al.  Meta-analysis: recent developments in quantitative methods for literature reviews. , 2001, Annual review of psychology.

[12]  Sally Glen,et al.  Problem-based Learning in Nursing: A New Model for a New Context? , 2000 .

[13]  J. Colliver.  Effectiveness of Problem‐based Learning Curricula: Research and Theory , 2000, Academic medicine : journal of the Association of American Medical Colleges.

[14]  C. Ham.  Health services research methods: a guide to best practice , 2000 .

[15]  F. Wolf.  Lessons to be learned from evidence-based medicine: practice and promise of evidence-based medicine and evidence-based education , 2000 .

[16]  Iain Burns,et al.  A New Model for a New Context , 2000 .

[17]  David Gijbels,et al.  Does problem-based learning educate problem-solvers? A meta-analysis on the effects of problem-based learning , 2000 .

[18]  Kay Wilkie,et al.  The Nature of Problem-based Learning , 2000 .

[19]  Cindy E. Hmelo-Silver,et al.  Problem-based Learning: A Research Perspective on Learning Interactions , 2000 .

[20]  H. Barrows.  Problem-Based Learning Applied to Medical Education , 2000 .

[21]  Ronald L. VanSickle,et al.  Problem-Solving Instruction and Students' Acquisition, Retention and Structuring of Economics Knowledge. , 2000 .

[22]  E. Benjamin,et al.  Implementing practice guidelines for diabetes care using problem-based learning. A prospective controlled trial using firm systems. , 1999, Diabetes care.

[23]  K. Patel.  Physicians for the 21St Century , 1999, Evaluation & the health professions.

[24]  Finch.  The effect of problem‐based learning on the academic performance of students studying podiatric medicine in Ontario , 1999, Medical education.

[25]  J. Kaczorowski,et al.  Problem-based small-group learning via the Internet among community family physicians: a randomized controlled trial. , 1999, M.D. computing : computers in medical practice.

[26]  W. Antepohl,et al.  Problem‐based learning versus lecture‐based learning in a course of basic pharmacology: a controlled, randomized study , 1999, Medical education.

[27]  G. Maudsley,et al.  Do we all mean the same thing by "problem-based learning"? A review of the concepts and a formulation of the ground rules. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[28]  Thomas A. Schwandt.  The Interpretive Review of Educational Matters: Is There Any Other Kind? , 1998 .

[29]  D. Langille,et al.  Comparison of problem‐based learning and lecture format in continuing medical education on headache diagnosis and management , 1998, Medical education.

[30]  R. Robbs,et al.  A Comparison of Problem-Based Learning and Standard Curriculum Students: Three Years of Retrospective Data , 1998 .

[31]  Cindy E. Hmelo,et al.  Problem-Based Learning: Effects on the Early Acquisition of Cognitive Skill in Medicine , 1998 .

[32]  George Davey Smith,et al.  meta-analysis bias in location and selection of studies , 1998 .

[33]  Albert J. J. A. Scherpbier,et al.  An Analysis of Progress Test Results of PBL and Non-PBL Students. , 1998 .

[34]  Lambert Schuwirth,et al.  An approach to the assessment of medical problem solving : Computerised Case-based Testing , 1998 .

[35]  M. Egger,et al.  Bias in location and selection of studies. , 1998, BMJ.

[36]  John D. Bransford,et al.  A theory-driven approach to assessing the cognitive effects of PBL , 1997 .

[37]  Amy V. Blue,et al.  Problem-based learning and performance-based testing: effective alternatives for undergraduate surgical education and assessment of student performance , 1997 .

[38]  L. Schuwirth,et al.  An international comparison of knowledge levels of medical students: the Maastricht Progress Test , 1996, Medical education.

[39]  H P Boshuizen,et al.  The development of diagnostic competence: comparison of a problem‐based, an integrated, and a conventional medical curriculum , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[40]  D. Parish,et al.  Problem-based advance cardiac life support. , 1996, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[41]  B F Richards,et al.  Ratings of students' performances in a third‐year internal medicine clerkship: a comparison between problem‐based and lecture‐based curricula , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[42]  J. A. Moran,et al.  Measuring the effectiveness of a pilot continuing medical education program. , 1996, Canadian family physician Medecin de famille canadien.

[43]  C. V. D. Vleuten,et al.  The assessment of professional competence: Developments, research and practical implications , 1996 .

[44]  C. V. D. van der Vleuten,et al.  The assessment of professional competence: Developments, research and practical implications. , 1996, Advances in health sciences education : theory and practice.

[45]  A D Oxman,et al.  Changing physician performance. A systematic review of the effect of continuing medical education strategies. , 1995, JAMA.

[46]  Chris McManus.  New Pathways to Medical Education: Learning to Learn at Harvard Medical School , 1995 .

[47]  M. Clarke,et al.  Identifying relevant studies for systematic reviews , 1995, BMJ.

[48]  C. Borland,et al.  Effect Size , 2019, SAGE Research Methods Foundations.

[49]  G. Moore,et al.  The influence of the New Pathway curriculum on Harvard medical students , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[50]  K. Dickersin,et al.  Systematic Reviews: Identifying relevant studies for systematic reviews , 1994 .

[51]  J. Premi,et al.  Practice‐based small‐group CME , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[52]  R. Mitchell.  The development of the cognitive behavior survey to assess medical student learning , 1994 .

[53]  L. Berkson.  Problem‐based learning: have the expectations been met? , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[54]  H. Schmidt.  Foundations of problem‐based learning: some explanatory notes , 1993, Medical education.

[55]  S. Kalishman,et al.  Performances on the NBME I, II, and III by medical students in the problem‐based learning and conventional tracks at the University of New Mexico , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[56]  D. T. Vernon,et al.  Does problem‐based learning work? A meta‐analysis of evaluative research , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[57]  F. Wolf.  Problem‐based learning and meta‐analysis: can we see the forest through the trees? , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[58]  Dean T. Jamison,et al.  World development report 1993 : investing in health , 1993 .

[59]  R B Haynes,et al.  Effect of problem-based, self-directed undergraduate education on life-long learning. , 1993, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[60]  M. Albanese,et al.  Problem‐based Learning: A Review of Literature on Its Outcomes and Implementation Issues , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[61]  Increasing the relevance of education for health professionals. Report of a WHO Study Group on Problem-Solving Education for the Health Professions. , 1993, World Health Organization technical report series.

[62]  H. Schmidt,et al.  The psychological basis of problem‐based learning: a review of the evidence , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[63]  G. Pigeon,et al.  From traditional to problem‐based learning: a case report of complete curriculum reform , 1992, Medical education.

[64]  D. T. Vernon,et al.  Problem‐based learning in two behavioral sciences courses at the University of Missouri‐Columbia , 1992, Academic medicine : journal of the Association of American Medical Colleges.

[65]  G. E. Fryer,et al.  Student preference for a case-based vs. lecture instructional format. , 1991, Journal of dental education.

[66]  R. Bridgham,et al.  The effect of curriculum era on NBME Part I outcomes in a problem-based versus a traditional curriculum track. , 1991, Academic medicine : journal of the Association of American Medical Colleges.

[67]  S Tolnai,et al.  Continuing medical education and career choice among graduates of problem‐based and traditional curricula , 1991, Medical education.

[68]  V L Patel,et al.  Effects of conventional and problem‐based medical curricula on problem solving , 1991, Academic medicine : journal of the Association of American Medical Colleges.

[69]  P. Rangachari,et al.  Design of a problem-based undergraduate course in pharmacology: implications for the teaching of physiology. , 1991, The American journal of physiology.

[70]  R. Bone,et al.  An experiment in medical education. A critical analysis using traditional criteria. , 1991, JAMA.

[71]  G. Moore.  The effect of compulsory participation of medical students in problem‐based learning , 1991, Medical education.

[72]  D. Puett,et al.  The endocrine module: An integrated course for first‐year medical students combining lecture‐based and modified problem‐based curricula , 1991 .

[73]  M. al-Haddad,et al.  Problem-based curriculum: outcome evaluation. , 1991, Medical teacher.

[74]  Douglas G. Altman,et al.  Practical statistics for medical research , 1990 .

[75]  K. Glanz,et al.  Problem‐based learning: cognitive retention and cohort traits of randomly selected participants and decliners , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[76]  J. Maxwell,et al.  A study of non‐volunteer faculty in a problem‐based curriculum , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[77]  R. S. Donner,et al.  Problem-based learning: an assessment of its feasibility and cost. , 1990, Human pathology.

[78]  S. Kalishman,et al.  Residency performance of graduates from a problem‐based and a conventional curriculum , 1990, Medical education.

[79]  Pathology education in a problem‐based medical curriculum , 1990 .

[80]  A. Dietrich,et al.  Adapting problem-based learning to a traditional curriculum: teaching about prevention. , 1990, Family practice research journal.

[81]  J. Bulte,et al.  Effects of the vocational training of general practice consultation skills and medical performance , 1989, Medical education.

[82]  H. Walton,et al.  Essentials of problem‐based learning , 1989, Medical education.

[83]  V R Neufeld,et al.  The McMaster M.D. program: a case study of renewal in medical education , 1989, Academic medicine : journal of the Association of American Medical Colleges.

[84]  J. Samet,et al.  The New Mexico experiment: educational innovation and institutional change , 1989, Academic medicine : journal of the Association of American Medical Colleges.

[85]  New directions for medical education : problem-based learning and community-oriented medical education , 1989 .

[86]  P. Blumberg,et al.  A comparison of student satisfaction with their preclinical environment in a traditional and a problem based curriculum. , 1988, Research in medical education : proceedings of the ... annual Conference. Conference on Research in Medical Education.

[87]  J. Nolte,et al.  Shifting toward problem-based learning in a medical school neurobiology course. , 1988, Research in medical education : proceedings of the ... annual Conference. Conference on Research in Medical Education.

[88]  C. Goldsmith,et al.  Billing patterns of general practitioners and family physicians in Ontario: a comparison of graduates of McMaster Medical School with graduates of other Ontario medical schools. , 1988, Research in medical education : proceedings of the ... annual Conference. Conference on Research in Medical Education.

[89]  J. Olson,et al.  The McMaster philosophy: a student's perspective on implementation , 1987, Medical education.

[90]  K E Lewis,et al.  The problem-based learning approach in baccalaureate nursing education: how effective is it? , 1987, Nursing papers. Perspectives en nursing.

[91]  Problem‐based learning of psychopathology in a traditional curriculum using multiple conceptual models , 1987, Medical education.

[92]  D. Schoen.  Educating the reflective practitioner , 1987 .

[93]  H S Barrows,et al.  A taxonomy of problem‐based learning methods , 1986, Medical education.

[94]  D. Newble,et al.  The approaches to learning of students in a traditional and in an innovative problem‐based medical school , 1986, Medical education.

[95]  J. Haf,et al.  Effect of two preclinical curricula on NBME Part I examination performance. , 1986, Journal of medical education.

[96]  Kathleen Saunders,et al.  The Library in a Problem-Based Curriculum , 1985 .

[97]  B. Hamad.  Problem‐based education in Gezira, Sudan , 1985, Medical education.

[98]  Arthur Kaufman,et al.  Implementing problem-based medical education : lessons from successful innovations , 1985 .

[99]  C R Coles,et al.  Differences between conventional and problem‐based curricula in their students' approaches to studying , 1985, Medical education.

[100]  D Mårtenson,et al.  Medical chemistry: evaluation of active and problem‐oriented teaching methods , 1985, Medical education.

[101]  D. Newble,et al.  The learning style of medical students , 1985, Medical education.

[102]  H F Claessen,et al.  Recall of medical information by students and doctors , 1985, Medical education.

[103]  W. Hays.  Summing Up: The Science of Reviewing Research. , 1985 .

[104]  R. Mcauley,et al.  Faculty perceptions of the McMaster M.D. program. , 1984, Journal of medical education.

[105]  C E Engel,et al.  Student perceptions of the learning environment in a new medical school , 1984, Medical education.

[106]  B. Bloom.  The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring , 1984 .

[107]  David B. Pillemer,et al.  Summing Up: The Science of Reviewing Research , 1984 .

[108]  Physicians for the twenty-first century. Report of the Project Panel on the General Professional Education of the Physician and College Preparation for Medicine. , 1984, Journal of medical education.

[109]  S. Dubovsky,et al.  Evaluating medical school graduates of a reduced course program: a preliminary report. , 1984, Journal of medical education.

[110]  N. Vu,et al.  How medical students learn. , 1983, Journal of medical education.

[111]  C. Woodward,et al.  The content of the medical curriculum at McMaster University: graduates' evaluation of their preparation for postgraduate training , 1983, Medical education.

[112]  H. Schmidt.  Problem‐based learning: rationale and description , 1983, Medical education.

[113]  C. Woodward,et al.  Perspectives of graduates two or five years after graduation from a three-year medical school. , 1982, Journal of medical education.

[114]  C. Woodward,et al.  Unravelling the meaning of global comparative ratings of interns. , 1981, Annual Conference on Research in Medical Education. Conference on Research in Medical Education.

[115]  G. Colditz,et al.  The students' view of an innovative undergraduate medical course: the first year at the University of Newcastle, N.S.W. † , 1980, Medical education.

[116]  David L. Post,et al.  Medical Problem Solving: An Analysis of Clinical Reasoning , 1979 .

[117]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[118]  L. Vygotsky.  Mind in Society: The Development of Higher Psychological Processes: Harvard University Press , 1978 .

[119]  L. S. Vygotskiĭ,et al.  Mind in society : the development of higher psychological processes , 1978 .

[120]  H S Barrows,et al.  The portable patient problem pack: a problem-based learning unit. , 1977, Journal of medical education.

[121]  H. Morgan.  A Problem‐oriented independent studies programme in basic medical sciences. , 1977, Medical education.

[122]  J. K. Robertson.  An experiment in medical education. , 1954, The British journal of radiology.

[123]  J. Dewey.  Logic, the theory of inquiry , 1938 .