Implementation Problems in Meta-Analysis

The paper probes implementation problems in meta-analysis by comparing six quantitative reviews of the research on the validity of student ratings of instructional effectiveness. The disparate conclusions of the reviews are explained by comparing the syntheses at each step in a meta-analysis: specifying inclusion criteria, locating studies, coding study features, calculating individual study outcomes, and data analysis. In addition, the resolution of implementation problems is explored, and several suggestions are offered for improving the practice of quantitative synthesis including the following: techniques for the specification of inclusion criteria, guidelines for the systematic summary of study features, a call for the analysis of statistical power and the reduction of Type II errors, and a discussion of the analytical problems posed by the presence of between- and within-study findings.

[1]  A. W. Bendig.  Student Achievement in Introductory Psychology and Student Ratings of the Competence and Empathy of Their Instructors , 1953 .

[2]  A. W. Bendig.  The Relation of Level of Course Achievement to Students' Instructor and Course Ratings in Introductory Psychology , 1953 .

[3]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[4]  Joseph E. Morsh,et al.  Student Achievement as a Measure of Instructor Effectiveness. , 1956 .

[5]  W. Borg,et al.  Comparison between a Performance Test and Criteria of Instructor Effectiveness , 1956 .

[6]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[7]  Royal P. Murdock.  The Effect of Student Ratings of Their Instructor on the Student's Achievement and Rating. Final Report. , 1969 .

[8]  W. R. Mann,et al.  Student Ratings of Teacher Effectiveness: Validity Studies1 , 1971 .

[9]  B. Rodin,et al.  Student evaluations of teachers. , 1972, Science.

[10]  Peter W. Frey,et al.  Student Ratings of Teaching: Validity of Several Rating Factors , 1973, Science.

[11]  P K Gessner,et al.  Evaluation of instruction. , 1973, Science.

[12]  R. Turner,et al.  Relationships Between College Student Ratings of Instructors and Residual Learning. , 1973 .

[13]  D. H. Sorge,et al.  Verbal Behavior of College Instructors and Attendant Effect Upon Student Attitudes and Achievement. , 1973 .

[14]  J. Soper.  Soft Research on a Hard Subject: Student Evaluations Reconsidered , 1973 .

[15]  A. Sullivan,et al.  Validity of student evaluation of teaching and the characteristics of successful instructors. , 1974 .

[16]  Kenneth O. Doyle,et al.  Student Ratings as Criteria for Effective Teaching , 1974 .

[17]  Rebecca Bryson.  Teacher Evaluations and Student Learning: A Reexamination. , 1974 .

[18]  H. Marsh,et al.  Validity and usefulness of student evaluations of instructional quality. , 1975 .

[19]  F. Costin,et al.  The Student as Consumer of the Teaching Process1 , 1975 .

[20]  Peter W. Frey,et al.  Student Ratings of Instruction: Validation Research1 , 1975 .

[21]  P. W. Frey,et al.  Validity of Student Instructional Ratings: Does Timing Matter?. , 1976 .

[22]  Roger Gene Hoffman.  Student Descriptions of Instructional Characteristics as Relevant Indices of Teaching Effectiveness , 1976 .

[23]  J. Centra.  Student Ratings of Instruction and Their Relationship to Student Learning , 1976 .

[24]  G. T. Endo,et al.  A Validation Study of Course Evaluation Ratings. , 1976 .

[25]  A Comparison of the Criterion Validity of Two Types of Student Response Inventories for Appraising Instruction. , 1976 .

[26]  Albert B. Smith,et al.  A study of the validity of four types of student ratings of college teaching assessed on a criterion of student achievement gains , 1976 .

[27]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[28]  N. R. Ellis,et al.  Evaluating the Teaching of Introductory Psychology , 1977 .

[29]  G. Glass.  9: Integrating Findings: The Meta-Analysis of Research , 1977 .

[30]  H. Eysenck.  An exercise in mega-silliness. , 1978 .

[31]  J. Palmer,et al.  Leniency, Learning, and Evaluations. , 1978 .

[32]  W. McKeachie,et al.  A Small Study Assessing Teacher Effectiveness: Does Learning Last?. , 1978 .

[33]  K. Doyle,et al.  Student, peer, and self evaluations of college instructors , 1978 .

[34]  R. G. Hoffman,et al.  Variables Affecting University Student Ratings of Instructor Behavior1 , 1978 .

[35]  W. D. Morgan,et al.  An Educational Production Function Approach to Teaching Effectiveness and Evaluation , 1978 .

[36]  R. Rosenthal.  Combining results of independent studies. , 1978 .

[37]  F. Costin.  Do Student Ratings of College Teachers Predict Student Achievement? , 1978 .

[38]  K. Doyle,et al.  Validity and Generalizability of Student Ratings From Between-Classes and Within-Class Data , 1979 .

[39]  Jere Brophy,et al.  Teacher Behavior and Student Learning. , 1979 .

[40]  C. I. Chase,et al.  Validity of Student Ratings of Faculty. Indiana Studies in Higher Education, Number Forty. , 1979 .

[41]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[42]  B. Bolton,et al.  Ratings of Instruction, Examination Performance, and Subsequent Enrollment in Psychology Courses , 1979 .

[43]  Larry A. Braskamp,et al.  Student Ratings and Instructor Self-ratings and Their Relationship to Student Achievement , 1979 .

[44]  Gregg B. Jackson,et al.  Methods for Integrative Reviews , 1980 .

[45]  C. Orpen.  Student Evaluation of Lecturers as an Indicator of Instructional Quality: A Validity Study , 1980 .

[46]  Herbert W. Marsh,et al.  Validity of Students' Evaluations of Teaching Effectiveness: Cognitive and Affective Criteria. , 1980 .

[47]  G. Glass,et al.  Meta-analysis in social research , 1981 .

[48]  P. A. Cohen.  Student Ratings of Instruction and Student Achievement: A Meta-analysis of Multisection Validity Studies , 1981 .

[49]  James A. Neal,et al.  A Selective Review of the Validity of Student Ratings of Teaching , 1982 .

[50]  W. Hays,et al.  Statistics (3rd ed.). , 1982 .

[51]  P. A. Cohen.  Validity of Student Ratings in Psychology Courses: A Research Synthesis , 1982 .

[52]  Gregg B. Jackson,et al.  Meta-Analysis: Cumulating Research Findings Across Studies , 1982 .

[53]  Philip C. Abrami,et al.  Educational Seduction , 1982 .

[54]  J. A. Neal,et al.  The Validity and Accuracy of Student Ratings of Instruction: A Reply to Peter A. Cohen , 1983 .

[55]  Peter A. Cohen,et al.  Comment on a Selective Review of the Validity of Student Ratings of Teaching , 1983 .

[56]  James A. Kulik,et al.  Effects of Coaching Programs on Achievement Test Performance , 1983 .

[57]  F. Bryant,et al.  Methodological issues in the meta‐analysis of quasi‐experiments , 1984 .

[58]  James A. Kulik,et al.  Meta-Analysis in Education: A Reply to Slavin , 1984 .

[59]  R. Rosenthal.  Meta-analytic procedures for social research , 1984 .

[60]  L. McCallum.  A meta-analysis of course evaluation data and its use in the tenure decision , 1984 .

[61]  Robert E. Slavin,et al.  Meta-Analysis in Education: How Has It Been Used? , 1984 .

[62]  Donald B. Rubin,et al.  Meta-Analytic Procedures for Combining Studies With Multiple Effect Sizes , 1986 .

[63]  Robert L. Bangert-Drowns,et al.  Review of developments in meta-analytic method. , 1986 .

[64]  Robert E. Slavin,et al.  Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional Reviews , 1986 .

[65]  Carl J. Huberty,et al.  On Statistical Testing , 1987 .

[66]  Robert E. Slavin,et al.  Best-Evidence Synthesis: Why Less Is More , 1987 .

[67]  David B. Pillemer,et al.  Summing Up: The Science of Reviewing Research , 1984 .

[68]  Betsy Jane Becker,et al.  Modeling multivariate effect sizes. , 1988 .

[69]  S. Chow.  Significance Test or Effect Size ? , 2022 .