Five Methodology Errors in Educational Research: The Pantheon of Statistical Significance and Other Faux Pas.

After presenting a general linear model as a framework for discussion, this paper reviews five methodology errors that occur in educational research: (1) the use of stepwise methods; (2) the failure to consider in result interpretation the context specificity of analytic weights (e.g., regression beta weights, factor pattern coefficients, discriminant function coefficients, canonical function coefficients) that are part of all parametric quantitative analyses; (3) the failure to interpret both weights and structure coefficients as part of result interpretation; (4) the failure to recognize that reliability is a characteristic of scores, and not of tests; and (5) the incorrect interpretation of statistical significance and the related failure to report and interpret the effect sizes present in all quantitative analysis. In several cases small heuristic discriminant analysis data sets are presented to make the discussion of each of these five methodology errors more concrete and accessible. Four appendixes contain computer programs for some of the analyses. (Contains 19 tables, 1 figure, and 143 references.) (SLD) ******************************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. ********************************************************************************

[1]  Edward L. Vockell,et al.  Perceptions of Document Quality and Use by Educational Decision Makers and Researchers 1 , 1974 .

[2]  B. Thompson.  Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms , 1996 .

[3]  Monica J. Harris.  Significance Tests are Not Enough , 1991 .

[4]  Bruce Thompson,et al.  The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives , 1993 .

[5]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[6]  Patricia B. Elmore,et al.  Research Note: Statistical Methods Employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research From 1978 to 1987 , 1988 .

[7]  Xitao Fan,et al.  Canonical correlation analysis and structural equation modeling: What do they have in common? , 1997 .

[8]  L. Cronbach.  Beyond the Two Disciplines of Scientific Psychology. , 1975 .

[9]  R. Frick,et al.  The appropriate use of null hypothesis testing. , 1996 .

[10]  R J Harris,et al.  A Canonical Cautionary. , 1989, Multivariate Behavioral Research.

[11]  Bruce Thompson,et al.  EDITORIAL COMMENT: GUEST EDITORIAL MISUSE OF ANCOVA AND RELATED “STATISTICAL CONTROL” PROCEDURES , 1992 .

[12]  Bruce Thompson,et al.  Seminal Readings on Reliability and Validity: A "Hit Parade" Bibliography , 1996 .

[13]  J. Levin,et al.  Reflections on Statistical and Substantive Significance, with a Slice of Replication. , 1997 .

[14]  C. Fornell,et al.  Canonical Correlation Analysis As A Special Case Of A Structural Relations Model. , 1981, Multivariate behavioral research.

[15]  William L. Hays.  Statistics (3rd ed.). , 1990 .

[16]  Ulrich Rendtel,et al.  Editorial , 2014, Journal of basic microbiology.

[17]  Gene V. Glass,et al.  Policy for the Unpredictable (Uncertainty Research and Policy) , 1979 .

[18]  F. Borgen,et al.  Uses of discriminant analysis following MANOVA: Multivariate statistics for multivariate purposes. , 1978 .

[19]  T. Vacha-Haase,et al.  Reliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies , 1998 .

[20]  B. Thompson.  Two and One‐Half Decades of Leadership in Measurement and Evaluation , 1992 .

[21]  Susan R. Davis,et al.  Trends in Reporting Psychometric Properties of Scales Used in Counseling Psychology Research. , 1990 .

[22]  J. Hunter.  Needed: A Ban on the Significance Test , 1997 .

[23]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[24]  P. Meehl.  Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. , 1978 .

[25]  Robert Morris William Travers.  How Research Has Changed American Schools: A History from 1840 to the Present , 1983 .

[26]  John M. Ladbury,et al.  Measurement and evaluation of a TEM , 1990 .

[27]  Stephen Olejnik,et al.  Planning Educational Research: Determining the Necessary Sample Size. , 1984 .

[28]  Robert Rosenthal,et al.  Effect sizes: Pearson's correlation, its display via the BESD, and alternative indices. , 1991 .

[29]  William Meredith,et al.  Canonical correlations with fallible data , 1964 .

[30]  Carl J. Huberty,et al.  Discriminant Analysis Via Statistical Packages , 1997 .

[31]  Nancy J. Emmons.  Statistical Methods Used in "American Educational Research Journal,""Journal of Educational Psychology," and "Sociology of Education" from 1972 through 1987. , 1990 .

[32]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[33]  Joel R. Levin,et al.  Book Reviews , 1998 .

[34]  Clifford E. Lunneborg,et al.  Bootstrap Applications for the Behavioral Sciences , 1987 .

[35]  B. Thompson.  Canonical Correlation Analysis: Uses and Interpretation , 1984 .

[36]  James P. Shaver,et al.  What Statistical Significance Testing Is, and What It Is Not , 1993 .

[37]  Bruce Thompson,et al.  The Importance of Structure Coefficients in Regression Research , 1985 .

[38]  E. Scott Huebner,et al.  Correlates of life satisfaction in children. , 1991 .

[39]  W. W. Rozeboom.  The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[40]  R. L. Hagen.  In praise of the null hypothesis statistical test. , 1997 .

[41]  Bruce Thompson,et al.  Discstra: A Computer Program that Computes Bootstrap Resampling Estimates of Descriptive Discriminant Analysis Function and Structure Coefficients and Group Centroids , 1992 .

[42]  Bruce Thompson,et al.  A Primer on the Logic and Use of Canonical Correlation Analysis. , 1991 .

[43]  Bruce Thompson.  The concept of statistical significance testing , 1994 .

[44]  Patricia Snyder,et al.  Evaluating Results Using Corrected and Uncorrected Effect Size Estimates , 1993 .

[45]  N. Gage.  Hard Gains in the Soft Sciences: The Case of Pedagogy , 1985 .

[46]  Bruce Thompson.  Common Methodology Mistakes in Dissertations, Revisited. , 1994 .

[47]  Bruce Thompson,et al.  Program Facstrap: A Program that Computes Bootstrap Estimates of Factor Structure , 1988 .

[48]  Jacob Cohen.  Multiple regression as a general data-analytic system. , 1968 .

[49]  M. Levine.  Canonical Analysis and Factor Comparison , 1977 .

[50]  Glenn L. Rowley,et al.  The Reliability of Observational Measures. , 1976 .

[51]  Bruce Thompson,et al.  The Importance of Structure Coefficients in Structural Equation Modeling Confirmatory Factor Analysis , 1997 .

[52]  R. Falk,et al.  Significance Tests Die Hard , 1995 .

[53]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[54]  B. Thompson,et al.  Factor Analytic Evidence for the Construct Validity of Scores: A Historical Overview and Some Guidelines , 1996 .

[55]  K. Holzinger,et al.  A study in factor analysis : the stability of a bi-factor solution , 1939 .

[56]  Bruce Thompson,et al.  Alphamax: A Program that Maximizes Coefficient Alpha by Selective Item Deletion , 1990 .

[57]  Bruce W. Hall,et al.  Evaluation of Published Educational Research: A National Survey1 , 1975 .

[58]  Bruce W. Tuckman.  A Proposal for Improving the Quality of Published Educational Research , 1990 .

[59]  Bruce Thompson.  Alternate Methods for Analyzing Data from Education Experiments , 1985 .

[60]  Bruce Thompson,et al.  Why Won't Stepwise Methods Die? , 1989 .

[61]  Norman Cliff,et al.  Analyzing Multivariate Data , 1987 .

[62]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[63]  E. Scott Huebner,et al.  Burnout among School Psychologists: An Exploratory Investigation into Its Nature, Extent, and Correlates. , 1992 .

[64]  J. Elashoff,et al.  Multiple Regression in Behavioral Research. , 1974 .

[65]  Bruce Thompson,et al.  Exploring the Replicability of a Study's Results: Bootstrap Statistics for the Multivariate Case , 1995 .

[66]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[67]  Xitao Fan,et al.  Canonical Correlation Analysis as a General Analytical Model. , 1992 .

[68]  Bruce Thompson,et al.  Stepwise Regression and Stepwise Discriminant Analysis Need Not Apply here: A Guidelines Editorial , 1995 .

[69]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[70]  C. J. Huberty,et al.  Applied Discriminant Analysis , 1994 .

[71]  Sandra H. Eason,et al.  Why Generalizability Theory Yields Better Results than Classical Test Theory. , 1989 .

[72]  Anne L. Schneider,et al.  Policy Implications of Using Significance Tests in Evaluation Research , 1984 .

[73]  Brian M. Reinhardt,et al.  Factors Affecting Coefficient Alpha: A Mini Monte Carlo Study. , 1991 .