From significant difference to significant sameness: Proposing a paradigm shift in business research

Complaints about the value of academic business research in addressing real world issues are common. A change in research paradigms—from significant difference to significant sameness—is necessary to improve this situation. The present paper challenges research orthodoxy as representing poor scientific practice and advocates an entirely different paradigm based on a sophisticated replication tradition following a meta-methodology of critical realism. A comprehensive analysis across philosophical, methodological, and statistical literatures develops this argument.

[1]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[2]  Russell S. Winer,et al.  Experimentation in the 21st century: The importance of external validity , 1999 .

[3]  Mark A. Mone,et al.  THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH , 1996 .

[4]  R. Bhaskar.  The Possibility of Naturalism , 2014 .

[5]  Janet Hoek,et al.  The Tea Bag Experiment: More Evidence on Incentives in Mail Surveys , 1998 .

[6]  Rory A. Fisher,et al.  The Arrangement of Field Experiments , 1992 .

[7]  J. Armstrong,et al.  Replications and Extensions in Marketing - Rarely Published But Quite Contrary , 1994 .

[8]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[9]  Berth Danermark,et al.  Explaining Society , 2001 .

[10]  L. Magnani.  Abduction, Reason, and Science. Process of Discovery and Explanation , 2001 .

[11]  Kenneth J. Ottenbacher,et al.  The Power of Replications and Replications of Power , 1996 .

[12]  A. Ehrenberg,et al.  Predictability and prediction , 1993 .

[13]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[14]  R. Nickerson,et al.  Null hypothesis significance testing: a review of an old and continuing controversy. , 2000, Psychological methods.

[15]  A. Ehrenberg,et al.  The Design of Replicated Studies , 1993 .

[16]  J. A. Nelder,et al.  Statistics, Science and Technology , 1986 .

[17]  Anthony D. Cox,et al.  The Effects of Background Music in Advertising: A Reassessment , 1989 .

[18]  Clark Glymour,et al.  Hypothetico-Deductivism Is Hopeless , 1980, Philosophy of Science.

[19]  B. Haig.  An abductive theory of scientific method. , 2005, Psychological methods.

[20]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[21]  Robert E. Krebs,et al.  Scientific Laws, Principles, and Theories: A Reference Guide , 2000 .

[22]  Raymond Hubbard,et al.  An empirical comparison of published replication research in accounting, economics, finance, management, and marketing , 1996 .

[23]  Randall L. Schultz,et al.  A Study of Marketing Generalizations , 1980 .

[24]  M. Resnik,et al.  Aspects of Scientific Explanation. , 1966 .

[25]  Michael E. Raynor,et al.  The Innovator's Solution: Creating and Sustaining Successful Growth , 2003 .

[26]  A. F. Chalmers,et al.  What Is This Thing Called Science , 1976 .

[27]  Eric W. K. Tsang,et al.  Replication and Theory Development in Organizational Science: A Critical Realist Perspective , 1999 .

[28]  John W. Tukey,et al.  We Need Both Exploratory and Confirmatory , 1980 .

[29]  William H. Starbuck,et al.  Keeping a Butterfly and an Elephant in a House of Cards: The Elements of Exceptional Success , 1993 .

[30]  Chris Chatfield,et al.  Confessions of a pragmatic statistician , 2002 .

[31]  William D. Wells,et al.  The Perils of N = 1 , 2001 .

[32]  Daniel M. Hausman,et al.  The Inexact and Separate Science of Economics , 1993 .

[33]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[34]  Jerker Denrell.  Vicarious Learning, Undersampling of Failure, and the Myths of Management , 2003, Organ. Sci..

[35]  E. A. Locke.  The Case for Inductive Theory Building† , 2007 .

[36]  P. Meehl.  Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[37]  J. Scott Armstrong,et al.  Replication research's disturbing trend , 2007 .

[38]  Leonard N. Reid,et al.  Replication in Advertising Research: 1977, 1978, 1979 , 1981 .

[39]  R. Frick,et al.  The appropriate use of null hypothesis testing. , 1996 .

[40]  B. McKelvey.  Organizational Systematics: Taxonomic Lessons from Biology , 1978 .

[41]  Henry H. Bauer,et al.  Scientific Literacy and the Myth of the Scientific Method , 1992 .

[42]  S. Psillos.  Scientific Realism: How Science Tracks Truth , 1999 .

[43]  Frank M. Bass,et al.  Empirical Generalizations and Marketing Science: A Personal View , 1995 .

[44]  George A. Akerlof.  How Human Psychology Drives the Economy and Why It Matters , 2009 .

[45]  J. Losee.  A historical introduction to the philosophy of science , 1972 .

[46]  Robert Brown.  Explanation in Social Science , 1963 .

[47]  Peter November,et al.  Seven Reasons Why Marketing Practitioners Should Ignore Marketing Academic Research , 2004 .

[48]  Jay Brand,et al.  File Drawer Problem , 2022, The SAGE Encyclopedia of Research Design.

[49]  J. H. Steiger.  Structural Model Evaluation and Modification: An Interval Estimation Approach. , 1990, Multivariate behavioral research.

[50]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[51]  M. J. Bayarri,et al.  Confusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical Statistical Testing , 2003 .

[52]  P. Barwise.  Good Empirical Generalizations , 1995 .

[53]  John G. Lynch,et al.  Validity and the research process , 1986 .

[54]  A. Sayer,et al.  Realism and Social Science , 1999 .

[55]  S. F. Gardial,et al.  Methods of Knowledge Development in Marketing and Macromarketing , 1990 .

[56]  D. Mccloskey,et al.  The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives , 2008 .

[57]  Alan Ryan.  The philosophy of the social sciences , 1970 .

[58]  C. Chatfield.  Model uncertainty, data mining and statistical inference , 1995 .

[59]  Daniel E. Vetter,et al.  Replication in strategic management: scientific testing for validity, generalizability, and usefulness , 1998 .

[60]  John E. Hunter,et al.  The Desperate Need for Replications , 2001 .

[61]  R. Lindsay,et al.  Publication System Biases Associated with the Statistical Testing Paradigm , 1994 .

[62]  A. Ehrenberg.  Theory or Well-Based Results: Which Comes First? , 1994 .

[63]  B. Flyvbjerg,et al.  Making Social Science Matter: Why Social Inquiry Fails and How It Can Succeed Again , 2001 .

[64]  George A. Akerlof,et al.  Animal spirits : how human psychology drives the economy, and why it matters for global capitalism : with a new preface by the authors , 2010 .

[65]  K. Popper,et al.  Conjectures and Refutations , 1963 .

[66]  T. Lawson.  Economics and Reality , 1997 .

[67]  John D. Aram,et al.  Bridging Scholarship in Management: Epistemological Reflections , 2003 .

[68]  Scott M. Fuess.  On Replications in Business and Economics Research: The QJBE Case , 1996 .

[69]  J. Ravetz.  Scientific Knowledge and Its Social Problems , 2020 .

[70]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[71]  Gerald J. Gorn,et al.  The Effects of Music in Advertising on Choice Behavior: A Classical Conditioning Approach: , 1982 .

[72]  J. Lynch.  Theory and external validity , 1999 .

[73]  Michael McAleer,et al.  Simplicity, scientific inference and econometric modelling , 1995 .

[74]  Raymond Hubbard,et al.  Journal Prestige and the Publication Frequency of Replication Research in the Finance Literature , 1997 .

[75]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[76]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[77]  Daniel E. Vetter,et al.  Replications in the Finance Literature: An Empirical Study , 1991 .

[78]  Frank Yates,et al.  The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics , 1951 .

[79]  A. V. D. Ven,et al.  Knowledge for Theory and Practice , 2006 .

[80]  Andrew Sayer,et al.  Method in Social Science: A Realist Approach , 1984 .

[81]  James O. Berger,et al.  Statistical Analysis and the Illusion of Objectivity , 1988 .

[82]  Stewart Clegg,et al.  Revising the Boundaries: Management Education and Learning in a Postpositivist World , 2003 .

[83]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[84]  Daniel E. Vetter,et al.  The Publication Incidence of Replications and Critical Commentary in Economics , 1992 .

[85]  C. Whitbeck,et al.  A Realist Theory of Science. , 1977 .

[86]  George E. P. Box,et al.  Statistics and Quality Improvement , 1994 .

[87]  J. Tukey.  Conclusions vs Decisions , 1960 .

[88]  Ronald N. Giere,et al.  Science without laws , 1999 .

[89]  M. Dunn,et al.  Conducting Marketing Science: The Role of Replication in the Research Process , 2000 .

[90]  S. Keen,et al.  经济学辨析：社会科学的—袭皇帝新衣 = Debunking Economics : The Naked Emperor of the Social Sciences , 2001 .

[91]  W E Deming,et al.  On probability as a basis for action. , 1975, Methods of information in medicine. Supplement.

[92]  Clayton M. Christensen,et al.  Why hard-nosed executives should care about management theory. , 2003, Harvard business review.

[93]  David Otley,et al.  Management control and performance management: whence and whither? , 2003 .

[94]  Matthew Longshore Smith,et al.  Overcoming theory-practice inconsistencies: Critical realism and information systems research , 2006, Inf. Organ..

[95]  David J. Ketchen,et al.  Organizational configurations and performance: the role of statistical power in extant research , 1999 .

[96]  John Hicks,et al.  Causality in Economics , 1980 .

[97]  R. Murray Lindsay,et al.  Reconsidering the status of tests of significance: An alternative criterion of adequacy☆ , 1995 .

[98]  William Q. Meeker,et al.  Assumptions for statistical inference , 1993 .

[99]  William H. Starbuck,et al.  The production of knowledge : the challenge of social science research , 2006 .

[100]  D. Hambrick.  THE FIELD OF MANAGEMENT'S DEVOTION TO THEORY: TOO MUCH OF A GOOD THING? , 2007 .

[101]  M. J. Bayarri,et al.  Bayesian Design of “Successful” Replications , 2002 .

[102]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[103]  W. Bennis,et al.  How business schools lost their way. , 2005, Harvard business review.

[104]  Peter Lipton,et al.  Inference to the best explanation , 1993 .

[105]  Shelby D. Hunt,et al.  Controversy in marketing theory : for reason, realism, truth, and objectivity , 2003 .

[106]  J. Nelder,et al.  Statistics for the millennium. From statistics to statistical science. Commentary. Author's of reply , 1999 .

[107]  G. Harman.  The Inference to the Best Explanation , 1965 .

[108]  Patrick Baert,et al.  Contemporary philosophy of social science , 1991 .

[109]  Geoff Easton,et al.  Marketing: a Critical Realist approach , 2002 .

[110]  Louis Phlips,et al.  Applied consumption analysis , 1974 .

[111]  Raymond Hubbard,et al.  How the Emphasis on ‘Original’ Empirical Marketing Research Impedes Knowledge Development , 2002 .

[112]  C Poole,et al.  Low P-Values or Narrow Confidence Intervals: Which Are More Durable? , 2001, Epidemiology.

[113]  J. Ziman,et al.  Science and Its Critics@@@Reliable Knowledge: An Exploration of the Grounds for Belief in Science , 1979 .

[114]  H. Keuzenkamp,et al.  Probability, econometrics and truth: the methodology of economics , 2000 .

[115]  P. Ormerod.  The Death of Economics , 1994 .