Experimental designs in management and leadership research: Strengths, limitations, and recommendations for improving publishability

Abstract Despite the renewed interest in the use of experimental designs in the fields of leadership and management over the past few decades, these designs are still relatively underutilized. Although there are several potential reasons for this, chief among them is misunderstanding the value of these designs. The purpose of this article is to review the role of laboratory, field, and quasi-experimental designs in management and leadership research. We first discuss the primary goals of experimental studies. Next, we examine the characteristics of experimental designs and how to distinguish laboratory, field, and quasi-experiments from one another and from non-experimental studies. Following these discussions, we provide examples of each type of experimental design and discuss their relative strengths and limitations. Finally, we discuss steps that researchers can take to increase the probability of having articles reporting experiments accepted by leadership and management journals.

[1]  Bruce Shearer,et al.  Piece Rates, Fixed Wages and Incentives: Evidence from a Field Experiment , 2004 .

[2]  Thomas D. Cook,et al.  Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. , 1972 .

[3]  S. Sitkin,et al.  Big‐B versus Big‐O: what is organizational about organizational behavior?* , 2001 .

[4]  S. Kerr,et al.  Leader initiating structure: A reconciliation of discrepant research results and some empirical tests☆ , 1976 .

[5]  Elizabeth A Stuart,et al.  Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. , 2010, Psychological methods.

[6]  R. Hertwig,et al.  Experimental practices in economics: A methodological challenge for psychologists? , 2001, Behavioral and Brain Sciences.

[7]  Jerald Greenberg,et al.  Situated Experiments in Organizations: Transplanting the Lab to the Field , 2004 .

[8]  J. Schaubroeck,et al.  Can Good Citizens Lead the Way in Providing Quality Service? A Field Quasi Experiment , 2001 .

[9]  Paul R. Sackett,et al.  Balancing Treatment and Control Groups in Quasi‐Experiments: An Introduction to Propensity Scoring , 2013 .

[10]  J. C. Wofford,et al.  Laboratory research on charismatic leadership , 1999 .

[11]  Philippe Jacquart,et al.  On making causal claims: A review and recommendations , 2010 .

[12]  Lawrence R. James,et al.  The unmeasured variables problem in path analysis. , 1980 .

[13]  Iwan Barankay,et al.  Incentives for Managers and Inequality Among Workers: Evidence from a Firm Level Experiment , 2006, SSRN Electronic Journal.

[14]  Huiwen Lian,et al.  Why Are Abusive Supervisors Abusive? A Dual-system Self-control Model , 2016 .

[15]  D. A. Kenny,et al.  Correlation and causality , 1979 .

[16]  David A. Hofmann,et al.  Outsourcing inspiration: The performance effects of ideological messages from leaders and beneficiaries , 2011 .

[17]  D. A. Kenny,et al.  Estimating and testing mediation and moderation in within-subject designs. , 2001, Psychological methods.

[18]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[19]  Paul R. Rosenbaum,et al.  Comparison of Multivariate Matching Methods: Structures, Distances, and Algorithms , 1993 .

[20]  Ricky W. Griffin,et al.  Laboratory research in management: Misconceptions and missed opportunities , 1991 .

[21]  Leonard Bickman,et al.  Applied Research Design: A Practical Approach , 2009 .

[22]  D. Mackinnon Introduction to Statistical Mediation Analysis , 2008 .

[23]  J. Antonakis On doing better science: From thrill of discovery to policy implications , 2017 .

[24]  J. Pearl Causality: Models, Reasoning and Inference , 2000 .

[25]  Mary E. Zellmer-Bruhn,et al.  From the Editors: Experimental designs in international business research , 2016 .

[26]  William R. Shadish,et al.  Randomized Controlled Studies and Alternative Designs in Outcome Studies , 2011 .

[27]  Robert Folger,et al.  Third parties' reactions to the abusive supervision of coworkers. , 2015, The Journal of applied psychology.

[28]  Arie W. Kruglanski,et al.  The Human Subject in the Psychology Experiment: Fact and Artifact , 1975 .

[29]  Donald P. Schwab,et al.  Research Methods for Organizational Studies , 1998 .

[30]  Jason A. Colquitt,et al.  From the Editors Publishing Laboratory Research in AMJ: A Question of When, Not If , 2008 .

[31]  Matthew Semadeni,et al.  The perils of endogeneity and instrumental variables in strategy research: : Understanding through simulations , 2014 .

[32]  J. Lynch On the External Validity of Experiments in Consumer Research , 1982 .

[33]  Bruce A. Thyer,et al.  Quasi-Experimental Research Designs , 2012 .

[34]  M. Orne On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[35]  O. Bandiera,et al.  Field Experiments with Firms , 2011, SSRN Electronic Journal.

[36]  D. Eden Leadership and expectations: Pygmalion effects and other self-fulfilling prophecies in organizations , 1992 .

[37]  Michael E. Gordon,et al.  On the virtues of laboratory babies and student bath water: A reply to Dobbins, Lane, and Steiner , 1988 .

[38]  Scott B. MacKenzie,et al.  Recommendations for Creating Better Concept Definitions in the Organizational, Behavioral, and Social Sciences , 2016 .

[39]  M. Gordon,et al.  The “Science of the Sophomore” Revisited: from Conjecture to Empiricism , 1986 .

[40]  Ronald F. Piccolo,et al.  The relative impact of complementary leader behaviors: Which matter most? , 2012 .

[41]  Bruce J. Avolio,et al.  Experimentally Analyzing the Impact of Leader Positivity on Follower Positivity and Performance , 2011 .

[42]  Elizabeth M. Campbell,et al.  Directive versus Empowering Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity , 2013 .

[43]  Ralph Hertwig,et al.  The Costs of Deception: Evidence from Psychology , 2001 .

[44]  Peter J. Frost,et al.  A laboratory study of charismatic leadership. , 1989 .

[45]  P. Cheng From covariation to causation: A causal power theory. , 1997 .

[46]  William M. K. Trochim Regression Discontinuity Design , 2001 .

[47]  Chester A. Schriesheim,et al.  Differences in Factor Structure across Three Versions of the Ohio State Leadership Scales. , 1975 .

[48]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[49]  Joeri Hofmans,et al.  Task complexity and transformational leadership: the mediating role of leaders’ state core self-evaluations , 2015 .

[50]  Irving M. Lane,et al.  A note on the role of laboratory methodologies in applied behavioural research: Don't throw out the baby with the bath water , 1988 .

[51]  Peter E. Kennedy A Guide to Econometrics , 1979 .

[52]  Arjen van Witteloostuijn,et al.  Toward Experimental International Business: Unraveling fundamental causal linkages , 2015 .

[53]  Daan van Knippenberg,et al.  License to Fail? How Leader Group Prototypicality Moderates the Effects of Leader Performance on Perceptions of Leadership Effectiveness , 2008 .

[54]  Robert F. Boruch,et al.  Randomized controlled trials for evaluation and planning , 2009 .

[55]  Gertrude Himmelfarb,et al.  Essays On Freedom And Power , 2010 .

[56]  John O. Summers,et al.  Checking the Success of Manipulations in Marketing Experiments , 1986 .

[57]  Robert G. Lord,et al.  The utility of experimental research in the study of transformational/charismatic leadership , 1999 .

[58]  Frank S. Freeman,et al.  Interpretation of educational measurements , 1928 .

[59]  Stephen E. Humphrey,et al.  TRAIT AND BEHAVIORAL THEORIES OF LEADERSHIP: AN INTEGRATION AND META-ANALYTIC TEST OF THEIR RELATIVE VALIDITY , 2011 .

[60]  T. Allen,et al.  The effects of organizational citizenship behavior on performance judgments: a field study and a laboratory experiment. , 1998, The Journal of applied psychology.

[61]  Alan G. Sawyer Demand Artifacts in Laboratory Experiments in Consumer Research , 1975 .

[62]  Peter D. Harms,et al.  Reconciling the Two Disciplines of Organisational Science: A Comparison of Findings from Lab and Field Research , 2015 .

[63]  James J. Lindsay,et al.  Research in the Psychological Laboratory , 1999 .

[64]  James G. Hunt,et al.  The effects of visionary and crisis-responsive charisma on followers: An experimental examination of two kinds of charismatic leadership , 1999 .

[65]  D. S. Derue,et al.  A quasi-experimental study of after-event reviews and leadership development. , 2012, The Journal of applied psychology.

[66]  W. Shadish,et al.  The renaissance of field experimentation in evaluating interventions. , 2009, Annual review of psychology.

[67]  Gilbert A. Churchill A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[68]  Daniel R. Ilgen,et al.  A century of work teams in the Journal of Applied Psychology. , 2017, The Journal of applied psychology.

[69]  Chester A. Schriesheim,et al.  Field studies of French and Raven's bases of power: Critique, reanalysis, and suggestions for future research. , 1985 .

[70]  Dov Eden,et al.  Field Experiments in Organizations , 2017 .

[71]  Herbert L. Smith 6. Matching with Multiple Controls to Estimate Treatment Effects in Observational Studies , 1997 .

[72]  Robert Rosenthal,et al.  The Volunteer Subject , 1976 .

[73]  Scott B. MacKenzie,et al.  The dangers of poor construct conceptualization , 2003 .

[74]  Eugene F. Stone-Romero,et al.  Trends in Research Design and Data Analytic Strategies in Organizational Research , 1995 .

[75]  Daniel R. Ilgen,et al.  Laboratory Research: A Question of When, Not If. , 1985 .

[76]  Samuel Bendahan,et al.  Leader corruption depends on power and testosterone , 2015 .

[77]  D. Campbell Factors relevant to the validity of experiments in social settings. , 1957, Psychological bulletin.

[78]  Julian C. Jamison,et al.  To Deceive or Not to Deceive: The Effect of Deception on Behavior in Future Laboratory Experiments , 2006 .

[79]  James T. Austin,et al.  History of Research Methods in Industrial and Organizational Psychology: Measurement, Design, Analysis , 2008 .

[80]  Earl R. Babbie,et al.  The practice of social research , 1969 .

[81]  Scott B. MacKenzie,et al.  Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques , 2011, MIS Q..

[82]  S. Sitkin,et al.  A Critical Assessment of Charismatic—Transformational Leadership Research: Back to the Drawing Board? , 2013 .

[83]  Dov Eden,et al.  Self-fulfilling prophecies in organizations. , 2003 .

[84]  Michael G. Findley,et al.  Field Experiments in Strategy Research , 2014, SSRN Electronic Journal.

[85]  L. Postman,et al.  The probability approach and nomothetic theory. , 1955, Psychological review.

[86]  Mingxiang Li,et al.  Using the Propensity Score Method to Estimate Causal Effects , 2013 .

[87]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[88]  Scott B. MacKenzie,et al.  Are we really measuring what we say we're measuring? Using video techniques to supplement traditional construct validation procedures. , 2013, The Journal of applied psychology.

[89]  K. M. Bartol,et al.  Linking Empowering Leadership and Employee Creativity: The Influence of Psychological Empowerment, Intrinsic Motivation, and Creative Process Engagement , 2010 .

[90]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[91]  L. Berkowitz,et al.  External validity is more than skin deep: Some answers to criticisms of laboratory experiments. , 1982 .

[92]  A. Kingstone,et al.  Attention, Researchers! It Is Time to Take a Look at the Real World , 2003 .

[93]  William Oakes,et al.  External validity and the use of real people as subjects. , 1972 .

[94]  J. Antonakis,et al.  Leadership Process Models: A Review and Synthesis , 2017 .

[95]  D. Mook,et al.  In defense of external invalidity. , 1983 .

[96]  D. Borsboom Measuring the mind: Conceptual issues in contemporary psychometrics , 2005 .

[97]  Jennifer S. Mueller Finding New Kinds of Needles in Haystacks: Experimentation in the Course of Abduction , 2018, Academy of Management Discoveries.

[98]  R. Harré,et al.  The explanation of social behaviour , 1973 .

[99]  G. Mitchell Revisiting Truth or Triviality , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[100]  Simon S. K. Lam,et al.  A field experiment testing frontline opinion leaders as change agents. , 2000, The Journal of applied psychology.

[101]  Marilynn B. Brewer,et al.  Principles and Methods of Social Research , 1986 .

[102]  B. Avolio,et al.  Impact of Transformational Leadership on Follower Development and Performance: A Field Experiment , 2002 .

[103]  M. Orne Demand Characteristics and the Concept of Quasi-Controls1 , 2009 .

[104]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[105]  Ralph L. Rosnow,et al.  Essentials of Behavioral Research: Methods and Data Analysis , 1984 .

[106]  David M. McCord,et al.  Evaluating the College Sophomore Problem: The Case of Personality and Politics , 2010, The Journal of psychology.

[107]  Edwin A. Locke,et al.  Resolving scientific disputes by the joint design of crucial experiments by the antagonists: Application to the Erez–Latham dispute regarding participation in goal setting. , 1988 .

[108]  Ralph Hertwig,et al.  Deception in Experiments: Revisiting the Arguments in Its Defense , 2008 .

[109]  K. Vohs,et al.  Psychology as the Science of Self-reports and Finger Movements Whatever Happened to Actual Behavior? , 2022 .

[110]  Robert F. Kidd,et al.  Manipulation checks: Advantage or disadvantage? , 1976 .

[111]  R. L. Dipboye,et al.  Research settings in industrial and organizational psychology: Are findings in the field more generalizable than in the laboratory? , 1979 .

[112]  R. A. Jones,et al.  Research Methods in the Social and Behavioral Sciences , 1985 .

[113]  R. Suddaby Editor's Comments: Construct Clarity in Theories of Management and Organization , 2010 .

[114]  Ethan R. Burris,et al.  Managerial modes of influence and counterproductivity in organizations: a longitudinal business-unit-level investigation. , 2007, The Journal of applied psychology.

[115]  K. Murphy,et al.  Evaluating the Performance of Paper People , 1986 .

[116]  Rolf van Dick,et al.  Dying for charisma: Leaders' inspirational appeal increases post-mortem , 2017 .

[117]  Derrick M. Anderson,et al.  Unfulfilled Promise: Laboratory experiments in public management research , 2015 .

[118]  T. Judge,et al.  Personality and leadership: a qualitative and quantitative review. , 2002, The Journal of applied psychology.

[119]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[120]  R. Peterson On the Use of College Students in Social Science Research: Insights from a Second‐Order Meta‐analysis , 2001 .

[121]  Jerald Greenberg,et al.  The College Sophomore as Guinea Pig: Setting the Record Straight , 1987 .

[122]  Roy Suddaby,et al.  Construct Clarity in Theories of Management and Organization , 2010 .

[123]  Patrick J. Rosopa,et al.  Research design options for testing mediation models and their implications for facets of validity , 2010 .

[124]  D. A. Kenny,et al.  Reflections on Mediation , 2008 .

[125]  Frank R. Kardes,et al.  In Defense of Experimental Consumer Psychology , 1996 .

[126]  Dustin K. Jundt,et al.  Functional versus dysfunctional team change: Problem diagnosis and structural feedback for self-managed teams , 2013 .

[127]  M. Zanna,et al.  Establishing a causal chain: why experiments are often more effective than mediational analyses in examining psychological processes. , 2005, Journal of personality and social psychology.

[128]  Jim McCambridge,et al.  The Effects of Demand Characteristics on Research Participant Behaviours in Non-Laboratory Settings: A Systematic Review , 2012, PloS one.

[129]  Denise M. Rousseau Envisioning Evidence-Based Management , 2012 .

[130]  D P Schultz,et al.  The human subject in psychological research. , 1969, Psychological bulletin.

[131]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[132]  P. Spirtes,et al.  Causation, prediction, and search , 1993 .

[133]  J. Lucas,et al.  Theory-Testing, Generalization, and the Problem of External Validity* , 2003 .

[134]  Helen Kelley,et al.  Research Commentary - Generalizability of Information Systems Research Using Student Subjects - A Reflection on Our Practices and Recommendations for Future Research , 2012, Inf. Syst. Res..

[135]  Charles R. Schwenk Why sacrifice rigour for relevance? A proposal for combining laboratory and field research in strategic management , 1982 .

[136]  Scott Highhouse Designing Experiments That Generalize , 2009 .

[137]  Dustin K. Jundt,et al.  Goal setting in teams: The impact of learning and performance goals on process and performance , 2013 .

[138]  Harold Sigall,et al.  Measures of Independent Variables and Mediators Are Useful in Social Psychology Experiments: But Are They Necessary? , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[139]  Gina S. Matkin,et al.  Applying mixed methods to leadership research: A review of current practices , 2012 .

[140]  Q. Mcnemar,et al.  Opinion-attitude methodology. , 1946, Psychological bulletin.

[141]  David de Vaus,et al.  Research Design in Social Research , 2001 .

[142]  Frank J. Landy,et al.  Another look at contrast effects in the employment interview. , 1973 .

[143]  Colin F. Camerer,et al.  The Promise and Success of Lab-Field Generalizability in Experimental Economics: A Critical Reply to Levitt and List , 2011 .

[144]  Christopher W. Bauman,et al.  Blame the shepherd not the sheep: Imitating higher-ranking transgressors mitigates punishment for unethical behavior , 2016 .

[145]  Jean M. Bartunek,et al.  Evidence-Based Management: Foundations, Development, Controversies and Future , 2017 .

[146]  Murray Webster,et al.  Why Do Experiments , 2014 .

[147]  Todd Rogers,et al.  Innovation with field experiments: Studying organizational behaviors in actual organizations , 2017 .

[148]  Ira J. Firestone,et al.  Implications of representativeness for generalizability of field and laboratory research findings. , 1980 .

[149]  Neal Schmitt,et al.  Student Guinea Pigs: Porcine Predictors and Particularistic Phenomena , 1987 .

[150]  Nathan P. Podsakoff,et al.  Effects of organizational citizenship behaviors on selection decisions in employment interviews. , 2011, The Journal of applied psychology.

[151]  R. L. Henshel,et al.  The purposes of laboratory experimentation and the virtues of deliberate artificiality. , 1980 .

[152]  P. Henry College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice , 2008 .

[153]  Scott B. MacKenzie,et al.  Working memory: theories, models, and controversies. , 2012, Annual review of psychology.

[154]  Howard Gadlin,et al.  Through the one-way mirror: The limits of experimental self-reflection. , 1975 .

[155]  Ronald F. Piccolo,et al.  Transformational and transactional leadership: a meta-analytic test of their relative validity. , 2004, The Journal of applied psychology.

[156]  J. Heckman,et al.  Lab Experiments Are a Major Source of Knowledge in the Social Sciences , 2009, Science.

[157]  D. O. Sears College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. , 1986 .

[158]  Eva M. Hyatt,et al.  A Critical Appraisal of Demand Artifacts in Consumer Research , 1991 .

[159]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[160]  P. M. Podsakoff,et al.  Speaking more broadly: an examination of the nature, antecedents, and consequences of an expanded set of employee voice behaviors. , 2014, The Journal of applied psychology.

[161]  William M. Holmes,et al.  Using Propensity Scores in Quasi-Experimental Designs , 2013 .

[162]  Bernardo F. Quiroga,et al.  On doing relevant and rigorous experiments: Review and recommendations , 2018, Journal of Operations Management.

[163]  R. Rosenthal,et al.  Covert communication in the psychological experiment. , 1967, Psychological bulletin.

[164]  Timothy D. Wilson,et al.  Experimentation in social psychology. , 1998 .

[165]  Bradley L. Kirkman,et al.  Recognizing "me" benefits "we": Investigating the positive spillover effects of formal individual recognition in teams. , 2016, The Journal of applied psychology.

[166]  Toby D. Wall,et al.  The Neglected Science and Art of Quasi-Experimentation , 2009 .

[167]  D. McKenzie,et al.  Returns to Capital in Microenterprises: Evidence from a Field Experiment , 2007, SSRN Electronic Journal.

[168]  T. Judge,et al.  Intelligence and leadership: a quantitative review and test of theoretical propositions. , 2004, The Journal of applied psychology.

[169]  Eugene F. Stone‐Romero,et al.  The Relative Validity and Usefulness of Various Empirical Research Designs , 2008 .

[170]  Simon S. K. Lam,et al.  Embracing transformational leadership: team values and the impact of leader behavior on team performance. , 2007, The Journal of applied psychology.

[171]  Lewis A. Taylor,et al.  Method Myopia , 2003 .

[172]  Hannah R. Rothstein,et al.  Synthesizing results of multiple randomized experiments to establish causality in mediation testing , 2015 .