Innovation with field experiments: Studying organizational behaviors in actual organizations

Abstract Organizational scholarship centers on understanding organizational context, usually captured through field studies, as well as determining causality, typically with laboratory experiments. We argue that field experiments can bridge these approaches, bringing causality to field research and developing organizational theory in novel ways. We present a taxonomy that proposes when to use an audit field experiment (AFE), procedural field experiment (PFE) or innovation field experiment (IFE) in organizational research and argue that field experiments are more feasible than ever before. With advances in technology, behavioral data has become more available and randomized changes are easier to implement, allowing field experiments to more easily create value—and impact—for scholars and organizations alike.

[1]  Marc Bendick,et al.  Measuring employment discrimination through controlled experiments , 1994, Race and Ethnic Conflict.

[2]  David G. Rand,et al.  Cooperating with the future , 2014, Nature.

[3]  Michael Luca,et al.  How to Design (and Analyze) a Business Experiment , 2015 .

[4]  J. V. Maanen,et al.  Toward a theory of organizational socialization , 1977 .

[5]  U. Gneezy,et al.  Lab in the Field: Measuring Preferences in the Wild , 2016 .

[6]  Erik Brynjolfsson,et al.  Big data: the management revolution. , 2012, Harvard business review.

[7]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[8]  Jeffrey R. Kling,et al.  Mechanism Experiments and Policy Evaluations , 2011 .

[9]  R. Gephart Qualitative Research and the Academy of Management Journal , 2004 .

[10]  A. Deaton,et al.  Understanding and Misunderstanding Randomized Controlled Trials , 2016, Social science & medicine.

[11]  T. Jick Mixing Qualitative and Quantitative Methods: Triangulation in Action. , 1979 .

[12]  M. Banaji,et al.  Naturals and strivers: Preferences and beliefs about sources of achievement , 2011 .

[13]  Bradley R. Staats,et al.  Breaking Them in or Eliciting Their Best? Reframing Socialization around Newcomers’ Authentic Self-expression , 2013 .

[14]  G. Johns The Essential Impact of Context on Organizational Behavior , 2006 .

[15]  Adam M. Grant Leading with Meaning: Beneficiary Contact, Prosocial Impact, and the Performance Effects of Transformational Leadership , 2012 .

[16]  R. Landis,et al.  Our Scholarly Practices Are Derailing Our Progress: The Importance of “Nothing” in the Organizational Sciences , 2013, Industrial and Organizational Psychology.

[17]  Leif D. Nelson,et al.  Shared Social Responsibility: A Field Experiment in Pay-What-You-Want Pricing and Charitable Giving , 2010, Science.

[18]  S. Robinson Trust and Breach of the Psychological Contract , 1996 .

[19]  Thomas G Rundall,et al.  Evidence-based management. , 2007, Hospitals & health networks.

[20]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[21]  K. Vohs,et al.  Psychology as the Science of Self-reports and Finger Movements Whatever Happened to Actual Behavior? , 2022 .

[22]  J. Stein,et al.  A Macroprudential Approach to Financial Regulation , 2010 .

[23]  Deepak Malhotra,et al.  When 3 + 1 > 4: Gift Structure and Reciprocity in the Field , 2015, Manag. Sci..

[24]  Chia-Jung Tsay,et al.  Creating Reciprocal Value Through Operational Transparency , 2015, Manag. Sci..

[25]  David G. Rand,et al.  Think global, act local: Preserving the global commons , 2016, Scientific Reports.

[26]  W. Shadish,et al.  The renaissance of field experimentation in evaluating interventions. , 2009, Annual review of psychology.

[27]  Silvana Trimi,et al.  Big-data applications in the government sector , 2014, Commun. ACM.

[28]  Maryam Kouchaki,et al.  Work-report formats and overbilling: How unit-reporting vs. cost-reporting increases accountability and decreases overbilling , 2015 .

[29]  G. Harrison,et al.  Field experiments , 1924, The Journal of Agricultural Science.

[30]  John A. List,et al.  Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off , 2011 .

[31]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[32]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[33]  Adam M. Grant,et al.  Impact and the art of motivation maintenance: The eVects of contact with beneWciaries on persistence behavior , 2006 .

[34]  Doreen Schweizer,et al.  Analyzing Quantitative Behavioral Observation Data , 2016 .

[35]  H. Tajfel Social Psychology of Intergroup Relations , 1982 .

[36]  Barry M. Staw,et al.  What Theory is Not , 1995 .

[37]  Angus Deaton Instruments, Randomization, and Learning about Development , 2010 .

[38]  Gerald R. Salancik,et al.  Field Stimulations for Organizational Behavior Research. , 1979 .

[39]  M. Patton,et al.  Qualitative evaluation and research methods , 1992 .

[40]  K. Weick What Theory Is Not, Theorizing Is , 1995 .

[41]  Peter Bergman,et al.  Is This Technology Useless? How Seemingly Irrelevant Factors Affect Adoption and Efficacy , 2017 .

[42]  The Trouble With JDM: Some Limitations to the Influence of JDM on Organizational Research , 2010, Industrial and Organizational Psychology.

[43]  Maria R. Ibanez,et al.  Behavioral Empirics and Field Experiments , 2018, The Handbook of Behavioral Operations.

[44]  B. Avolio,et al.  Impact of Transformational Leadership on Follower Development and Performance: A Field Experiment , 2002 .

[45]  Jennifer A. Chatman,et al.  Full-Cycle Micro-Organizational Behavior Research , 2005 .

[46]  Iris Bohnet,et al.  What Works: Gender Equality by Design , 2016 .

[47]  C. Goldin,et al.  Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians , 1997 .

[48]  Jennifer A. Chatman,et al.  PEOPLE AND ORGANIZATIONAL CULTURE: A PROFILE COMPARISON APPROACH TO ASSESSING PERSON-ORGANIZATION FIT , 1991 .

[49]  Lamar Pierce,et al.  Dishonesty in the Name of Equity , 2009, Psychological science.

[50]  Adam D. Galinsky,et al.  Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/ejsp.324 Power, optimism, and risk-taking , 2022 .

[51]  D. Rousseau Psychological and implied contracts in organizations , 1989 .

[52]  John P. A. Ioannidis,et al.  A manifesto for reproducible science , 2017, Nature Human Behaviour.

[53]  D. Campbell,et al.  Unobtrusive Measures: Nonreactive Research in the Social Sciences , 1966 .

[54]  KimGang-Hoon,et al.  Big-data applications in the government sector , 2014 .

[55]  Denise M. Rousseau,et al.  CHANGING OBLIGATIONS AND THE PSYCHOLOGICAL CONTRACT: A LONGITUDINAL STUDY , 1994 .

[56]  Massachusetts Clean Water Trust. 2015 Annual Report , 2015 .

[57]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[58]  H. Ono,et al.  The Review of Black Political Economy , 2003 .

[59]  Chia-Jung Tsay Sight over sound in the judgment of music performance , 2013, Proceedings of the National Academy of Sciences.

[60]  Alex Pentland,et al.  Big Data and Management , 2014 .

[61]  A. Strauss,et al.  The Discovery of Grounded Theory , 1967 .

[62]  J K Drummond,et al.  The scientific approach. , 1965, South African medical journal = Suid-Afrikaanse tydskrif vir geneeskunde.

[63]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[64]  R. Dawes,et al.  Belief in the unstructured interview: The persistence of an illusion , 2013, Judgment and Decision Making.

[65]  Megan A. K. Peters,et al.  Perceptual confidence neglects decision-incongruent evidence in the brain , 2017, Nature Human Behaviour.

[66]  Ron Kohavi,et al.  The Surprising Power of Online Experiments , 2017 .

[67]  Donald T. Campbell,et al.  Methods for the experimenting society , 1991 .

[68]  Christopher D. Chambers,et al.  Redefine statistical significance , 2017, Nature Human Behaviour.

[69]  Rachel Glennerster,et al.  Running Randomized Evaluations: A Practical Guide , 2013 .

[70]  N. Srinivasan,et al.  Registered Replication Report: Rand, Greene, and Nowak (2012) , 2017, Perspectives on psychological science : a journal of the Association for Psychological Science.

[71]  R. Landis,et al.  When is Nothing Something? Editorial for the Null Results Special Issue of Journal of Business and Psychology , 2014 .

[72]  H. Russell Bernard,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2000 .

[73]  Olivia Freeman,et al.  Talking points personal outcomes approach: practical guide. , 2012 .

[74]  Barry M. Staw The experimenting organization , 1977 .

[75]  F. Gino,et al.  Dishonest Deed, Clear Conscience: When Cheating Leads to Moral Disengagement and Motivated Forgetting , 2011, Personality & social psychology bulletin.

[76]  Alex O Holcombe,et al.  An Introduction to Registered Replication Reports at Perspectives on Psychological Science , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[77]  Nina Mazar,et al.  Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end , 2012, Proceedings of the National Academy of Sciences.

[78]  E. Suchman Evaluative Research: Principles and Practice in Public Service and Social Action Progr , 1968 .

[79]  K. Vohs,et al.  Psychology as the Science of Self-Reports and Finger Movements , 2010 .

[80]  K. Eisenhardt Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[81]  H. Parsons What Happened at Hawthorne? , 1974, Science.

[82]  Michael I. Norton,et al.  The Labor Illusion: How Operational Transparency Increases Perceived Value , 2011, Manag. Sci..

[83]  R. Suddaby From the Editors: What Grounded Theory is Not , 2006 .

[84]  O. Hauser,et al.  Failure to CAPTCHA Attention: Null Results from an Honesty Priming Experiment in Guatemala , 2017, Behavioral sciences.

[85]  J. Moon,et al.  Corporate Social Responsibility , 2004 .

[86]  Wanda J. Orlikowski,et al.  Theorizing Practice and Practicing Theory , 2011, Organ. Sci..

[87]  Matteo M. Galizzi,et al.  On the External Validity of Social Preference Games: A Systematic Lab-Field Study , 2018, Manag. Sci..

[88]  Ronald E. Purser,et al.  Deconstructing the Lone Genius Myth: Toward a Contextual View of Creativity , 1995 .

[89]  A. Parry Handbook of Qualitative Research , 2002 .

[90]  Muriel Niederle,et al.  Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible , 2015 .

[91]  Fred A. Mael,et al.  Social identity theory and the organization , 1989 .

[92]  E. King,et al.  Field Experiments on Sensitive Organizational Topics , 2013 .

[93]  Matteo M. Galizzi,et al.  Like ripples on a pond: behavioral spillovers and their implications for research and policy , 2015 .

[94]  D. Malhotra,et al.  Trust and Collaboration in the Aftermath of Conflict: The Effects of Contract Structure , 2011 .

[95]  Lorne Campbell,et al.  Registered Replication Report , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[96]  Dov Eden,et al.  Field Experiments in Organizations , 2017 .

[97]  Bruce Western,et al.  Discrimination in a Low-Wage Labor Market , 2009, American sociological review.

[98]  Leif D. Nelson,et al.  Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” , 2014 .

[99]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[100]  Katherine L. Milkman,et al.  Temporal Distance and Discrimination , 2012, Psychological science.

[101]  Barry M. Staw,et al.  Stumbling Toward a Social Psychology of Organizations: An Autobiographical Look at the Direction of Organizational Research , 2016 .

[102]  Michael G. Findley,et al.  Field Experiments in Strategy Research , 2014, SSRN Electronic Journal.

[103]  Stefan Thau,et al.  Experiments in organizational behavior , 2014 .

[104]  Chia-Jung Tsay Privileging Naturals Over Strivers , 2016, Personality & social psychology bulletin.

[105]  Jane Sell,et al.  Laboratory experiments in the social sciences , 2007 .

[106]  Francesca Gino,et al.  Behavioral Ethics: Toward a Deeper Understanding of Moral Judgment and Dishonesty , 2012 .

[107]  A. Edmondson Psychological Safety and Learning Behavior in Work Teams , 1999 .

[108]  Keystone State,et al.  2015 Annual Report , 2015 .

[109]  D. Campbell Reforms as experiments , 1969 .

[110]  Donald P. Green,et al.  Field Experiments: Design, Analysis, and Interpretation , 2012 .

[111]  Vanessa C. Burbano Social Responsibility Messages and Worker Wage Requirements: Field Experimental Evidence from Online Labor Marketplaces , 2015, Organ. Sci..

[112]  Andrew Leigh,et al.  Does Ethnic Discrimination Vary Across Minority Groups? Evidence from a Field Experiment , 2012 .

[113]  Amy Ednnondson,et al.  Psychological Safety and Learning Behavior in Work , 2008 .

[114]  Leif D. Nelson,et al.  Pay-what-you-want, identity, and self-signaling in markets , 2012, Proceedings of the National Academy of Sciences.

[115]  J. Windeler [External validity]. , 2008, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen.

[116]  Charles A. Holt,et al.  Discrimination: Experimental Evidence from Psychology and Economics , 2006 .

[117]  R. I. Sutton,et al.  Organizational behavior: linking individuals and groups to organizational contexts. , 1993, Annual review of psychology.

[118]  J. Hackman,et al.  Employee reactions to job characteristics. , 1971 .

[119]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[120]  Rachel Glennerster,et al.  The Generalizability Puzzle , 2017 .

[121]  Donald P. Green,et al.  Enough Already about “Black Box” Experiments: Studying Mediation Is More Difficult than Most Scholars Suppose , 2009 .

[122]  Todd Bodner,et al.  Newcomer adjustment during organizational socialization: a meta-analytic review of antecedents, outcomes, and methods. , 2007, The Journal of applied psychology.

[123]  I SAMUELS,et al.  Reticular mechanisms and behavior. , 1959, Psychological bulletin.

[124]  Adam D. Galinsky,et al.  Power, Propensity to Negotiate, and Moving First in Competitive Interactions , 2007, Personality & social psychology bulletin.

[125]  Invisible Inequality Leads to Punishing the Poor and Rewarding the Rich , 2016 .

[126]  Erin L. Krupka,et al.  License to Cheat: Voluntary Regulation and Ethical Behavior , 2013, Manag. Sci..

[127]  D. Malhotra,et al.  The Effects of Contracts on Interpersonal Trust , 2002 .