Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets

ABSTRACT: Online labor markets allow rapid recruitment of large numbers of workers for very low pay. Although online workers are often used as research participants, there is little evidence that they are motivated to make costly choices to forgo wealth or leisure that are often central to addressing accounting research questions. Thus, we investigate the validity of using online workers as a proxy for non-experts when accounting research designs use more demanding tasks than these workers typically complete. Three experiments examine the costly choices of online workers relative to student research participants. We find that online workers are at least as willing as students to make costly choices, even at significantly lower wages. We also find that online workers are sensitive to performance-based wages, which are just as effective in inducing high effort as high fixed wages. We discuss implications of our results for conducting accounting research with online workers. Data Availability: Contact the au...

[1]  C. Frith,et al.  The Role of Working Memory in Visual Selective Attention , 2001, Science.

[2]  Jonathan H. Grenier,et al.  The Effects of Accounting Standard Precision, Auditor Task Expertise, and Judgment Frameworks on Audit Firm Litigation Exposure , 2015, Current Issues in Auditing.

[3]  Geoffrey B. Sprinkle,et al.  The effects of monetary incentives on effort and task performance: theories, evidence, and a framework for research , 2002 .

[4]  William D. Brink.  The Effects of Sharing the Consequences of Tax Evasion and Regret Salience on Taxpayer Compliance , 2014 .

[5]  Lisa Koonce,et al.  The Effects of Norms on Investor Reactions to Derivative Use , 2015 .

[6]  Jeremy Douthit,et al.  The Robustness of Honesty Effects on Budget Proposals when the Superior has Rejection Authority , 2015 .

[7]  E. Deci,et al.  A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. , 1999, Psychological bulletin.

[8]  V. Smith,et al.  Social Distance and Other-Regarding Behavior in Dictator Games: Reply , 1999 .

[9]  Willie Choi,et al.  Strategy Selection, Surrogation, and Strategic Performance Measurement Systems , 2012 .

[10]  Kristy L. Towry,et al.  Does the Communication of Causal Linkages Improve Employee Effort Allocations and Firm Performance? An Experimental Investigation , 2012 .

[11]  Amar Cheema,et al.  Data collection in a flat world: the strengths and weaknesses of mechanical turk samples , 2013 .

[12]  A. Rustichini,et al.  Pay Enough or Don't Pay at All , 2000 .

[13]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[14]  Donald V. Moser,et al.  Honesty in Managerial Reporting , 2001 .

[15]  Marie-Claire Villeval,et al.  Does Monitoring Decrease Work Effort? The Complementarity between Agency and Crowding-Out Theories , 2004, Games Econ. Behav..

[16]  M. W. Nelson,et al.  Experimental Research in Financial Accounting , 2001 .

[17]  Christopher P. Cerasoli,et al.  Intrinsic motivation and extrinsic incentives jointly predict performance: a 40-year meta-analysis. , 2014, Psychological bulletin.

[18]  Barry L. Lewis,et al.  AN EXPERIMENT TESTING THE BEHAVIORAL EQUIVALENCE OF STRATEGICALLY EQUIVALENT EMPLOYMENT CONTRACTS , 1989 .

[19]  E. Fehr,et al.  Psychological Foundations of Incentives , 2002 .

[20]  Bryan K. Church,et al.  Shared Interest and Honesty in Budget Reporting , 2012 .

[21]  Geoffrey B. Sprinkle,et al.  A Review of the Effects of Financial Incentives on Performance in Laboratory Tasks: Implications for Management Accounting , 2000 .

[22]  Sandra Slaughter,et al.  Understanding the Motivations, Participation, and Performance of Open Source Software Developers: A Longitudinal Study of the Apache Projects , 2006, Manag. Sci..

[23]  James H. Long,et al.  Online Instrument Delivery and Participant Recruitment Services: Emerging Opportunities for Behavioral Accounting Research , 2014 .

[24]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[25]  Max H. Bazerman,et al.  'A MATTER OF TRUST' : EFFECTS OF COMMUNICATION ON THE EFFICIENCY AND DISTRIBUTION OF OUTCOMES , 1998 .

[26]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[27]  V. Smith,et al.  Social distance and other-regarding behavior in dictator games , 2000 .

[28]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[29]  Andrew H. Newman,et al.  A Marathon, a Series of Sprints, or Both? Tournament Horizon and Dynamic Task Complexity in Multi-Period Settings , 2016 .

[30]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[31]  Gerald R. Ferris,et al.  Distance in organizations , 1993 .

[32]  Kristy L. Towry,et al.  Contracting on Contemporaneous versus Forward‐Looking Measures: An Experimental Investigation* , 2008 .

[33]  A. Isen,et al.  The Influence of Positive Affect on Intrinsic and Extrinsic Motivation: Facilitating Enjoyment of Play, Responsible Work Behavior, and Self-Control , 2005 .

[34]  Jane Kennedy,et al.  Are M.B.A. Students a Good Proxy for Nonprofessional Investors , 2007 .

[35]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[36]  Geoffrey B. Sprinkle.  Perspectives on experimental research in managerial accounting , 2003 .

[37]  Clara Xiaoling Chen,et al.  Can Wages Buy Honesty? The Relationship Between Relative Wages and Employee Theft , 2012 .

[38]  Charalampos Mainemelis,et al.  When The Muse Takes It All: A Model For The Experience of Timelessness in Organizations , 2001 .

[39]  Stanley Baiman,et al.  AGENCY RESEARCH IN MANAGERIAL ACCOUNTING: A SECOND LOOK. , 1990 .

[40]  Kristina Rennekamp.  Processing Fluency and Investors’ Reactions to Disclosure Readability: disclosure readability and investors’ reactions , 2012 .

[41]  D. Pfaff,et al.  Agency theory, performance evaluation, and the hypothetical construct of intrinsic motivation , 2002 .

[42]  Joseph F. Brazel,et al.  Electronic versus Face-to-Face Review: The Effects of Alternative Forms of Review on Auditors' Performance , 2003 .

[43]  William B. Tayler,et al.  Lost in Translation: The Effects of Incentive Compensation on Strategy Surrogation , 2012 .

[44]  Accounting, Principal-Agent Theory, and Self-Interested Behavior , 1992 .

[45]  Geoffrey B. Sprinkle,et al.  The effect of rankings on honesty in budget reporting , 2014 .

[46]  Scott Clifford,et al.  Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies , 2014, Journal of Experimental Political Science.

[47]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[48]  Frederick W. Rankin,et al.  The Effect of Honesty and Superior Authority on Budget Proposals , 2008 .

[49]  E. Fehr.  A Theory of Fairness, Competition and Cooperation , 1998 .

[50]  Steven J. Kachelmeier,et al.  Measuring and Motivating Quantity, Creativity, or Both , 2008 .

[51]  Frederick W. Rankin,et al.  The Effect of Information Systems on Honesty in Managerial Reporting: A Behavioral Perspective Honesty in Accounting and Control: A Discussion of "The Effect of Information Systems on Honesty in Managerial Reporting: A Behavioral Perspective" , 2006 .

[52]  Michael G. Williamson,et al.  Reward System Design and Group Creativity: An Experimental Investigation , 2012 .

[53]  Yu-An Sun,et al.  The Effects of Performance-Contingent Financial Incentives in Online Labor Markets , 2013, AAAI.

[54]  C. Stumpp,et al.  The Impact of Water Table Drawdown and Drying on Subterranean Aquatic Fauna in In-Vitro Experiments , 2013, PloS one.

[55]  S. Mark Young,et al.  Judgment and decision-making research in accounting and auditing: Experimental incentive-contracting research in management accounting , 1995 .

[56]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[57]  Joseph S. Valacich,et al.  The Influence of Task Interruption on Individual Decision Making: An Information Overload Perspective , 1999 .

[58]  Iris Bohnet,et al.  The sound of silence in prisoner's dilemma and dictator games , 1999 .

[59]  Frederick W. Rankin,et al.  The Effect of Information Systems on Honesty in Managerial Reporting: A Behavioral Perspective , 2006 .

[60]  Agnès Festré,et al.  Theory and Evidence in Psychology and Economics About Motivation Crowding Out: A Possible Convergence? , 2015 .

[61]  Susan D. Krische.  Investment Experience, Financial Literacy, and Investment-Related Judgments , 2018 .

[62]  Daniel G. Goldstein,et al.  Honesty in an Online Labor Market , 2011, Human Computation.

[63]  Robert H. Ashton,et al.  Judgment and decision-making research in accounting and auditing: Acknowledgments , 1995 .

[64]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[65]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[66]  J. McGonigal.  Reality Is Broken: Why Games Make Us Better and How They Can Change the World , 2011 .

[67]  Mark Penno,et al.  Asymmetry Of Pre-Decision Information And Managerial Accounting , 1984 .

[68]  Marco Fondi,et al.  Deciphering the Role of RND Efflux Transporters in Burkholderia cenocepacia , 2011, PloS one.

[69]  Robert Libby,et al.  Determinants of judgment performance in accounting settings: Ability, knowledge, motivation , 1993 .

[70]  J. George,et al.  Work Interrupted: A Closer Look at the Role of Interruptions in Organizational Life , 2003 .

[71]  Nicholas Seybert,et al.  Impaired Judgment: The Effects of Asset Impairment Reversibility and Cognitive Dissonance on Future Investment , 2014 .

[72]  Terry E. Duncan,et al.  Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: a confirmatory factor analysis. , 1989, Research quarterly for exercise and sport.

[73]  A. Acquisti,et al.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk , 2013, Behavior Research Methods.

[74]  Roderick M. Kramer,et al.  Trust and distrust in organizations: emerging perspectives, enduring questions. , 1999, Annual review of psychology.

[75]  Steven J. Kachelmeier,et al.  Negotiated Transfer Pricing: Is Fairness Easier Said than Done? , 2002 .

[76]  Steven J. Kachelmeier,et al.  Attracting Creativity: The Initial and Aggregate Effects of Contract Selection on Creativity-Weighted Productivity , 2010 .

[77]  Kristopher J Preacher,et al.  Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models , 2008, Behavior research methods.

[78]  D. Hoffman,et al.  The Influence of Goal-Directed and Experiential Activities on Online Flow Experiences , 2003 .

[79]  C. B. Colby.  The weirdest people in the world , 1973 .

[80]  Stanley Baiman,et al.  PRE-DECISION INFORMATION AND PARTICIPATIVE MANAGEMENT CONTROL-SYSTEMS , 1983 .

[81]  J. Rubin,et al.  Negotiation , 1983 .

[82]  Yongxia Skadberg,et al.  Visitors' flow experience while browsing a Web site: its measurement, contributing factors and consequences , 2004, Comput. Hum. Behav..

[83]  U. Fischbacher,et al.  Why Social Preferences Matter - the Impact of Non-Selfish Motives on Competition, Cooperation and Incentives , 2002 .

[84]  V. S. Maas,et al.  How Control System Design Influences Performance Misreporting , 2013 .

[85]  Donna L. Hoffman,et al.  Flow Online: Lessons Learned and Future Prospects , 2009 .

[86]  I. Solomon,et al.  Theory and Experimentation in Studies of Audit Judgments and Decisions: Avoiding Common Research Traps , 2001 .

[87]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[88]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[89]  George A. Akerlof.  The Market for “Lemons”: Quality Uncertainty and the Market Mechanism , 1970 .

[90]  J. H. Davis,et al.  An Integrative Model Of Organizational Trust , 1995 .