An Introduction to Conducting Email Audit Studies

This chapter offers the first general introduction to conducting email audit studies. It provides an overview of the steps involved from experimental design to empirical analysis. It then offers detailed recommendations about email address collection, email delivery, and email analysis, which are usually the three most challenging points of an audit study. The focus here is on providing a set of primarily technical recommendations to researchers who might want to conduct an email audit study. The chapter concludes by suggesting several ways that email audit studies can be adapted to investigate a broader range of social phenomena.

[1]  Muriel Niederle,et al.  Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible , 2015 .

[2]  S. Baert.  Wage subsidies and hiring chances for the disabled: some causal evidence , 2014, The European Journal of Health Economics.

[3]  Michael Vlassopoulos,et al.  Racial Discrimination in Local Public Services: A Field Experiment in the US , 2015, SSRN Electronic Journal.

[4]  C. Grose.  Field Experimental Work on Political Institutions , 2014 .

[5]  S. Baert,et al.  Getting Grey Hairs in the Labour Market: An Alternative Experiment on Age Discrimination , 2016, SSRN Electronic Journal.

[6]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[7]  David E. Broockman,et al.  Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators , 2011 .

[8]  S. M. Gaddis.  Discrimination in the Credential Society: An Audit Study of Race and College Selectivity in the Labor Market , 2014 .

[9]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[10]  Marc Ratkovic,et al.  Estimating treatment effect heterogeneity in randomized program evaluation , 2013, 1305.5682.

[11]  Katherine L. Milkman,et al.  What happens before? A field experiment exploring how pay and representation differentially shape bias on the pathway into organizations. , 2015, The Journal of applied psychology.

[12]  Katherine L. Milkman,et al.  Temporal Distance and Discrimination , 2012, Psychological science.

[13]  J. Sharman.  Shopping for Anonymous Shell Companies: An Audit Study of Anonymity and Crime in the International Financial System , 2010 .

[14]  J. Heckman.  Detecting Discrimination , 1998 .

[15]  Bernie Hogan,et al.  Racial and Ethnic Biases in Rental Housing: An Audit Study of Online Apartment Listings , 2011 .

[16]  Brad J. Bushman,et al.  You've got mail: Using e-mail to examine the effect of prejudiced attitudes on discrimination against Arabs , 2004 .

[17]  D. Pager,et al.  The Sociology of Discrimination: Racial Discrimination in Employment, Housing, Credit, and Consumer Markets. , 2008, Annual review of sociology.

[18]  Sharon L. Lohr,et al.  Sampling: Design and Analysis , 1999 .

[19]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[20]  David S. Pedulla.  Emerging Frontiers in Audit Study Research: Mechanisms, Variation, and Representativeness , 2018 .

[21]  Winston T. Lin,et al.  Standard Operating Procedures: A Safety Net for Pre-Analysis Plans , 2016, PS: Political Science & Politics.

[22]  R. J. Hauck.  Protecting Human Research Participants, IRBs, and Political Science Redux: Editor's Introduction , 2008, PS: Political Science & Politics.

[23]  Michael G. Findley,et al.  Causes of noncompliance with international law: : a field experiment on anonymous incorporation , 2015 .

[24]  David Neumark,et al.  Sex Discrimination in Restaurant Hiring: An Audit Study , 1995 .

[25]  Daniel M. Butler.  Representing the Advantaged: How Politicians Reinforce Inequality , 2014 .

[26]  Donald P. Green,et al.  Field Experiments: Design, Analysis, and Interpretation , 2012 .

[27]  Kp Suresh.  An overview of randomization techniques: An unbiased assessment of outcome in clinical research , 2011, Journal of human reproductive sciences.

[28]  Daniel M. Butler,et al.  Moving Beyond Measurement: Adapting Audit Studies to Test Bias-Reducing Interventions , 2017, Journal of Experimental Political Science.

[29]  P. Riach,et al.  Deceptive Field Experiments of Discrimination: Are They Ethical? , 2004 .

[30]  Scott Desposato.  Ethics and experiments : problems and solutions for social scientists and policy professionals , 2015 .

[31]  Stephen L. Ross,et al.  Discrimination in Metropolitan Housing Markets: National Results from Phase 1 of the Housing Discrimination Study (HDS) , 2002 .

[32]  Ryan A. Beasley,et al.  Computerizing Audit Studies. , 2009, Journal of economic behavior & organization.

[33]  Ali Ahmed,et al.  Are gay men and lesbians discriminated against in the hiring process , 2013 .

[34]  Mia Costa,et al.  How Responsive are Political Elites? A Meta-Analysis of Experiments on Public Officials* , 2017, Journal of Experimental Political Science.

[35]  Lina Andersson,et al.  Does age matter for employability? A field experiment on ageism in the Swedish labour market , 2012 .

[36]  S. M. Gaddis,et al.  Arab American Housing Discrimination, Ethnic Competition, and the Contact Hypothesis , 2015 .

[37]  J. Yinger,et al.  What Have We Learned from Paired Testing in Housing Markets , 2015 .

[38]  K. Schoors,et al.  First Depressed, then Discriminated Against? , 2014, SSRN Electronic Journal.

[39]  Benjamin A. Olken,et al.  Promises and Perils of Pre-analysis Plans , 2015 .

[40]  D. Yanow,et al.  Reforming Institutional Review Board Policy: Issues in Implementation and Field Research , 2008, PS: Political Science & Politics.

[41]  S. M. Gaddis.  An Introduction to Audit Studies in the Social Sciences , 2017 .

[42]  David E. Broockman.  Black Politicians Are More Intrinsically Motivated to Advance Blacks’ Interests: A Field Experiment Manipulating Political Incentives , 2013 .

[43]  P. Riach,et al.  Field Experiments of Discrimination in the Market Place , 2002 .