Do Employers “Walk the Talk” After All? An Illustration of Methods for Assessing Signals in Underpowered Designs

This article presents two alternative methods to null hypothesis significance testing (NHST) for improving inferences from underpowered research designs. Post hoc design analysis (PHDA) assesses wh...

[1]  Klaus Fiedler,et al.  The Long Way From α-Error Control to Validity Proper , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[2]  Colin Jerolmack,et al.  Talk Is Cheap , 2014 .

[3]  M. G. Pittau,et al.  A weakly informative default prior distribution for logistic and other regression models , 2008, 0901.4011.

[4]  Bruce Western,et al.  Bayesian Analysis for Sociologists , 1999 .

[5]  G. Cumming.  The New Statistics: Why and How , 2013 .

[6]  Sander Greenland,et al.  Null misinterpretation in statistical testing and its impact on health risk assessment. , 2011, Preventive medicine.

[7]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[8]  Torrin M. Liddell,et al.  Bayesian data analysis for newcomers , 2018, Psychonomic bulletin & review.

[9]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[10]  S. Reich.  An exception to the rule: Belief in redeemability, desistance signals, and the employer’s decision to hire a job applicant with a criminal record , 2017 .

[11]  Min-Sun Kim,et al.  Attitude-Behavior Relations: A Meta-Analysis of Attitudinal Relevance and Topic , 1993 .

[12]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[13]  J. Berger.  The case for objective Bayesian analysis , 2006 .

[14]  Gary King,et al.  Logistic Regression in Rare Events Data , 2001, Political Analysis.

[15]  Brian A. Nosek,et al.  An open investigation of the reproducibility of cancer biology research , 2014, eLife.

[16]  Sander Greenland,et al.  Nonsignificance plus high power does not imply support for the null over the alternative. , 2012, Annals of epidemiology.

[17]  C. Ferguson,et al.  A Vast Graveyard of Undead Theories , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[18]  L. Oakes.  Sample size, statistical power, and false conclusions in infant looking-time research. , 2017, Infancy : the official journal of the International Society on Infant Studies.

[19]  John K Kruschke,et al.  Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[20]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[21]  Ernest C Davenport,et al.  Phi/Phimax: Review and Synthesis , 1991 .

[22]  Daniele Fanelli,et al.  Negative results are disappearing from most disciplines and countries , 2011, Scientometrics.

[23]  S. Goodman,et al.  Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations , 2016, European Journal of Epidemiology.

[24]  D. Heisey,et al.  The Abuse of Power , 2001 .

[25]  Jeremy Freese,et al.  Replication Standards for Quantitative Social Science , 2007 .

[26]  D. Lakens,et al.  Sailing From the Seas of Chaos Into the Corridor of Stability , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[27]  C. Uggen,et al.  Statistical Power in Experimental Audit Studies , 2016 .

[28]  Yan Ma,et al.  Meta-analysis of Proportions of Rare Events–A Comparison of Exact Likelihood Methods with Robust Variance Estimation , 2016, Commun. Stat. Simul. Comput..

[29]  Stephen Vaisey.  The “Attitudinal Fallacy” Is a Fallacy , 2014 .

[30]  J. Freese.  Overcoming Objections to Open-Source Social Science , 2007 .

[31]  S. Maxwell,et al.  Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? , 2015, The American psychologist.

[32]  J. Gill.  The Insignificance of Null Hypothesis Significance Testing , 1999 .

[33]  Jonathan J Deeks,et al.  Much ado about nothing: a comparison of the performance of meta‐analytical methods with rare events , 2007, Statistics in medicine.

[34]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[35]  Yiming Wang,et al.  Assessing Regional Cerebral Blood Flow in Depression Using 320-Slice Computed Tomography , 2014, PloS one.

[36]  D. Pager,et al.  Walking the Talk? What Employers Say Versus What They Do , 2005 .

[37]  D. Rosenthal,et al.  Quality-space theory in olfaction , 2014, Front. Psychol..

[38]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[39]  D. Pager.  The Mark of a Criminal Record1 , 2003, American Journal of Sociology.

[40]  Franz J. Neyer,et al.  A Gentle Introduction to Bayesian Analysis: Applications to Developmental Research , 2013, Child development.

[41]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[42]  J. Freese,et al.  Replication in Social Science , 2017 .

[43]  Miguel A. Vadillo,et al.  Underpowered samples, false negatives, and unconscious learning , 2015, Psychonomic bulletin & review.

[44]  L. Brown,et al.  Interval Estimation for a Binomial Proportion , 2001 .

[45]  S. J. Kraus,et al.  Attitudes and the Prediction of Behavior: A Meta-Analysis of the Empirical Literature , 1990 .

[46]  Daniel James O'Keefe,et al.  Brief Report: Post Hoc Power, Observed Power, A Priori Power, Retrospective Power, Prospective Power, Achieved Power: Sorting Out Appropriate Uses of Statistical Power Analyses , 2007 .

[47]  K. McKelvey,et al.  Robust Detection of Rare Species Using Environmental DNA: The Importance of Primer Specificity , 2013, PloS one.

[48]  Yuk Lee,et al.  Correlation Coefficients for Binary Data In Factor Analysis , 2010 .

[49]  C. R. Tittle.  The Arrogance of Public Sociology , 2004 .

[50]  Torrin M. Liddell,et al.  The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective , 2016, Psychonomic bulletin & review.

[51]  Paul A Murtaugh,et al.  In defense of P values. , 2014, Ecology.

[52]  D. Albarracín,et al.  Forming attitudes that predict future behavior: a meta-analysis of the attitude-behavior relation. , 2006, Psychological bulletin.

[53]  Nicholas Occhiuto.  Investing in Independent Contract Work , 2017 .

[54]  J. Carlin,et al.  Beyond Power Calculations , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[55]  B. Jasny,et al.  Again, and Again, and Again … , 2011 .

[56]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[57]  Malay Ghosh,et al.  Objective Priors: An Introduction for Frequentists , 2011, 1108.2120.