Transparency of CHI Research Artifacts: Results of a Self-Reported Survey

Several fields of science are experiencing a “replication crisis” that has negatively impacted their credibility. Assessing the validity of a contribution via replicability of its experimental evidence and reproducibility of its analyses requires access to relevant study materials, data, and code. Failing to share them limits the ability to scrutinize or build-upon the research, ultimately hindering scientific progress. Understanding how the diverse research artifacts in HCI impact sharing can help produce informed recommendations for individual researchers and policy-makers in HCI. Therefore, we surveyed authors of CHI 2018–2019 papers, asking if they share their papers’ research materials and data, how they share them, and why they do not. The results (34% response rate) show that sharing is uncommon, partly due to misunderstandings about the purpose of sharing and reliable hosting. We conclude with recommendations for fostering open research practices. This paper and all data and materials are freely available at https://osf.io/3bu6t. Author

[1]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[2]  Erik Schultes,et al.  The FAIR Guiding Principles for scientific data management and stewardship , 2016, Scientific Data.

[3]  Jeffrey Nichols,et al.  RepliCHI SIG: from a panel to a new submission venue for replication , 2012, CHI Extended Abstracts.

[4]  E. Wagenmakers,et al.  Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). , 2011, Journal of personality and social psychology.

[5]  Stephan Lewandowsky,et al.  The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review , 2016, Royal Society Open Science.

[6]  Florence Debarre,et al.  The Availability of Research Data Declines Rapidly with Article Age , 2013, Current Biology.

[7]  B. Englich,et al.  No Replication, No Trust? How Low Replicability Influences Trust in Psychology , 2020, Social Psychological and Personality Science.

[8]  Jamie Donatuto,et al.  Conducting Research with Tribal Communities: Sovereignty, Ethics, and Data-Sharing Issues , 2011, Environmental health perspectives.

[9]  Tamarinde L Haven,et al.  Preregistering qualitative research , 2019, Accountability in research.

[10]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[11]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[12]  Brad Novotny,et al.  Working Party on the Protection of Individuals with regard to the Processing of Personal Data , 2015 .

[13]  Dennis McCafferty.  Should code be released? , 2010, Commun. ACM.

[14]  Jakob Grue Simonsen,et al.  Is once enough?: on the extent and content of replications in human-computer interaction , 2014, CHI.

[15]  Michael C. Frank,et al.  Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition , 2018, Royal Society Open Science.

[16]  Michael B. Yaffe,et al.  Reproducibility in science , 2015, Science Signaling.

[17]  Adrian Friday,et al.  Evaluating HCI Research beyond Usability , 2018, CHI Extended Abstracts.

[18]  Shion Guha,et al.  Moving Transparent Statistics Forward at CHI , 2017, CHI Extended Abstracts.

[19]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[20]  Shion Guha,et al.  Special Interest Group on Transparent Statistics in HCI , 2016, CHI Extended Abstracts.

[21]  Marilyn Tremaine.  CHI '01 Extended Abstracts on Human Factors in Computing Systems , 2001, CHI Extended Abstracts.

[22]  Christopher D. Chambers,et al.  Transparency and Openness Promotion (TOP) Guidelines , 2014 .

[23]  C. Haeussler.  Information-Sharing in Academia and the Industry: A Comparative Study , 2010 .

[24]  Robbie C. M. van Aert,et al.  Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking , 2016, Front. Psychol..

[25]  Lewis L. Chuang,et al.  Transparency and Openness Promotion Guidelines for HCI , 2018, CHI Extended Abstracts.

[26]  Steve Haroz,et al.  Open Practices in Visualization Research : Opinion Paper , 2018, 2018 IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV).

[27]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[28]  Prasad Patil,et al.  A statistical definition for reproducibility and replicability , 2016, bioRxiv.

[29]  Shion Guha,et al.  Qualitative Methods for CSCW: Challenges and Opportunities , 2019, CSCW Companion.

[30]  C. Ferguson.  "Everybody knows psychology is not a real science": Public perceptions of psychology and how we can improve our relationship with policymakers, the scientific community, and the general public. , 2015, The American psychologist.

[31]  Eric W. D. Rozier,et al.  Reproducibility, correctness, and buildability: The three principles for ethical public dissemination of computer science and engineering research , 2014, 2014 IEEE International Symposium on Ethics in Science, Technology and Engineering.

[32]  Carl Gutwin,et al.  HARK No More: On the Preregistration of CHI Experiments , 2018, CHI.

[33]  Stefan Schutt,et al.  The Challenge of Technology Research in Sensitive Settings: Case Studies in 'ensitive HCI' , 2015, OZCHI.

[34]  Michael S. Bernstein,et al.  RepliCHI - CHI should be replicating and validating results more: discuss , 2011, CHI Extended Abstracts.

[35]  C. Tenopir,et al.  Data Sharing by Scientists: Practices and Perceptions , 2011, PloS one.

[36]  Michelle N. Meyer,et al.  Practical Tips for Ethical Data Sharing , 2018 .

[37]  Florian Echtler,et al.  Open Source, Open Science, and the Replication Crisis in HCI , 2018, CHI Extended Abstracts.

[38]  H. Waitzkin,et al.  On Studying the Discourse of Medical Encounters: A Critique of Quantitative and Qualitative Methods and a Proposal for Reasonable Compromise , 1990, Medical care.

[39]  Susann Fiedler,et al.  Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency , 2016, PLoS biology.

[40]  Celia Kitzinger,et al.  Anonymising interview data: challenges and compromise in practice , 2015, Qualitative research : QR.

[41]  Pawel W. Wozniak,et al.  Designing for Reproducibility: A Qualitative Study of Challenges and Opportunities in High Energy Physics , 2019, CHI.

[42]  D. Borsboom,et al.  The poor availability of psychological research data for reanalysis. , 2006, The American psychologist.

[43]  Hans Ekkehard Plesser,et al.  Reproducibility vs. Replicability: A Brief History of a Confused Terminology , 2018, Front. Neuroinform..

[44]  Sameer Patil,et al.  Local Standards for Anonymization Practices in Health, Wellness, Accessibility, and Aging Research at CHI , 2019, CHI.

[45]  Stuart Reeves,et al.  RepliCHI: the workshop II , 2014, CHI Extended Abstracts.

[46]  Eric-Jan Wagenmakers,et al.  Data Sharing in Psychology: A Survey on Barriers and Preconditions , 2018 .

[47]  G. Nilsonne,et al.  Data sharing in qualitative research : opportunities and concerns , 2018 .

[48]  Neil Malhotra,et al.  Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results , 2015, Political Analysis.

[49]  Nick Barnes.  Publish your computer code: it is good enough , 2010, Nature.

[50]  Randall J. LeVeque,et al.  Top Ten Reasons to Not Share Your Code ( and why you should anyway ) , 2011 .

[51]  O WobbrockJacob,et al.  Research contributions in human-computer interaction , 2016 .

[52]  Shion Guha,et al.  Special Interest Group on Transparent Statistics Guidelines , 2018, CHI Extended Abstracts.

[53]  A. Treloar,et al.  Open Data in Global Environmental Research: The Belmont Forum’s Open Data Survey , 2016, PloS one.

[54]  Max L. Wilson,et al.  RepliCHI: the workshop , 2013, CHI Extended Abstracts.

[55]  Lex Nederbragt,et al.  Good enough practices in scientific computing , 2016, PLoS Comput. Biol..

[56]  Stuart Barber,et al.  Choosing a Method to Reduce Selection Bias: A Tool for Researchers , 2015 .

[57]  Benedikt Fecher,et al.  What Drives Academic Data Sharing? , 2014, PloS one.

[58]  James Fogarty,et al.  Code and Contribution in Interactive Systems Research , 2017 .