AID: An Automated Detector for Gender-Inclusivity Bugs in OSS Project Pages

The tools and infrastructure used in tech, including Open Source Software (OSS), can embed "inclusivity bugs"- features that disproportionately disadvantage particular groups of contributors. To see whether OSS developers have existing practices to ward off such bugs, we surveyed 266 OSS developers. Our results show that a majority (77%) of developers do not use any inclusivity practices, and 92% of respondents cited a lack of concrete resources to enable them to do so. To help fill this gap, this paper introduces AID, a tool that automates the GenderMag method to systematically find gender-inclusivity bugs in software. We then present the results of the tool's evaluation on 20 GitHub projects. The tool achieved precision of 0.69, recall of 0.92, an F-measure of 0.79 and even captured some inclusivity bugs that human GenderMag teams missed.

[1]  Alexiei Dingli,et al.  USEFul: A Framework to Mainstream Web Site Usability Through Automated Evaluation , 2012 .

[2]  Gayna Williams,et al.  Are you sure your software is gender-neutral? , 2014, INTR.

[3]  Shuming Shi,et al.  CASE: Context-Aware Semantic Expansion , 2020, AAAI.

[4]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[5]  Marilyn Hughes Blackmon,et al.  Automated Cognitive Walkthrough for the Web (AutoCWW) , 2002 .

[6]  Leonard R. Kasday,et al.  A tool to evaluate universal Web accessibility , 2000, CUU '00.

[7]  Margaret M. Burnett,et al.  Gender pluralism in problem-solving software , 2011, Interact. Comput..

[8]  Rick Spencer,et al.  The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company , 2000, CHI.

[9]  Margaret M. Burnett,et al.  Semi-Automating (or not) a Socio-Technical Method for Socio-Technical Systems , 2018, 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[10]  Nicola Marsden,et al.  Cognitive walkthrough of a learning management system with gendered personas , 2018, GenderIT.

[11]  Sven Apel,et al.  Discipline Matters: Refactoring of Preprocessor Directives in the #ifdef Hell , 2018, IEEE Transactions on Software Engineering.

[12]  Peter G. Polson,et al.  An automated cognitive walkthrough , 1991, CHI '91.

[13]  Sean A. Munson,et al.  Unequal Representation and Gender Stereotypes in Image Search Results for Occupations , 2015, CHI.

[14]  Christophe Kolski,et al.  State of the Art on the Cognitive Walkthrough Method, Its Variants and Evolutions , 2010, Int. J. Hum. Comput. Interact..

[15]  Simone Stumpf,et al.  From GenderMag to InclusiveMag: An Inclusive Design Meta-Method , 2019, 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[16]  Clayton Lewis,et al.  AutoCWW : Automated Cognitive Walkthrough for the Web , 2002 .

[17]  Xeniya Kondrat.  Gender and Video Games: How Is Female Gender Generally Represented in Various Genres of Video Games? , 2015 .

[18]  Susan T. Dumais,et al.  Latent Semantic Indexing (LSI) and TREC-2 , 1993, TREC.

[19]  Marco Aurélio Gerosa,et al.  How Gender-Biased Tools Shape Newcomer Experiences in OSS Projects , 2020, IEEE Transactions on Software Engineering.

[20]  Steven Abney,et al.  Part-of-Speech Tagging and Partial Parsing , 1997 .

[21]  Margaret M. Burnett,et al.  GenderMag: A Method for Evaluating Software's Gender Inclusiveness , 2016, Interact. Comput..

[22]  Shaowen Bardzell,et al.  Gender-Inclusive HCI Research and Design: A Conceptual Review , 2020, Found. Trends Hum. Comput. Interact..

[23]  Mary Czerwinski,et al.  Gender differences and programming environments: across programming populations , 2010, ESEM '10.

[24]  Kelly Blincoe,et al.  Understanding the popular users: Following, affiliation influence and leadership on GitHub , 2016, Inf. Softw. Technol..

[25]  Margaret M. Burnett,et al.  Finding Gender-Inclusiveness Software Issues with GenderMag: A Field Investigation , 2016, CHI.

[26]  Melody Y. Ivory.  Web TANGO: towards automated comparison of information-centric web site designs , 2000, CHI Extended Abstracts.

[27]  Robert W. Bowdidge,et al.  Why don't software developers use static analysis tools to find bugs? , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[28]  Steve Young,et al.  Corpus-based methods in language and speech processing , 1997 .

[29]  Gillian Dobbie,et al.  Automated Usability Testing Framework , 2008, AUIC.

[30]  Margaret M. Burnett,et al.  Pedagogical Content Knowledge for Teaching Inclusive Design , 2018, ICER.

[31]  Valentina Grigoreanu,et al.  Informal cognitive walkthroughs (ICW): paring down and pairing up for an agile world , 2013, CHI.

[32]  Y. Raghu Reddy,et al.  Usability Evaluation Framework for Mobile Apps using Code Analysis , 2018, EASE.

[33]  Catarina Gralha,et al.  Analysing Gender Differences in Building Social Goal Models: A Quasi-Experiment , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[34]  Margaret M. Burnett,et al.  Open Source Barriers to Entry, Revisited: A Sociotechnical Perspective , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[35]  Joan Meyers-Levy,et al.  Revisiting gender differences: What we know and what lies ahead☆ , 2015 .

[36]  Margaret M. Burnett,et al.  Gender HCl and microsoft: Highlights from a longitudinal study , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[37]  Sogol Balali,et al.  Recommending Tasks to Newcomers in OSS Projects: How Do Mentors Handle It? , 2020, OpenSym.

[38]  Mary Shaw,et al.  Prospects for an engineering discipline of software , 1990, IEEE Software.

[39]  Cindy Grimm,et al.  Identifying Gender Differences in Information Processing Style, Self-efficacy, and Tinkering for Robot Tele-operation , 2018, 2018 15th International Conference on Ubiquitous Robots (UR).

[40]  A. Bandura.  Social Foundations of Thought and Action , 1986 .

[41]  Igor Steinmacher,et al.  What Attracts Newcomers to Onboard on OSS Projects? TL;DR: Popularity , 2019, OSS.

[42]  David M. Nichols,et al.  Supporting Gender-Neutral Digital Library Creation: A Case Study Using the GenderMag Toolkit , 2016, ICADL.

[43]  Marco Aurélio Gerosa,et al.  Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects , 2015, CSCW.

[44]  Joakim Nivre,et al.  An Efficient Algorithm for Projective Dependency Parsing , 2003, IWPT.

[45]  Lara Letaw,et al.  Engineering Gender-Inclusivity into Software: Ten Teams' Tales from the Trenches , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[46]  Steven E. Stemler.  Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation A Comparison of Consensus, Consistency, and Measurement A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability Approaches to Estimating Interrater Reliabilit , 2022 .

[47]  Andrew Sears,et al.  Heuristic Walkthroughs: Finding the Problems Without the Noise , 1997, Int. J. Hum. Comput. Interact..

[48]  Margaret Burnett,et al.  From Gender Biases to Gender-Inclusive Design: An Empirical Investigation , 2019, CHI.

[49]  Margaret M. Burnett,et al.  Tinkering and gender in end-user programmers' debugging , 2006, CHI.

[50]  James Pustejovsky,et al.  Lexical Knowledge Representation and Natural Language Processing , 1993, Artif. Intell..

[51]  Christopher F. Karpowitz,et al.  Gender Inequality in Deliberative Participation , 2012, American Political Science Review.

[52]  Rachael Tatman,et al.  Gender and Dialect Bias in YouTube’s Automatic Captions , 2017, EthNLP@EACL.

[53]  Margaret M. Burnett,et al.  GenderMag experiences in the field: The whole, the parts, and the workload , 2016, 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[54]  Martha Cleveland-Innes,et al.  Revisiting methodological issues in transcript analysis: Negotiated coding and reliability , 2006, Internet High. Educ..

[55]  Margaret M. Burnett.  "Womenomics" and gender-inclusive software: what software engineers need to know (invited talk) , 2016, SIGSOFT FSE.