How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow

To ensure the quality of its shared knowledge, Stack Overflow encourages users to revise answers through a badge system, which is based on quantitative measures (e.g., a badge is awarded after revising more than 500 answers). Prior studies show that badges can positively steer the user behavior on Stack Overflow (e.g., increasing user participation). However, little is known whether revision-related badges have a negative impact on the quality of revisions since some studies show that certain users may game incentive systems to gain rewards. In this study, we analyze 3,871,966 revision records that are collected from 2,377,692 Stack Overflow answers. We find that: 1) Users performed a much larger than usual revisions on the badge-awarding days compared to normal days; 25% of the users did not make any more revisions once they received their first revision-related badge. 2) Performing more revisions than usual in a single day increased the likelihood of such revisions being rolled back (e.g., due to undesired or incorrect revisions). 3) Users were more likely to perform text and small revisions if they performed many revisions in a single day. Our findings are concurred by the Stack Overflow community, and they highlight the need for changes to the current badge system in order to provide a better balance between the quality and quantity of revisions.

[1]  Donald J. Wheeler,et al.  Understanding Statistical Process Control , 1986 .

[2]  Ee-Peng Lim,et al.  Measuring article quality in wikipedia: models and evaluation , 2007, CIKM '07.

[3]  Gilad Mishne,et al.  Finding high-quality content in social media , 2008, WSDM '08.

[4]  Huseyin Cavusoglu,et al.  Can Gamification Motivate Voluntary Contributions?: The Case of StackOverflow Q&A Community , 2015, CSCW Companion.

[5]  K. Gwet.  Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity , 2002 .

[6]  Nicole Novielli,et al.  Mining Successful Answers in Stack Overflow , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[7]  Julie Jones,et al.  We don't need no stinkin' badges: examining the social role of badges in the Huffington Post , 2012, CSCW '12.

[8]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[9]  R. Tibshirani,et al.  Improvements on Cross-Validation: The 632+ Bootstrap Method , 1997 .

[10]  Aniket Kittur,et al.  Harnessing the wisdom of crowds in wikipedia: quality through coordination , 2008, CSCW.

[11]  James D. Herbsleb,et al.  Social coding in GitHub: transparency and collaboration in an open software repository , 2012, CSCW.

[12]  Chris Parnin,et al.  "We Don't Do That Here": How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities , 2018, CHI.

[13]  Chanchal Kumar Roy,et al.  Answering questions about unanswered questions of Stack Overflow , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[14]  Jure Leskovec,et al.  Steering user behavior with badges , 2013, WWW.

[15]  Paul A. Watters,et al.  Statistics in a nutshell - a desktop quick reference , 2008 .

[16]  Shaul Fox,et al.  Escalation Behavior as a Specific Case of Goal-Directed Activity: A Persistence Paradigm , 2002 .

[17]  Feng Xu,et al.  Want a Good Answer? Ask a Good Question First! , 2013, ArXiv.

[18]  Feng Xu,et al.  Detecting high-quality posts in community question answering sites , 2015, Inf. Sci..

[19]  Ahmed E. Hassan,et al.  An empirical study of dormant bugs , 2014, MSR 2014.

[20]  Gerald Penn,et al.  Collaborative editing for improved usefulness and usability of transcript-enhanced webcasts , 2008, CHI.

[21]  E. Churchill,et al.  Badges in Social Media: A Social Psychological Perspective , 2011 .

[22]  Michele Lanza,et al.  Improving Low Quality Stack Overflow Post Detection , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[23]  Gang Wang,et al.  Towards Monetary Incentives in Social Q&A Services , 2017 .

[24]  Scott Grant,et al.  Encouraging user behaviour with achievements: An empirical study , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[25]  Eirini Kalliamvakou,et al.  Open Source-Style Collaborative Development Practices in Commercial Projects Using GitHub , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[26]  Rebecca Shields,et al.  Digital badges – rewards for learning? , 2017, Education and Information Technologies.

[27]  Haiyi Zhu,et al.  Is It Good to Be Like Wikipedia?: Exploring the Trade-offs of Introducing Collaborative Editing Model to Q&A Sites , 2015, CSCW.

[28]  Alberto Bacchelli,et al.  Quality Questions Need Quality Code: Classifying Code Fragments on Stack Overflow , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[29]  Michael W. Godfrey,et al.  An Empirical Study on the Practice of Maintaining Object-Relational Mapping Code in Java Systems , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[30]  Forrest Shull,et al.  Defect categorization: making use of a decade of widely varying historical data , 2008, ESEM '08.

[31]  Sheizaf Rafaeli,et al.  Predictors of answer quality in online Q&A sites , 2008, CHI.

[32]  Gang Wang,et al.  Analyzing Payment Based Question and Answering Service , 2017, ArXiv.

[33]  Alberto Bacchelli,et al.  Expectations, outcomes, and challenges of modern code review , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[34]  Lei Li,et al.  Answer Quality Characteristics and Prediction on an Academic Q&A Site: A Case Study on ResearchGate , 2015, WWW.

[35]  Chanchal Kumar Roy,et al.  An Insight into the Unresolved Questions at Stack Overflow , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[36]  Nathaniel Hudson,et al.  Understanding triggers for clarification requests in community-based software help forums , 2015, 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[37]  Ahmed E. Hassan,et al.  Understanding the factors for fast answers in technical Q&A websites , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[38]  Michele Lanza,et al.  Understanding and Classifying the Quality of Technical Forum Questions , 2014, 2014 14th International Conference on Quality Software.

[39]  Nicole Novielli,et al.  How to ask for technical help? Evidence-based guidelines for writing questions on Stack Overflow , 2017, Inf. Softw. Technol..

[40]  Robert E. Kraut,et al.  Why pay?: exploring how financial incentives are used for question & answer , 2010, CHI.

[41]  Kalina Yacef,et al.  Collaborative Writing Support Tools on the Cloud , 2011, IEEE Transactions on Learning Technologies.

[42]  Christoph Treude,et al.  How do programmers ask and answer questions on the web?: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[43]  Paul Benjamin Lowry,et al.  A Taxonomy of Collaborative Writing to Improve Empirical Research, Writing Practice, and Tool Development , 2004 .

[44]  Ahmed E. Hassan,et al.  Automated Verification of Load Tests Using Control Charts , 2011, 2011 18th Asia-Pacific Software Engineering Conference.

[45]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..