How do Practitioners Capture and Utilize User Feedback During Continuous Software Engineering?

Continuous software engineering (CSE) evolved as a process for rapid software evolution. Continuous delivery enables developers to frequently retrieve user feedback on the latest software increment. Developers use these insights for requirements validation and verification. Despite the importance of users, reports about user feedback in CSE practice are sparse. We conducted 20 interviews with practitioners from 17 companies that apply CSE. We asked practitioners how they capture and utilize user feedback. In this paper, we detail the practitioners' answers by posing three research questions. To improve continuous user feedback capture and utilization with respect to requirements engineering, we derived five recommendations: First, internal sources should be approached, as they provide a rich source of user feedback; second, existing tool support should be adapted and extended to automate user feedback processing; third, a concept of reference points should be established to relate user feedback to requirements; fourth, the utilization of user feedback for requirements validation should be increased; and last, the interaction with user feedback should be enabled and supported by increasing developer-user communication. We conclude that a continuous user understanding activity can improve requirements engineering by contributing to both the completeness and correctness of requirements.

[1]  Laura Lehtola,et al.  Investigating the State of User Involvement in Practice , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[2]  Reinhold Plösch,et al.  Operationalised product quality models and assessment: The Quamoco approach , 2014, Inf. Softw. Technol..

[3]  Norbert Seyff,et al.  Providing a User Forum is not enough: First Experiences of a Software Company with CrowdRE , 2017, 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW).

[4]  Xavier Franch,et al.  FAME: Supporting Continuous Requirements Elicitation by Combining User Feedback and Monitoring , 2018, 2018 IEEE 26th International Requirements Engineering Conference (RE).

[5]  Casper Lassenius,et al.  Problems, causes and solutions when adopting continuous delivery - A systematic literature review , 2017, Inf. Softw. Technol..

[6]  Pasi Kuvaja,et al.  Continuous deployment of software intensive products and services: A systematic mapping study , 2017, J. Syst. Softw..

[7]  Bernd Brügge,et al.  User Feedback in Mobile Development , 2014, MobileDeLi '14.

[8]  Austen Rainer,et al.  Case Study Research in Software Engineering - Guidelines and Examples , 2012 .

[9]  Eduard C. Groen,et al.  Users — The Hidden Software Product Quality Experts?: A Study on How App Users Report Quality Aspects in Online Reviews , 2017, 2017 IEEE 25th International Requirements Engineering Conference (RE).

[10]  Klaas-Jan Stol,et al.  Continuous software engineering: A roadmap and agenda , 2017, J. Syst. Softw..

[11]  Jan Ole Johanssen,et al.  Continuous software engineering and its support by usage and decision knowledge: An interview study with practitioners , 2019, J. Softw. Evol. Process..

[12]  Jan Bosch,et al.  Towards Data-Driven Product Development: A Multiple Case Study on Post-deployment Data Usage in Software-Intensive Embedded Systems , 2013, LESS.

[13]  Jan Bosch,et al.  Continuous Software Engineering: An Introduction , 2014, Continuous Software Engineering.

[14]  Janis A. Bubenko,et al.  Challenges in requirements engineering , 1995, Proceedings of 1995 IEEE International Symposium on Requirements Engineering (RE'95).

[15]  Richard Baskerville,et al.  Agile requirements engineering practices and challenges: an empirical study , 2007, Inf. Syst. J..

[16]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[17]  Qin Liu,et al.  Task Analysis-Based User Event Logging for Mobile Applications , 2017, 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW).

[18]  Maleknaz Nayebi,et al.  Toward Data-Driven Requirements Engineering , 2016, IEEE Software.

[19]  Jan Ole Johanssen,et al.  Decision Knowledge Triggers in Continuous Software Engineering , 2018, 2018 IEEE/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE).

[20]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[21]  Amy J. Ko,et al.  A case study of post-deployment user feedback triage , 2011, CHASE.

[22]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[23]  Bernd Brügge,et al.  Rugby: an agile process model based on continuous delivery , 2014, RCoSE 2014.

[24]  Liming Zhu,et al.  Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices , 2017, IEEE Access.

[25]  Mathias Meyer,et al.  Continuous Integration and Its Tools , 2014, IEEE Software.

[26]  Romain Rouvoy,et al.  App Store 2.0: From Crowdsourced Information to Actionable Feedback in Mobile Ecosystems , 2017, IEEE Software.

[27]  Kurt Matzler,et al.  THE KANO MODEL: HOW TO DELIGHT YOUR CUSTOMERS , 1996 .

[28]  Walid Maalej,et al.  How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews , 2014, 2014 IEEE 22nd International Requirements Engineering Conference (RE).

[29]  Jan Ole Johanssen,et al.  How do Practitioners Manage Decision Knowledge during Continuous Software Engineering? (S) , 2019, SEKE.

[30]  Brian Fitzgerald,et al.  The ABC of Software Engineering Research , 2018, ACM Trans. Softw. Eng. Methodol..

[31]  Björn Regnell,et al.  Challenges in Market-Driven Requirements Engineering - an Industrial Interview Study , 2002 .

[32]  Jan Ole Johanssen,et al.  Feature Crumbs: Adapting Usage Monitoring to Continuous Software Engineering , 2018, PROFES.

[33]  Stephan Krusche,et al.  Context-Aware User Feedback in Continuous Software Evolution , 2016, 2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED).

[34]  Jan Ole Johanssen,et al.  Continuous Thinking Aloud , 2019, 2019 IEEE/ACM Joint 4th International Workshop on Rapid Continuous Software Engineering and 1st International Workshop on Data-Driven Decisions, Experimentation and Evolution (RCoSE/DDrEE).

[35]  Lin Ma,et al.  Unpacking Tacit Knowledge for Requirements Engineering , 2013, Managing Requirements Knowledge.

[36]  Jan Ole Johanssen,et al.  Practitioners' eye on continuous software engineering: an interview study , 2018, ICSSP.

[37]  Alessia Knauss On the usage of context for requirements elicitation: End-user involvement in IT ecosystems , 2012, 2012 20th IEEE International Requirements Engineering Conference (RE).

[38]  Jan Bosch,et al.  Modeling continuous integration practice differences in industry software development , 2014, J. Syst. Softw..

[39]  Marco Torchiano,et al.  Six reasons for rejecting an industrial survey paper , 2013, 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI).

[40]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[41]  Norbert Seyff,et al.  Feedback Gathering from an Industrial Point of View , 2017, 2017 IEEE 25th International Requirements Engineering Conference (RE).

[42]  Lianping Chen,et al.  The Journal of Systems and Software , 2022 .

[43]  Philipp Leitner,et al.  Continuous Experimentation: Challenges, Implementation Techniques, and Current Research , 2018, IEEE Software.

[44]  Mark Keil,et al.  The user–developer communication process: a critical case study , 2003, Inf. Syst. J..

[45]  Michael D. Myers,et al.  The qualitative interview in IS research: Examining the craft , 2007, Inf. Organ..

[46]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[47]  Xavier Franch,et al.  Data-Driven Requirements Engineering in Agile Projects: The Q-Rapids Approach , 2017, 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW).

[48]  Anind K. Dey Enabling the use of context in interactive applications , 2000, CHI Extended Abstracts.

[49]  Jan Ole Johanssen,et al.  Toward Usability Problem Identification Based on User Emotions Derived from Facial Expressions , 2019, 2019 IEEE/ACM 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion).

[50]  Bernd Brügge,et al.  User involvement in software evolution practice: A case study , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[51]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[52]  Walid Maalej,et al.  When users become collaborators: towards continuous and context-aware user input , 2009, OOPSLA Companion.