Analytics for Product Planning: In-Depth Interview Study with SaaS Product Managers

SaaS cloud computing, in contrast to packaged products, enables permanent contact between users of a software product and the product-owning company. When planning the development and evolution of a software product, a product manager depends on reliable information about feature attractiveness. So far, planning decisions were based on stakeholder opinion and the customer's willingness to buy. Whether or not a feature actually is used was out of consideration. Analytics that measure the interaction between users and the SaaS gives product managers unprecedented access to information about product usage. To understand whether and how SaaS analytics can be used for product planning decision, we performed 17 in-depth interviews with experienced managers of SaaS products and analyzed the results analyzed with a mixed-method strategy. The empirical results characterize the relevance of a broad range of analytics for product planning decisions, and the strengths and limitations of an analytics-based product planning approach.

[1]  Ameeta Soni,et al.  Successfully launching your product: getting it right , 2004 .

[2]  Slinger Jansen,et al.  The Agile Requirements Refinery: Applying SCRUM Principles to Software Product Management , 2009, 2009 Third International Workshop on Software Product Management.

[3]  Andreas Holzinger,et al.  Usability engineering methods for software developers , 2005, CACM.

[4]  Avinash Kaushik,et al.  Web Analytics: An Hour a Day , 2007 .

[5]  Ian Thomas,et al.  Business-Driven Product Planning Using Feature Vectors and Increments , 2002, IEEE Softw..

[6]  Hakan Erdogmus,et al.  Cloud Computing: Does Nirvana Hide behind the Nebula? , 2009, IEEE Softw..

[7]  Sungjoo Lee,et al.  Technology roadmapping for R&D planning: The case of the Korean parts and materials industry , 2007 .

[8]  Daniel P. Siewiorek,et al.  A resource allocation model for QoS management , 1997, Proceedings Real-Time Systems Symposium.

[9]  Neal Leavitt,et al.  Is Cloud Computing Really Ready for Prime Time? , 2009, Computer.

[10]  Daniel A. Menascé,et al.  QoS Issues in Web Services , 2002, IEEE Internet Comput..

[11]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[12]  Inge van de Weerd,et al.  Developing a Maturity Matrix for Software Product Management , 2010, ICSOB.

[13]  Martin Höst,et al.  An analytical model for requirements selection quality evaluation in product software development , 2003, Proceedings. 11th IEEE International Requirements Engineering Conference, 2003..

[14]  H. Kittlaus,et al.  Software Product Management and Pricing: Key Success Factors for Software Organizations , 2009 .

[15]  Avinash Kaushik,et al.  Web Analytics 2.0: The Art of Online Accountability and Science of Customer Centricity , 2009 .

[16]  Hans Christian Benestad,et al.  A comparison of model-based and judgment-based release planning in incremental software projects , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[17]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[18]  C. Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[19]  Christof Ebert,et al.  The impacts of software product management , 2007, J. Syst. Softw..

[20]  Günther Ruhe,et al.  The art and science of software release planning , 2005, IEEE Software.

[21]  Ville Heikkilä,et al.  Rigorous Support for Flexible Planning of Product Releases - A Stakeholder-Centric Approach and Its Initial Evaluation , 2010, 2010 43rd Hawaii International Conference on System Sciences.

[22]  Mike Kuniavsky,et al.  Observing the User Experience: A Practitioner's Guide to User Research (Morgan Kaufmann Series in Interactive Technologies) (The Morgan Kaufmann Series in Interactive Technologies) , 2003 .

[23]  Günther Ruhe,et al.  Ad Hoc Versus Systematic Planning of Software Releases - A Three-Staged Experiment , 2006, PROFES.

[24]  Brian Clifton,et al.  Advanced Web Metrics with Google Analytics , 2008 .

[25]  Inge van de Weerd,et al.  The Influence of Situational Factors in Software Product Management: An Empirical Study , 2008, 2008 Second International Workshop on Software Product Management.

[26]  Claes Wohlin,et al.  A product management challenge: Creating software product value through requirements selection , 2008, J. Syst. Archit..

[27]  Markus Fiedler,et al.  Quality of Experience from user and network perspectives , 2010, Ann. des Télécommunications.

[28]  Günther Ruhe,et al.  Release planning process improvement - an industrial case study , 2006, Softw. Process. Improv. Pract..

[29]  Günther Ruhe,et al.  Supporting Software Release Planning Decisions for Evolving Systems , 2005, 29th Annual IEEE/NASA Software Engineering Workshop.

[30]  David Probert,et al.  Strategic Roadmapping: A Workshop-based Approach for Identifying and Exploring Strategic Issues and Opportunities , 2007 .

[31]  Ranjit Bose,et al.  Advanced analytics: opportunities and challenges , 2009, Ind. Manag. Data Syst..

[32]  Du Wan Cheun,et al.  A Quality Model for Evaluating Software-as-a-Service in Cloud Computing , 2009, 2009 Seventh ACIS International Conference on Software Engineering Research, Management and Applications.

[33]  Ron Kohavi,et al.  Emerging trends in business analytics , 2002, CACM.

[34]  Michael A. Cusumano,et al.  The Changing Software Business: Moving from Products to Services , 2008, Computer.

[35]  Neil A. M. Maiden,et al.  Power and politics in requirements engineering: embracing the dark side? , 2012, Requirements Engineering.

[36]  R. Feldt,et al.  A systematic review on strategic release planning models , 2010, Inf. Softw. Technol..

[37]  Alistair Croll,et al.  Complete Web monitoring , 2009 .

[38]  Bojan Cukic,et al.  Log-Based Reliability Analysis of Software as a Service (SaaS) , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[39]  M. Houtermans,et al.  Reliability Engineering & Data Collection , 2007, Second International Conference on Systems (ICONS'07).

[40]  J. Jiao,et al.  Product portfolio planning with customer-engineering interaction , 2005 .

[41]  Günther Ruhe,et al.  A systematic approach for solving the wicked problem of software release planning , 2007, Soft Comput..

[42]  R. Cooper,et al.  New Product Portfolio Management : Practices and Performance , 1999 .

[43]  Günther Ruhe,et al.  Lightweight Replanning of Software Product Releases , 2006, 2006 International Workshop on Software Product Management (IWSPM'06 - RE'06 Workshop).

[44]  Dietmar Pfahl,et al.  Operational Planning, Re-planning and Risk Analysis for Software Releases , 2007, PROFES.

[45]  Jason Burby,et al.  Actionable Web Analytics: Using Data to Make Smart Business Decisions , 2007 .

[46]  Samuel Fricker,et al.  Release Planning with Feature Trees: Industrial Case , 2012, REFSQ.

[47]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[48]  Richard A. Parker,et al.  Designing and Conducting Survey Research: A Comprehensive Guide , 1992 .

[49]  Vidyanand Choudhary,et al.  Software as a Service: Implications for Investment in Software Development , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[50]  Anne H. H. Ngu,et al.  QoS computation and policing in dynamic web service selection , 2004, WWW Alt. '04.

[51]  Inge van de Weerd,et al.  Towards a Reference Framework for Software Product Management , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[52]  Björn Regnell,et al.  Overscoping: Reasons and consequences — A case study on decision making in software product management , 2010, 2010 Fourth International Workshop on Software Product Management.

[53]  Mike Kuniavsky,et al.  Observing the User Experience: A Practitioner's Guide to User Research (Second Edition) , 2013, IEEE Transactions on Professional Communication.

[54]  Thomas Zimmermann,et al.  Analytics for software development , 2010, FoSER '10.

[55]  Koo Ping Shung,et al.  APPLICATION OF ANALYTICS IN BUSINESS STRATEGY , 2012 .

[56]  Björn Regnell,et al.  Introducing Tool Support for Retrospective Analysis of Release Planning Decisions , 2006, PROFES.

[57]  O. H. Bray,et al.  Fundamentals of technology roadmapping , 1997 .

[58]  Laura Johnson,et al.  How Many Interviews Are Enough? , 2006 .

[59]  Antony Tang,et al.  Building roadmaps: a knowledge sharing perspective , 2011, SHARK '11.

[60]  Björn Regnell,et al.  Investigating Upstream versus Downstream Decision-Making in Software Product Management , 2009, 2009 Third International Workshop on Software Product Management.

[61]  Björn Regnell,et al.  Exploring Software Product Management decision problems with constraint solving - opportunities for prioritization and release planning , 2011, 2011 Fifth International Workshop on Software Product Management (IWSPM).

[62]  Tony Gorschek,et al.  Requirements Abstraction Model , 2005, Requirements Engineering.

[63]  Casper Lassenius,et al.  An Approach to Product Roadmapping in Small Software Product Businesses , 2002 .

[64]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .