Closing the Loop: Evaluating a Measurement Instrument for Maturity Model Design

To support the systematic improvement of business intelligence (BI) in organizations, we have designed and refined a BI maturity model (BIMM) and a respective measurement instrument (MI) in prior research. In this study, we devise an evaluation strategy, and evaluate the validity of the designed measurement artifact. Through cluster analysis of maturity assessments of 92 organizations, we identify characteristic BI maturity scenarios and representative cases for the relevant scenarios. For evaluating the designed instrument, we compare its results with insights obtained from in-depth interviews in the respective companies. A close match between our model's quantitative maturity assessments and the maturity levels from the qualitative analyses indicates that the MI correctly assesses BI maturity. The applied evaluation approach has the potential to be re-used in other design research studies where the validity of utility claims is often hard to prove.

[1]  R. Nolan,et al.  Managing the Four Stages of EDP Growth , 1974 .

[2]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[3]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[4]  Linda S. Lotto.  Qualitative Data Analysis: A Sourcebook of New Methods , 1986 .

[5]  RICHARD C. DUBES,et al.  How many clusters are best? - An experiment , 1987, Pattern Recognit..

[6]  David W. Gerbing,et al.  An Updated Paradigm for Scale Development Incorporating Unidimensionality and Its Assessment , 1988 .

[7]  Robert K. Kazanjian,et al.  An empirical test of a stage of growth progression model , 1989 .

[8]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[9]  Mark C. Paulk,et al.  Capability Maturity Model , 1991 .

[10]  E. Moak,et al.  The evolution of information management , 1993 .

[11]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[12]  Thilini Ariyachandra,et al.  Data Warehousing Stages of Growth , 2001, Inf. Syst. Manag..

[13]  J Crawford,et al.  Project Management Maturity Model , 2001 .

[14]  C. Fox,et al.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2001 .

[15]  Hisham M. Haddad,et al.  A survey of industrial experiences with CMM and the teaching of CMM practices , 2002 .

[16]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[17]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[18]  Omar A. El Sawy,et al.  Assessing Information System Design Theory in Perspective: How Useful Was our 1992 Initial Rendition? , 2004 .

[19]  James E. Cates,et al.  The Ladder of Business Intelligence (LOBI): a framework for enterprise IT planning and architecture , 2005, Int. J. Bus. Inf. Syst..

[20]  Jerry N. Luftman.  Key Issues for IT Executives 2004 , 2005, MIS Q. Executive.

[21]  Michael Rosemann,et al.  Understanding the Main Phases of Developing a Maturity Assessment Model , 2005 .

[22]  Steve Williams,et al.  The Profit Impact of Business Intelligence , 2006 .

[23]  Atish P. Sinha,et al.  Data Warehousing Process Maturity: An Exploratory Study of Factors Influencing User Perceptions , 2006, IEEE Transactions on Engineering Management.

[24]  Jochen Töpfer.  Active Enterprise Intelligence , 2008 .

[25]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[26]  Jörg Becker,et al.  Developing Maturity Models for IT Management , 2009, Bus. Inf. Syst. Eng..

[27]  Marco R. Spruit,et al.  BIDM - The Business Intelligence Development Model , 2010, ICEIS.

[28]  Marlies van Steenbergen,et al.  The Design of Focus Area Maturity Models , 2010, DESRIST.

[29]  Robert Winter,et al.  Business Intelligence Maturity Models: An Overview , 2010 .

[30]  Barbara Wixom,et al.  The BI-Based Organization , 2010, Int. J. Bus. Intell. Res..

[31]  Min-Hooi Chuah.  An enterprise business intelligence maturity model (EBIMM): Conceptual framework , 2010, 2010 Fifth International Conference on Digital Information Management (ICDIM).

[32]  Björn Niehaves,et al.  Maturity Models in Information Systems Research: Literature Search and Analysis , 2011, Commun. Assoc. Inf. Syst..

[33]  Zahir Irani,et al.  Business Intelligence Maturity: The Economic Transitional Context Within Slovenia , 2011, Inf. Syst. Manag..

[34]  Robert Winter,et al.  Using Quantitative Analyses to Construct a Capability Maturity Model for Business Intelligence , 2012, 2012 45th Hawaii International Conference on System Sciences.

[35]  Atish P. Sinha,et al.  A Model of Data Warehousing Process Maturity , 2012, IEEE Transactions on Software Engineering.

[36]  Jan vom Brocke,et al.  Evaluations in the Science of the Artificial - Reconsidering the Build-Evaluate Pattern in Design Science Research , 2012, DESRIST.

[37]  Jan Pries-Heje,et al.  A Comprehensive Framework for Evaluation in Design Science Research 1 , 2022 .

[38]  David Raber.  Evaluation of a Business Intelligence Maturity Model: The Reliability Perspective , 2013 .

[39]  Robert Winter,et al.  Towards The Measurement Of Business Intelligence Maturity , 2013, ECIS.