Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems

Mixed methods research is an approach that combines quantitative and qualitative research methods in the same research inquiry. Such work can help develop rich insights into various phenomena of interest that cannot be fully understood using only a quantitative or a qualitative method. Notwithstanding the benefits and repeated calls for such work, there is a dearth of mixed methods research in information systems. Building on the literature on recent methodological advances in mixed methods research, we develop a set of guidelines for conducting mixed methods research in IS. We particularly elaborate on three important aspects of conducting mixed methods research: (1) appropriateness of a mixed methods approach; (2) development of meta-inferences (i.e., substantive theory) from mixed methods research; and (3) assessment of the quality of meta-inferences (i.e., validation of mixed methods research). The applicability of these guidelines is illustrated using two published IS papers that used mixed methods.

[1]  Marie-Claude Boudreau,et al.  Enacting Integrated Information Technology: A Human Agency Perspective , 2005, Organ. Sci..

[2]  AngSoon,et al.  IT Outsourcing Success , 2004 .

[3]  Geoff Walsham,et al.  The Emergence of Interpretivism in IS Research , 1995, Inf. Syst. Res..

[4]  C. Teddlie,et al.  Foundations of mixed methods research , 2013 .

[5]  Mark Keil,et al.  A Framework for Assessing the Reliability of Software Project Status Reports , 2002 .

[6]  Freddy Winston Castro Explaining Society, Critical Realism in the Social Sciences , 2002 .

[7]  Ioanna D. Constantiou,et al.  Exploring the influence of reference situations and reference pricing on mobile service user behaviour , 2006, Eur. J. Inf. Syst..

[8]  Caroline Stenbacka Qualitative research requires quality concepts of its own , 2001 .

[9]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[10]  J. Greene Is Mixed Methods Social Inquiry a Distinctive Methodology? , 2008 .

[11]  Amrit Tiwana,et al.  Relative importance of evaluation criteria for enterprise systems: a conjoint study , 2006, Inf. Syst. J..

[12]  Anne Beaudry,et al.  The Other Side of Acceptance: Studying the Direct and Indirect Effects of Emotions on Information Technology Use , 2010, MIS Q..

[13]  Viswanath Venkatesh,et al.  A Longitudinal Investigation of Personal Computers in Homes: Adoption Determinants and Emerging Challenges , 2001, MIS Q..

[14]  Anol Bhattacherjee,et al.  Understanding Changes in Belief and Attitude Toward Information Technology Usage: A Theoretical Model and Longitudinal Test , 2004, MIS Q..

[15]  Rajiv Sabherwal,et al.  Organizational Knowledge Management: A Contingency Perspective , 2001, J. Manag. Inf. Syst..

[16]  A. Sayer,et al.  Realism and Social Science , 1999 .

[17]  Blake Ives,et al.  Trust and the Unintended Effects of Behavior Control in Virtual Teams , 2003, MIS Q..

[18]  John Mingers,et al.  Re-Establishing the Real: Critical Realism and Information Systems , 2004 .

[19]  Y. Fried,et al.  Location, location, location: contextualizing organizational research* , 2001 .

[20]  Line Dubé,et al.  Rigor in Information Systems Positivist Case Research: Current Practices , 2003, MIS Q..

[21]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[22]  Richard T. Watson,et al.  Web Home Page Complexity and Communication Effectiveness , 2001, J. Assoc. Inf. Syst..

[23]  Bonnie Kaplan,et al.  Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study , 1988, MIS Q..

[24]  James D. Herbsleb,et al.  Familiarity, Complexity, and Team Performance in Geographically Distributed Software Development , 2007, Organ. Sci..

[25]  John Mingers,et al.  The paucity of multimethod research: a review of the information systems literature , 2003, Inf. Syst. J..

[26]  Lois-Ellin Datta,et al.  Paradigm wars: A basis for peaceful coexistence and beyond , 1994 .

[27]  M. D. Myers,et al.  Qualitative Research in Information Systems: A Reader , 2002 .

[28]  Radhika Santhanam,et al.  Postimplementation Knowledge Transfers to Users and Information Technology Professionals , 2007, J. Manag. Inf. Syst..

[29]  J. Kirk,et al.  Reliability and Validity in Qualitative Research , 1985 .

[30]  James D. Herbsleb,et al.  Team Knowledge and Coordination in Geographically Distributed Software Development , 2007, J. Manag. Inf. Syst..

[31]  John Mingers,et al.  Real-izing information systems: critical realism as an underpinning philosophy for information systems , 2004, Inf. Organ..

[32]  Detmar W. Straub,et al.  IT Outsourcing Success: A Psychological Contract Perspective , 2004, Inf. Syst. Res..

[33]  Stanley Houston,et al.  Beyond Social Constructionism: Critical Realism and Social Work , 2001 .

[34]  Richard Baskerville,et al.  Generalizing Generalizability in Information Systems Research , 2003, Inf. Syst. Res..

[35]  Ashley A. Bush,et al.  A Comparison of Transaction Cost, Agency, and Knowledge-Based Predictors of IT Outsourcing Decisions: A U.S.-Japan Cross-Cultural Field Study , 2007, J. Manag. Inf. Syst..

[36]  Abbas Tashakkori,et al.  Mixed Methodology: Combining Qualitative and Quantitative Approaches , 1998 .

[37]  Viswanath Venkatesh,et al.  Assimilation of Interorganizational Business Process Standards , 2007, Inf. Syst. Res..

[38]  Jos van Hillegersberg,et al.  Change factors requiring agility and implications for IT , 2006, Eur. J. Inf. Syst..

[39]  Geoffrey S. Hubona,et al.  A Scientific Basis for Rigor in Information Systems Research , 2009, MIS Q..

[40]  Michael Grimsley,et al.  e-Government information systems: Evaluation-led design for public value and client trust , 2007, Eur. J. Inf. Syst..

[41]  R. Bhaskar A realist theory of science , 1976 .

[42]  A. O’Cathain,et al.  Combining Qualitative and Quantitative Methods , 2007 .

[43]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[44]  Maurice Landry,et al.  A disciplined methodological pluralism for mis research , 1992 .

[45]  D. Prasad Data Collection Strategies in Mixed Method Research , 2012 .

[46]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[47]  J. Brannen The practice of a mixed methods research strategy: personal, professional and project considerations , 2008 .

[48]  K. Perreault,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2011 .

[49]  M. Lynne Markus,et al.  Power, politics, and MIS implementation , 1987, CACM.

[50]  Tracy Ann Sykes,et al.  Impacts of Information and Communication Technology Implementations on Employees' Jobs in Service Organizations in India: A Multi‐Method Longitudinal Field Study , 2010 .

[51]  P. J. Runkel,et al.  A guide to usage for writers and students in the social sciences , 1984 .

[52]  Suzanne Rivard,et al.  A Multilevel Model of Resistance to Information Technology Implementation , 2005, MIS Q..

[53]  C. Wight,et al.  After Postpositivism? The Promises of Critical Realism , 2000 .

[54]  M. Alvesson,et al.  Constructing mystery: Empirical matters in theory development , 2007 .

[55]  Ray Hackney,et al.  Towards an e-Government efficiency agenda: the impact of information and communication behaviour on e-Reverse auctions in public sector procurement , 2007, Eur. J. Inf. Syst..

[56]  Manfred Max Bergman,et al.  7 Quality of Inferences in Mixed Methods Research: Calling for an Integrative Framework , 2008 .

[57]  Geoff Walsham,et al.  Doing interpretive research , 2006, Eur. J. Inf. Syst..

[58]  Marios Koufaris,et al.  Applying the Technology Acceptance Model and Flow Theory to Online Consumer Behavior , 2002, Inf. Syst. Res..

[59]  Deborah Compeau,et al.  Computer Self-Efficacy: Development of a Measure and Initial Test , 1995, MIS Q..

[60]  Angelika Dimoka,et al.  The Nature and Role of Feedback Text Comments in Online Marketplaces: Implications for Trust Building, Price Premiums, and Seller Differentiation , 2006, Inf. Syst. Res..

[61]  Jay F. Nunamaker,et al.  Interactions Between System Evaluation and Theory Testing: A Demonstration of the Power of a Multifaceted Approach to Information Systems Research , 2006, J. Manag. Inf. Syst..

[62]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[63]  I. Newman,et al.  Mixed Methods Research , 2008 .

[64]  Peter H. Rossi The war between the quals and the quants: Is a lasting peace possible? , 1994 .

[65]  D. Mertens Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods , 1997 .

[66]  Wanda J. Orlikowski,et al.  Studying Information Technology in Organizations: Research Approaches and Assumptions , 1991, Inf. Syst. Res..

[67]  Anne Beaudry,et al.  Understanding User Responses to Information Technology: A Coping Model of User Adaption , 2005, MIS Q..

[68]  Chris Westrup,et al.  Making ERPs work: accountants and the introduction of ERP systems , 2005, Eur. J. Inf. Syst..

[69]  M. Sandelowski,et al.  The problem of rigor in qualitative research , 1986, ANS. Advances in nursing science.

[70]  Hun Choi,et al.  Contribution to Quality of Life: A New Outcome Variable for Mobile Data Service , 2007, J. Assoc. Inf. Syst..

[71]  Jennifer Caroline Greene,et al.  Defining and describing the paradigm issue in mixed‐method evaluation , 1997 .

[72]  Jon Heales,et al.  Reliability factors in business software: volatility, requirements and end-users , 2002, Inf. Syst. J..

[73]  Viswanath Venkatesh,et al.  Job Characteristics and Job Satisfaction: Understanding the Role of Enterprise Resource , 2010, MIS Q..

[74]  T. Jick Mixing Qualitative and Quantitative Methods: Triangulation in Action. , 1979 .

[75]  Detmar W. Straub,et al.  Trust and TAM in Online Shopping: An Integrated Model , 2003, MIS Q..

[76]  Brian Detlor Internet‐based information systems use in organizations: an information studies perspective , 2003, Inf. Syst. J..

[77]  John Mingers,et al.  Multimethodology: Towards a framework for mixing methodologies , 1997 .

[78]  Bongsik Shin,et al.  An Exploratory Investigation of System Success Factors in Data Warehousing , 2003, J. Assoc. Inf. Syst..

[79]  Michael D. Myers,et al.  Qualitative Research in Information Systems , 1997, MIS Q..

[80]  E. Burton Swanson,et al.  Organizing Visions for Information Technology and the Information Systems Executive Response , 2003, J. Manag. Inf. Syst..

[81]  Viswanath Venkatesh,et al.  Explaining physicians' use of EMR systems and performance in the shakedown phase , 2011, J. Am. Medical Informatics Assoc..

[82]  Dorothy E. Leidner,et al.  Leadership Effectiveness in Global Virtual Teams , 2002, J. Manag. Inf. Syst..

[83]  John Mingers,et al.  Multi-Paradigm Multimethodology , 1997 .

[84]  Monica J. Garfield,et al.  The Adoption and Use of GSS in Project Teams: Toward More Participative Processes and Outcomes , 2003, MIS Q..

[85]  Ernest R. House,et al.  Integrating the quantitative and qualitative , 1994 .

[86]  Mark Lycett,et al.  Towards the development of a social capital approach to evaluating change management interventions , 2005, Eur. J. Inf. Syst..

[87]  R. Hyman Quasi-Experimentation: Design and Analysis Issues for Field Settings (Book) , 1982 .

[88]  John W. Creswell,et al.  Editorial: Mixed Methodology Across Disciplines , 2008 .

[89]  J. Mack,et al.  When Subordinates Become IT Contractors: Persistent Managerial Expectations in IT Outsourcing , 2003 .

[90]  A. Bryman Barriers to Integrating Quantitative and Qualitative Research , 2007 .

[91]  Valerie J. Caracelli,et al.  Toward a Conceptual Framework for Mixed-Method Evaluation Designs , 1989 .

[92]  JINWEI CAO,et al.  Interactions Between System Evaluation And Theory Testing: A Demonstration of the Power of a Mulitfaceted Approach to Systems Research , 2006 .

[93]  Paul A. Pavlou,et al.  Understanding and Predicting Electronic Commerce Adoption: An Extension of the Theory of Planned Behavior , 2006, MIS Q..

[94]  Anna Sidorova,et al.  Uncovering the Intellectual Core of the Information Systems Discipline , 2008, MIS Q..

[95]  유창조 Naturalistic Inquiry , 2022, The SAGE Encyclopedia of Research Design.

[96]  Jennifer Caroline Greene,et al.  Mixed Methods in Social Inquiry , 2007 .

[97]  Heshan Sun,et al.  Causal Relationships between Perceived Enjoyment and Perceived Ease of Use: An Alternative Approach , 2006, J. Assoc. Inf. Syst..

[98]  Abbas Tashakkori,et al.  Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective , 2003 .

[99]  Sharon F. Rallis,et al.  Qualitative and quantitative inquiries are not incompatible: A call for a new partnership , 1994 .

[100]  N. Denzin,et al.  The Research Act , 1978 .

[101]  BoudreauMarie-Claude,et al.  Enacting Integrated Information Technology , 2005 .

[102]  Detmar W. Straub,et al.  Structural Equation Modeling and Regression: Guidelines for Research Practice , 2000, Commun. Assoc. Inf. Syst..

[103]  Hsin Hsin Chang,et al.  Technical and management perceptions of enterprise information system importance, implementation and benefits , 2006, Inf. Syst. J..

[104]  Allen S. Lee A Scientific Methodology for MIS Case Studies , 1989, MIS Q..

[105]  K. Howe Against the Quantitative-Qualitative Incompatibility Thesis or Dogmas Die Hard , 1988 .

[106]  E. A. Locke The Case for Inductive Theory Building† , 2007 .

[107]  A. Onwuegbuzie,et al.  The Validity Issue in Mixed Research , 2006 .

[108]  Detmar W. Straub,et al.  Validation Guidelines for IS Positivist Research , 2004, Commun. Assoc. Inf. Syst..

[109]  Keith Punch,et al.  Introduction to Social Research: Quantitative and Qualitative Approaches , 1998 .

[110]  John W. Creswell,et al.  Designing and Conducting Mixed Methods Research , 2006 .

[111]  Jan Pries-Heje,et al.  Aligning Software Processes with Strategy , 2006, MIS Q..

[112]  Detmar W. Straub,et al.  Validating Instruments in MIS Research , 1989, MIS Q..

[113]  Viswanath Venkatesh,et al.  Model of Adoption and Technology in Households: A Baseline Model Test and Extension Incorporating Household Life Cycle , 2005, MIS Q..

[114]  Daniel Robey,et al.  Research Commentary: Diversity in Information Systems Research: Threat, Promise, and Responsibility , 1996, Inf. Syst. Res..

[115]  Mark Lycett,et al.  A programme management approach for ensuring curriculum coherence in IS (higher) education , 2007, Eur. J. Inf. Syst..

[116]  Robert W. Zmud,et al.  Managing Peer-to-Peer Conflicts in Disruptive Information Technology Innovations: The Case of Software Reuse , 2006, MIS Q..

[117]  J. P. Shim,et al.  Retailer Web Site Influence On Customer Shopping: Exploratory Study on Key Factors of Customer Satisfaction , 2002, J. Assoc. Inf. Syst..

[118]  Gary Garrison,et al.  Research Note - A Model of Conflict, Leadership, and Performance in Virtual Teams , 2008, Inf. Syst. Res..

[119]  N. Denzin,et al.  Handbook of Qualitative Research , 1994 .

[120]  Sharon F. Rallis,et al.  The Qualitative-Quantitative Debate: New Perspectives. , 1994 .

[121]  Egon G. Guba,et al.  What Have We Learned About Naturalistic Evaluation , 1987 .

[122]  Viswanath Venkatesh,et al.  Where To Go From Here? Thoughts on Future Directions for Research on Individual-Level Technology Adoption with a Focus on Decision Making , 2006, Decis. Sci..

[123]  A. Strauss,et al.  Discovery of Substantive Theory: A Basic Strategy Underlying Qualitative Research , 1965 .

[124]  G. Johnson The essential impact of context on organizational behavior , 2006 .

[125]  Allen S. Lee Rigor and relevance in MIS research: beyond the approach of positivism alone , 1999 .

[126]  C. Whitbeck,et al.  A Realist Theory of Science. , 1977 .

[127]  Michael D. Myers,et al.  A Set of Principles for Conducting Critical Research in Information Systems , 2011, MIS Q..

[128]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[129]  Detmar W. Straub,et al.  Validating IS Positivist Instrumentation: 1997-2001 , 2004 .

[130]  John Mingers,et al.  Critical realism and information systems: brief responses to Monod and Klein , 2004, Inf. Organ..

[131]  R. Weber Editor's comments: the rhetoric of positivism versus interpretivism: a personal view , 2004 .

[132]  R. Keeney,et al.  The Value of Internet Commerce to the Customer , 1999 .

[133]  T. Thrall Work redesign. , 2003, Hospitals & health networks.

[134]  Mark Keil,et al.  The user–developer communication process: a critical case study , 2003, Inf. Syst. J..

[135]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[136]  John Mingers,et al.  Combining IS Research Methods: Towards a Pluralist Methodology , 2001, Inf. Syst. Res..

[137]  J. Maxwell Understanding and Validity in Qualitative Research , 1992 .

[138]  Mark Keil,et al.  Reporting bad news on software projects: the effects of culturally constituted views of face‐saving , 2007, Inf. Syst. J..

[139]  Irit Hadar,et al.  Applying ontology-based rules to conceptual modeling: a reflection on modeling decision making , 2007, Eur. J. Inf. Syst..

[140]  R. Bhaskar,et al.  Critical Realism , 2011 .

[141]  E. Guba,et al.  Paradigmatic Controversies, Contradictions, and Emerging Confluences. , 2005 .

[142]  Soon Ang,et al.  Work Outcomes and Job Design for Contract Versus Permanent Information Systems Professionals on Software Development Teams , 2001, MIS Q..

[143]  M. Patton,et al.  Qualitative evaluation and research methods , 1992 .

[144]  Ashley A. Bush,et al.  Reconciling user and project manager perceptions of IT project risk: a Delphi study 1 , 2002, Inf. Syst. J..

[145]  Viswanath Venkatesh,et al.  Model of Acceptance with Peer Support: A Social Network Perspective to Understand Employees' System Use , 2009, MIS Q..

[146]  Marianne W. Lewis,et al.  Metatriangulation: Building Theory from Multiple Paradigms , 1999 .

[147]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[148]  Mark Keil,et al.  Keeping Mum as the Project Goes Under: Toward an Explanatory Model , 2001, J. Manag. Inf. Syst..