Experimental research and multivariate statistical analyses in business information systems research - a meta-review

Experimental research has been widely used within the information systems IS field. Using a meta-review, this study examines the use of experimental research in the IS field during the period 1990 to 2013 and investigates the utilisation of statistical techniques, particularly multivariate analyses. This study provides important information regarding the pattern of experimental research in the IS field i.e., experimental designs, treatments, tasks, incentives and participants. This study finds that multivariate analysis of variance MANOVA is frequently used as the multivariate statistical technique in experimental IS research. While the use of multivariate analyses is common in recent experimental IS research, concerns remain about the potential inappropriate use of measurements and violations of statistical assumptions required by the various multivariate analyses, for example, using ordinal scales rather than interval scales and issues surrounding the normality of data. This article is intended to offer guidelines to help future researchers be aware of the statistical issues and to mitigate the shortcomings that arise due to inappropriate use of measurements and violations of the statistical assumptions in experimental IS research.

[1]  Ozgur Turetken,et al.  A multi-attribute, multi-weight clustering approach to managing "e-mail overload" , 2006, Decis. Support Syst..

[2]  Jacob M. Rose.  The effects of multimedia-induced affective states on recall and decision-making by individual investors , 2001, Int. J. Account. Inf. Syst..

[3]  Sally Wright,et al.  Are Financial Auditors Overconfident in Their Ability to Assess Risks Associated with Enterprise Resource Planning Systems? , 2004, J. Inf. Syst..

[4]  Denis Cousineau,et al.  Outliers detection and treatment: a review , 2010 .

[5]  C. J. Wolfe,et al.  Why Wait? Modeling Factors that Influence the Decision of When to Learn a New Use of Technology , 2006, J. Inf. Syst..

[6]  Jill M. D'Aquila,et al.  Accounting Students as Surrogates for Accounting Professionals When Studying Ethical Dilemmas: A Cautionary Note , 2003 .

[7]  Edgar H. Sibley,et al.  A profile of information systems research published in Information & Management , 2007, Inf. Manag..

[8]  Robert D. St. Louis,et al.  Asynchronous implementation of the nominal group technique: is it effective? , 2000, Decis. Support Syst..

[9]  Jack Shih-Chieh Hsu,et al.  Understanding the role of computer-mediated counter-argument in countering confirmation bias , 2012, Decis. Support Syst..

[10]  Vicky Arnold,et al.  The impact of tagging qualitative financial information on investor decision making: Implications for XBRL , 2012, Int. J. Account. Inf. Syst..

[11]  G. Norman.  Likert scales, levels of measurement and the “laws” of statistics , 2010, Advances in health sciences education : theory and practice.

[12]  J. Jaccard,et al.  LISREL Approaches to Interaction Effects in Multiple Regression , 1998 .

[13]  Jan Mendling,et al.  A study on the effects of routing symbol design on process model comprehension , 2013, Decis. Support Syst..

[14]  Malcolm Smith,et al.  Research Methods in Accounting , 2003 .

[15]  Paul R. Sackett,et al.  OUTLIER DETECTION AND TREATMENT IN I/O PSYCHOLOGY: A SURVEY OF RESEARCHER BELIEFS AND AN EMPIRICAL ILLUSTRATION , 2006 .

[16]  R. Gnanadesikan,et al.  Multivariate Analysis of Variance (MANOVA) , 1962 .

[17]  Galit Shmueli,et al.  Predictive Analytics in Information Systems Research , 2010, MIS Q..

[18]  W. James Potter,et al.  Rethinking validity and reliability in content analysis , 1999 .

[19]  James E. Hunton,et al.  Decision aid reliance: Modeling the effects of decision aid reliability and pressures to perform on reliance behavior , 2011, Int. J. Account. Inf. Syst..

[20]  Ramesh Sharda,et al.  Should I send this message? Understanding the impact of interruptions, social hierarchy and perceived task complexity on user performance and perceived workload , 2013, Decis. Support Syst..

[21]  Richard B. Dull,et al.  Web-based financial statements: hypertext links to footnotes and their effect on decisions , 2003, Int. J. Account. Inf. Syst..

[22]  Carl J. Huberty,et al.  Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses , 1998 .

[23]  S. Leung.  A Comparison of Psychometric Properties and Normality in 4-, 5-, 6-, and 11-Point Likert Scales , 2011 .

[24]  J. Stevens.  Applied Multivariate Statistics for the Social Sciences , 1986 .

[25]  C. J. Wolfe,et al.  Knowledge Sharing: The Effects of Incentives, Environment, and Person , 2008, J. Inf. Syst..

[26]  James E. Hunton,et al.  Retraction: Investigating the Impact of Auditor-Provided Systems Reliability Assurance on Potential Service Recipients , 2015, J. Inf. Syst..

[27]  Gregory A. Liyanarachchi,et al.  Comparing the investment decisions of accounting practitioners and students: an empirical study on the adequacy of student surrogates , 2005 .

[28]  Xiang Fang,et al.  An empirical study of web site navigation structures' impacts on web site usability , 2007, Decis. Support Syst..

[29]  Robin R. Pennington,et al.  Managing impressions using distorted graphs of income and earnings per share: The role of memory , 2009, Int. J. Account. Inf. Syst..

[30]  Panos Markopoulos,et al.  Powerful and consistent analysis of likert-type rating scales , 2010, CHI.

[31]  Mark A. Fuller,et al.  Investigating the effects of computer mediated interruptions: An analysis of task characteristics and interruption frequency on financial performance , 2009, Int. J. Account. Inf. Syst..

[32]  Helen Kelley,et al.  Research Commentary - Generalizability of Information Systems Research Using Student Subjects - A Reflection on Our Practices and Recommendations for Future Research , 2012, Inf. Syst. Res..

[33]  Mukhtar M. Ali,et al.  Robustness to nonnormality of regression F-tests , 1996 .

[34]  Jin-Ting Zhang,et al.  A note on the modified two-way MANOVA tests , 2012 .

[35]  Ruth C. King,et al.  Social responsibility and stakeholder influence: Does technology matter during stakeholder deliberation with high-impact decisions? , 2010, Decis. Support Syst..

[36]  Mun Y. Yi,et al.  Untangling the antecedents of initial trust in Web-based health information: The roles of argument quality, source expertise, and user perceptions of information quality and risk , 2013, Decis. Support Syst..

[37]  Joey F. George,et al.  Countering the anchoring and adjustment bias with decision support systems , 2000, Decis. Support Syst..

[38]  Ramesh Sharda,et al.  Development of a fisheye-based information search processing aid (FISPA) for managing information overload in the web environment , 2004, Decis. Support Syst..

[39]  E. Vance Wilson,et al.  Internet-based knowledge acquisition: Task complexity and performance , 2012, Decis. Support Syst..

[40]  G. Vining,et al.  Data Analysis: A Model-Comparison Approach , 1989 .

[41]  David A. Wood,et al.  Accounting Program Research Rankings by Topical Area and Methodology , 2010 .

[42]  James A. Senn,et al.  Research in Management Information Systems: The Minnesota Experiments , 1977 .

[43]  Paul Jen-Hwa Hu,et al.  The Effects of Process Accountability on Individuals' Use of a Familiar Technology , 2011, J. Inf. Syst..

[44]  Mark D. Miller,et al.  Examining differences across journal rankings , 2005, CACM.

[45]  Yadolah Dodge.  The Concise Encyclopedia of Statistics , 2008 .

[46]  K. Thomas,et al.  Cognitive Elements of Empowerment: An “Interpretive” Model of Intrinsic Task Motivation , 1990 .

[47]  Hans van der Heijden.  Mobile decision support for in-store purchase decisions , 2006, Decis. Support Syst..

[48]  Paul John Steinbart,et al.  The Effects of Explanation Type and User Involvement on Learning from and Satisfaction with Expert Systems , 1998 .

[49]  Elmar Kiesling,et al.  A comparison of representations for discrete multi-criteria decision problems , 2013, Decis. Support Syst..

[50]  Marilyn M. Greenstein,et al.  Extending the Accounting Brand to Privacy Services , 2003, J. Inf. Syst..

[51]  Jane Kennedy,et al.  Are M.B.A. Students a Good Proxy for Nonprofessional Investors , 2007 .

[52]  E. S. Vasu.  Multiple Regression with Varying Levels of Correlation among Predictors: Monte Carlo Sampling from Normal and Non-Normal Populations , 1978 .

[53]  D. Ariely,et al.  Effort for Payment , 2004, Psychological science.

[54]  Guillermo Vallejo,et al.  Robust tests for multivariate factorial designs under heteroscedasticity , 2011, Behavior Research Methods.

[55]  D. A. Kenny,et al.  Consequences of violating the independence assumption in analysis of variance. , 1986 .

[56]  Holmes W. Finch.  Comparison of the Performance of Nonparametric and Parametric MANOVA Test Statistics when Assumptions Are Violated , 2005 .

[57]  Joa Sang Lim,et al.  Judgmental forecasting with interactive forecasting support systems , 1996, Decis. Support Syst..

[58]  Alan R. Dennis,et al.  Conducting Experimental Research in Information Systems , 2001, Commun. Assoc. Inf. Syst..

[59]  Per E. Pedersen,et al.  An exploratory study of customers' perception of company web sites offering various interactive applications: moderating effects of customers' Internet experience , 2004, Decis. Support Syst..

[60]  Suzanne L. Lowensohn,et al.  An Examination of Faculty Perceptions of Academic Journal Quality within Five Specialized Areas of Accounting Research , 2006 .

[61]  Ian Ayres,et al.  Super Crunchers: Why Thinking-By-Numbers Is the New Way to Be Smart , 2007 .

[62]  Francesco Battaglia,et al.  Evolutionary Statistical Procedures: An Evolutionary Computation Approach to Statistical Procedures Designs and Applications , 2011 .

[63]  Judy Robertson,et al.  Likert-type scales, statistical methods, and effect sizes , 2012, Commun. ACM.

[64]  Tatsuya Kubokawa,et al.  Tests for multivariate analysis of variance in high dimension under non-normality , 2013, J. Multivar. Anal..

[65]  Karen Corral,et al.  The impact of alternative diagrams on the accuracy of recall: A comparison of star-schema diagrams and entity-relationship diagrams , 2006, Decis. Support Syst..

[66]  Stephen F. Davis,et al.  Handbook of Research Methods in Experimental Psychology , 2003 .

[67]  J. Tirole,et al.  Intrinsic and Extrinsic Motivation , 2003 .

[68]  Clark Hampton,et al.  Determinants of reliance: An empirical test of the theory of technology dominance , 2005, Int. J. Account. Inf. Syst..

[69]  Alison Parkes,et al.  The effect of task-individual-technology fit on user attitude and performance: An experimental investigation , 2013, Decis. Support Syst..

[70]  Gerardine DeSanctis,et al.  ICIS Paper: Methodological Issues in Experimental IS Research: Experiences and Recommendations , 1985, MIS Q..

[71]  Jan Recker,et al.  Scientific Research in Information Systems - A Beginner's Guide , 2012, Progress in IS.

[72]  T. R. Knapp.  Treating ordinal scales as interval scales: an attempt to resolve the controversy. , 1990, Nursing research.

[73]  Emanuel Schmider,et al.  Is It Really Robust , 2010 .

[74]  Geoffrey B. Sprinkle,et al.  The effects of monetary incentives on effort and task performance: theories, evidence, and a framework for research , 2002 .

[75]  Cheryl L. Dunn,et al.  Perceived semantic expressiveness of accounting systems and task accuracy effects , 2000, Int. J. Account. Inf. Syst..

[76]  Steven E. Kaplan,et al.  A Web assurance services model of trust for B2C e-commerce , 2003, Int. J. Account. Inf. Syst..

[77]  G. Walls.  Is systematic quantitative research scientifically rigorous? Methodological and statistical considerations , 2012, Journal of the American Psychoanalytic Association.

[78]  Tina M. Loraas,et al.  Learning new uses of technology while on an audit engagement: Contextualizing general models to advance pragmatic understanding , 2010, Int. J. Account. Inf. Syst..

[79]  A. Rustichini,et al.  Pay Enough or Don't Pay at All , 2000 .