Quantitative Longitudinal Research: A Review of IS Literature, and a Set of Methodological Guidelines

Data captured at different points in time provides the basis for longitudinal research. It is unquestioned that several IS phenomena deal with change over time such as post-adoption behavior with respect to IT artifacts. However, cross-sectional research designs are predominantly applied in the IS field up till now. This paper is therefore written not only to motivate the IS community to apply longitudinal research to time-variant IS phenomena but also to discuss common pitfalls. For this purpose, we outline various longitudinal studies and provide four guidelines that should be considered during their planning. In particular, common methodological issues like space and amount of repeated observations or attrition are discussed. Finally, an overview of common longitudinal research questions and corresponding methods of longitudinal analyses is provided.

[1]  Wai Fong Boh,et al.  The Career Paths Less (or More) Traveled: A Sequence Analysis of IT Career Histories, Mobility Patterns, and Career Success , 2012, MIS Q..

[2]  Xiaojun Zhang,et al.  "Doctors Do Too Little Technology": A Longitudinal Field Study of an Electronic Healthcare System Implementation , 2011, Inf. Syst. Res..

[3]  Everett M. Rogers,et al.  Diffusion of innovations (5. ed.) , 2003 .

[4]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[5]  Robert E. Ployhart,et al.  The “Quick Start Guide” for Conducting and Publishing Longitudinal Research , 2011 .

[6]  Gerardine DeSanctis,et al.  ICIS Paper: Methodological Issues in Experimental IS Research: Experiences and Recommendations , 1985, MIS Q..

[7]  G. Molenberghs,et al.  Coping With Memory Effect and Serial Correlation When Estimating Reliability in a Longitudinal Framework , 2010 .

[8]  Benjamin T. Hazen,et al.  Diffusion of Innovations and the Theory of Planned Behavior in Information Systems Research: A Metaanalysis , 2014, Communications of the Association for Information Systems.

[9]  K. Perreault,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2011 .

[10]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[11]  Daniel John Zizzo,et al.  Experimenter demand effects in economic experiments , 2008 .

[12]  Matthew K. O. Lee,et al.  Digital Inclusiveness--Longitudinal Study of Internet Adoption by Older Adults , 2006, J. Manag. Inf. Syst..

[13]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[14]  Jonathan P. Doh,et al.  The evolution of risk in information systems offshoring: the impact of home country risk, firm learning, and competitive dynamics , 2009 .

[15]  Andrew B. Whinston,et al.  Capacity Provision Networks: Foundations of Markets for Sharable Resources in Distributed Computational Economies , 2008, Inf. Syst. Res..

[16]  R. Steyer,et al.  A latent state-trait model (not only) for social desirability. , 1993 .

[17]  F. Heider The psychology of interpersonal relations , 1958 .

[18]  Michael Eid,et al.  Latent state–trait theory and research in personality and individual differences , 1999 .

[19]  Pankaj Setia,et al.  How Peripheral Developers Contribute to Open-Source Software Development , 2012, Inf. Syst. Res..

[20]  Naresh K. Malhotra,et al.  Research Note - Two Competing Perspectives on Automatic Use: A Theoretical and Empirical Comparison , 2005, Inf. Syst. Res..

[21]  Lesley Willcoxson,et al.  Progress in the IT/business relationship: a longitudinal assessment , 2004, J. Inf. Technol..

[22]  Robert E. Ployhart,et al.  Longitudinal Research: The Theory, Design, and Analysis of Change , 2010 .

[23]  J. Oud,et al.  Continuous time modelling with individually varying time intervals for oscillating and non-oscillating processes. , 2013, The British journal of mathematical and statistical psychology.

[24]  Linda M. Collins,et al.  Using Modern Missing Data Methods with Auxiliary Variables to Mitigate the Effects of Attrition on Statistical Power , 2012 .

[25]  Heshan Sun,et al.  A Longitudinal Study of Herd Behavior in the Adoption and Continued Use of Technology , 2013, MIS Q..

[26]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[27]  Xin Lai,et al.  A Monte Carlo Study of the Effects of Common Method Variance on Significance Testing and Parameter Bias in Hierarchical Linear Modeling , 2013 .

[28]  Brian S. Butler,et al.  Team Cognition: Development and Evolution in Software Project Teams , 2007, J. Manag. Inf. Syst..

[29]  S. Geisser,et al.  On methods in the analysis of profile data , 1959 .

[30]  J. Buckler Growth at adolescence , 2007 .

[31]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[32]  J J McArdle,et al.  Latent growth curves within developmental structural equation models. , 1987, Child development.

[33]  William J. McGuire,et al.  The Current Status of Cognitive Consistency Theories1 , 1966 .

[34]  Michael D. Myers,et al.  A Set of Principles for Conducting Critical Research in Information Systems , 2011, MIS Q..

[35]  Lei Chi,et al.  Understanding Postadoptive Behaviors in Information Systems Use: A Longitudinal Analysis of System Use Problems in the Business Intelligence Context , 2012, J. Manag. Inf. Syst..

[36]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[37]  Janice Baranowski,et al.  Story Immersion in a Health Videogame for Childhood Obesity Prevention. , 2012, Games for health journal.

[38]  Viswanath Venkatesh,et al.  Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology , 2012, MIS Q..

[39]  Matthias Trier,et al.  Research Note - Towards Dynamic Visualization for Understanding Evolution of Digital Communication Networks , 2008, Inf. Syst. Res..

[40]  J. Graham,et al.  Missing data analysis: making it work in the real world. , 2009, Annual review of psychology.

[41]  T. Mitchell,et al.  Building Better Theory: Time and The Specification of When Things Happen , 2001 .

[42]  S. Maxwell,et al.  Bias in Cross-Sectional Analyses of Longitudinal Mediation: Partial and Complete Mediation Under an Autoregressive Model , 2011, Multivariate behavioral research.

[43]  H. Huynh,et al.  Conditions under Which Mean Square Ratios in Repeated Measurements Designs Have Exact F-Distributions , 1970 .

[44]  Franz W. Kellermanns,et al.  The longitudinal impact of enterprise system users’ pre-adoption expectations and organizational support on post-adoption proficient usage , 2014, EJIS.

[45]  Leo J. Th. van der Kamp,et al.  Longitudinal Data Analysis: Designs, Models and Methods , 1999 .

[46]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[47]  Naresh K. Malhotra,et al.  A Longitudinal Model of Continued IS Use: An Integrative View of Four Mechanisms Underlying Postadoption Phenomena , 2005, Manag. Sci..

[48]  Frank Q. Fu,et al.  Motivating Salespeople to Sell New Products: the Relative Influence of Attitudes, Subjective Norms, and Self-efficacy Literature Review: New Product Performance, Tpb, and the Sales Force Intentions and Sales Performance the Relative Influence of Attitudes, Subjective Norms, and Self-efficacy the Mod , 2022 .

[49]  E. Rogers,et al.  Diffusion of Innovations , 1964 .

[50]  J. S. Sodhi,et al.  Using Internet and Mobile Phone Technology to Deliver an Automated Physical Activity Program: Randomized Controlled Trial , 2007, Journal of medical Internet research.

[51]  Scott E. Maxwell,et al.  Internal Invalidity in Pretest-Posttest Self-Report Evaluations and a Re-evaluation of Retrospective Pretests , 1979 .

[52]  S. Maxwell,et al.  Bias in cross-sectional analyses of longitudinal mediation. , 2007, Psychological methods.

[53]  R. Bakeman Recommended effect size statistics for repeated measures designs , 2005, Behavior research methods.

[54]  H. Weiss,et al.  An Examination of the Joint Effects of Affective Experiences and Job Beliefs on Job Satisfaction and Variations in Affective Experiences over Time. , 1999, Organizational behavior and human decision processes.

[55]  Sung S. Kim The integrative framework of technology use: an extension and test , 2009 .

[56]  James R Cook Engaged Scholarship: A Guide for Organizational and Social Research , 2014 .

[57]  L. Cronbach,et al.  How we should measure "change": Or should we? , 1970 .

[58]  Frank M. Bass,et al.  Comments on "A New Product Growth for Model Consumer Durables The Bass Model" , 2004, Manag. Sci..

[59]  Michael Frankfurter,et al.  Applied Longitudinal Data Analysis Modeling Change And Event Occurrence , 2016 .

[60]  P. Gagné,et al.  The Effect of Sample Size on Latent Growth Models. , 2003 .

[61]  A 15-year longitudinal test of the theory of planned behaviour to predict physical activity in a randomized national sample of Canadian adults , 2012 .

[62]  Raghu Garud,et al.  The innovation journey , 1999 .

[63]  F. Anseel,et al.  Response Rates in Organizational Science, 1995–2008: A Meta-analytic Review and Guidelines for Survey Researchers , 2010 .

[64]  Masumi Iida,et al.  A Procedure for Evaluating Sensitivity to Within-Person Change: Can Mood Measures in Diary Studies Detect Change Reliably? , 2006, Personality & social psychology bulletin.

[65]  Erja Mustonen-Ollila,et al.  Why organizations adopt information system process innovations: a longitudinal study using Diffusion of Innovation theory , 2003, Inf. Syst. J..

[66]  H. Raghav Rao,et al.  Trust and Satisfaction, Two Stepping Stones for Successful E-Commerce Relationships: A Longitudinal Exploration , 2009, Inf. Syst. Res..

[67]  P. Dubno Attitudes toward women executives: A longitudinal approach. , 1985 .

[68]  Arvind K. Tripathi,et al.  Understanding Willingness-to-Pay Formation of Repeat Bidders in Sequential Online Auctions , 2010, Inf. Syst. Res..

[69]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[70]  Vijay Mahajan,et al.  New Product Diffusion Models in Marketing: A Review and Directions for Research: , 1990 .

[71]  Gang Peng,et al.  Computer skills, non-routine tasks, and wage premium: A longitudinal study , 2011, J. Strateg. Inf. Syst..

[72]  Henk Folmer,et al.  Reply to Steele & Ferrer: Modeling Oscillation, Approximately or Exactly? , 2011, Multivariate behavioral research.

[73]  Daniel A. Newman Longitudinal Modeling with Randomly and Systematically Missing Data: A Simulation of Ad Hoc, Maximum Likelihood, and Multiple Imputation Techniques , 2003 .

[74]  Jos W. R. Twisk,et al.  Applied Longitudinal Data Analysis for Epidemiology: References , 2013 .

[75]  M. Griffin,et al.  Longitudinal relationships between core self-evaluations and job satisfaction. , 2012, The Journal of applied psychology.

[76]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[77]  L. Yates Interpretive claims and methodological warrant in small-number qualitative, longitudinal research , 2003 .

[78]  Janni Nielsen,et al.  European Conference on Information Systems (ECIS) , 2008 .

[79]  Abraham Seidmann,et al.  An Empirical Investigation of the Value of Integrating Enterprise Information Systems: The Case of Medical Imaging Informatics , 2009, J. Manag. Inf. Syst..

[80]  Radhika Santhanam,et al.  The Effects of Social Network Structure on Enterprise Systems Success: A Longitudinal Multilevel Analysis , 2012, Inf. Syst. Res..

[81]  Terry C. Blum,et al.  Assessing the Non-Random Sampling Effects of Subject Attrition in Longitudinal Research , 1996 .

[82]  R. Steyer,et al.  Is it Possible to Feel Good and Bad at the Same Time? New Evidence on the Bipolarity of Mood-state Dimensions , 2004 .

[83]  Jon Jasperson,et al.  Managing change in an information systems development organization: understanding developer transitions from a structured to an object‐oriented development environment 1 , 2008, Inf. Syst. J..

[84]  Jay J. Shen,et al.  Financial position and adoption of electronic health records: a retrospective longitudinal study. , 2012, Journal of health care finance.

[85]  D. Dimitrov,et al.  Pretest-posttest designs and measurement of change. , 2003, Work.

[86]  Steven H. Dahlquist,et al.  An assessment of hierarchical linear modeling in international business, management, and marketing , 2013 .

[87]  Neil Anderson,et al.  The Power of Momentum: A New Model of Dynamic Relationships between Job Satisfaction Change and Turnover Intentions , 2011 .

[88]  D P MacKinnon,et al.  Maximizing the Usefulness of Data Obtained with Planned Missing Value Patterns: An Application of Maximum Likelihood Procedures. , 1996, Multivariate behavioral research.

[89]  Daymont Tn,et al.  The research uses of the National Longitudinal Surveys: an update , 1983 .

[90]  Mark A. Serva,et al.  Using Latent Growth Modeling to Understand Longitudinal Effects in MIS Theory: A Primer , 2011, Commun. Assoc. Inf. Syst..

[91]  ZhangXiaojun,et al.  Doctors Do Too Little Technology , 2011 .

[92]  Kerrie L. Unsworth,et al.  Changing behaviour: Increasing the effectiveness of workplace interventions in creating pro‐environmental behaviour change , 2013 .

[93]  Viswanath Venkatesh,et al.  Digital Divide Initiative Success in Developing Countries: A Longitudinal Field Study in a Village in India , 2013, Inf. Syst. Res..

[94]  Frank M. Bass,et al.  A New Product Growth for Model Consumer Durables , 2004, Manag. Sci..

[95]  Jonas Hedman,et al.  Technology Use as Consumption: A Longitudinal Study of Smart Phones , 2009, ICIS.

[96]  C. Moorman,et al.  Cross-Sectional versus Longitudinal Survey Research: Concepts, Findings, and Guidelines , 2008 .

[97]  Peter H. Gray,et al.  Social Influence and Knowledge Management Systems Use: Evidence from Panel Data , 2013, MIS Q..

[98]  R. Lerner,et al.  Problematics of Time and Timing in the Longitudinal Study of Human Development: Theoretical and Methodological Issues , 2009, Human Development.

[99]  Izak Benbasat,et al.  Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation , 1991, Inf. Syst. Res..

[100]  Gerald C. Kane,et al.  Membership Turnover and Collaboration Success in Online Communities: Explaining Rises , 2011, MIS Q..

[101]  Anthony S. Bryk,et al.  Toward a More Appropriate Conceptualization of Research on School Effects: A Three-Level Hierarchical Linear Model , 1988, American Journal of Education.

[102]  Claudio Vitari,et al.  A longitudinal analysis of trajectory changes in the software industry: the case of the content management application segment , 2009, Eur. J. Inf. Syst..

[103]  P. Shrout,et al.  Assessing the reliability of change: A comparison of two measures of adult attachment , 2013 .

[104]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..