Anticipating, avoiding, and alleviating measurement error: A synthesis of the literature with practical recommendations

Researchers’ ability to draw inferences from their empirical work hinges on the degree of measurement error. The literature in Information Systems and other behavioural disciplines describes a plethora of sources of error. While it helps researchers deal with them when taking specific steps in the measurement process, like modelling constructs, developing instruments, collecting data, and analysing data, it does not provide an overall guide to help them prevent and deal with measurement error. This paper presents a synthesis of the insights in the literature through a decomposition of the logic of measurement. It shows how researchers can classify sources of error, evaluate their impact, and refine their measurement plans, in terms of specific steps or overall measurement approaches. We hope this will aid researchers in anticipating, avoiding, and alleviating error in measurement, and in drawing valid research conclusions.

[1]  Ryan T. Wright,et al.  Assessing Common Method Bias: Problems with the ULMC Technique , 2012, MIS Q..

[2]  Adamantios Diamantopoulos,et al.  MIMIC models, formative indicators and the joys of research , 2013 .

[3]  Denny Borsboom,et al.  Frontiers of Test Validity Theory: Measurement, Causation, and Meaning , 2013 .

[4]  A. Burton-Jones Minimizing Method Bias Through Programmatic Research , 2009 .

[5]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[6]  Detmar W. Straub,et al.  Trust and TAM in Online Shopping: An Integrated Model , 2003, MIS Q..

[7]  R. Bagozzi,et al.  On the nature and direction of relationships between constructs and measures. , 2000, Psychological methods.

[8]  Cheryl Burke Jarvis,et al.  A Critical Review of Construct Indicators and Measurement Model Misspecification in Marketing and Consumer Research , 2003 .

[9]  Detmar W. Straub,et al.  The Critical Importance of Construct Measurement Specification: A Response to Aguirre-Urreta and Marakas , 2012, MIS Q..

[10]  Scott B. MacKenzie,et al.  Working memory: theories, models, and controversies. , 2012, Annual review of psychology.

[11]  Arun Rai,et al.  Discovering Unobserved Heterogeneity in Structural Equation Models to Avert Validity Threats , 2013, MIS Q..

[12]  R. Fisher Social Desirability Bias and the Validity of Indirect Questioning , 1993 .

[13]  D. Paulhus Measurement and control of response bias. , 1991 .

[14]  Thurasamy Ramayah,et al.  Testing and Controlling for Common Method Variance: A Review of Available Methods , 2017 .

[15]  D. Dillman Mail and internet surveys , 1999 .

[16]  Detmar W. Straub,et al.  Specifying Formative Constructs in Information Systems Research , 2007, MIS Q..

[17]  Gilbert A. Churchill A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[18]  Terry Anthony Byrd,et al.  A methodology for construct development in MIS research , 2005, Eur. J. Inf. Syst..

[19]  L. Mandolesi Physiology of Behavior , 2002 .

[20]  P. Chisnall Mail and Internet Surveys: The Tailored Design Method , 2007, Journal of Advertising Research.

[21]  Michael A. Olson,et al.  Implicit measures in social cognition. research: their meaning and use. , 2003, Annual review of psychology.

[22]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[23]  Terry M. Coalter,et al.  Context, Cognition, and Common Method Variance: Psychometric and Verbal Protocol Evidence , 1996 .

[24]  William R. Shadish,et al.  The person–situation debate: A critical multiplist perspective. , 1986 .

[25]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[26]  Jessica Bagger,et al.  The Balanced Inventory of Desirable Responding (BIDR) , 2007 .

[27]  Jason Bennett Thatcher,et al.  Conceptualizing models using multidimensional constructs: a review and guidelines for their use , 2012, Eur. J. Inf. Syst..

[28]  David J. Ketchen,et al.  The Use of Archival Proxies in Strategic Management Studies , 2013 .

[29]  David E. Kanouse,et al.  Controlling for Acquiescence Response Set in Scale Development , 1982 .

[30]  Gary Goertz,et al.  Social Science Concepts: A User's Guide , 2005 .

[31]  Tibert Verhagen,et al.  Toward a Better Use of the Semantic Differential in IS Research: An Integrative Framework of Suggested Action , 2015, J. Assoc. Inf. Syst..

[32]  J. Schafer,et al.  Missing data: our view of the state of the art. , 2002, Psychological methods.

[33]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[34]  Scott B. MacKenzie,et al.  Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques , 2011, MIS Q..

[35]  N. Schwarz,et al.  Context Effects in Social and Psychological Research , 1992 .

[36]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[37]  L. Havlicek,et al.  Effect of the violation of assumptions upon significance levels of the Pearson r. , 1977 .

[38]  Edward E. Rigdon,et al.  Rethinking Partial Least Squares Path Modeling: In Praise of Simple Methods , 2012 .

[39]  Cheryl Burke Jarvis,et al.  The Negative Consequences of Measurement Model Misspecification: A Response to Aguirre-Urreta and Marakas , 2012, MIS Q..

[40]  Adamantios Diamantopoulos,et al.  Incorporating Formative Measures into Covariance-Based Structural Equation Models , 2011, MIS Q..

[41]  Kenneth S. Law,et al.  Toward A Taxonomy of Multidimensional Constructs , 1998 .

[42]  R. Lennox,et al.  Conventional wisdom on measurement: A structural equation perspective. , 1991 .

[43]  A. Greenwald,et al.  Measuring individual differences in implicit cognition: the implicit association test. , 1998, Journal of personality and social psychology.

[44]  George M. Marakas,et al.  Revisiting Bias Due to Construct Misspecification: Different Results from Considering Coefficients in Standardized Form , 2012, MIS Q..

[45]  Shirley Gregor,et al.  Eight Obstacles to Overcome in the Theory Testing Genre , 2014, J. Assoc. Inf. Syst..

[46]  Timothy D. Wilson,et al.  Experimentation in social psychology. , 1998 .

[47]  S. Messick Meaning and Values in Test Validation: The Science and Ethics of Assessment , 1989 .

[48]  Anil K. Bera,et al.  A test for normality of observations and regression residuals , 1987 .

[49]  Robert J. Vokurka,et al.  The empirical assessment of construct validity , 1998 .

[50]  J. Edwards Multidimensional Constructs in Organizational Behavior Research: An Integrative Analytical Framework , 2001 .

[51]  Jeffrey R. Edwards,et al.  To prosper, organizational psychology should … overcome methodological barriers to progress , 2008 .

[52]  Karl Wennberg,et al.  Entrepreneurship research through databases: Measurement and design issues , 2005 .

[53]  Ali F. Farhoomand,et al.  Improving management information systems research: question order effects in surveys , 1997, Inf. Syst. J..

[54]  D. Watson,et al.  Constructing validity: Basic issues in objective scale development , 1995 .

[55]  Fritz Drasgow,et al.  A Meta-Analytic Study of Social Desirability Distortion in Computer- Administered Questionnaires, Traditional Questionnaires, and Interviews , 1999 .

[56]  Edward E. Rigdon Lee, Cadogan, and Chamberlain: an excellent point . . . But what about that iceberg? , 2013 .

[57]  Nathan S. Hartman,et al.  Method Variance and Marker Variables: A Review and Comprehensive CFA Marker Technique , 2010 .

[58]  J. P. Peter Construct Validity: A Review of Basic Issues and Marketing Practices , 1981 .

[59]  W. Meredith Measurement invariance, factor analysis and factorial invariance , 1993 .

[60]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[61]  Edward E. Rigdon,et al.  Rethinking Partial Least Squares Path Modeling: Breaking Chains and Forging Ahead , 2014 .

[62]  Lee J. Cronbach,et al.  Construct validation after thirty years. , 1989 .

[63]  S. Segerstrom,et al.  Heart Rate Variability Reflects Self-Regulatory Strength, Effort, and Fatigue , 2007, Psychological science.

[64]  Andrew Burton-Jones,et al.  Thinking About Measures and Measurement in Positivist Research: A Proposal for Refocusing on Fundamentals , 2017, Inf. Syst. Res..

[65]  Sander Zwanenburg,et al.  How to Tie a Construct to Indicators: Guidelines for Valid Measurement , 2015, ICIS.

[66]  B. Muthén BEYOND SEM: GENERAL LATENT VARIABLE MODELING , 2002 .

[67]  S. Haynes,et al.  Content validity in psychological assessment: A functional approach to concepts and methods. , 1995 .

[68]  H. Kraemer,et al.  Summated Rating Scale Construction An Introduction , 2009 .

[69]  M. Ronald Buckley,et al.  Measurement Error and Theory Testing in Consumer Research: An Illustration of the Importance of Construct Validation , 1988 .

[70]  Angelika Dimoka,et al.  How to Conduct a Functional Magnetic Resonance (fMRI) Study in Social Science Research , 2012, MIS Q..

[71]  Michael J. Zyphur,et al.  Is Quantitative Research Ethical? Tools for Ethically Practicing, Evaluating, and Using Quantitative Research , 2017 .

[72]  M. Bradley,et al.  Remembering pictures: pleasure and arousal in memory. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[73]  M. Lindell,et al.  Accounting for common method variance in cross-sectional research designs. , 2001, The Journal of applied psychology.

[74]  Edward E. Rigdon,et al.  Comment on “Improper use of endogenous formative variables” , 2014 .

[75]  G. J. Mellenbergh,et al.  Generalized linear item response theory. , 1994 .

[76]  Richard P. Bagozzi,et al.  Measurement and Meaning in Information Systems and Organizational Research: Methodological and Philosophical Foundations , 2011, MIS Q..

[77]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[78]  D. Campbell Factors relevant to the validity of experiments in social settings. , 1957, Psychological bulletin.