A Tool for Addressing Construct Identity in Literature Reviews and Meta-Analyses

The problem of detecting whether two behavioral constructs reference the same real-world phenomenon has existed for over 100 years. Discordant naming of constructs is here termed the construct identity fallacy (CIF). We designed and evaluated the construct identity detector (CID), the first tool with large-scale construct identity detection properties and the first tool that does not require respondent data. Through the adaptation and combination of different natural language processing (NLP) algorithms, six designs were created and evaluated against human expert decisions. All six designs were found capable of detecting construct identity, and a design combining two existing algorithms significantly outperformed the other approaches. A set of follow-up studies suggests the tool is valuable as a supplement to expert efforts in literature review and metaanalysis. Beyond design science contributions, this article has important implications related to the taxonomic structure of social and behavioral science constructs, for the jingle and jangle fallacy, the core of the Information Systems nomological network, and the inaccessibility of social and behavioral science knowledge. In sum, CID represents an important, albeit tentative, step toward discipline-wide identification of construct identities

[1]  Albert H. Segars,et al.  An Empirical Examination of the Concern for Information Privacy Instrument , 2002, Inf. Syst. Res..

[2]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[3]  Kristopher J Preacher,et al.  Repairing Tom Swift's Electric Factor Analysis Machine , 2003 .

[4]  S ValacichJoseph,et al.  Uncovering the intellectual core of the information systems discipline , 2008 .

[5]  George A. Miller,et al.  Using Corpus Statistics and WordNet Relations for Sense Identification , 1998, CL.

[6]  J. Loevinger.  Objective Tests as Instruments of Psychological Theory , 1957 .

[7]  Michael Healy,et al.  Theory and Applications of Ontology: Computer Applications , 2010 .

[8]  Marie Johnston,et al.  Theoretical framework and methodological development of common subjective health outcome measures in osteoarthritis: a critical review , 2007, Health and quality of life outcomes.

[9]  S. T. Dumais,et al.  Using latent semantic analysis to improve access to textual information , 1988, CHI '88.

[10]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[11]  Hock-Hai Teo,et al.  The Value of Privacy Assurance: An Exploratory Field Experiment , 2007, MIS Q..

[12]  Atreyi Kankanhalli,et al.  Investigating user resistance to information systems implementation: a status quo bias perspective , 2009 .

[13]  Michael E. Lesk,et al.  Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone , 1986, SIGDOC '86.

[14]  James Birt,et al.  Theory Identity: A Machine-Learning Approach , 2014, 2014 47th Hawaii International Conference on System Sciences.

[15]  Christian Swann,et al.  Comparing two measures of mental toughness , 2011 .

[16]  Raymond L. Debus,et al.  Evaluation of the Big-Two-Factor Theory of Academic Motivation Orientations: An Evaluation of Jingle-Jangle Fallacies , 2003 .

[17]  Frank S. Freeman,et al.  Interpretation of educational measurements , 1928 .

[18]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[19]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[20]  S. Whiteside,et al.  The Five Factor Model and impulsivity: using a structural model of personality to understand impulsivity , 2001 .

[21]  D. Swanson.  Fish Oil, Raynaud's Syndrome, and Undiscovered Public Knowledge , 2015, Perspectives in biology and medicine.

[22]  H. Jeff Smith,et al.  Information Privacy: Measuring Individuals' Concerns About Organizational Practices , 1996, MIS Q..

[23]  William Empson,et al.  朦胧的七种类型 = Seven types of ambiguity , 1930 .

[24]  J. J. Po-An Hsieh,et al.  ScholarWorks @ Georgia State University , 2016 .

[25]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[26]  Samuel B. Bacharach,et al.  Organizational Theories: Some Criteria for Evaluation , 1989 .

[27]  Philip Resnik,et al.  Using Information Content to Evaluate Semantic Similarity in a Taxonomy , 1995, IJCAI.

[28]  Jay F. Nunamaker,et al.  Detecting Fake Websites: The Contribution of Statistical Learning Theory , 2010, MIS Q..

[29]  A. Burton-Jones.  Minimizing Method Bias Through Programmatic Research , 2009 .

[30]  Dennis F. Galletta,et al.  How Endogenous Motivations Influence User Intentions: Beyond the Dichotomy of Extrinsic and Intrinsic User Motivations , 2008, J. Manag. Inf. Syst..

[31]  George A. Miller,et al.  Introduction to WordNet: An On-line Lexical Database , 1990 .

[32]  Kai R. T. Larsen,et al.  Addicted to constructs: science in reverse? , 2013, Addiction.

[33]  Kai R. T. Larsen,et al.  A Taxonomy of Antecedents of Information Systems Success: Variable Analysis Studies , 2003, J. Manag. Inf. Syst..

[34]  Victor R. Prybutok,et al.  Latent Semantic Analysis: five methodological recommendations , 2012, Eur. J. Inf. Syst..

[35]  Kai R. Larsen,et al.  9. A Mathematical Approach to Categorization and Labeling of Qualitative Data: The Latent Categorization Method , 2004 .

[36]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[37]  Walt Detmar Meurers.  Diagnosing Meaning Errors in Short Answers to Reading Comprehension Questions , 2008 .

[38]  D. Gefen,et al.  E-commerce: the role of familiarity and trust , 2000 .

[39]  Naresh K. Malhotra,et al.  Internet Users' Information Privacy Concerns (IUIPC): The Construct, the Scale, and a Causal Model , 2004, Inf. Syst. Res..

[40]  Izak Benbasat,et al.  The Effect of Multimedia on Perceived Equivocality and Perceived Usefulness of Information Systems , 2000, MIS Q..

[41]  Pierre Baldi,et al.  A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval , 2010, Bioinform..

[42]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[43]  Zuhair Bandar,et al.  Sentence similarity based on semantic nets and corpus statistics , 2006, IEEE Transactions on Knowledge and Data Engineering.

[44]  P. Alexander,et al.  A Motivated Exploration of Motivation Terminology. , 2000, Contemporary educational psychology.

[45]  Detmar W. Straub,et al.  Specifying Formative Constructs in Information Systems Research , 2007, MIS Q..

[46]  Jane Webster,et al.  An Investigation of Information Systems Use Patterns: Technological Events as Triggers, the Effect of Time, and Consequences for Performance , 2013, MIS Q..

[47]  Chockalingam Viswesvaran,et al.  An Evaluation of Construct Validity: What Is This Thing Called Emotional Intelligence? , 2005 .

[48]  Robert F. Easley,et al.  Relating Collaborative Technology Use to Teamwork Quality and Performance: An Empirical Analysis , 2003, J. Manag. Inf. Syst..

[49]  Charu C. Aggarwal,et al.  A Survey of Text Clustering Algorithms , 2012, Mining Text Data.

[50]  Anna Sidorova,et al.  Uncovering the Intellectual Core of the Information Systems Discipline , 2008, MIS Q..

[51]  Detmar W. Straub,et al.  Information Technology Adoption Across Time: A Cross-Sectional Comparison of Pre-Adoption and Post-Adoption Beliefs , 1999, MIS Q..

[52]  J. Block.  A contrarian view of the five-factor approach to personality description. , 1995, Psychological bulletin.

[53]  Martha Palmer,et al.  Verb Semantics and Lexical Selection , 1994, ACL.

[54]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[55]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[56]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[57]  Carlo Strapparava,et al.  Corpus-based and Knowledge-based Measures of Text Semantic Similarity , 2006, AAAI.

[58]  Steven L. Alter.  Architecture of Sysperanto: A Model-Based Ontology of the IS Field , 2005, Commun. Assoc. Inf. Syst..

[59]  David W. Conrath,et al.  Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy , 1997, ROCLING/IJCLCLP.

[60]  Derek Partridge,et al.  Representation of Knowledge , 1996, Artificial Intelligence.

[61]  Rick S Zimmerman,et al.  Health Behavior Theory and cumulative knowledge regarding health behaviors: are we moving in the right direction? , 2005, Health education research.

[62]  M. Zuckerman.  Behavioral Expressions and Biosocial Bases of Sensation Seeking , 1994 .

[63]  H. Raghav Rao,et al.  Trust and Satisfaction, Two Stepping Stones for Successful E-Commerce Relationships: A Longitudinal Exploration , 2009, Inf. Syst. Res..

[64]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[65]  Naftali Tishby,et al.  Sufficient Dimensionality Reduction , 2003, J. Mach. Learn. Res..

[66]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[67]  Erkki Sutinen,et al.  Comparison of Dimension Reduction Methods for Automated Essay Grading , 2008, J. Educ. Technol. Soc..

[68]  Susan T. Dumais,et al.  The latent semantic analysis theory of knowledge , 1997 .

[69]  Jingjing Li,et al.  Tracking Behavioral Construct Use through Citations: A Relation Extraction Approach , 2013, ICIS.

[70]  Ryad Titah,et al.  Measure for Measure: A two study multi-trait multi-method investigation of construct validity in IS research , 2013, Comput. Hum. Behav..

[71]  Denny Borsboom,et al.  The end of construct validity. , 2009 .

[72]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[73]  Steven L. Alter.  The Knowledge Cube: Scaffolding for a Body of Knowledge about Information Systems , 2012, ECIS.

[74]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[75]  Alan R. Hevner,et al.  POSITIONING AND PRESENTING DESIGN SCIENCE RESEARCH FOR MAXIMUM IMPACT 1 , 2013 .

[76]  Jingjing Li,et al.  Establishing Nomological Networks for Behavioral Science: a Natural Language Processing Based Approach , 2011, ICIS.

[77]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[78]  A. Kaplan.  The Conduct of Inquiry: Methodology for Behavioural Science , 1965 .

[79]  R. Ryan Nelson,et al.  Educational Needs as Perceived by IS and End-User Personnel: A Survey of Knowledge and Skill Requirements , 1991, MIS Q..

[80]  Jörg Becker,et al.  Comprehensible Predictive Models for Business Processes , 2016, MIS Q..

[81]  Michael J. Gallivan,et al.  ORGANIZATIONS: A MULTILEVEL PERSPECTIVE , 2007 .

[82]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[83]  Wolfgang Ketter,et al.  Competitive Benchmarking: An IS Research Approach to Address Wicked Problems with Big Data and Analytics , 2015, MIS Q..

[84]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[85]  P. Alexander,et al.  Coming to Terms: How Researchers in Learning and Literacy Talk About Knowledge , 1991 .

[86]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[87]  Wynne W. Chin,et al.  A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte Carlo Simulation Study and an Electronic - Mail Emotion/Adoption Study , 2003, Inf. Syst. Res..

[88]  G. Tellis,et al.  Mining Marketing Meaning from Online Chatter: Strategic Brand Analysis of Big Data Using Latent Dirichlet Allocation , 2014 .

[89]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[90]  T. Landauer.  LSA as a Theory of Meaning , 2007 .

[91]  Mimi Bong,et al.  Problems in academic motivation research and advantages and disadvantages of their solutions. , 1996 .