What Drives Key Informant Accuracy?

In an effort to establish and enhance the accuracy of key informant data, organizational survey studies are increasingly relying on triangulation techniques by including supplemental data sources that complement information acquired from key informants. Despite the growing popularity of triangulation, little guidance exists as to when and how it should be conducted. Addressing this gap, the authors develop hypotheses linking a comprehensive set of study characteristics at the construct, informant, organizational, and industry levels to key informant accuracy. Two studies test these hypotheses. The first study is a meta-analysis of triangulation applications. Using data from 127 studies published in six major marketing and management journals, the authors identify antecedents to key informant reliability. The second study, using eight multi-informant data sets, analyzes antecedents to key informant validity. The results from these studies inform survey researchers as to which conditions particularly call for the use of triangulation. The authors conclude by offering guidelines on when and how to employ triangulation techniques.

[1]  Terry M. Coalter,et al.  Context, Cognition, and Common Method Variance: Psychometric and Verbal Protocol Evidence , 1996 .

[2]  John C. Narver,et al.  Does Competitive Environment Moderate the Market Orientation-Performance Relationship? , 1994 .

[3]  D. Paulhus.  Measurement and control of response bias. , 1991 .

[4]  Daniel A. Levinthal,et al.  ABSORPTIVE CAPACITY: A NEW PERSPECTIVE ON LEARNING AND INNOVATION , 1990 .

[5]  Kristin Smith-Crowe,et al.  Accurate tests of statistical significance for r(WG) and average deviation interrater agreement indexes. , 2003, The Journal of applied psychology.

[6]  W. P. Barnett,et al.  The Dynamics of Competitive Intensity , 1997 .

[7]  R. Tweedie,et al.  A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis , 2000 .

[8]  Rory A. Fisher,et al.  Theory of Statistical Estimation , 1925, Mathematical Proceedings of the Cambridge Philosophical Society.

[9]  K. Sutcliffe.  What Executives Notice: Accurate Perceptions in Top Management Teams , 1994 .

[10]  Ishtiaq P. Mahmood,et al.  Inter-industry differences in profitability: the legacy of the structure–efficiency debate revisited , 2009 .

[11]  W. R. Dillon,et al.  An Integrative Look at the Use of Additive and Multiplicative Covariance Structure Models in the Analysis of MTMM Data , 1992 .

[12]  P. Bliese.  Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. , 2000 .

[13]  D. Hambrick.  Strategic Awareness Within Top Management Teams. , 1980 .

[14]  C. Moorman,et al.  Cross-Sectional versus Longitudinal Survey Research: Concepts, Findings, and Guidelines , 2008 .

[15]  M. Ronald Buckley,et al.  Measurement Error and Theory Testing in Consumer Research: An Illustration of the Importance of Construct Validation , 1988 .

[16]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[17]  William R. Dillon,et al.  On the Use of Confirmatory Measurement Models in the Analysis of Multiple-Informant Reports , 1990 .

[18]  John M. Mezias,et al.  Opening Pandora's Box: Studying the Accuracy of Managers' Perceptions , 1996 .

[19]  R. Bagozzi,et al.  On the evaluation of structural equation models , 1988 .

[20]  David B. Pillemer,et al.  Very Long-Term Memories of the First Year in College , 1988 .

[21]  J. Steenkamp,et al.  A Review and Evaluation of Meta-Analysis Practices in Management Research , 2009 .

[22]  Michael Eid,et al.  A multitrait-multimethod model with minimal assumptions , 2000 .

[23]  R. Hastie,et al.  The relationship between memory and judgment depends on whether the judgment task is memory-based or on-line , 1986 .

[24]  R. Shiffrin,et al.  A retrieval model for both recognition and recall. , 1984, Psychological review.

[25]  L. Stern,et al.  Conducting Interorganizational Research Using Key Informants , 1993 .

[26]  William H. Glick,et al.  Common Methods Bias: Does Common Methods Variance Really Bias Results? , 1998 .

[27]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .

[28]  J.J.H. Bartholomée,et al.  Campell, Donald T. , 2005 .

[29]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[30]  Steven E. Scullen,et al.  A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. , 2002, Psychological methods.

[31]  Jeffrey M. Wooldridge,et al.  Introductory Econometrics: A Modern Approach , 1999 .

[32]  D. Campbell,et al.  Unobtrusive Measures: Nonreactive Research in the Social Sciences , 1966 .

[33]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[34]  James J. Chrisman,et al.  Corporate diversification: Entry, strategy, and performance , 1984 .

[35]  William H. Glick,et al.  Conceptualizing and Measuring Organizational and Psychological Climate: Pitfalls in Multilevel Research , 1985 .

[36]  Roger Tourangeau,et al.  Contributions of survey research to the understanding of memory , 1993 .

[37]  L. Phillips.  Assessing Measurement Error in Key Informant Reports: A Methodological Note on Organizational Analysis in Marketing , 1981 .

[38]  Alvin J. Silk,et al.  Measure Unreliability: A Hidden Threat to Cross-National Marketing Research? , 1981 .

[39]  Deborah F. Goodman,et al.  A meta-analysis of interrater and internal consistency reliability of selection interviews. , 1995 .

[40]  N. Denzin,et al.  The Research Act , 1978 .

[41]  Shahid A. Zia,et al.  Competitive Strategy: Techniques for Analyzing Industries & Competitors , 2013 .

[42]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[43]  David M. Sanbonmatsu,et al.  On the automatic activation of attitudes. , 1986, Journal of personality and social psychology.

[44]  P. Slovic.  Convergent validation of risk taking measures. , 1962, Journal of abnormal and social psychology.

[45]  Bryan Dawson,et al.  Method Effects, Measurement Error, and Substantive Conclusions , 2010 .

[46]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[47]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[48]  S. Sharp,et al.  Explaining heterogeneity in meta-analysis: a comparison of methods. , 1997, Statistics in medicine.

[49]  Richard P. Bagozzi,et al.  Multitrait-Multimethod Matrices in Consumer Research , 1991 .

[50]  B. Golden.  The past is the past--or is it? The use of retrospective accounts as indicators of past strategy. , 1992, Academy of Management journal. Academy of Management.

[51]  Coldplay,et al.  X/Y , 2020, The A–Z of Intermarriage.

[52]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[53]  H W Marsh,et al.  Overcoming Problems in Confirmatory Factor Analyses of MTMM Data: The Correlated Uniqueness Model and Factorial Invariance. , 1992, Multivariate behavioral research.

[54]  Jack M. Feldman,et al.  Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. , 1988 .

[55]  Sandy D. Jap.  Pie-Expansion Efforts: Collaboration Processes in Buyer–Supplier Relationships , 1999 .

[56]  George R. Franke,et al.  Salesperson Adaptive Selling Behavior and Customer Orientation: A Meta-Analysis , 2006 .

[57]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[58]  Joseph A. Cote,et al.  Estimating Trait, Method, and Error Variance: Generalizing across 70 Construct Validation Studies , 1987 .

[59]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[60]  Alvin J. Silk,et al.  Measuring Influence in Organizational Purchase Decisions , 1982 .

[61]  S. Rogelberg,et al.  Introduction Understanding and Dealing With Organizational Survey Nonresponse , 2007 .

[62]  G. Huber,et al.  Retrospective reports of strategic‐level managers: Guidelines for increasing their accuracy , 1985 .

[63]  Neil M. A. Hauenstein,et al.  Interrater Agreement Reconsidered: An Alternative to the rwg Indices , 2005 .

[64]  Christian Homburg,et al.  Responsiveness to Customers and Competitors:The Role of Affective and Cognitive Organizational Systems , 2007 .

[65]  S. Kozlowski,et al.  A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. , 1992 .

[66]  John Seidler,et al.  On Using Informants: A Technique for Collecting Quantitative Data and Controlling Measurement Error in Organization Analysis , 1974 .

[67]  L. James,et al.  rwg: An assessment of within-group interrater agreement. , 1993 .

[68]  Richard P. Bagozzi,et al.  Assessing Construct Validity in Organizational Research , 1991 .

[69]  James G. March,et al.  Crossroads---Organizational Performance as a Dependent Variable , 1997 .

[70]  Peter L. Rogers,et al.  Measurement and Control , 1993 .

[71]  T. Jick.  Mixing Qualitative and Quantitative Methods: Triangulation in Action. , 1979 .

[72]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[73]  Seymour Sudman,et al.  A methodological assessment of the use of key informants , 1975 .