Countering method bias in questionnaire-based user studies

Purpose – This paper seeks to discuss reliability problems associated with questionnaires, commonly employed in library and information science. It aims to focus on the effects of “common method variance” (CMV), which is a form of bias, and ways of countering these effects.Design/methodology/approach – The paper critically reviews the use of existing tools for demonstrating reliability in questionnaire‐based studies. In particular, it focuses on Cronbach's alpha, “Harman's single factor test” and Lindell and Whitney's “marker variable” approach. The paper introduces an illustrative case study based on the work on metacognition and web searching. It goes on to make recommendations for researchers considering using a questionnaire‐based approach.Findings – CMV is a problem affecting questionnaire‐based studies in different disciplines across social and information science. Where questionnaire items are more abstract, CMV has been found to be more of a problem. The widely used Cronbach alpha measure, of the ...

[1]  Yazdan Mansourian,et al.  Contextual elements and conceptual components of information visibility on the web , 2008, Libr. Hi Tech.

[2]  Chulmo Koo,et al.  Exploring the effect of strategic positioning on firm performance in the e-business context , 2008, Int. J. Inf. Manag..

[3]  Vandra L. Huber,et al.  Situational moderators of leader reward and punishment behaviors: fact or fiction? , 1984, Organizational behavior and human performance.

[4]  H. Harman.  Modern factor analysis , 1961 .

[5]  Gregory Schraw,et al.  Assessing metacognitive awareness , 1994 .

[6]  Irina Saur-Amaral,et al.  Leader Self-Reported Emotional Intelligence and Perceived Employee Creativity: An Exploratory Study , 2007 .

[7]  J. Metcalfe,et al.  Metacognition : knowing about knowing , 1994 .

[8]  Achilles A. Armenakis,et al.  The Influence of Personality Differences Between Subordinates and Supervisors on Perceptions of LMX , 2008 .

[9]  Naresh K. Malhotra,et al.  Common Method Variance in IS Research: A Comparison of Alternative Approaches and a Reanalysis of Past Research , 2006, Manag. Sci..

[10]  Barry Eaglestone,et al.  Towards "metacognitively aware" IR systems: an initial user study , 2009, J. Documentation.

[11]  Rebecca A. Thacker,et al.  An Examination of the Relationship Between Upward Influence Tactics and Assessments of Promotability , 1995 .

[12]  James Flynn,et al.  Levels of strategic purchasing: Impact on supply integration and performance , 2006 .

[13]  R. Handfield,et al.  Environmental Management Systems and Green Supply Chain Management: Complements for Sustainability? , 2008 .

[14]  William D. Todor,et al.  Relationships Between Leader Reward and Punishment Behavior and Group Processes and Productivity , 1985 .

[15]  Vijay Sethi,et al.  Environmental assessment in strategic information systems planning , 2005, Int. J. Inf. Manag..

[16]  William H. Glick,et al.  Common Methods Bias: Does Common Methods Variance Really Bias Results? , 1998 .

[17]  M. Hannafin,et al.  Cognitive strategies and learning from the world wide web , 1997 .

[18]  Amy B. Woszczynski,et al.  The Problem of Common Method Variance in IS Research , 2004 .

[19]  Philip H. Winne,et al.  Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. , 2007, The British journal of educational psychology.

[20]  Ludwig Bstieler,et al.  Influence of Tie Strength and Behavioural Factors on Effective Knowledge Acquisition: A Study of Korean New Product Alliances , 2008 .

[21]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[22]  P. M. Podsakoff,et al.  Self-Reports in Organizational Research: Problems and Prospects , 1986 .

[23]  Katherine M. White,et al.  Utilizing the Theory of Planned Behavior to Inform Change Management , 2008 .

[24]  Rongqiu Chen,et al.  Examining the Mechanism of the Value Co-Creation with Customers , 2008 .

[25]  Osman M. Karatepe,et al.  Work-Family Conflict and Facilitation in the Hotel Industry , 2008 .

[26]  C. Schriesheim.  The Similarity of Individual Directed and Group Directed Leader Behavior Descriptions , 1979 .

[27]  J. Schriesheim.  The Social Context of Leader-Subordinate Relations: An Investigation of the Effects of Group Cohesiveness , 1980 .

[28]  Peter Vlerick,et al.  The Love of Money and Pay Level Satisfaction: Measurement and Functional Equivalence in 29 Geopolitical Entities around the World , 2006, Management and Organization Review.

[29]  Michael Song,et al.  Conflict management and innovation performance: An integrated contingency perspective , 2006 .

[30]  E. Coiera,et al.  Impact of Web Searching and Social Feedback on Consumer Decision Making: A Prospective Online Experiment , 2008, Journal of medical Internet research.

[31]  Aamir Ali Chughtai,et al.  Impact of Job Involvement on In-Role Job Performance and Organizational Citizenship Behaviour , 2008 .

[32]  J. Flavell.  Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry. , 1979 .

[33]  P. Alexander,et al.  The Interaction of Domain-Specific and Strategic Knowledge in Academic Performance , 1988 .

[34]  Ann L. Brown.  Learning, remembering, and understanding , 1982 .

[35]  Nigel Ford,et al.  Modeling cognitive processes in information seeking: From Popper to Pask , 2004, J. Assoc. Inf. Sci. Technol..

[36]  M. Lindell,et al.  Accounting for common method variance in cross-sectional research designs. , 2001, The Journal of applied psychology.

[37]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[38]  J. Wagner,et al.  Percept-Percept Inflation in Microorganizational Research: An Investigation of Prevalence and Effect , 1994 .

[39]  Donald T. Campbell,et al.  Testing a Single-Factor Model as an Alternative to the Misuse of Partial Correlations in Hypothesis-Testing Research , 1970 .

[40]  Albert L. Lederer,et al.  Rapid business and IT change: drivers for strategic information systems planning? , 2008, Eur. J. Inf. Syst..

[41]  Jamal Abedi,et al.  Reliability and Validity of a State Metacognitive Inventory: Potential for Alternative Assessment , 1996 .

[42]  Hale Kaynak,et al.  The cross-functional coordination between operations, marketing, purchasing and engineering and the impact on performance , 2008, Int. J. Manuf. Technol. Manag..

[43]  Liana Kourteli,et al.  Scanning the business external environment for information: evidence from Greece , 2005, Inf. Res..

[44]  Joseph A. Cote,et al.  Estimating Trait, Method, and Error Variance: Generalizing across 70 Construct Validation Studies: , 1987 .

[45]  E. B. Andersen,et al.  Modern factor analysis , 1961 .

[46]  A. Brown.  Metacognition, executive control, self-regulation, and other more mysterious mechanisms , 1987 .

[47]  Hans De Witte,et al.  Employability and Employees’ Well‐Being: Mediation by Job Insecurity , 2008 .

[48]  A. Carr,et al.  Communication methods, information sharing, supplier development and performance , 2007 .

[49]  William P. Dunlap,et al.  Partialling Factor Scores Does Not Control Method Variance: A Reply to Podsakoff and Todor , 1986 .

[50]  Charles N. Greene,et al.  An Evaluation of Causal Models Linking the Received Role with Job Satisfaction. , 1973 .

[51]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .