Exploring the Relationship between User Satisfaction and Relevance in Information Systems

The goal of this research was to better understand the relationship between relevance and user-satisfaction, the two predominant aspects of user-based performance in information systems. This project unconfounds relevance and user-satisfaction assessments of system performance at the retrieved item level. To minimize the idiosyncracies of any one system, a generalized, naturalistic information system was employed in this study. Respondents completed sense-making timeline questionnaires in which they described a recent need they had for geographic information. Retrieved documents from the generalized system consisted of the responses users obtained while resolving their information needs. Respondents directly provided process, product, cost-benefit, and overall satisfaction assessments with the generalized geographic system. Relevance judgments of retrieved items were obtained through content analysis from sense-making questionnaires as a secondary observation technique. The content analysis provided relevance values on both five-category and two-category scales. Results indicate that relevance has strong relationships (gamma values from 0.74 to 0.89) with process, product and overall user satisfaction measures while relevance and cost-benefit satisfaction measures have no significant relationship (gamma value of 0.049). This analysis also indicates that neither relevance nor user-satisfaction subsumes the other concept, and that understanding the proper units of analysis for these measures helps resolve the paradox of the management information system and information science literatures not informing each other concerning user-based information system performance measures.

[1]  Constance A. Mellon.  Naturalistic Inquiry for Library Science: Methods and Applications for Research, Evaluation, and Teaching , 1990 .

[2]  Thomas J. Froehlich,et al.  Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance research , 1994 .

[3]  Michael J. Ginzberg,et al.  Finding an adequate measure of or/ms effectiveness , 1978 .

[4]  Robert S. Taylor,et al.  Value-Added Processes in Information Systems , 1987 .

[5]  R. Weber.  Basic Content Analysis , 1986 .

[6]  Melvin J. Voigt,et al.  Progress in Communication Sciences , 1982 .

[7]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[8]  T. Park.  The Nature of Relevance in Information Retrieval: An Empirical Study , 1993, The Library Quarterly.

[9]  Myron Henry Gluck.  Understanding performance in information systems : an investigation of system and user views of geographic information , 1993 .

[10]  Michael B. Eisenberg.  Measuring relevance judgments , 1988, Inf. Process. Manag..

[11]  J. Janes.  Other people's judgments: a comparison of users' and others' judgments of document relevance, topicality, and utility , 1994 .

[12]  Louise T. Su.  The Relevance of Recall and Precision in User Evaluation , 1994, J. Am. Soc. Inf. Sci..

[13]  Taemin Kim Park.  Toward a theory of user-based relevance: a call for a new paradigm of inquiry , 1994 .

[14]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[15]  Joseph W. Janes.  On the Distribution of Relevance Judgments. , 1993 .

[16]  N. Melone.  A theoretical assessment of the user-satisfaction construct in information systems research , 1990 .

[17]  Robert W. Zmud,et al.  AN EMPIRICAL INVESTIGATION OF THE DIMENSIONALITY OF THE CONCEPT OF INFORMATION , 1978 .

[18]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[19]  R. Oliver.  Measurement and evaluation of satisfaction processes in retail settings. , 1981 .

[20]  Gregory B. Newby.  Toward a Reassessment of Individual Differences for Information Systems: The Power of User-Based Situational Predictors. , 1991 .

[21]  B. Dervin,et al.  Information needs and uses. , 1986 .

[22]  Patricia Tobaka Fletcher.  An examination of situational dimensions in the information behaviors of general managers , 1990 .

[23]  W. DeSarbo,et al.  Response Determinants in Satisfaction Judgments , 1988 .

[24]  Joseph Janes,et al.  Relevance Judgments of Actual Users and Secondary Judges: A Comparative Study , 1992, The Library Quarterly.

[25]  R. Oliver.  A Cognitive Model of the Antecedents and Consequences of Satisfaction Decisions , 1980 .

[26]  E. Burton Swanson,et al.  A Note on Information Attributes , 1985, J. Manag. Inf. Syst..

[27]  Sammy W. Pearson,et al.  Development of a Tool for Measuring and Analyzing Computer User Satisfaction , 1983 .

[28]  Jack D. Glazier,et al.  Qualitative Research in Information Management , 1992 .

[29]  Michael Sanford Nilan.  Structural constraints and situational information seeking : a test of two predictors in a sense-making context , 1985 .

[30]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[31]  Louise T. Su.  Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[32]  C. W. Cleverdon,et al.  Evaluation Tests of Information Retrieval Systems , 1970 .

[33]  Charles A. Gallagher.  Perceptions of the Value of a Management Information System , 1974 .

[34]  A. Liebetrau.  Measures of association , 1983 .

[35]  Michael B. Eisenberg,et al.  A re-examination of relevance: toward a dynamic, situational definition , 1990, Inf. Process. Manag..

[36]  William S. Cooper,et al.  A definition of relevance for information retrieval , 1971, Inf. Storage Retr..

[37]  Stuart A. Sutton.  The role of attorney mental models of law in case relevance determinations: an exploratory analysis , 1994 .

[38]  Brenda Dervin,et al.  From the mind’s eye of the user: The sense-making qualitative-quantitative methodology. , 1992 .

[39]  Christine L Borgman.  Human-computer interaction with information retrieval systems: Understanding complex communication behavior. , 1986 .

[40]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[41]  Myke Gluck.  Understanding performance in information systems: blending relevance and competence , 1995 .

[42]  Daniel Robey,et al.  User Attitudes and Management Information System Use , 1979 .

[43]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[44]  E. B. Swanson,et al.  Management Information Systems: Appreciation and Involvement , 1974 .

[45]  S. P. Harter.  Psychological relevance and information science , 1992 .

[46]  Judith Anne Tessier.  Toward the understanding of user satisfaction: a multiattribute study of user evaluations of computer-based literature searches in medical libraries , 1981 .

[47]  H. T. Reynolds,et al.  Analysis of nominal data , 1977 .

[48]  Robert W. Zmud,et al.  An Attribute Space for Organizational Communication Channels , 1990, Inf. Syst. Res..