Evaluation of the Impact of User-Cognitive Styles on the Assessment of Text Summarization

Text summarization techniques have been found to be effective with regard to helping users find relevant information faster. The effectiveness and efficiency of a user's performance in an information-seeking task can greatly be improved if he/she needs to only look at a summary that includes the relevant information presented in his/her preferred manner. On the other hand, if the main idea is misrepresented and/or omitted altogether from a summary, it may take users more time to solve a target problem or, even worse, lead users to make incorrect decisions. There is an important need to design a personalized text summarization system that takes into account both what a user is currently interested in and how a user perceives information. The latter factor is referred to as a user's cognitive styles. Although there are some existing approaches that have employed a user's interests to help in the design of a personalized text summarization system, there has been inadequate focus on exploring cognitive styles. This paper aims at studying the impact of a user's cognitive styles when assessing multidocument summaries. In particular, we choose two dimensions of a user's cognitive style - the analytic/wholist and verbal/imagery dimensions - and study their impacts on how a user assesses a summary that was generated from a set of documents. In particular, the type of a document set refers to whether the set's content is loosely or closely related. We use a document set type to explore if there are any differences in the users' assessments of summaries that were generated from sets of different types. The results of this paper show that different users have different assessments with regard to information coverage and the way that information is presented in both loosely and closely related document sets. In addition, we found that the coherency ratings that were given to summaries from the two types of document sets were significantly different between the analytic and wholist groups. This result leads us to investigate the impact of a user's cognitive styles and the following two factors that directly relate to the coherence of a summary: 1) graph entropy and 2) the percentage of stand-alone concepts. We found that these two factors and a user's cognitive styles affect a user's ratings on the coherency of a summary.

[1]  E. Santos,et al.  Impacts of User Modeling on Personalization of Information Retrieval : An Evaluation with Human Intelligence Analysts , 2005 .

[2]  Carol Tenopir,et al.  Users' interaction with World Wide Web resources: an exploratory study using a holistic approach , 2000, Inf. Process. Manag..

[3]  D. Cook.  Learning and Cognitive Styles in Web-Based Learning: Theory, Evidence, and Application , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[4]  R. Riding,et al.  Cognitive Styles—an overview and integration , 1991 .

[5]  David Miller,et al.  Web search strategies and human individual differences: Cognitive and demographic factors, Internet attitudes, and approaches , 2005, J. Assoc. Inf. Sci. Technol..

[6]  Peter Brusilovsky,et al.  Preface to Special Issue on User Modeling for Web Information Retrieval , 2004, User Modeling and User-Adapted Interaction.

[7]  Daniel Dominic Sleator,et al.  Parsing English with a Link Grammar , 1995, IWPT.

[8]  A. Field.  Discovering statistics using SPSS for Windows. , 2000 .

[9]  Nigel Ford,et al.  Learning Styles and Strategies of Postgraduate Students , 1985 .

[10]  Eugene Santos,et al.  KAVANAH: AN ACTIVE USER INTERFACE INFORMATION RETRIEVAL APPLICATION , 2001 .

[11]  Andy P. Field,et al.  Discovering Statistics Using SPSS for Windows: Advanced Techniques for Beginners , 2000 .

[12]  Eugene Santos,et al.  User Modelling for Intent Prediction in Information Analysis , 2003 .

[13]  Eugene Santos,et al.  Evaluation of the Effects of User-Sensitivity on Text Summarization , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[14]  Martin Hassel.  Resource Lean and Portable Automatic Text Summarization , 2007 .

[15]  Eugene Santos,et al.  Capturing User Intent for Information Retrieval , 2004 .

[16]  Mark T. Maybury,et al.  Advances in Automatic Text Summarization , 1999 .

[17]  Sanghee Yeo.  Handbook of individual differences, learning, and instruction. , 2010, Korean journal of medical education.

[18]  George Ghinea,et al.  Measuring quality of perception in distributed multimedia: Verbalizers vs. imagers , 2008, Comput. Hum. Behav..

[19]  Eugene Santos,et al.  Modeling Users for Adaptive Information Retrieval by Capturing User Intent , 2009 .

[20]  Richard Riding,et al.  Cognitive style, gender and learning from multi-media materials in 11-year-old children , 1999, Br. J. Educ. Technol..

[21]  Inderjeet Mani,et al.  The Challenges of Automatic Summarization , 2000, Computer.

[22]  James P. Callan,et al.  Passage-level evidence in document retrieval , 1994, SIGIR '94.

[23]  Richard J. Riding,et al.  The Effect of Cognitive Style on the Preferred Format of Instructional Material , 1997 .

[24]  Eugene Santos,et al.  Impacts of Analysts' Cognitive Styles on the Analytic Process , 2010, 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[25]  Noémie Elhadad.  User-Sensitive Text Summarization , 2004, AAAI.

[26]  R. Riding,et al.  Cognitive Styles and Learning Strategies , 2013 .

[27]  Mark Sanderson,et al.  Advantages of query biased summaries in information retrieval , 1998, SIGIR '98.

[28]  Martin Graff,et al.  Learning from web-based instructional systems and cognitive style , 2003, Br. J. Educ. Technol..

[29]  C. A. Moore,et al.  Field-Dependent and Field-Independent Cognitive Styles and Their Educational Implications , 1977 .

[30]  Ian J. Deary,et al.  The reliability of Riding’s Cognitive Style Analysis test , 2003 .

[31]  Eugene Santos,et al.  Empirical Evaluation of Adaptive User Modeling in a Medical Information Retrieval Application , 2003, User Modeling.

[32]  G. Pask.  STYLES AND STRATEGIES OF LEARNING , 1976 .

[33]  Elizabeth Boling,et al.  Information-conveying approaches and cognitive styles of mental modeling in a hypermedia-based learning environment , 2008 .

[34]  Pablo Gervás,et al.  User-model based personalized summarization , 2007, Inf. Process. Manag..

[35]  David Miller,et al.  The role of individual differences in Internet searching: an empirical study , 2001 .

[36]  Massih-Reza Amini,et al.  The use of unlabeled data to improve supervised learning for text summarization , 2002, SIGIR '02.

[37]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[38]  George V. Kissel.  The effect of computer experience on subjective and objective software usability measures , 1995, CHI '95.

[39]  Marc H. Rosa.  Relationships between Cognitive Styles and Reading Comprehension of Expository Text of African American Male Students , 1994 .

[40]  Robert W. Bailey,et al.  Performance vs. Preference , 1993 .

[41]  Frances Wood,et al.  Information skills, searching behaviour and cognitive styles for student-centred learning: a computer-assisted learning approach , 1996, J. Inf. Sci..

[42]  Amanda Spink,et al.  Information seeking and mediated searching. Part 4. Cognitive styles in information seeking , 2002, J. Assoc. Inf. Sci. Technol..

[43]  Mirella Lapata,et al.  Graph Connectivity Measures for Unsupervised Word Sense Disambiguation , 2007, IJCAI.

[44]  Hoa Trang Dang,et al.  Overview of DUC 2005 , 2005 .

[45]  Eugene Santos,et al.  What Makes a Good Summary , 2009 .

[46]  Manuel J. Maña López,et al.  Using and Evaluating User Directed Summaries to Improve Information Access , 1999, ECDL.

[47]  Martin Graff,et al.  Cognitive Style and Attitudes Towards Using Online Learning and Assessment Methods , 2003 .

[48]  Enrique Frías-Martínez,et al.  Automatic cognitive style identification of digital library users for personalization , 2007, J. Assoc. Inf. Sci. Technol..

[49]  Pablo Gervás,et al.  Evaluation of a System for Personalized Summarization of Web Contents , 2005, User Modeling.

[50]  Robert D. Macredie,et al.  Cognitive styles and hypermedia navigation: Development of a learning model , 2002, J. Assoc. Inf. Sci. Technol..