Analysis of PubMed User Sessions Using a Full-Day PubMed Query Log: A Comparison of Experienced and Nonexperienced PubMed Users

Background PubMed is the largest biomedical bibliographic information source on the Internet. PubMed has been considered one of the most important and reliable sources of up-to-date health care evidence. Previous studies examined the effects of domain expertise/knowledge on search performance using PubMed. However, very little is known about PubMed users’ knowledge of information retrieval (IR) functions and their usage in query formulation. Objective The purpose of this study was to shed light on how experienced/nonexperienced PubMed users perform their search queries by analyzing a full-day query log. Our hypotheses were that (1) experienced PubMed users who use system functions quickly retrieve relevant documents and (2) nonexperienced PubMed users who do not use them have longer search sessions than experienced users. Methods To test these hypotheses, we analyzed PubMed query log data containing nearly 3 million queries. User sessions were divided into two categories: experienced and nonexperienced. We compared experienced and nonexperienced users per number of sessions, and experienced and nonexperienced user sessions per session length, with a focus on how fast they completed their sessions. Results To test our hypotheses, we measured how successful information retrieval was (at retrieving relevant documents), represented as the decrease rates of experienced and nonexperienced users from a session length of 1 to 2, 3, 4, and 5. The decrease rate (from a session length of 1 to 2) of the experienced users was significantly larger than that of the nonexperienced groups. Conclusions Experienced PubMed users retrieve relevant documents more quickly than nonexperienced PubMed users in terms of session length.

[1]  Pier Luca Lanzi,et al.  Mining interesting knowledge from weblogs: a survey , 2005, Data Knowl. Eng..

[2]  Boryung Ju,et al.  Does domain knowledge matter: Mapping users' expertise to their information interactions , 2007, J. Assoc. Inf. Sci. Technol..

[3]  Bradley M. Hemminger,et al.  Mining connections between chemicals, proteins, and diseases extracted from Medline annotations , 2010, J. Biomed. Informatics.

[4]  Damien Palacio,et al.  Query Operators Shown Beneficial for Improving Search Results , 2011, TPDL.

[5]  Soohyung Joo,et al.  Factors affecting the selection of search tactics: Tasks, knowledge, process, and systems , 2012, Inf. Process. Manag..

[6]  Sophia P Gladding,et al.  Should We Google It? Resource Use by Internal Medicine Residents for Point-of-Care Clinical Decision Making , 2013, Academic medicine : journal of the Association of American Medical Colleges.

[7]  Zhiyong Lu,et al.  Semi-automatic semantic annotation of PubMed queries: A study on quality, efficiency, satisfaction , 2011, J. Biomed. Informatics.

[8]  R. Thiele,et al.  Speed, accuracy, and confidence in Google, Ovid, PubMed, and UpToDate: results of a randomised trial , 2010, Postgraduate Medical Journal.

[9]  Stephen J. Westerman,et al.  Individual differences in human-computer interaction , 1993 .

[10]  Bryce Allen,et al.  Topic Knowledge and Online Catalog Search Formulation , 1991, The Library Quarterly.

[11]  Zhiyong Lu,et al.  Identifying related journals through log analysis , 2009, Bioinform..

[12]  Marie-Christine Jaulent,et al.  Improving information retrieval using Medical Subject Headings Concepts: a test case on rare and chronic diseases. , 2012, Journal of the Medical Library Association : JMLA.

[13]  Daqing He,et al.  Detecting session boundaries from Web user logs , 2000 .

[14]  Jean-François Rouet,et al.  The Skills of Document Use: From Text Comprehension to Web-Based Learning , 2006 .

[15]  R Haux,et al.  Knowledge retrieval as one type of knowledge-based decision support in medicine: results of an evaluation study. , 1996, International journal of bio-medical computing.

[16]  Thomas Agoritsas,et al.  Sensitivity and Predictive Value of 15 PubMed Search Strategies to Answer Clinical Questions Rated Against Full Systematic Reviews , 2012, Journal of medical Internet research.

[17]  Darlene Chapman.  Advanced search features of PubMed. , 2009, Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l'Academie canadienne de psychiatrie de l'enfant et de l'adolescent.

[18]  Susan Wiedenbeck,et al.  PATTERNS OF INFORMATION SEEKING ON THE WEB: A QUALITATIVE STUDY OF DOMAIN EXPERTISE AND WEB EXPERTISE , 2003 .

[19]  C Nankivell,et al.  Networked information and clinical decision making: the experience of Birmingham Heartlands and Solihull National Health Service Trust (Teaching) , 2001, Medical education.

[20]  Kyung-Sun Kim,et al.  Cognitive style and on-line database search experience as predictors of Web search performance , 2000, J. Am. Soc. Inf. Sci..

[21]  Lotty Hooft,et al.  What Are the Barriers to Residents' Practicing Evidence-Based Medicine? A Systematic Review , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[22]  Yan Zhang,et al.  Patterns of Journal Use by Scientists through Three Evolutionary Phases , 2003, D Lib Mag..

[23]  W R Hersh,et al.  How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review. , 1998, JAMA.

[24]  Zhiyong Lu,et al.  Evaluation of query expansion using MeSH in PubMed , 2009, Information Retrieval.

[25]  Filip Radlinski,et al.  Query chains: learning to rank from implicit feedback , 2005, KDD '05.

[26]  Zhiyong Lu,et al.  Understanding PubMed® user search behavior through log analysis , 2009, Database J. Biol. Databases Curation.

[27]  R Brian Haynes,et al.  Retrieving Clinical Evidence: A Comparison of PubMed and Google Scholar for Quick Clinical Searches , 2013, Journal of medical Internet research.

[28]  Klaus Nordhausen,et al.  Modeling successful performance in Web searching , 2006, J. Assoc. Inf. Sci. Technol..

[29]  Amanda Spink,et al.  Real life, real users, and real needs: a study and analysis of user queries on the web , 2000, Inf. Process. Manag..

[30]  Fang Liu,et al.  SLIM: an alternative Web interface for MEDLINE/PubMed searches – a preliminary study , 2005, BMC Medical Informatics Decis. Mak..

[31]  Stefano Bonassi,et al.  Development of search filters for retrieval of literature on the molecular epidemiology of cancer. , 2010, Mutation research.

[32]  S. Fincher,et al.  Designing for Expert Information Finding Strategies , 2005 .

[33]  Kambiz Bahaadinbeigy,et al.  MEDLINE versus EMBASE and CINAHL for telemedicine searches. , 2010, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[34]  Mattox.  Welcome to ARCHIVES CME , 2000, Archives of otolaryngology--head & neck surgery.

[35]  D. Tabatabai,et al.  How experts and novices search the Web , 2005 .

[36]  Eve-Marie Lacroix,et al.  The US National Library of Medicine in the 21st century: expanding collections, nontraditional formats, new audiences. , 2002, Health information and libraries journal.

[37]  Illhoi Yoo,et al.  Recent research for MEDLINE/PubMed: short review , 2010, DTMBIO '10.

[38]  R. Cullen,et al.  In search of evidence: family practitioners' use of the Internet for clinical information. , 2002, Journal of the Medical Library Association : JMLA.

[39]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988, J. Am. Soc. Inf. Sci..

[40]  Don R. Swanson,et al.  Information Retrieval as a Trial-And-Error Process , 1977, The Library Quarterly.

[41]  Charles P. Friedman,et al.  Research Paper: Factors Associated with Success in Searching MEDLINE and Applying Evidence to Answer Clinical Questions , 2002, J. Am. Medical Informatics Assoc..

[42]  Randy R Richter,et al.  Using MeSH (Medical Subject Headings) to Enhance PubMed Search Strategies for Evidence-Based Practice in Physical Therapy , 2011, Physical Therapy.

[43]  Doug Downey,et al.  Models of Searching and Browsing: Languages, Studies, and Application , 2007, IJCAI.

[44]  Rebecca Nugent,et al.  Medical literature searches: a comparison of PubMed and Google Scholar. , 2012, Health information and libraries journal.

[45]  Gary Marchionini,et al.  Information Seeking in Electronic Environments , 1995 .

[46]  Christine Ros,et al.  Effects of domain knowledge on reference search with the PubMed database: An experimental study , 2009, J. Assoc. Inf. Sci. Technol..

[47]  Karen M. Drabenstott,et al.  Do nondomain experts enlist the strategies of domain experts? , 2003, J. Assoc. Inf. Sci. Technol..

[48]  Paolo Gardois,et al.  Effectiveness of bibliographic searches performed by paediatric residents and interns assisted by librarians. A randomised controlled trial. , 2011, Health information and libraries journal.

[49]  Judit Bar-Ilan,et al.  Preference for electronic format of scientific journals—A case study of the Science Library users at the Hebrew University , 2005 .

[50]  Elizabeth D. Liddy,et al.  The effects of expertise and feedback on search term selection and subsequent learning , 2005, J. Assoc. Inf. Sci. Technol..

[51]  Zhiyong Lu,et al.  Finding Query Suggestions for PubMed , 2009, AMIA.

[52]  Swapnesh C. Patel,et al.  Effectiveness of expert semantic knowledge as a navigational aid within hypertext , 1998, Behav. Inf. Technol..

[53]  Amit X. Garg,et al.  Impact of PubMed search filters on the retrieval of evidence by physicians , 2012, Canadian Medical Association Journal.

[54]  Stephen P. Harter.  Online Searching Styles: An Exploratory Study. , 1984 .

[55]  Roy Rada,et al.  Interacting With Hypertext: A Meta-Analysis of Experimental Studies , 1996, Hum. Comput. Interact..

[56]  Masoomeh Faghankhani,et al.  A comparison of answer retrieval through four evidence-based textbooks (ACP PIER, Essential Evidence Plus, First Consult, and UpToDate): A randomized controlled trial , 2011, Medical teacher.

[57]  Steven W. Brown,et al.  The effects and interaction of spatial visualization and domain expertise on information seeking , 2004, Comput. Hum. Behav..

[58]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. , 1993 .

[59]  K. Lasserre,et al.  Expert searching in health librarianship: a literature review to identify international issues and Australian concerns. , 2012, Health information and libraries journal.

[60]  Zhiyong Lu,et al.  Improving accuracy for identifying related PubMed queries by an integrated approach , 2009, J. Biomed. Informatics.

[61]  A. Sood,et al.  Literature search using PubMed: an essential tool for practicing evidence- based medicine. , 2006, The Journal of the Association of Physicians of India.

[62]  Bernard J. Jansen.  Limits of the Web Log Analysis Artifacts , 2006 .

[63]  David A. Cook,et al.  Features of Effective Medical Knowledge Resources to Support Point of Care Learning: A Focus Group Study , 2013, PloS one.

[64]  W. Hersh,et al.  Factors associated with successful answering of clinical questions using an information retrieval system. , 2002, Bulletin of the Medical Library Association.

[65]  Amanda Spink,et al.  Searching heterogeneous collections on the Web: behaviour of Excite users , 1998, Inf. Res..

[66]  Amanda Spink,et al.  Defining a session on Web search engines , 2007, J. Assoc. Inf. Sci. Technol..

[67]  C J Walker,et al.  A study to enhance clinical end-user MEDLINE search skills: design and baseline findings. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[68]  Karen S Davies.  Physicians and their use of information: a survey comparison between the United States, Canada, and the United Kingdom. , 2011, Journal of the Medical Library Association : JMLA.

[69]  K. A. McKibbon,et al.  Online access to MEDLINE in clinical settings. A study of use and usefulness. , 1990, Annals of internal medicine.

[70]  Christoph Hölscher,et al.  Web search behavior of Internet experts and newbies , 2000, Comput. Networks.

[71]  D Fitzgerald,et al.  How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. , 1990, Computers and biomedical research, an international journal.

[72]  Karen Markey.  Twenty-five years of end-user searching, Part 1: Research findings , 2007 .

[73]  M L Pao,et al.  Factors affecting students' use of MEDLINE. , 1993, Computers and biomedical research, an international journal.

[74]  Christine Ros,et al.  How do scientists select articles in the PubMed database? An empirical study of criteria and strategies , 2012 .

[75]  James E. Pitkow,et al.  Characterizing Browsing Strategies in the World-Wide Web , 1995, Comput. Networks ISDN Syst..

[76]  S. D. De Groote,et al.  Measuring use patterns of online journals and databases. , 2003, Journal of the Medical Library Association : JMLA.

[77]  Gary Marchionini.  Information Seeking in Full-Text End-User-Oriented Search Systems: The Roles of Domain and Search Expertise , 1993 .

[78]  Christine Ros,et al.  The use of online electronic information resources in scientific research: The case of neuroscience , 2007 .

[79]  Andrei Broder,et al.  A taxonomy of web search , 2002, SIGF.

[80]  Jaime Teevan,et al.  Query log analysis: social and technological challenges , 2007, SIGF.

[81]  Tracy Y Allen,et al.  How to find evidence when you need it, part 2: a clinician's guide to MEDLINE: the basics. , 2002, Annals of emergency medicine.

[82]  Monika Henzinger,et al.  Analysis of a very large web search engine query log , 1999, SIGF.

[83]  Christopher C. Yang,et al.  Mining related queries from search engine query logs , 2006, WWW '06.

[84]  Yen-Jen Oyang,et al.  Relevant term suggestion in interactive web search based on contextual information in query session logs , 2003, J. Assoc. Inf. Sci. Technol..

[85]  Daniel Gayo-Avello,et al.  A survey on session detection methods in query logs and a proposal for future evaluation , 2009, Inf. Sci..

[86]  Elmer V. Bernstam,et al.  A day in the life of PubMed: analysis of a typical day's query log. , 2007, Journal of the American Medical Informatics Association : JAMIA.

[87]  Barbara M. Wildemuth,et al.  Medical Students' Personal Knowledge, Searching Proficiency, and Database Use in Problem Solving , 1995, J. Am. Soc. Inf. Sci..

[88]  Abu Saleh Mohammad Mosa,et al.  A Study on Pubmed Search Tag Usage Pattern: Association Rule Mining of a Full-day Pubmed Query Log , 2013, BMC Medical Informatics and Decision Making.

[89]  Hamid Reza Baradaran,et al.  To Compare PubMed Clinical Queries and UpToDate in Teaching Information Mastery to Clinical Residents: A Crossover Randomized Controlled Trial , 2011, PloS one.

[90]  Betty K. Oldroyd.  Study of strategies used in online searching 5: differences between the experienced and the inexperienced searcher , 1984 .

[91]  W. Hersh,et al.  Use of a multi-application computer workstation in a clinical setting. , 1994, Bulletin of the Medical Library Association.