Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review.

CONTEXT The quality of consumer health information on the World Wide Web is an important issue for medicine, but to date no systematic and comprehensive synthesis of the methods and evidence has been performed. OBJECTIVES To establish a methodological framework on how quality on the Web is evaluated in practice, to determine the heterogeneity of the results and conclusions, and to compare the methodological rigor of these studies, to determine to what extent the conclusions depend on the methodology used, and to suggest future directions for research. DATA SOURCES We searched MEDLINE and PREMEDLINE (1966 through September 2001), Science Citation Index (1997 through September 2001), Social Sciences Citation Index (1997 through September 2001), Arts and Humanities Citation Index (1997 through September 2001), LISA (1969 through July 2001), CINAHL (1982 through July 2001), PsychINFO (1988 through September 2001), EMBASE (1988 through June 2001), and SIGLE (1980 through June 2001). We also conducted hand searches, general Internet searches, and a personal bibliographic database search. STUDY SELECTION We included published and unpublished empirical studies in any language in which investigators searched the Web systematically for specific health information, evaluated the quality of Web sites or pages, and reported quantitative results. We screened 7830 citations and retrieved 170 potentially eligible full articles. A total of 79 distinct studies met the inclusion criteria, evaluating 5941 health Web sites and 1329 Web pages, and reporting 408 evaluation results for 86 different quality criteria. DATA EXTRACTION Two reviewers independently extracted study characteristics, medical domains, search strategies used, methods and criteria of quality assessment, results (percentage of sites or pages rated as inadequate pertaining to a quality criterion), and quality and rigor of study methods and reporting. DATA SYNTHESIS Most frequently used quality criteria used include accuracy, completeness, readability, design, disclosures, and references provided. Fifty-five studies (70%) concluded that quality is a problem on the Web, 17 (22%) remained neutral, and 7 studies (9%) came to a positive conclusion. Positive studies scored significantly lower in search (P =.02) and evaluation (P =.04) methods. CONCLUSIONS Due to differences in study methods and rigor, quality criteria, study population, and topic chosen, study results and conclusions on health-related Web sites vary widely. Operational definitions of quality criteria are needed.

[1]  Health information during a week of television. , 1972, The New England journal of medicine.

[2]  R. Ostman,et al.  Quality of information in mass media: a barrier to the dental health education of the public. , 1974, Journal of public health dentistry.

[3]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[4]  L Dalton,et al.  Evaluating printed health information for consumers. , 1981, Bulletin of the Medical Library Association.

[5]  Paskoff Bm.  Accuracy of telephone reference service in health sciences libraries. , 1991 .

[6]  G. Guyatt,et al.  Validation of an index of the quality of review articles. , 1991, Journal of clinical epidemiology.

[7]  J A Seaboldt,et al.  Comparison of information obtained from a Usenet newsgroup and from drug information centers. , 1997, American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists.

[8]  What's on the Internet? Services for women affected by HIV and AIDS. , 1997, Health care for women international.

[9]  K Davison,et al.  The quality of dietary information on the World Wide Web. , 1997, Clinical performance and quality health care.

[10]  J C Wyatt,et al.  Commentary: measuring quality and impact of the world wide web , 1997, BMJ.

[11]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[12]  W R Hersh,et al.  Applicability and quality of information for answering clinical questions on the Web. , 1998, JAMA.

[13]  Maugans Ta,et al.  The internet as a pediatric neurosurgery information resource. , 1998 .

[14]  E. Coiera.  Information epidemics, economics, and immunity on the internet , 1998 .

[15]  Coverage and quality of oral cancer information in the popular press: 1987-98. , 1998, Journal of public health dentistry.

[16]  H. J. Mcclung,et al.  The Internet as a source for current patient information. , 1998, Pediatrics.

[17]  C Boyer,et al.  The Health On the Net Code of Conduct for medical and health Websites , 1998, Comput. Biol. Medicine.

[18]  D Charnock,et al.  DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. , 1999, Journal of epidemiology and community health.

[19]  Angel A Hernández-Borges,et al.  Can Examination of WWW Usage Statistics and other Indirect Quality Indicators Help to Distinguish the Relative Quality of Medical websites? , 1999, Journal of medical Internet research.

[20]  J van der Lei,et al.  Rx medication information for the public and the WWW: quality issues. , 1999, Medical informatics and the Internet in medicine.

[21]  A Coulter,et al.  Sharing decisions with patients: is the information good enough? , 1999, BMJ.

[22]  G Eysenbach,et al.  Labeling and Filtering of Medical Information on the Internet , 1999, Methods of Information in Medicine.

[23]  B. Bloom,et al.  Internet Availability of Prescription Pharmaceuticals to the Public , 1999, Annals of Internal Medicine.

[24]  D A Asch,et al.  Direct sale of sildenafil (Viagra) to consumers over the Internet. , 1999, The New England journal of medicine.

[25]  B O'Mahony,et al.  Irish health care Web sites: a review. , 1999, Irish medical journal.

[26]  Paul Kim,et al.  Published criteria for evaluating health related web sites: review , 1999, BMJ.

[27]  Mark A. Musen,et al.  The low availability of metadata elements for evaluating the quality of medical information on the World Wide Web , 1999, AMIA.

[28]  C L Hatfield,et al.  Quality of consumer drug information provided by four Web sites. , 1999, American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists.

[29]  Gunther Eysenbach,et al.  Online Prescribing of Sildanefil (Viagra) on the World Wide Web , 1999, Journal of medical Internet research.

[30]  B. Wright,et al.  Management advice for children with chronic fatigue syndrome: a systematic study of information from the internet , 1999, Irish Journal of Psychological Medicine.

[31]  C. Lee Giles,et al.  Accessibility of information on the web , 1999, Nature.

[32]  Loren H. Rieseberg,et al.  How reliable is science information on the web? , 1999, Nature.

[33]  Patrice Degoulet,et al.  A critical appraisal of the use of Internet for calculating cardiovascular risk , 1999, AMIA.

[34]  J. Biermann,et al.  Evaluation of cancer information on the Internet , 1999, Cancer.

[35]  M A Graber,et al.  Readability levels of patient education material on the World Wide Web. , 1999, The Journal of family practice.

[36]  Assessing the quality of newspaper medical advice columns for elderly readers. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[37]  T. Magee,et al.  Varicose veins on the internet. , 2000, European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery.

[38]  D. Fitzmaurice,et al.  A systematic review of patient information leaflets for hypertension , 2000, Journal of Human Hypertension.

[39]  S. Brewster,et al.  Urology and the Internet: an evaluation of Internet use by urology patients and of information available on urological topics , 2000, BJU international.

[40]  L Hoffman-Goetz,et al.  Quality of Breast Cancer Sites on the World Wide Web , 2000, Canadian journal of public health = Revue canadienne de sante publique.

[41]  Quality of medical information about menorrhagia on the worldwide web , 2000, BJOG : an international journal of obstetrics and gynaecology.

[42]  M H Oermann,et al.  Quality of care information for consumers on the Internet. , 2000, Journal of nursing care quality.

[43]  C. Estrada,et al.  Anticoagulant Patient Information Material Is Written at High Readability Levels , 2000, Stroke.

[44]  P. Lurie,et al.  The Content and Accessibility of Sex Education Information on the Internet , 2000, Health education & behavior : the official publication of the Society for Public Health Education.

[45]  Review and evaluation of anatomy sites on the Internet (updated 1999) , 2000, Surgical and Radiologic Anatomy.

[46]  M. Bogenschutz.  Drug Information Libraries on the Internet , 2000, Journal of psychoactive drugs.

[47]  T Delamothe,et al.  Quality of websites: kitemarking the west wind , 2000, BMJ : British Medical Journal.

[48]  Gunther Eysenbach,et al.  Consumer health informatics , 2000, BMJ : British Medical Journal.

[49]  P. Huynh,et al.  Evaluation of the quality of self-education mammography material available for patients on the Internet. , 2000, Academic radiology.

[50]  H. Christensen,et al.  Quality of web based information on treatment of depression: cross sectional survey , 2000, BMJ : British Medical Journal.

[51]  M. Latthe,et al.  Quality of information on emergency contraception on the Internet. , 2000, The British journal of family planning.

[52]  D. Bozentka,et al.  Evaluating the Source and Content of Orthopaedic Information on the Internet: The Case of Carpal Tunnel Syndrome* , 2000, The Journal of bone and joint surgery. American volume.

[53]  Y. L. Jiang,et al.  Quality evaluation of orthodontic information on the World Wide Web. , 2000, American journal of orthodontics and dentofacial orthopedics : official publication of the American Association of Orthodontists, its constituent societies, and the American Board of Orthodontics.

[54]  Quality of information on female sterilisation on the Internet. , 2000, Journal of obstetrics and gynaecology : the journal of the Institute of Obstetrics and Gynaecology.

[55]  Orthopdie im World Wide Web: Universitts-Homepages der G7-Staaten , 2000 .

[56]  J. Miles,et al.  Slimming on the Internet , 2000, Journal of the Royal Society of Medicine.

[57]  F L Wilson,et al.  An analysis of the readability and cultural sensitivity of information on the National Cancer Institute's Web site: CancerNet. , 2000, Oncology nursing forum.

[58]  Feili Tu,et al.  It Is Not Just a Matter of Ethics: A Survey of the Provision of Health Disclaimers, Caveats, and Other Health-related Alerts in Consumer Health Information on Eating Disorders on the Internet , 2000 .

[59]  K A Arndt,et al.  Accessibility and reliability of cutaneous laser surgery information on the World Wide Web. , 2000, Journal of the American Academy of Dermatology.

[60]  S. Peroutka.  Analysis of Internet Sites for Headache , 2001, Cephalalgia : an international journal of headache.

[61]  Dan Brickley,et al.  A metadata vocabulary for self- and third-party labeling of health web-sites: Health Information Disclosure, Description and Evaluation Language (HIDDEL) , 2001, AMIA.

[62]  Information about age-related macular degeneration on the Internet. , 2001 .

[63]  L. Barot,et al.  The Internet as a source of information on breast augmentation. , 2001, Plastic and reconstructive surgery.

[64]  R. Kravitz,et al.  Health information on the Internet: accessibility, quality, and readability in English and Spanish. , 2001, JAMA.

[65]  T. L. Lissman,et al.  A critical review of internet information about depression. , 2001, Psychiatric services.

[66]  Is current information available useful for patients and their families? , 2001, Annals of the Royal College of Surgeons of England.

[67]  Gunther Eysenbach,et al.  An Ontology of Quality Initiatives and a Model for Decentralized, Collaborative Quality Management on the (Semantic) World Wide Web , 2001, Journal of medical Internet research.

[68]  Ahmad Risk,et al.  Review Of Internet Health Information Quality Initiatives , 2001, Journal of medical Internet research.

[69]  G ter Riet,et al.  Comparison of search strategies and quality of medical information of the Internet: a study relating to ankle sprain. , 2001, Injury.

[70]  E. Boyer,et al.  Web sites with misinformation about illicit drugs. , 2001, The New England journal of medicine.

[71]  D M D'Alessandro,et al.  The readability of pediatric patient education materials on the World Wide Web. , 2001, Archives of pediatrics & adolescent medicine.

[72]  A. Sing,et al.  Problems and risks of unsolicited e-mails in patient-physician encounters in travel medicine settings. , 2006, Journal of travel medicine.

[73]  C. Corpron,et al.  Evaluation of pediatric surgery information on the Internet. , 2001, Journal of pediatric surgery.

[74]  G. Eysenbach,et al.  Ethical issues in qualitative research on internet communities , 2001, BMJ : British Medical Journal.