Web page quality: can we measure it and what do we find? A report of exploratory findings.

BACKGROUND The aim of this study was to report exploratory findings from an attempt to quantify the quality of a sample of World Wide Web (WWW) pages relating to MMR vaccine that a typical user might locate. METHOD Forty pages obtained from a search of the WWW using two search engines and the search expression 'mmr vaccine' were analysed using a standard proforma. The proforma looked at the information the pages contained in terms of three categories: content, authorship and aesthetics. The information from each category was then quantified into a summary statistic, and receiver operating characteristic (ROC) curves were generated using a 'gold standard' of quality derived from the published literature. Optimal cut-off points for each of the three sections were calculated that best discriminated 'good' from 'bad' pages. Pages were also assessed as to whether they were pro- or anti-vaccination. RESULTS For this sample, the combined contents and authorship score, with a cut-off of five, was a good discriminator, having 88 per cent sensitivity and 92 per cent specificity. Aesthetics was not a good discriminator. In the sample, 32.5 per cent of pages were pro-vaccination; 42.5 per cent were anti-vaccination and 25 per cent were neutral. The relative risk of being of poor quality if anti-vaccination was 3.3 (95 per cent confidence interval 1.8, 6.1). CONCLUSION The sample of Web pages did contain some quality information on MMR vaccine. It also contained a great deal of misleading, inaccurate data. The proforma, combined with a knowledge of the literature, may help to distinguish between the two.

[1]  Alex M. Andrew,et al.  Medical Information on the Internet: A Guide for Health Professionals , 2002 .

[2]  Paul Kim,et al.  Published criteria for evaluating health related web sites: review , 1999, BMJ.

[3]  A. Sabrá,et al.  Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children , 1998, The Lancet.

[4]  Selway.  MMR vaccination and autism 1998. Medical practitioners need to give more than reassurance. , 1998, BMJ.

[5]  A R Jadad,et al.  Rating health information on the Internet: navigating to knowledge or to Babel? , 1998, JAMA.

[6]  E. Miller,et al.  Impact of anti-vaccine movements on pertussis control: the untold story , 1998, The Lancet.

[7]  MMR vaccine coverage falls after adverse publicity. , 1998, Communicable disease report. CDR weekly.

[8]  S M Dorman,et al.  Evaluating health-related Web sites. , 1997, The Journal of school health.

[9]  P. Impicciatore,et al.  Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home , 1997, BMJ.

[10]  Abby Day,et al.  A model for monitoring Web site effectiveness , 1997, Internet Res..

[11]  H. Eric Branscomb.  Casting Your Net: A Student's Guide to Research on the Internet , 1997 .

[12]  G D Lundberg,et al.  Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. , 1997, JAMA.

[13]  Ann Bowling,et al.  Research Methods in Health: Investigating Health and Health Services , 1997 .

[14]  L. Lynd,et al.  Drug and Poison Information Resources on the Internet, Part 2: Identification and Evaluation , 1996, Pharmacotherapy.

[15]  H Bower,et al.  Internet sees growth of unverified health claims , 1996, BMJ.

[16]  M Pallen,et al.  Guide to the Internet: The world wide web , 1995, BMJ.

[17]  E K Shultz,et al.  Multivariate receiver-operating characteristic curve analysis: prostate cancer screening as an example. , 1995, Clinical chemistry.

[18]  M. Zweig,et al.  Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. , 1993, Clinical chemistry.

[19]  J R Beck,et al.  The use of relative operating characteristic (ROC) curves in test performance evaluation. , 1986, Archives of pathology & laboratory medicine.

[20]  J. Hanley,et al.  A method of comparing the areas under receiver operating characteristic curves derived from the same cases. , 1983, Radiology.

[21]  J. Swets.  ROC analysis applied to the evaluation of medical imaging techniques. , 1979, Investigative radiology.

[22]  L B Lusted,et al.  Radiographic applications of receiver operating characteristic (ROC) curves. , 1974, Radiology.