Proteomics research to discover markers: what can we learn from Netflix?

BACKGROUND Research in the field of proteomics to discover markers for detection of cancer has produced disappointing results, with few markers gaining US Food and Drug Administration approval, and few claims borne out when subsequently tested in rigorous studies. What is the role of better mathematical or statistical analysis in improving the situation? CONTENT This article examines whether a recent successful Netflix-sponsored competition using mathematical analysis to develop a prediction model for movie ratings of individual subscribers can serve to improve studies of markers in the field of proteomics. Netflix developed a database of movie preferences of individual subscribers using a longitudinal cohort research design. Groups of researchers then competed to develop better ways to analyze the data. Against this background, the strengths and weaknesses of research design are reviewed, contrasting the Netflix design with that of studies of biomarkers to detect cancer. Such biomarker studies generally have less-strong design, lower numbers of outcomes, and greater difficulty in even just measuring predictors and outcomes, so the fundamental data that will be used in mathematical analysis tend to be much weaker than in other kinds of research. CONCLUSIONS If the fundamental data that will be analyzed are not strong, then better analytic methods have limited use in improving the situation. Recognition of this situation is an important first step toward improving the quality of clinical research about markers to detect cancer.

[1]  W. Kannel,et al.  An evaluation of follow-up methods in the Framingham Heart Study. , 1967, American Journal of Public Health and the Nations Health.

[2]  The case-control study: consensus and controversy. , 1979, Journal of chronic diseases.

[3]  J. Gohagan,et al.  The Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial of the National Cancer Institute: history, organization, and status. , 2000, Controlled clinical trials.

[4]  P C Prorok,et al.  Design of the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial. , 2000, Controlled clinical trials.

[5]  David F Ransohoff,et al.  Challenges and opportunities in evaluating diagnostic tests. , 2002, Journal of clinical epidemiology.

[6]  Sudhir Srivastava,et al.  Markers for early detection of cancer: Statistical guidelines for nested case-control studies , 2002, BMC medical research methodology.

[7]  D. Ransohoff.  Developing Molecular Biomarkers for Cancer , 2003, Science.

[8]  D. Rennie,et al.  The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. , 2003, Annals of internal medicine.

[9]  M. Radmacher,et al.  Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. , 2003, Journal of the National Cancer Institute.

[10]  D. Ransohoff.  Rules of evidence for cancer molecular-marker discovery and validation , 2004, Nature Reviews Cancer.

[11]  David F Ransohoff,et al.  Evaluating discovery-based research: when biologic reasoning cannot work. , 2004, Gastroenterology.

[12]  W. Sauerbrei,et al.  REporting recommendations for tumor MARKer prognostic studies (REMARK) , 2005, Nature Clinical Practice Oncology.

[13]  D. Ransohoff.  Bias as a threat to the validity of cancer molecular-marker research , 2005, Nature reviews. Cancer.

[14]  B. H. Patterson,et al.  Evaluating markers for the early detection of cancer: overview of study designs and methods , 2006, Clinical trials.

[15]  D. Levy,et al.  Multiple biomarkers for the prediction of first major cardiovascular events and death. , 2006, The New England journal of medicine.

[16]  D. Morrow,et al.  Benchmarks for the Assessment of Novel Cardiovascular Biomarkers , 2007, Circulation.

[17]  A. Dupuy,et al.  Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting. , 2007, Journal of the National Cancer Institute.

[18]  David F Ransohoff,et al.  How to improve reliability and efficiency of research about molecular markers: roles of phases, guidelines, and study design. , 2007, Journal of clinical epidemiology.

[19]  David F Ransohoff,et al.  The process to discover and develop biomarkers for cancer: a work in progress. , 2008, Journal of the National Cancer Institute.

[20]  Michelle A. Anderson,et al.  A Mouse to Human Search for Plasma Proteome Changes Associated with Pancreatic Tumor Development , 2008, PLoS medicine.

[21]  Holly Janes,et al.  Pivotal Evaluation of the Accuracy of a Biomarker Used for Classification or Prediction: Standards for Study Design , 2008, Journal of the National Cancer Institute.

[22]  Christoph H Borchers,et al.  Multi-site assessment of the precision and reproducibility of multiple reaction monitoring–based measurements of proteins in plasma , 2009, Nature Biotechnology.

[23]  D. Ransohoff,et al.  Sources of bias in specimens for research about molecular markers for cancer. , 2010, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[24]  Anne-Marie Welling,et al.  Work in progress. , 2012, Nursing standard (Royal College of Nursing (Great Britain) : 1987).