SIMON, an Automated Machine Learning System, Reveals Immune Signatures of Influenza Vaccine Responses

Machine learning holds considerable promise for understanding complex biological processes such as vaccine responses. Capturing interindividual variability is essential to increase the statistical power necessary for building more accurate predictive models. However, available approaches have difficulty coping with incomplete datasets which is often the case when combining studies. Additionally, there are hundreds of algorithms available and no simple way to find the optimal one. Here, we developed Sequential Iterative Modelling “OverNight” or SIMON, an automated machine learning system that compares results from 128 different algorithms and is particularly suitable for datasets containing many missing values. We applied SIMON to data from five clinical studies of seasonal influenza vaccination. The results reveal previously unrecognized CD4+ and CD8+ T cell subsets strongly associated with a robust antibody response to influenza antigens. These results demonstrate that SIMON can greatly speed up the choice of analysis modalities. Hence, it is a highly useful approach for data-driven hypothesis generation from disparate clinical datasets. Our strategy could be used to gain biological insight from ever-expanding heterogeneous datasets that are publicly available.

[1]  Barry Smith,et al.  ImmPort, toward repurposing of open access immunological assay data for translational and clinical research , 2018, Scientific Data.

[2]  L. Grohskopf,et al.  Background Document for “Prevention and Control of Seasonal Influenza with Vaccines: Recommendations of the Advisory Committee on Immunization Practices—United States, 2017-18 Influenza Season” Introduction , 2017 .

[3]  Patrick Dunn,et al.  The 10,000 Immunomes Project: A resource for human immunology , 2017, bioRxiv.

[4]  Alicia M. Fry,et al.  Influenza Vaccine Effectiveness in the United States during the 2015–2016 Season , 2017, The New England journal of medicine.

[5]  H. Maecker,et al.  Immune Checkpoint Function of CD85j in CD8 T Cell Differentiation and Aging , 2017, Front. Immunol..

[6]  Ankit N Khambhati,et al.  Crowdsourcing seizure detection: algorithm development and validation on human implanted device recordings , 2017, Brain : a journal of neurology.

[7]  Raphael Gottardo,et al.  Multicohort analysis reveals baseline transcriptional predictors of influenza vaccination responses , 2017, Science Immunology.

[8]  R. Cox,et al.  Long-term Maintenance of the Influenza-Specific Cross-Reactive Memory CD4+ T-Cell Responses Following Repeated Annual Influenza Vaccination , 2016, The Journal of infectious diseases.

[9]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[10]  C. Doglioni,et al.  IL‐17 superfamily cytokines modulate normal germinal center B cell migration , 2016, Journal of leukocyte biology.

[11]  Di Wu,et al.  CXCR5+ follicular cytotoxic T cells control viral infection in B cell follicles , 2016, Nature Immunology.

[12]  Ting Ni,et al.  Follicular CXCR5-expressing CD8+ T cells curtail chronic viral infection , 2016, Nature.

[13]  Matheus C. Bürger,et al.  Defining CD8+ T cells that provide the proliferative burst after PD-1 therapy , 2016, Nature.

[14]  Randal S. Olson,et al.  Automating Biomedical Data Science Through Tree-Based Pipeline Optimization , 2016, EvoApplications.

[15]  Reiner Schulz,et al.  Adjuvanted influenza-H1N1 vaccination reveals lymphoid signatures of age-dependent early responses and of clinical adverse events , 2016, Nature Immunology.

[16]  B. Zheng,et al.  IL-17A Promotes Pulmonary B-1a Cell Differentiation via Induction of Blimp-1 Expression during Influenza Virus Infection , 2016, PLoS pathogens.

[17]  Wei Liu,et al.  Learning to Hash for Indexing Big Data—A Survey , 2015, Proceedings of the IEEE.

[18]  Eva K. Lee,et al.  Systems Analysis of Immunity to Influenza Vaccination across Multiple Years and in Diverse Populations Reveals Shared Molecular Signatures. , 2015, Immunity.

[19]  Minghui Wang,et al.  Efficient Test and Visualization of Multi-Set Intersections , 2015, Scientific Reports.

[20]  B. Pulendran,et al.  Systems vaccinology: Enabling rational vaccine design with systems biological approaches. , 2015, Vaccine.

[21]  Ash A. Alizadeh,et al.  Large-Scale and Comprehensive Immune Profiling and Functional Analysis of Normal Human Aging , 2015, PloS one.

[22]  S. Stevanović,et al.  High-density preculture of PBMCs restores defective sensitivity of circulating CD8 T cells to virus- and tumor-derived antigens. , 2015, Blood.

[23]  Anne M Johnson,et al.  Natural T Cell-mediated Protection against Seasonal and Pandemic Influenza. Results of the Flu Watch Cohort Study. , 2015, American journal of respiratory and critical care medicine.

[24]  H. Maecker,et al.  Cytokine-stimulated Phosphoflow of PBMC Using CyTOF Mass Cytometry. , 2015, Bio-protocol.

[25]  Max Kuhn,et al.  caret: Classification and Regression Training , 2015 .

[26]  Mark M. Davis,et al.  Cytomegalovirus infection enhances the immune response to influenza , 2015, Science Translational Medicine.

[27]  H. Maecker,et al.  Phenotyping of Live Human PBMC using CyTOF™ Mass Cytometry. , 2015, Bio-protocol.

[28]  Atul J. Butte,et al.  Variation in the Human Immune System Is Largely Driven by Non-Heritable Influences , 2015, Cell.

[29]  A. Osterhaus,et al.  Human Influenza A Virus-Specific CD8+ T-Cell Response Is Long-lived. , 2015, The Journal of infectious diseases.

[30]  Mark M. Davis,et al.  CD161 Defines a Transcriptional and Functional Phenotype across Distinct Human T Cell Lineages , 2014, Cell reports.

[31]  Mark M. Davis,et al.  Apoptosis and other immune biomarkers predict influenza vaccine responsiveness , 2014, Molecular Systems Biology.

[32]  B. Pulendran Systems vaccinology: Probing humanity’s diverse immune systems with vaccines , 2014, Proceedings of the National Academy of Sciences.

[33]  Yuri Kotliarov,et al.  Global Analyses of Human Immune Variation Reveal Baseline Predictors of Postvaccination Responses , 2014, Cell.

[34]  David Gomez-Cabrero,et al.  Data integration in the era of omics: current and future challenges , 2014, BMC Systems Biology.

[35]  Mark M. Davis,et al.  Systems analysis of sex differences reveals an immunosuppressive role for testosterone in the response to influenza vaccination , 2013, Proceedings of the National Academy of Sciences.

[36]  S. McWeeney,et al.  A systems framework for vaccine design , 2013, Current Opinion in Immunology.

[37]  Sean C. Bendall,et al.  Normalization of mass cytometry data with bead standards , 2013, Cytometry. Part A : the journal of the International Society for Analytical Cytology.

[38]  Mark M. Davis,et al.  Apoptosis and other immune biomarkers predict influenza vaccine responsiveness , 2013, Molecular systems biology.

[39]  Christoforos Anagnostopoulos,et al.  When is the area under the receiver operating characteristic curve an appropriate measure of classifier performance? , 2013, Pattern Recognit. Lett..

[40]  Virginia Pascual,et al.  Induction of ICOS+CXCR3+CXCR5+ TH Cells Correlates with Antibody Responses to Influenza Vaccination , 2013, Science Translational Medicine.

[41]  F. McCoy,et al.  Janus-faced PIDD: a sensor for DNA damage-induced cell death or survival? , 2012, Molecular cell.

[42]  Yael Rosenberg-Hasson,et al.  The Stanford Data Miner: a novel approach for integrating and exploring heterogeneous immunological data , 2012, Journal of Translational Medicine.

[43]  J. Oxford,et al.  Preexisting influenza-specific CD4+ T cells correlate with disease protection against influenza challenge in humans , 2012, Nature Medicine.

[44]  Hermann Einsele,et al.  Preculture of PBMCs at high cell density increases sensitivity of T-cell responses, revealing cytokine release by CD28 superagonist TGN1412. , 2011, Blood.

[45]  B. Pulendran,et al.  Systems vaccinomics: the road ahead for vaccinology. , 2011, Omics : a journal of integrative biology.

[46]  Eva K. Lee,et al.  Systems Biology of Seasonal Influenza Vaccination in Humans , 2011, Nature Immunology.

[47]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[48]  Xavier Robin,et al.  pROC: an open-source package for R and S+ to analyze and compare ROC curves , 2011, BMC Bioinformatics.

[49]  Witold R. Rudnicki,et al.  Feature Selection with the Boruta Package , 2010 .

[50]  Tero Aittokallio,et al.  Dealing with missing values in large-scale studies: microarray data imputation and beyond , 2010, Briefings Bioinform..

[51]  David J. Hand,et al.  Measuring classifier performance: a coherent alternative to the area under the ROC curve , 2009, Machine Learning.

[52]  Danila Valmori,et al.  Human memory FOXP3+ Tregs secrete IL-17 ex vivo and constitutively express the TH17 lineage-specific transcription factor RORγt , 2009, Proceedings of the National Academy of Sciences.

[53]  T. Strutt,et al.  Tc17, a Unique Subset of CD8 T Cells That Can Protect against Lethal Influenza Challenge1 , 2009, The Journal of Immunology.

[54]  András Kocsor,et al.  ROC analysis: applications to the classification of biological sequences and 3D structures , 2008, Briefings Bioinform..

[55]  R. Jacobson,et al.  Heterogeneity in Vaccine Immune Response: The Role of Immunogenetics and the Emerging Field of Vaccinomics , 2007, Clinical pharmacology and therapeutics.

[56]  Veronica D. Gonzalez,et al.  CXCR5+ CCR7– CD8 T cells are early effector memory cells that infiltrate tonsil B cell follicles , 2007, European journal of immunology.

[57]  Morten Nielsen,et al.  Large-scale validation of methods for cytotoxic T-lymphocyte epitope prediction , 2007, BMC Bioinformatics.

[58]  T. Gingeras,et al.  CD127 expression inversely correlates with FoxP3 and suppressive function of human CD4+ T reg cells , 2006, The Journal of experimental medicine.

[59]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[60]  C Zimmer,et al.  Glioma assessment using quantitative blood volume maps generated by T1-weighted dynamic contrast-enhanced magnetic resonance imaging: a receiver operating characteristic study , 2006, Acta radiologica.

[61]  D. Richman,et al.  Memory CD8+ T cells vary in differentiation phenotype in different persistent virus infections , 2002, Nature Medicine.

[62]  Alex E. Lash,et al.  Gene Expression Omnibus: NCBI gene expression and hybridization array data repository , 2002, Nucleic Acids Res..

[63]  Malbea A Lapete,et al.  Prevention and Control of Influenza Recommendations of the Advisory Committee on Immunization Practices ( ACIP ) , 2004 .

[64]  R. Tibshirani,et al.  Significance analysis of microarrays applied to the ionizing radiation response , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[65]  F. Sallusto,et al.  Two subsets of memory T lymphocytes with distinct homing potentials and effector functions , 1999, Nature.

[66]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[67]  Stephen V. Stehman,et al.  Selecting and interpreting measures of thematic classification accuracy , 1997 .

[68]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[69]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[70]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[71]  David W. Aha,et al.  A Comparative Evaluation of Sequential Feature Selection Algorithms , 1995, AISTATS.

[72]  G. K. Hirst THE QUANTITATIVE DETERMINATION OF INFLUENZA VIRUS AND ANTIBODIES BY MEANS OF RED CELL AGGLUTINATION , 1942, The Journal of experimental medicine.