Large AI Models in Health Informatics: Applications, Challenges, and the Future

Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which often reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A concrete example is the recent debut of ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our life. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multimodality data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents an up-to-date comprehensive review of large AI models, from background to their applications. We identify seven key sectors that large AI models are applicable and might have substantial influence, including 1) molecular biology and drug discovery; 2) medical diagnosis and decision-making; 3) medical imaging and vision; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges in health informatics, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.

[1]  Benny P. L. Lo,et al.  A Step Towards Conditional Autonomy - Robotic Appendectomy , 2023, IEEE Robotics and Automation Letters.

[2]  Henrique Pondé de Oliveira Pinto,et al.  GPT-4 Technical Report , 2023, 2303.08774.

[3]  Wenpin Hou,et al.  GeneTuring tests GPT models in genomics , 2023, bioRxiv.

[4]  X. Li,et al.  Empowering Beginners in Bioinformatics with ChatGPT , 2023, bioRxiv.

[5]  Chenfei Wu,et al.  Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models , 2023, ArXiv.

[6]  Mehdi S. M. Sajjadi,et al.  PaLM-E: An Embodied Multimodal Language Model , 2023, ICML.

[7]  Tara N. Sainath,et al.  Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages , 2023, ArXiv.

[8]  Li Dong,et al.  Language Is Not All You Need: Aligning Perception with Language Models , 2023, NeurIPS.

[9]  Naman Goyal,et al.  LLaMA: Open and Efficient Foundation Language Models , 2023, ArXiv.

[10]  Li Fei-Fei,et al.  MimicPlay: Long-Horizon Imitation Learning by Watching Human Play , 2023, ArXiv.

[11]  Jindong Wang,et al.  On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective , 2023, ArXiv.

[12]  André Susano Pinto,et al.  Tuning computer vision models with task rewards , 2023, ICML.

[13]  Xi Ouyang,et al.  ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models , 2023, ArXiv.

[14]  Sjoerd van Steenkiste,et al.  Scaling Vision Transformers to 22 Billion Parameters , 2023, ICML.

[15]  Kyle Lam,et al.  ChatGPT: the future of discharge summaries? , 2023, The Lancet. Digital health.

[16]  S. Savarese,et al.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models , 2023, ArXiv.

[17]  Terry Yue Zhuo,et al.  Exploring AI Ethics of ChatGPT: A Diagnostic Analysis , 2023, ArXiv.

[18]  F. Ciravegna,et al.  Medical artificial intelligence is as much social as it is technological , 2023, Nature Machine Intelligence.

[19]  Timo I. Denk,et al.  MusicLM: Generating Music From Text , 2023, ArXiv.

[20]  Christopher D. Manning,et al.  DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature , 2023, ArXiv.

[21]  Jayesh K. Gupta,et al.  ClimaX: A foundation model for weather and climate , 2023, ArXiv.

[22]  J. El-Khoury,et al.  Evaluating the Performance of ChatGPT in Ophthalmology , 2023, medRxiv.

[23]  Michael W. Spratling,et al.  Data Augmentation Alone Can Improve Adversarial Training , 2023, ICLR.

[24]  Tiffany H. Kung,et al.  Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models , 2022, medRxiv.

[25]  Radu Tudor Ionescu,et al.  Diffusion Models in Vision: A Survey , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Peizhen Bai,et al.  Interpretable bilinear attention network with domain adaptation improves drug–target prediction , 2022, Nature Machine Intelligence.

[27]  Bingbing Ni,et al.  MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification , 2021, Scientific Data.

[28]  Benny P. L. Lo,et al.  Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring , 2021, IEEE Transactions on Cybernetics.

[29]  Jiming Liu,et al.  Reinforcement Learning in Healthcare: A Survey , 2019, ACM Comput. Surv..

[30]  Tianming Liu,et al.  ChatAug: Leveraging ChatGPT for Text Data Augmentation , 2023, ArXiv.

[31]  M. Ingrisch,et al.  ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports , 2022, European radiology.

[32]  Hyung Won Chung,et al.  Large language models encode clinical knowledge , 2022, Nature.

[33]  S. Levine,et al.  RT-1: Robotics Transformer for Real-World Control at Scale , 2022, Robotics: Science and Systems.

[34]  Amelia Villegas-Morcillo,et al.  ManyFold: an efficient and flexible library for training and validating protein folding models , 2022, Bioinformatics.

[35]  Abdelrahman M. Shaker,et al.  UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation , 2022, IEEE transactions on medical imaging.

[36]  Jong Wook Kim,et al.  Robust Speech Recognition via Large-Scale Weak Supervision , 2022, ICML.

[37]  Colin B. Compas,et al.  A large language model for electronic health records , 2022, npj Digital Medicine.

[38]  Brian D. Weitzner,et al.  OpenFold: Retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization , 2022, bioRxiv.

[39]  Jie Tang,et al.  Improved the Protein Complex Prediction with Protein Language Models , 2022, bioRxiv.

[40]  Jifeng Dai,et al.  Towards All-in-One Pre-Training via Maximizing Multi-Modal Mutual Information , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  François Ferland,et al.  A Review on the Use of Mobile Service Robots in Elderly Care , 2022, Robotics.

[42]  Hongsheng Li,et al.  InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Alexander M. Rush,et al.  BLOOM: A 176B-Parameter Open-Access Multilingual Language Model , 2022, ArXiv.

[44]  Andrew M. Dai,et al.  Scaling Instruction-Finetuned Language Models , 2022, ArXiv.

[45]  F. Rodriguez y Baena,et al.  Modular robotic platform for precision neurosurgery with a bio-inspired needle: System overview and first in-vivo deployment , 2022, PLoS ONE.

[46]  Jimeng Sun,et al.  MedCLIP: Contrastive Learning from Unpaired Medical Images and Text , 2022, EMNLP.

[47]  Ludwig Schmidt,et al.  LAION-5B: An open large-scale dataset for training next generation image-text models , 2022, NeurIPS.

[48]  P. Chambon,et al.  Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains , 2022, ArXiv.

[49]  Li Fei-Fei,et al.  VIMA: General Robot Manipulation with Multimodal Prompts , 2022, ArXiv.

[50]  George M. Church,et al.  Single-sequence protein structure prediction using a language model and deep learning , 2022, Nature Biotechnology.

[51]  Yaniv Taigman,et al.  AudioGen: Textually Guided Audio Generation , 2022, ICLR.

[52]  Lisa Anne Hendricks,et al.  Improving alignment of dialogue agents via targeted human judgements , 2022, ArXiv.

[53]  P. Rajpurkar,et al.  Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors , 2022, ML4H@NeurIPS.

[54]  Shenmin Zhang,et al.  BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining , 2022, Briefings Bioinform..

[55]  P. Rajpurkar,et al.  Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning , 2022, Nature Biomedical Engineering.

[56]  Tsung-Hui Chang,et al.  Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training , 2022, MICCAI.

[57]  Ashish V. Thapliyal,et al.  PaLI: A Jointly-Scaled Multilingual Language-Image Model , 2022, arXiv.org.

[58]  O. S.,et al.  Accurate prediction of protein structures and interactions using a three-track neural network , 2022, Yearbook of Paediatric Endocrinology.

[59]  D. Fox,et al.  Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation , 2022, CoRL.

[60]  David Grangier,et al.  AudioLM: A Language Modeling Approach to Audio Generation , 2022, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[61]  Guolin Ke,et al.  Uni-Fold: An Open-Source Platform for Developing Protein Folding Models beyond AlphaFold , 2022, bioRxiv.

[62]  Benny P. L. Lo,et al.  Clustering Egocentric Images in Passive Dietary Monitoring with Self-Supervised Learning , 2022, 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI).

[63]  Alistair E. W. Johnson,et al.  BRAX, Brazilian labeled chest x-ray dataset , 2022, Scientific data.

[64]  Jian Peng,et al.  High-resolution de novo structure prediction from primary sequence , 2022, bioRxiv.

[65]  Weiqi Wang,et al.  Diffsound: Discrete Diffusion Model for Text-to-sound Generation , 2022, ArXiv.

[66]  O. Winther,et al.  Can large language models reason about medical questions? , 2022, Patterns.

[67]  Dianhai Yu,et al.  HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddle , 2022, ArXiv.

[68]  Tao Shen,et al.  E2Efold-3D: End-to-End Deep Learning Method for accurate de novo RNA 3D Structure Prediction , 2022, ArXiv.

[69]  Jing Yu Koh,et al.  Scaling Autoregressive Models for Content-Rich Text-to-Image Generation , 2022, Trans. Mach. Learn. Res..

[70]  J. Dean,et al.  Emergent Abilities of Large Language Models , 2022, Trans. Mach. Learn. Res..

[71]  D. Sontag,et al.  Large language models are few-shot clinical information extractors , 2022, EMNLP.

[72]  David J. Fleet,et al.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding , 2022, NeurIPS.

[73]  Sergio Gomez Colmenarejo,et al.  A Generalist Agent , 2022, Trans. Mach. Learn. Res..

[74]  Xi Victoria Lin,et al.  OPT: Open Pre-trained Transformer Language Models , 2022, ArXiv.

[75]  Prafulla Dhariwal,et al.  Hierarchical Text-Conditional Image Generation with CLIP Latents , 2022, ArXiv.

[76]  Andrew M. Dai,et al.  PaLM: Scaling Language Modeling with Pathways , 2022, J. Mach. Learn. Res..

[77]  S. Levine,et al.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances , 2022, CoRL.

[78]  Zhaoping Xiong,et al.  PanGu Drug Model: learn a molecule like a human , 2022, bioRxiv.

[79]  D. Sancarlo,et al.  Emotion Recognizing by a Robotic Solution Initiative (EMOTIVE Project) , 2022, Sensors.

[80]  Qingxiong Tan,et al.  Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions , 2022, bioRxiv.

[81]  J. Leskovec,et al.  LinkBERT: Pretraining Language Models with Document Links , 2022, ACL.

[82]  Lisa Anne Hendricks,et al.  Training Compute-Optimal Large Language Models , 2022, ArXiv.

[83]  Vikash Kumar,et al.  R3M: A Universal Visual Representation for Robot Manipulation , 2022, CoRL.

[84]  Michael J. Black,et al.  LocATe: End-to-end Localization of Actions in 3D with Transformers , 2022, ArXiv.

[85]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[86]  Yang You,et al.  FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours , 2022, ArXiv.

[87]  S. Hoi,et al.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation , 2022, ICML.

[88]  Dale Schuurmans,et al.  Chain of Thought Prompting Elicits Reasoning in Large Language Models , 2022, NeurIPS.

[89]  Reza Yazdani Aminabadi,et al.  Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model , 2022, ArXiv.

[90]  A. Krieger,et al.  Autonomous robotic laparoscopic surgery for intestinal anastomosis , 2022, Science Robotics.

[91]  Junzhou Huang,et al.  DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery - A Focus on Affinity Prediction Problems with Noise Annotations , 2022, ArXiv.

[92]  Aaron B. Adcock,et al.  Revisiting Weakly Supervised Pre-Training of Visual Perception Models , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[93]  Renelito Delos Santos,et al.  LaMDA: Language Models for Dialog Applications , 2022, ArXiv.

[94]  Trevor Darrell,et al.  A ConvNet for the 2020s , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[95]  G. Buel,et al.  Can AlphaFold2 predict the impact of missense mutations on structure? , 2022, Nature Structural & Molecular Biology.

[96]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Prafulla Dhariwal,et al.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models , 2021, ICML.

[98]  Marcus Rohrbach,et al.  FLAVA: A Foundational Language And Vision Alignment Model , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[99]  Diego de Las Casas,et al.  Improving language models by retrieving from trillions of tokens , 2021, ICML.

[100]  Liunian Harold Li,et al.  Grounded Language-Image Pre-training , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[101]  B. Landman,et al.  Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[102]  Li Dong,et al.  Swin Transformer V2: Scaling Up Capacity and Resolution , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[103]  Juan Pino,et al.  XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale , 2021, INTERSPEECH.

[104]  Ross B. Girshick,et al.  Masked Autoencoders Are Scalable Vision Learners , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[105]  Zhenguo Li,et al.  FILIP: Fine-grained Interactive Language-Image Pre-Training , 2021, ICLR.

[106]  Benny P. L. Lo,et al.  Egocentric Human Trajectory Forecasting With a Wearable Camera and Multi-Modal Fusion , 2021, IEEE Robotics and Automation Letters.

[107]  Alexander M. Rush,et al.  Multitask Prompted Training Enables Zero-Shot Task Generalization , 2021, ICLR.

[108]  James M. Rehg,et al.  Ego4D: Around the World in 3,000 Hours of Egocentric Video , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[109]  Bolei Zhou,et al.  PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks , 2021, IEEE Robotics and Automation Letters.

[110]  Adams Wei Yu,et al.  SimVLM: Simple Visual Language Model Pretraining with Weak Supervision , 2021, ICLR.

[111]  S. Ovchinnikov,et al.  ColabFold: making protein folding accessible to all , 2022, Nature Methods.

[112]  Bjoern H Menze,et al.  The Medical Segmentation Decathlon , 2021, Nature Communications.

[113]  Alexander Kolesnikov,et al.  Scaling Vision Transformers , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[114]  Qi Tian,et al.  Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation , 2021, ECCV Workshops.

[115]  U. Deva Priyakumar,et al.  LigGPT: Molecular Generation using a Transformer-Decoder Model , 2021 .

[116]  Daguang Xu,et al.  UNETR: Transformers for 3D Medical Image Segmentation , 2021, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[117]  Christopher D. Manning,et al.  Contrastive Learning of Medical Visual Representations from Paired Images and Text , 2020, MLHC.

[118]  Jianfeng Gao,et al.  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing , 2020, ACM Trans. Comput. Heal..

[119]  Jeff Wu,et al.  WebGPT: Browser-assisted question-answering with human feedback , 2021, ArXiv.

[120]  Jeffrey J. Gray,et al.  Deciphering antibody affinity maturation with language models and weakly supervised learning , 2021, ArXiv.

[121]  Po-Sen Huang,et al.  Scaling Language Models: Methods, Analysis & Insights from Training Gopher , 2021, ArXiv.

[122]  Junzhou Huang,et al.  scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data , 2021, bioRxiv.

[123]  Liang Huang,et al.  Transformer-Based Generative Model Accelerating the Development of Novel BRAF Inhibitors , 2021, ACS omega.

[124]  Lu Yuan,et al.  Florence: A New Foundation Model for Computer Vision , 2021, ArXiv.

[125]  Karan Desai,et al.  RedCaps: web-curated image-text data created by the people, for the people , 2021, NeurIPS Datasets and Benchmarks.

[126]  Blake Hannaford,et al.  A decade retrospective of medical robotics research from 2010 to 2020 , 2021, Science Robotics.

[127]  D. Hassabis,et al.  Protein complex prediction with AlphaFold-Multimer , 2021, bioRxiv.

[128]  S. Yeung,et al.  GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[129]  Dieter Fox,et al.  CLIPort: What and Where Pathways for Robotic Manipulation , 2021, CoRL.

[130]  Michael S. Bernstein,et al.  On the Opportunities and Risks of Foundation Models , 2021, ArXiv.

[131]  Jiaxin Zheng,et al.  Extracting Predictive Representations from Hundreds of Millions of Molecules. , 2021, The journal of physical chemistry letters.

[132]  Chung-Cheng Chiu,et al.  w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training , 2021, 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).

[133]  Xin Shen,et al.  Efficient Medical Image Segmentation Based on Knowledge Distillation , 2021, IEEE Transactions on Medical Imaging.

[134]  Oriol Vinyals,et al.  Highly accurate protein structure prediction with AlphaFold , 2021, Nature.

[135]  Ruslan Salakhutdinov,et al.  HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units , 2021, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[136]  F. Pan,et al.  Algebraic graph-assisted bidirectional transformers for molecular property prediction , 2021, Nature Communications.

[137]  Diane M. Korngiebel,et al.  Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare delivery , 2021, npj Digital Medicine.

[138]  Pieter Abbeel,et al.  Decision Transformer: Reinforcement Learning via Sequence Modeling , 2021, NeurIPS.

[139]  Dong-Sheng Cao,et al.  MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction , 2021, Briefings Bioinform..

[140]  Sen Song,et al.  An effective self-supervised framework for learning expressive molecular global representations to drug discovery , 2021, Briefings Bioinform..

[141]  Aiping Lu,et al.  ADMETlab 2.0: an integrated online platform for accurate and comprehensive predictions of ADMET properties , 2021, Nucleic Acids Res..

[142]  Lihi Zelnik-Manor,et al.  ImageNet-21K Pretraining for the Masses , 2021, NeurIPS Datasets and Benchmarks.

[143]  Quoc V. Le,et al.  EfficientNetV2: Smaller Models and Faster Training , 2021, ICML.

[144]  D. Rueckert,et al.  Federated deep learning for detecting COVID-19 lung abnormalities in CT: a privacy-preserving multinational validation study , 2021, npj Digital Medicine.

[145]  Chunhua Shen,et al.  CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation , 2021, MICCAI.

[146]  Armand Joulin,et al.  Self-supervised Pretraining of Visual Features in the Wild , 2021, ArXiv.

[147]  Jiecao Chen,et al.  WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning , 2021, SIGIR.

[148]  Enhua Wu,et al.  Transformer in Transformer , 2021, NeurIPS.

[149]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[150]  Alec Radford,et al.  Zero-Shot Text-to-Image Generation , 2021, ICML.

[151]  Vishal M. Patel,et al.  Medical Transformer: Gated Axial-Attention for Medical Image Segmentation , 2021, MICCAI.

[152]  Radu Soricut,et al.  Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[153]  Yundong Zhang,et al.  TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation , 2021, MICCAI.

[154]  John F. Canny,et al.  MSA Transformer , 2021, bioRxiv.

[155]  Quoc V. Le,et al.  Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision , 2021, ICML.

[156]  Yan Wang,et al.  TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation , 2021, ArXiv.

[157]  Shekoofeh Azizi,et al.  Big Self-Supervised Models Advance Medical Image Classification , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[158]  Wen Gao,et al.  Pre-Trained Image Processing Transformer , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[159]  Silvio C. E. Tosatto,et al.  Pfam: The protein families database in 2021 , 2020, Nucleic Acids Res..

[160]  Robert D. Finn,et al.  RNAcentral 2021: secondary structure integration, improved sequence search and new member databases , 2020, Nucleic Acids Res..

[161]  Mingyue Zheng,et al.  DrugSpaceX: a large screenable and synthetically tractable database extending drug space , 2020, Nucleic Acids Res..

[162]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[163]  A. Ng,et al.  MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models , 2020, 2010.05352.

[164]  Zhenming Liu,et al.  CMNPD: a comprehensive marine natural products database towards facilitating drug discovery from the ocean , 2020, Nucleic Acids Res..

[165]  Ziqian Xie,et al.  Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction , 2020, npj Digital Medicine.

[166]  Jimeng Sun,et al.  MolTrans: Molecular Interaction Transformer for drug–target interaction prediction , 2020, Bioinform..

[167]  Myle Ott,et al.  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences , 2019, Proceedings of the National Academy of Sciences.

[168]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[169]  Malaikannan Sankarasubbu,et al.  BioELECTRA:Pretrained Biomedical text Encoder using Discriminators , 2021, BIONLP.

[170]  Guotong Xie,et al.  Learn molecular representations from large-scale unlabeled molecules for drug discovery , 2020, ArXiv.

[171]  Qiang Chen,et al.  IPN-V2 and OCTA-500: Methodology and Dataset for Retinal Image Segmentation , 2020, ArXiv.

[172]  Marwin H. S. Segler,et al.  Molecular representation learning with language models and domain-relevant auxiliary tasks , 2020, ArXiv.

[173]  Yurii S Moroz,et al.  ZINC20 - A Free Ultralarge-Scale Chemical Database for Ligand Discovery , 2020, J. Chem. Inf. Model..

[174]  Yang Zhang,et al.  Bio-Megatron: Larger Biomedical Domain Language Model , 2020, EMNLP.

[175]  Shuang Yu,et al.  Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representations , 2020, MICCAI.

[176]  B. Rost,et al.  ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing , 2020, bioRxiv.

[177]  Mark Chen,et al.  Generative Pretraining From Pixels , 2020, ICML.

[178]  Yatao Bian,et al.  Self-Supervised Graph Transformer on Large-Scale Molecular Data , 2020, NeurIPS.

[179]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[180]  T. Burki A new paradigm for drug development , 2020, The Lancet Digital Health.

[181]  Doug Downey,et al.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks , 2020, ACL.

[182]  Paolo Fiorini,et al.  Autonomous task planning and situation awareness in robotic surgery* , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[183]  Kaiming He,et al.  Designing Network Design Spaces , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[184]  A. Wong,et al.  COVID-Net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images , 2020, Scientific Reports.

[185]  Kristina Lerman,et al.  Tracking Social Media Discourse About the COVID-19 Pandemic: Development of a Public Coronavirus Twitter Data Set , 2020, JMIR public health and surveillance.

[186]  Nikhil Naik,et al.  ProGen: Language Modeling for Protein Generation , 2020, bioRxiv.

[187]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[188]  S. Gelly,et al.  Big Transfer (BiT): General Visual Representation Learning , 2019, ECCV.

[189]  L. Grummer-Strawn,et al.  Dynamics of the double burden of malnutrition and the changing nutrition reality , 2019, The Lancet.

[190]  Daria Grechishnikova,et al.  Transformer neural network for protein-specific de novo drug generation as a machine translation problem , 2019, Scientific Reports.

[191]  Ross B. Girshick,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[192]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[193]  Kazem Rahimi,et al.  BEHRT: Transformer for Electronic Health Records , 2019, Scientific Reports.

[194]  Jaewoo Kang,et al.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining , 2019, Bioinform..

[195]  Antonio Pertusa,et al.  PadChest: A large chest x-ray image dataset with multi-label annotated reports , 2019, Medical Image Anal..

[196]  Shion Honda,et al.  SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery , 2019, ArXiv.

[197]  Junzhou Huang,et al.  SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction , 2019, BCB.

[198]  Nima Tajbakhsh,et al.  Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis , 2019, MICCAI.

[199]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[200]  Matt J. Kusner,et al.  A Model to Search for Synthesizable Molecules , 2019, NeurIPS.

[201]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[202]  Ronan Collobert,et al.  wav2vec: Unsupervised Pre-training for Speech Recognition , 2019, INTERSPEECH.

[203]  Wei-Hung Weng,et al.  Publicly Available Clinical BERT Embeddings , 2019, Proceedings of the 2nd Clinical Natural Language Processing Workshop.

[204]  Oscar Franzén,et al.  PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data , 2019, Database J. Biol. Databases Curation.

[205]  Kai Ma,et al.  Med3D: Transfer Learning for 3D Medical Image Analysis , 2019, ArXiv.

[206]  Christopher D. Manning,et al.  GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[207]  Jacob Rosen,et al.  Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[208]  Yifan Yu,et al.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison , 2019, AAAI.

[209]  Quoc V. Le,et al.  GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism , 2018, ArXiv.

[210]  Hojung Nam,et al.  DeepConv-DTI: Prediction of drug-target interactions via deep learning with convolution on protein sequences , 2018, PLoS Comput. Biol..

[211]  Evan Bolton,et al.  PubChem 2019 update: improved access to chemical data , 2018, Nucleic Acids Res..

[212]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[213]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[214]  C. Deane,et al.  Observed Antibody Space: A Resource for Data Mining Next-Generation Sequencing of Antibody Repertoires , 2018, The Journal of Immunology.

[215]  Le Lu,et al.  DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning , 2018, Journal of medical imaging.

[216]  Radu Soricut,et al.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning , 2018, ACL.

[217]  Kaiming He,et al.  Exploring the Limits of Weakly Supervised Pretraining , 2018, ECCV.

[218]  Harald Kittler,et al.  Descriptor : The HAM 10000 dataset , a large collection of multi-source dermatoscopic images of common pigmented skin lesions , 2018 .

[219]  F. von Delft,et al.  Where is crystallography going? , 2018, Acta crystallographica. Section D, Structural biology.

[220]  J. Söding,et al.  Clustering huge protein sequence sets in linear time , 2018, bioRxiv.

[221]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[222]  Michael C. Yip,et al.  Robot Autonomy for Surgery , 2017, The Encyclopedia of Medical Robotics.

[223]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[224]  A. Ng,et al.  MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs. , 2017 .

[225]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[226]  Chen Sun,et al.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[227]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[228]  Shane Legg,et al.  Deep Reinforcement Learning from Human Preferences , 2017, NIPS.

[229]  Peter Kazanzides,et al.  Medical robotics—Regulatory, ethical, and legal considerations for increasing levels of autonomy , 2017, Science Robotics.

[230]  Yash Goyal,et al.  Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering , 2016, International Journal of Computer Vision.

[231]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[232]  Chris Maloney,et al.  PubMed Central , 2017 .

[233]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[234]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[235]  Michael S. Bernstein,et al.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations , 2016, International Journal of Computer Vision.

[236]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[237]  Michael S. Bernstein,et al.  Visual7W: Grounded Question Answering in Images , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[238]  David A. Shamma,et al.  YFCC100M , 2015, Commun. ACM.

[239]  K. Bhaskaran,et al.  Data Resource Profile: Clinical Practice Research Datalink (CPRD) , 2015, International journal of epidemiology.

[240]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[241]  Xinlei Chen,et al.  Microsoft COCO Captions: Data Collection and Evaluation Server , 2015, ArXiv.

[242]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[243]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[244]  Peter B. McGarvey,et al.  UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches , 2014, Bioinform..

[245]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[246]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[247]  S. Scheres,et al.  How cryo-EM is revolutionizing structural biology. , 2015, Trends in biochemical sciences.

[248]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[249]  Peter Young,et al.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions , 2014, TACL.

[250]  Stephen M. Moore,et al.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository , 2013, Journal of Digital Imaging.

[251]  Robert Petryszak,et al.  UniChem: a unified chemical structure cross-referencing and identifier tracking system , 2013, Journal of Cheminformatics.

[252]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[253]  John P. Overington,et al.  ChEMBL: a large-scale bioactivity database for drug discovery , 2011, Nucleic Acids Res..

[254]  Vicente Ordonez,et al.  Im2Text: Describing Images Using 1 Million Captioned Photographs , 2011, NIPS.

[255]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[256]  Cathy H. Wu,et al.  The Universal Protein Resource (UniProt) , 2004, Nucleic Acids Res..

[257]  Cathy H. Wu,et al.  UniProt: the Universal Protein knowledgebase , 2004, Nucleic Acids Res..

[258]  K. Wüthrich The way to NMR structures of proteins , 2001, Nature Structural Biology.