Embedding of semantic predications

This paper concerns the generation of distributed vector representations of biomedical concepts from structured knowledge, in the form of subject-relation-object triplets known as semantic predications. Specifically, we evaluate the extent to which a representational approach we have developed for this purpose previously, known as Predication-based Semantic Indexing (PSI), might benefit from insights gleaned from neural-probabilistic language models, which have enjoyed a surge in popularity in recent years as a means to generate distributed vector representations of terms from free text. To do so, we develop a novel neural-probabilistic approach to encoding predications, called Embedding of Semantic Predications (ESP), by adapting aspects of the Skipgram with Negative Sampling (SGNS) algorithm to this purpose. We compare ESP and PSI across a number of tasks including recovery of encoded information, estimation of semantic similarity and relatedness, and identification of potentially therapeutic and harmful relationships using both analogical retrieval and supervised learning. We find advantages for ESP in some, but not all of these tasks, revealing the contexts in which the additional computational work of neural-probabilistic modeling is justified.

[1]  Trevor Cohen,et al.  Real, Complex, and Binary Semantic Vectors , 2012, QI.

[2]  Geoffrey E. Hinton,et al.  Learning Distributed Representations of Concepts Using Linear Relational Embedding , 2001, IEEE Trans. Knowl. Data Eng..

[3]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[4]  Marcelo Fiszman,et al.  The interaction of domain knowledge and linguistic structure in natural language processing: interpreting hypernymic propositions in biomedical text , 2003, J. Biomed. Informatics.

[5]  Paul Smolensky,et al.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1990, Artif. Intell..

[6]  Marc Weeber,et al.  Using concepts in literature-based discovery: simulating Swanson's Raynaud-fish oil and migraine-magnesium discoveries , 2001 .

[7]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[8]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[9]  Pentti Kanerva,et al.  Binary Spatter-Coding of Ordered K-Tuples , 1996, ICANN.

[10]  M. Schuemie,et al.  Defining a Reference Set to Support Methodological Research in Drug Safety , 2013, Drug Safety.

[11]  Michael W. Berry,et al.  Mathematical Foundations Behind Latent Semantic Analysis , 2007 .

[12]  Gunnar Rätsch,et al.  Knowledge Transfer with Medical Language Embeddings , 2016, ArXiv.

[13]  Reed McEwan,et al.  Corpus domain effects on distributional semantic modeling of medical terms , 2016, Bioinform..

[14]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[15]  Gerard de Melo,et al.  Medical Concept Embeddings via Labeled Background Corpora , 2016, LREC.

[16]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[17]  Wei Xu,et al.  Can artificial neural networks learn language models? , 2000, INTERSPEECH.

[18]  Danqi Chen,et al.  Reasoning With Neural Tensor Networks for Knowledge Base Completion , 2013, NIPS.

[19]  Thomas C. Rindflesch,et al.  Predicting High-Throughput Screening Results With Scalable Literature-Based Discovery Methods , 2014, CPT: pharmacometrics & systems pharmacology.

[20]  Guido Zuccon,et al.  Medical Semantic Similarity with a Neural Language Model , 2014, CIKM.

[21]  Jason Weston,et al.  Learning Structured Embeddings of Knowledge Bases , 2011, AAAI.

[22]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[23]  Xin Rong,et al.  word2vec Parameter Learning Explained , 2014, ArXiv.

[24]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[25]  Trevor Cohen,et al.  Classification-by-Analogy: Using Vector Representations of Implicit Relationships to Identify Plausibly Causal Drug/Side-effect Relationships , 2016, AMIA.

[26]  Peter Bruza,et al.  Semantic Oscillations: Encoding Context and Structure in Complex Valued Holographic Vectors , 2010, AAAI Fall Symposium: Quantum Informatics for Cognitive, Social, and Semantic Processes.

[27]  Peter Davies,et al.  Discovering discovery patterns with predication-based Semantic Indexing , 2012, J. Biomed. Informatics.

[28]  Carol Friedman,et al.  Exploiting Semantic Relations for Literature-Based Discovery , 2006, AMIA.

[29]  Trevor Cohen,et al.  Reflective Random Indexing and indirect inference: A scalable method for discovery of implicit connections , 2010, J. Biomed. Informatics.

[30]  Tony A. Plate,et al.  Analogy retrieval and processing with distributed vector representations , 2000, Expert Syst. J. Knowl. Eng..

[31]  Michael N. Jones,et al.  OrBEAGLE: Integrating Orthography into a Holographic Model of the Lexicon , 2011, ICANN.

[32]  Magnus Sahlgren,et al.  The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces , 2006 .

[33]  Alessandro Lenci,et al.  Distributional semantics in linguistic and cognitive research , 2008 .

[34]  Simon D. Levy,et al.  Vector Symbolic Architectures: A New Building Material for Artificial General Intelligence , 2008, AGI.

[35]  Anders Holst,et al.  Random indexing of text samples for latent semantic analysis , 2000 .

[36]  Terrence Adam,et al.  Semantic Similarity and Relatedness between Clinical Terms: An Experimental Study. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[37]  Ross W. Gayler.  Vector Symbolic Architectures answer Jackendoff's challenges for cognitive neuroscience , 2004, ArXiv.

[38]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[39]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[40]  Trevor Cohen,et al.  Discovery at a distance: Farther journeys in predication space , 2012, 2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops.

[41]  J. Neumann,et al.  The Logic of Quantum Mechanics , 1936 .

[42]  Pentti Kanerva,et al.  What We Mean When We Say "What's the Dollar of Mexico?": Prototypes and Mapping in Concept Space , 2010, AAAI Fall Symposium: Quantum Informatics for Cognitive, Social, and Semantic Processes.

[43]  Lorenzo Rosasco,et al.  Holographic Embeddings of Knowledge Graphs , 2015, AAAI.

[44]  Paul Thagard,et al.  Integrating structure and meaning: a distributed model of analogical mapping , 2001 .

[45]  Dominic Widdows,et al.  Orthogonal Negation in Vector Spaces for Modelling Word-Meanings and Document Retrieval , 2003, ACL.

[46]  Trevor Cohen,et al.  Predication-based Semantic Indexing: Permutations as a Means to Encode Predications in Semantic Space , 2009, AMIA.

[47]  Halil Kilicoglu,et al.  SemMedDB: a PubMed-scale repository of biomedical semantic predications , 2012, Bioinform..

[48]  Trevor Cohen,et al.  Empirical distributional semantics: Methods and biomedical applications , 2009, J. Biomed. Informatics.

[49]  Hua Xu,et al.  Identifying Plausible Adverse Drug Reactions Using Knowledge Extracted from the Literature , 2014, AMIA.

[50]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[51]  Trevor Cohen,et al.  Reasoning with vectors: A continuous model for fast robust inference , 2015, Log. J. IGPL.

[52]  Tony A. Plate,et al.  Holographic Reduced Representation: Distributed Representation for Cognitive Structures , 2003 .

[53]  Trevor Cohen,et al.  Finding Schizophrenia's Prozac Emergent Relational Similarity in Predication Space , 2011, QI.

[54]  P. Bork,et al.  A side effect resource to capture phenotypic effects of drugs , 2010, Molecular systems biology.

[55]  Trevor Cohen,et al.  Orthogonality and Orthography: Introducing Measured Distance into Semantic Space , 2013, QI.

[56]  Joshua B. Tenenbaum,et al.  Modelling Relational Data using Bayesian Clustered Tensor Factorization , 2009, NIPS.

[57]  Trevor Cohen,et al.  Many Paths Lead to Discovery: Analogical Retrieval of Cancer Therapies , 2012, QI.

[58]  Dmitri A. Rachkovskij,et al.  Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning , 2001, Neural Computation.

[59]  Trevor Cohen,et al.  Embedding Probabilities in Predication Space with Hermitian Holographic Reduced Representations , 2015, QI.

[60]  Omer Levy,et al.  Improving Distributional Similarity with Lessons Learned from Word Embeddings , 2015, TACL.

[61]  Jimeng Sun,et al.  Medical Concept Representation Learning from Electronic Health Records and its Application on Heart Failure Prediction , 2016, ArXiv.

[62]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[63]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Praveen Paritosh,et al.  Freebase: a collaboratively created graph database for structuring human knowledge , 2008, SIGMOD Conference.

[65]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[66]  Stephen I. Gallant,et al.  Representing Objects, Relations, and Sequences , 2013, Neural Computation.

[67]  Todd R. Johnson,et al.  Retrofitting Word Vectors of MeSH Terms to Improve Semantic Similarity Measures , 2016, Louhi@EMNLP.

[68]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.

[69]  Omer Levy,et al.  word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method , 2014, ArXiv.

[70]  Dominic Widdows,et al.  Semantic Vectors: a Scalable Open Source Package and Online Technology Management Application , 2008, LREC.

[71]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .