Natural Language Reasoning, A Survey

This survey paper proposes a clearer view of natural language reasoning in the field of Natural Language Processing (NLP), both conceptually and practically. Conceptually, we provide a distinct definition for natural language reasoning in NLP, based on both philosophy and NLP scenarios, discuss what types of tasks require reasoning, and introduce a taxonomy of reasoning. Practically, we conduct a comprehensive literature review on natural language reasoning in NLP, mainly covering classical logical reasoning, natural language inference, multi-hop question answering, and commonsense reasoning. The paper also identifies and views backward reasoning, a powerful paradigm for multi-step reasoning, and introduces defeasible reasoning as one of the most important future directions in natural language reasoning research. We focus on single-modality unstructured natural language text, excluding neuro-symbolic techniques and mathematical reasoning.

[1]  Noah D. Goodman,et al.  Why think step-by-step? Reasoning emerges from the locality of experience , 2023, ArXiv.

[2]  Marco Tulio Ribeiro,et al.  Sparks of Artificial General Intelligence: Early experiments with GPT-4 , 2023, ArXiv.

[3]  E. Cambria,et al.  Logical Reasoning over Natural Language as Knowledge Representation: A Survey , 2023, ArXiv.

[4]  Dan Su,et al.  A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity , 2023, IJCNLP.

[5]  Li Dong,et al.  Language Models as Inductive Reasoners , 2022, ArXiv.

[6]  Seyed Mehran Kazemi,et al.  LAMBADA: Backward Chaining for Automated Reasoning in Natural Language , 2022, arXiv.org.

[7]  K. Chang,et al.  Towards Reasoning in Large Language Models: A Survey , 2022, ArXiv.

[8]  Se-Young Yun,et al.  Large Language Models Are Reasoning Teachers , 2022, ArXiv.

[9]  Xiang Ren,et al.  APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning , 2022, ArXiv.

[10]  Fei Huang,et al.  Reasoning with Language Model Prompting: A Survey , 2022, ArXiv.

[11]  O. Yu. Golovneva,et al.  ALERT: Adapt Language Models to Reasoning Tasks , 2022, Annual Meeting of the Association for Computational Linguistics.

[12]  Lucie Charlotte Magister,et al.  Teaching Small Language Models to Reason , 2022, ArXiv.

[13]  Mrinmaya Sachan,et al.  Distilling Multi-Step Reasoning Capabilities of Large Language Models into Smaller Models via Semantic Decompositions , 2022, arXiv.org.

[14]  D. Schuurmans,et al.  What learning algorithm is in-context learning? Investigations with linear models , 2022, ICLR.

[15]  Greg Durrett,et al.  Complementary Explanations for Effective In-Context Learning , 2022, Annual Meeting of the Association for Computational Linguistics.

[16]  Matt Gardner,et al.  CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation , 2022, EMNLP.

[17]  D. Roth,et al.  Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts , 2022, EMNLP.

[18]  Dong Yu,et al.  MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure , 2022, EMNLP.

[19]  Oyvind Tafjord,et al.  Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning , 2022, EMNLP.

[20]  William Yang Wang,et al.  WikiWhy: Answering and Explaining Cause-and-Effect Questions , 2022, ICLR.

[21]  S. Gu,et al.  Large Language Models Can Self-Improve , 2022, EMNLP.

[22]  Andrew M. Dai,et al.  Scaling Instruction-Finetuned Language Models , 2022, ArXiv.

[23]  T. Zhao,et al.  Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers , 2022, ACL.

[24]  Quoc V. Le,et al.  Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them , 2022, ACL.

[25]  Noah A. Smith,et al.  Measuring and Narrowing the Compositionality Gap in Language Models , 2022, ArXiv.

[26]  Alexander J. Smola,et al.  Automatic Chain of Thought Prompting in Large Language Models , 2022, ICLR.

[27]  He He,et al.  Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought , 2022, ICLR.

[28]  Shafiq R. Joty,et al.  FOLIO: Natural Language Reasoning with First-Order Logic , 2022, ArXiv.

[29]  M. Shanahan,et al.  Faithful Reasoning Using Large Language Models , 2022, ArXiv.

[30]  M. Witbrock,et al.  Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation , 2022, NeSy.

[31]  James L. McClelland,et al.  Language models show human-like content effects on reasoning , 2022, ArXiv.

[32]  J. Dean,et al.  Emergent Abilities of Large Language Models , 2022, Trans. Mach. Learn. Res..

[33]  Swaroop Mishra,et al.  Is a Question Decomposition Unit All We Need? , 2022, EMNLP.

[34]  Danqi Chen,et al.  Generating Natural Language Proofs with Verifier-Guided Search , 2022, EMNLP.

[35]  Xiang Ren,et al.  RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners , 2022, EMNLP.

[36]  S. Gu,et al.  Large Language Models are Zero-Shot Reasoners , 2022, NeurIPS.

[37]  Ronan Le Bras,et al.  Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations , 2022, EMNLP.

[38]  Guy Van den Broeck,et al.  On the Paradox of Learning to Reason from Data , 2022, IJCAI.

[39]  Yu Cao,et al.  Interpretable Proof Generation via Iterative Backward Reasoning , 2022, NAACL.

[40]  D. Schuurmans,et al.  Least-to-Most Prompting Enables Complex Reasoning in Large Language Models , 2022, ICLR.

[41]  I. Higgins,et al.  Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning , 2022, ICLR.

[42]  Andrew O. Arnold,et al.  Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner , 2022, NAACL-HLT.

[43]  Bing Qin,et al.  e-CARE: a New Dataset for Exploring Explainable Causal Reasoning , 2022, ACL.

[44]  Hongming Zhang,et al.  METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation , 2022, NAACL-HLT.

[45]  Santiago Ontañón,et al.  LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models , 2022, ArXiv.

[46]  Noah D. Goodman,et al.  STaR: Bootstrapping Reasoning With Reasoning , 2022, 2203.14465.

[47]  Matt Gardner,et al.  Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets , 2022, ACL.

[48]  Michael Witbrock,et al.  AbductionRules: Training Transformers to Explain Unexpected Inputs , 2022, FINDINGS.

[49]  D. Schuurmans,et al.  Self-Consistency Improves Chain of Thought Reasoning in Language Models , 2022, ICLR.

[50]  Xiang Ren,et al.  FaiRR: Faithful and Robust Deductive Reasoning over Natural Language , 2022, ACL.

[51]  Cornelia Caragea,et al.  SciNLI: A Corpus for Natural Language Inference on Scientific Text , 2022, ACL.

[52]  Liqiang Nie,et al.  MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning , 2022, FINDINGS.

[53]  Dale Schuurmans,et al.  Chain of Thought Prompting Elicits Reasoning in Large Language Models , 2022, NeurIPS.

[54]  Weizhu Chen,et al.  Reasoning Like Program Executors , 2022, EMNLP.

[55]  Sang Michael Xie,et al.  An Explanation of In-context Learning as Implicit Bayesian Inference , 2021, ICLR.

[56]  Ronan Le Bras,et al.  Generated Knowledge Prompting for Commonsense Reasoning , 2021, ACL.

[57]  Ashish Sabharwal,et al.  ♫ MuSiQue: Multihop Questions via Single-hop Question Composition , 2021, TACL.

[58]  Sujian Li,et al.  Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues , 2021, ACL.

[59]  Yejin Choi,et al.  Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines , 2021, ACL.

[60]  Li Dong,et al.  Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers , 2023, ACL.

[61]  Christopher D. Manning,et al.  GreaseLM: Graph REASoning Enhanced Language Models , 2022, ICLR.

[62]  Duyu Tang,et al.  Analytical Reasoning of Text , 2022, NAACL-HLT.

[63]  Xuanjing Huang,et al.  CQG: A Simple and Effective Controlled Generation Framework for Multi-hop Question Generation , 2022, ACL.

[64]  Eduard Hovy,et al.  Think about it! Improving defeasible reasoning by first modeling the question scenario. , 2021, EMNLP.

[65]  Niranjan Balasubramanian,et al.  Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension , 2021, EMNLP.

[66]  Nanyun Peng,et al.  Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning , 2021, EMNLP.

[67]  Alyssa Lees,et al.  ReasonBERT: Pre-trained to Reason with Distant Supervision , 2021, EMNLP.

[68]  Eunsol Choi,et al.  CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge , 2021, NeurIPS Datasets and Benchmarks.

[69]  Seung-won Hwang,et al.  Robustifying Multi-hop QA through Pseudo-Evidentiality Training , 2021, ACL.

[70]  Dragomir R. Radev,et al.  DocNLI: A Large-scale Dataset for Document-level Natural Language Inference , 2021, FINDINGS.

[71]  Niranjan Balasubramanian,et al.  TellMeWhy: A Dataset for Answering Why-Questions in Narratives , 2021, FINDINGS.

[72]  Yejin Choi,et al.  CommonsenseQA 2.0: Exposing the Limits of AI through Gamification , 2021, NeurIPS Datasets and Benchmarks.

[73]  Debanjan Ghosh,et al.  Figurative Language in Recognizing Textual Entailment , 2021, FINDINGS.

[74]  Prateek Yadav,et al.  multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning , 2021, NAACL.

[75]  Yongjie Huang,et al.  Breadth First Reasoning Graph for Multi-hop Question Answering , 2021, NAACL.

[76]  Mihai Surdeanu,et al.  Explainable Multi-hop Verbal Reasoning Through Internal Monologue , 2021, NAACL.

[77]  Eduard Hovy,et al.  Could you give me a hint ? Generating inference graphs for defeasible reasoning , 2021, FINDINGS.

[78]  Swarat Chaudhuri,et al.  Flexible Generation of Natural Language Deductions , 2021, EMNLP.

[79]  Oyvind Tafjord,et al.  Explaining Answers with Entailment Trees , 2021, EMNLP.

[80]  J. Leskovec,et al.  QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering , 2021, NAACL.

[81]  Jonathan Berant,et al.  Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies , 2021, Transactions of the Association for Computational Linguistics.

[82]  Alborz Geramifard,et al.  DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue , 2021, ACL.

[83]  Yejin Choi,et al.  Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences , 2020, EMNLP.

[84]  Peter Clark,et al.  ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language , 2020, FINDINGS.

[85]  Yue Zhang,et al.  Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts , 2020, AAAI.

[86]  William W. Cohen,et al.  Differentiable Open-Ended Commonsense Reasoning , 2020, NAACL.

[87]  Wenhu Chen,et al.  Unsupervised Multi-hop Question Answering by Question Generation , 2020, NAACL.

[88]  Christopher D. Manning,et al.  Answering Open-Domain Questions of Varying Reasoning Steps from Text , 2020, EMNLP.

[89]  Yejin Choi,et al.  COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs , 2020, AAAI.

[90]  Gregor Betz,et al.  Critical Thinking for Language Models , 2020, IWCS.

[91]  Dawn Song,et al.  Measuring Massive Multitask Language Understanding , 2020, ICLR.

[92]  Bill Yuchen Lin,et al.  RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms , 2020, EMNLP.

[93]  Daniel Khashabi,et al.  Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models , 2021, NAACL.

[94]  Dinesh Garg,et al.  Explanations for CommonsenseQA: New Dataset and Models , 2021, ACL.

[95]  Ting Liu,et al.  Learning Event Graph Knowledge for Abductive Reasoning , 2021, ACL.

[96]  Akiko Aizawa,et al.  Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , 2020, COLING.

[97]  Yejin Choi,et al.  Thinking Like a Skeptic: Defeasible Inference in Natural Language , 2020, FINDINGS.

[98]  Eduardo Blanco,et al.  An Analysis of Natural Language Inference Benchmarks through the Lens of Negation , 2020, EMNLP.

[99]  Yejin Choi,et al.  Social Chemistry 101: Learning to Reason about Social and Moral Norms , 2020, EMNLP.

[100]  Mohit Bansal,et al.  ConjNLI: Natural Language Inference over Conjunctive Sentences , 2020, EMNLP.

[101]  Yejin Choi,et al.  Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning , 2020, EMNLP.

[102]  Parisa Kordjamshidi,et al.  SRLGRN: Semantic Role Labeling Graph Reasoning Network , 2020, EMNLP.

[103]  Harsh Jhamtani,et al.  Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering , 2020, EMNLP.

[104]  Mohit Bansal,et al.  PRover: Proof Generation for Interpretable Reasoning over Rules , 2020, EMNLP.

[105]  Chris Callison-Burch,et al.  Reasoning about Goals, Steps, and Temporal Ordering with WikiHow , 2020, EMNLP.

[106]  Hanmeng Liu,et al.  LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning , 2020, IJCAI.

[107]  Wei Liu,et al.  Low-Resource Generation of Multi-hop Reasoning Questions , 2020, ACL.

[108]  Jonathan Berant,et al.  Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge , 2020, ArXiv.

[109]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[110]  Ashish Sabharwal,et al.  Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning , 2020, EMNLP.

[111]  Rajarshi Das,et al.  ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning , 2020, EMNLP.

[112]  Jun Yan,et al.  Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering , 2020, EMNLP.

[113]  Nicola Pellicano,et al.  Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning , 2020, ACL.

[114]  Kyunghyun Cho,et al.  Unsupervised Question Decomposition for Question Answering , 2020, EMNLP.

[115]  Oyvind Tafjord,et al.  Transformers as Soft Reasoners over Language , 2020, IJCAI.

[116]  Bill Yuchen Lin,et al.  CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning , 2020, FINDINGS.

[117]  Jiashi Feng,et al.  ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning , 2020, ICLR.

[118]  Paul N. Bennett,et al.  Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention , 2020, ICLR.

[119]  Yejin Choi,et al.  PIQA: Reasoning about Physical Commonsense in Natural Language , 2019, AAAI.

[120]  Noah A. Smith,et al.  Social Bias Frames: Reasoning about Social and Power Implications of Language , 2019, ACL.

[121]  Ashish Sabharwal,et al.  QASC: A Dataset for Question Answering via Sentence Composition , 2019, AAAI.

[122]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[123]  Kentaro Inui,et al.  R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason , 2019, ACL.

[124]  Benjamin Van Durme,et al.  Uncertain Natural Language Inference , 2019, ACL.

[125]  Wenhu Chen,et al.  TabFact: A Large-scale Dataset for Table-based Fact Verification , 2019, ICLR.

[126]  Doug Downey,et al.  Abductive Commonsense Reasoning , 2019, ICLR.

[127]  P. Dangerfield.  Logic , 1996, Aristotle and the Stoics.

[128]  Yejin Choi,et al.  Counterfactual Story Reasoning and Generation , 2019, EMNLP.

[129]  Yejin Choi,et al.  Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning , 2019, EMNLP.

[130]  Matt Gardner,et al.  Reasoning Over Paragraph Effects in Situations , 2019, EMNLP.

[131]  Joelle Pineau,et al.  CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text , 2019, EMNLP.

[132]  Ming-Wei Chang,et al.  Natural Questions: A Benchmark for Question Answering Research , 2019, TACL.

[133]  Johan Bos,et al.  Can Neural Networks Understand Monotonicity Reasoning? , 2019, BlackboxNLP@ACL.

[134]  Luke Zettlemoyer,et al.  E3: Entailment-driven Extracting and Editing for Conversational Machine Reading , 2019, ACL.

[135]  David Vilares,et al.  HEAD-QA: A Healthcare Dataset for Complex Reasoning , 2019, ACL.

[136]  Hannaneh Hajishirzi,et al.  Multi-hop Reading Comprehension through Question Decomposition and Rescoring , 2019, ACL.

[137]  Sameer Singh,et al.  Compositional Questions Do Not Necessitate Multi-hop Reasoning , 2019, ACL.

[138]  Richard Socher,et al.  Explain Yourself! Leveraging Language Models for Commonsense Reasoning , 2019, ACL.

[139]  Mohit Bansal,et al.  Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA , 2019, ACL.

[140]  Tassilo Klein,et al.  Attention Is (not) All You Need for Commonsense Reasoning , 2019, ACL.

[141]  Lei Li,et al.  Dynamically Fused Graph Network for Multi-hop Reasoning , 2019, ACL.

[142]  Ali Farhadi,et al.  HellaSwag: Can a Machine Really Finish Your Sentence? , 2019, ACL.

[143]  Johan Bos,et al.  HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning , 2019, *SEMEVAL.

[144]  Greg Durrett,et al.  Understanding Dataset Design Choices for Multi-hop Reasoning , 2019, NAACL.

[145]  Yoav Artzi,et al.  A Corpus for Reasoning about Natural Language Grounded in Photographs , 2018, ACL.

[146]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[147]  Yejin Choi,et al.  Social IQA: Commonsense Reasoning about Social Interactions , 2019, EMNLP 2019.

[148]  Yejin Choi,et al.  ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning , 2019, AAAI.

[149]  Jonathan Berant,et al.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge , 2019, NAACL.

[150]  Thomas Lukasiewicz,et al.  e-SNLI: Natural Language Inference with Natural Language Explanations , 2018, NeurIPS.

[151]  Yoshua Bengio,et al.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , 2018, EMNLP.

[152]  Guillaume Lample,et al.  XNLI: Evaluating Cross-lingual Sentence Representations , 2018, EMNLP.

[153]  R. Sarpong,et al.  Bio-inspired synthesis of xishacorenes A, B, and C, and a new congener from fuscol† †Electronic supplementary information (ESI) available. See DOI: 10.1039/c9sc02572c , 2019, Chemical science.

[154]  Bhavana Dalvi,et al.  Reasoning about Actions and State Changes by Injecting Commonsense Knowledge , 2018, EMNLP.

[155]  Guillaume Bouchard,et al.  Interpretation of Natural Language Rules in Conversational Machine Reading , 2018, EMNLP.

[156]  Yejin Choi,et al.  SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference , 2018, EMNLP.

[157]  Peter Clark,et al.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , 2018, EMNLP.

[158]  Rachel Rudinger,et al.  Hypothesis Only Baselines in Natural Language Inference , 2018, *SEMEVAL.

[159]  Yejin Choi,et al.  Event2Mind: Commonsense Inference on Events, Intents, and Reactions , 2018, ACL.

[160]  Peter Clark,et al.  SciTaiL: A Textual Entailment Dataset from Science Question Answering , 2018, AAAI.

[161]  Masatoshi Tsuchiya,et al.  Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment , 2018, LREC.

[162]  Oren Etzioni,et al.  Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , 2018, ArXiv.

[163]  Sebastian Riedel,et al.  Constructing Datasets for Multi-hop Reading Comprehension Across Documents , 2017, TACL.

[164]  Benno Stein,et al.  The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants , 2017, NAACL.

[165]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[166]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[167]  Adam Wierman,et al.  Thinking Fast and Slow , 2017, SIGMETRICS Perform. Evaluation Rev..

[168]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[169]  Catherine Havasi,et al.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge , 2016, AAAI.

[170]  Nathanael Chambers,et al.  A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories , 2016, NAACL.

[171]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[172]  Philipp Koehn,et al.  Synthesis Lectures on Human Language Technologies , 2016 .

[173]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[174]  Ido Dagan,et al.  Recognizing Textual Entailment: Models and Applications , 2013, Recognizing Textual Entailment: Models and Applications.

[175]  Zornitsa Kozareva,et al.  SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning , 2011, *SEMEVAL.

[176]  W. Marsden.  I and J , 2012 .

[177]  Gerhard Weikum,et al.  Robust Disambiguation of Named Entities in Text , 2011, EMNLP.

[178]  Akshay Java,et al.  The ICWSM 2009 Spinn3r Dataset , 2009 .

[179]  Hugo Liu,et al.  ConceptNet — A Practical Commonsense Reasoning Tool-Kit , 2004 .

[180]  Risto Hilpinen,et al.  Peirce's logic , 2004, The Rise of Modern Logic: From Leibniz to Frege.

[181]  John Woods,et al.  What Is Reasoning? What Is an Argument?* , 2001 .

[182]  A. Macintyre.  The Oxford Dictionary of Philosophy , 1996 .

[183]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[184]  H. Putnam,et al.  Reasoning and the Logic of Things. The Cambridge Conferences Lectures of 1898. , 1993 .

[185]  Paul Thagard,et al.  Epistemology and cognition , 1991 .

[186]  Critical thinking as argument analysis? , 1989 .

[187]  Maurice A. Finocchiaro,et al.  Informal Logic and the Theory of Reasoning , 1984 .

[188]  Ian Hacking,et al.  A concise introduction to logic , 1972 .

[189]  H. Eysenck,et al.  Thinking movement and the creation of dance through numbers , 2006 .

[190]  A. D. Ritchie.  The Dictionary of Philosophy , 1945, Nature.

[191]  J. Locke.  An Essay concerning Human Understanding , 1924, Nature.