EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models

Large Language Models (LLMs) usually suffer from knowledge cutoff or fallacy issues, which means they are unaware of unseen events or generate text with incorrect facts owing to the outdated/noisy data. To this end, many knowledge editing approaches for LLMs have emerged -- aiming to subtly inject/edit updated knowledge or adjust undesired behavior while minimizing the impact on unrelated inputs. Nevertheless, due to significant differences among various knowledge editing methods and the variations in task setups, there is no standard implementation framework available for the community, which hinders practitioners to apply knowledge editing to applications. To address these issues, we propose EasyEdit, an easy-to-use knowledge editing framework for LLMs. It supports various cutting-edge knowledge editing approaches and can be readily apply to many well-known LLMs such as T5, GPT-J, LlaMA, etc. Empirically, we report the knowledge editing results on LlaMA-2 with EasyEdit, demonstrating that knowledge editing surpasses traditional fine-tuning in terms of reliability and generalization. We have released the source code on GitHub at https://github.com/zjunlp/EasyEdit, along with Google Colab tutorials and comprehensive documentation for beginners to get started. Besides, we present an online system for real-time knowledge editing, and a demo video at http://knowlm.zjukg.cn/easyedit.mp4.

[1]  Shenqi Jing,et al.  AsdKB: A Chinese Knowledge Base for the Early Screening and Diagnosis of Autism Spectrum Disorder , 2023, SEMWEB.

[2]  Wayne Xin Zhao,et al.  Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation , 2023, ArXiv.

[3]  Roberta Raileanu,et al.  Challenges and Applications of Large Language Models , 2023, ArXiv.

[4]  Eric Michael Smith,et al.  Llama 2: Open Foundation and Fine-Tuned Chat Models , 2023, ArXiv.

[5]  Cheng Chang,et al.  Secrets of RLHF in Large Language Models Part I: PPO , 2023, ArXiv.

[6]  Nelson F. Liu,et al.  Lost in the Middle: How Language Models Use Long Contexts , 2023, TACL.

[7]  Xindong Wu,et al.  Unifying Large Language Models and Knowledge Graphs: A Roadmap , 2023, IEEE Transactions on Knowledge and Data Engineering.

[8]  Akshay Krishna Sheshadri,et al.  Editing Commonsense Knowledge in GPT , 2023, ArXiv.

[9]  Lei Li,et al.  Can We Edit Factual Knowledge by In-Context Learning? , 2023, EMNLP.

[10]  Peng Wang,et al.  Editing Large Language Models: Problems, Methods, and Opportunities , 2023, ArXiv.

[11]  Andrew M. Dai,et al.  PaLM 2 Technical Report , 2023, ArXiv.

[12]  A. Globerson,et al.  Dissecting Recall of Factual Associations in Auto-Regressive Language Models , 2023, ArXiv.

[13]  Jacob Andreas,et al.  Inspecting and Editing Knowledge Representations in Language Models , 2023, 2304.00740.

[14]  Wayne Xin Zhao,et al.  A Survey of Large Language Models , 2023, ArXiv.

[15]  Haitao Zheng,et al.  Parameter-efficient fine-tuning of large-scale pre-trained language models , 2023, Nature Machine Intelligence.

[16]  Jie Zhou,et al.  Transformer-Patcher: One Mistake worth One Neuron , 2023, ICLR.

[17]  Mohit Bansal,et al.  Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models , 2023, ArXiv.

[18]  Fei Huang,et al.  Reasoning with Language Model Prompting: A Survey , 2022, ACL.

[19]  Shuohang Wang,et al.  Retrieval Augmentation for Commonsense Reasoning: A Unified Approach , 2022, EMNLP.

[20]  Arnab Sen Sharma,et al.  Mass-Editing Memory in a Transformer , 2022, ICLR.

[21]  Christopher D. Manning,et al.  Memory-Based Model Editing at Scale , 2022, ICML.

[22]  Yoav Goldberg,et al.  Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space , 2022, EMNLP.

[23]  Dipankar Ray,et al.  ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection , 2022, ACL.

[24]  Haitao Zheng,et al.  Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models , 2022, ArXiv.

[25]  David Bau,et al.  Locating and Editing Factual Associations in GPT , 2022, NeurIPS.

[26]  Pascale Fung,et al.  Survey of Hallucination in Natural Language Generation , 2022, ACM Comput. Surv..

[27]  Diego de Las Casas,et al.  Improving language models by retrieving from trillions of tokens , 2021, ICML.

[28]  Li Dong,et al.  Knowledge Neurons in Pretrained Transformers , 2021, ACL.

[29]  Nicola De Cao,et al.  Editing Factual Knowledge in Language Models , 2021, EMNLP.

[30]  Chuanqi Tan,et al.  KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction , 2021, WWW.

[31]  Stella Biderman,et al.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , 2021 .

[32]  D. Klein,et al.  Calibrate Before Use: Improving Few-Shot Performance of Language Models , 2021, ICML.

[33]  Omer Levy,et al.  Transformer Feed-Forward Layers Are Key-Value Memories , 2020, EMNLP.

[34]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[35]  Fabio Petroni,et al.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks , 2020, NeurIPS.

[36]  Ming-Wei Chang,et al.  REALM: Retrieval-Augmented Language Model Pre-Training , 2020, ICML.

[37]  Colin Raffel,et al.  How Much Knowledge Can You Pack into the Parameters of a Language Model? , 2020, EMNLP.

[38]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[39]  Frank F. Xu,et al.  How Can We Know What Language Models Know? , 2019, Transactions of the Association for Computational Linguistics.

[40]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[41]  R'emi Louf,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[42]  Guilin Qi,et al.  Knowledge graph construction from multiple online encyclopedias , 2019, World Wide Web.

[43]  Sebastian Riedel,et al.  Language Models as Knowledge Bases? , 2019, EMNLP.

[44]  Omer Levy,et al.  Zero-Shot Relation Extraction via Reading Comprehension , 2017, CoNLL.

[45]  A. R. Mcclure.  Empirical Methods , 1948 .

[46]  A. Globerson,et al.  Evaluating the Ripple Effects of Knowledge Editing in Language Models , 2023, TACL.

[47]  Aitor Lewkowycz,et al.  Effect of scale on catastrophic forgetting in neural networks , 2022, ICLR.

[48]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[49]  Aidong Zhang,et al.  A Survey on Context Learning , 2017, IEEE Transactions on Knowledge and Data Engineering.