license: cc-by-nc-sa-4.0
Editing Conceptual Knowledge for Large Language Models
Overview β’ How To Use β’ Citation β’ Paper β’ Website
π‘ Conceptual Knowledge Editing
Task Definition
Concept is a generalization of the world in the process of cognition, which represents the shared features and essential characteristics of a class of entities.
Therefore, the endeavor of concept editing aims to modify the definition of concepts, thereby altering the behavior of LLMs when processing these concepts.
Evaluation
To analyze conceptual knowledge modification, we adopt the metrics for factual editing (the target is the concept $C$ rather than factual instance $t$).
Reliability: the success rate of editing with a given editing descriptionGeneralization: the success rate of editing within the editing scopeLocality: whether the model's output changes after editing for unrelated inputs
Concept Specific Evaluation Metrics
Instance Change: capturing the intricacies of these instance-level changesConcept Consistency: the semantic similarity of generated concept definition
π Usage
π Current Implementation
As the main Table of our paper, four editing methods are supported for conceptual knowledge editing.
| Method | GPT-2 | GPT-J | LlaMA2-13B-Chat | Mistral-7B-v0.1 |
|---|---|---|---|---|
| FT | β | β | β | β |
| ROME | β | β | β | β |
| MEMIT | β | β | β | β |
| PROMPT | β | β | β | β |
π» Run
You can follow EasyEdit to run the experiments.
π Citation
Please cite our paper if you use ConceptEdit in your work.
@misc{wang2024editing,
title={Editing Conceptual Knowledge for Large Language Models},
author={Xiaohan Wang and Shengyu Mao and Ningyu Zhang and Shumin Deng and Yunzhi Yao and Yue Shen and Lei Liang and Jinjie Gu and Huajun Chen},
year={2024},
eprint={2403.06259},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
π Acknowledgement
We would like to express our sincere gratitude to DBpediaοΌWikidataοΌOntoProbe-PLMs and ROME.
Their contributions are invaluable to the advancement of our work.