Enhancing text comprehension via fusing pre-trained language model with knowledge graph

Abstract

Pre-trained language models (PLMs) such as BERT and GPTs capture rich linguistic and syntactic knowledge from pre-training over large-scale text corpora, which can be further fine-tuned for specific downstream tasks. However, these models still have limitations as they rely on knowledge gained from plain text and ignore structured knowledge such as knowledge graphs (KGs). Recently, there has been a growing trend of explicitly integrating KGs into PLMs to improve their performance. For instance, K-BERT incorporates KG triples as domain-specific supplements into input sentences. Nevertheless, we have observed that such methods do not consider the semantic relevance between the introduced knowledge and the original input sentence, leading to the issue of knowledge impurities. To address this issue, we propose a semantic matching-based approach that enriches the input text with knowledge extracted from an external KG. The architecture of our model comprises three components: the knowledge retriever (KR), the knowledge injector (KI), and the knowledge aggregator (KA). The KR, built upon the sentence representation learning model (i.e. CoSENT), retrieves triples with high semantic relevance to the input sentence from an external KG to alleviate the issue of knowledge impurities. The KI then integrates the retrieved triples from the KR into the input text by converting the original sentence into a knowledge tree with multiple branches, the knowledge tree is transformed into an accessible sequence of text that can be fed into the KA. Finally, the KA takes the flattened knowledge tree and passes it through an embedding layer and a masked Transformer encoder. We conducted extensive evaluations on eight datasets covering five text comprehension tasks, and the experimental results demonstrate that our approach exhibits competitive advantages over popular knowledge-enhanced PLMs such as K-BERT and ERNIE.