Zero-shot Neural Passage Retrieval with a Pre-trained Language Model

May 17, 2023

Kioxia is developing Memory-Centric AI*¹ that can search through and reference its memory base. We present our recent research result on passage retrieval task, one of the things we are working on.

Passage retrieval is the task of finding relevant passages corresponding to a given question (Fig.1). It has been shown that pre-trained*² neural language models can perform as a neural passage retriever when fine-tuned*³ for retrieval. However, the existing neural retrievers do not perform well especially when a named entity (such as a name of person) in a question is a dominant clue for retrieval^[1]. To address the issue, several approaches have been proposed to modify the method of fine-tuning for retrieval.

In contrast, our idea is opposite to the conventional one: employing a pre-trained language model without fine-tuning. Our intuition is that; a pre-trained language model should have rich knowledge including named entities and the rich knowledge would be forgotten during fine-tuning*⁴, resulting in poor retrieval performance on entity-centric questions. However, it has not been clear how to leverage a pre-trained language model for retrieval without fine-tuning.

We propose a method that allows a pre-trained language model to be used for retrieval without fine-tuning (Fig.2). A pre-trained language model encodes the named entities extracted from a question and passages into contextualized representations (vectors). Our method requires only one language model, while conventional methods require to fine-tune two language models, one for questions and the other for passages (Fig.3).

Our proposed method achieves almost comparable performance (67.1%) to a state-of-the-art retriever (74.5%) and shows the best performance for two relations out of 24 relations, when evaluated on the question-answering dataset ^[1] consisting of a total of 24 relations (e.g., place of birth) ^{[2, 3]}. Conventional methods require creation of a dataset in a specific domain, such as medical domain, to build a neural retriever that works in the domain. Our method, on the other hand, demonstrates the possibility of retrieval without building the additional dataset, as long as the language model is pre-trained in the domain.