论文标题
在基于变压器的编码器中延迟交互层以进行有效的开放域问题回答
Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering
论文作者
论文摘要
大规模文档语料库（例如Wikipedia）上的开放域问答（ODQA）是计算机科学的关键挑战。尽管基于变压器的语言模型（例如BERT）在Squad上显示了超越人类在文本中提取答案的能力，但面对更大的搜索空间时，它们的复杂性很高。解决此问题的最常见方法是添加初步信息检索步骤，以大量过滤语料库，仅保留相关段落。在本文中，我们提出了一个更直接和互补的解决方案，其中包括应用基于变压器模型的架构的通用变化，以延迟输入子部分之间的关注并允许对计算的更有效管理。所得的变体与采掘任务上的原始模型具有竞争力，并且在ODQA设置上允许在许多情况下进行大幅度加速，甚至可以提高性能。
Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although transformer-based language models such as Bert have shown on SQuAD the ability to surpass humans for extracting answers in small passages of text, they suffer from their high complexity when faced to a much larger search space. The most common way to tackle this problem is to add a preliminary Information Retrieval step to heavily filter the corpus and only keep the relevant passages. In this paper, we propose a more direct and complementary solution which consists in applying a generic change in the architecture of transformer-based models to delay the attention between subparts of the input and allow a more efficient management of computations. The resulting variants are competitive with the original models on the extractive task and allow, on the ODQA setting, a significant speedup and even a performance improvement in many cases.
