Extractive Russian Text Summarization as Greedy Sentence Sequence Continuation Search with Probabilities from Pretrained Language Models

Abstract

Pretrained language models based on Transformer have defined new state-of-the-art result on a wide range of tasks being finetuned or used as embedders. Models with Multi-head Self-Attention mechanism have defined a new standart of quality on text summarization task in English, while Transformer based models potential for extractive Russian text summarization has been barely explored. We propose a new method for extractive Russian text summarization, reducing the task to the selection of the most probable sequence of sentences. The new method beats ROUGE-1 and ROUGE-L scores of other models such as SummaRuNNer, and mBART evaluated on Gazeta dataset and is more preferred in human evaluation poll.