LordBERT: Embedding Long Text by Segment Ordering with BERT

Although BERT has achieved significant improvements on many downstream NLP tasks, it has difficulty handling long text because of its quadratic computation complexity. A typical approach to this issue is splitting the input into shorter segments and utilizing order-independent attention mechanism to conduct intersegment interaction, but the approach ignores the segment order information, which is greatly beneficial for capturing implicit relations across different segments. To address this problem, we propose a novel multi-task learning framework, named LordBERT, which fully exploits both intra- and inter-segment information in long text by segment ordering with BERT. LordBERT learns segment-level representations from segments through BERT and a reasoner, and utilizes an auxiliary segment ordering module to reorder disordered segments. With this module, the model implicitly encodes intersegment relations and global information of long text into segment representations. The downstream task and the ordering task are jointly optimized during training, while for inferencing we mainly conduct the downstream task. Experimental results show that LordBERT outperforms the state-of-the-art models by up to 0.58% in accuracy for text classification tasks on long text.