32 15 v 3 [ cs . C L ] 1 4 D ec 2 01 4 Sequence to Sequence Learning with Neural Networks

Deep Neural Networks (DNNs) are powerful models that have ac hieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be us ed to map sequences to sequences. In this paper, we present a general end-to-end ap proach to sequence learning that makes minimal assumptions on the sequence str ucture. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LS TM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT’14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the L STM’s BLEU score was penalized on out-of-vocabulary words. Additiona lly, the LSTM did not have difficulty on long sentences. For comparison, a phraseb d SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementione d SMT system, its BLEU score increases to 36.5, which is close to the previous b est result on this task. The LSTM also learned sensible phrase and sentence rep resentations that are sensitive to word order and are relatively invariant to t he active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM’s pe rformance markedly, because doing so introduced many short term dependencies be tween the source and the target sentence which made the optimization problem easier.