Pipeline Signed Japanese Translation Focusing on a Post-positional Particle Complement and Conjugation in a Low-resource Setting

Because sign language is a visual language, the translation of it into spoken language is typically performed through an intermediate representation called gloss notation. In sign language, function words, such as particles and determiners, are not explicitly expressed, and there is little or no concept of morphological inflection in sign language. Therefore, gloss notation does not include such linguistic constructs. Because of these factors, we argue that sign language translation is effectively processed by taking advantage of the similarities and differences between sign language and its spoken counterpart. We thus propose a pipeline translation method that clearly focuses on the difference between spoken Japanese and signed Japanese written in gloss notation. Specifically, our method first uses statistical machine translation (SMT) to map glosses to corresponding spoken language words. We then use three transformer-based seq2seq models trained using a large out-ofdomain monolingual Japanese corpus to complement postpositional particles and estimate conjugations for the verbs, adjectives, and auxiliary verbs in the first translation. We apply the seq2seq models in sequence until the translation converges. Our experimental results show that the proposed method performs robustly on the low-resource corpus and is +4.4/+4.9 points above the SMT baseline for BLEU-3/4.