RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning

In recent studies it has been shown that Multilingual language models under perform their monolingual counterparts (Conneau et al., 2020). It is also a well known fact that training and maintaining monolingual models for each language is a costly and time consuming process. Roman Urdu is a resource starved language used popularly on social media platforms and chat apps. In this research we propose a novel dataset of scraped tweets containing 54M tokens and 3M sentences. Additionally we also propose RUBERT a bilingual Roman Urdu model created by additional pretraining of English BERT (Devlin et al., 2019). We compare its performance with a monolingual Roman Urdu BERT trained from scratch and a multilingual Roman Urdu BERT created by additional pretraining of Multilingual BERT (mBERT (Devlin et al., 2019)). We show through our experiments that additional pretraining of the English BERT produces the most notable performance improvement.