Detecting Romanized Thai tokens in social media texts

Social media contents were created by a large number of users or writers. Additionally, each of them has their own writing styles, which depend on their creative thinking or attitudes. As commonly found in online social networks of Thai users, typed texts sometimes include Thai words that were transliterated with Roman letters. Therefore, text-to-speech systems cannot pronounce these transliterated tokens correctly. In this work, we propose and evaluate statistical methods for detecting Romanized Thai tokens. Both context-dependent and context-free classification features are proposed. Real social network texts are used for constructing the training set and the test set. Human subjects can detect Thai Romanized tokens at 91.16% accuracy on average when adjacent contexts are hidden while the accuracy is at 99.41% with contexts. With the proposed features, a decision tree-based classifier and an N-gram-based classifier yield 87.63% and 74.42% accuracy, respectively. In the later case, the accuracy increases to 82.60% when the tokens' existence in English dictionaries is considered. Combining the two methods results in a detection accuracy of 89.36%.