Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification

A meme is an part of media created to share an opinion or emotion across the internet. Due to its popularity, memes have become the new forms of communication on social media. However, due to its nature, they are being used in harmful ways such as trolling and cyberbullying progressively. Various data modelling methods create different possibilities in feature extraction and turning them into beneficial information. The variety of modalities included in data plays a significant part in predicting the results. We try to explore the significance of visual features of images in classifying memes. Memes are a blend of both image and text, where the text is embedded into the image. We try to incorporate the memes as troll and non-trolling memes based Siddhanth U Hegde University Visvesvaraya College of Engineering, Bangalore University siddhanthhegde227@gmail.com Adeep Hande Indian Institute of Information Technology Tiruchirappalli, Tamil Nadu, India adeeph18c@iiitt.ac.in Ruba Priyadharshini ULTRA Arts and Science College, Madurai, Tamil Nadu, India rubapriyadharshini.a@gmail.com Sajeetha Thavareesan, Ratnasingam Sakuntharaj Eastern University, Sri Lanka {sajeethas,sakuntharaj}@esn.ac.lk Sathiyaraj Thangasamy Sri Krishna Adithya College of Arts and Science, Coimbatore, Tamil Nadu, India. sathiyarajt@skacas.ac.in B Bharathi SSN College of Engineering, Tamil Nadu, India. bharathib@ssn.edu.in Bharathi Raja Chakravarthi* Insight SFI Research Centre for Data Analytics, National University of Ireland Galway, Galway, Ireland bharathi.raja@insight-centre.org on the images and the text on them. However, the images are to be analysed and combined with the text to increase performance. Our work illustrates different textual analysis methods and contrasting multimodal methods ranging from simple merging to cross attention to utilising both worlds’ best visual and textual features. The fine-tuned cross-lingual language model, XLM, performed the best in textual analysis, and the multimodal transformer performs the best in multimodal analysis.

[1]  Jean H. French.  Image-based memes as sentiment predictors , 2017, 2017 International Conference on Information Society (i-Society).

[2]  Eva Schlinger,et al.  How Multilingual is Multilingual BERT? , 2019, ACL.

[3]  Qinyu Que.  Simon @ DravidianLangTech-EACL2021: Meme Classification for Tamil with BERT , 2021, DRAVIDIANLANGTECH.

[4]  Georg Heigold,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2021, ICLR.

[5]  Veselin Stoyanov,et al.  Unsupervised Cross-lingual Representation Learning at Scale , 2019, ACL.

[6]  B. B,et al.  SSNCSE_NLP@DravidianLangTech-EACL2021: Meme classification for Tamil using machine learning approach , 2021, DRAVIDIANLANGTECH.

[7]  Keiron O'Shea,et al.  An Introduction to Convolutional Neural Networks , 2015, ArXiv.

[8]  Manoj Jayabalan,et al.  Neural Machine Translation: A Review of the Approaches , 2019 .

[9]  Quoc V. Le,et al.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators , 2020, ICLR.

[10]  Yang Bai,et al.  HUB@DravidianLangTech-EACL2021: Meme Classification for Tamil Text-Image Fusion , 2021, DRAVIDIANLANGTECH.

[11]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[12]  L. Grundlingh,et al.  Memes as speech acts , 2018 .

[13]  Yiming Yang,et al.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context , 2019, ACL.

[14]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Atreyee Dey,et al.  MuRIL: Multilingual Representations for Indian Languages , 2021, ArXiv.

[16]  D. Karthika Renuka,et al.  Spam Classification Based on Supervised Learning Using Machine Learning Techniques , 2011, 2011 International Conference on Process Automation, Control and Computing.

[17]  Keping Li,et al.  Network-Based Bag-of-Words Model for Text Classification , 2020, IEEE Access.

[18]  P. Sivakumar,et al.  A REVIEW ON IMAGE SEGMENTATION TECHNIQUES , 2016 .

[19]  Hyeseung Park,et al.  Detection of Abandoned and Stolen Objects Based on Dual Background Model and Mask R-CNN , 2020, IEEE Access.

[20]  Mohammed Moshiul Hoque,et al.  NLP-CUET@DravidianLangTech-EACL2021: Investigating Visual and Textual Features to Identify Trolls from Multimodal Social Media Memes , 2021, DRAVIDIANLANGTECH.

[21]  Adithya Avvaru,et al.  BERT at SemEval-2020 Task 8: Using BERT to Analyse Meme Emotions , 2020, SemEval@COLING.

[22]  Paul Buitelaar,et al.  A Dataset for Troll Classification of TamilMemes , 2020, WILDRE.

[23]  Jong-Seok Lee,et al.  EmbraceNet: A robust deep learning architecture for multimodal classification , 2019, Inf. Fusion.

[24]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[25]  Bharathi Raja Chakravarthi,et al.  UVCE-IIITT@DravidianLangTech-EACL2021: Tamil Troll Meme Classification: You need to Pay more Attention , 2021, DRAVIDIANLANGTECH.

[26]  R. Milner.  FCJ-156 Hacking the Social: Internet Memes, Identity Antagonism, and the Logic of Lulz. , 2013 .

[27]  Seth Flaxman,et al.  Multimodal Sentiment Analysis To Explore the Structure of Emotions , 2018, KDD.

[28]  Yuhao Du,et al.  Understanding Visual Memes: An Empirical Analysis of Text Superimposed on Memes Shared on Twitter , 2020, ICWSM.

[29]  Codewithzichao@DravidianLangTech-EACL2021: Exploring Multimodal Transformers for Meme Classification in Tamil Language , 2021, DRAVIDIANLANGTECH.

[30]  Wenzhong Guo,et al.  Deep Multimodal Representation Learning: A Survey , 2019, IEEE Access.

[31]  S. Chitrakala,et al.  A survey on extractive text summarization , 2017, 2017 International Conference on Computer, Communication and Signal Processing (ICCCSP).

[32]  Pengshuai Yin,et al.  Deep Guidance Network for Biomedical Image Segmentation , 2020, IEEE Access.

[33]  Bharathi Raja Chakravarthi,et al.  IIITK@DravidianLangTech-EACL2021: Offensive Language Identification and Meme Classification in Tamil, Malayalam and Kannada , 2021, DRAVIDIANLANGTECH.

[34]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[35]  Naveen Arivazhagan,et al.  Language-agnostic BERT Sentence Embedding , 2020, ArXiv.

[36]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[37]  Bharathi Raja Chakravarthi,et al.  Findings of the Shared Task on Troll Meme Classification in Tamil , 2021, DRAVIDIANLANGTECH.

[38]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Christo Kirov,et al.  Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset , 2020, LREC.

[41]  L. Shifman,et al.  Talking It Personally: Features of Successful Political Posts on Facebook , 2018, Social Media + Society.

[42]  Venkatesh,et al.  Fine-tuned MobileNet Classifier for Classification of Strawberry and Cherry Fruit Types , 2021, 2021 International Conference on Computer Communication and Informatics (ICCCI).

[43]  Sunil Saumya,et al.  IIIT_DWD@EACL2021: Identifying Troll Meme in Tamil using a hybrid deep learning approach , 2021, DRAVIDIANLANGTECH.