Research Projects

In this project, we aim to investigate harmfulness within memes as an umbrella effect that can help detect, trace, and moderate any form of illicit content being disseminated via memes. We aim to address critical challenges like abstract obscurity, multimodal amalgamation, contextual dependency, and cross-modal (dis-)association. To this end, we propose to (a) characterize memetic harmfulness and targeted category detection, (b) model harmfully targeted entity detection, and (c) meme contextualization. To achieve these objectives, we will first collect a large-scale dataset, which will involve annotations for harmfulness intensity, protected category and entity assignment, and natural language explanations. As part of harmfulness and category type detection in memes, we will analyze harmfulness cues within memes by examining the dataset cross-sectioned by various domains, categories, and topics. The distributional analyses of the dataset will follow this. The next step involves designing an effective multimodal fusion strategy that factors in and fuses localized and global perspectives conveyed within memes. The intra/inter-modal fusion strategies would be directed towards systematically fusing the multimodal analogies encoded by multiple modalities within memes. Finally, the multimodal representations thus learned will be used toward multiclass and multi-label classification objectives. Towards harmful target detection, the first step would be to segregate the dataset, w.r.t entities that are exposed differently (harmfully vs. non-harmfully) during the training phase, to establish generalizability during the test time. As part of the modeling effort, we aim to learn an entity dictionary before contextualizing it using encoded world knowledge. Then we would incorporate the embedded harmfulness via the verbal cues embedded within the meme. Finally, the visual features encoded would be fused via an effective technique like adaptive multimodal low-rank bilinear pooling towards eventual binary classification for harmful vs. non-harmful targeting. Our final objective towards contextualizing memes entails working with a multimodal dataset with manually annotated natural language rationales to justify semantic role label assignments to different entities mentioned. This would be realized by establishing a multi-task learning-based framework, wherein related tasks would be jointly optimized against. In the process, we would realize the optimized explanation generation model, which could be an auto-regressive language model. This project aims to achieve an automated, comprehensive multimodal meme analysis system that can help detect, track, and contextualize harmful memes for building assistive moderation technology. This project intends to generate IPs, patents, publications, and collaborations as outcomes from the initiatives aligned w.r.t the set objectives.