Research Projects

Current scene graph generation (SGG) methods are limited by closed vocabularies, restricting their ability to recognize novel objects and relationships in real-world scenarios. While natural language captions provide coarse image descriptions, they lack the structured representation needed for compositional reasoning and complex visual understanding tasks. Scene graphs offer explicit triplet structures (subject-predicate-object) that enable contextual disambiguation, compositional reasoning, and superior performance in downstream applications like visual question answering and robot manipulation. Scientific Objectives: This research proposes a novel Masked-object Open-Vocabulary Scene Graph Generation (Masked OvSGG) framework that combines vision-language alignment with masked modeling to create robust, generalizable scene understanding systems. The primary objective is to develop a model that can recognize novel objects and relationships while maintaining robustness to partial observations through self-supervised masked learning. Hypothesis/Model: We hypothesize that randomly masking object nodes during training and forcing the model to predict them from contextual cues will improve structural understanding and generalization to unseen categories. By integrating pretrained vision-language models (VLMs) with masked autoencoder principles, the model will learn context-aware representations that surpass current open-vocabulary SGG methods. Main Experiments: (1) Develop masking operators for scene graphs with formal mathematical definitions, (2) Implement vision-language alignment using CLIP-style contrastive learning for both objects and relations, (3) Design composite loss functions combining contrastive alignment and masked reconstruction objectives, (4) Conduct extensive experiments on Visual Genome, COCO, and GQA datasets using standard SGG metrics (Recall@K, mean Recall@K), (5) Perform ablation studies comparing masking strategies, distillation techniques, and LLM-enhanced prompts, (6) Evaluate open-vocabulary performance on seen vs. unseen categories. Significance: This research addresses fundamental limitations in current SGG methods by introducing self-supervised learning principles to structured scene understanding. The expected breakthrough lies in creating the first SGG system that simultaneously handles novel vocabularies and partial observations, potentially revolutionizing applications in autonomous systems, medical imaging, and human-computer interaction. The framework's ability to generalize to new object-relation combinations without retraining could significantly reduce annotation costs and enable deployment in diverse, evolving environments.