Research Projects

Video data constitutes over 70% of global internet traffic, underscoring the need for highly efficient compression techniques to address bandwidth and storage challenges. Traditional video compression methods, though advancing, are struggling to keep pace with the rapid surge in video traffic. Generative compression approaches, leveraging state-of-the-art diffusion models that are capable of producing photorealistic visual data, present a promising alternative. Unlike traditional lossy compression, which irreversibly degrades visual quality to reduce file sizes, generative models synthesize new pixels, thereby producing high quality results while achieving substantial size reductions. However, the reverse diffusion process in such models requires careful conditioning to preserve fidelity to the source content and is computationally intensive due to its iterative noise-removal mechanism. This project proposes a novel generative compression framework by conditioning a latent diffusion model on both the latent representation of the visual content and the spatial perceptual quality distribution derived from prior diffusion steps. This targeted conditioning focuses the model on reconstructing visually complex regions that typically exhibit greater degradation in quality-agnostic frameworks. We propose to derive this conditioning information by developing a model for predicting the quality of AI generated visual content at different levels of visual quality, corresponding to the different stages of the reverse diffusion process. We also propose to compress videos by for predicting video frames in a generative fashion, using motion cues to condition the generation in addition to their spatial quality maps. To improve the computational cost of frame prediction for video compression through diffusion, the proposed project also aims to reduce the number of reverse diffusion steps for frame generation by mixing noise and previous frame predictions of the model in a motion guided manner, so that the computation of the reverse diffusion steps is directed towards reconstructing regions with higher uncertainty, manifested by their higher motion content. Validation of the novel ideas proposed in this project through comprehensive experimentation and analysis would reduce bandwidth, power, and storage costs associated with video data transmission, promote sustainability, and enhance user experiences in low bandwidth environments.