Research Projects

The problem of novel view synthesis is fundamental in computer vision and graphics and has several applications in augmented and virtual reality, robotics, autonomous driving and telepresence. The advent of neural radiance fields (NeRFs) had a significant impact on the quality of novel renders while allowing for small model sizes. Although NeRFs achieve high quality novel views, they are extremely slow in terms of training and rendering times, owing to the need for multiple queries to a large neural network. In this context 3D Gaussian splatting (3DGS) models allow for extremely fast rendering at high visual quality. However, these 3DGS models typically require a large number of input views for training. Their performance degrades significantly as the number of input views reduces. Secondly, 3DGS models are primarily designed for static scenes. Their extension to dynamic scenes requires the introduction of non-trivial modeling capabilities. One of the major distortions that occur while training 3DGS models under sparse input conditions is the appearance of floating objects (or floaters) with incorrect depths. Several recent methods seek to improve 3DGS under sparse conditions by mitigating such distortions. However, they either suffer from the use of depth priors that may not generalize well for diverse scenes, or tend to smooth out the reconstructions leading to loss of details. Thus, there is a need to address the shortcomings of existing sparse-input 3DGS methods in balancing the recovery of details and removing floaters. We observe that there are largely two key levers for training 3DGS models in the sparse setting. Firstly, the densification step is very important in the sparse setting due to the limited number of initalization points. Due to the lack of sufficient constraints imposed by limited views, uncontrolled densification can lead to floaters or objects are incorrect depths. This motivates the design of controlled densification models in the sparse input case. The other popular lever is the design of priors. In this context, we further see two large sets of priors. One of the goals of our proposal is to investigate the relevance of generative image or video priors. Alternately, we also wish to leverage the success of feedforward 3DGS models and leverage priors about scenes learnt from them for superior novel view synthesis in the sparse setting. Next, we note that that there is a severe limitation in terms of the availability of datasets that are particularly relevant for sparse input novel view synthesis. The camera spacing in current datasets does not realistically reflect the challenges that one might face when a scene ought to be reconstructed using a handful of camera views. We seek to design a dataset that carefully considers the challenges of sparse input novel view synthesis. The final major focus of our proposal is the extension of 3DGS to dynamic scenes. Many existing dynamic scene models primarily simulate motion via time-varying opacity and color, leading to supervision from very few frames, insufficient densification, and blurred reconstructions. Our primary goal is to first explore how to achieve better densification in the dynamic regions to allow for superior reconstruction of details. Secondly, current models do not effectively model motion, leading to ineffective use of Gaussians. As the duration of the video increases, such inefficient modeling can lead to extremely large model sizes. Our main goal in this proposal is to enable the Gaussians to move for long duration sequences (of the order of a few minutes), while keeping the model size under control. We believe that this can be achieved by enabling the Gaussians to move to model object motion instead of other mechanisms adopted by existing dynamic 3DGS models. Our proposal will have a significant impact on sparse input and dynamic 3D Gaussian splatting making such models more readily deployable in practical applications.