Research Projects

Overparameterized deep neural networks and generative models, like diffusion models and large language models, are central to modern machine learning, achieving impressive performance through extensive parameterization. However, limited theoretical understanding constrains further progress, and the large number of parameters complicates optimization and deployment on resource-limited devices. This research addresses these challenges by investigating (i) the statistical foundations of gradient-based optimization and feature learning in these models, (ii) the capabilities of generative models for sampling and learning dynamical systems, and (iii) iterative model compression methods for personalized, on-device learning. Our primary objective is to comprehend how optimization techniques, such as gradient descent (GD) and stochastic gradient descent (SGD), traverse the complex non-convex landscape of these models. We aim to delve into the feature learning mechanisms within deep neural networks and generative models during training with GD or SGD. This insight into the dynamics of GD and SGD within discriminative (CNN, Resnet) and generative (diffusion model, LLMs) models can guide us in designing superior algorithms, consequently enhancing training efficiency and model performance. Furthermore, our research seeks to delve into the generative properties of overparametrized deep generative models. We aim to gain novel insights into sampling from these models and refine MCMC sampling algorithms using them. We would also investigate the effectiveness of LLMs in learning dynamical system. By focusing on theoretical aspects, this research aims to answer key questions like Model Representations, Generalization and Stability, Memory and Long-Range Dependencies and Interpretability and Structure. Additionally, the project addresses the challenge of deploying and training highly over-parametrized models on mobile and edge devices by developing sparse iterative methods and novel model compression techniques tailored for such models, supported with strong theoretical guarantees. Investigating the relationship between model compressibility, sparsity, and generalization behavior can lead to more effective compression methods and compressed training methods, enhancing overall model performance and energy efficiency. This research has the potential to significantly impact machine learning by advancing the understanding of optimization, generalization, and compression in overparameterized models. This could lead to more efficient, scalable training algorithms and better model compression methods, particularly for real-time, high-performance applications in mobile and edge settings. Furthermore, insights into the generative capabilities of models like LLMs could expand their use in scientific modeling, especially for complex systems like dynamical systems, potentially impacting healthcare, finance, and engineering with accessible machine learning solutions.