Research Projects

Machine Learning (ML) hinges on optimization algorithms to efficiently train predictive models across domains. This proposal tries to advance the optimization for ML with two work packages. The first work package is focused on understanding implicit biases in optimization algorithms. As part of the second work package, I will be devising performative optimization schemes for the scenarios of decision-dependent data distribution. Contributions made through the proposed work will advance the foundational understanding of ML algorithms and develop efficient optimization schemes for real-world problems. The scientific objectives along with key hypotheses to be tested for the two work packages are listed as follows. Overparameterized models have demonstrated remarkable predictive power across a wide range of tasks. Although optimization problems involving overparameterized models admit many solutions, prior works have studied implicit regularization (or bias) with popular optimizers [1]. The objective of Work Package 1 is to extend the understanding of these implicit biases to underexplored settings including the study with accelerated variant of gradient descent, with relaxed conditions on the learning rate, with optimization over non-Euclidean geometry, in problems other than classification and in multi-objective optimization. This will also include studying implicit regularization with more general loss functions like the Fenchel Young loss or Fitzpatrick loss. I will also explore the interplay between inductive biases (eg. choice of architecture) and explicit regularization (eg. dropout scheme) on the implicit bias. Finally, I plan to use the tools for inverse gradient flow to uncover possible implicit regularization during training. Work Package 2 tries to advance the existing optimization schemes in performative optimization that deals with the case of non-stationary decision-dependent data distribution and has led to the study of performative optimizers. The convergence analysis of these optimizers have been studied for a restricted class of loss functions [2], which I plan to extend. A significant part of this work package will also be to uncover possible implicit biases in these performative optimization algorithms. To the best of my knowledge, prior works have not looked at implicit regularization of performative optimizers. The significance of the proposal is summarized as follows. Work Package 1 will expand our theoretical understanding of implicit biases present in optimization algorithms. Work Package 2 focuses on expanding the theory for performative optimization and investigating possible implicit biases in these. Together, these will enrich the foundational theory of ML optimization and yield algorithms poised for real‐world impact. [1] Gal Vardi. On the Implicit Bias in Deep-Learning Algorithms (a survey). CACM, 2023. [2] Li et al. Stochastic Optimization Schemes for Performative Prediction with Nonconvex Loss. NeurIPS, 2024.