Research Projects

Stochastic approximation (SA) was proposed by H. Robbins and S. Monro in a paper published in 1951. This algorithm involves incremental updates and works with noisy cost observations while not knowing the cost function. It has been found to be extremely useful in a number of applications where one does not know the underlying system dynamics/model but wants to find say the optimum of a function or its fixed point under noisy observations. It finds applications in areas such as adaptive control, stochastic optimization (SO), reinforcement learning (RL), machine learning, applications of which are also studied in various engineering domains. Over a number of years, the foundations of SA have been significantly strengthened, however, there are still gaps that remain which this project aims to fill. An important goal will be to provide verifiable sufficient conditions for stability and convergence of non-smooth objectives and set-valued maps. It will also generalize the noise model to Markov noise as needed for online RL, unlike standard analyses that involve just martingale difference noise. For instance, online RL applications typically involve Markov noise which the standard approaches do not cater to. This analysis will also be provided for multi-timescale SA involving multiple coupled recursions operating on different timescales as is done in actor-critic RL algorithms. We shall then pursue the design and analysis of algorithms for control and optimization of random dynamical systems. In particular, we shall devise algorithms for problems of optimization under uncertainty as well as RL. For problems of SO, one of our goals will be to devise algorithms that have lower bias in their gradient/Hessian estimators and which improve accuracy and convergence speed over others. We shall further develop algorithms for global optimization of nonlinear (non-convex) objectives where we shall build on local search algorithms by incorporating multiple initial starts as a first step. Towards this end, we shall also borrow ideas from discrete stochastic optimization as well as bandit algorithms. We shall devise RL algorithms for safe exploration as safety is a prime concern in many applications where training has to be done online on expensive hardware or real systems. We shall also devise algorithms for multi-agent systems in both cooperative and competitive settings as well as when the RL controllers are cooperative but arranged in a hierarchy. In all of this, we shall also provide sound theoretical analyses of the developed algorithms in addition to testing them on various settings. Finally, we shall study the applications of RL and SO algorithms in the domain of smart grids. Here we shall study problems of energy management, scheduling of flexible energy demand, grid voltage and frequency control, as well as of security in information exchange. This setting will be an important benchmark for testing many of our algorithms that will be developed.