Research Projects

Deep Reinforcement Learning (RL) has been widely successful in handling complex sequential decision-making tasks by integrating neural networks with RL methodologies. Nevertheless, training these deep RL models is often computationally demanding and requires careful tuning of hyperparameters. Transfer learning has emerged as a powerful strategy to mitigate these challenges by allowing previously acquired knowledge to be reused in related tasks, reducing the need to train models entirely from scratch. For example, imagine a household robot trained to transport an object to a designated spot. If the target location shifts slightly, the robot’s original policy may not perform well in this altered scenario. Rather than retraining the model completely, it would be more practical to build upon the insights from the pre-trained model. Additionally, it is generally preferable to use a faster, more computationally efficient model to adjust to this new task setting for faster adaptability. One effective approach to transfer learning in RL is to utilize the internal representations formed within neural networks, particularly at the output of the last hidden layer. These activations serve as refined representations of the state, capturing essential input features. This enables RL agents to generalize and adapt their learned behaviours across tasks with similar structures. For instance, in the household robot example, these learned representations from initial training could encapsulate spatial or object-related information, allowing the robot to adapt effectively to new task variations using simpler models. However, in standard deep RL models, state representations often exhibit high correlation, which reduces their suitability for transfer learning, especially when adapting to lightweight models like linear function approximators. To bridge this gap, we aim to build a novel training paradigm designed not only to optimize the primary task but also to yield state representations that are inherently suited for transfer. Although representation learning has been explored in the literature for improving learning speed and stability, limited attention has been given to its specific role in transfer learning. Our project aims to formulate new loss functions for RL algorithms, with Deep Q-Networks (DQN) in particular, by imposing additional constraints on state representations. These constraints will promote distinctive and optimal state representations that improve transferability across tasks. To implement this, we will employ two-timescale stochastic approximation methods, where the final layer weights are updated at a faster timescale, while the state representations adjust at a slower pace, ensuring smoother convergence. We will then apply this adaptive framework to standard RL benchmarks, including Open AI Gym environments, Atari games and real-life household robotic tasks, to validate its robustness and efficiency in enhancing transfer learning outcomes.