Research Projects

The aim of the proposed project is to advance the frameworks and techniques for developing real-world robot control. Robotics is a discipline that aims to build machines that automate human-like physical tasks such as object manipulation, locomotion, etc. It has been a major field of study in engineering and has been instrumental in automating applications such as manufacturing. The use cases for robotics extend beyond manufacturing to delivery, surveillance, search and rescue, crisis management, space, etc. Thus, robotics has a wide array of applications of relevance. Thus, this project aims to advance the design techniques for control and at the same time advance the state-of-the-art of AI in robotics. Specifically, the project aims to develop general-purpose AI models akin to large language models that are tunable to specific robotic tasks with minimal effort. Tuning a model to carry out reasoning, planning and the low-level control for any given task is cumbersome and typically requires multiple iterations. The proposed project aims to address these limitations by developing AI models and methods that will simplify the application of AI to the physical world and robotic tasks akin to Large Language Models in language processing. The project aims to use a hybrid approach: Vision Language Model (VLM) for reasoning and planning and a low-level controller for controlling the dynamics of the robot. The proposal is inspired by human physiology where the VLM is akin to the human brain that plans at the high level and the low-level controller is akin to the motor control in a human. The viability of the proposal as a pathway for general robotic intelligence stems from the fact that our proposed flow of control is similar in construct to how control flows in humans. The VLM will be integrated with a low-level controller where the VLM will play the role of a high-level planner, and the lower-level controller will play the role of the follower of the plan that executes the plan. We will test two approaches, one where a VLM is directly integrated with a low-level controller that is trained to execute the plan of the VLM and second where the VLM is co-trained with the low-level controller. Here, the aim is to leverage the implicit intelligence and world knowledge of LMs. The project aims to address the following challenges pertinent to developing such models: How to ground the LM in robotic reasoning? How to ensure the LM in conjunction with the low-level controller is reliable and does not hallucinate in robotic planning and execution? How to ensure performance is consistent? How to distill the knowledge into a smaller model that can be embedded in robot hardware? The training and initial testing will be carried out on virtual environments. Once the methods are validated on simulation, the final testing will be carried out on real robots. The project will lead to methods for developing models with general robotic intelligence.

Artificial Intelligence, Sensorimotor intelligence, Robotics, Deep Reinforcement Learning