Research Projects

A modern approach to building machine learning models involves training on offline data, followed by learning from online interactions. For example, recommender engines are initially built on offline logged data collected through beta testing on paid users. Then, after deployment, the systems are updated sequentially on data collected through online user interactions. Training models on user data, however, can risk exposing sensitive information, such as gender or income, which violates user privacy. While offline data is considered free of privacy concerns, online data poses significant privacy risks. The same issue applies to AI-powered chatbots, which are initially pre-trained on public demonstration data and then fine-tuned on private data from real-time user interactions. The fine-tuning step, using bandit optimization and reinforcement learning algorithms, can amplify potential breaches of users’ private data. Differential Privacy ensures that no adversary can extract too much information about a user’s data from a trained model. This is done by adding noise during training, but it can reduce the model's accuracy, as the noise perturbs underlying patterns in the dataset. Using offline public data, which poses no privacy risks, can help mitigate this issue by providing the agent with an initial dataset to start the learning algorithm. Although there has been recent progress in this direction for models trained with supervised learning, little is understood about models fine-tuned with online learning, including large language models. Our first objective is to develop differentially private bandit optimization and reinforcement learning algorithms when the learning agent has access to offline public data, and theoretically upper-bound their error rates. To characterize the performance of our offline data-assisted online optimization algorithms, we will design baselines using either pure offline or pure online strategies and compare those with our algorithms on practical recommender system datasets. The second objective is to design differentially private policy fine-tuning algorithms for aligning large language models with human feedback by leveraging offline demonstration data and assessing their empirical performance on widely used alignment tasks using open-source LLMs. On achieving the first objective, we will have a fundamental understanding of the optimal error rate of privacy-preserving bandit and RL optimization algorithms with side access to offline data. We will also uncover whether it is possible to achieve higher accuracy than purely offline and online baselines. On achieving the second objective, we will have a complete understanding of LLM alignment algorithms that do not compromise the privacy of chatbot users. We will also have a prototype of a privacy-preserving AI chatbot, which will pave the way for developing private, trustworthy, and safe AI systems.