Research Projects

Rationale: Recent advances in brain-computer interfaces (BCIs) have enabled non-motor-based communication, with covert speech recognition as a key breakthrough. However, spoken words alone often lack emotional context, which is essential for meaningful communication. Integrating covert speech with emotion recognition can enhance human-like interactions, particularly for clinical and robotic applications. This study aims to develop a multi-modal BCI system by combining covert speech recognition and emotion analysis using hybrid EEG-fNIRS signals, leveraging EEG’s temporal precision and fNIRS’s spatial accuracy. It aims to improve real-time communication and emotional understanding in emergency care and clinical settings, especially for patients with motor speech disorders. Objectives: • Phase 1: Further develop and test a deep learning-based covert speech recognition system for 15 essential English words using EEG-fNIRS signals (Ongoing under DST-ICMR funding). • Phase 2: Develop an emotion recognition system for six primary emotions (happiness, sadness, anger, fear, surprise, neutrality) using EEG-fNIRS signals and culturally relevant stimuli from the Affective Film Dataset from India (AFDI). • Phase 3: Evaluate the system's feasibility and accuracy in patients with motor speech disorders. Methodology: • Phase 1 (Ongoing): Data collection and analysis of EEG-fNIRS signals from 75 participants performing covert articulation tasks. • Phase 2: o Design: Cross-sectional study with 75 healthy adults (18–40 years, MoCA > 26). o Setup: EEG-fNIRS recordings during emotion-inducing videos (AFDI dataset). o Procedure: Baseline signals, video-triggered emotions, signal processing, and CNN-based classification; accuracy validated using subject-specific and generalization methods. • Phase 3: o Design: Analytical study on 11 patients with motor speech disorders and 11 controls. o Procedure: Covert speech and emotion tasks while recording EEG-fNIRS signals; metrics include accuracy, precision, F1-score, and AUC. • Outcomes: o Development of a real-time, non-invasive BCI system capable of decoding covert speech and emotions. o Generation of an EEG-fNIRS dataset relevant to the Indian context. o Advancement of BCI research for healthcare, robotics, and affective computing applications. This study bridges critical gaps in communication technology, providing tools for empathetic clinical care and enabling human-like interactions in robotics and AI. Combining speech and emotion decoding enhances communication for individuals with speech impairments and paves the way for advanced neurotechnological solutions