Emotion-Aware Digital Twin for a Large Language Model-Based Personalized Therapy Solution

Abstract

Mental health disorders are increasing worldwide, yet many individuals still face limited access to timely mental health support. At the same time, wearable devices such as smartwatches enable continuous collection of physiological signals that may provide useful indicators of affective states in everyday life. This thesis explores how wearable-based emotion recognition can be integrated with retrieval-augmented large language models (RAG-LLMs) and a digital twin framework to provide accessible, personalized, and context-aware mental well-being support anytime and anywhere. First, the thesis introduces WARM-VR, a new affective computing dataset designed to address limitations of existing resources by combining immersive stimuli with wearable physiological sensing and self-report. The dataset includes recordings from 31 participants and integrates multimodal signals with arousal, valence, and relaxation annotations to support reproducible affect research in more realistic settings. Second, the thesis develops and evaluates multiple deep learning architectures for emotion recognition from Photoplethysmography (PPG)-based wearables, including hybrid CNN–LSTM–TCN models as well as Transformer-based and Mamba-based approaches, with attention to noise, subject variability, and class imbalance. Third, the thesis investigates the use of Retrieval-Augmented Generation (RAG) to reduce hallucination and improve reliability in mental-health-oriented conversational systems, and introduces an evaluation framework that distinguishes between retrieval-grounded factual accuracy and therapist-like conversational support across different LLMs. Building on these components, the thesis proposes UbiMyTherapist (You Be My Therapist), a ubiquitous emotion-aware digital twin framework designed to operate continuously alongside the user by combining wearable-based emotion estimation, user history, and psychological knowledge bases, enabling reactive conversational support and proactive interventions. A proof-of-concept prototype is implemented using an agentic orchestration layer and a vector-based RAG pipeline grounded in Cognitive Behavioral Therapy (CBT) content. Finally, a user case study with 24 participants compares four system configurations: baseline GPT-4o, prompt-engineered GPT-4o, RAG-based GPT-4o, and the full digital twin prototype. The results from the pilot study show that integrating retrieval grounding with user context and emotional-state history improves perceived therapist-like conversation, empathy, personalization, and overall user experience.