Affective State and Pain Estimation Through Facial Emotion Analysis

Emotional state analysis is essential for understanding human emotions and their influence on health and well-being. Conventional methods rely on static models, often failing to account for the dynamic nature of human emotions and their context-dependent characteristics. This study proposes a method for estimating a patient’s emotional state using their facial expressions by utilising a CNN-BiLSTM cascade to classify facial expressions in real-time. This study uses the Extended Denver Intensity of Spontaneous Facial Action (DISFA+) and Extended Cohn-Kanade (CK+) datasets for experimentation. The emotion estimation model reported an accuracy of 85% on a sample size of 2,425 (DISFA+) annotated with seven basic emotions and further reported an 80% accuracy on CK+ dataset annotated with eight emotions. Estimated emotions are further mapped into emotional state estimates (comfortable vs uncomfortable categories), utilizing the mapping present in the literature as a way to monitor and interpret patients’ emotional states during online consultations. The study additionally evaluated the models using the UNBC McMaster pain datasets. DISFA+ model classified 18% of unlabelled samples as anger, 70% as disgust, and 10% as sadness. In contrast, the CK+ model classified 50% as anger, 40% as disgust, and 5% as sadness. This research highlights the strong correlation between facial expressions of anger and disgust in individuals experiencing pain. The study contributes to the field of affective computing in healthcare by improving the assessment of emotional states and pain.