A Temporal Approach to Facial Emotion Expression Recognition

Systems embedded with facial emotion expression recognition models enable the application of emotion-related knowledge to improve human and computer interaction and in doing so, users have a satisfying experience. Facial expressions exhibited by individuals are mostly used as non-verbal cues of communication. It is envisaged that accurate and real-time estimation of expressions and/or emotional changes will improve existing online platforms. However, further mapping of estimated expressions to emotions is highly useful in many applications such as sentiment analysis, market analysis, student comprehension among others. Feedback based on estimated emotions plays a crucial role in improving the usability of such models. However, there have been no or limited feedback mechanisms incorporated into these models. The proposed work, therefore, investigates the use of deep learning to identify and estimate emotional changes in human faces and further analysis of estimated emotions to provide feedback. The methodology involves a temporal approach including a VGG-19 pre-trained network for feature extraction, a BiLSTM architecture for facial emotion expression recognition, and mapping criteria to map estimated expressions and the resultant emotion (positive, negative, neutral). The CNN-BiLSTM model achieved an accuracy of 91% on a test set consisting of seven basic emotions of anger, disgust, fear, happy, surprise, sadness and neutral from the Denver Intensity of Spontaneous Facial Action (DISFA) data. The data set for affective States in E-Environment(DAiSEE) labeled with boredom, frustration, confusion, and engagement was used to further test the proposed model to estimate the seven basic expressions and re-evaluate the mapping model used for mapping expressions to emotions.