Estimating Students’ Learning Affects: An Approach Based on the Recognition of Facial Emotion Expressions

Online education has experienced rapid development owing to its significance as a potential solution to teaching and learning under the critical conditions caused by Covid-19. A major obstacle in this form of learning is that online classes lack direct, timely and effective communication and feedback to teachers. The use of machine learning algorithms for estimating facial emotion expressions of students during teaching sessions has garnered interest from researchers in the last decade; however, there has been no or limited feedback mechanisms incorporated into these models. In the present study, we explored the use of deep learning to identify emotional changes in students’ faces and use them to estimate the learning affect experienced by the students. This involved implementing a CNN-BiLSTM model for emotion expression recognition and mapping of identified emotions into positive, negative and neutral learning affects, and for further affect analysis. This model was trained, validated and tested by using the extended DISFA database. A test accuracy of 92 per cent on a sample size of 2 274 was reported. The classified emotions were then mapped into learning affects, based on mappings provided in the literature. The model was further tested on live samples (collected in a laboratory set-up) to ascertain the validity of the mappings. It is envisaged that the analysis of the learning affects through facial emotion changes will potentially pave the way for timely and appropriate feedback to teachers on the learning affect experienced by students, potentially improving the feedback mechanism of the existing e-learning platforms.