Affects Analysis: A Temporal Approach to Estimate Students' Learning

The dissimilarity between real and virtual environments is getting closer rapidly. More people are working on computers to accomplish a variety of tasks. Tasks like shopping, banking, learning, teaching, etc. are all being carried out online. The education sector is one of the sectors that has adapted to the needs and obstacles created by the impossibility to present a face to face sessions, which was necessitated because of the global pandemic. These obstacles have led to more options for online teaching and learning. Unfortunately, the underlying experiences, feelings, emotions, and moods of students during learning are not adequately captured by teachers, unlike in the classroom environment where a teacher can observe students through visual and audio cues such as eye gaze, body language, facial expressions, speech, etc. Facial emotion expressions recognition during online learning has been a subject of research in the field of machine learning, but most have not adequately considered analyzing a student's affect states with the possibility of providing appropriate feedback. This work, therefore, intends to contribute to the existing knowledge by proposing a temporal approach to estimate a student's learning affects. A model, that comprises of a video input of facial emotion expressions, feature extraction using ResNet-50 pretrained CNN model, and a bi-directional Long Short-Term Memory classifier to detect and classify affect states of boredom, confusion, frustration, and engagement for estimating student's learning is proposed. The baseline model was trained, validated, and tested using the DAiSEE dataset and shows a test accuracy of 88% on 6546 test samples. The work further maps the affect states into positive and negative learning affects. It is anticipated that such a model will serve as feedback to the teachers and enhance the estimation of the students' learning, especially during e-learning sessions.