Adaptive Reasoning: An Affect Related Feedback Approach for Enhanced E-Learning

Recognition of affective states to enhance e-learning platforms has been a topic of machine learning research. Compared to other input modalities, facial expressions have the potential to reveal nonverbal cues about a learner’s learning affect. However, most studies were limited in their analysis of learning affects exhibited by a learner with the possibility of providing appropriate feedback to teachers and learners. This work proposes an adaptive reasoning mechanism that considers the estimated affective states and learning affect in generating feedback with reasoning incorporated. This work utilizes a Convolutional Neural Network- Bidirectional Long-Short Term Memory (CNN-BiLSTM) cascade framework for affective states analysis through processing a live/stored observation of a learner in the form of a temporal signal. Using the proposed ensemble, four affective states were estimated, namely boredom, confusion, frustration, and engagement. Dataset for Affective States in E-Environment (DAiSEE) was used to train, validate, and test the baseline model, which reported an accuracy of 86% on 4305 test samples. In the next stage, mappings between estimated affective states and learning affects (i.e. positive, negative and neutral) were established based on an adaptive mapping mechanism, to consolidate the mapping between affective states and learning affects. Live testing and survey feedback were then used to further validate, adapt and amend the feedback process. Incorporating and interpreting the estimated affective states and learning affect is imperative in providing information to both teachers and learners, and hence potentially improve the existing e-learning platforms.