Facial emotion recognition using temporal relational network: an application to E-learning

Anil Pise, Hima Vadapalli, Ian Sanders

Research output: Contribution to journalArticlepeer-review

18 Citations (Scopus)


E-learning enables the dissemination of valuable academic information to all users regardless of where they are situated. One of the challenges faced by e-learning systems is the lack of constant interaction between the user and the system. This observability feature is an essential feature of a typical classroom setting and a means of detecting or observing feature reactions and thus such features in the form of expressions should be incorporated into an e-learning platform. The proposed solution is the implementation of a deep-learning-based facial image analysis model to estimate the learning affect and to reflect on the level of student engagement. This work proposes the use of a Temporal Relational Network (TRN), for identifying the changes in the emotions on students’ faces during e-learning session. It is observed that TRN sparsely samples individual frames and then learns their causal relations, which is much more efficient than sampling dense frames and convolving them. In this paper, single-scale and multi-scale temporal relations are considered to achieve the proposed goal. Furthermore, a Multi-Layer Perceptron (MLP) is also tested as a baseline classifier. The proposed framework is end-to-end trainable for video-based Facial Emotion Recognition (FER). The proposed FER model was tested on the open-source DISFA+ database. The TRN based model showed a significant reduction in the length of the feature set which were effective in recognizing expressions. It is observed that the multi-scale TRN has produced better accuracy than the single-scale TRN and MLP with an accuracy of 92.7%, 89.4%, and 86.6% respectively.

Original languageEnglish
Pages (from-to)26633-26653
Number of pages21
JournalMultimedia Tools and Applications
Issue number19
Publication statusPublished - Aug 2022
Externally publishedYes


  • Deep learning
  • E-learning
  • Learning affect
  • Relational reasoning
  • Segmentation
  • Temporal relational network

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Facial emotion recognition using temporal relational network: an application to E-learning'. Together they form a unique fingerprint.

Cite this