Fine-Tuned CNNs with Self-Attention Mechanism for Enhanced Facial Expression Recognition
DOI:
https://doi.org/10.59461/ijdiic.v4i2.166Keywords:
Facial Emotions Recognition, Emotion recognition, Image analysis, CNN, Facial dynamicsAbstract
The growing need for facial emotion recognition in various domains, particularly in online education, has driven advancements in Artificial Intelligence (AI) and computer vision. Facial expressions are a vital source of nonverbal communication as they convey a wide range of emotions through subtle changes in facial features. Recent developments in Deep Learning (DL) and Convolutional Neural Networks (CNNs) have opened new avenues for analyzing and interpreting human emotions. This study proposes a novel CNN-based real-time facial expression recognition (FER) framework tailored for online education systems. The framework incorporates dynamic region attention and self-attention mechanisms, enabling the model to focus on key facial regions that vary in importance depending on emotional context. The proposed model is fine-tuned to enhance its capability to identify facial expressions in various situations by integrating these methods with transfer learning. Experimental results demonstrate that the model achieves an accuracy of 83% using FER 2013, surpassing traditional static image-based techniques. This study proposes to bridge the gap in facial expression observation in online education, facilitating educators with valuable visions into pupil sentiments to advance learning consequences.
Downloads
References
E. Pranav, S. Kamal, C. Satheesh Chandran, and M. H. Supriya, “Facial Emotion Recognition Using Deep Convolutional Neural Network,” in 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), IEEE, Mar. 2020, pp. 317–320. doi: 10.1109/ICACCS48705.2020.9074302.
W. Mellouk and W. Handouzi, “Facial emotion recognition using deep learning: review and insights,” Procedia Comput. Sci., vol. 175, pp. 689–694, 2020, doi: 10.1016/j.procs.2020.07.101.
C. Pramerdorfer and M. Kampel, “Facial Expression Recognition using Convolutional Neural Networks: State of the Art,” Dec. 2016. http://arxiv.org/abs/1612.02903
K. Sarvakar, R. Senkamalavalli, S. Raghavendra, J. Santosh Kumar, R. Manjunath, and S. Jaiswal, “Facial emotion recognition using convolutional neural networks,” Mater. Today Proc., vol. 80, pp. 3560–3564, 2023, doi: 10.1016/j.matpr.2021.07.297.
M. M. Taghi Zadeh, M. Imani, and B. Majidi, “Fast Facial emotion recognition Using Convolutional Neural Networks and Gabor Filters,” in 2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI), IEEE, Feb. 2019, pp. 577–581. doi: 10.1109/KBEI.2019.8734943.
W. Gu, C. Xiang, Y. V. Venkatesh, D. Huang, and H. Lin, “Facial expression recognition using radial encoding of local Gabor features and classifier synthesis,” Pattern Recognit., vol. 45, no. 1, pp. 80–91, Jan. 2012, doi: 10.1016/j.patcog.2011.05.006.
A. J. O’Toole and C. D. Castillo, “Face Recognition by Humans and Machines: Three Fundamental Advances from Deep Learning,” Annu. Rev. Vis. Sci., vol. 7, no. 1, pp. 543–570, Sep. 2021, doi: 10.1146/annurev-vision-093019-111701.
H. Y. Patil, A. G. Kothari, and K. M. Bhurchandi, “Expression invariant face recognition using local binary patterns and contourlet transform,” Optik (Stuttg)., vol. 127, no. 5, pp. 2670–2678, Mar. 2016, doi: 10.1016/j.ijleo.2015.11.187.
George, A., H. Wimme, and C.M. Rebman, “Artifical Intelligence Facial Expression Recognition for Emotion Detection: Performance and Acceptance,” Issues Inf. Syst., 2020, doi: 10.48009/4_iis_2020_81-91.
M. Liu, S. Li, S. Shan, and X. Chen, “AU-inspired Deep Networks for Facial Expression Feature Learning,” Neurocomputing, vol. 159, pp. 126–136, Jul. 2015, doi: 10.1016/j.neucom.2015.02.011.
Y. Gan, “Facial Expression Recognition Using Convolutional Neural Network,” in Proceedings of the 2nd International Conference on Vision, Image and Signal Processing, New York, NY, USA: ACM, Aug. 2018, pp. 1–5. doi: 10.1145/3271553.3271584.
P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller, and S. Zafeiriou, “End-to-End Multimodal Emotion Recognition Using Deep Neural Networks,” IEEE J. Sel. Top. Signal Process., vol. 11, no. 8, pp. 1301–1309, Dec. 2017, doi: 10.1109/JSTSP.2017.2764438.
M. Karnati, A. Seal, D. Bhattacharjee, A. Yazidi, and O. Krejcar, “Understanding Deep Learning Techniques for Recognition of Human Emotions Using Facial Expressions: A Comprehensive Survey,” IEEE Trans. Instrum. Meas., vol. 72, pp. 1–31, 2023, doi: 10.1109/TIM.2023.3243661.
Y. Liu and G. Fu, “Emotion recognition by deeply learned multi-channel textual and EEG features,” Futur. Gener. Comput. Syst., vol. 119, pp. 1–6, Jun. 2021, doi: 10.1016/j.future.2021.01.010.
R. Pathar, A. Adivarekar, A. Mishra, and A. Deshmukh, “Human Emotion Recognition using Convolutional Neural Network in Real Time,” in 2019 1st International Conference on Innovations in Information and Communication Technology (ICIICT), IEEE, Apr. 2019, pp. 1–7. doi: 10.1109/ICIICT1.2019.8741491.
P. Tarnowski, M. Kołodziej, A. Majkowski, and R. J. Rak, “Emotion recognition using facial expressions,” Procedia Comput. Sci., vol. 108, pp. 1175–1184, 2017, doi: 10.1016/j.procs.2017.05.025.
S. Minaee, M. Minaei, and A. Abdolrashidi, “Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network,” Sensors, vol. 21, no. 9, p. 3046, Apr. 2021, doi: 10.3390/s21093046.
K. P and A. T, “Group Facial Emotion Analysis System Using Convolutional Neural Network,” in 2020 4th International Conference on Trends in Electronics and Informatics (ICOEI)(48184), IEEE, Jun. 2020, pp. 643–647. doi: 10.1109/ICOEI48184.2020.9143037.
Y. Khaireddin and Z. Chen, “Facial Emotion Recognition: State of the Art Performance on FER2013,” May 2022. http://arxiv.org/abs/2105.03588
A. Bhandari and N. R. Pal, “Can edges help convolution neural networks in emotion recognition?,” Neurocomputing, vol. 433, pp. 162–168, Apr. 2021, doi: 10.1016/j.neucom.2020.12.092.
C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, “Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,” Artif. Intell. Rev., vol. 43, no. 2, pp. 155–177, Feb. 2015, doi: 10.1007/s10462-012-9368-5.
J. Cai, O. Chang, X.-L. Tang, C. Xue, and C. Wei, “Facial Expression Recognition Method Based on Sparse Batch Normalization CNN,” in 2018 37th Chinese Control Conference (CCC), IEEE, Jul. 2018, pp. 9608–9613. doi: 10.23919/ChiCC.2018.8483567.
M. H. Alkawaz, D. Mohamad, A. H. Basori, and T. Saba, “Blend Shape Interpolation and FACS for Realistic Avatar,” 3D Res., vol. 6, no. 1, p. 6, Mar. 2015, doi: 10.1007/s13319-015-0038-7.
E. Sariyanidi, H. Gunes, and A. Cavallaro, “Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 6, pp. 1113–1133, Jun. 2015, doi: 10.1109/TPAMI.2014.2366127.
M. Mohammadpour, H. Khaliliardali, S. M. R. Hashemi, and M. M. AlyanNezhadi, “Facial emotion recognition using deep convolutional networks,” in 2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI), IEEE, Dec. 2017, pp. 0017–0021. doi: 10.1109/KBEI.2017.8324974.
U. F. Ikwuanusi, “Real Time Classification of Facial Expressions for Effective and Intelligent Video Communication,” Electron. Theses Diss., p. 1076, 2023.
A. A. Pise et al., “Methods for Facial Expression Recognition with Applications in Challenging Situations,” Comput. Intell. Neurosci., vol. 2022, pp. 1–17, May 2022, doi: 10.1155/2022/9261438.
S. Li and W. Deng, “Deep Facial Expression Recognition: A Survey,” IEEE Trans. Affect. Comput., vol. 13, no. 3, pp. 1195–1215, Jul. 2022, doi: 10.1109/TAFFC.2020.2981446.
M. Long and Y. Zeng, “Detecting Iris Liveness with Batch Normalized Convolutional Neural Network,” Comput. Mater. Contin., vol. 58, no. 2, pp. 493–504, 2019, doi: 10.32604/cmc.2019.04378.
I. J. Goodfellow et al., “Challenges in Representation Learning: A Report on Three Machine Learning Contests,” 2013, pp. 117–124. doi: 10.1007/978-3-642-42051-1_16.
T. Connie, M. Al-Shabi, W. P. Cheah, and M. Goh, “Facial Expression Recognition Using a Hybrid CNN–SIFT Aggregator,” 2017, pp. 139–149. doi: 10.1007/978-3-319-69456-6_12.
J. Liao, Y. Lin, T. Ma, S. He, X. Liu, and G. He, “Facial Expression Recognition Methods in the Wild Based on Fusion Feature of Attention Mechanism and LBP,” Sensors, vol. 23, no. 9, p. 4204, Apr. 2023, doi: 10.3390/s23094204.
V. S. Amal, S. Suresh, and G. Deepa, “Real-Time Emotion Recognition from Facial Expressions Using Convolutional Neural Network with Fer2013 Dataset,” 2022, pp. 541–551. doi: 10.1007/978-981-16-3675-2_41.
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Rabika Khalid, Atta Ur Rahman, Sania Ali , Bibi Saqia
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
