No Access

Facial Emotion Recognition Using an Ensemble of Multi-Level Convolutional Neural Networks

Hai-Duong Nguyen

School of Electronics and Computer Engineering, Chonnam National University, Gwangju, South Korea

E-mail Address: nhduong_3010@live.com

Search for more papers by this author

Soonja Yeom

School of Engineering and ICT, University of Tasmania, Hobart, Australia

E-mail Address: soonja.yeom@utas.edu.au

Search for more papers by this author

Guee-Sang Lee

School of Electronics and Computer Engineering, Chonnam National University, Gwangju, South Korea

E-mail Address: gslee@jnu.ac.kr

Search for more papers by this author

Hyung-Jeong Yang

School of Electronics and Computer Engineering, Chonnam National University, Gwangju, South Korea

E-mail Address: hjyang@jnu.ac.kr

Search for more papers by this author

In-Seop Na

Software Convergence Education Institute, Chosun University, Gwangju, South Korea

E-mail Address: ypencil@chosun.ac.kr

Search for more papers by this author

, and

Soo-Hyung Kim

http://orcid.org/0000-0003-3575-5035

School of Electronics and Computer Engineering, Chonnam National University, Gwangju, South Korea

E-mail Address: shkim@jnu.ac.kr

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0218001419400159Cited by:43 (Source: Crossref)

This article is part of the issue:

Special Issue on the International Conference on Pattern Recognition and Artificial Intelligence (ICPRAI 2018)
Guest Editors: Ching Yee Suen (Concordia University, Canada), Pong C. Yuen (Hong Kong Baptist University), Adam Krzyzak (Concordia University, Canada), Simone Marinai (Università degli Studi di Firenze, Italy) and Patrick S. P. Wang (Northeastern University, USA)

Abstract

Emotion recognition plays an indispensable role in human–machine interaction system. The process includes finding interesting facial regions in images and classifying them into one of seven classes: angry, disgust, fear, happy, neutral, sad, and surprise. Although many breakthroughs have been made in image classification, especially in facial expression recognition, this research area is still challenging in terms of wild sampling environment. In this paper, we used multi-level features in a convolutional neural network for facial expression recognition. Based on our observations, we introduced various network connections to improve the classification task. By combining the proposed network connections, our method achieved competitive results compared to state-of-the-art methods on the FER2013 dataset.

Keywords:

References

1. M. Al-Shabi, W. P. Cheah and T. Connie , Facial expression recognition using a hybrid CNN-SIFT aggregator, Int. Workshop on Multi-Disciplinary Trends in Artificial Intelligence (Springer, Gadong, Brunei, 2017), pp. 139–149. Google Scholar
2. A. Breeze, Alastair breeze computer scientist portfolio and blog (2016), https://www.alastairbreeze.com/experiments/machine-learning/facial-keypoint-detection/. Google Scholar
3. P. Carrier and A. Courville, Challenges in representation learning: Facial expression recognition challenge (2013), https://goo.gl/kVzT48. Google Scholar
4. F. Chollet et al., Keras (2015), https://github.com/fchollet/keras. Google Scholar
5. P. Ekman and W. V. Friesen , Constants across cultures in the face and emotion, J. Pers. Social Psychol. 17(2) (1971) 124–129. Web of Science, Google Scholar
6. P. Ekman and W. V. Friesen, Facs - facial action coding system (1977), https://www.cs.cmu.edu/ face/facs.htm. Google Scholar
7. M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman , The pascal visual object classes challenge: A retrospective, Int. J. Comput. Vis. 111(1) (2015) 98–136. Web of Science, Google Scholar
8. K. He, X. Zhang, S. Ren and J. Sun , Deep residual learning for image recognition, in Int. Conf. Comput. Vis. Pattern Recognition (CVPR) (Las Vegas, USA, 2016), pp. 770–778. Google Scholar
9. G. Huang, Z. Liu, L. van der Maaten and K. Q. Weinberger , Densely connected convolutional networks, Int. Conf. Computer Vision and Pattern Recognition (CVPR) (Honolulu, Hawaii, USA, 2017), pp. 4700–4708. Google Scholar
10. B. K. Kim, J. Roh, S. Y. Dong and S. Y. Lee , Hierarchical committee of deep convolutional neural networks for robust facial expression recognition, J. Multimodal User Interfaces 10(2) (2016). Web of Science, Google Scholar
11. A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images, technical report, University of Toronto (2009). Google Scholar
12. A. Krizhevsky, I. Sutskever and G. E. Hinton , Imagenet classification with deep convolutional neural networks, Int. Conf. Neural Information Processing Systems (NIPS) (Lake Tahoe, Nevada, USA, 2012), pp. 1097–1105. Google Scholar
13. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner , Gradient-based learning applied to document recognition, Proc. IEEE 86(11) (1998) 2278–2324. Web of Science, Google Scholar
14. H. Lee, R. Grosse, R. Ranganath and A. Y. Ng , Unsupervised learning of hierarchical sentations with convolutional deep belief networks, Commun. ACM 54(10) (2011) 95–103. Web of Science, Google Scholar
15. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews , The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, Int. Conf. Computer Vision and Pattern Recognition (CVPR) - Workshops (San Francisco, CA, USA, 2010), pp. 94–101. Google Scholar
16. S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh and J. F. Cohn , Disfa: A spontaneous facial action intensity database, IEEE Trans. Affective Comput. 4(2) (2013) 151–160. Web of Science, Google Scholar
17. D. McDuff, R. el Kaliouby, T. Senechal, M. Amr, J. F. Cohn and R. Picard , Affectiva-mit facial expression dataset (am-fed): Naturalistic and spontaneous facial expressions collected “ in-the-wild”, Int. Conf. Computer Vision and Pattern Recognition (CVPR) - Workshops (Portland, Oregon, USA, 2013), pp. 881–888. Google Scholar
18. A. Mollahosseini, D. Chan and M. H. Mahoor , Going deeper in facial expression recognition using deep neural networks, Winter Conf. Applications of Computer Vision (WACV) (Lake Placid, NY, USA, 2016), pp. 1–10. Google Scholar
19. M. Pantic, M. Valstar, R. Rademaker and L. Maat , Web-based database for facial expression analysis, Int. Conf. Multimedia and Expo (Amsterdam, Netherlands, 2005), pp. 317–321. Google Scholar
20. K. Raghavendra et al., keras-vis (2017), https://github.com/raghakot/keras-vis. Google Scholar
21. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg and L. Fei-Fei , Imagenet large scale visual recognition challenge, Int. J. Comput. Vis. 115(3) (2015) 211–252. Web of Science, Google Scholar
22. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh and D. Batra , Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization, Int. Conf. Computer Vision (ICCV) (Venice, Italy, 2017), pp. 618–626. Google Scholar
23. C. Shan, S. Gong and P. W. McOwan , Facial expression recognition based on local binary patterns: A comprehensive study, Image Vision Comput. 27(6) (2009) 803–816. Web of Science, Google Scholar
24. K. Simonyan and A. Zisserman , Very deep convolutional networks for large-scale image recognition, Int. Conf. Learning Representations (ICLR) (San Diego, California, USA, 2015), pp. 1–14. Google Scholar
25. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov , Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15(1) (2014) 1929–1958. Google Scholar
26. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich , Going deeper with convolutions, Int. Conf. Computer Vision and Pattern Recognition (CVPR) (Boston, Massachusetts, USA, 2015), pp. 1–9. Google Scholar
27. Y. Tang , Deep learning using linear support vector machines, Workshop on Challenges in Representation Learning, International Conference on Machine Learning (ICML) (Atlanta, USA, 2013). Google Scholar
28. Y. Tian, T. Kanade and J. F. Cohn , Recognizing action units for facial expression analysis, IEEE Trans. Pattern Analysis Mach. Intell. 23(2) (2001) 97–115. Web of Science, Google Scholar
29. J. Wang and C. Yuan , Facial expression recognition with multi-scale convolution neural network, in Pacific-Rim Conf. Adv. Multimedia Information Processing (Xi'an, China, 2016), pp. 376–385. Google Scholar
30. Z. Yu and C. Zhang , Image based static facial expression recognition with multiple deep network learning, Int. Conf. Multimodal Interaction (ICMI) (Seattle, Washington, USA, 2015), pp. 435–442. Google Scholar
31. X. Zhang, M. H. Mahoor and S. M. Mavadati , Facial expression recognition using $l_{p}$ -norm mkl multiclass-svm, Mach. Vis. Appl. 26 (2015) 467–483. Web of Science, Google Scholar
32. S. Zhou, Y. Liang, J. Wan and S. Z. Li , Facial expression recognition based on multi-scale cnns, in Chin. Conf. Biometric Recognition (Chengdu, China, 2016), pp. 503–510. Google Scholar