Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Facial Expression Recognition Method Using Two-Stream Convolutional Networks in Natural Scenes

  • Received : 2020.09.04
  • Accepted : 2020.11.08
  • Published : 2021.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

Aiming at the problem that complex external variables in natural scenes have a greater impact on facial expression recognition results, a facial expression recognition method based on two-stream convolutional neural network is proposed. The model introduces exponentially enhanced shared input weights before each level of convolution input, and uses soft attention mechanism modules on the space-time features of the combination of static and dynamic streams. This enables the network to autonomously find areas that are more relevant to the expression category and pay more attention to these areas. Through these means, the information of irrelevant interference areas is suppressed. In order to solve the problem of poor local robustness caused by lighting and expression changes, this paper also performs lighting preprocessing with the lighting preprocessing chain algorithm to eliminate most of the lighting effects. Experimental results on AFEW6.0 and Multi-PIE datasets show that the recognition rates of this method are 95.05% and 61.40%, respectively, which are better than other comparison methods.

Keywords

References

  1. N. P. Gopalan, S. Bellamkonda, and V. S. Chaitanya, "Facial expression recognition using geometric landmark points and convolutional neural networks," in Proceedings of 2018 International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, 2018, pp. 1149-1153.
  2. Y. He and X. He, "Facial expression recognition based on multi-feature fusion and HOSVD," in Proceedings of 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu, China, 2019, pp. 638-643. https://doi.org/10.1109/ITNEC.2019.8729003
  3. S. Wang, B. Pan, H. Chen, and Q. Ji, "Thermal augmented expression recognition," IEEE Transactions on Cybernetics, vol. 48, no. 7, pp. 2203-2214, 2018. https://doi.org/10.1109/TCYB.2017.2786309
  4. J. Ueda and K. Okajima, "Face morphing using average face for subtle expression recognition," in Proceedings of 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), Dubrovnik, Croatia, 2019, pp. 187-192. https://doi.org/10.1109/ISPA.2019.8868931
  5. N. Song, H. Yang, and P. Wu, "A gesture-to-emotional speech conversion by combining gesture recognition and facial expression recognition," in Proceedings of 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), Beijing, China, 2018, pp. 1-6. https://doi.org/10.1109/ACIIAsia.2018.8470350
  6. B. K. Kim, J. Roh, S. Y. Dong, and S. Y. Lee, "Hierarchical committee of deep convolutional neural networks for robust facial expression recognition," Journal on Multimodal User Interfaces, vol. 10, no. 2, pp. 173-189, 2016. https://doi.org/10.1007/s12193-015-0209-0
  7. G. Pons and D. Masip, "Supervised committee of convolutional neural networks in automated facial expression analysis," IEEE Transactions on Affective Computing, vol. 9, no. 3, pp. 343-350, 2018. https://doi.org/10.1109/TAFFC.2017.2753235
  8. L. Chen, M. Zhou, W. Su, M. Wu, J. She, and K. Hirota, "Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction," Information Sciences, vol. 428, pp. 49- 61, 2018. https://doi.org/10.1016/j.ins.2017.10.044
  9. P. Liu, S. Han, Z. Meng, and Y. Tong, "Facial expression recognition via a boosted deep belief network," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, 2014, pp. 1805-1812. https://doi.org/10.1109/CVPR.2014.233
  10. P. Hu, D. Cai, S. Wang, A. Yao, and Y. Chen, "Learning supervised scoring ensemble for emotion recognition in the wild," in Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK, 2017, pp. 553-560.
  11. Y. Fan, J. C. Lam, and V. O. Li, "Video-based emotion recognition using deeply-supervised neural networks," in Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, 2018, pp. 584-588.
  12. S. Eleftheriadis, O. Rudovic, and M. Pantic, "Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition," IEEE Transactions on Image Processing, vol. 24, no. 1, pp. 189-204, 2014. https://doi.org/10.1109/TIP.2014.2375634