ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Robust Facial Expression Recognition Based on Local Directional Pattern

  • Jabid, Taskeed (Department of Computer Engineering, Kyung Hee University) ;
  • Kabir, Md. Hasanul (Department of Computer Engineering, Kyung Hee University) ;
  • Chae, Oksam (Department of Computer Engineering, Kyung Hee University)
  • Received : 2010.03.15
  • Accepted : 2010.08.02
  • Published : 2010.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Automatic facial expression recognition has many potential applications in different areas of human computer interaction. However, they are not yet fully realized due to the lack of an effective facial feature descriptor. In this paper, we present a new appearance-based feature descriptor, the local directional pattern (LDP), to represent facial geometry and analyze its performance in expression recognition. An LDP feature is obtained by computing the edge response values in 8 directions at each pixel and encoding them into an 8 bit binary number using the relative strength of these edge responses. The LDP descriptor, a distribution of LDP codes within an image or image patch, is used to describe each expression image. The effectiveness of dimensionality reduction techniques, such as principal component analysis and AdaBoost, is also analyzed in terms of computational cost saving and classification accuracy. Two well-known machine learning methods, template matching and support vector machine, are used for classification using the Cohn-Kanade and Japanese female facial expression databases. Better classification accuracy shows the superiority of LDP descriptor against other appearance-based feature descriptors.

Keywords

References

  1. Y.L. Tian et al., "Real World Real-time Automatic Recognition of Facial Expressions," Proc. IEEE Workshop Performance Evaluation of Tracking and Surveillance, 2003.
  2. C. Shan, S. Gong, and P.W. McOwan, "Robust Facial Expression Recognition using Local Binary Patterns," Proc. IEEE Int. Conf. Image Process., 2005, pp. 914-917.
  3. M.Z. Uddin, J.J. Lee, and T.S. Kim, "An Enhanced Independent Component-Based Human Facial Expression Recognition from Video," IEEE Trans. Consum. Electron., vol. 55, no. 4, 2009, pp. 2216-2224. https://doi.org/10.1109/TCE.2009.5373791
  4. M.C. Hwang et al., "Person Identification System for Future Digital TV with Intelligence," IEEE Trans. Consum. Electron., vol. 53, no. 1, 2007, pp. 218-226. https://doi.org/10.1109/TCE.2007.339528
  5. P. Corcoran et al., "Biometric Access Control for Digital Media Streams in Home Networks," IEEE Trans. Consum. Electron., vol. 53, no. 3, 2007, pp. 917-925. https://doi.org/10.1109/TCE.2007.4341566
  6. M. Pantic and L.J.M. Rothkrantz, "Automatic Analysis of Facial Expressions: The State of the Art," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, 2000, pp. 1424-1445. https://doi.org/10.1109/34.895976
  7. B. Fasel and J. Luettin, "Automatic Facial Expression Analysis: A Survey," Pattern Recog., vol. 36, no. 1, 2003, pp. 259-275. https://doi.org/10.1016/S0031-3203(02)00052-3
  8. G. Zhao and M. Pietikainen, "Boosted Multi-resolution Spatio- Temporal Descriptors for Facial Expression Recognition," Pattern Recognit. Lett., vol. 30, no. 12, 2009, pp. 1117-1127. https://doi.org/10.1016/j.patrec.2009.03.018
  9. C. Shan, S. Gong, and P.W. McOwan, "Facial Expression Recognition based on Local Binary Patterns: A Comprehensive Study," Image Vision Comput., vol. 27, no. 6, May 2009, pp. 803- 816. https://doi.org/10.1016/j.imavis.2008.08.005
  10. K.H. Choi et al., "A Probabilistic Network for Facial Feature Verification," ETRI J., vol. 25, no. 2, Apr. 2003, pp. 140-143. https://doi.org/10.4218/etrij.03.0202.0205
  11. K.H. Kim et al., "Facial Feature Extraction Based on Private Energy Map in DCT Domain," ETRI J., vol. 29, no. 2, Apr. 2007, pp. 243-245. https://doi.org/10.4218/etrij.07.0206.0218
  12. Y. Tian, T. Kanade, and J.F. Cohn, "Facial Expression Analysis," Handbook of Face Recognition, Springer, Oct. 2003.
  13. Z. Zhang et al., "Comparison between Geometry-Based and Gabor-wavelets-based Facial Expression Recognition Using Multi-layer Perceptron," Proc. IEEE Int. Conf. Auto. Face Gesture Recog., Apr. 1998, pp. 454-459.
  14. P. Ekman and W. Friesen, Facial Action Coding System: A Technique for Measurement of Facial Movement, Consulting Psychologists Press, 1978.
  15. M. Valstar, I. Patras, and M. Pantic, "Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data," IEEE CVPR Workshop, vol. 3, 2005, pp. 76-84.
  16. M. Valstar and M. Pantic, "Fully Automatic Facial Action Unit Detection and Temporal Analysis," IEEE CVPR Workshop, June 2006, p. 149.
  17. M.A. Turk and A.P. Pentland, "Face Recognition Using Eigenfaces," Proc. Comput. Vision Pattern Recog., 1991, pp. 586-591.
  18. C. Padgett and G. Cottrell, "Representation Face Images for Emotion Classification," Advances Neural Inf. Process. Syst., vol. 9, Cambridge, MA, MIT Press, 1997.
  19. M.S. Bartlett, J.R. Movellan, and T.J. Sejnowski, "Face Recognition by Independent Component Analysis," IEEE Trans. Neural Networks, vol. 13, no. 6, 2002, pp. 1450-1464. https://doi.org/10.1109/TNN.2002.804287
  20. C.C. Fa and F.Y. Shin, "Recognizing Facial Action Units using Independent Component Analysis and Support Vector Machine," Pattern Recog., vol. 39, no. 9, 2006, pp. 1795-1798. https://doi.org/10.1016/j.patcog.2006.03.017
  21. M.J. Lyons, J. Budynek, and S. Akamatsu, "Automatic Classification of Single Facial Images," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 12, 1999, pp. 1357-1362. https://doi.org/10.1109/34.817413
  22. G. Donato et al., "Classifying Facial Actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, 1999, pp. 974-989. https://doi.org/10.1109/34.799905
  23. T. Ojala and M. Pietikainen, "Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, 2002, pp. 971-987. https://doi.org/10.1109/TPAMI.2002.1017623
  24. X. Feng, M. Pietikainen, and A. Hadid, "Facial Expression Recognition with Local Binary Patterns and Linear Programming," Pattern Recog. Image Anal., vol. 15, no. 2, 2005, pp. 546-548.
  25. G. Zhao and M. Pietikainen, "Dynamic Texture Recognition using Local Binary Patterns with An Application to Facial Expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, 2007, pp. 915-928. https://doi.org/10.1109/TPAMI.2007.1110
  26. H. Zhou, R. Wang, and C. Wang, "A Novel Extended Local Binary Pattern Operator for Texture Analysis," Inf. Science, vol. 178, no. 22, 2008, pp. 4314-4325. https://doi.org/10.1016/j.ins.2008.07.015
  27. T. Jabid, M.H. Kabir, and O.S. Chae, "Local Directional Pattern (LDP) for Face Recognition," IEEE Int. Conf. Consum. Electron., 2010, pp. 329-330.
  28. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive Database for Facial Expression Analysis," IEEE Int. Conf. Autom. Face Gesture Recog., Mar. 2000, pp. 46-53.
  29. S. Zhao, Y. Gao, and B. Zhang, "Sobel-LBP," Int. Conf. Image Process., 2008, pp. 2144-2147.
  30. R. Mattivi and L. Shao, "Human Action Recognition Using LBPTOP as Sparse Spatio-Temporal Feature Descriptor," Int. Conf. Comput. Anal. Image Pattern, 2009, pp. 740-747.
  31. W.K. Pratt, Digital Image Processing, Wiley, New York, 1978.
  32. S.W. Lee, "Off-line Recognition of Totally Unconstrained Handwritten Numerals Using Multilayer Cluster Neural Network," IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 6, 1996, pp. 648-652. https://doi.org/10.1109/34.506416
  33. T. Jabid, M.H. Kabir, and O.S. Chae, "Local Directional Pattern (LDP): A Robust Image Descriptor for Object Recognition," IEEE Int. Conf. Adv. Video and Signal-Based Surveillance, 2010, pp. 482-487.
  34. D. Lowe, "Distinctive Image Features from Scale Invariant Key Points," Int. J. Comput. Vision, vol. 60, no. 2, 2004, pp. 91-110. https://doi.org/10.1023/B:VISI.0000029664.99615.94
  35. T. Ahonen, A. Hadid, and M. Pietikainen, "Face Description with Local Binary Patterns: Application to Face Recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 12, 2006, pp. 2037-2041. https://doi.org/10.1109/TPAMI.2006.244
  36. S. Gundimada and V.K. Asari, "Facial Recognition Using Multisensor Images Based on Localized Kernel Eigen Spaces," IEEE Trans. Image Process., vol. 18, no. 6, 2009, pp. 1314-1325. https://doi.org/10.1109/TIP.2009.2016713
  37. C.A. Kumar, "Analysis of Unsupervised Dimensionality Reduction Techniques," Comput. Sci. Inf. Syst., vol. 6, no. 2, Dec. 2009, pp. 217-227. https://doi.org/10.2298/CSIS0902217K
  38. Y. Freund and R.E. Schapire, "A Decision-Theoretic Generalization of On-line Learning and an Application to Boosting," Computational Learning Theory, 1995, pp. 23-37.
  39. R.E. Schapire and Y. Singer, "Improved Boosting Algorithms using Confidence-Rated Predictions," Maching Learning, vol. 37, no.3, 1999, pp. 297-336. https://doi.org/10.1023/A:1007614523901
  40. C. Cortes and V. Vapnik, "Support Vector Networks," Machine Learning, vol. 20, no. 3, 1995, pp. 273-297.
  41. C.W. Hsu and C.J. Lin, "A Comparison on Methods for Multiclass Support Vector Machines," IEEE Trans. Neural Networks, vol. 13, no. 2, 2002, pp. 415-425. https://doi.org/10.1109/72.991427
  42. Z. Niu et al., "2D Cascaded AdaBoost for Eye Localization," Proc. IEEE Int. Conf. Pattern Recog., 2006, pp. 1216-1219.
  43. M.S. Bartlett et al., "Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior," IEEE Conf. Computer Vision and Pattern Recog., 2005, pp. 568-573.
  44. Y. Tian, "Evaluation of Face Resolution for Expression Analysis," CVPR Workshop Face Process. Video, 2004, p. 82.
  45. Y. Ono, FT. Okabe, and Y. Sato, "Gaze Estimation from Low Resolution Images," IEEE Pacific-Rim Symp. Image Video Technol., 2006, pp. 178-188.
  46. J.N. Bassili, "Emotion Recognition: The Role of Facial Movement and the Relative Importance of Upper and Lower Areas of the Face," J. Personality Social Psychology, vol. 37, no. 11, 1979, pp. 2049-2058. https://doi.org/10.1037/0022-3514.37.11.2049

Cited by

  1. Gradient directional pattern: A robust feature descriptor for facial expression recognition vol.48, pp.19, 2010, https://doi.org/10.1049/el.2012.1841
  2. Local Directional Number Pattern for Face Analysis: Face and Expression Recognition vol.22, pp.5, 2013, https://doi.org/10.1109/tip.2012.2235848
  3. Automated Facial Expression Recognition Using Gradient-Based Ternary Texture Patterns vol.2013, pp.None, 2010, https://doi.org/10.1155/2013/831747
  4. Centralized Gradient Pattern for Face Recognition vol.ed96, pp.3, 2010, https://doi.org/10.1587/transinf.e96.d.538
  5. Modified Local Directional Pattern 영상을 이용한 얼굴인식 vol.2, pp.3, 2010, https://doi.org/10.3745/ktsde.2013.2.3.205
  6. Development of Mobile Social Network Systems Using Real-Time Facial Authentication and Collaborative Recommendations vol.9, pp.12, 2010, https://doi.org/10.1155/2013/820979
  7. Extraction of User Preference for Video Stimuli Using EEG-Based User Responses vol.35, pp.6, 2013, https://doi.org/10.4218/etrij.13.0113.0194
  8. Zernike moments and LDP-weighted patches for content-based image retrieval vol.8, pp.3, 2010, https://doi.org/10.1007/s11760-013-0561-z
  9. 얼굴 표정인식을 위한 2D-DCT 특징추출 방법 vol.3, pp.3, 2010, https://doi.org/10.3745/ktsde.2014.3.3.135
  10. Facial Expression Recognition Based on the Texture Features of Global Principal Component and Local Boundary vol.548, pp.None, 2014, https://doi.org/10.4028/www.scientific.net/amm.548-549.1110
  11. Facial Expression Recognition via Non-Negative Least-Squares Sparse Coding vol.5, pp.2, 2010, https://doi.org/10.3390/info5020305
  12. Signed Local Directional Pattern을 이용한 강력한 얼굴 표정인식 vol.51, pp.6, 2010, https://doi.org/10.5573/ieie.2014.51.6.089
  13. Extreme Learning Machine Ensemble Using Bagging for Facial Expression Recognition vol.10, pp.3, 2010, https://doi.org/10.3745/jips.02.0004
  14. Eigen directional bit-planes for robust face recognition vol.60, pp.4, 2014, https://doi.org/10.1109/tce.2014.7027346
  15. Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection vol.21, pp.6, 2010, https://doi.org/10.1007/s00530-014-0400-2
  16. Feast: face and emotion analysis system for smart tablets vol.74, pp.21, 2010, https://doi.org/10.1007/s11042-014-2082-3
  17. Automatic facial expression recognition using features of salient facial patches vol.6, pp.1, 2015, https://doi.org/10.1109/taffc.2014.2386334
  18. Human Facial Expression Recognition Using Stepwise Linear Discriminant Analysis and Hidden Conditional Random Fields vol.24, pp.4, 2010, https://doi.org/10.1109/tip.2015.2405346
  19. Text and User Generic Model for Writer Verification Using Combined Pen Pressure Information From Ink Intensity and Indented Writing on Paper vol.45, pp.3, 2010, https://doi.org/10.1109/thms.2014.2380828
  20. Facial expression recognition using ASM-based post-processing technique vol.26, pp.3, 2016, https://doi.org/10.1134/s105466181603010x
  21. Active Shape Model을 이용한 외형기반 얼굴표정인식에 관한 연구 vol.5, pp.1, 2010, https://doi.org/10.3745/ktsde.2016.5.1.43
  22. SIFT 기술자를 이용한 얼굴 표정인식 vol.5, pp.2, 2016, https://doi.org/10.3745/ktsde.2016.5.2.89
  23. Binocular Responses for No-Reference 3D Image Quality Assessment vol.18, pp.6, 2016, https://doi.org/10.1109/tmm.2016.2542580
  24. Novel Method for Face Recognition using Laplacian of Gaussian Mask with Local Contour Pattern vol.10, pp.11, 2016, https://doi.org/10.3837/tiis.2016.11.022
  25. An Emotion Recognition System for Mobile Applications vol.5, pp.None, 2010, https://doi.org/10.1109/access.2017.2672829
  26. A Facial-Expression Monitoring System for Improved Healthcare in Smart Cities vol.5, pp.None, 2010, https://doi.org/10.1109/access.2017.2712788
  27. Offline writer verification based on forensic expertise: Analyzing multiple characters by combining the shape and advanced pen pressure information vol.22, pp.2, 2017, https://doi.org/10.3408/jafst.731
  28. A comparative study of the use of local directional pattern for texture-based informal settlement classification vol.15, pp.3, 2010, https://doi.org/10.1016/j.jart.2016.12.009
  29. Palmprint Recognition Based on Complete Direction Representation vol.26, pp.9, 2010, https://doi.org/10.1109/tip.2017.2705424
  30. Directional Age-Primitive Pattern (DAPP) for Human Age Group Recognition and Age Estimation vol.12, pp.11, 2010, https://doi.org/10.1109/tifs.2017.2695456
  31. Local Directional Ternary Pattern for Facial Expression Recognition vol.26, pp.12, 2010, https://doi.org/10.1109/tip.2017.2726010
  32. Improved gradient local ternary patterns for facial expression recognition vol.2017, pp.None, 2017, https://doi.org/10.1186/s13640-017-0190-5
  33. Evaluating real-life performance of the state-of-the-art in facial expression recognition using a novel YouTube-based datasets vol.77, pp.1, 2010, https://doi.org/10.1007/s11042-016-4321-2
  34. Novel directional patterns and a Generalized Supervised Dimension Reduction System (GSDRS) for facial emotion recognition vol.77, pp.8, 2010, https://doi.org/10.1007/s11042-017-5141-8
  35. Dimensionality reduced local directional number pattern for face recognition vol.9, pp.1, 2010, https://doi.org/10.1007/s12652-016-0408-x
  36. Modified dimensionality reduced local directional pattern for facial analysis vol.9, pp.3, 2010, https://doi.org/10.1007/s12652-017-0473-9
  37. Angled local directional pattern for texture analysis with an application to facial expression recognition vol.12, pp.5, 2010, https://doi.org/10.1049/iet-cvi.2017.0340
  38. Kernel Embedding Multiorientation Local Pattern for Image Representation vol.48, pp.4, 2010, https://doi.org/10.1109/tcyb.2017.2682272
  39. Facial expression recognition based on Gabor features of salient patches and ACI-LBP vol.34, pp.4, 2010, https://doi.org/10.3233/jifs-17422
  40. Facial Expression Recognition under Difficult Conditions: A Comprehensive Study on Edge Directional Texture Patterns vol.28, pp.2, 2010, https://doi.org/10.2478/amcs-2018-0030
  41. Accurate and robust facial expression recognition system using real-time YouTube-based datasets vol.48, pp.9, 2010, https://doi.org/10.1007/s10489-017-1121-y
  42. Weighted Gradient Feature Extraction Based on Multiscale Sub-Blocks for 3D Facial Recognition in Bimodal Images vol.9, pp.3, 2010, https://doi.org/10.3390/info9030048
  43. Palmprint recognition based on discriminant multiscale representation vol.27, pp.5, 2010, https://doi.org/10.1117/1.jei.27.5.053032
  44. Local Dominant Directional Symmetrical Coding Patterns for Facial Expression Recognition vol.2019, pp.None, 2010, https://doi.org/10.1155/2019/3587036
  45. An Approach to Share Self-Taught Knowledge between Home IoT Devices at the Edge vol.19, pp.4, 2010, https://doi.org/10.3390/s19040833
  46. Facial expression recognition based on weighted adaptive symmetric CBP-TOP vol.36, pp.3, 2010, https://doi.org/10.3233/jifs-18696
  47. Regional adaptive affinitive patterns (RADAP) with logical operators for facial expression recognition vol.13, pp.5, 2010, https://doi.org/10.1049/iet-ipr.2018.5683
  48. Facial expression recognition based on improved completed local ternary patterns vol.15, pp.3, 2010, https://doi.org/10.1007/s11801-019-8136-z
  49. Novel multispectral face descriptor using orthogonal walsh codes vol.13, pp.7, 2010, https://doi.org/10.1049/iet-ipr.2018.6423
  50. DLGBD: A directional local gradient based descriptor for face recognition vol.78, pp.12, 2010, https://doi.org/10.1007/s11042-018-6967-4
  51. Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction vol.19, pp.13, 2019, https://doi.org/10.3390/s19132844
  52. A novel maximum and minimum response-based Gabor (MMRG) feature extraction method for facial expression recognition vol.78, pp.16, 2010, https://doi.org/10.1007/s11042-019-7646-9
  53. Facial Expression Recognition: A Survey vol.11, pp.10, 2010, https://doi.org/10.3390/sym11101189
  54. Multiparameter Space Decision Voting and Fusion Features for Facial Expression Recognition vol.2020, pp.None, 2010, https://doi.org/10.1155/2020/8886872
  55. Multimodal biometric system combining left and right palmprints vol.48, pp.1, 2010, https://doi.org/10.1108/idd-01-2019-0011
  56. FTSH: a framework for transition from square image processing to hexagonal image processing vol.79, pp.11, 2010, https://doi.org/10.1007/s11042-019-08487-z
  57. A comprehensive survey on automatic facial action unit analysis vol.36, pp.5, 2020, https://doi.org/10.1007/s00371-019-01707-5
  58. Template matching and machine learning-based robust facial expression recognition system using multi-level Haar wavelet vol.42, pp.4, 2020, https://doi.org/10.1080/1206212x.2017.1395134
  59. Robust local oriented patterns for ear recognition vol.79, pp.41, 2010, https://doi.org/10.1007/s11042-020-09456-7
  60. Rapid facial expression recognition under part occlusion based on symmetric SURF and heterogeneous soft partition network vol.79, pp.41, 2010, https://doi.org/10.1007/s11042-020-09566-2
  61. A dynamic inverse distance weighting-based local face descriptor vol.79, pp.41, 2020, https://doi.org/10.1007/s11042-020-09581-3
  62. Subspace learning for facial expression recognition: an overview and a new perspective vol.10, pp.None, 2010, https://doi.org/10.1017/atsip.2020.27
  63. A novel approach for facial expression recognition using local binary pattern with adaptive window vol.80, pp.2, 2010, https://doi.org/10.1007/s11042-020-09663-2
  64. Interval graph of facial regions with common intersection salient points for identifying and classifying facial expression vol.80, pp.3, 2010, https://doi.org/10.1007/s11042-020-09806-5
  65. Facial expression recognition using singular values and wavelet‐based LGC‐HD operator vol.10, pp.2, 2010, https://doi.org/10.1049/bme2.12012
  66. Facial Emotion Recognition Using Transfer Learning in the Deep CNN vol.10, pp.9, 2021, https://doi.org/10.3390/electronics10091036
  67. Triple-Type Feature Extraction for Palmprint Recognition vol.21, pp.14, 2021, https://doi.org/10.3390/s21144896
  68. Local Neighbourhood Edge Responsive Image Descriptor for Texture Classification Using Gaussian Mutated JAYA Optimization Algorithm vol.46, pp.9, 2021, https://doi.org/10.1007/s13369-021-05417-w
  69. Face Recognition System Using Local Features Fusion for Multi-Masks vol.2107, pp.1, 2010, https://doi.org/10.1088/1742-6596/2107/1/012044
  70. A Novel deep neural network-based emotion analysis system for automatic detection of mild cognitive impairment in the elderly vol.468, pp.None, 2022, https://doi.org/10.1016/j.neucom.2021.10.038
  71. Enhanced global and local face feature extraction for effective recognition of facial emotions vol.34, pp.5, 2022, https://doi.org/10.1002/cpe.6701