The Journal of Korean Institute of Information Technology (한국정보기술학회논문지)

Korean Institute of Information Technology (한국정보기술학회)
권호 표지
DOI QR코드

DOI QR Code

Analysis of the Relative Importance of HRV Metrics to Predict Emotion by Using Valence-Arousal Driven Neural Network

감정예측을 위한 심박변이도 변수의 상대적 중요도 분석: Valence-Arousal 기반의 인공신경망 관점으로

  • Park, Sung Soo ;
  • Lee, Kun Chang (SKK Business School/SAIHST(Samsung Advanced Institute for Health Sciences & Technology), Sungkyunkwan University)
  • 박성수 (성균관대학교 경영대학) ;
  • 이건창 (성균관대학교 경영대학/삼성융합의과학원)
  • Received : 2017.10.19
  • Accepted : 2018.04.09
  • Published : 2018.04.30
⟨ Previous Next ⟩

Abstract

Previous emotion prediction methods based on the categorized emotions have shown difficulty in recognizing relative importance of the input variables. This study proposes a new emotion prediction mechanism in which HRV (Heart Rate Variability) metrics and continuous valence-arousal scores are used to formulate the emotion prediction neural network. The Garson's algorithm was adopted to compute relative importance of input HRV metrics. The physiological data collected from 50 participants was used to obtain that the relative importance of Mean RR is 35.6%, the best among a number of input HRV metrics. Meanwhile. the frequency domain HRV metrics such as VLF, LF, and HF have low relative importance of 3.2%, 3.6%, and 3.0%, respectively.

범주화된 감정예측을 이용한 기존연구는 각 입력 변수가 예측에 기여하는 상대적 중요도를 정확히 파악하기 어렵다. 본 연구는 연속형인 Valence-Arousal 기반의 예측모형을 생성하고, Garson의 알고리즘을 적용하여 감정예측 입력변수에 대한 상대적 중요도를 제시한다. 이를 위해 피험자 50명으로부터 수집한 심박변이도(HRV, Heart Rate Variability) 변수와 인공신경망을 사용하여 감정예측 모형을 생성하였다. 실험결과 다양한 심박변이도 중 Mean RR의 상대 중요도가 35.6%으로서 감정예측에 가장 높은 기여를 하는 것으로 확인되었다. 한편, 주파수 영역 심박변이도 지표인 VLF, LF, HF의 감정예측시 상대적 중요도는 각각 3.2%, 3.6%, 3.0%로 낮은 것으로 확인되었다.

Keywords

Acknowledgement

Supported by : 한국연구재단

References

  1. R. W. Picard, "Affective computing: challenges", International Journal of Human-Computer Studies, Vol. 59, No. 1, pp. 55-64, Jul. 2003. https://doi.org/10.1016/S1071-5819(03)00052-1
  2. C. Maaoui and A. Pruski, "Emotion recognition through physiological signals for human-machine communication", Cutting Edge Robotics, pp. 317-333, Sep. 2010.
  3. J. S. Park, and Y. H. Seo, "Acoustic Informationbased Emotion Recognition for Human-Robot Interaction," The Journal of Korean Institute of Information Technology, Vol. 9, No. 6, pp. 39-46, Jun. 2011.
  4. E. C. Nook, K. A. Lindquist, and J. Zaki, "A new look at emotion perception: Concepts speed and shape facial emotion recognition", Emotion, Vol. 15, No. 5, pp. 569-578, Oct. 2015. https://doi.org/10.1037/a0039166
  5. C. N. Anagnostopoulos, T. Iliou and I. Giannoukos, "Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011", Artificial Intelligence Review, Vol. 43, No. 2, pp. 155-177, Feb. 2015. https://doi.org/10.1007/s10462-012-9368-5
  6. M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, "Analysis of EEG signals and facial expressions for continuous emotion detection", IEEE Transactions on Affective Computing, Vol. 7, No. 1, pp. 17-28, May 2016. https://doi.org/10.1109/TAFFC.2015.2436926
  7. C. D. Katsis, N. S. Katertsidis and D. I. Fotiadis, "An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders", Biomedical Signal Processing and Control, Vol. 6, No. 3, pp. 261-268, Jul. 2011. https://doi.org/10.1016/j.bspc.2010.12.001
  8. F. A. Russo, N. N. Vempala, and G. M. Sandstrom, "Predicting musically induced emotions from physiological inputs: Linear and neural network models", Frontiers in Psychology, Vol. 4, No. 468, Aug. 2013.
  9. P. A. Kragel and K. S. LaBar, "Multivariate pattern classification reveals autonomic and experiential representations of discrete emotions", Emotion, Vol. 13, No. 4, pp. 681-690, Aug. 2013. https://doi.org/10.1037/a0031820
  10. J. Selvaraj, M. Murugappan, K. Wan, and S. Yaacob, "Classification of emotional states from electrocardiogram signals: a non-linear approach based on hurst", Biomedical engineering online, Vol. 12, No. 44, May 2013.
  11. D. Lee and and S. G. Lee, "The EEG Signal Based Emotion Recognition Using the EMD", The Journal of Korean Institute of Information Technology, Vol. 12, No. 6, pp. 47-53, Jun. 2014.
  12. M. J. Song, M. Y. Kim, I. S. Sim, and W. S. Kim, "Evaluation of Horticultural Therapy on the Emotional Improvement of Depressed Patients by Using Heart Rate Variability", Korean Journal of Horticultural Science & Technology, Vol. 28, No. 6, pp. 1066-1071, Dec. 2010.
  13. N. Ravaja, "Contributions of psychophysiology to media research: Review and recommendations", Media Psychology, Vol. 6, No. 2, pp. 193-235, Dec. 2004. https://doi.org/10.1207/s1532785xmep0602_4
  14. M. S. Kim, Y. N. Kim, and Y. C. Cho, "Electrocardiographic characteristics of significant factors of detected atrial fibrillation using WEMS", Journal of the Korea Industrial Information Systems Research, Vol. 20, No. 6, pp. 37-46, Dec. 2015. https://doi.org/10.9723/jksiis.2015.20.6.037
  15. S. N. Yu and S. F. Chen, "Emotion state identification based on heart rate variability and genetic algorithm", 2015 37th IEEE Conference on Engineering in Medicine and Biology Society (EMBC), pp. 538-541, Aug. 2015.
  16. R. Rakshit, V. R. Reddy, and P. Deshpande, "Emotion detection and recognition using HRV features derived from photoplethysmogram signals", In Proceedings of the 2nd workshop on Emotion Representations and Modelling for Companion Systems, Tokyo, Japan, Nov. 2016.
  17. H. W. Guo, Y. S. Huang, C. H. Lin, J. C. Chien, K. Haraikawa, and J. S. Shieh, "Heart Rate Variability Signal Features for Emotion Recognition by Using Principal Component Analysis and Support Vectors Machine", IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE), Taichung, Taiwan, pp. 274-277, Oct. 2016.
  18. P. Ekman, W. V. Friesen, M. O'Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, and K. Scherer, "Universals and cultural differences in the judgments of facial expressions of emotion", Journal of personality and social psychology, Vol. 53, No. 4, pp. 712-717, Dec. 1987. https://doi.org/10.1037/0022-3514.53.4.712
  19. J. A. Russell, "A circumplex model of affect", Journal of Personality and Social Psychology, Vol. 39, pp. 1161-1178, Dec. 1980. https://doi.org/10.1037/h0077714
  20. J. Kim and E. Andre, "Emotion recognition based on physiological changes in music listening", IEEE transactions on pattern analysis and machine intelligence, Vol. 30, No. 12, pp. 2067-2083, Feb. 2008. https://doi.org/10.1109/TPAMI.2008.26
  21. T. G. M. Vrijkotte, L. J. P. van Doornen, and E. J. C. de Geus, "Effects of Work Stress on Ambulatory Blood Pressure, Heart Rate, and Heart Rate Variability", Hypertension, Vol. 35, pp. 880-886, Apr. 2000. https://doi.org/10.1161/01.HYP.35.4.880
  22. M. Swangnetr and D. B. Kaber, "Emotional state classification in patient-robot interaction using wavelet analysis and statistics-based feature selection", IEEE Transactions on Human-Machine Systems, Vol. 43, No. 1, pp. 63-75, Sep. 2013. https://doi.org/10.1109/TSMCA.2012.2210408
  23. G. Valenza, L. Citi, A. Lanata, E. P. Scilingo, and R. Barbieri, "Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics", Scientific reports, Vol. 4, No. 4998, pp. 1-13, May 2014.
  24. M. Nardelli, G. Valenza, A. Greco, A. Lanata, and E. P. Scilingo, "Recognizing emotions induced by affective sounds through heart rate variability", IEEE Transactions on Affective Computing, Vol. 6, No. 4, pp. 385-394, May 2015. https://doi.org/10.1109/TAFFC.2015.2432810
  25. S. S. Park and K. C. Lee, "Emotion prediction neural network to understand how emotion is predicted by using heart rate variability measurements", Journal of The Korea Society of Computer and Information, Vol. 22, No. 7, pp. 75-82, Jul. 2017. https://doi.org/10.9708/JKSCI.2017.22.05.075
  26. S. S. Park and K. C. Lee, "Convergence Implementing Emotion Prediction Neural Network Based on Heart Rate Variability (HRV)", Under review in Journal of The Korea Convergence Society, Mar. 2018.
  27. M. M. Bradley and P. J. Lang, "Measuring emotion: the self-assessment manikin and the semantic differential", Journal of behavior therapy and experimental psychiatry, Vol. 25, No. 1, pp. 49-59, Mar. 1994. https://doi.org/10.1016/0005-7916(94)90063-9
  28. U. R. Acharya, K. P. Joseph, N. Kannathal, C. M. Lim, and J. S. Suri, "Heart rate variability: a review", Medical and biological engineering and computing, Vol. 44, No. 12, pp. 1031-1051, Dec. 2006. https://doi.org/10.1007/s11517-006-0119-0
  29. M. P. Tarvainen, J. P. Niskanen, J. A. Lipponen, P. O. Ranta-Aho, and P. A. Karjalainen, "Kubios HRV - heart rate variability analysis software", Computer methods and programs in biomedicine, Vol. 113, No. 1, pp. 210-220, Jan. 2014. https://doi.org/10.1016/j.cmpb.2013.07.024
  30. C. M. Bishop, "Neural Networks for Pattern Recognition", Oxford university press, pp. 77-111, Dec. 1995.
  31. R. Hecht-Nielsen, "Theory of the Back Propagation Neural Network", In Proceeding of the International Joint Conference on Neural Networks (IJCNN), New York, pp. 593-605, Dec. 1989.
  32. A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion recognition using bio-sensors: First steps towards an automatic system", In Tutorial and research workshop on affective dialogue systems, Springer Berlin Heidelberg, pp. 36-48, Jun. 2004.
  33. G. D. Garson, "Interpreting neural-network connection weights", AI Expert, Vol. 6, pp. 47-51, Apr. 1991.