The Journal of Korea Robotics Society (로봇학회논문지)

Korea Robotics Society (한국로봇학회)
권호 표지
DOI QR코드

DOI QR Code

3D Convolutional Neural Networks based Fall Detection with Thermal Camera

열화상 카메라를 이용한 3D 컨볼루션 신경망 기반 낙상 인식

  • Kim, Dae-Eon (Ujin Technology) ;
  • Jeon, BongKyu (Robotics Program, KAIST(Korea Advanced Institute of Science and Technology)) ;
  • Kwon, Dong-Soo (Department of Mechanical Engineering, KAIST(Korea Advanced Institute of Science and Technology))
  • Received : 2018.02.03
  • Accepted : 2018.02.19
  • Published : 2018.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

This paper presents a vision-based fall detection system to automatically monitor and detect people's fall accidents, particularly those of elderly people or patients. For video analysis, the system should be able to extract both spatial and temporal features so that the model captures appearance and motion information simultaneously. Our approach is based on 3-dimensional convolutional neural networks, which can learn spatiotemporal features. In addition, we adopts a thermal camera in order to handle several issues regarding usability, day and night surveillance and privacy concerns. We design a pan-tilt camera with two actuators to extend the range of view. Performance is evaluated on our thermal dataset: TCL Fall Detection Dataset. The proposed model achieves 90.2% average clip accuracy which is better than other approaches.

Keywords

References

  1. A. L. Hendrich, P. S. Bender, and A. Nyhuis, "Validation of the Hendrich II Fall Risk Model: A large concurrent case/control study of hospitalized patients," Applied Nursing Research, vol. 16, no. 1, pp. 9-21, February, 2003. https://doi.org/10.1053/apnr.2003.016009
  2. A. Kalache, D. Fu, S. Yoshida, W. Al-Faisal, L. Beattie, W. Chodzko-Zajko, H. Fu, K. James, S. Kalula, B. Krishnaswamy, N. Kronfol, P. Marin, I. Pike, D. Rose, V. Scott, J. Stevens, C. Todd, and G. Usha, "World Health Organisation Global Report on Falls Prevention in Older Age," Geneva: WHO 2007, World Health Organization, Department of Ageing & Life Course, ed., 2007.
  3. L. Z. Rubenstein, "Falls in older people: epidemiology, risk factors and strategies for prevention," Age and Ageing, vol. 35, no. suppl_2, pp. ii37-ii41, 2006.
  4. M. Mubashir, L. Shao, and L. Seed, "A survey on fall detection: Principles and approaches," Neurocomputing, vol. 100, no. 16, pp. 144-152, 2013. https://doi.org/10.1016/j.neucom.2011.09.037
  5. J. G. Shin, "Prevalence and management of falls among elderly in a hospital," Unpublished master's thesis, Hanyang University, pp. 1-63, 2011.
  6. H. Rimminen, J. Lindstrom, M. Linnavuo, and R. Sepponen, "Detection of falls among the elderly by a floor sensor using the electric near field," IEEE Transactions on Information Technology in Biomedicine, vol. 14, no. 6, pp. 1475-1476, 2010. https://doi.org/10.1109/TITB.2010.2051956
  7. S. Shan and T. Yuan, "A wearable pre-impact fall detector using feature selection and support vector machine," IEEE 10th INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING, Beijing, China, 2010, doi: 10.1109/ICOSP.2010.5656840.
  8. B. Kaluza and M. Lustrek, "Fall detection and activity recognition methods for the confidence project: a survey," 10th International Multiconference, Ljubljana, Slovenia, pp. 22-25, 2008.
  9. R. Igual, C. Medrano, and I. Plaza, "Challenges, issues and trends in fall detection systems," BioMedical Engineering, [Online], vol. 12, no. 1, pp. 66, DOI. Available: https://doi.org/10.1186/1475-925X-12-66.
  10. M. A. Habib, M. S. Mohktar, S. B. Kamaruzzaman, K. S. Lim, T. M. Pin, and F. Ibrahim, "Smartphone-Based Solutions for Fall Detection and Prevention: Challenges and Open Issues," Sensors 2014, vol. 14, no. 4, pp. 7181-7208, 2014. https://doi.org/10.3390/s140407181
  11. M. K. Mulcahy and S. Kurkovsky. "Automatic Fall Detection using Mobile Devices," 2015 12th International Conference on Information Technology, Las Vegas, NV, USA, 2015, doi: 10.1109/ITNG.2015.98.
  12. Z. Fu, E. Culurciello, P. Lichtsteiner, and Tobi Delbruck, "Fall detection using an address-event temporal contrast vision sensor," 2008 IEEE International Symposium on Circuits and Systems, Seattle, WA, USA, 2008, doi: 10.1109/ISCAS.2008.4541445.
  13. S.-G. Miaou , P.-H. Sung, a nd C .-Y. Huang, "A c ustomized human fall detection system using omni-camera images and personal information," 1st Transdisciplinary Conference on Distributed Diagnosis and Home Healthcare, Arlington, VA, USA, 2006, doi: 10.1109/DDHH.2006.1624792.
  14. W. K. Wong, H. L. Lim, C. K. Loo, and W. S. Lim, "Home alone faint detection surveillance system using thermal camera," 2010 Second International Conference on Computer Research and Development, Kuala Lumpur, Malaysia, 2010, doi: 10.1109/ICCRD.2010.163.
  15. G. Mastorakis and D. Makris, "Fall detection system using Kinect's infrared sensor," Journal of Real-Time Image Processing, vol. 9, no. 4, pp. 635-646, December, 2014. https://doi.org/10.1007/s11554-012-0246-9
  16. S. Vadivelu, S. Ganesan, O. V. R. Murthy, and A. Dhall, "Thermal Imaging Based Elderly Fall Detection," Computer Vision - ACCV 2016 Workshops, Taipei, Taiwan, pp. 541-553, 2016.
  17. P. Scovanner, S. Ali, and M. Shah, "A 3-dimensional sift descriptor and its application to action recognition," 15th ACM international conference on Multimedia, Augsburg, Germany, pp. 357-360, 2007.
  18. A. Klaeser, M. Marszalek, and C. Schmid, "A spatio-temporal descriptor based on 3d-gradients," British Machine Conference, pp. 99.1-99.10, 2008.
  19. H. Wang and C. Schmid, "Action recognition with improved trajectories," 2013 IEEE International Conference on Computer Vision, Sydney, Australia, 2013, doi: 10.1109/ICCV.2013.441.
  20. S. Ji, W. Xu, M. Yang, and Kai Yu, "3D convolutional neural networks for human action recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 221-231, Jan., 2013. https://doi.org/10.1109/TPAMI.2012.59
  21. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3d convolutional networks," 2015 IEEE International Conference on Computer Vision, Santiago, Chile, 2015, doi: 10.1109/ICCV.2015.510.
  22. K. Simonyan and A. Zisserman. "Two-stream convolutional networks for action recognition in videos," Advances in Neural Information Processing Systems 27, Montreal, Canada, pp. 568-576, 2014.
  23. P. Molchanov, S. Gupta, K. Kim, and J. Kautz, "Hand gesture recognition with 3D convolutional neural networks," 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA, 2015, doi: 10.1109/CVPRW.2015.7301342.
  24. M. Baccouche,F. Mamalet, C. Wolf, C. Garcia, and A. Baskurt, "Sequential deep learning for human action recognition," International Workshop on Human Behavior Understanding, 2011, doi: 10.1007/978-3-642-25446-8_4.
  25. K. Schindler and L. van Gool, "Action snippets: How many frames does human action recognition require?," 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008, doi: 10.1109/CVPR.2008.4587730.
  26. A. Sixsmith, N. Johnson, and R. Whatmore, "Pyroelectric IR sensor arrays for fall detection in the older population," Journal de Physique IV, Vol. 128, pp. 153-160, September, 2005.
  27. S. Kido, T. Miyasaka, T. Tanaka, T. Shimizu, and T. Saga, "Fall detection in toilet rooms using thermal imaging sensors," 2009 IEEE/SICE International Symposium on System Integration, Tokyo, Japan, 2009, doi: 10.1109/SI.2009.5384550.
  28. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, no. 6, pp. 84-90, June, 2012.
  29. L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of Machine Learning Research 9, pp. 2579-2605, Nov., 2008.