Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Depth Image Restoration Using Generative Adversarial Network

Generative Adversarial Network를 이용한 손실된 깊이 영상 복원

  • Nah, John Junyeop (Inha University, Department of Information and Communication Engineering) ;
  • Sim, Chang Hun (Inha University, Department of Information and Communication Engineering) ;
  • Park, In Kyu (Inha University, Department of Information and Communication Engineering)
  • 나준엽 (인하대학교 정보통신공학과) ;
  • 심창훈 (인하대학교 정보통신공학과) ;
  • 박인규 (인하대학교 정보통신공학과)
  • Received : 2018.05.08
  • Accepted : 2018.07.16
  • Published : 2018.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a method of restoring corrupted depth image captured by depth camera through unsupervised learning using generative adversarial network (GAN). The proposed method generates restored face depth images using 3D morphable model convolutional neural network (3DMM CNN) with large-scale CelebFaces Attribute (CelebA) and FaceWarehouse dataset for training deep convolutional generative adversarial network (DCGAN). The generator and discriminator equip with Wasserstein distance for loss function by utilizing minimax game. Then the DCGAN restore the loss of captured facial depth images by performing another learning procedure using trained generator and new loss function.

본 논문에서는 generative adversarial network (GAN)을 이용한 비감독 학습을 통해 깊이 카메라로 깊이 영상을 취득할 때 발생한 손실된 부분을 복원하는 기법을 제안한다. 제안하는 기법은 3D morphable model convolutional neural network (3DMM CNN)와 large-scale CelebFaces Attribute (CelebA) 데이터 셋 그리고 FaceWarehouse 데이터 셋을 이용하여 학습용 얼굴 깊이 영상을 생성하고 deep convolutional GAN (DCGAN)의 생성자(generator)와 Wasserstein distance를 손실함수로 적용한 구별자(discriminator)를 미니맥스 게임기법을 통해 학습시킨다. 이후 학습된 생성자와 손실 부분을 복원해주기 위한 새로운 손실함수를 이용하여 또 다른 학습을 통해 최종적으로 깊이 카메라로 취득된 얼굴 깊이 영상의 손실 부분을 복원한다.

Keywords

References

  1. K. Xu, J. Zhou, and Z. Wang, "A method of hole-filling for the depth map generated by Kinect with moving objects detection," Proceeding of IEEE international Symposium on Broadband Multimedia Systems and Broadcasting, pp. 1-5, June 2012.
  2. L. Feng, L.-M. Po, X. Xu, K.-H. Ng, C.-H. Cheung, and K.-w. Cheung, "An adaptive background biased depth map hole-filling method for Kinect," Proceeding of IEEE Industrial Electronics Society, pp. 2366-2371, November 2013.
  3. S. Ikehata, J. Cho, and K. Aizawa, "Depth map inpainting and super-resolution based on internal statistics of geometry and appearance," Proceeding of IEEE International Conference on Image Processing, pp. 938-942, September 2013.
  4. A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep convolutional generative adversarial networks," Proceeding of International Conference on Learning Representations, May 2016.
  5. R. A. Yeh, C. Chen, T. Yian Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do, "Semantic image inpainting with deep generative models," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
  6. M. Arjovsky, S.Chintala, and L. Bottou, "Wasserstein GAN," Proceeding of International Conference on Machine Learning, vol. 70, pp.214-223, August. 2017.
  7. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," Proceeding of Advances in Neural Information Processing Systems, December 2014.
  8. A. T. Tran, T. Hassner, I. Masi, and G. Medioni, "Regressing robust and discriminative 3D morphable models with a very deep neural net-work," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
  9. C. Cao, Y. Weng, S. Zhou, Y. Tong, and K.Zhou, "FaceWarehouse: a 3D facial expression database for visual computing," IEEE Transaction on Visualization and Computer Graphics, vol. 20, no. 3, pp. 413-425, March 2014. https://doi.org/10.1109/TVCG.2013.249
  10. P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter, "A 3D face model for pose and illumination invariant face recognition," Proceeding of IEEE International Conference on Advanced Video and Signal Based Surveillance, October 2009.
  11. $Intel^{(R)}$ $RealSense^{TM}$ Camera SR300, https://software.intel.com/sites/default/files/managed/0c/ec/realsense-sr300-product-data-sheet-rev-1-0.pdf (accessed August 13, 2018).