Journal of the Institute of Electronics Engineers of Korea SP (대한전자공학회논문지SP)

The Institute of Electronics and Information Engineers (대한전자공학회)
권호 표지

High-resolution Depth Generation using Multi-view Camera and Time-of-Flight Depth Camera

다시점 카메라와 깊이 카메라를 이용한 고화질 깊이 맵 제작 기술

  • Kang, Yun-Suk (Realistic Broadcasting Research Center, Gwangju Institute of Science and Technology) ;
  • Ho, Yo-Sung (Realistic Broadcasting Research Center, Gwangju Institute of Science and Technology)
  • 강윤석 (광주과학기술원 실감방송연구센터) ;
  • 호요성 (광주과학기술원 실감방송연구센터)
  • Received : 2011.08.29
  • Published : 2011.11.25
Download PDF
⟨ Previous Next ⟩

Abstract

The depth camera measures range information of the scene in real time using Time-of-Flight (TOF) technology. Measured depth data is then regularized and provided as a depth image. This depth image is utilized with the stereo or multi-view image to generate high-resolution depth map of the scene. However, it is required to correct noise and distortion of TOF depth image due to the technical limitation of the TOF depth camera. The corrected depth image is combined with the color image in various methods, and then we obtain the high-resolution depth of the scene. In this paper, we introduce the principal and various techniques of sensor fusion for high-quality depth generation that uses multiple camera with depth cameras.

깊이 카메라는 Time-of-Flight (TOF) 기술을 이용하여 장면 내 물체들의 거리 정보를 실시간으로 측정하며, 측정된 값은 깊이 영상으로 출력되어 양안식 혹은 다시점 카메라와 함께 장면의 고화질 깊이 맵을 제작하는 데 사용된다. 그러나 깊이 카메라 자체가 가지는 기술적 한계로 인하여 영상에 잡음과 왜곡이 포함되어 있기 때문에 이를 효과적으로 제거할 수 있는 기술이 요구되며, 처리 된 깊이 카메라 영상은 다양한 방법으로 색상 영상과 융합되어 장면의 깊이 정보를 생성할 수 있다. 본 논문에서는 이와 같이 다시 점 카메라와 깊이 카메라를 함께 사용하여 고화질의 깊이 정보를 획득할 수 있는 혼합형 카메라 방식의 원리와 깊이 영상 처리 및 깊이 생성을 위한 기술 동향을 설명한다.

Keywords

References

  1. S. Saxena, M, Sun, and A. Y. Ng, "Make 3D: Learning 3D Scene Structure from a Single Still Image," IEEE Transactions on Pattern Analysis and Intelligence, vol. 31, no. 5, pp. 824-840, May 2009. https://doi.org/10.1109/TPAMI.2008.132
  2. J. Sun, N.N. Zheng, and H.Y. Shum, "Stereo Matching Using Belief Propagation," IEEE Transactions of Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, May 2003. https://doi.org/10.1109/TPAMI.2003.1206509
  3. J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, "A State of the Art in Structured Light Patterns for Surface Profilometry," Pattern Recognition, vol. 43, issue 8, pp. 2666-2680, Aug. 2010. https://doi.org/10.1016/j.patcog.2010.03.004
  4. ZCam Product Data Sheet, StudioGE, 2009.
  5. S.Y. Kim, E.K. Lee, and Y.S. Ho, "Generation of ROI Enhanced Depth Maps Using Stereoscopic Cameras and a Depth Camera," IEEE Transactions on Broadcasting, vol. 54, no. 4, pp. 732-740, Dec. 2008.
  6. A. Wang, T. Qiu, and L. Shao, "A Simple Method of Radial Distortion Correction with Centre of Distortion Estimation," Journal of Mathematical Imaging and Vision, vol. 35, no. 3, pp. 165-172, July 2009. https://doi.org/10.1007/s10851-009-0162-1
  7. J.H Cho, I.Y. Jang, S.M. Kim, and K.H. Lee, "Depth Image Processing Technique for Representing Human Actors in 3DTV Using Single Depth Camera," Proc. of 3DTV Conference, pp. 1-4, May 2007.
  8. J. Zhu, L. Wang, R. Yang, and J. Davis, "Fusion of Time-of-Flight Depth and Stereo for High Accuracy Depth Maps," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 231-236, June 2008.
  9. SR4000 data sheet, Mesa Imaging, 2010.
  10. A. Frick, B. Bartczack and R. Koch, "3D-TV LDV Content Generation with a Hybrid TOF-Multi Camera Rig," Proc. of 3DTV Conference, pp. 1-4, June 2010.
  11. Y.S. Kang and Y.S. Ho, "Disparity Map Generation for Color Image Using TOF Depth Camera," Proc. of 3DTV Conference, pp. 1-4, May 2011.
  12. Q. Yang, K. Tan, B. Culbertson, and J. Apostolopoulos, "Fusion of Active and Passive Sensors for Fast 3D Capture," Proc. of IEEE International Workshop on Multimedia Signal Processing, pp. 1-6, Oct. 2010.
  13. Y. Ho and Y. Kang, "Multi-view Depth Generation using Multi-Depth Camera System," Proc. of International Conference on 3D Systems and Applications, pp. 67-70, May 2010.
  14. E. Lee and Y. Ho, "Generation of High-Quality Depth Maps Using Hybrid Camera System for 3-D Video," Journal of Visual Communication and Image Representation, vol. 22, issue 1, pp. 73-84, Jan. 2011. https://doi.org/10.1016/j.jvcir.2010.10.006