Image ProcessingNo Access

Real-Time Calibration and Registration Method for Indoor Scene with Joint Depth and Color Camera

Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream Data North, China University of Technology, Beijing, P. R. China

State Key Laboratory of Virtual Reality Technology and Systems Beihang University, Beijing, P. R. China

E-mail Address: fqzhang@ncut.edu.cn

Search for more papers by this author

Tingshen Lei

Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream Data North, China University of Technology, Beijing, P. R. China

Search for more papers by this author

Jinhong Li

Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream Data North, China University of Technology, Beijing, P. R. China

Search for more papers by this author

Xingquan Cai

Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream Data North, China University of Technology, Beijing, P. R. China

Search for more papers by this author

Xuqiang Shao

State Key Laboratory of Virtual Reality Technology and Systems Beihang University, Beijing, P. R. China

School of Control and Computer Engineering, North China Electric Power University, Baoding, P. R. China

Search for more papers by this author

Jian Chang

National Centre for Computer Animation, Bournemouth University, Poole, UK

Search for more papers by this author

, and

Feng Tian

State Key Laboratory of Virtual Reality Technology and Systems Beihang University, Beijing, P. R. China

School of Computer and Information Technology, Northeast Petroleum University, Daqing, P. R. China

Search for more papers by this author

https://doi.org/10.1142/S0218001418540216Cited by:11 (Source: Crossref)

Abstract

Traditional vision registration technologies require the design of precise markers or rich texture information captured from the video scenes, and the vision-based methods have high computational complexity while the hardware-based registration technologies lack accuracy. Therefore, in this paper, we propose a novel registration method that takes advantages of RGB-D camera to obtain the depth information in real-time, and a binocular system using the Time of Flight (ToF) camera and a commercial color camera is constructed to realize the three-dimensional registration technique. First, we calibrate the binocular system to get their position relationships. The systematic errors are fitted and corrected by the method of B-spline curve. In order to reduce the anomaly and random noise, an elimination algorithm and an improved bilateral filtering algorithm are proposed to optimize the depth map. For the real-time requirement of the system, it is further accelerated by parallel computing with CUDA. Then, the Camshift-based tracking algorithm is applied to capture the real object registered in the video stream. In addition, the position and orientation of the object are tracked according to the correspondence between the color image and the 3D data. Finally, some experiments are implemented and compared using our binocular system. Experimental results are shown to demonstrate the feasibility and effectiveness of our method.

Keywords:

References

1. F. Arrigoni, B. Rossi, and A. Fusiello , Global registration of 3d point sets via lrs decomposition, in Part of the Lecture Notes in Computer Science Book Series, Vol. 9908 (Springer, Cham, 2016). Google Scholar
2. B. Bartczak, I. Schiller, C. Beder and R. Koch , Integration of a time-of-flight camera into a mixed reality system for handling dynamic scenes, moving view points and occlusions in real-time, in Proc. 4th Int. Symp. 3D Data Processing, Visualization and Transmission, PVT Workshop (Atlanta, GA, USA, 2008). Google Scholar
3. G. Bartili et al., Emergency medicine training with gesture driven interactive 3D simulations, in Proc. ACM Workshop on User Experience in e-learning and Augmented Technologies in Education, Nara, Japan, 2012, pp. 25–30. Google Scholar
4. Y. Bok, H. G. Jeon and I. S. Kweon , Geometric calibration of micro-lens-based light field cameras using line features, IEEE Trans. Pattern Anal. Mach. Intell. 39 (2) (2017) 287–300. Web of Science, Google Scholar
5. S. Bouaziz, A. Tagliasacchi and H. Li , Modern techniques and applications for real-time non-rigid registration, in Proc. SIGGRAPH ASIA Courses, 5–8 December 2016, ACM, New York, Article No. 11. Google Scholar
6. Z. Cheng, J. Ren, J. Shen and H. Miao , Building a large scale test collection for effective benchmarking of mobile landmark search, in Proc. Int. Conf. Multimedia Modeling, Springer, Berlin, Heidelberg, 2013, pp. 36–46. Google Scholar
7. J. C. K. Chow and D. D. Lichti , Photogrammetric bundle adjustment with self-calibration of the primesense 3D camera technology: Microsoft Kinect, IEEE Access, 22 (2013) 465–474. Web of Science, Google Scholar
8. A. I. Comport et al., Real-time marker-less tracking for augmented reality: the virtual visual serving framework, IEEE Trans. Vis. Comput. Graphics, 12 (4) (2006) 615–628. Web of Science, Google Scholar
9. W. Darwish et al., A new calibration method for commercial rgb-d sensors, Sensors, 17 (6) (2017) 1204. Web of Science, Google Scholar
10. F. Destelle et al., A Multi-modal 3D capturing platform for learning and preservation of traditional sports and games, in Proc. 23rd ACM Int. Conf. Multimedia, 26–30 October 2015, Brisbane, Australia, pp. 747–748. Google Scholar
11. GoPro. https://gopro.com/. Google Scholar
12. E. Guido, H. Matthijs and G. Rob , Automatic calibration of stationary surveillance cameras in the wild, in Eur. Conf. Computer Vision (ECCV), Springer, Cham, 2016, pp. 743–759. Google Scholar
13. K. Guo, F. Xu and X. Liu , Real-time geometry, albedo and motion reconstruction using a single RGBD camera, J. ACM Trans. Graphics, 36 (4) (2017) 44a. Google Scholar
14. C. Harris and M. Stephens , A combined corner and edge detector, in Proc. Fourth Alvey Vision Conf. Sheffield, UK, 1988, pp. 147–151. Google Scholar
15. C. D. Herrera, J. Kannala and J. Heikkila , Joint depth and color camera calibration with distortion correction, IEEE Trans. Pattern Anal. Mach. Intell. 34 (2012) 2058–2064. Web of Science, Google Scholar
16. S. Huang, X. Ying, J. Rong, Z. Shang and H. Zha , Camera calibration from periodic motion of a pedestrian, in IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 27–30 June 2016, Las Vegas, NV, pp. 3025–3033. Google Scholar
17. B. Huhle, T. Schairer, P. Jenke and W. Strasser , Robust non-local denoising of colored depth data, in IEEE Conf. Computer Vision and Recognition, Workshop on ToF-Camera Based Computer Vision, 23–28 June 2008, Anchorage, AK, USA, pp. 1–8. Google Scholar
18. IDS Imaging Development Systems GmbH, Obersulm, http://www.ids-imaging.com. Google Scholar
19. M. Iindner, A. Kolb and K. Hartmann , Data fusion of PMD-based distance-information and high resolution RGB-images, in IEEE Symp. Signals Circuits and Systems (ISSCS), 13–14 July 2007, Iasi, Romania, pp. 121–124. Google Scholar
20. M. Iindner and A. Kolb , Lateral and depth calibration of PMD-distance sensors, in Proc. Int. Symp. Visual Computing, LNCS (Springer, 2006), pp. 524–533. Google Scholar
21. H. Ishiguro, Y. Masashi and T. Saburo , Omni-directional stereo, IEEE Trans. Pattern Anal. Mach. Intell. 14 (2) (1992) 257–262. Web of Science, Google Scholar
22. H. G. Jeon et al., Accurate depth map estimation from a lenslet light field camera, in IEEE Int. Conf. Computer Vision and Pattern Recognition, 7–12 June 2015, Boston, MA, pp. 1547–1555. Google Scholar
23. A. Kolb, E. Barth, R. Koch and R. Larsen , Time-of-flight sensors in computer graphics, Comput. Graph. Forum, 29 (1) (2010) 141–159. Web of Science, Google Scholar
24. R. Lange, 3D Time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology, PhD dissertation, University of Siegen (2000). Google Scholar
25. J.-K. Lee, J. Yea, M.-G. Park and K.-J. Yoon , Joint layout estimation and global multi-view registration for indoor reconstruction, in 2017 IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 162–171. Google Scholar
26. T. Li et al., Reconstructing hand poses using visible light, Proc. ACM on Interaction, Mobile, Wearable and Ubiquitous Technol. 3 (1) (2017) 71–90. Google Scholar
27. C. Li, F. Zhong and X. Qin , Accurate 3D head pose estimation with noisy RGBD images, in Proc. 33rd Computer Graphics Int. (ACM, New York, 2016), pp. 37–40. Google Scholar
28. W. Lin et al., Intuitive 3D flight gaming with tangible objects, in Proc. ACM SIGGRAPH Posters, 24–28 July 2016, New York. Google Scholar
29. L. Ma, C. Kerl, J. Stueckler and D. Cremers , Cpa-slam: Consistent plane-model alignment for direct rgb-d slam, in 2016 IEEE Int. Conf. Robotics and Automation (ICRA), 16–21 May 2016, Stockholm, Sweden. Google Scholar
30. Meet Kinect for Windows, Available online: https://dev.windows.com/en-us/kinect, 2016. Google Scholar
31. D. Moreno, F. Calakli and G. Taubin , Unsynchronized structured light, ACM Trans. Graph. 34 (6) (2015) 178. Web of Science, Google Scholar
32. S. Nousias et al., Corner-based geometric calibration of multi-focus plenoptic cameras, in IEEE Int. Conf. Computer Vision (ICCV), Santiago, Chile, 2015, pp. 957–965. Google Scholar
33. OpenCV Tools, http://graphicon.ru/oldgr/en/research/calibration/opencv.html. Google Scholar
34. D. Scaramuzza, Omnidirectional Vision: From calibration to root motion estimation, Ph.D. thesis, Swiss Federal Institute of Technology Zurich (ETHZ) (2008). Google Scholar
35. G. Simon, A. W. Fitzgibbon and A. Zisserman , Augmented reality camera tracking with homo-graphics augmented reality, Proc. IEEE and ACM Int. Symp. 22 (6) (2000) 120–128. Google Scholar
36. J. Tai, S. Tsang, C. Lin and K. Song , Real-time image tracking for automatic traffic monitoring and enforcement application, Image Vis. Comput. 22 (6) (2004) 485–501. Web of Science, Google Scholar
37. S. Tang, Q. Zhu, W. Chen, W. Darwish, B. Wu, H. Hu and M. Chen , Enhanced RGB-D mapping method for detailed 3D indoor and outdoor modeling, Sensors, 16 (2016) 1589. Web of Science, Google Scholar
38. P. Tissainayagam and D. Suter , Assessing the performance of Comer detector for point feature tracking applications, J. Image Vis. Comput. 22 (8) (2004) 663–679. Web of Science, Google Scholar
39. Y.-T. Wang, C.-A. Shen and J.-S. Yang , Calibrated kinect sensors for robot simultaneous localization and mapping, in Proc. 2014 19th Int. Conf. Methods and Models in Automation and Robotics (MMAR) (IEEE, Miedzyzdroje, Poland, 2014). Google Scholar
40. K. Wang, G. Zhang and H. Bao , Robust 3D reconstruction with an RGB-D camera, IEEE Trans. Image Process. 23 (2014) 4893–4906. Web of Science, Google Scholar
41. C. Wu, A. Quigley and H. David , Out of sight: A toolkit for tracking occluded human joint positions, Pers. Ubiquitous Comput. 21 (1) (2017) 125–135. Web of Science, Google Scholar
42. Xtion PRO LIVE, Available online: https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/, 2016. Google Scholar
43. Z. Zhang , Flexible camera calibration by viewing a plane from unknown orientations, in Proc. 7th IEEE. Int. Conf. Comput. Vis. 20–27 September 1999, Corfu, Greece, pp. 666–673. Google Scholar
44. C. Zhang and Z. Zhang , Calibration between depth and color sensors for commodity depth cameras, in Proc. IEEE Int. Conf. Multimedia and Expo (ICME), 11–15 July 2011, Barcelona, Spain, pp. 47–64. Google Scholar
45. J. Zhu, I. Wang, R. Yang and J. Davis , Fusion of time-of-flight depth and stereo for high accuracy depth maps, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 23–28 June 2008, Anchorage, AK. Google Scholar