SPECIAL ISSUE: Video Analysis and Understanding for Surveillance Applications; Edited by L. Wang, Q. Wu, M. Li, J. Gonzàlez and X. GengNo Access

GEO-BASED AERIAL SURVEILLANCE VIDEO PROCESSING FOR SCENE UNDERSTANDING AND OBJECT TRACKING

JIANGJIAN XIAO

Sarnoff Corporation, 201 Washington Road, Princeton, NJ 08536, USA

Search for more papers by this author

HUI CHENG

Sarnoff Corporation, 201 Washington Road, Princeton, NJ 08536, USA

Search for more papers by this author

FENG HAN

Sarnoff Corporation, 201 Washington Road, Princeton, NJ 08536, USA

Search for more papers by this author

, and

HARPREET SAWHNEY

Sarnoff Corporation, 201 Washington Road, Princeton, NJ 08536, USA

Search for more papers by this author

https://doi.org/10.1142/S0218001409007582Cited by:3 (Source: Crossref)

Abstract

This paper presents an approach to extract semantic layers from aerial surveillance videos for scene understanding and object tracking. The input videos are captured by low flying aerial platforms and typically consist of strong parallax from non-ground-plane structures as well as moving objects. Our approach leverages the geo-registration between video frames and reference images (such as those available from Terraserver and Google satellite imagery) to establish a unique geo-spatial coordinate system for pixels in the video. The geo-registration process enables Euclidean 3D reconstruction with absolute scale unlike traditional monocular structure from motion where continuous scale estimation over long periods of time is an issue. Geo-registration also enables correlation of video data to other stored information sources such as GIS (Geo-spatial Information System) databases. In addition to the geo-registration and 3D reconstruction aspects, the other key contributions of this paper also include: (1) providing a reliable geo-based solution to estimate camera pose for 3D reconstruction, (2) exploiting appearance and 3D shape constraints derived from geo-registered videos for labeling of structures such as buildings, foliage, and roads for scene understanding, and (3) elimination of moving object detection and tracking errors using 3D parallax constraints and semantic labels derived from geo-registered videos. Experimental results on extended time aerial video data demonstrates the qualitative and quantitative aspects of our work.

Keywords:

References

S. Ali, V. Reilly and M. Shah, Computer Vision and Pattern Recognition 1 (2007), DOI: 10.1109/CVPR.2007.383070. Google Scholar
C. Baillardet al., ISPRGIS 69 (1999). Google Scholar
T. Belli, M. Cord and M. Jordan, 3d data reconstruction and modeling for urban scene analysis, Workshop of Automatic Extraction of Man-Made Objects from Aerial and Space Images (III) (2001) pp. 125–134. Google Scholar
N. Haala and C. Brenner, ISPRS J. Photogrammetry and Rem. Sens. 54, 130 (1999), DOI: 10.1016/S0924-2716(99)00010-6. Crossref, Web of Science, Google Scholar
R. I. Hartley and A. Zisserman , Multiple View Geometry in Computer Vision , 2nd edn. ( Cambridge University Press , 2004 ) . Crossref, Google Scholar
J. Kanget al., Detection and tracking of moving objects from a moving platform in presence of strong parallax, IEEE Int. Conf. Computer Vision (2005) pp. 10–17. Google Scholar
R. Kasturi, Performance evaluation protocol for face, person and vehicle detection & tracking in video analysis and content extraction, Computer Science and Engineering, University of South Florida, Tampa, FL, USA, January 2006 . Google Scholar
V. Kolmogorov and R. Zabih, Multi-camera scene reconstruction via graph cut, Proc. ECCV (2002) pp. 82–96. Google Scholar
D. Lowe, Int. J. Comput. Vis. 60, 91 (2004), DOI: 10.1023/B:VISI.0000029664.99615.94. Crossref, Web of Science, Google Scholar
A. Pereraet al., Computer Vision and Pattern Recognition 201 (2006). Google Scholar
A. Pereraet al., Computer Vision and Pattern Recognition 201 (2006). Google Scholar
J. Porwayet al., Computer Vision and Pattern Recognition 1 (2008), DOI: 10.1109/CVPR.2008.4587359. Google Scholar
H. Sawhney, Y. Guo and R. Kumar, IEEE Trans. Patt. Anal. Mach. Intell. 22, 1191 (2000), DOI: 10.1109/34.879803. Crossref, Web of Science, Google Scholar
H. Sawhney, S. Hsu and R. Kumar, IEEE Trans. Patt. Anal. Mach. Intell. 22, 1191 (2000), DOI: 10.1109/34.879803. Crossref, Web of Science, Google Scholar
H. Shum and R. Szeliski, Int. J. Comput. Vis. 16(1), 63 (2000). Web of Science, Google Scholar
H. Tanget al., Dynamic 3d urban scene modeling using multiple pushbroom mosaics, Third Int. Symp. 3D Data Processing, Visualization and Transmission (2006) pp. 456–463. Google Scholar
C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, Int. Conf. Comput. Vis. (1998) pp. 839–846. Google Scholar
Z. Tu and S. Zhu, IEEE Trans. Patt. Anal. Mach. Intell. 24, 657 (2002). Web of Science, Google Scholar
V. Verma, R. Kumar and S. Hsu, Computer Vision and Pattern Recognition 2213 (2006). Google Scholar
J. Xiao and M. Shah, IEEE Trans. Patt. Anal. Mach. Intell. 27(10), 972 (2005). Google Scholar
J. Xiaoet al., Vehicle and person tracking in UAV videos, Multimodal Technologies for Perception of Humans: International Evaluation Workshops CLEAR and RT (2007) pp. 203–214. Google Scholar
A. Yilmaz, O. Javed and M. Shah, ACM J. Comput. Surv. 38, (2006). Google Scholar
Z. Zhang, IEEE Trans. Patt. Anal. Mach. Intell. 22(11), 1330 (2001), DOI: 10.1109/34.888718. Crossref, Web of Science, Google Scholar