We propose semantic grid, a spatial 2D map of the environment around an autonomous vehicle consisting of cells which represent the semantic information of the corresponding region such as car, road, vegetation, bikes, etc. It consists of an integration of an occupancy grid, which computes the grid states with a Bayesian filter approach, and semantic segmentation information from monocular RGB images, which is obtained with a deep neural network. The network fuses the information and can be trained in an end-to-end manner. The output of the neural network is refined with a conditional random field. The proposed method is tested in various datasets (KITTI dataset, Inria-Chroma dataset and SYNTHIA) and different deep neural network architectures are compared.

This paper was recommended for publication in its revised form by editorial board member, Han Wang.

Keywords:

References

1. H. P. Moravec , Sensor fusion in certainty grids for mobile robots, AI Magazine 9(2) (1988) 61. Google Scholar
2. L. Rummelhard, A. Nègre and C. Laugier , Conditional monte carlo dense occupancy tracker, Intelligent Transportation Systems Conf. (IEEE, 2015), pp. 2485–2490. Crossref, Google Scholar
3. M. Perrollaz, J.-D. Yoder, A. Spalanzani and C. Laugier , Using the disparity space to compute occupancy grids from stereo-vision, Int. Conf. Intelligent Robots and Systems (IEEE, 2010), pp. 2721–2726. Crossref, Google Scholar
4. J. D. Adarve, M. Perrollaz, A. Makris and C. Laugier , Computing occupancy grids from multiple sensors using linear opinion pools, Int. Conf. Robotics and Automation (IEEE, 2012), pp. 4074–4079. Crossref, Google Scholar
5. O. Erkent, C. Wolf, C. Laugier, D. Gonzalez and V. Cano , Semantic grid estimation with a hybrid bayesian and deep neural network approach, Int. Conf. Intelligent Robots and Systems, (IEEE, 2018), pp. 1–8. Crossref, Google Scholar
6. Ö. Erkent, C. Wolf and C. Laugier , Semantic grid estimation with occupancy grids and semantic segmentation networks, Int. Conf. Control, Automation, Robotics and Vision (Singapore, 2018), pp. 1051–1056. Crossref, Google Scholar
7. P. Krahenbuhl and V. Koltun , Efficient inference in fully connected crfs with gaussian edge potentials, Conf. Neural Inf. Process. Syst. (Spain, 2011), pp. 109–117. Google Scholar
8. S. Ren, K. He, R. Girshick and J. Sun , Faster r-cnn: Towards real-time object detection with region proposal networks, Adv. in Neural Inf. Process. Syst. (2015) 91–99. Google Scholar
9. F. Tung and J. J. Little , MF3D: Model-free 3D semantic scene parsing, Int. Conf. Robotics and Automation (Singapore, 2017), pp. 4596–4603. Crossref, Google Scholar
10. P. Babahajiani, L. Fan, J. K. Kämäräinen and M. Gabbouj , Urban 3D segmentation and modelling from street view images and LiDAR point clouds, Mach. Vision Appl. 28(7) (2017) 1–16. Crossref, Google Scholar
11. J. Dequaire, P. Ondrúška, D. Rao, D. Wang and I. Posner , Deep tracking in the wild: End-to-end tracking using recurrent neural networks, Int. J. Robot. Res. (2017) p. 0278364917710543. Google Scholar
12. L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille , Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, (IEEE, 2017), pp. 99, Pattern Anal. Mach. Intell. Google Scholar
13. C. Coué, C. Pradalier, C. Laugier, T. Fraichard and P. Bessière , Bayesian occupancy filtering for multitarget tracking: An automotive application, Int. J. Robot. Res. 25(1) (2006) 19–30. Crossref, Google Scholar
14. B. Fulkerson, A. Vedaldi and S. Soatto , Class segmentation and object localization with superpixel neighborhoods, Int. Conf. Computer Vision (Japan, 2009), pp. 670–677. Crossref, Google Scholar
15. M. J. Shotton and R. Cipolla , Semantic texton forests for image categorization and segmentation, Conf. Computer Vision and Pattern Recoginition, (2008). Crossref, Google Scholar
16. J. Shotton, J. Winn, C. Rother and A. Criminisi , Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context, Int. J. Comput. Vision 81(1) (2009) 2–23. Crossref, Google Scholar
17. C. Farabet, C. Couprie, L. Najman and Y. LeCun , Learning Hierarchical Features for Scene Labeling, Pattern Anal. Mach. Intell. (2013). Crossref, Google Scholar
18. J. Long, E. Shelhamer and T. Darrell , Fully convolutional networks for semantic segmentation, Conf. Computer Vision and Pattern Recoginition (USA, 2015), pp. 3431–3440. Crossref, Google Scholar
19. V. Badrinarayanan, A. Kendall and R. Cipolla , Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Pattern Anal. Mach. Intell. 39(12) (2017) 2481–2495. Crossref, Google Scholar
20. O. Ronneberger, P. Fischer and T. Brox , U-net: Convolutional networks for biomedical image segmentation, Int. Conf. Medical Image Computing and Computer-Assisted Intervention (Germany, 2015), pp. 234–241. Crossref, Google Scholar
21. D. Fourure, R. Emonet, E. Fromont, D. Muselet, A. Tremeau and C. Wolf , Residual Conv-Deconv Grid Network for Semantic Segmentation, British Machine Vision Conf. (UK, 2017). Crossref, Google Scholar
22. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., Imagenet large scale visual recognition challenge, Int. J. Comput. Vision 115(3) (2015) 211–252. Crossref, Google Scholar
23. F. Yu and V. Koltun , Multi-scale context aggregation by dilated convolutions, Int. Conf. Learning Representations (ICLR) (Puerto Rico, 2016). Google Scholar
24. G. Lin, C. Shen, A. Van Den Hengel and I. Reid , Exploring context with deep structured models for semantic segmentation, IEEE Pattern Anal. Mach. Intell. 40(6) (2017) 1352–1366. Crossref, Google Scholar
25. H. Noh, S. Hong and B. Han , Learning deconvolution network for semantic segmentation, Int. Conf. Computer Vision (Chile, 2015), pp. 1520–1528. Crossref, Google Scholar
26. K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014). Google Scholar
27. S. Thrun, W. Burgard and D. Fox , Probabilistic Robotics (MIT press, Cambridge, MA, 2005), p. 02139. Google Scholar
28. M. Jaderberg, K. Simonyan, A. Zisserman and K. Kavukcuoglu , Spatial transformer networks, Conf. Neural Information Processing System (Canada, 2015), pp. 2017–2025. Google Scholar
29. C. Rother, V. Kolmogorov and A. Blake , Grabcut: Interactive foreground extraction using iterated graph cuts, ACM Trans. Graphics (TOG), 23(3), ACM (2004), 309–314. Crossref, Google Scholar
30. A. Geiger, P. Lenz and R. Urtasun , Are we ready for autonomous driving? The kitti vision benchmark suite, Conf. Computer Vision and Pattern Recoginition (USA, 2012), pp. 3354–3361. Crossref, Google Scholar
31. R. Zhang, S. A. Candra, K. Vetter and A. Zakhor , Sensor Fusion for Semantic Segmentation of Urban Scenes, Int. Conf. Robotics and Automation, (2015) 1850–1857. Crossref, Google Scholar
32. G. Ros, L. Sellart, J. Materzynska, D. Vazquez and A. M. Lopez , The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes, The IEEE Conf. Computer Vision and Pattern Recognition (CVPR) (USA, June 2016), pp. 3234–3243. Google Scholar
33. K. He, X. Zhang, S. Ren and J. Sun , Deep residual learning for image recognition, IEEE Conf. Computer Vision and Pattern Recoginition, (2016) 770–778. Crossref, Google Scholar
34. D. Eigen and R. Fergus , Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture, Int. Conf. Computer Vision, (2015) 2650–2658. Crossref, Google Scholar