Special Issue: Selected Papers from 3rd IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR 2020); Guest Editors: M.-C. Hu and W. HürstNo Access

US2RO: Union of Superpoints to Recognize Objects

Marcel Tiator

Department of Media, University of Applied Sciences Düsseldorf, Düsseldorf 40476, Germany

E-mail Address: marcel.tiator@hs-duesseldorf.de

Search for more papers by this author

Anna Maria Kerkmann

Department of Computer Science, Heinrich Heine University Düsseldorf, Düsseldorf 40225, Germany

E-mail Address: anna.kerkmann@hhu.de

Search for more papers by this author

Christian Geiger

Department of Media, University of Applied Sciences Düsseldorf, Düsseldorf 40476, Germany

E-mail Address: geiger@hs-duesseldorf.de

Search for more papers by this author

, and

Paul Grimm

Computer Sciences, Fulda University of Applied Sciences, Fulda 36037, Germany

E-mail Address: paul.grimm@cs.hs-fulda.de

Search for more papers by this author

https://doi.org/10.1142/S1793351X21400146Cited by:0 (Source: Crossref)

Abstract

The creation of interactive virtual reality (VR) applications from 3D scanned content usually includes a lot of manual and repetitive work. Our research aim is to develop agents that recognize objects to enhance the creation of interactive VR applications. We trained partition agents in our superpoint growing environment that we extended with an expert function. This expert function solves the sparse reward signal problem of the previous approaches and enables to use a variant of imitation learning and deep reinforcement learning with dense feedback. Additionally, the function allows to calculate a performance metric for the degree of imitation for different partitions. Furthermore, we introduce an environment to optimize the superpoint generation. We trained our agents with 1182 scenes of the ScanNet data set. More specifically, we trained different neural network architectures with 1170 scenes and tested their performance with 12 scenes. Our intermediate results are promising such that our partition system might be able to assist the VR application development from 3D scanned content in near future.

Keywords:

References

1. Q. Wang, Y. Tan and Z. Mei , Computational methods of acquisition and processing of 3D point cloud data for construction applications, Arch. Comput. Methods Eng. 27 (2020) 479–499. Crossref, Web of Science, Google Scholar
2. C. R. Qi, H. Su, M. Kaichun and L. J. Guibas , PointNet: Deep learning on point sets for 3D classification and segmentation, in Proc Conf. Computer Vision and Pattern Recognition, IEEE, Honolulu, Hawaii, 2017. Google Scholar
3. C. R. Qi, L. Yi, H. Su and L. J. Guibas , PointNet $+ +$ : Deep hierarchical feature learning on point sets in a metric space, in Proc. Conf. Neural Information Processing Systems, IEEE, Honolulu, Hawaii, 2017. Google Scholar
4. S. Chen, L. Zheng, Y. Zhang, Z. Sun and K. Xu , VERAM: View-enhanced recurrent attention model for 3D shape classification, IEEE Trans. Visual. Comput. Graph. 25(12) (2019) 3244–3257. Crossref, Web of Science, Google Scholar
5. K. Zhang, M. Hao, J. Wang, C. W. de Silva and C. Fu, Linked dynamic graph CNN: Learning on point cloud via linking hierarchical features, arXiv:1904.10014 [cs] (2019). Google Scholar
6. D. Maturana and S. Scherer , VoxNet: A 3D convolutional neural network for real-time object recognition, in Proc. Int. Conf. on Intelligent Robots and Systems, IEEE, Hamburg, Germany, 2015. Crossref, Google Scholar
7. M. Tiator, C. Geiger and P. Grimm , Point cloud segmentation with deep reinforcement learning, in Proc. 24th European Conf. on Artificial Intelligence, IOS Press, Santiago de Compostela, Spain, 2020. Google Scholar
8. M. Tiator, C. Geiger and P. Grimm , Point cloud segmentation: Solving a perceptual grouping task with deep reinforcement learning, in Proc. 17th Workshop of Virtual and Augmented Reality, Gesellschaft für Informatik e.V., Trier, Germany, 2020. Google Scholar
9. J. Papon, A. Abramov, M. Schoeler and F. Worgotter , Voxel cloud connectivity segmentation - supervoxels for point clouds, in Proc. Conf. Computer Vision and Pattern Recognition, IEEE, Portland, Oregon, 2013. Crossref, Google Scholar
10. A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser and M. Niebner , ScanNet: richly-annotated 3D reconstructions of indoor scenes, in Proc. Conf. on Computer Vision and Pattern Recognition, IEEE, Honolulu, Hawaii, 2017. Crossref, Google Scholar
11. R. B. Rusu and S. Cousins , 3D is here: Point Cloud Library (PCL), in Proc. Int. Conf. Robotics and Automation, IEEE, Shanghai, China, 2011. Crossref, Google Scholar
12. Q.-Y. Zhou, J. Park and V. Koltun, Open3D: A modern library for 3D data processing, technical report (2018). Google Scholar
13. L. Landrieu and M. Boussaha , Point cloud oversegmentation with graph-structured deep metric learning, in Proc. Conf. Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, 2019, pp. 7432–7441. Crossref, Google Scholar
14. J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, Proximal policy optimization algorithms, arXiv: 1707.06347 [cs] (2017). Google Scholar
15. A. Richtsfeld, T. Morwald, J. Prankl, M. Zillich and M. Vincze , Segmentation of unknown objects in indoor environments, in Proc. Int. Conf. Intelligent Robots and Systems, IEEE, Algarve, Portugal, 2012. Crossref, Google Scholar
16. Y. Xu, L. Hoegner, S. Tuttas and U. Stilla , Voxel- and graph-based point cloud segmentation of 3D scenes using perceptual grouping laws, ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences IV-1/W1(1W1), 2017, pp. 43–50. Crossref, Google Scholar
17. Y. Toda, Z. Ju, H. Yu, N. Takesue, K. Wada and N. Kubota , Real-time 3D point cloud segmentation using growing neural gas with utility, in Proc. 9th Int. Conf. Human System Interactions, IEEE, Portsmouth, UK, 2016, pp. 418–422. Crossref, Google Scholar
18. M. Tiator, A. M. Kerkmann, C. Geiger and P. Grimm , Using semantic segmentation to assist the creation of interactive VR applications, in Proc. 3rd Int. Conf. Artificial Intelligence & Virtual Reality, IEEE, Utrecht, Netherlands, 2020. Crossref, Google Scholar
19. F. Engelmann, T. Kontogianni, A. Hermans and B. Leibe , Exploring spatial context for 3D semantic segmentation of point clouds, Int. Conf. Computer Vision Workshops, 2017, pp. 716–724. Crossref, Google Scholar
20. F. Engelmann, T. Kontogianni, J. Schult and B. Leibe , Know what your neighbors do: 3D semantic segmentation of point clouds, in European Conf. Computer Vision, 2018, pp. 395–409. Google Scholar
21. Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein and J. M. Solomon , Dynamic graph CNN for learning on point clouds, ACM Trans. Graph. 38 (2018) 1–12. Web of Science, Google Scholar
22. X. Ye, J. Li, H. Huang, L. Du and X. Zhang , 3D recurrent neural networks with context fusion for point cloud semantic segmentation, in European Conf. Computer Vision, 2018, pp. 415–430. Crossref, Google Scholar
23. G. Huang, Z. Liu, L. Van Der Maaten and K. Q. Weinberger , Densely connected convolutional networks, in Proc. Conf. Computer Vision and Pattern Recognition, IEEE, Honolulu, Hawaii, 2017, pp. 2261–2269. Crossref, Google Scholar
24. H. Li, Z. Xu, G. Taylor, C. Studer and T. Goldstein , Visualizing the loss landscape of neural nets, in Proc. 32nd Conf. Neural Information Processing Systems (Curran Associates, Montréal, 2018), pp. 6389–6399. Google Scholar
25. K. He, X. Zhang, S. Ren and J. Sun , Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, in Proc. Int. Conf. Computer Vision, IEEE, Santiago, Chile, 2015, pp. 1026–1034. Crossref, Google Scholar
26. S. K. Kumar, On weight initialization in deep neural networks, arXiv: 1704.08863 [cs] (2017). Google Scholar
27. R. J. Williams , Simple statistical gradient-following algorithms for connectionist reinforcement learning, Mach. Learn. 8(3–4) (1992) 229–256. Crossref, Web of Science, Google Scholar
28. R. Huang, W. Zhang, A. Kundu, C. Pantofaru, D. A. Ross, T. Funkhouser and A. Fathi , An LSTM approach to temporal 3D object detection in LiDAR point clouds, in Proc. 16th European Conf. Computer Vision, 12362 LNCS (Springer, 2020), pp. 266–282. Crossref, Google Scholar
29. M. Najibi, G. Lai, A. Kundu, Z. Lu, V. Rathod, T. Funkhouser, C. Pantofaru, D. Ross, L. S. Davis and A. Fathi , DOPS: Learning to detect 3D objects and predict their 3D shapes, in Proc. Conf. Computer Vision and Pattern Recognition, 2020, pp. 11910–11919. Crossref, Google Scholar
30. I. Armeni, S. Sax, A. R. Zamir and S. Savarese, Joint 2D-3D-semantic data for indoor scene understanding, arXiv: 1702.01105 [cs] (2017). Google Scholar
31. F. Poux, C. Mattes and L. Kobbelt , Unsupervised segmentation of indoor 3D point cloud: Application to object-based classification, Int. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences — ISPRS Archives, Vol. 44, 2020, pp. 111–118. Crossref, Google Scholar