World Scientific Annual Review of Artificial Intelligence

3D deep learning, i.e., deep learning with 3D data, has been attracting increasing attention due to its tremendous research values and broad real-world applications. In this paper, we first provide an overview of 3D data, its unique characteristics, and the challenges associated, as well as the wide range of applications of 3D data in various fields. We then introduce the fundamentals of 3D deep learning, including the basic concepts, the key technical challenges, the recent developments, and the emerging real-world applications of 3D deep learning.

Understanding 3D scenes, such as semantic segmentation and instance identification within point clouds, typically demands extensive annotated datasets. However, generating point-by-point labels is an overly laborious process. While recent techniques have been developed to train 3D networks with a minimal fraction of labeled points, our method, dubbed “One Thing One Click,” simplifies this by requiring just a single label per object. To effectively utilize these sparse annotations during network training, we’ve crafted an innovative self-training strategy. This involves alternating between training phases and label spreading, powered by a graph propagation module. Additionally, we integrate a relation network to create category-specific prototypes, improving pseudo label accuracy and steering the training process. Our approach also seamlessly integrates with 3D instance segmentation, incorporating a point-clustering technique. Our method demonstrates superior performance over other weakly supervised strategies for 3D semantic and instance segmentation, as evidenced by tests on both ScanNet-v2 and S3DIS datasets. Remarkably, the efficacy of our self-training method with limited annotations rivals that of fully supervised models. Codes and models are available at https://github.com/liuzhengzhe/One-Thing-One-Click.

Deep learning models are the state of the art for semantic segmentation of point clouds, the success of which relies on the availability of large-scale annotated datasets. However, it can be prohibitively costly to prepare such datasets. In this work, we propose a holistic active learning (AL) approach to maximize model performance given limited annotation budgets. We investigate the appropriate sample granularity for active selection under the realistic “click” measurement of annotation cost, and demonstrate that superpoint-based selection allows for most efficient usage of the limited budget, when compared with point-level, polygon-level and instance/shape-level selection. We further propose new objective for AL acquisition function and exploit local consistency constraints to boost the performance of our superpoint-based approach. We evaluate our methods on three benchmark datasets, ShapeNet and PartNet and S3DIS. The results demonstrate that AL is an effective strategy to address the high annotation costs in semantic point cloud segmentation.

Self-supervised learning methods are able to learn latent features from unlabeled data samples by designing pre-text tasks, and hence have attracted a great deal of interest in terms of sample-efficient learning. As a promising scheme of self-supervised learning, masked autoencoding has significantly advanced natural language processing and computer vision, but has not been introduced to point cloud yet. In this paper, we propose a novel scheme of masked autoencoders for 3D point cloud self-supervised learning, addressing the special challenges posed by point cloud, including leakage of location information and uneven information density. Concretely, we divide the input point cloud into irregular point patches and randomly mask them at a high ratio. Then, a standard Transformer-based autoencoder, with an asymmetric design and a shifting mask tokens operation, learns high-level latent features from unmasked point patches, aiming to reconstruct the masked point patches. Extensive experiments show that our approach is efficient during pre-training and generalizes well on various downstream tasks. Apart from the proposed method, we will also introduce potential directions for 3D point cloud self-supervised learning, including improvements for masked autoencoding, developments for point cloud scene understanding, etc.

3D point cloud deep learning has received significant attention due to its wide applications, such as augmented/mixed reality, autonomous vehicles/drones, and robots. Despite its remarkable accuracy, the computational cost and sparse nature of 3D point cloud deep learning models greatly hinder their use in real-world, latency-sensitive applications. In this paper, we develop efficient algorithms, systems, and hardware for 3D deep learning to overcome the computational challenges (especially sparsity) of 3D point cloud data, making it more applicable in a broader range of real-world scenarios. From the algorithm perspective, we introduce novel 3D building blocks (PVCNN and SPVNAS^∗, https://pvnas.mit.edu) that are tailor-made for point cloud data to reduce sparse and irregular overheads. On the system level, we develop a high-performance library (TorchSparse^∗, https://torchsparse.mit.edu) that can handle sparse and irregular workloads on general-purpose hardware, which is typically optimized for dense and regular workloads. Furthermore, we develop a specialized hardware accelerator (PointAcc^∗, https://pointacc.mit.edu) that can support various types of point cloud deep learning primitives and mitigate memory bottlenecks for sparse point cloud computing.

A key aspect of efficient point cloud processing is sampling, which significantly reduces computational overhead, memory requirements, and redundancy. In this paper, we first present a comprehensive review and in-depth analysis of existing representative heuristic-based and learning-based sampling methods, highlighting their merits and limitations. Subsequently, we introduce an efficient semantic segmentation method and an object detection method for large-scale 3D point clouds containing millions of points, with the point cloud sampling strategy serving as a crucial component. Through a detailed examination of the RandLA-Net and IA-SSD algorithms, we aim to provide valuable insights for future research and development in this domain. Our overarching goal is to advance the efficiency, accuracy, and scalability of point cloud processing techniques, ultimately benefiting a wide range of applications across various domains.

In this paper, we propose a new approach for learning shape implicit neural representations (INRs) from point cloud data that do not require normal vectors as input. We show that our method, which uses a soft constraint on the divergence of the distance function to the shape’s surface, can produce smooth solutions that accurately orient gradients to match the unknown normal at each point, even outperforming methods that use normal vectors directly. This work extends the latest work on divergence-guided sinusoidal activation INRs [Y. Ben-Shabat, C. H. Koneputugodage and S. Gould, Proc IEEE/CVF Conf Computer Vision and Pattern Recognition, 2022, pp. 19323–19332], to Gaussian activation INRs and provides extended theoretical analysis and results. We evaluate our approach on tasks related to surface reconstruction and shape space learning.

Deep neural networks for 3D point cloud classification, such as PointNet, have been demonstrated to be vulnerable to adversarial attacks. Current adversarial defenders often learn to denoise the (attacked) point clouds by reconstruction, and then feed them to the classifiers as input. In contrast to the literature, we propose a novel bilevel optimization framework for robust point cloud classification, where the internal optimization can effectively defend the adversarial attacks as denoising and the external optimization can learn the classifiers accordingly. As a demonstration, we further propose an effective and efficient instantiation of our approach, namely, Lattice Point Classifier (LPC), based on structured sparse coding in the permutohedral lattice and 2D convolutional neural networks (CNNs) that integrates both internal and external optimization problems into end-to-end trainable network architectures. We demonstrate state-of-the-art robust point cloud classification performance on ModelNet40 and ScanNet under seven different attackers. Our demo code is available at: https://github.com/Zhang-VISLab.

Exploiting geometric features is a common approach to enhance monocular 3D object detection. However, their performance is limited due to the absence of depth information. To address this limitation, an external depth estimator can be employed to predict depth, but this approach significantly reduces the efficiency and flexibility of the model. Instead of relying on a costly depth estimator, we propose a depth-aware monocular 3D object detector that is trained using augmented training data. Specifically, we utilize reference images and their corresponding depth maps to train an efficient rendering module, which synthesizes a variety of photo-realistic images with different virtual depths. By learning from these images, the detector adapts its features to depth variations. Furthermore, we introduce an auxiliary module that guides the network to learn more informative representations from the depth images. Both modules are removed after training, resulting in no additional computational overhead during the final deployment.

In this paper, we employ the latest developments in 3D semi-supervised learning to create cutting-edge deep learning models for 3D object detection and segmentation of buried structures in high-resolution X-ray semiconductor scans. We illustrate our approach to locating the region of interest of High Bandwidth Memory (HBM) structures and their individual components and identifying various defects. We showcase how semi-supervised learning is utilized to capitalize on the vast amounts of available unlabeled data to enhance both detection and segmentation performance. Additionally, we explore the benefits of contrastive learning in the data pre-selection for our detection model and multi-scale Mean-Teacher training paradigm in 3D semantic segmentation to achieve better performance compared to the state of the art. We also provide an objective comparison for metrology-based defect detection with a 3D classification network. Our extensive experiments have shown that our approach outperforms the state of the art by up to 16% on object detection and 7.8% on semantic segmentation. Our fully-automated custom metrology package shows a mean error of less than 2 $\mu$m for key features such as bond line thickness and provides better defect detection performance than the direct 3D classification approach. Overall, our method achieves state-of-the-art performance and can be used to improve the accuracy and efficiency of a wide range of failure analysis applications in semiconductor manufacturing. Finally, we also increase the segmentation models flexibility and adaptability to new data. We propose a generic training strategy and a new loss function that reduces the training time by 60% and the required amount of data by 48% making the training process more efficient.

Machine learning approaches have significantly advanced the 3D medical images analysis, such as the CT and MRI scans, which enables improved diagnosis and treatment evaluation. These image volumes provide rich spatial context for understanding the internal brain and body anatomies. Typical medical image analysis tasks, such as segmentation, reconstruction and registration, are essential for characterizing this context. Related to 3D data formats, meshes, point clouds and others are used to represent the anatomical structures, each with unique applications. To better capture the spatial information and address data scarcity, self- and semi-supervised learning methods have emerged. However, efficient 3D representation learning remains challenging. Recently, Transformers have shown promise, leveraging the self-attention mechanisms that perform well on transfer learning and self-supervised methods. These techniques are applied for medical domains without extensive manual labeling. This work explores data-efficient models, scalable deep learning, semantic context utilization and transferability in 3D medical image analysis. We also evaluated the foundational models, self-supervised pre- training, transfer learning and prompt tuning, thus advancing this critical field.