Kimia Noorbakhsh
Author
Introduction
With major recent developments in 3D sensors, large-scale 3D geometry datasets, and advances in deep 3D learning, 3D computer vision is playing an increasingly important role in many fields. This includes a varied range of cutting-edge applications spanning robotics, autonomous navigation and localization, 3D scene reconstruction, 3D scene understanding, 3D tracking and surveillance, digital city modeling, and virtual and augmented reality.
The goal of this Research Topic is to present high-quality research on 3D computer vision that addresses challenging problems in 3D data processing, proposes new developments for reliable 3D data acquisition, and advances the state of the art in 3D applications.
Some of the areas of focus are:
- 3D analysis and modeling (3D deep learning and 3D machine learning, 3D recognition, 3D Segmentation, 3D detection, 3D registration)
- 3D acquisition (calibration, structure from motion and SLAM, computational photography, 3D reconstruction)
- 3D applications (robotics, medical applications, sports applications, digital fabrication)
In this notebook, we will focus on the first application.
3D Data Representation
- 3D Point Clouds
- 3D Models
- 3D Mesh
- Voxel-based models
- Parametric Model (CAD)
- Other representations
Show a code to read the ModelNet40 dataset and visualize some of its samples.
In [ ]:
# pointnet
Deep Learning on Unsorted Sets (The rise of Pointnet)
From a data structure point of view, a point cloud is an unordered set of vectors. While most works in deep learning focus on regular input representations like sequences (in speech and language processing), images, and volumes (video or 3D data), not much work has been done in deep learning on point sets.
Pointnet Architecture:
Pointnet is a deep learning framework that directly consumes unordered point sets as inputs. A point cloud is represented as a set of 3D points ${P_i| i = 1, \dots, n}$, where each point $P_i$ is a vector of its ($x, y, z$) coordinate plus extra feature channels such as color, normal, etc. For simplicity and clarity, unless otherwise noted, we only use the ($x, y, z$) coordinate as our point’s channels.
For the object classification task, the input point cloud is either directly sampled from a shape or pre-segmented from a scene point cloud. Our proposed deep network outputs $k$ scores for all the $k$ candidate classes. For semantic segmentation, the input can be a single object for part region segmentation or a sub-volume from a 3D scene for object region segmentation. Our model will output $n \times m$ scores for each of the n points and each of the $m$ semantic subcategories.
In [ ]:
# Show the code of pointnet archtecture and evaluate a pre-trained model on the ModelNet40 data.
Introduce other useful point cloud models and link to their papers.
3D Adversarial Attacks
Reference to the original 3D adversarial attack paper and introduce the procedure.
Papers and Labs regarding 3D Computer vision
List of papers:
Labs and groups working on 3D Computer Vision:
References
- https://towardsdatascience.com/how-to-represent-3d-data-66a0f6376afb
- Reference 2
In [ ]:
