The team behind PyTorch has announced the release of PyTorch3D – a modular and efficient library for 3D deep learning.
PyTorch3D offers differentiable operators for 3D deep learning, a fast and modular differentiable renderer for meshes and point clouds as well as 3D data structures. According to Facebook AI developers who designed and built Pytorch3D, the reason for its introduction were the engineering challenges in 3D deep learning, which created a large gap between 2D and 3D deep learning over the years.
PyTorch3D supports batches of heterogeneous data and many of the differentiable operators are implemented with custom CUDA kernels in order to improve efficiency. The key feature of the new library is the rendering engine for point clouds and meshes. The engine was implemented in a scalable and modular way by decomposing the rendering pipeline into stages (rasterization, lighting, shading, and blending).
Compared to existing open-source packages for 3D deep learning and compared to naive PyTorch implementations, PyTorch3D provides a factor of x10 speedup with the optimized 3D operators. Researchers also experimented with unsupervised shape prediction and showed that PyTorch3D and its features improve the state-of-the-art performance for unsupervised 3D mesh and point cloud prediction on the ShapeNet dataset.
The implementation of PyTorch3D is open-source and can be found here. Details on all the new features and the implementation can be found in the paper published in arxiv.