OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation

 Tong Wu2 , Jiarui Zhang1,3 , Xiao Fu1 , Yuxin Wang1,4 , Jiawei Ren5 , Liang Pan5 , 

Wayne Wu1 , Lei Yang1,3 , Jiaqi Wang1 , Chen Qian1 , Dahua Lin1,2, Ziwei Liu5

 1Shanghai Artificial Intelligence Laboratory, 2The Chinese University of Hong Kong, 3SenseTime Research, 

4Hong Kong University of Science and Technology,5S-Lab, Nanyang Technological University 

{wt020,dhlin}@ie.cuhk.edu.hk, zjr954@163.com, ywangom@connect.ust.hk, 

wuwenyan0503@gmail.com, {fuxiao,wangjiaqi,qianchen}@pjlab.org.cn, 

yanglei@sensetime.com, jiawei011@e.ntu.edu.sg, {liang.pan,ziwei.liu}@ntu.edu.sg

Abstract

Recent advances in modeling 3D objects mostly rely on synthetic datasets due to the lack of large-scale realscanned 3D databases. To facilitate the development of 3D perception, reconstruction, and generation in the real world, we propose OmniObject3D, a large vocabulary 3D object dataset with massive high-quality real-scanned 3D objects. OmniObject3D has several appealing properties: 1) Large Vocabulary: It comprises 6,000 scanned objects in 190 daily categories, sharing common classes with popBCorresponding authors. https://omniobject3d.github.io/ ular 2D datasets (e.g., ImageNet and LVIS), benefiting the pursuit of generalizable 3D representations. 2) Rich Annotations: Each 3D object is captured with both 2D and 3D sensors, providing textured meshes, point clouds, multiview rendered images, and multiple real-captured videos. 3) Realistic Scans: The professional scanners support highquality object scans with precise shapes and realistic appearances. With the vast exploration space offered by OmniObject3D, we carefully set up four evaluation tracks: a) robust 3D perception, b) novel-view synthesis, c) neural surface reconstruction, and d) 3D object generation. Extensive studies are performed on these four benchmarks, revealing 1 arXiv:2301.07525v2 [cs.CV] 11 Apr 2023 new observations, challenges, and opportunities for future research in realistic 3D vision.