RGB-Pointmap Pretraining for Unified 3D Scene Understanding

A short video walkthrough of UniScene3D qualitative results and scene understanding behavior.

The scenes and text queries are drawn from wild, out-of-domain data, underscoring UniScene3D's strong generalization capability.

Abstract

Pretraining 3D encoders through alignment with Contrastive Language–Image Pre-training (CLIP) has emerged as a promising direction to learn generalizable representations for 3D scene understanding. In this paper, we propose UniScene3D, a transformer-based framework that learns unified 3D scene representations from multi-view RGB–Pointmap by leveraging the priors of a pretrained 2D foundation model. For robust RGB-Pointmap representation learning, we introduce novel cross-view geometric alignment and grounded view alignment to enforce geometry and semantic consistency across views. Extensive low-shot and task-specific fine-tuning across viewpoint grounding, scene retrieval, scene classification, and 3D visual question answering achieves state-of-the-art performance. These results establish UniScene3D as an effective framework for unified 3D scene understanding.

Key Takeaways

• Core Question: Unlike 2D vision, 3D scene understanding still lacks a generalizable encoder like CLIP, largely due to the scarcity of large-scale 3D pretraining data. This raises the question: can a 2D vision encoder be extended into a general 3D scene encoder without extensive 3D pretraining?
• Preliminary Finding: Pointmaps encode world-frame geometry like point clouds while preserving an image-like format compatible with 2D vision models. Our initial study shows that pretrained 2D vision weights are also beneficial for learning pointmap features.
• Model Contribution: UniScene3D extends pretrained CLIP models to learn unified 3D scene representations from pixel-aligned, multi-view RGB-Pointmaps by jointly encoding geometry and appearance.
• Key Training Idea: We introduce cross-view geometric alignment and grounded view alignment to enforce geometric and semantic consistency across viewpoints.
• Result: The learned representations effectively combine complementary information from images and pointmaps, generalize across diverse scenes, and transfer well to a broad range of downstream 3D tasks.

Method

UniScene3D takes multi-view image-pointmap pairs, fuses appearance and geometry at the input stage, and learns a unified representation with view-level and scene-level alignment objectives.

Figure 2. The model combines early RGB-geometry fusion with cross-view geometric alignment, grounded view alignment, view-level alignment, and scene-level alignment.

Qualitative Examples

These examples show how UniScene3D uses appearance and geometry together to retrieve views that match language queries.

‹

›

Use the arrow buttons to browse the qualitative visualization pages.

Evaluation

UniScene3D is evaluated on viewpoint grounding, scene retrieval, scene type classification, and 3D VQA.

Main Findings

Conclusion

UniScene3D is a 3D representation learning method that jointly models geometry and appearance from multi-view RGB-Pointmaps. By leveraging rich cross-view context, it introduces cross-view geometric alignment and grounded view alignment objectives for pretraining.

Across multiple 3D scene understanding benchmarks, UniScene3D achieves state-of-the-art performance while remaining robust to varying numbers of input views. Looking forward, a natural next step is to scale UniScene3D to larger model sizes and explore its use as a general vision backbone for 3D foundation models.

Citation

@inproceedings{mao2026uniscene3d,
  title     = {RGB-Pointmap Pretraining for Unified 3D Scene Understanding},
  author    = {Mao, Ye and Luo, Weixun and Huang, Ranran and Jing, Junpeng and Mikolajczyk, Krystian},
  booktitle = {arXiv},
  year      = {2026}
}