Omnidirectional 3D Scene Reconstruction from Single Image

Yang, Ren; Li, Jiahao; Lu, Yan

Omnidirectional 3D Scene Reconstruction from Single Image

Ren Yang, Jiahao Li, Yan Lu

Microsoft Research Asia

Paper NeurIPS 2025

Omni3D reconstructs an omnidirectional 3D scene from a single input image, rendering novel views across the full front and back hemispheres.

Omni3D reconstructs a full omnidirectional 3D scene from a single image, enabling free rendering of novel views across the entire front and back hemispheres.

Abstract

Reconstruction of 3D scenes from a single image is a crucial step towards enabling next-generation AI-powered immersive experiences. However, existing diffusion-based methods often struggle with reconstructing omnidirectional scenes due to geometric distortions and inconsistencies across the generated novel views, hindering accurate 3D recovery. To overcome this challenge, we propose Omni3D, an approach designed to enhance the geometric fidelity of diffusion-generated views for robust omnidirectional reconstruction. Our method leverages priors from pose estimation techniques, such as MASt3R, to iteratively refine both the generated novel views and their estimated camera poses. Specifically, we minimize the 3D reprojection errors between paired views to optimize the generated images, and simultaneously, correct the pose estimation based on the refined views. This synergistic optimization process yields geometrically consistent views and accurate poses, which are then used to build an explicit 3D Gaussian Splatting representation capable of omnidirectional rendering. Experimental results validate the effectiveness of Omni3D, demonstrating significantly advanced 3D reconstruction quality in the omnidirectional space, compared to previous state-of-the-art methods.

Highlights

Omnidirectional from One Image

Omni3D lifts a single image into a complete omnidirectional 3D Gaussian Splatting scene, rendering high-quality novel views across the full sphere — not just forward-facing angles.

Pose-View Optimization

A synergistic PVO module iteratively refines both diffusion-generated views and their camera poses by minimizing 3D reprojection errors, yielding geometrically consistent views and accurate poses.

State-of-the-Art Quality

Omni3D outperforms prior methods on Tanks and Temples, Mip-NeRF 360, and DL3DV, improving PSNR by up to 1.45 dB while remaining efficient thanks to a parallelized design.

Method

Overall framework of Omni3D showing the three view-generation stages, the Pose-View Optimization module, and 3D Gaussian Splatting reconstruction.

(a) The overall Omni3D framework generates novel views in omnidirectional space across three stages. In each stage, generated views are refined by the Pose-View Optimization (PVO) module before the final stage represents the scene with 3DGS. (b) The pairwise iterative PVO module estimates camera poses and intrinsics, then overfits a lightweight network that optimizes each generated view by minimizing 3D reprojection errors, updating poses and intrinsics until convergence.

Starting from a single input image, Omni3D uses a Multi-View Diffusion (MVD) model to synthesize novel views along cardinal orbits, progressively covering the front, side, and back hemispheres across three stages. Because diffusion-generated views often suffer from geometric distortions and inconsistencies, each set of views is processed by the Pose-View Optimization (PVO) module.

PVO leverages strong geometric priors from pose estimation (MASt3R) and adopts a progressive pairing scheme with a sliding window (default window size N = 12). For each view pair, a lightweight network learns a homography, a flow map, and a residual to correct the generated view, while an iterative loop alternates between optimizing the view content (with poses fixed) and updating the camera poses and intrinsics (from the refined view). Poses reliably converge after three updates.

The pairing scheme also enables strong parallelism: independent view pairs sharing a reference are optimized concurrently, so the full pipeline requires only 24 serial PVO computations across all stages. Finally, the geometrically consistent views and accurate poses are used to train an explicit 3D Gaussian Splatting representation for free omnidirectional rendering.

Results

We evaluate omnidirectional 3D reconstruction on the Tanks and Temples, Mip-NeRF 360, and DL3DV datasets, rendering views from the reconstructed 3DGS at groundtruth camera poses. Omni3D consistently outperforms ZeroNVS, ViewCrafter, and LiftImage3D across all datasets and metrics.

Quantitative Comparison (Omnidirectional Space)

Method	Tanks and Temples			Mip-NeRF 360			DL3DV
Method	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
ZeroNVS	12.67	0.4647	0.7506	13.40	0.2413	0.8299	11.28	0.4725	0.7074
ViewCrafter	13.91	0.4714	0.5886	14.06	0.2420	0.7649	16.61	0.6185	0.3883
LiftImage3D	14.85	0.4841	0.5781	14.27	0.2491	0.6479	16.21	0.6020	0.4844
Omni3D (Ours)	16.30	0.5308	0.5166	15.89	0.2859	0.6369	17.08	0.6649	0.3348

Visual Comparison

Rendered views of Omni3D and compared approaches. Omni3D produces higher visual quality with fewer distortions and better geometric accuracy relative to the groundtruth.

User Study

In a user study, 10 non-expert raters scored reconstructed 3D scenes from 0 (poorest) to 10 (perfect) using rendered omnidirectional trajectory videos. Omni3D was clearly preferred, consistent with the numerical results.

Method	Tanks and Temples	Mip-NeRF 360	DL3DV
ZeroNVS	1.0	1.3	0.8
ViewCrafter	4.3	4.7	7.4
LiftImage3D	5.1	4.5	5.8
Omni3D (Ours)	7.6	7.9	8.2

Ablation Study

Ablation showing 3D-reprojected views before and after Pose-View Optimization, with geometric errors corrected after PVO.

3D-reprojected views before (left) and after (right) PVO. PVO corrects geometric errors (highlighted in red), improving consistency between optimized and reference views — crucial for accurate 3DGS reconstruction.

Effect of PVO & Generalization

Setting	PSNR↑	SSIM↑	LPIPS↓
Omni3D w/o PVO	15.56	0.5198	0.5346
Omni3D	16.30	0.5308	0.5166
LiftImage3D	14.85	0.4841	0.5781
LiftImage3D + PVO	15.28	0.4964	0.5446

Pose-Update Iterations in PVO

Iterations	PSNR↑	SSIM↑	LPIPS↓
0 (w/o PVO)	15.56	0.5198	0.5346
1	15.62	0.5207	0.5325
2	15.91	0.5254	0.5296
3 (default)	16.30	0.5308	0.5166
4	16.33	0.5311	0.5162

PVO improves PSNR by 0.74 dB over the no-PVO baseline and generalizes to other MVD backbones (e.g., LiftImage3D). Pose estimates converge after three updates.

Efficiency

Thanks to the parallelized PVO design, Omni3D reconstructs a full omnidirectional 3DGS scene in about 34 minutes on 8×A100 GPUs — faster than ZeroNVS and LiftImage3D — while covering the entire omnidirectional space rather than only forward-facing views.

Method (8×A100)	MVD	Pose calc.	3DGS	Total
ZeroNVS	–	–	–	133.7 min
ViewCrafter	2.1 min	–	12.8 min	14.9 min
LiftImage3D	3.5 min	1.5 min	67.4 min	72.4 min
Omni3D (Ours)	10.8 min	10.5 min	12.8 min	34.1 min

BibTeX

@inproceedings{Yang2025Omni3D,
  title     = {Omnidirectional 3D Scene Reconstruction from Single Image},
  author    = {Yang, Ren and Li, Jiahao and Lu, Yan},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2025},
}