Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set

NeurIPS 2024

Wenyuan Zhang1, Yu-Shen Liu1, Zhizhong Han2
1School of Software, Tsinghua University, 2Wayne State University
Paper Code

Abstract

It is vital to infer a signed distance function (SDF) for multi-view based surface reconstruction. 3D Gaussian splatting (3DGS) provides a novel perspective for volume rendering, and shows advantages in rendering efficiency and quality. Although 3DGS provides a promising neural rendering option, it is still hard to infer SDFs for surface reconstruction with 3DGS due to the discreteness, the sparseness, and the off-surface drift of 3D Gaussians. To resolve these issues, we propose a method that seamlessly merge 3DGS with the learning of neural SDFs. Our key idea is to more effectively constrain the SDF inference with the multi-view consistency. To this end, we dynamically align 3D Gaussians on the zero-level set of the neural SDF, and then render the aligned 3D Gaussians through the differentiable rasterization. Meanwhile, we update the neural SDF by pulling neighboring space to the pulled 3D Gaussians, which progressively refine the signed distance field near the surface. With both differentiable pulling and splatting, we jointly optimize 3D Gaussians and the neural SDF with both RGB and geometry constraints, which recovers more accurate, smooth, and complete surfaces with more geometry details. Our numerical and visual comparisons show our superiority over the state-of-the-art results on the widely used benchmarks.

Method

In this paper, we propose to seamlessly combine 3D Gaussians with the learning of neural SDFs. Our method provides a novel perspective to jointly learn 3D Gaussians and neural SDFs by more effectively using multi-view consistency and imposing geometry constraints.

Here is an overview of our method. We (a) pull 3D Gaussians onto the zero-level set for splatting, while (b) pulling the neighboring space onto the Gaussian disks for SDF inference. To better facilitate this procedure, we introduce three constraints: (c) push the Gaussians to become disks; (d) encourage the disk to be a tangent plane on the zero-level set; (e) constrain the query points to be pulled along the shortest path.

Visualization Results

Comparison on DTU Dataset

Visual comparisons on open surface reconstructions with error maps on DeepFashion3D dataset. Note that NeAT uses additional mask supervision. The transition from blue to yellow indicates small to large reconstruction errors.

Comparison on Tanks and Temples Dataset

Comparison on Mip-NeRF 360 Dataset

BibTeX

@inproceedings{zhang2024gspull,
      title = {Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set},
      author = {Wenyuan Zhang and Yu-Shen Liu and Zhizhong Han},
      booktitle = {Advances in Neural Information Processing Systems},
      year = {2024},
    }