3D Gaussian Splatting — Real-Time Radiance Field Rendering

Introduction

3D Gaussian Splatting represents scenes using anisotropic 3D Gaussians that are optimized from multi-view images via differentiable rendering. Unlike neural radiance fields that require expensive ray marching, Gaussian splatting uses fast GPU rasterization to achieve real-time rendering while matching or exceeding NeRF quality.

What 3D Gaussian Splatting Does

• Reconstructs 3D scenes from calibrated multi-view photographs
• Renders novel viewpoints at 30+ FPS at 1080p resolution on consumer GPUs
• Optimizes position, covariance, color, and opacity of millions of 3D Gaussians
• Supports adaptive density control to add or remove Gaussians during training
• Exports trained scenes for real-time interactive viewers

Architecture Overview

The method initializes sparse 3D Gaussians from Structure-from-Motion point clouds. Each Gaussian stores position, anisotropic covariance (rotation + scale), opacity, and spherical harmonic color coefficients. A tile-based rasterizer projects and alpha-composites sorted Gaussians per pixel. Gradients flow through the differentiable rasterizer to optimize all parameters, while adaptive density control splits, clones, or prunes Gaussians based on view-space gradients and opacity.

Self-Hosting & Configuration

• Requires COLMAP-processed images (camera poses and sparse point cloud)
• CUDA 11.6+ and a GPU with 12+ GB VRAM for training typical scenes
• Training takes 20-40 minutes on an RTX 3090 for bounded scenes
• Configure iterations, densification intervals, and SH degree in training args
• Viewer application uses OpenGL for real-time scene exploration

Key Features

• Real-time rendering via efficient tile-based GPU rasterization
• Differentiable rendering enables end-to-end optimization from images
• Adaptive density control automatically refines scene representation
• Anisotropic Gaussians capture both fine detail and smooth surfaces
• Compact representation compared to voxel or MLP-based methods

Comparison with Similar Tools

• NeRF (Instant-NGP) — ray-marching MLP approach; slower rendering but compact model size
• Nerfstudio — framework for multiple NeRF methods; Gaussian splatting is faster at inference
• 3DGS variants (Mip-Splatting, SuGaR) — extensions addressing aliasing or mesh extraction
• Point-based rendering — earlier point splatting lacked differentiable optimization
• Plenoxels — voxel-based radiance fields; similar speed but more memory

FAQ

Q: What input data format is required? A: A set of images with camera poses from COLMAP, or any SfM tool that produces a compatible sparse reconstruction.

Q: How much disk space does a trained model use? A: Typically 50-500 MB depending on scene complexity and number of Gaussians.

Q: Can it handle dynamic scenes? A: The base method is for static scenes. Extensions like Dynamic 3D Gaussians add temporal modeling.

Q: What hardware is needed for real-time viewing? A: Any modern GPU supporting OpenGL 4.5 or Vulkan can render trained scenes in real-time.

Sources

• https://github.com/graphdeco-inria/gaussian-splatting
• https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/