Marigold — Diffusion-Based Monocular Depth and Surface Normal Estimation

Introduction

Marigold is a monocular depth estimation method that fine-tunes a latent diffusion model (Stable Diffusion) using only synthetic depth data. By leveraging the rich visual priors learned during text-to-image pre-training, Marigold produces depth maps with exceptional detail and generalization, earning a CVPR 2024 Oral presentation and Best Paper Award candidacy.

What Marigold Does

• Estimates dense depth maps from a single RGB image
• Predicts surface normals from a single image using the same diffusion framework
• Preserves fine-grained detail at object boundaries and thin structures
• Generalizes zero-shot to diverse real-world scenes without domain-specific training data
• Supports ensemble inference for improved accuracy via multi-sample denoising

Architecture Overview

Marigold builds on the Stable Diffusion v2 architecture. The RGB input image is encoded into the latent space, and a depth map is generated through an iterative denoising process conditioned on the image. The model is fine-tuned on synthetic depth data (Hypersim and Virtual KITTI) by treating depth prediction as an image-to-image translation task in latent space. A latent consistency model (LCM) variant reduces the denoising steps from 50 to as few as 1-4, enabling near real-time inference. Surface normal estimation uses the same approach with normal maps as the target modality.

Self-Hosting & Configuration

• Install via pip install diffusers and load from Hugging Face Hub with MarigoldDepthPipeline
• Pre-trained checkpoints are available in float16 for memory-efficient inference
• Requires a CUDA GPU with at least 6 GB VRAM for the LCM variant
• Adjust num_inference_steps (1-50) and ensemble_size (1-10) to trade speed for quality
• Supports batch processing for multiple images in a single pipeline call

Key Features

• Diffusion prior: inherits rich scene understanding from Stable Diffusion pre-training
• Synthetic-only fine-tuning: no real-world depth ground truth needed, yet generalizes broadly
• LCM variant enables 1-4 step inference for near real-time depth estimation
• Dual modality: same framework handles both depth and surface normal prediction
• Integrated into Hugging Face Diffusers for easy adoption and community support

Comparison with Similar Tools

• MiDaS — discriminative model producing relative depth; Marigold achieves finer boundary detail through diffusion
• Depth Anything — large-scale discriminative depth model with strong generalization; faster but with less detailed boundaries
• Depth Pro — Apple's metric depth model; outputs absolute scale, while Marigold produces affine-invariant relative depth
• GeoWizard — concurrent diffusion-based geometry estimation; Marigold is more widely adopted with Hugging Face integration
• ZoeDepth — combines relative and metric depth estimation; Marigold focuses on maximizing detail quality

FAQ

Q: Does Marigold output metric depth? A: By default Marigold produces affine-invariant relative depth. A scale-and-shift alignment step can calibrate it to metric depth if ground truth is available.

Q: How fast is the LCM variant? A: The LCM checkpoint runs in 1-4 denoising steps, taking approximately 0.5 seconds per image on a modern GPU compared to several seconds for the full diffusion model.

Q: Can I use Marigold for 3D reconstruction? A: Yes. The depth and normal maps can be combined with camera parameters to generate point clouds or meshes for downstream 3D applications.

Q: What license is Marigold released under? A: Marigold is released under the Apache 2.0 license. Model weights follow the Stable Diffusion license terms.

Sources

• https://github.com/prs-eth/Marigold
• https://marigoldmonodepth.github.io/