# Marigold — Diffusion-Based Monocular Depth and Surface Normal Estimation

> Marigold repurposes the Stable Diffusion image generator for monocular depth estimation, producing detailed depth and surface normal maps from single images with fine-grained boundary preservation.

## Install

Save in your project root:

# Marigold — Diffusion-Based Monocular Depth and Surface Normal Estimation

## Quick Use
```bash
pip install diffusers torch
python -c "
import torch
from diffusers import MarigoldDepthPipeline
pipe = MarigoldDepthPipeline.from_pretrained('prs-eth/marigold-depth-lcm-v1-0', torch_dtype=torch.float16).to('cuda')
from diffusers.utils import load_image
image = load_image('https://marigoldmonodepth.github.io/images/einstein.jpg')
result = pipe(image)
result.prediction[0].save('depth.png')
"
```

## Introduction
Marigold is a monocular depth estimation method that fine-tunes a latent diffusion model (Stable Diffusion) using only synthetic depth data. By leveraging the rich visual priors learned during text-to-image pre-training, Marigold produces depth maps with exceptional detail and generalization, earning a CVPR 2024 Oral presentation and Best Paper Award candidacy.

## What Marigold Does
- Estimates dense depth maps from a single RGB image
- Predicts surface normals from a single image using the same diffusion framework
- Preserves fine-grained detail at object boundaries and thin structures
- Generalizes zero-shot to diverse real-world scenes without domain-specific training data
- Supports ensemble inference for improved accuracy via multi-sample denoising

## Architecture Overview
Marigold builds on the Stable Diffusion v2 architecture. The RGB input image is encoded into the latent space, and a depth map is generated through an iterative denoising process conditioned on the image. The model is fine-tuned on synthetic depth data (Hypersim and Virtual KITTI) by treating depth prediction as an image-to-image translation task in latent space. A latent consistency model (LCM) variant reduces the denoising steps from 50 to as few as 1-4, enabling near real-time inference. Surface normal estimation uses the same approach with normal maps as the target modality.

## Self-Hosting & Configuration
- Install via `pip install diffusers` and load from Hugging Face Hub with `MarigoldDepthPipeline`
- Pre-trained checkpoints are available in float16 for memory-efficient inference
- Requires a CUDA GPU with at least 6 GB VRAM for the LCM variant
- Adjust `num_inference_steps` (1-50) and `ensemble_size` (1-10) to trade speed for quality
- Supports batch processing for multiple images in a single pipeline call

## Key Features
- Diffusion prior: inherits rich scene understanding from Stable Diffusion pre-training
- Synthetic-only fine-tuning: no real-world depth ground truth needed, yet generalizes broadly
- LCM variant enables 1-4 step inference for near real-time depth estimation
- Dual modality: same framework handles both depth and surface normal prediction
- Integrated into Hugging Face Diffusers for easy adoption and community support

## Comparison with Similar Tools
- **MiDaS** — discriminative model producing relative depth; Marigold achieves finer boundary detail through diffusion
- **Depth Anything** — large-scale discriminative depth model with strong generalization; faster but with less detailed boundaries
- **Depth Pro** — Apple's metric depth model; outputs absolute scale, while Marigold produces affine-invariant relative depth
- **GeoWizard** — concurrent diffusion-based geometry estimation; Marigold is more widely adopted with Hugging Face integration
- **ZoeDepth** — combines relative and metric depth estimation; Marigold focuses on maximizing detail quality

## FAQ
**Q: Does Marigold output metric depth?**
A: By default Marigold produces affine-invariant relative depth. A scale-and-shift alignment step can calibrate it to metric depth if ground truth is available.

**Q: How fast is the LCM variant?**
A: The LCM checkpoint runs in 1-4 denoising steps, taking approximately 0.5 seconds per image on a modern GPU compared to several seconds for the full diffusion model.

**Q: Can I use Marigold for 3D reconstruction?**
A: Yes. The depth and normal maps can be combined with camera parameters to generate point clouds or meshes for downstream 3D applications.

**Q: What license is Marigold released under?**
A: Marigold is released under the Apache 2.0 license. Model weights follow the Stable Diffusion license terms.

## Sources
- https://github.com/prs-eth/Marigold
- https://marigoldmonodepth.github.io/

---
Source: https://tokrepo.com/en/workflows/asset-937f4d38
Author: AI Open Source