# Hallo — Hierarchical Audio-Driven Portrait Animation

> A method for animating portrait images into talking videos with expressive facial movements and natural head motion, driven by audio input.

## Install

Save as a script file and run:

# Hallo — Hierarchical Audio-Driven Portrait Animation

## Quick Use
```bash
git clone https://github.com/fudan-generative-vision/hallo.git
cd hallo
pip install -r requirements.txt
python scripts/inference.py --source_image portrait.png --driving_audio speech.wav
```

## Introduction
Hallo generates realistic talking portrait videos from a single image and an audio clip. Developed at Fudan University, it uses a hierarchical audio-conditioned diffusion model to produce expressive facial animations with natural lip sync and head movement.

## What Hallo Does
- Animates a single portrait photo into a talking head video
- Generates expressive lip movements synchronized to speech
- Produces natural head poses and facial expression changes
- Supports various portrait styles including artistic illustrations
- Outputs high-resolution video with temporal consistency

## Architecture Overview
Hallo builds on a latent diffusion framework with hierarchical audio conditioning. Audio features are injected at multiple levels: low-level features drive lip movements while high-level features control head pose and expression. A temporal attention module ensures smooth frame-to-frame transitions, and a reference network preserves the source identity.

## Self-Hosting & Configuration
- Requires Python 3.10+ with PyTorch 2.x and CUDA
- GPU with at least 12 GB VRAM recommended
- Download pre-trained weights from the Hugging Face model hub
- Configure output resolution and frame rate in the inference script
- Supports both single-image and batch inference modes

## Key Features
- High-fidelity output rivaling commercial portrait animation tools
- Hierarchical conditioning for independent control of lips, expression, and pose
- Works on photos, paintings, and AI-generated portraits
- Temporal attention for artifact-free video output
- Clean codebase built on the diffusers library

## Comparison with Similar Tools
- **SadTalker** — uses 3D coefficients; Hallo uses diffusion for higher visual fidelity
- **Wav2Lip** — lip-only editing; Hallo generates full portrait animation
- **LivePortrait** — real-time focus; Hallo prioritizes output quality
- **AniPortrait** — similar approach; Hallo offers stronger expression diversity

## FAQ
**Q: What image types work best?**
A: Clear frontal or near-frontal portraits with visible facial features.

**Q: Can it handle long audio clips?**
A: Yes, it processes audio in segments and stitches results seamlessly.

**Q: Is it fast enough for real-time use?**
A: No, inference is offline; real-time use requires optimization or distillation.

**Q: Does it preserve the source identity well?**
A: Yes, a reference network ensures strong identity preservation.

## Sources
- https://github.com/fudan-generative-vision/hallo
- https://fudan-generative-vision.github.io/hallo/

---
Source: https://tokrepo.com/en/workflows/asset-dbac96d0
Author: Script Depot