# AnimateDiff — Plug-and-Play Animation for Diffusion Models

> A plug-and-play motion module that turns community text-to-image Stable Diffusion models into animation generators without additional training. ICLR 2024 Spotlight paper.

## Install

Save the content below to `.claude/skills/` or append to your `CLAUDE.md`:

# AnimateDiff — Plug-and-Play Animation for Diffusion Models

## Quick Use
```bash
pip install diffusers torch
python -c "
from diffusers import AnimateDiffPipeline, MotionAdapter, DDIMScheduler
adapter = MotionAdapter.from_pretrained('guoyww/animatediff-motion-adapter-v1-5-3')
pipe = AnimateDiffPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', motion_adapter=adapter)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
output = pipe('a photo of a cat walking', num_frames=16)
output.frames[0][0].save('frame0.png')
"
```

## Introduction
AnimateDiff is a motion module framework that adds temporal animation capabilities to existing Stable Diffusion models. Instead of training a video model from scratch, AnimateDiff inserts learnable motion modules into frozen text-to-image models, enabling any community checkpoint or LoRA to generate animated sequences while preserving its visual style.

## What AnimateDiff Does
- Adds temporal motion to any Stable Diffusion 1.5 or SDXL checkpoint without retraining
- Generates short animated sequences (typically 16-32 frames) from text prompts
- Preserves the visual style of base models, LoRAs, and textual inversions during animation
- Supports MotionLoRA for training custom motion patterns with minimal data
- Integrates with ComfyUI and AUTOMATIC1111 WebUI via community extensions

## Architecture Overview
AnimateDiff inserts temporal attention layers (motion modules) between the spatial self-attention blocks of a frozen Stable Diffusion UNet. These modules learn motion dynamics from video data while the original image model weights remain unchanged. At inference, the motion modules coordinate frame-to-frame consistency, producing coherent animations. The plug-and-play design means one trained motion module works across thousands of community model variants.

## Self-Hosting & Configuration
- Install via pip with diffusers: `pip install diffusers[torch]`
- Download motion adapter weights from Hugging Face (v1.5 or v2 variants)
- Combine with any SD 1.5 checkpoint: community models, custom LoRAs, and embeddings all work
- Configure frame count, FPS, and guidance scale for desired animation length and style
- Use ComfyUI-AnimateDiff-Evolved for a visual node-based animation workflow

## Key Features
- Works with thousands of existing community Stable Diffusion models out of the box
- No video training data needed to animate a specific model checkpoint
- MotionLoRA enables custom motion training with as few as 50 video clips
- Native Hugging Face diffusers integration for programmatic use
- Active ecosystem of ComfyUI and WebUI extensions with advanced controls

## Comparison with Similar Tools
- **CogVideo** — dedicated video generation model trained end-to-end; AnimateDiff retrofits animation onto existing image models
- **Stable Video Diffusion** — image-to-video from Stability AI; AnimateDiff offers text-to-animation with community model compatibility
- **Open-Sora** — Sora-style video generation; AnimateDiff is lighter and integrates with the existing SD ecosystem
- **Deforum** — frame-by-frame animation via prompt interpolation; AnimateDiff learns actual motion dynamics for smoother results
- **Wan2.1** — standalone video generator; AnimateDiff uniquely preserves the style of any base image model

## FAQ
**Q: Does AnimateDiff work with SDXL models?**
A: Yes. AnimateDiff v3 and community adapters support SDXL, though SD 1.5 adapters have more options and are more mature.

**Q: How many frames can I generate?**
A: The default motion modules handle 16-32 frames well. Longer sequences are possible with sliding window approaches like SparseCtrl or FreeNoise.

**Q: Can I use ControlNet with AnimateDiff?**
A: Yes. SparseCtrl and community extensions allow combining ControlNet conditioning with AnimateDiff for controlled animations guided by depth maps, poses, or edges.

**Q: What resolution and FPS are typical outputs?**
A: Standard output is 512x512 at 8 fps for SD 1.5. Higher resolutions are possible with SDXL adapters. Output can be interpolated to higher FPS with frame interpolation tools.

## Sources
- https://github.com/guoyww/AnimateDiff
- https://animatediff.github.io/

---
Source: https://tokrepo.com/en/workflows/animatediff-plug-play-animation-diffusion-models-04d7fee0
Author: AI Open Source