# IP-Adapter — Image Prompt Adapter for Diffusion Models

> A lightweight adapter that enables pre-trained text-to-image diffusion models to generate images conditioned on a reference image prompt alongside text.

## Install

Save in your project root:

# IP-Adapter — Image Prompt Adapter for Diffusion Models

## Quick Use
```python
from diffusers import StableDiffusionPipeline
from ip_adapter import IPAdapter

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
ip_model = IPAdapter(pipe, "ip-adapter_sd15.bin")
images = ip_model.generate(pil_image=ref_img, prompt="a painting in the style", num_samples=1)
```

## Introduction
IP-Adapter is a lightweight adapter module from Tencent AI Lab that adds image prompt capability to pre-trained text-to-image diffusion models. It lets you guide image generation using a reference image in addition to or instead of text, enabling style transfer, subject-driven generation, and visual conditioning.

## What IP-Adapter Does
- Enables image-conditioned generation in Stable Diffusion models
- Supports combining image and text prompts for hybrid conditioning
- Transfers style, composition, or subject from a reference image
- Works with SD 1.5, SDXL, and other compatible diffusion models
- Provides fine-grained and full variants for different control levels

## Architecture Overview
IP-Adapter introduces a decoupled cross-attention mechanism. The original text cross-attention layers remain frozen, and a parallel set of cross-attention layers is added for image features. Image embeddings are extracted by a CLIP image encoder and injected through these new layers. Only the adapter weights are trained, keeping the base model intact.

## Self-Hosting & Configuration
- Compatible with the Hugging Face diffusers library
- Download adapter weights from the project Hugging Face page
- Requires a CLIP image encoder alongside the diffusion model
- GPU with at least 8 GB VRAM for SD 1.5 or 12 GB for SDXL
- Adjustable image prompt strength via a scaling parameter

## Key Features
- Lightweight adapter adding only a few trainable parameters
- Decoupled cross-attention preserving original text capabilities
- Multiple model variants: full, plus, and face-specific adapters
- Compatible with ControlNet and other diffusion add-ons
- Broad community adoption in ComfyUI and A1111 WebUI

## Comparison with Similar Tools
- **ControlNet** — spatial conditioning (edges, depth); IP-Adapter conditions on semantic content
- **InstantID** — face-specific identity transfer; IP-Adapter is more general-purpose
- **Style Transfer** — classical methods; IP-Adapter leverages diffusion model capabilities
- **Textual Inversion** — requires training per concept; IP-Adapter works zero-shot

## FAQ
**Q: Can I use IP-Adapter with SDXL?**
A: Yes, dedicated SDXL-compatible adapter weights are provided.

**Q: Does it replace text prompts entirely?**
A: No, it complements text prompts — you can use both simultaneously.

**Q: How many reference images does it need?**
A: A single reference image is sufficient for conditioning.

**Q: Can I combine it with ControlNet?**
A: Yes, IP-Adapter and ControlNet can be used together for layered control.

## Sources
- https://github.com/tencent-ailab/IP-Adapter
- https://ip-adapter.github.io/

---
Source: https://tokrepo.com/en/workflows/asset-dd3edd0e
Author: AI Open Source