# LLaVA — Large Language and Vision Assistant

> An open-source multimodal model that connects a vision encoder with a large language model for general-purpose visual and language understanding. LLaVA achieves strong results on multimodal benchmarks with a simple architecture.

## Install

Save the content below to `.claude/skills/` or append to your `CLAUDE.md`:

# LLaVA — Large Language and Vision Assistant

## Quick Use
```bash
git clone https://github.com/haotian-liu/LLaVA.git && cd LLaVA
pip install -e .
python -m llava.serve.cli 
  --model-path liuhaotian/llava-v1.5-7b 
  --image-file "https://example.com/photo.jpg"
```

## Introduction
LLaVA (Large Language-and-Vision Assistant) is a multimodal AI model that combines a pre-trained CLIP vision encoder with a large language model through a simple projection layer. It enables conversational interactions about images, documents, and visual content.

## What LLaVA Does
- Answers open-ended questions about images in natural language
- Describes, reasons about, and analyzes visual content in detail
- Supports multi-turn conversations with image context retention
- Provides a web-based demo, CLI interface, and API server
- Offers multiple model sizes from 7B to 34B parameters

## Architecture Overview
LLaVA connects a frozen CLIP ViT-L/14 vision encoder to a LLaMA or Vicuna language model via a trainable linear projection layer. Image features are projected into the language model's token embedding space and concatenated with text tokens. Training proceeds in two stages: first aligning vision-language features on image-caption pairs, then instruction tuning on multimodal conversations.

## Self-Hosting & Configuration
- Requires Python 3.10+ and PyTorch 2.0+
- Model weights are available on Hugging Face in multiple sizes
- 7B model needs approximately 16 GB VRAM; 13B needs 28 GB
- Gradio web UI available via `python -m llava.serve.gradio_web_server`
- Supports 4-bit and 8-bit quantization for reduced memory usage

## Key Features
- Simple two-stage training: alignment then instruction tuning
- Achieves competitive scores on MMBench, SEED-Bench, and other multimodal benchmarks
- Multiple model variants: LLaVA-1.5, LLaVA-1.6 (LLaVA-NeXT) with dynamic resolution
- Efficient training requiring only 1 day on 8x A100 GPUs for the full pipeline
- Supports both Gradio web interface and OpenAI-compatible API serving

## Comparison with Similar Tools
- **GPT-4V** — proprietary multimodal model with broader capabilities; LLaVA is fully open source and self-hostable
- **InternVL** — strong open-source alternative with different vision encoder choices
- **Qwen-VL** — Alibaba's multimodal model; competitive performance with different training data
- **MiniGPT-4** — earlier open multimodal approach; LLaVA offers simpler architecture and better performance

## FAQ
**Q: Can LLaVA process video?**
A: The base model handles single images. LLaVA-NeXT-Video extends the architecture to video frames.

**Q: What languages does LLaVA support?**
A: Primarily English, though the underlying LLM may handle other languages with reduced quality.

**Q: Can I fine-tune LLaVA on custom data?**
A: Yes. The repository includes scripts for both stages of training on custom image-text datasets.

**Q: How does LLaVA compare to commercial APIs?**
A: LLaVA-1.5-13B achieves results competitive with early GPT-4V on several benchmarks while running locally.

## Sources
- https://github.com/haotian-liu/LLaVA
- https://llava-vl.github.io/

---
Source: https://tokrepo.com/en/workflows/llava-large-language-vision-assistant-9fff047d
Author: AI Open Source