# LLM Council — Multi-Model Answer Aggregation by Karpathy

> Open-source framework that queries multiple LLMs in parallel and aggregates their answers to produce higher-quality, more reliable responses.

## Install

Save as a script file and run:

# LLM Council — Multi-Model Answer Aggregation

## Quick Use
```bash
git clone https://github.com/karpathy/llm-council.git
cd llm-council
pip install -r requirements.txt
python council.py "What causes aurora borealis?"
```

## Introduction
LLM Council, created by Andrej Karpathy, sends your question to multiple LLMs simultaneously and synthesizes their responses into a single, higher-quality answer. The approach leverages model diversity to reduce hallucinations and improve factual accuracy.

## What LLM Council Does
- Queries multiple LLMs in parallel with the same prompt
- Collects and compares responses across model providers
- Synthesizes a consensus answer from diverse model outputs
- Highlights areas of agreement and disagreement between models
- Supports configurable model lists and API endpoints

## Architecture Overview
The system dispatches a user query to a configurable set of LLM APIs concurrently using async Python. Responses are collected and fed into a synthesis stage where a judge model evaluates, reconciles, and merges the answers. The final output represents a consensus view weighted by response quality and inter-model agreement.

## Self-Hosting & Configuration
- Requires Python 3.10+ with asyncio support
- API keys for each model provider set via environment variables
- Model list is configurable in a JSON or YAML config file
- Supports OpenAI, Anthropic, Google, and OpenAI-compatible endpoints
- Rate limiting and retry logic are built into the dispatcher

## Key Features
- Parallel multi-model querying for fast aggregate answers
- Automatic disagreement detection across model responses
- Configurable synthesis strategy (majority vote, judge model, merge)
- Cost tracking across multiple API providers
- Extensible to any OpenAI-compatible endpoint including local models

## Comparison with Similar Tools
- **LiteLLM** — unified LLM proxy; Council adds multi-model aggregation and synthesis
- **Portkey AI Gateway** — routing and caching; Council focuses on answer quality via consensus
- **OpenRouter** — model access unification; Council runs models in parallel and merges results
- **Manifest** — cost-optimized routing; Council prioritizes answer accuracy over cost

## FAQ
**Q: How many models should I use?**
A: Three to five models provide good diversity without excessive cost; diminishing returns above seven.

**Q: Does it work with local models?**
A: Yes, any OpenAI-compatible API endpoint works, including Ollama and vLLM.

**Q: How is the final answer selected?**
A: A configurable synthesis step uses a judge model to merge and reconcile the best parts of each response.

**Q: What is the latency impact?**
A: Queries run in parallel, so total latency is roughly that of the slowest model plus synthesis time.

## Sources
- https://github.com/karpathy/llm-council
- https://github.com/karpathy/llm-council/blob/master/README.md

---
Source: https://tokrepo.com/en/workflows/asset-5c5fc87c
Author: Script Depot