LLM Council — Multi-Model Answer Aggregation by Karpathy

Introduction

LLM Council, created by Andrej Karpathy, sends your question to multiple LLMs simultaneously and synthesizes their responses into a single, higher-quality answer. The approach leverages model diversity to reduce hallucinations and improve factual accuracy.

What LLM Council Does

• Queries multiple LLMs in parallel with the same prompt
• Collects and compares responses across model providers
• Synthesizes a consensus answer from diverse model outputs
• Highlights areas of agreement and disagreement between models
• Supports configurable model lists and API endpoints

Architecture Overview

The system dispatches a user query to a configurable set of LLM APIs concurrently using async Python. Responses are collected and fed into a synthesis stage where a judge model evaluates, reconciles, and merges the answers. The final output represents a consensus view weighted by response quality and inter-model agreement.

Self-Hosting & Configuration

• Requires Python 3.10+ with asyncio support
• API keys for each model provider set via environment variables
• Model list is configurable in a JSON or YAML config file
• Supports OpenAI, Anthropic, Google, and OpenAI-compatible endpoints
• Rate limiting and retry logic are built into the dispatcher

Key Features

• Parallel multi-model querying for fast aggregate answers
• Automatic disagreement detection across model responses
• Configurable synthesis strategy (majority vote, judge model, merge)
• Cost tracking across multiple API providers
• Extensible to any OpenAI-compatible endpoint including local models

Comparison with Similar Tools

• LiteLLM — unified LLM proxy; Council adds multi-model aggregation and synthesis
• Portkey AI Gateway — routing and caching; Council focuses on answer quality via consensus
• OpenRouter — model access unification; Council runs models in parallel and merges results
• Manifest — cost-optimized routing; Council prioritizes answer accuracy over cost

FAQ

Q: How many models should I use? A: Three to five models provide good diversity without excessive cost; diminishing returns above seven.

Q: Does it work with local models? A: Yes, any OpenAI-compatible API endpoint works, including Ollama and vLLM.

Q: How is the final answer selected? A: A configurable synthesis step uses a judge model to merge and reconcile the best parts of each response.

Q: What is the latency impact? A: Queries run in parallel, so total latency is roughly that of the slowest model plus synthesis time.

Sources

• https://github.com/karpathy/llm-council
• https://github.com/karpathy/llm-council/blob/master/README.md