Run LLMs Locally

What's in this pack

These seven runners cover the full spectrum: from "I want a chat window on my laptop" to "I'm putting Llama 3 behind a load balancer for 10k QPS."

Why local matters in 2026

Three forces have collapsed the cost gap between cloud APIs and self-hosted inference.

First, model quality. Open weights from Meta (Llama), Mistral, Qwen, and DeepSeek now match GPT-4-class capability on most reasoning and coding tasks. The penalty for not paying OpenAI is no longer a quality penalty.

Second, hardware. A single RTX 4090 runs Llama 3 70B at usable speed via llama.cpp's GGUF Q4 quantization. Apple Silicon laggards finally got unified memory — an M3 Max runs 70B locally without thermal throttle. Even mid-range gaming laptops handle 8B models in real time.

Third, privacy and compliance. Healthcare, legal, finance, and EU GDPR-bound shops can't send PII to a third-party API. Local inference is the only legal path. The same applies to coding agents — most enterprises ban Cursor/Copilot from touching proprietary repos.

Install in one command

# Install the whole pack
tokrepo install pack/local-llm-runners

# Or pick the one runner you actually need
tokrepo install ollama
tokrepo install open-webui
tokrepo install tgi

Each asset's TokRepo page bundles the install command, the recommended config, and the model-pull command for the most common Llama / Qwen / DeepSeek weights.

Common pitfalls

• VRAM accounting: a "7B" model takes ~14 GB at FP16, ~4 GB at Q4. Always check the quantization file name before downloading.
• Context window vs RAM: a 32k context on a 7B model can use as much VRAM as the weights themselves. Lower the context if you OOM.
• Open WebUI on top of Ollama: Open WebUI talks OpenAI protocol, so you must enable the OpenAI compatibility endpoint on Ollama (OLLAMA_HOST=0.0.0.0) — many tutorials skip this.
• TGI vs vLLM: TGI shines for HuggingFace-hosted models with sharded weights; vLLM is faster for raw throughput. Don't pick TGI just because it's older.
• Model licensing: Llama 3 is permissive but not MIT. Check the license before commercial deployment, especially for downstream fine-tunes.

Relationship to other packs

The local-LLM-runners pack is the runtime layer. To make it useful end-to-end:

• Pair with the AI Second Brain pack — Logseq + Khoj indexing your notes against a local Ollama.
• Pair with LLM Eval & Guardrails to verify your local model isn't regressing vs the closed-source baseline.
• Pair with the Document AI Pipeline to feed PDFs into local inference instead of sending them to a vendor.

Together these three packs give you a fully air-gapped knowledge stack that never phones home. The boundary is clean: runners do inference, the eval pack scores quality, the second-brain pack handles retrieval, and the doc pipeline turns files into chunks. Mix and match by your privacy and latency targets, then layer Ollama or TGI underneath as the engine.

When to pick which runner

• Single-developer laptop, mostly chat: Ollama plus Jan as the UI. Five-minute install, GGUF Q4 weights, runs offline on the plane.
• Team behind a VPN, shared GPU server: TGI or vLLM behind a load balancer, Open WebUI as the team-facing front end with SSO. One model, many users, no per-seat OpenAI bill.
• Mobile app demo or browser-only inference: MLC-LLM. Compiles weights to WebGPU/Metal/Vulkan and runs without a server at all — useful for offline mobile prototypes.
• Research lab fine-tuning on consumer GPUs: Text Generation WebUI. Built-in LoRA training, ExLlama backend, exotic loaders for the half-broken model checkpoints HuggingFace ships every week.