Privacy-First Local AI — Pile IA 100% Locale

What's in this pack

This pack is for the buyer who reads "we may use your data to improve our service" in a vendor contract and walks away. Law firms working privileged matters. Hospitals handling PHI. Government agencies on air-gapped networks. Cross-border traders whose contract terms can't sit on an OpenAI server in Texas. The whole point is no network call to anyone's API at inference time.

The stack splits into three layers plus the protocol glue:

• Runtime layer — the model engine that loads weights and serves tokens. Ollama, LM Studio, MLC-LLM live here.
• UI layer — what a non-technical lawyer or doctor actually opens. Open WebUI, Jan, GPT4All, LibreChat, Text Generation WebUI are all variants of "local ChatGPT."
• RAG layer — connects the model to your own documents (case files, patient charts, customs declarations). Khoj indexes a folder and serves it back to any local model.
• Speech layer — Faster Whisper transcribes meetings, depositions, and consultations locally so the audio never reaches a cloud STT vendor.

This is the privacy-first sibling of our local-first-ai pack (which optimizes for a developer's personal rig) and local-llm-runners (which compares engines side-by-side). This one is opinionated about the regulated-industry deployment shape — what to install when the buyer's compliance officer is the one signing off.

Install in this order

1. Ollama — start here. Single binary, runs on macOS / Linux / Windows, pulls a model with ollama pull llama3.1 and serves an OpenAI-compatible HTTP API on localhost. Everything downstream — UI, RAG, agents — can be pointed at Ollama as if it were OpenAI, which means you can swap any cloud setup to fully local by changing one base URL.
2. LM Studio — the GUI alternative to Ollama for non-CLI users. Same job (load GGUF weights, serve a local OpenAI-compatible endpoint), but with a desktop app for model search, download, and chat. Use this on a lawyer's or doctor's workstation; use Ollama on the server.
3. MLC-LLM — when Ollama is too slow on your hardware. Compiles models with TVM for native execution on Apple Silicon, NVIDIA, AMD, and even WebGPU. Worth the extra setup pain if your throughput target is real-time chat for 10+ concurrent users on a single workstation.
4. Open WebUI — the ChatGPT clone. Docker-compose, points at Ollama's port, gives you multi-user accounts, chat history, model picker, document upload. This is what your end users actually open. The most-deployed local chat UI in regulated environments because of the per-user permission model.
5. Jan — desktop app version of Open WebUI's idea. Cross-platform Electron client with built-in model downloader. Use this when each user gets their own laptop and you don't want to run a central server.
6. GPT4All — same niche as Jan, different lineage (Nomic). Slightly smaller binary, opinionated default model picks, runs on CPU-only machines reasonably well. Worth installing on the ancient ThinkPads still on every loan officer's desk.
7. LibreChat — the multi-provider chat UI for organizations that want one frontend across local + (carefully whitelisted) cloud. Useful in hybrid setups: privileged matters route to Ollama, general research routes to a contract-bound cloud provider. Per-user routing rules.
8. Text Generation WebUI — the power-user fallback. Supports more model formats than anyone else (GGUF, GPTQ, AWQ, EXL2, transformers), exposes raw sampling parameters, and runs LoRA adapters. Install this when a researcher needs to fine-tune a model on internal data and still keep the workflow private.
9. Khoj — the RAG layer. Watches a folder (case files, patient charts, customs PDFs), embeds them locally, serves them back via a chat UI or an API any other tool can call. Configure it to use Ollama for both embeddings and generation and your documents never leave the box.
10. Faster Whisper — meeting / deposition / consultation transcription. 4x faster than vanilla Whisper at the same accuracy. Pipe its output into Khoj and you get a fully local "AI that joined the meeting" without sending audio to AssemblyAI, Deepgram, or OpenAI.

How they fit together

   ┌──────────────────────────────────────────────────────────┐
   │  Your workstation / on-prem server (no outbound network) │
   └──────────────────────────────────────────────────────────┘

   Laptop GPU / server GPU / Apple Silicon NPU
             │
             ▼
   ┌──────────────────────────────┐
   │  Runtime: Ollama / LM Studio │  ◄── MLC-LLM if you need
   │  (OpenAI-compatible API)     │      real-time multi-user
   └──────────────────────────────┘
             │
   ┌─────────┴─────────────────────────┐
   ▼                                   ▼
  Chat UIs                          RAG layer
   ├─ Open WebUI  (browser, multi-user)  Khoj  ◄── folder watch
   ├─ Jan  (desktop client)               │       (case files,
   ├─ GPT4All  (low-spec laptops)         │        patient docs,
   ├─ LibreChat  (hybrid routing)         │        customs PDFs)
   └─ Text Generation WebUI (power user)  ▼
                                       Local vector DB
                                       (SQLite / Qdrant on disk)
             │
             ▼
   Faster Whisper ─► transcripts ─► Khoj ─► chat answers your
   (audio in, text out, all local)            recorded meetings

The pattern is rigid on purpose: one runtime at the bottom (so every downstream tool points to the same OpenAI-compatible URL), one or more UIs chosen by user audience, one RAG service if documents matter, one speech engine if meetings matter. Don't run two competing runtimes — that's how you end up with 30 GB of duplicate model weights on the same machine.

Tradeoffs you'll hit

• Hardware floor is real — a 7B model in 4-bit quantization wants ~5 GB RAM, runs at 15-30 tok/s on M1/M2. A 70B model wants 40 GB+ and runs at 2-5 tok/s on M3 Max — usable for batch, painful for interactive chat. Don't promise the compliance officer GPT-4 quality from a 16 GB MacBook.
• Quality vs frontier models — a strong open model (Llama 3.1 70B, Qwen 2.5 72B) is roughly GPT-4-class for short Q&A and noticeably weaker on long-form reasoning. For privileged drafting, expect to do 1-2 more revision passes than you would with a frontier cloud model. Most regulated buyers accept this; the alternative is sending privileged data to a third party.
• Maintenance cost — you now own the deployment. Model updates, GPU driver upgrades, vector DB compaction, log rotation — all yours. Budget 0.2-0.5 FTE of IT time per 50 active users, more in year one.
• Multi-user concurrency — Ollama serializes requests by default. For more than ~5 concurrent users on a single GPU, switch to a real inference server (vLLM, TGI) behind Open WebUI. The pack tools are correct for the single-team scale; the same UIs work in front of vLLM when you grow.
• Air-gap vs partial isolation — "fully local" splits into two postures. Air-gapped: machine has no network at all, models pre-loaded via USB, updates via approval workflow. Network-isolated: machine has network for OS updates only, all AI traffic stays on localhost. The second is much easier; the first is what some government / defense buyers actually require.

Common pitfalls

• M1 / 16 GB MacBook trying to run a 70B model — won't fit. It will load via mmap then thrash the SSD. Stick to 7-13B models on 16 GB, 30-34B on 32 GB, 70B on 64 GB+. Apple's unified memory makes this easier than discrete GPUs, but the math is still the math.
• Wrong-language model picks — most open base models are English-heavy. For Chinese legal text, pick Qwen 2.5 (Chinese-strong by design). For Japanese medical notes, pick a fine-tune like ELYZA. Choosing Llama 3.1 for a non-English corpus and then complaining about quality is the most common failure mode.
• RAG chunking strategy — naive 1000-token chunks destroy contract clauses (each clause is its own logical unit) and medical chart entries (each entry is timestamped and atomic). Configure Khoj's chunker to split on document-specific boundaries before you embed 50,000 case files and discover retrieval is useless.
• Forgetting to disable telemetry — Open WebUI, Jan, Text Generation WebUI all ship with optional anonymous telemetry. Turn it off explicitly in config and verify with a network sniffer. The whole point of this stack is no outbound calls; a single forgotten checkbox undermines the compliance story.
• Storing PHI / privileged data in cleartext on disk — your model weights might be local, but if Khoj's vector DB sits unencrypted on the same disk, a stolen laptop leaks everything anyway. Enable full-disk encryption (FileVault, LUKS, BitLocker) on every machine in the stack. This is the audit finding that gets caught last, after every other privacy control passes.