[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"pack-detail-local-llm-runners-zh":3,"seo:pack:local-llm-runners:zh":74},{"code":4,"message":5,"data":6},200,"操作成功",{"pack":7},{"slug":8,"icon":9,"tone":10,"status":11,"status_label":12,"title":13,"description":14,"items":15,"install_cmd":73},"local-llm-runners","💻","#374151","stable","稳定","本机跑 LLM","Ollama \u002F GPT4All \u002F MLC-LLM \u002F Jan \u002F Open WebUI \u002F Text Generation WebUI \u002F TGI — \"不要 API key，用我的 GPU\" 全方案。",[16,28,36,44,51,58,65],{"id":17,"uuid":18,"slug":19,"title":20,"description":21,"author_name":22,"view_count":23,"vote_count":24,"lang_type":25,"type":26,"type_label":27},771,"4cecf968-aa84-47ec-9f32-c3b11432c18f","ollama-model-library-best-ai-models-local-use-4cecf968","Ollama Model Library — Best AI Models for Local Use","Curated guide to the best models available on Ollama for coding, chat, and reasoning. Compare Llama, Mistral, Gemma, Phi, and Qwen models for local AI development.","Skill Factory",490,0,"en","skill","Skill",{"id":29,"uuid":30,"slug":31,"title":32,"description":33,"author_name":34,"view_count":35,"vote_count":24,"lang_type":25,"type":26,"type_label":27},274,"f493abd9-0870-49b3-a04b-719ee2a5df0f","gpt4all-run-llms-privately-your-desktop-f493abd9","GPT4All — Run LLMs Privately on Your Desktop","GPT4All runs large language models privately on everyday desktops and laptops without GPUs or API calls. 77.2K+ GitHub stars. Desktop app + Python SDK, LocalDocs for private data. MIT licensed.","AI Open Source",303,{"id":37,"uuid":38,"slug":39,"title":40,"description":41,"author_name":42,"view_count":43,"vote_count":24,"lang_type":25,"type":26,"type_label":27},232,"735f5a27-07d6-47ac-8377-e29be76a9452","mlc-llm-universal-llm-deployment-engine-735f5a27","MLC-LLM — Universal LLM Deployment Engine","Deploy any LLM on any hardware — phones, browsers, GPUs, CPUs. Compiles models for native performance on iOS, Android, WebGPU, CUDA, Metal, and Vulkan. 22K+ stars.","Script Depot",336,{"id":45,"uuid":46,"slug":47,"title":48,"description":49,"author_name":34,"view_count":50,"vote_count":24,"lang_type":25,"type":26,"type_label":27},282,"11107806-c69a-4b75-8360-d0504ff602d7","text-generation-webui-local-llm-chat-interface-11107806","Text Generation WebUI — Local LLM Chat Interface","Text Generation WebUI is a Gradio interface for running LLMs locally. 46.4K+ GitHub stars. Multiple backends, vision, training, image gen, OpenAI-compatible API. 100% offline.",401,{"id":52,"uuid":53,"slug":54,"title":55,"description":56,"author_name":34,"view_count":57,"vote_count":24,"lang_type":25,"type":26,"type_label":27},278,"7b703194-ec0f-4244-a98e-3ec206a883b8","jan-offline-ai-desktop-app-full-privacy-7b703194","Jan — Offline AI Desktop App with Full Privacy","Jan is an open-source ChatGPT alternative that runs LLMs locally with full privacy. 41.4K+ GitHub stars. Desktop app for Windows\u002FmacOS\u002FLinux, OpenAI-compatible API, MCP support. Apache 2.0.",323,{"id":59,"uuid":60,"slug":61,"title":62,"description":63,"author_name":42,"view_count":64,"vote_count":24,"lang_type":25,"type":26,"type_label":27},218,"5d37ffb8-d351-4fb1-8665-bef4db25b275","open-webui-self-hosted-ai-chat-interface-5d37ffb8","Open WebUI — Self-Hosted AI Chat Interface","User-friendly, self-hosted AI chat interface. Supports Ollama, OpenAI, Anthropic, and any OpenAI-compatible API. RAG, web search, voice, image gen, and plugins. 129K+ stars.",348,{"id":66,"uuid":67,"slug":68,"title":69,"description":70,"author_name":71,"view_count":72,"vote_count":24,"lang_type":25,"type":26,"type_label":27},1303,"e08ad222-37db-11f1-9bc6-00163e2b0d79","text-generation-inference-tgi-hugging-face-production-llm-e08ad222","Text Generation Inference (TGI) — Hugging Face Production LLM Server","TGI is Hugging Face's production-grade LLM inference server. It powers HF Inference Endpoints with continuous batching, tensor parallelism, quantization, and OpenAI-compatible APIs — handling thousands of requests per second.","Hugging Face",434,"tokrepo install pack\u002Flocal-llm-runners",{"pageType":75,"pageKey":8,"locale":76,"title":77,"metaDescription":78,"h1":13,"tldr":79,"bodyMarkdown":80,"faq":81,"schema":97,"internalLinks":107,"citations":120,"wordCount":133,"generatedAt":134},"pack","zh","本机跑 LLM：7 个开源运行器 · TokRepo","Ollama \u002F GPT4All \u002F MLC-LLM \u002F Jan \u002F Open WebUI \u002F Text Generation WebUI \u002F TGI — 用自己的 GPU 跑 LLM，不要 API key。TokRepo 一条命令装完。","七个经过实战的开源 LLM 运行时，覆盖笔记本聊天、GPU 服务器、移动端、Web UI、生产端点全场景。TokRepo 一条命令装齐。","## 这个 pack 装了什么\n\n| # | 运行器 | 最适合 | 后端 |\n|---|---|---|---|\n| 1 | Ollama | Mac\u002FLinux\u002FWindows 一行 CLI | llama.cpp |\n| 2 | GPT4All | 桌面应用，不要 GPU | llama.cpp + GGUF |\n| 3 | MLC-LLM | iOS \u002F Android \u002F WebGPU | TVM 编译器 |\n| 4 | Jan | 桌面版 ChatGPT 替代 | llama.cpp + 远程 API |\n| 5 | Open WebUI | 给任意 OpenAI 兼容运行器套 ChatGPT 风格 UI | 反向代理 Ollama\u002FvLLM\u002FTGI |\n| 6 | Text Generation WebUI | 研究级 UI，可做 LoRA 训练 | transformers + ExLlama + llama.cpp |\n| 7 | Hugging Face TGI | 生产级服务，支持连续批处理 | Rust + Python，多 GPU |\n\n七个运行器把全光谱覆盖了：从「想在笔记本上有个聊天窗口」到「Llama 3 要塞在负载均衡后撑 10k QPS」。\n\n## 为什么 2026 年要本地化\n\n三股力量把云 API 和自托管推理的成本差抹平了。\n\n第一，模型质量。Meta（Llama）、Mistral、Qwen、DeepSeek 的开源权重在多数推理和编码任务上已经追上 GPT-4 级别。不付 OpenAI 钱不再意味着质量打折。\n\n第二，硬件。一张 RTX 4090 用 llama.cpp 的 GGUF Q4 量化跑 Llama 3 70B 速度可用。Apple Silicon 终于上了统一内存 —— M3 Max 跑 70B 不发热掉速。中端游戏本也能流畅跑 8B。\n\n第三，隐私合规。医疗、法律、金融、欧盟 GDPR 圈的公司不能把 PII 发给第三方 API，本地推理是唯一合法路径。编码 agent 同理 —— 多数企业禁止 Cursor\u002FCopilot 接触私有仓库。\n\n## 一条命令装齐\n\n```bash\n# 装整个 pack\ntokrepo install pack\u002Flocal-llm-runners\n\n# 或者只装真正需要的那个\ntokrepo install ollama\ntokrepo install open-webui\ntokrepo install tgi\n```\n\n每个资产的 TokRepo 页面都打包好了安装命令、推荐配置，以及主流 Llama \u002F Qwen \u002F DeepSeek 权重的 model-pull 命令。\n\n## 常见坑\n\n- **显存账要算清楚**：「7B」模型 FP16 大约 14 GB，Q4 大约 4 GB。下载前先看量化后缀\n- **上下文窗口 vs 显存**：7B 模型开 32k 上下文，KV cache 可能跟权重一样大。OOM 就先把上下文压低\n- **Open WebUI 套在 Ollama 上**：Open WebUI 走 OpenAI 协议，必须在 Ollama 上开 OpenAI 兼容端点（`OLLAMA_HOST=0.0.0.0`），多数教程跳过了这步\n- **TGI vs vLLM**：HuggingFace 分片权重选 TGI，纯吞吐选 vLLM。不要因为 TGI 老就默认选它\n- **模型许可**：Llama 3 是宽松但不是 MIT。商用部署前看许可证，做下游 fine-tune 尤其要看\n\n## 跟其他 pack 的关系\n\nlocal-llm-runners 是*运行时*层。要端到端可用，要搭配：\n\n- 配 **AI 第二大脑** pack —— Logseq + Khoj 把笔记索引进本地 Ollama\n- 配 **LLM 评测 & 护栏** —— 验证本地模型相对闭源基线没退化\n- 配 **文档 AI 流水线** —— 把 PDF 喂进本地推理而不是发给 vendor\n\n三个 pack 凑起来就是完全断网也能跑、不回拨厂商的知识栈。边界清楚：运行器做推理，评测 pack 打分，第二大脑 pack 做检索，文档流水线把文件切块。按隐私和延迟目标自由组合，下层垫 Ollama 或 TGI 当引擎。\n\n## 怎么挑哪个运行器\n\n- **单人笔记本、主要聊天**：Ollama + Jan 当 UI。五分钟装好，GGUF Q4 权重，飞机上离线也能跑\n- **VPN 后团队共享 GPU 服务器**：TGI 或 vLLM 套在负载均衡后，Open WebUI 当团队前端 + SSO。一个模型多用户用，省掉按座位的 OpenAI 账单\n- **移动端 demo 或纯浏览器推理**：MLC-LLM。把权重编译到 WebGPU\u002FMetal\u002FVulkan，完全无服务器跑 —— 离线移动端原型最好用\n- **研究实验室在消费级 GPU 上做 fine-tune**：Text Generation WebUI。内建 LoRA 训练、ExLlama 后端，还能加载 HuggingFace 每周丢出来那些半坏的 checkpoint",[82,85,88,91,94],{"q":83,"a":84},"这套真免费？还是有隐藏成本？","七个运行器全开源免费。成本是硬件 —— 选模型权重要看你 GPU 显存够不够。消费级 RTX 3090\u002F4090（24GB）跑 7B-13B 流畅，70B 配合激进量化可用。M 系列 Mac 走 Metal。要 GPU 算力可以租 Runpod 或 Vast.ai，长跑成本远低于 OpenAI API。",{"q":86,"a":87},"Ollama 还是 Jan 该从哪个开始？","你常驻终端、想要 OpenAI 兼容 HTTP 给应用调用，选 Ollama。想要一键桌面聊天像 ChatGPT 那样的体验，选 Jan。很多人两个一起跑：Ollama 当引擎，Jan 或 Open WebUI 当 UI。它们共享 Ollama 本地的 GGUF 模型库。",{"q":89,"a":90},"能搭 Cursor \u002F Codex CLI 用吗？","可以 —— Cursor 和 Codex CLI 都接受任意 OpenAI 兼容端点。在设置里把 URL 指向 http:\u002F\u002Flocalhost:11434\u002Fv1（Ollama）或对应端口。Cursor 设置里叫 Custom OpenAI URL。注意：本地 7B 跟 GPT-4 在长上下文重构上有差距，要生产级代码改动建议用 70B+。",{"q":92,"a":93},"跟 LLM 评测 & 护栏 pack 有啥区别？","本 pack 是负责跑模型的运行时，评测 pack 是给模型输出打分的。两者互补：先在这装个运行器，然后用 DeepEval\u002FPromptfoo 跑分，确保把云模型换成本地模型不会退化。多数想本地化的团队两个 pack 都要装。",{"q":95,"a":96},"装完最大的坑是什么？","上下文窗口没设到符合显存预算。默认值保守（2k-4k），但如果你加载一个 32k 训练的模型再喂满上下文，KV cache 会爆掉，跑一半 OOM。上线前永远在真实负载下看一眼 `nvidia-smi`。",{"@context":98,"@type":99,"name":100,"description":101,"numberOfItems":102,"publisher":103},"https:\u002F\u002Fschema.org","CollectionPage","Run LLMs Locally","Seven open-source runners that let you serve LLMs on your own GPU or laptop, no API key required.",7,{"@type":104,"name":105,"url":106},"Organization","TokRepo","https:\u002F\u002Ftokrepo.com",[108,112,116],{"url":109,"anchor":110,"reason":111},"\u002Fzh\u002Fpacks\u002Fai-second-brain","AI 第二大脑","本地 LLM 是隐私后端首选",{"url":113,"anchor":114,"reason":115},"\u002Fzh\u002Fpacks\u002Fllm-eval-guardrails","LLM 评测与护栏","本地模型质量评估",{"url":117,"anchor":118,"reason":119},"\u002Fzh\u002Ftools\u002Follama","Ollama","本 pack 里最受欢迎的运行器",[121,125,129],{"claim":122,"source_name":123,"source_url":124},"Ollama is an open-source local LLM runtime with a public model library","ollama\u002Follama on GitHub","https:\u002F\u002Fgithub.com\u002Follama\u002Follama",{"claim":126,"source_name":127,"source_url":128},"Hugging Face Text Generation Inference (TGI) is the production-grade serving backend","huggingface\u002Ftext-generation-inference","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftext-generation-inference",{"claim":130,"source_name":131,"source_url":132},"Open WebUI provides a ChatGPT-style UI on top of any OpenAI-compatible runner","open-webui\u002Fopen-webui","https:\u002F\u002Fgithub.com\u002Fopen-webui\u002Fopen-webui",536,"2026-05-02T15:00:00Z"]