Colibri — Run 744B Parameter LLMs on Consumer Hardware in Pure C

Introduction

Colibri is a minimalist LLM inference engine written in pure C with zero external dependencies. Its key innovation is streaming Mixture-of-Experts layers from disk on demand, allowing models with hundreds of billions of parameters to run on consumer hardware without requiring the entire model to fit in RAM.

What Colibri Does

• Runs 744B MoE models on machines with 25GB RAM by streaming experts from SSD
• Compiles to a single static binary with no dependencies
• Supports quantized model formats for reduced disk footprint
• Provides interactive chat and batch completion modes
• Achieves usable token generation speeds on consumer NVMe drives

Architecture Overview

Colibri implements a memory-mapped expert streaming system. Only the active experts for each token are loaded into RAM, while inactive experts remain on disk. The engine uses a custom memory allocator optimized for the cyclic access pattern of transformer layers, and exploits OS page cache for frequently activated experts.

Self-Hosting & Configuration

• Compile with make on any system with a C compiler (gcc, clang, MSVC)
• Download compatible model weights in GGUF or Colibri native format
• Configure RAM budget and thread count via command-line flags
• NVMe SSD strongly recommended for acceptable generation speed
• No CUDA or GPU required, though GPU offloading is optional

Key Features

• Zero dependencies: single C file compiles anywhere
• Expert streaming enables models far larger than available RAM
• Sub-second time-to-first-token for cached conversation contexts
• Supports both interactive and batch inference modes
• Memory-safe implementation with bounds checking in debug builds

Comparison with Similar Tools

• llama.cpp — requires model to mostly fit in RAM vs disk-streaming architecture
• ExLlamaV2 — GPU-focused with Python vs CPU-first with pure C
• vLLM — server-grade multi-GPU vs single-machine consumer hardware
• llamafile — single-file distribution vs separate engine and weights
• MLC-LLM — compiled model graphs vs interpreted streaming approach

FAQ

Q: What generation speed can I expect? A: On a modern NVMe SSD, expect 2-5 tokens/second for a 744B model. Smaller models are proportionally faster.

Q: Which models are supported? A: Any GGUF-format MoE model. GLM-5.2 744B is the primary target, with community support for others.

Q: Do I need a GPU? A: No. Colibri is designed for CPU inference with disk streaming. Optional GPU offloading is available for acceleration.

Q: How much disk space do the model weights need? A: A Q4 quantized 744B model requires approximately 400GB of disk space.

Sources

• https://github.com/JustVugg/colibri
• https://github.com/JustVugg/colibri#readme