# Colibri — Run 744B Parameter LLMs on Consumer Hardware in Pure C

> A tiny inference engine written in pure C with zero dependencies that streams expert layers from disk, enabling 744B MoE models to run on machines with as little as 25GB RAM.

## Install

Save as a script file and run:

# Colibri — Run 744B Parameter LLMs on Consumer Hardware in Pure C

## Quick Use
```bash
git clone https://github.com/JustVugg/colibri.git
cd colibri && make
./colibri --model /path/to/glm-5.2-744b/ --prompt "Hello"
```

## Introduction
Colibri is a minimalist LLM inference engine written in pure C with zero external dependencies. Its key innovation is streaming Mixture-of-Experts layers from disk on demand, allowing models with hundreds of billions of parameters to run on consumer hardware without requiring the entire model to fit in RAM.

## What Colibri Does
- Runs 744B MoE models on machines with 25GB RAM by streaming experts from SSD
- Compiles to a single static binary with no dependencies
- Supports quantized model formats for reduced disk footprint
- Provides interactive chat and batch completion modes
- Achieves usable token generation speeds on consumer NVMe drives

## Architecture Overview
Colibri implements a memory-mapped expert streaming system. Only the active experts for each token are loaded into RAM, while inactive experts remain on disk. The engine uses a custom memory allocator optimized for the cyclic access pattern of transformer layers, and exploits OS page cache for frequently activated experts.

## Self-Hosting & Configuration
- Compile with make on any system with a C compiler (gcc, clang, MSVC)
- Download compatible model weights in GGUF or Colibri native format
- Configure RAM budget and thread count via command-line flags
- NVMe SSD strongly recommended for acceptable generation speed
- No CUDA or GPU required, though GPU offloading is optional

## Key Features
- Zero dependencies: single C file compiles anywhere
- Expert streaming enables models far larger than available RAM
- Sub-second time-to-first-token for cached conversation contexts
- Supports both interactive and batch inference modes
- Memory-safe implementation with bounds checking in debug builds

## Comparison with Similar Tools
- **llama.cpp** — requires model to mostly fit in RAM vs disk-streaming architecture
- **ExLlamaV2** — GPU-focused with Python vs CPU-first with pure C
- **vLLM** — server-grade multi-GPU vs single-machine consumer hardware
- **llamafile** — single-file distribution vs separate engine and weights
- **MLC-LLM** — compiled model graphs vs interpreted streaming approach

## FAQ
**Q: What generation speed can I expect?**
A: On a modern NVMe SSD, expect 2-5 tokens/second for a 744B model. Smaller models are proportionally faster.

**Q: Which models are supported?**
A: Any GGUF-format MoE model. GLM-5.2 744B is the primary target, with community support for others.

**Q: Do I need a GPU?**
A: No. Colibri is designed for CPU inference with disk streaming. Optional GPU offloading is available for acceleration.

**Q: How much disk space do the model weights need?**
A: A Q4 quantized 744B model requires approximately 400GB of disk space.

## Sources
- https://github.com/JustVugg/colibri
- https://github.com/JustVugg/colibri#readme

---
Source: https://tokrepo.com/en/workflows/asset-fec89962
Author: Script Depot