kimi-k3-in-c — Run a Trillion-Parameter Model on One CPU in Pure C99

Introduction

kimi-k3-in-c is a from-scratch C99 implementation that runs the 2.78-trillion-parameter Kimi K3 mixture-of-experts model on a single CPU with approximately 8 GB of RAM. It achieves this by streaming expert weights from disk on demand, quantizing to MXFP4, and using hand-tuned SIMD kernels. No BLAS library, no ML framework, and no GPU required.

What kimi-k3-in-c Does

• Loads and runs inference on the full Kimi K3 model using only CPU and disk
• Streams MoE expert weights from storage to keep resident memory under 8.24 GB
• Quantizes weights to MXFP4 format for compact storage and fast arithmetic
• Generates text token by token with configurable sampling parameters
• Compiles on any platform with a C99 compiler — Linux, macOS, Windows, even ARM boards

Architecture Overview

The implementation treats the model as a disk-backed data structure. Only the active experts for each token are loaded into memory, while inactive experts remain on disk. The inference loop uses AVX2 and NEON SIMD intrinsics for vectorized matrix multiplication, with fallback scalar paths for platforms without SIMD support. The entire codebase is a single-file compilation unit with zero external dependencies — just C99 and the standard library.

Self-Hosting & Configuration

• Clone the repository and run make to compile the single C file
• Download the quantized model weights into the weights directory
• Adjust the number of threads with the --threads flag to match your CPU core count
• Set temperature, top-k, and top-p via command-line arguments
• Works on x86-64 (with AVX2) and ARM64 (with NEON) architectures

Key Features

• Zero dependencies: no BLAS, no framework, no GPU — just a C compiler and disk space
• Streams 2.78T parameters from disk so only active experts occupy memory
• MXFP4 quantization keeps model storage compact without severe quality loss
• Hand-tuned SIMD kernels for AVX2 and ARM NEON
• Single-file portable C99 that compiles anywhere

Comparison with Similar Tools

• llama.cpp — supports many models but targets smaller architectures; kimi-k3-in-c tackles a trillion-parameter MoE
• Colibri — also does disk-streamed MoE inference; kimi-k3-in-c is pure C99 with zero deps
• vLLM — GPU-based serving engine; kimi-k3-in-c runs entirely on CPU
• Turbo Fieldfare — Metal-accelerated Gemma inference on Mac; kimi-k3-in-c is platform-agnostic C

FAQ

Q: How fast is inference? A: Token generation is slower than GPU-based engines but viable for batch or offline workloads.

Q: Can I run it on a Raspberry Pi? A: In principle yes (ARM NEON is supported), but you need enough disk and patience for the throughput.

Q: Does quantization hurt quality? A: MXFP4 introduces some quality loss. The repository includes benchmark comparisons.

Q: Why C99 instead of C++? A: Maximum portability and simplicity. C99 compilers are available on every platform.

Sources

• https://github.com/FareedKhan-dev/kimi-k3-in-c
• https://github.com/MoonshotAI/Kimi-K3