Candle — Minimalist Machine Learning Framework for Rust

What it is

Candle is a Rust-native machine learning framework focused on inference performance, small binaries, and serverless deployment. Built by Hugging Face, it runs Llama, Whisper, Stable Diffusion, and other PyTorch models in pure Rust without requiring Python.

Candle is designed for ML engineers and systems developers who need fast inference with minimal dependencies, especially for edge deployment, WebAssembly targets, or serverless functions where Python runtimes add overhead.

How it saves time or tokens

Candle produces small, self-contained binaries that start in milliseconds compared to Python-based inference servers that load heavy runtimes. No Python dependency chain means no version conflicts, no pip install issues, and predictable builds. CUDA and Metal support provides GPU acceleration on par with PyTorch for supported models.

How to use

1. Add Candle to your Rust project:

cargo add candle-core candle-nn candle-transformers

1. Run a pre-built example:

cargo run --example llama -- --prompt 'Hello, world'

1. Use the tensor API:

use candle_core::{Device, Tensor};

fn main() -> anyhow::Result<()> {
    let device = Device::Cpu;
    let a = Tensor::randn(0f32, 1., (3, 3), &device)?;
    let b = Tensor::randn(0f32, 1., (3, 3), &device)?;
    let c = a.matmul(&b)?;
    println!("{c}");
    Ok(())
}

Example

// Load and run Whisper for speech-to-text
use candle_transformers::models::whisper;

fn transcribe(audio_path: &str) -> anyhow::Result<String> {
    let device = Device::Cpu;
    let model = whisper::model::Whisper::load(
        "openai/whisper-base",
        &device,
    )?;
    let result = model.transcribe(audio_path)?;
    Ok(result.text)
}

Related on TokRepo

• AI coding tools — ML and AI development frameworks
• Local LLM tools — running models locally

Common pitfalls

• Expecting full training support: Candle is optimized for inference, not large-scale training
• Not enabling the cuda or metal feature flags for GPU acceleration
• Trying to load PyTorch checkpoints directly without converting to safetensors format first