oMLX — LLM Inference Server with Continuous Batching for Apple Silicon

Introduction

oMLX is an LLM inference server designed specifically for Apple Silicon Macs. It leverages the MLX framework to deliver fast, efficient model serving with features like continuous batching and SSD-based KV cache offloading, all controllable from a native macOS menu bar interface.

What oMLX Does

• Serves LLM models locally on Apple Silicon with an OpenAI-compatible API
• Uses continuous batching to handle multiple concurrent requests efficiently
• Offloads KV cache to SSD when GPU memory is limited, enabling larger context windows
• Provides a macOS menu bar app for one-click model management and server control
• Supports quantized MLX models from Hugging Face for reduced memory footprint

Architecture Overview

oMLX runs as a local HTTP server backed by Apple's MLX framework for GPU-accelerated inference on M-series chips. It implements continuous batching so incoming requests are merged into running batches without waiting for previous sequences to finish. When the unified memory is exhausted, the KV cache spills transparently to NVMe SSD, trading some latency for the ability to serve models or context lengths that would otherwise not fit in RAM.

Self-Hosting & Configuration

• Requires a Mac with Apple Silicon (M1 or later) and macOS 13+
• Install via Homebrew or download the app bundle from releases
• Point the server at any MLX-format model directory or Hugging Face repo ID
• Configure port, max concurrency, and cache directory through CLI flags or the menu bar UI
• Supports environment variables for headless or remote-access deployment

Key Features

• Native Apple Silicon acceleration via MLX for maximum throughput per watt
• Continuous batching scheduler that maximizes GPU utilization across requests
• SSD-backed KV cache extends effective context far beyond available RAM
• OpenAI-compatible API endpoint for drop-in integration with existing tools
• Menu bar app provides visual model switching, memory stats, and log monitoring

Comparison with Similar Tools

• Ollama — Cross-platform and broader model support, but lacks continuous batching and SSD caching on Mac
• llama.cpp — Highly portable C++ inference, but requires manual server setup and tuning for Apple Silicon
• MLX Examples — Official Apple examples for MLX models, but no production server features like batching or caching
• LM Studio — Polished desktop UI with broad model support, but closed source and less configurable

FAQ

Q: Which Mac models are supported? A: Any Mac with Apple Silicon (M1, M2, M3, M4 series) running macOS 13 Ventura or later.

Q: Can I use models from Hugging Face directly? A: Yes, point oMLX at any mlx-community model repo ID and it downloads and caches the model automatically.

Q: How does SSD caching affect performance? A: SSD caching adds some latency compared to pure RAM inference, but it enables serving models or context lengths that would otherwise be impossible on your hardware.

Q: Is there an API compatible with OpenAI clients? A: Yes, oMLX exposes a /v1/chat/completions endpoint that works with any OpenAI-compatible client library.

Sources

• https://github.com/jundot/oMLX
• https://mlx.readthedocs.io/