RunAnywhere SDKs — Run AI Models Locally on Any Device

Introduction

RunAnywhere SDKs let developers embed local AI inference into mobile, desktop, and web applications without sending data to the cloud. The toolkit wraps optimized C++ inference engines behind platform-native bindings, so the same model runs on an iPhone, an Android phone, or a browser tab.

What RunAnywhere Does

• Provides unified SDKs for iOS (Swift), Android (Kotlin), React Native, Flutter, and web
• Runs LLMs like Llama, Qwen, and Gemma with quantized weights for on-device speed
• Supports diffusion models for local image generation on mobile GPUs
• Offers a chat completion API that mirrors the OpenAI interface for easy migration
• Handles model downloading, caching, and lifecycle management automatically

Architecture Overview

The core is a C++ inference engine built on llama.cpp and custom GGUF model loaders. Platform SDKs bind to this core via JNI on Android, Swift C interop on iOS, and WebAssembly plus WebGPU on the web. Models are fetched on first use, cached locally, and loaded into memory on demand. The engine selects the best available backend (Metal, Vulkan, CPU NEON) at runtime.

Self-Hosting & Configuration

• Install the platform-specific SDK package via npm, CocoaPods, or Gradle
• Call the init method with a model identifier to download and cache the model
• Configure quantization level (Q4, Q5, Q8) to trade quality for speed and memory
• Set context window size and max tokens per response via the options object
• Use the streaming API for token-by-token output in chat interfaces

Key Features

• One API across six platforms including React Native and Flutter
• Automatic backend selection for Metal, Vulkan, WebGPU, or CPU
• Built-in model management with download progress callbacks
• Streaming and non-streaming chat completion interfaces
• Multimodal support for vision-language models on supported hardware

Comparison with Similar Tools

• Ollama — desktop-focused server; RunAnywhere targets mobile and web natively
• llama.cpp — the underlying C++ engine; RunAnywhere adds platform SDKs and model management
• MLX — Apple Silicon only; RunAnywhere runs on Android and web too
• MediaPipe LLM — Google's on-device inference; RunAnywhere supports more model formats
• ExecuTorch — PyTorch's mobile runtime; RunAnywhere offers higher-level chat APIs

FAQ

Q: What models are supported? A: Any GGUF-format model works. The SDK ships with optimized configs for Llama 3, Qwen 2.5, Gemma 2, and Phi-3.

Q: How much RAM do models need on mobile? A: A Q4 quantized 3B model needs roughly 2 GB of RAM. Smaller 1B models run under 1 GB, suitable for most modern phones.

Q: Does it work offline after the initial model download? A: Yes. Once cached, inference runs entirely on-device with no network required.

Q: Can I fine-tune models through the SDK? A: No. The SDK is inference-only. Fine-tune models externally, convert to GGUF, and load them with the custom model path option.

Sources

• https://github.com/RunanywhereAI/runanywhere-sdks
• https://runanywhere.ai/docs