Ollama — Run LLMs Locally

What it is

Ollama is a tool for running open-source large language models on your local machine. It handles model downloading, quantization, memory management, and serving behind a simple CLI and HTTP API. You run ollama run llama3.1 and get an interactive chat session. You call localhost:11434/api/generate and get a streaming inference API. No cloud accounts, no API keys, no usage fees.

Ollama supports a wide range of models including Llama 3.1, Mistral, Gemma, Code Llama, Phi, and community-contributed models. It runs on macOS, Linux, and Windows, with automatic GPU acceleration on Apple Silicon (Metal), NVIDIA (CUDA), and AMD (ROCm) hardware.

How it saves time or tokens

Every API call to a cloud LLM costs tokens and money. For development workflows where you make hundreds of test calls per day (prompt iteration, function calling tests, embedding generation), those costs add up. Ollama eliminates per-token costs entirely by running inference on hardware you already own.

The time savings come from zero network latency for local inference, no rate limiting, and no API key management. For privacy-sensitive tasks like processing internal documents or generating code from proprietary codebases, local inference means the data never leaves your machine.

How to use

1. Install Ollama with one command:

```bash

curl -fsSL https://ollama.com/install.sh | sh

```

On macOS, you can also download the desktop app from ollama.com.

1. Run a model:

```bash

ollama run llama3.1

```

This downloads the model on first run (typically 4-8 GB) and starts an interactive chat.

1. Use the HTTP API for programmatic access:

```bash

curl http://localhost:11434/api/generate -d '{

"model": "llama3.1",

"prompt": "Explain quicksort in 3 sentences."

}'

```

Example

Using Ollama with Python via the official library:

import ollama

# Simple completion
response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'What is a closure in JavaScript?'}]
)
print(response['message']['content'])

# Streaming
for chunk in ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Write a haiku about code.'}],
    stream=True
):
    print(chunk['message']['content'], end='', flush=True)

Popular models and their approximate sizes:

Related on TokRepo

• Local LLM Tools — Compare Ollama with other local inference options like LM Studio, llama.cpp, and vLLM.
• Local LLM: Ollama — Deep dive into Ollama workflows and configurations on TokRepo.

Common pitfalls

• Running a 70B model on a machine with 16GB RAM. Large models require memory roughly equal to their download size. If you exceed available RAM, Ollama will use swap, which makes inference extremely slow. Start with 7B-8B models and scale up based on your hardware.
• Forgetting that Ollama serves on port 11434 by default. If another service uses that port, Ollama will fail to start. Set OLLAMA_HOST environment variable to change the port.
• Expecting cloud-level speed from consumer hardware. A 7B model on an M1 MacBook generates roughly 30-50 tokens per second. A 70B model on the same hardware generates 3-5 tokens per second. Set expectations based on your hardware.