Ray — Distributed Computing for Python and AI Workloads

What it is

Ray is an open-source unified framework for scaling Python and AI applications. It handles distributed training, hyperparameter search, large-scale data processing, and model serving under a single API. You turn any Python function into a distributed task with the @ray.remote decorator.

ML engineers, data scientists, and platform teams use Ray when their workloads outgrow a single machine. Ray's ecosystem includes Ray Train (distributed training), Ray Tune (hyperparameter optimization), Ray Serve (model serving), and Ray Data (distributed data processing).

How it saves time or tokens

Ray abstracts away the complexity of distributed computing. Instead of managing MPI, Dask clusters, or custom job schedulers, you write standard Python and Ray handles scheduling, fault tolerance, and resource allocation. The @ray.remote decorator converts a function call into a distributed task without rewriting your code.

For LLM workloads, Ray Serve can host multiple models behind a single endpoint with autoscaling, reducing infrastructure management overhead.

How to use

1. Install Ray:

pip install 'ray[default]'

1. Distribute a computation:

import ray
ray.init()

@ray.remote
def heavy(n):
    return sum(i * i for i in range(n))

futures = [heavy.remote(10_000_000) for _ in range(8)]
results = ray.get(futures)
print(sum(results))

1. This runs 8 parallel tasks across available CPU cores. On a cluster, it distributes across multiple nodes automatically.

Example

Serving a model with Ray Serve:

from ray import serve

@serve.deployment
class Classifier:
    def __init__(self):
        self.model = load_model()

    def __call__(self, request):
        data = request.json()
        return self.model.predict(data['input'])

serve.run(Classifier.bind())

This deploys a model endpoint with built-in autoscaling, health checks, and request batching.

Related on TokRepo

• AI Tools for Coding -- Development tools for building AI applications
• Local LLM Providers -- Run LLMs locally with distributed inference via Ray

Common pitfalls

• Ray's object store consumes significant memory. On machines with limited RAM, set object_store_memory in ray.init() to prevent out-of-memory errors.
• Serialization errors occur when passing non-picklable objects between tasks. Use Ray's built-in serialization or convert objects to serializable formats before passing.
• Starting Ray on a cluster requires matching Python and Ray versions across all nodes. Use Docker images or Conda environments to ensure consistency.