# TorchServe — Production Model Serving for PyTorch

> TorchServe is an open-source model serving framework for PyTorch that packages trained models into scalable HTTP and gRPC endpoints with built-in support for batching, logging, metrics, and multi-model management.

## Install

Save in your project root:

# TorchServe — Production Model Serving for PyTorch

## Quick Use
```bash
# Install TorchServe
pip install torchserve torch-model-archiver

# Package a model into a .mar archive
torch-model-archiver --model-name resnet 
  --version 1.0 
  --model-file model.py 
  --serialized-file resnet.pth 
  --handler image_classifier

# Start the server
torchserve --start --model-store model_store --models resnet=resnet.mar
```

## Introduction
TorchServe is developed jointly by AWS and Meta as the official model serving solution for PyTorch. It bridges the gap between training and production by providing a performant, configurable server that handles model packaging, versioning, dynamic batching, and monitoring without requiring custom serving infrastructure.

## What TorchServe Does
- Packages PyTorch models into versioned, deployable archives (.mar files) with dependencies included
- Serves models via REST and gRPC APIs with configurable endpoints for inference, management, and metrics
- Supports dynamic batching to aggregate requests and maximize GPU utilization
- Manages multiple models simultaneously with independent scaling and version control
- Provides built-in Prometheus metrics and logging for production observability

## Architecture Overview
TorchServe runs a Java-based frontend that accepts HTTP/gRPC requests and routes them to Python backend workers. Each model runs in its own worker process, enabling isolation and independent scaling. The torch-model-archiver tool packages model weights, handler code, and dependencies into a single .mar file. A management API allows loading, unloading, and scaling models at runtime without restarting the server.

## Self-Hosting & Configuration
- Install from PyPI: `pip install torchserve torch-model-archiver torch-workflow-archiver`
- Configure server settings in `config.properties`: worker count, batch size, timeouts, and GPU assignment
- Deploy models by placing .mar files in the model store directory and registering them via the management API
- Set up dynamic batching with `batch_size` and `max_batch_delay` parameters per model
- Run in Docker with the official `pytorch/torchserve` image for containerized deployments

## Key Features
- Dynamic batching aggregates individual requests into GPU-efficient batches automatically
- Custom handlers allow preprocessing, inference, and postprocessing logic in Python
- A/B testing and canary deployments through multi-version model management
- Snapshots capture server state for reproducible deployments and quick recovery
- Large model inference support with model parallelism across multiple GPUs

## Comparison with Similar Tools
- **vLLM** — Optimized for LLM inference with PagedAttention; TorchServe is a general-purpose serving framework for any PyTorch model
- **Triton Inference Server** — NVIDIA's multi-framework server; TorchServe is PyTorch-native with simpler setup for PyTorch-only deployments
- **BentoML** — Python-first serving with packaging; TorchServe provides tighter PyTorch integration with official support from Meta and AWS
- **TFServing** — TensorFlow's serving solution; TorchServe is the equivalent for the PyTorch ecosystem
- **Ray Serve** — General-purpose model serving on Ray; TorchServe is more focused with built-in PyTorch-specific optimizations

## FAQ
**Q: Does TorchServe support GPU inference?**
A: Yes. TorchServe automatically uses available GPUs. You can assign specific GPUs to specific models and configure the number of workers per GPU.

**Q: Can I serve multiple models simultaneously?**
A: Yes. TorchServe manages multiple models with independent worker pools. Use the management API to load, unload, and scale models at runtime.

**Q: What is a custom handler?**
A: A handler is a Python class that defines four methods: initialize (load model), preprocess (transform input), inference (run model), and postprocess (format output). TorchServe includes default handlers for common tasks like image classification and object detection.

**Q: How does dynamic batching work?**
A: TorchServe collects incoming requests up to a configurable batch size or timeout, then sends the batch to the model in a single forward pass. This maximizes GPU utilization when serving many concurrent requests.

## Sources
- https://github.com/pytorch/serve
- https://pytorch.org/serve/

---
Source: https://tokrepo.com/en/workflows/asset-6072344c
Author: AI Open Source