KServe — Scalable ML Model Serving on Kubernetes

Introduction

KServe (formerly KFServing) is the standard model inference platform on Kubernetes maintained by the CNCF. It abstracts the complexity of deploying ML models behind a simple InferenceService custom resource, handling autoscaling, canary rollouts, and multi-framework serving.

What KServe Does

• Deploys ML models as Kubernetes services with a single YAML manifest
• Autoscales inference workloads from zero to many replicas based on request load
• Supports canary and pinned rollout strategies for safe model updates
• Provides a standardized V2 inference protocol compatible with multiple frameworks
• Manages model transformers and explainers alongside predictors in one resource

Architecture Overview

KServe extends Kubernetes with the InferenceService CRD. The control plane reconciles desired state into Knative Services or raw Kubernetes Deployments. Each InferenceService can include a predictor (model server), transformer (pre/post-processing), and explainer (model interpretability). The data plane routes requests through an ingress gateway to the appropriate model pod.

Self-Hosting & Configuration

• Install via kubectl apply or Helm chart on any Kubernetes 1.25+ cluster
• Serverless mode uses Knative for scale-to-zero; RawDeployment mode works without Knative
• Model artifacts are loaded from S3, GCS, Azure Blob, or PVCs via a storage initializer
• GPU scheduling is handled by standard Kubernetes resource requests and node selectors
• Monitoring integrates with Prometheus and Grafana for latency, throughput, and error metrics

Key Features

• Multi-framework support: TensorFlow, PyTorch, scikit-learn, XGBoost, LightGBM, ONNX, vLLM, and Triton
• Scale-to-zero with Knative reduces infrastructure costs for infrequently accessed models
• Canary rollouts with traffic percentage splitting for safe model version transitions
• ModelMesh integration for high-density multi-model serving on shared infrastructure
• V2 Inference Protocol provides a standardized REST and gRPC API across all frameworks

Comparison with Similar Tools

• TensorFlow Serving — Single-framework; KServe provides a unified interface for 10+ ML frameworks
• Triton Inference Server — KServe can use Triton as a backend runtime while adding autoscaling and K8s-native management
• BentoML — Packaging and deployment tool; KServe focuses on Kubernetes-native orchestration and autoscaling
• Seldon Core — Similar Kubernetes model serving; KServe is the CNCF standard with broader community adoption
• Ray Serve — Python-native serving framework; KServe is Kubernetes-native with richer deployment strategies

FAQ

Q: Does KServe require Knative? A: No. KServe supports a RawDeployment mode that works without Knative, using standard Kubernetes Deployments and HPA.

Q: Can KServe serve LLMs? A: Yes. KServe integrates with vLLM, Hugging Face TGI, and Triton for serving large language models with GPU acceleration.

Q: How does scale-to-zero work? A: In Knative mode, KServe scales pods to zero after a configurable idle timeout and spins them back up on incoming requests.

Q: What model storage backends are supported? A: KServe supports S3, GCS, Azure Blob Storage, HDFS, and Kubernetes Persistent Volume Claims for model artifact storage.

Sources

• https://github.com/kserve/kserve
• https://kserve.github.io/website/