# KServe — Scalable ML Model Serving on Kubernetes

> KServe is a CNCF project that provides a standardized Kubernetes-native platform for deploying, scaling, and managing machine learning models in production with support for TensorFlow, PyTorch, XGBoost, vLLM, and custom inference runtimes.

## Install

Save as a script file and run:

# KServe — Scalable ML Model Serving on Kubernetes

## Quick Use
```bash
# Install KServe on an existing K8s cluster
kubectl apply -f https://github.com/kserve/kserve/releases/latest/download/kserve.yaml
# Deploy a scikit-learn model
cat <<EOFYAML | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
EOFYAML
# Test the endpoint
kubectl get inferenceservice sklearn-iris
```

## Introduction
KServe (formerly KFServing) is the standard model inference platform on Kubernetes maintained by the CNCF. It abstracts the complexity of deploying ML models behind a simple InferenceService custom resource, handling autoscaling, canary rollouts, and multi-framework serving.

## What KServe Does
- Deploys ML models as Kubernetes services with a single YAML manifest
- Autoscales inference workloads from zero to many replicas based on request load
- Supports canary and pinned rollout strategies for safe model updates
- Provides a standardized V2 inference protocol compatible with multiple frameworks
- Manages model transformers and explainers alongside predictors in one resource

## Architecture Overview
KServe extends Kubernetes with the InferenceService CRD. The control plane reconciles desired state into Knative Services or raw Kubernetes Deployments. Each InferenceService can include a predictor (model server), transformer (pre/post-processing), and explainer (model interpretability). The data plane routes requests through an ingress gateway to the appropriate model pod.

## Self-Hosting & Configuration
- Install via kubectl apply or Helm chart on any Kubernetes 1.25+ cluster
- Serverless mode uses Knative for scale-to-zero; RawDeployment mode works without Knative
- Model artifacts are loaded from S3, GCS, Azure Blob, or PVCs via a storage initializer
- GPU scheduling is handled by standard Kubernetes resource requests and node selectors
- Monitoring integrates with Prometheus and Grafana for latency, throughput, and error metrics

## Key Features
- Multi-framework support: TensorFlow, PyTorch, scikit-learn, XGBoost, LightGBM, ONNX, vLLM, and Triton
- Scale-to-zero with Knative reduces infrastructure costs for infrequently accessed models
- Canary rollouts with traffic percentage splitting for safe model version transitions
- ModelMesh integration for high-density multi-model serving on shared infrastructure
- V2 Inference Protocol provides a standardized REST and gRPC API across all frameworks

## Comparison with Similar Tools
- **TensorFlow Serving** — Single-framework; KServe provides a unified interface for 10+ ML frameworks
- **Triton Inference Server** — KServe can use Triton as a backend runtime while adding autoscaling and K8s-native management
- **BentoML** — Packaging and deployment tool; KServe focuses on Kubernetes-native orchestration and autoscaling
- **Seldon Core** — Similar Kubernetes model serving; KServe is the CNCF standard with broader community adoption
- **Ray Serve** — Python-native serving framework; KServe is Kubernetes-native with richer deployment strategies

## FAQ
**Q: Does KServe require Knative?**
A: No. KServe supports a RawDeployment mode that works without Knative, using standard Kubernetes Deployments and HPA.

**Q: Can KServe serve LLMs?**
A: Yes. KServe integrates with vLLM, Hugging Face TGI, and Triton for serving large language models with GPU acceleration.

**Q: How does scale-to-zero work?**
A: In Knative mode, KServe scales pods to zero after a configurable idle timeout and spins them back up on incoming requests.

**Q: What model storage backends are supported?**
A: KServe supports S3, GCS, Azure Blob Storage, HDFS, and Kubernetes Persistent Volume Claims for model artifact storage.

## Sources
- https://github.com/kserve/kserve
- https://kserve.github.io/website/

---
Source: https://tokrepo.com/en/workflows/e691472c-39c9-11f1-9bc6-00163e2b0d79
Author: Script Depot