KServe Model Serving Complete Guide: InferenceService, Canary Deployment, Transformer, and InferenceGraph Production Operations

• Introduction
• InferenceService CRD in Depth
  • Predictor / Transformer / Explainer Architecture
  • Supported Runtime Comparison
  • A Basic InferenceService YAML
  • LLM Serving on the vLLM Backend
• Canary Deployment Strategy
  • Gradual Rollout with canaryTrafficPercent
  • A Step-by-Step Promotion Script
• Implementing a Custom Transformer
  • Designing the Pre/Post-Processing Pipeline
  • Subclassing kserve.Model and Implementing the Handlers
  • An InferenceService YAML Including the Transformer
• InferenceGraph: DAG-Based Composite Inference
  • The 4 Node Types
  • A/B Testing and Ensemble Patterns
  • The Splitter Pattern for A/B Testing
• Autoscaling Strategy
  • Knative Pod Autoscaler (KPA) vs HPA vs KEDA
  • Scale-to-Zero for GPU Workloads
  • KEDA + vLLM Metric Autoscaling
• New Features in v0.15
  • Envoy AI Gateway Integration
  • LocalModelCache Multi-Node Groups
  • LLM Serving Optimizations
• Failure Cases and Troubleshooting
  • Case 1: A Transformer OOM Halting the Inference Pipeline
  • Case 2: Cold Start Delay Under Scale-to-Zero
  • Debugging Checklist
• Operational Considerations
  • GPU Node Scheduling and tolerations
  • Model Caching Strategy
  • Managing Resource Quotas
• References

Introduction

Training an ML model and serving it reliably in production are fundamentally different engineering problems. In training, GPU utilization and convergence speed are what matter; in serving, latency, throughput, version management, safe rollout, and failure recovery are decisive. In a composite inference pipeline where several models interact, simply putting a model behind Flask cannot cope with the operational complexity.

KServe (formerly KFServing) is a CNCF Incubating project designed to solve these production model serving problems in a Kubernetes-native way. You deploy models declaratively through the InferenceService CRD, respond to traffic swings with Knative-based autoscaling, roll out safely with Canary deployment, and build DAG-based composite inference with InferenceGraph.

KServe started in 2019 as part of the Kubeflow project under the name KFServing, and was rebranded to KServe in 2021 when it split off as an independent project. After joining the CNCF Sandbox in 2023 it grew quickly and was promoted to Incubating in 2025. It supports all the major inference runtimes - TFServing, TorchServe, Triton, vLLM - and, in step with the LLM era, is expanding to a vLLM backend and Envoy AI Gateway integration.

This article covers everything you need in practice, with code, from KServe's core architecture through to production operating strategy.

InferenceService CRD in Depth

Predictor / Transformer / Explainer Architecture

A KServe InferenceService is made up of three core components.

Client Request
      │
      ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│ Transformer │────▶│  Predictor  │────▶│  Explainer  │
│ (pre/post-  │     │ (model      │     │ (generates  │
│  processing)│◀────│  inference) │     │  the reason)│
└─────────────┘     └─────────────┘     └─────────────┘
      │
      ▼
Client Response

• Predictor: the core inference engine. It supports a range of runtimes including TFServing, TorchServe, Triton, XGBoost, LightGBM, Sklearn, and vLLM.
• Transformer: handles preprocessing of the inference request and post-processing of the response. Image resizing, tokenization, feature engineering and the like are separated from the Predictor so they can scale independently.
• Explainer: generates explanations of the inference result. Through Alibi Explainer, AIF360 and similar tools it provides the reasoning behind a model's prediction.

Supported Runtime Comparison

A Basic InferenceService YAML

The most basic form of an InferenceService defines only a Predictor. Below is an example serving a Sklearn model stored on S3.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: ml-serving
  annotations:
    serving.kserve.io/deploymentMode: Serverless
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 10
    scaleTarget: 5
    scaleMetric: concurrency
    model:
      modelFormat:
        name: sklearn
      storageUri: 's3://ml-models/sklearn/iris/v1'
      resources:
        requests:
          cpu: '500m'
          memory: '512Mi'
        limits:
          cpu: '1'
          memory: '1Gi'

When you apply this YAML, the KServe controller creates a Knative Service, sets up routing through an Istio VirtualService, and the Knative Pod Autoscaler autoscales based on concurrency.

LLM Serving on the vLLM Backend

From KServe v0.13, vLLM is supported as a first-class runtime. It exposes an OpenAI-compatible API automatically, so existing OpenAI client code can be used unchanged.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-vllm
  namespace: ml-serving
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 4
    model:
      modelFormat:
        name: vLLM
      storageUri: 'pvc://llm-model-cache/Meta-Llama-3.1-8B-Instruct'
      args:
        - '--max-model-len=8192'
        - '--gpu-memory-utilization=0.90'
        - '--enable-chunked-prefill'
        - '--max-num-batched-tokens=16384'
        - '--tensor-parallel-size=2'
      resources:
        requests:
          cpu: '8'
          memory: '32Gi'
          nvidia.com/gpu: '2'
        limits:
          cpu: '16'
          memory: '64Gi'
          nvidia.com/gpu: '2'
    tolerations:
      - key: 'nvidia.com/gpu'
        operator: 'Exists'
        effect: 'NoSchedule'
    nodeSelector:
      gpu-type: 'a100'

Pointing storageUri at a PVC lets you cache large LLM weights on the node in advance and load them quickly. tensor-parallel-size=2 spreads the model across 2 GPUs, getting past the memory limit of a single GPU.

Canary Deployment Strategy

Gradual Rollout with canaryTrafficPercent

The most dangerous moment when updating a model in production is right after the new version is deployed. KServe natively supports Canary deployment through the canaryTrafficPercent field, moving traffic to the new version step by step.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
  namespace: ml-serving
  annotations:
    serving.kserve.io/deploymentMode: Serverless
spec:
  predictor:
    canaryTrafficPercent: 10
    minReplicas: 2
    maxReplicas: 20
    model:
      modelFormat:
        name: sklearn
      storageUri: 's3://ml-models/fraud/v2'
      resources:
        requests:
          cpu: '1'
          memory: '2Gi'
        limits:
          cpu: '2'
          memory: '4Gi'

The configuration above routes only 10% of total traffic to the new model (v2) and sends the remaining 90% to the existing stable version. After deployment you validate the new version's performance through monitoring, then raise canaryTrafficPercent in stages.

A Step-by-Step Promotion Script

Automating the gradual rollout with a script reduces manual mistakes.

#!/bin/bash
# canary-promote.sh - gradual Canary promotion script
ISVC_NAME="fraud-detector"
NAMESPACE="ml-serving"
STAGES=(10 30 50 80 100)
MAX_ERROR_RATE=0.05
OBSERVE_MINUTES=10

for pct in "${STAGES[@]}"; do
  echo "=== Updating Canary traffic to ${pct}% ==="

  kubectl patch inferenceservice "$ISVC_NAME" -n "$NAMESPACE" \
    --type='json' \
    -p="[{\"op\": \"replace\", \"path\": \"/spec/predictor/canaryTrafficPercent\", \"value\": $pct}]"

  echo "Observing metrics for ${OBSERVE_MINUTES} minutes..."
  sleep $((OBSERVE_MINUTES * 60))

  # Query the canary error rate from Prometheus
  ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query" \
    --data-urlencode "query=rate(revision_request_count{revision_name=~\".*canary.*\",response_code!=\"200\"}[5m]) / rate(revision_request_count{revision_name=~\".*canary.*\"}[5m])" \
    | jq -r '.data.result[0].value[1] // "0"')

  if (( $(echo "$ERROR_RATE > $MAX_ERROR_RATE" | bc -l) )); then
    echo "Error rate ${ERROR_RATE} exceeds the threshold ${MAX_ERROR_RATE}. Rolling back."
    kubectl patch inferenceservice "$ISVC_NAME" -n "$NAMESPACE" \
      --type='json' \
      -p='[{"op": "replace", "path": "/spec/predictor/canaryTrafficPercent", "value": 0}]'
    exit 1
  fi

  echo "Error rate ${ERROR_RATE} - within the normal range. Moving to the next stage."
done

echo "=== Canary promotion complete. v2 is now handling 100% of traffic. ==="

This script shifts traffic from 10% to 100% in 5 stages, monitoring the error rate for 10 minutes at each stage. If the error rate exceeds 5% it rolls back to 0% automatically.

Implementing a Custom Transformer

Designing the Pre/Post-Processing Pipeline

In real production you have to convert the raw data a client sends (an image URL, text, JSON) into the tensor format the model expects. Separating the Transformer from the Predictor brings the following benefits.

• Independent scaling: when preprocessing is CPU-intensive and inference is GPU-intensive, each can have its own scaling policy
• Reusability: the same Transformer can be reused across several model versions
• Deployment independence: changes to Transformer logic can be shipped without redeploying the model

Subclassing kserve.Model and Implementing the Handlers

You implement a Custom Transformer by subclassing the kserve.Model class from the KServe Python SDK.

import kserve
from kserve import InferRequest, InferResponse, InferInput
from typing import Dict, List
import numpy as np
from PIL import Image
import requests
from io import BytesIO
import logging

logger = logging.getLogger(__name__)

class ImageTransformer(kserve.Model):
    """A Transformer that takes an image URL, preprocesses it, and passes it to the Predictor"""

    def __init__(self, name: str, predictor_host: str):
        super().__init__(name)
        self.predictor_host = predictor_host
        self.target_size = (224, 224)
        self.mean = np.array([0.485, 0.456, 0.406])
        self.std = np.array([0.229, 0.224, 0.225])
        self.ready = False

    def load(self):
        """Model initialization. Put warm-up logic here."""
        logger.info("ImageTransformer initialization complete")
        self.ready = True

    def preprocess(
        self, payload: Dict, headers: Dict = None
    ) -> InferRequest:
        """Take an image URL and convert it into a normalized tensor"""
        instances = payload.get("instances", [])
        processed_images = []

        for instance in instances:
            image_url = instance.get("image_url")
            response = requests.get(image_url, timeout=10)
            response.raise_for_status()

            image = Image.open(BytesIO(response.content)).convert("RGB")
            image = image.resize(self.target_size)

            # Convert to a numpy array, then normalize
            img_array = np.array(image, dtype=np.float32) / 255.0
            img_array = (img_array - self.mean) / self.std
            img_array = np.transpose(img_array, (2, 0, 1))  # CHW
            processed_images.append(img_array)

        input_tensor = np.stack(processed_images)

        infer_input = InferInput(
            name="input",
            shape=list(input_tensor.shape),
            datatype="FP32",
            data=input_tensor.tolist(),
        )
        return InferRequest(
            model_name=self.name,
            infer_inputs=[infer_input],
        )

    def postprocess(
        self, response: InferResponse, headers: Dict = None
    ) -> Dict:
        """Convert the model output into a human-readable form"""
        predictions = response.outputs[0].data

        class_names = ["cat", "dog", "bird", "fish", "other"]
        results = []

        for pred in predictions:
            if isinstance(pred, list):
                probs = np.array(pred)
            else:
                probs = np.array([pred])

            top_idx = int(np.argmax(probs))
            results.append({
                "class": class_names[top_idx],
                "confidence": float(probs[top_idx]),
                "all_scores": {
                    name: float(score)
                    for name, score in zip(class_names, probs)
                },
            })

        return {"predictions": results}


if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--predictor_host", required=True)
    parser.add_argument("--model_name", default="image-classifier")
    args = parser.parse_args()

    transformer = ImageTransformer(
        name=args.model_name,
        predictor_host=args.predictor_host,
    )
    transformer.load()
    kserve.ModelServer(workers=4).start([transformer])

An InferenceService YAML Including the Transformer

When you integrate the Transformer into the InferenceService, KServe sets up the internal routing between Transformer and Predictor automatically.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: image-classifier
  namespace: ml-serving
spec:
  transformer:
    minReplicas: 2
    maxReplicas: 15
    scaleTarget: 10
    scaleMetric: concurrency
    containers:
      - name: image-transformer
        image: registry.example.com/ml/image-transformer:v1.2.0
        args:
          - '--model_name=image-classifier'
        resources:
          requests:
            cpu: '1'
            memory: '2Gi'
          limits:
            cpu: '2'
            memory: '4Gi'
        env:
          - name: STORAGE_URI
            value: ''
  predictor:
    minReplicas: 1
    maxReplicas: 8
    model:
      modelFormat:
        name: pytorch
      storageUri: 's3://ml-models/image-classifier/resnet50-v2'
      resources:
        requests:
          cpu: '2'
          memory: '4Gi'
          nvidia.com/gpu: '1'
        limits:
          cpu: '4'
          memory: '8Gi'
          nvidia.com/gpu: '1'

In this setup the Transformer scales out to a maximum of 15 on CPU nodes, and the Predictor to a maximum of 8 on GPU nodes. When preprocessing is the bottleneck you only need to add Transformers, which is cost efficient.

InferenceGraph: DAG-Based Composite Inference

The 4 Node Types

InferenceGraph is KServe's advanced feature for chaining several InferenceServices into a DAG to form a composite inference pipeline. It was promoted to GA in v0.11 and supports 4 node types.

A/B Testing and Ensemble Patterns

A pattern used often in real production is the combination of an ensemble with A/B testing. For example, in a fraud detection system, running a rule-based model, an XGBoost model, and a deep learning model at the same time and ensembling their results can raise accuracy over any single model.

apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
  name: fraud-detection-ensemble
  namespace: ml-serving
  annotations:
    serving.kserve.io/propagateHeaders: 'x-request-id,x-trace-id'
spec:
  nodes:
    root:
      routerType: Sequence
      steps:
        - name: feature-enrichment
          serviceName: feature-enricher
          weight: 100
        - name: ensemble-node
          nodeName: model-ensemble
    model-ensemble:
      routerType: Ensemble
      steps:
        - name: xgboost-model
          serviceName: fraud-xgboost
          weight: 40
        - name: deep-model
          serviceName: fraud-deep-learning
          weight: 40
        - name: rule-engine
          serviceName: fraud-rule-engine
          weight: 20
    result-combiner:
      routerType: Sequence
      steps:
        - name: weighted-average
          serviceName: ensemble-combiner
          weight: 100

This InferenceGraph works as follows.

1. root node (Sequence): first feature-enricher turns the raw data into enriched features, then passes the result to the model-ensemble node.
2. model-ensemble node (Ensemble): the three models - XGBoost, deep learning, and the rule engine - run in parallel on the same input. Each model's weight is used as its weighting when the final results are combined.
3. The final result is produced as the weighted average of the three models.

The Splitter Pattern for A/B Testing

apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
  name: recommendation-ab-test
  namespace: ml-serving
spec:
  nodes:
    root:
      routerType: Splitter
      steps:
        - name: model-v1-stable
          serviceName: recommender-v1
          weight: 80
        - name: model-v2-experiment
          serviceName: recommender-v2
          weight: 20

The Splitter distributes 80% of traffic to the stable version and 20% to the experimental one. Unlike an InferenceService Canary, an InferenceGraph Splitter splits traffic between fully independent InferenceServices, which makes it suited to A/B testing across different model architectures.

Autoscaling Strategy

Knative Pod Autoscaler (KPA) vs HPA vs KEDA

KServe supports three autoscalers. Which one fits depends on the characteristics of the workload.

Scale-to-Zero for GPU Workloads

GPU instances are expensive, so Scale-to-Zero matters. But GPU models have long cold start times (from tens of seconds to a few minutes), so care is needed. When using KPA you have to set the scaledown delay appropriately.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: gpu-model
  namespace: ml-serving
  annotations:
    # Enable Scale-to-Zero (the default)
    serving.kserve.io/enable-scale-to-zero: 'true'
    # How long to wait after the last request before scaling to zero
    autoscaling.knative.dev/scale-to-zero-pod-retention-period: '15m'
    # Scaledown delay (prevents abrupt shrinking)
    autoscaling.knative.dev/scale-down-delay: '5m'
    # Number of requests handled concurrently (set low for GPU models)
    autoscaling.knative.dev/target: '2'
    autoscaling.knative.dev/metric: 'concurrency'
spec:
  predictor:
    minReplicas: 0
    maxReplicas: 4
    model:
      modelFormat:
        name: pytorch
      storageUri: 's3://ml-models/large-model/v1'
      resources:
        requests:
          nvidia.com/gpu: '1'
        limits:
          nvidia.com/gpu: '1'

KEDA + vLLM Metric Autoscaling

For LLM serving it is effective to scale on vLLM's own metrics (number of waiting requests, KV cache utilization) rather than on CPU or memory. KEDA's Prometheus Scaler lets you implement that.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llama3-vllm-scaler
  namespace: ml-serving
spec:
  scaleTargetRef:
    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    name: llama3-vllm
  minReplicaCount: 1
  maxReplicaCount: 8
  pollingInterval: 15
  cooldownPeriod: 300
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
          policies:
            - type: Pods
              value: 2
              periodSeconds: 60
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Pods
              value: 1
              periodSeconds: 120
  triggers:
    # Scaling based on the number of waiting vLLM requests
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: vllm_waiting_requests
        query: |
          avg(vllm:num_requests_waiting{model_name="llama3-vllm"})
        threshold: '5'
    # Scaling based on KV Cache utilization
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: vllm_kv_cache_usage
        query: |
          avg(vllm:gpu_cache_usage_perc{model_name="llama3-vllm"})
        threshold: '0.85'

This configuration triggers a scale-out if either of two conditions is met.

• Add a Pod when waiting vLLM requests exceed an average of 5.
• Add a Pod when GPU KV cache utilization exceeds 85%.

On scaledown, a 5-minute stabilization window and a policy of removing 1 Pod every 2 minutes are applied to prevent abrupt shrinking.

New Features in v0.15

Envoy AI Gateway Integration

The most notable feature in KServe v0.15 (June 2025) is Envoy AI Gateway integration. It provides routing and observability geared to LLM serving.

• Token-based rate limiting: caps token usage per minute or per hour by API key
• Model-level routing: routes to the right backend (vLLM, Triton, an external API) based on the request's model field
• Semantic Caching: caches responses to similar prompts, cutting cost and latency
• Usage Tracking: tracks token usage, latency, error rate and more per model, user, or team

LocalModelCache Multi-Node Groups

Downloading large LLM weights from S3 every time makes cold start take minutes. LocalModelCache in v0.15 caches the model on node-local disk in advance, cutting startup time dramatically.

apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelCache
metadata:
  name: llama3-cache
  namespace: ml-serving
spec:
  sourceModelUri: 's3://ml-models/Meta-Llama-3.1-8B-Instruct'
  nodeGroups:
    - name: a100-nodes
      nodeSelector:
        gpu-type: 'a100'
      persistentVolumeClaimSpec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: local-nvme

This configuration creates a 50Gi NVMe volume on every node labelled gpu-type: a100 and pre-downloads the model weights from S3. When a Pod starts it loads the weights from local disk instead of S3, so cold start drops from minutes to seconds.

LLM Serving Optimizations

v0.15 included a number of optimizations for LLM workloads.

• LoRA adapter hot-swapping: keeps the base model in place and swaps only the LoRA adapter dynamically. This serves per-customer fine-tuned models efficiently in a multi-tenant environment
• Speculative Decoding support: speculative decoding using a Draft Model on the vLLM backend improves token generation speed by 2-3x
• Prefix Caching: shares the KV cache of repeated prefixes such as the system prompt, shortening TTFT (Time To First Token)

Failure Cases and Troubleshooting

Case 1: A Transformer OOM Halting the Inference Pipeline

Symptom: in an image classification pipeline, the Transformer Pod was intermittently OOMKilled and the whole inference chain failed.

Cause analysis: when the Transformer loaded large images (8K resolution) into memory, PIL allocated memory equal to the decompressed original size. With 10 concurrent requests, 10 x 200MB = 2GB was needed at once, but the memory limit was set to 1Gi.

Fix:

# Apply the image size cap at the earliest preprocessing stage
from PIL import Image

# PIL DecompressionBomb protection setting
Image.MAX_IMAGE_PIXELS = 89_478_485  # about 9500x9500

def safe_load_image(image_bytes: bytes, max_size: int = 2048):
    """Memory-safe image load"""
    img = Image.open(BytesIO(image_bytes))

    # Resize immediately if the original exceeds max_size
    if max(img.size) > max_size:
        ratio = max_size / max(img.size)
        new_size = (int(img.width * ratio), int(img.height * ratio))
        img = img.resize(new_size, Image.LANCZOS)

    return img.convert("RGB")

In addition, the Transformer's memory limit was raised to 4Gi and the concurrency target lowered to 5 to cap concurrent processing.

Case 2: Cold Start Delay Under Scale-to-Zero

Symptom: a GPU model in the Scale-to-Zero state timed out after more than 60 seconds on the first request.

Cause analysis: the time to download the model weights (2GB) from S3 and load them into GPU memory exceeded Knative's default timeout (30 seconds).

Fix:

1. Extend progressDeadline to 600 seconds
2. Apply LocalModelCache to pre-cache the model on the node locally
3. Set minReplicas: 1 to keep at least 1 Pod running (where the cost is acceptable)

Debugging Checklist

These are the items to check systematically when debugging a KServe deployment.

1. Check the InferenceService status: confirm READY is True in kubectl get isvc -n ml-serving
2. Check Pod status: inspect the Pods with kubectl get pods -n ml-serving -l serving.kserve.io/inferenceservice=MODEL_NAME
3. Check events: look at the Events section of kubectl describe isvc MODEL_NAME -n ml-serving
4. Check the Knative Revision: inspect revision status with kubectl get revisions -n ml-serving
5. Check storage access: look for S3/GCS access errors in the StorageInitializer logs
6. Check Istio routing: inspect the traffic routing rules with kubectl get virtualservice -n ml-serving
7. Check for insufficient resources: check allocatable GPU and free memory with kubectl describe node

Operational Considerations

GPU Node Scheduling and tolerations

GPU nodes usually carry a taint, so tolerations and nodeSelector must be stated explicitly on the InferenceService. Omit them and the Pod stays in Pending.

spec:
  predictor:
    tolerations:
      - key: 'nvidia.com/gpu'
        operator: 'Exists'
        effect: 'NoSchedule'
    nodeSelector:
      gpu-type: 'a100'
    model:
      modelFormat:
        name: vLLM
      # ... model configuration

Model Caching Strategy

When running large models, a tiered caching strategy is essential.

• Tier 1 cache (node local): store model weights on NVMe SSD using LocalModelCache. The fastest load speed
• Tier 2 cache (PVC): cache on network storage using a PersistentVolumeClaim. Shareable across nodes
• Tier 3 origin (Object Storage): keep the model's original on S3, GCS and the like. The slowest, but the most durable

Managing Resource Quotas

Set a per-namespace ResourceQuota to prevent teams from contending over GPU resources.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: ml-serving-quota
  namespace: ml-serving
spec:
  hard:
    requests.cpu: '64'
    requests.memory: '256Gi'
    requests.nvidia.com/gpu: '8'
    limits.cpu: '128'
    limits.memory: '512Gi'
    limits.nvidia.com/gpu: '8'
    persistentvolumeclaims: '20'

This Quota limits the ml-serving namespace to a maximum of 8 GPUs, 128 CPU cores, and 512Gi of memory. Separating teams into their own namespaces and assigning each an appropriate Quota makes stable multi-tenant operation possible.

References

• KServe Documentation
• InferenceService CRD API
• Canary Rollout Strategy
• Data Plane Architecture
• InferenceGraph Concept
• KServe v0.15 Release | CNCF
• KServe Autoscaling with KEDA | Red Hat