Open-weight AI on Kubernetes: A Developer’s Guide to vLLM vs. KubeAI

Deploying open-weight AI models on Kubernetes has shifted from experimental to expected. This guide addresses the central question of what sits between model weights and the cluster—comparing vLLM as a high-performance inference engine, KubeAI as a Kubernetes-native model management layer, and how they work together on GKE.

Table of Contents

• Why Open-Weight AI on Kubernetes Matters Now
• Prerequisites
• Key Concepts Before You Choose
• vLLM on Kubernetes: Direct Deployment
• KubeAI on Kubernetes: The Platform Approach
• Head-to-Head Comparison: vLLM vs. KubeAI
• GKE-Specific Considerations
• Implementation Checklist
• Decision Framework: Which Should You Pick?
• Putting It Together

Why Open-Weight AI on Kubernetes Matters Now

Deploying open-weight AI models on Kubernetes has shifted from experimental to expected. Organizations running Llama 3, Mistral, Qwen, and Gemma in production need serving infrastructure that matches the maturity of their container orchestration platform. The term "open-weight" is deliberate and distinct from "open-source." These models publish their trained weights for download and use, but their training data, training code, and licenses vary. Llama 3 ships under a custom Meta license; Mistral models carry Apache 2.0 for certain variants; Gemma uses Google's permissive terms. Verify the license for each specific model variant at the Hugging Face model card before production use. None of these qualify as "open-source" under the OSI definition, but all grant enough access for production deployment.

The infrastructure convergence makes this practical. GPU node pools are first-class citizens on GKE, EKS, and AKS. Kubernetes scheduling understands NVIDIA device plugins, topology-aware placement, and multi-GPU allocation. The missing layer is what sits between the model weights and the cluster: the serving infrastructure that turns a downloaded checkpoint into a reliable, scalable API endpoint.

vLLM operates as a high-performance inference engine. KubeAI operates as a Kubernetes-native model management layer that can use vLLM as its backend. These are not always competitors.

That is the central question this guide addresses. vLLM operates as a high-performance inference engine. KubeAI operates as a Kubernetes-native model management layer that can use vLLM as its backend. These are not always competitors. Understanding when to use each, and when to layer them together, determines whether a deployment costs a team days or weeks.

Prerequisites

Before following the deployment steps in this guide, confirm the following:

• Kubernetes cluster ≥ 1.27
• NVIDIA GPU node pool provisioned with the appropriate GPU type (e.g., L4 or A100)
• NVIDIA device plugin for Kubernetes installed and nvidia.com/gpu resource visible on GPU nodes (kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]')
• kubectl configured and pointed at the target cluster
• Helm ≥ 3.x installed locally
• A Hugging Face account with a token that has accepted the license for mistralai/Mistral-7B-Instruct-v0.3 (this is a gated model; you must accept the license at huggingface.co before your token can download it)

Key Concepts Before You Choose

Inference Engine vs. Model Orchestrator

vLLM is a purpose-built inference engine. It implements PagedAttention for efficient KV cache memory management, continuous batching for maximizing GPU throughput, and tensor parallelism for distributing large models across multiple GPUs. A single vLLM process serves one base model (with optional LoRA adapters). It handles tokenization, scheduling, sampling, and memory allocation directly against the GPU hardware.

KubeAI is a Kubernetes operator and platform. It manages the model lifecycle through Custom Resource Definitions (CRDs), handles autoscaling decisions, and provides unified routing to multiple models through a single API gateway. Critically, KubeAI can use vLLM as its underlying inference engine. It does not replace vLLM's inference capabilities; it orchestrates them.

The practical analogy: vLLM is the worker process handling requests. KubeAI is the operator that deploys, scales, and routes to those workers, analogous to a Deployment controller combined with a load balancer.

What "Production-Ready" Means on Kubernetes

Production readiness for LLM serving on Kubernetes involves several dimensions that general web services do not face. You must set explicit nvidia.com/gpu resource requests and proper node affinity to land Pods on the right hardware. Autoscaling from zero is essential for cost control when GPU nodes are expensive (verify current pricing for your GPU type and region at your cloud provider's pricing calculator). An OpenAI-compatible API surface matters because application teams have standardized on that interface. Model versioning and rollout strategies determine whether a model update drops requests or completes them gracefully during the swap.

vLLM on Kubernetes: Direct Deployment

When to Use vLLM Directly

Direct vLLM deployment makes sense in specific scenarios. Teams needing maximum control over inference parameters, including sampling configurations, quantization settings, and custom LoRA adapter management, benefit from direct access. Single-model or small-model-count deployments (one to two models) do not justify the overhead of an orchestration layer. If your team already runs complex Kubernetes workloads and can build the supporting infrastructure (HPA, KEDA, Prometheus monitors) without outside help, direct deployment avoids an unnecessary abstraction. When the goal is squeezing every token per second out of the hardware, eliminating any proxy layer matters.

Deploying vLLM with a Kubernetes Manifest

First, create the namespace and the Hugging Face token Secret required by the Deployment. Never pass secrets as CLI literals — this exposes the token in shell history and process table. Instead, store the token in a file and create the Secret from that file:

kubectl create namespace ml-serving

# Store your HF token in a file (chmod 600, add to .gitignore)
# echo "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > ./hf-token.txt

kubectl create secret generic hf-token \
  --from-file=token=./hf-token.txt \
  -n ml-serving

Next, create the PersistentVolumeClaim. If you plan to run multiple replicas on different nodes, use ReadWriteMany with an RWX-capable StorageClass (e.g., Filestore on GKE, EFS on EKS). If RWX storage is unavailable, keep ReadWriteOnce but set replicas: 1, or use a StatefulSet with per-replica PVCs instead.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-model-cache
  namespace: ml-serving
spec:
  accessModes:
    - ReadWriteMany   # Requires a RWX-capable StorageClass (e.g., Filestore on GKE, EFS on EKS)
  storageClassName: standard-rwx   # Replace with your RWX StorageClass name
  resources:
    requests:
      storage: 100Gi

Apply the PVC:

kubectl apply -f vllm-model-cache-pvc.yaml -n ml-serving

The following manifest deploys vLLM serving mistralai/Mistral-7B-Instruct-v0.3 on a single A100 GPU. Replace the image tag with the current stable tag from the vLLM releases page; never use latest in production manifests.

Note: The nodeSelector label cloud.google.com/gke-accelerator is GKE-specific. EKS and AKS use different node labeling conventions for GPU types. GPU nodes commonly have taints (e.g., nvidia.com/gpu=present:NoSchedule) to prevent non-GPU workloads from landing on them; the tolerations section below is required to satisfy these taints.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-mistral-7b
  namespace: ml-serving
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-mistral-7b
  template:
    metadata:
      labels:
        app: vllm-mistral-7b
    spec:
      terminationGracePeriodSeconds: 300   # 5 min; tune based on max expected response time
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-tesla-a100
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
        # GKE-specific taint added to GPU nodes in some configurations:
        - key: "cloud.google.com/gke-accelerator"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.4   # Pin to current stable; verify at github.com/vllm-project/vllm/releases
          args:
            - "--model"
            - "mistralai/Mistral-7B-Instruct-v0.3"
            - "--tensor-parallel-size"
            - "1"
            - "--max-model-len"
            - "8192"
            - "--gpu-memory-utilization"
            - "0.90"
          ports:
            - containerPort: 8000
              name: http
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
          env:
            - name: HUGGING_FACE_HUB_TOKEN  # Verified correct name for vLLM >= 0.4.1
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 60        # 60 * 10s = 10 min startup budget
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 15
            failureThreshold: 40        # 30s + (40 * 15s) = ~10 min max wait
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120    # Allow full model load before liveness kicks in
            periodSeconds: 30
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]  # Allow load balancer to drain connections
          volumeMounts:
            - name: model-cache
              mountPath: /root/.cache/huggingface
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: vllm-model-cache
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-mistral-7b
  namespace: ml-serving
spec:
  selector:
    app: vllm-mistral-7b
  ports:
    - port: 8000
      targetPort: 8000
      name: http
  type: ClusterIP

Additionally, create a PodDisruptionBudget to prevent voluntary disruptions from causing a total outage:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-mistral-7b-pdb
  namespace: ml-serving
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-mistral-7b

Note: minAvailable: 1 with replicas: 1 means voluntary disruptions (node drain, rolling update) will be blocked until a replacement pod is ready. This is the correct behavior for a single-replica GPU deployment but will stall kubectl drain if the replacement pod cannot be scheduled (e.g., no free GPU node). Increase replicas and use minAvailable: 1 with replicas >= 2 for drainable production deployments.

Key details: gpu-memory-utilization at 0.90 instructs vLLM to pre-allocate 90% of available GPU VRAM for the KV cache after loading model weights. Values above 0.95 risk out-of-memory failures under high concurrency. The PersistentVolumeClaim avoids re-downloading the model on every Pod restart. If the PVC is empty on first start, vLLM will download ~14GB from Hugging Face Hub, which can take 5-20 minutes depending on network speed; the Pod will appear Running but the readiness probe will prevent traffic from being routed until the model is fully loaded. The Hugging Face token is stored as a Kubernetes Secret and injected as an environment variable.

Once the Deployment rolls out and the readiness probe passes, verify with a direct request. Since the Service uses ClusterIP, you need to port-forward to reach it from your local machine:

kubectl port-forward svc/vllm-mistral-7b 8001:8000 -n ml-serving &
VLLM_PF_PID=$!

curl -X POST http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

# When done:
# kill $VLLM_PF_PID

Scaling and Operational Concerns

Horizontal scaling means running multiple replicas behind the same Service. Each replica loads the full model into its own GPU. Kubernetes Services round-robin requests across replicas, though the actual distribution depends on kube-proxy mode (iptables vs. IPVS) and connection reuse patterns. This works but extracts an operational burden: every replica requires its own GPU, and no request-aware load balancing exists by default. Note that multi-replica scaling requires a ReadWriteMany PVC or per-replica storage via a StatefulSet; a ReadWriteOnce PVC will cause scheduling failures when replicas land on different nodes.

vLLM has no native scale-to-zero capability. Achieving zero-replica idle states requires pairing vLLM with KEDA or custom HPA metrics. vLLM exposes Prometheus metrics useful for autoscaling, but metric names are version-dependent and have changed across releases. Verify the exact metric names for your vLLM version with curl http://<vllm-pod>:8000/metrics | grep 'num_requests'. Building a KEDA scaler against these works but requires you to write and maintain custom KEDA ScaledObject manifests.

To update a model, edit the Deployment spec and wait for the new Pod to download and load the model before the old one terminates. The PodDisruptionBudget defined above helps prevent request failures during this process.

KubeAI on Kubernetes: The Platform Approach

When to Use KubeAI

KubeAI targets a different operational profile. If your team manages a catalog of models for different groups, such as a coding assistant, a summarization model, and an embedding model, centralized management through a single operator pays for itself quickly. Scale-to-zero comes built in, with no KEDA configuration needed, and that matters when GPU hours are the largest line item. The CRD-based workflow means platform engineers declare desired state through Model custom resources rather than building Deployment, Service, HPA, and monitoring manifests for each model. In practice, this collapses what would be four or five manifests per model into a single Model CR.

The CRD-based workflow means platform engineers declare desired state through Model custom resources rather than building Deployment, Service, HPA, and monitoring manifests for each model. In practice, this collapses what would be four or five manifests per model into a single Model CR.

Deploying KubeAI with Helm and a Model CR

Install KubeAI using Helm with a values file that targets a GKE cluster with GPU node pools and enables the vLLM backend. Specify the exact chart version; check available versions with helm search repo kubeai/kubeai --versions.

# kubeai-values.yaml
modelServerPods:
  nodeSelector:
    cloud.google.com/gke-accelerator: nvidia-l4
resourceProfiles:
  nvidia-gpu-l4:
    requests:
      cpu: "4"
      memory: "16Gi"
      nvidia.com/gpu: "1"
    limits:
      cpu: "8"
      memory: "32Gi"
      nvidia.com/gpu: "1"

helm repo add kubeai https://www.kubeai.org
helm repo update
helm install kubeai kubeai/kubeai --version <chart-version> \
  --namespace kubeai --create-namespace \
  -f kubeai-values.yaml

Replace <chart-version> with a specific version such as 0.10.0. Check KubeAI releases for the latest stable chart version.

Create the Hugging Face token Secret in the kubeai namespace. As with the vLLM deployment, never pass secrets as CLI literals:

# Store your HF token in a file (chmod 600, add to .gitignore)
# echo "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > ./hf-token.txt

kubectl create secret generic hf-token \
  --from-file=token=./hf-token.txt \
  -n kubeai

With KubeAI running, define a Model custom resource to deploy Mistral-7B using vLLM as the engine. Verify the API version stability in the KubeAI release notes for the installed chart version; CRD schemas may change between minor versions.

apiVersion: kubeai.org/v1
kind: Model
metadata:
  name: mistral-7b-instruct
  namespace: kubeai
  annotations:
    kubeai.org/scale-up-timeout: "300s"   # Verify annotation name in KubeAI CRD docs for your version
spec:
  url: "hf://mistralai/Mistral-7B-Instruct-v0.3"
  engine: VLLM
  resourceProfile: nvidia-gpu-l4:1
  minReplicas: 0
  maxReplicas: 3
  args:
    - "--max-model-len"
    - "8192"
    - "--gpu-memory-utilization"
    - "0.90"
  env:
    - name: HUGGING_FACE_HUB_TOKEN
      valueFrom:
        secretKeyRef:
          name: hf-token
          key: token

Verify this env schema against the installed KubeAI CRD version; the format follows the standard Kubernetes EnvVar spec.

Setting minReplicas: 0 enables scale-to-zero. When no requests arrive for the configured idle period, KubeAI terminates the inference Pod and releases the GPU. When a new request arrives, KubeAI spins up the Pod, loads the model, and serves the request. Be aware that the first request after scale-to-zero triggers a cold start that typically takes 30-120 seconds; if the client timeout is shorter than the cold-start time, the request will fail. Set client-side timeouts accordingly (e.g., --max-time 300 with curl).

Testing against KubeAI's gateway: first, confirm the gateway Service name after install, as the default Service name in the chart may differ from what is shown here:

kubectl get svc -n kubeai

Then port-forward to the gateway Service and test. Use a different local port than the vLLM port-forward to avoid bind conflicts:

kubectl port-forward svc/<gateway-service-name> 8002:80 -n kubeai &
KUBEAI_PF_PID=$!

curl -X POST http://localhost:8002/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  --max-time 300 \
  -d '{
    "model": "mistral-7b-instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

# When done:
# kill $KUBEAI_PF_PID

Replace <gateway-service-name> with the actual Service name from the kubectl get svc output.

Note the model field uses the name from the Model CR metadata, not the Hugging Face path. KubeAI routes the request to the correct backend based on this identifier.

What KubeAI Manages for You

KubeAI handles automatic Pod scheduling with GPU affinity derived from the resource profile. Cold-start latency is real: loading a 7B model from an SSD-backed PersistentVolume typically takes 30-120 seconds, and first-pull from Hugging Face Hub or slow network-attached storage may take 5-15 minutes. Size client timeouts accordingly. Scale-to-zero and scale-from-zero are first-class features, not bolt-on additions. The model catalog routes multiple models through a single gateway endpoint, with the model field in the request body determining which backend receives the request. KubeAI ships with preconfigured support for popular open-weight models, reducing boilerplate for common deployments.

Head-to-Head Comparison: vLLM vs. KubeAI

Feature Comparison Table

Performance Considerations

KubeAI adds a thin routing and proxy layer between the client and the vLLM inference process. For typical LLM workloads where per-request inference takes 100ms to several seconds, the proxy adds minimal latency. Benchmark your specific throughput target with curl -w '%{time_starttransfer}' against both direct vLLM and KubeAI-proxied endpoints to measure the actual difference in your environment.

Cold-start latency from scale-to-zero deserves explicit attention. The model load time dominates: 30 to 120 seconds for 7B parameter models when weights reside on a fast SSD-backed PersistentVolume. First-pull from Hugging Face Hub or slow network-attached storage may take 5-15 minutes. Teams using scale-to-zero must set client-side timeout expectations accordingly and consider keeping a minimum of one replica for latency-sensitive models.

When raw throughput tuning matters, such as maximizing tokens per second for a single high-traffic model, direct vLLM deployment with custom engine arguments and full access to sampling parameters provides the tightest control.

GKE-Specific Considerations

GPU Node Pools and Autopilot

On GKE, configuring GPU node pools requires selecting the accelerator type. L4 GPUs (24GB VRAM) handle models up to 13B parameters in fp16 precision, assuming no KV cache pressure from long context lengths. A100 80GB GPUs can serve 70B parameter models in fp16. A100 40GB GPUs require quantization (GPTQ/AWQ) to serve 70B models. Verify VRAM requirements for your precision setting. These estimates assume fp16 (2 bytes per parameter) precision. Quantized models (4-bit GPTQ/AWQ) may allow larger models on the same VRAM. H100 GPUs with multi-node configurations handle larger models. Spot (preemptible) GPU instances cost less than on-demand, but discount percentages vary by GPU type and region. Verify current pricing at the GCP Pricing Calculator. Spot instances risk interruption, making them suitable for development and burst workloads rather than steady production traffic.

GKE Autopilot supports GPU workloads but imposes constraints: resource requests must match specific machine profiles, and not all GPU types are available in all regions. Both vLLM and KubeAI function on Autopilot, but Autopilot's opinionated node management can conflict with fine-grained GPU scheduling. For example, Autopilot may reject Pods that request non-standard CPU/memory ratios alongside GPU resources.

GKE + KubeAI Integration Points

Workload Identity allows Pods to authenticate as Google Cloud service accounts, enabling secure model pulls from GCS buckets or Artifact Registry without embedding credentials. For the model gateway, an internal load balancer keeps traffic within the VPC.

Warning: If the networking.gke.io/load-balancer-type: Internal annotation is missing or misconfigured, the LoadBalancer Service will create a public IP, exposing the model gateway to the internet.

The following is a separate GKE-specific values file. Merge it with kubeai-values.yaml into a single file, or pass both to Helm with -f kubeai-values.yaml -f kubeai-gke-values.yaml.

Verify the correct values path for serviceAccountAnnotations in your KubeAI chart version before applying:

helm show values kubeai/kubeai --version <version> | grep -A10 serviceAccount

The path shown below may differ between chart versions; incorrect placement will cause Workload Identity to silently fail. After applying, verify with:

kubectl describe serviceaccount -n kubeai | grep "iam.gke.io/gcp-service-account"

# kubeai-gke-values.yaml
# REPLACE: use your GCP project ID in the service account email below
modelServerPods:
  nodeSelector:
    cloud.google.com/gke-accelerator: nvidia-l4
  serviceAccountAnnotations:
    iam.gke.io/gcp-service-account: kubeai-model-puller@my-project.iam.gserviceaccount.com
service:
  type: LoadBalancer
  annotations:
    networking.gke.io/load-balancer-type: Internal
resourceProfiles:
  nvidia-gpu-l4:
    requests:
      cpu: "4"
      memory: "16Gi"
      nvidia.com/gpu: "1"
    limits:
      cpu: "8"
      memory: "32Gi"
      nvidia.com/gpu: "1"

Implementation Checklist

1. ☐ Choose a model and verify GPU memory requirements. Estimate GPU VRAM as: (parameters × 2 bytes for fp16) + KV cache (~10-30% of VRAM, scaling with max-model-len) + ~2GB framework overhead. Validate empirically before production.
2. ☐ Provision GPU node pool (L4 for models up to 13B in fp16, A100-80GB for up to 70B in fp16, A100-40GB for quantized models, multi-node H100 for larger).
3. ☐ Decide: single-model (vLLM direct) or multi-model/scale-to-zero (KubeAI).
4. ☐ If vLLM direct: create Deployment, Service, PodDisruptionBudget, HPA/KEDA scaler, Prometheus ServiceMonitor.
5. ☐ If KubeAI: Helm install with pinned chart version, create Model CRs, configure autoscaling profile.
6. ☐ Validate OpenAI-compatible endpoint with a test request.
7. ☐ Configure monitoring: GPU utilization, request latency p50/p99, queue depth.
8. ☐ Set resource quotas and PodDisruptionBudgets for production stability.
9. ☐ Document cold-start latency for scale-to-zero models; set client timeout expectations.
10. ☐ Establish model update workflow (CR update for KubeAI; rolling deployment for vLLM).

Decision Framework: Which Should You Pick?

Use vLLM Directly When...

The deployment involves one or two models and the priority is maximum throughput tuning. The team already operates complex Kubernetes workloads and is comfortable building HPA policies, Prometheus scrapers, and custom rolling update strategies. Recent vLLM features (v0.5+) like speculative decoding or serving multiple LoRA adapters at scale require direct access to engine arguments.

Use KubeAI When...

The organization manages a catalog of models for multiple teams or use cases. GPU cost control via scale-to-zero is a hard requirement, not a nice-to-have. The team wants a faster path to production with less custom YAML and fewer moving parts to maintain. Using vLLM under the hood is acceptable, but managing it directly is not where the team should spend its engineering time.

Putting It Together

vLLM and KubeAI are complementary more than competitive. vLLM provides the inference performance layer. KubeAI provides the operational management layer. The decision is about operational investment versus abstraction: how much infrastructure plumbing a team wants to own versus delegate.

The decision is about operational investment versus abstraction: how much infrastructure plumbing a team wants to own versus delegate.

The manifests, Helm values, and curl commands in this guide are starting points. Adapt the resource profiles to the specific GPU hardware available, adjust autoscaling parameters based on observed traffic patterns, and use the checklist to track deployment completeness.

Pin your versions, run the checklist, and revisit when KubeAI's multi-cluster model distribution and vLLM's disaggregated serving hit stable releases.