From Ollama to vLLM: A Migration Guide for Growing Teams

How to Migrate from Ollama to vLLM

1. Audit your current Ollama models and map each to its HuggingFace equivalent.
2. Provision a GPU host with 16 GB+ VRAM and install nvidia-container-toolkit.
3. Deploy vLLM via Docker on a separate port alongside the running Ollama instance.
4. Build a Node.js proxy that translates Ollama request/response formats to OpenAI-compatible payloads.
5. Run shadow mode for 24–48 hours, logging vLLM latency without serving live traffic.
6. Route 10% of traffic to vLLM as a canary, monitoring errors and output quality.
7. Scale canary traffic to 50%, then 100%, restarting the proxy at each step.
8. Decommission Ollama after a 48-hour bake period with stable vLLM metrics.

Migrating from Ollama to vLLM solves the shared inference bottleneck, but the path between the two is poorly documented and full of subtle format mismatches that will break a working application. This guide provides a decision framework, a detailed architecture comparison, and a step-by-step migration with zero downtime using a Node.js proxy layer.

Table of Contents

• When Ollama Starts Holding Your Team Back
• Prerequisites
• Ollama vs. vLLM: Understanding the Architectural Differences
• Five Signs It's Time to Migrate
• Pre-Migration Setup
• Setting Up vLLM Alongside Ollama
• Building the Migration Proxy in Node.js
• Migrating Request and Response Formats
• Running the Canary Migration
• Post-Migration Optimization
• Complete Migration Checklist
• What Changes and What Doesn't

When Ollama Starts Holding Your Team Back

The scenario is immediately recognizable: three engineers hit the shared inference endpoint during a standup demo, and response latency spikes from two seconds to fifteen. Someone restarts the service. Someone else switches to a local instance. The demo stalls. Migrating from Ollama to vLLM solves this problem, but the path between the two is poorly documented and full of subtle format mismatches that will break a working application.

Ollama is a developer-friendly local LLM runner built for simplicity. It wraps model downloading, quantization, and serving into a single CLI tool. vLLM is a production-grade inference engine that uses continuous batching and PagedAttention to serve dozens or hundreds of concurrent users on the same GPU hardware. Both serve language models over HTTP, but the architectural differences between them dictate entirely different scaling characteristics.

This guide provides a decision framework, a detailed architecture comparison, and a step-by-step migration with zero downtime using a Node.js proxy layer. It targets teams of 3 to 15 engineers currently running Ollama in shared development or staging environments and evaluating production inference options.

Prerequisites

Before starting, confirm every item on this list:

• Node.js 18+ — all scripts in this guide require native fetch and ES module support. Run node --version to verify.
• Linux host with NVIDIA GPU — CUDA drivers do not work on Windows Docker without WSL2, and macOS does not support GPU passthrough.
• NVIDIA GPU with at least 16 GB VRAM (24 GB recommended for Llama 3 8B-class models).
• NVIDIA driver 525 or later (CUDA 12.x support, required by recent vLLM images).
• Install nvidia-container-toolkit and configure the Docker daemon for GPU access. See the installation guide.
• Docker Engine 24.0+ with Compose V2.
• Create a HuggingFace account and generate a valid access token (HF_TOKEN). For gated models like Meta Llama 3, accept the license agreement at https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct using the HuggingFace account associated with your token before the container can download the model.
• All JavaScript files in this guide use ES module syntax. Add "type": "module" to your package.json or use .mjs file extensions.

A minimal package.json for the project:

{
  "name": "ollama-vllm-proxy",
  "type": "module",
  "dependencies": {
    "express": "^4.18.0"
  }
}

Ollama vs. vLLM: Understanding the Architectural Differences

How Ollama Serves Models

Ollama's default behavior is single-model-at-a-time loading with sequential request processing. When a request arrives, Ollama loads the model into GPU memory (if not already resident), runs inference, and returns the result. Subsequent requests for a different model trigger an unload/load cycle. Its REST API is custom-designed for simplicity, exposing /api/generate for raw completions and /api/chat for conversational inference. Ollama loads and unloads models based on incoming request patterns, with configurable keep-alive durations. This design works well for a single developer running models locally but creates bottlenecks the moment multiple users share an endpoint.

How vLLM Serves Models

vLLM's architecture is fundamentally different. Its continuous batching engine groups incoming requests into dynamic batches, processing tokens across multiple requests simultaneously rather than sequentially. PagedAttention, the memory management system at vLLM's core, manages GPU memory in fixed-size blocks (analogous to virtual memory pages in an operating system). This eliminates memory fragmentation and allows the engine to serve an order of magnitude more concurrent sequences than naive memory allocation permits (see the PagedAttention paper for measured comparisons). The API is OpenAI-compatible out of the box, exposing /v1/chat/completions and /v1/completions. For multi-GPU setups, tensor parallelism splits model layers across devices. The entire engine operates asynchronously, meaning request handling never blocks on a single completion.

PagedAttention, the memory management system at vLLM's core, manages GPU memory in fixed-size blocks (analogous to virtual memory pages in an operating system). This eliminates memory fragmentation and allows the engine to serve an order of magnitude more concurrent sequences than naive memory allocation permits.

Side-by-Side Comparison Table

Five Signs It's Time to Migrate

Performance Red Flags

Three signals indicate the team has outgrown Ollama's architecture. When median response time exceeds 5 seconds for simple completions under concurrent load, the sequential processing model is the bottleneck. A single user may see acceptable latency, but concurrent requests queue linearly. Requests that queue and timeout when three or more team members hit the endpoint simultaneously confirm the problem: Ollama does not batch requests, so each one waits for the previous one to finish. GPU utilization pinned at 100% while throughput stays flat is the third signal. The GPU is fully occupied processing one request at a time, with no mechanism to amortize compute across multiple concurrent sequences.

Operational Red Flags

Beyond raw performance, operational friction signals readiness for migration. If the Node.js backend has accumulated hand-rolled request queuing logic, retry wrappers, or load balancing hacks to work around Ollama's concurrency limitations, that complexity becomes technical debt; a batching engine removes it. The need to serve multiple model versions simultaneously, whether for A/B testing or phased rollouts, also exceeds Ollama's single-model-at-a-time default behavior.

Pre-Migration Setup

Infrastructure Requirements

vLLM requires a GPU with at least 16 GB VRAM for production-relevant models in the 7B-8B parameter range. Practical choices for small teams include the NVIDIA A10G (24 GB, available on AWS G5 instances), the L4 (24 GB, available on GCP), or the RTX 4090 (24 GB, for on-premise setups). A Python 3.9+ environment is needed to run vLLM itself, even though the application stack is Node.js. Docker is the recommended isolation strategy, keeping the Python dependency chain entirely separate from the application runtime.

Auditing Your Current Ollama Usage

Before migrating, catalog everything. Identify which models are running, their quantization levels, and their parameter counts. Map each Ollama model name to its HuggingFace equivalent (for example, ollama run llama3:8b corresponds to meta-llama/Meta-Llama-3-8B-Instruct). Verify with ollama show llama3 before assuming model size, as the untagged llama3 alias resolves differently across Ollama versions. Document every API endpoint the Node.js backend calls, including any custom parameters passed through options.

The following script automates the inventory step by querying Ollama's local API:

// audit-ollama.js — Run with: node audit-ollama.js
// Requires Node.js 18+ (for native fetch) and "type": "module" in package.json
const OLLAMA_BASE = process.env.OLLAMA_HOST || "http://localhost:11434";

async function auditModels() {
  const tagsRes = await fetch(`${OLLAMA_BASE}/api/tags`);
  if (!tagsRes.ok) {
    throw new Error(`Failed to list models: HTTP ${tagsRes.status}`);
  }
  const { models } = await tagsRes.json();
  if (!Array.isArray(models)) {
    throw new Error("Unexpected response shape from /api/tags");
  }

  console.log(`Found ${models.length} model(s) in Ollama:
`);

  await Promise.all(
    models.map(async (model) => {
      const showRes = await fetch(`${OLLAMA_BASE}/api/show`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ model: model.name }),
      });
      if (!showRes.ok) {
        console.warn(`[WARN] Could not fetch details for ${model.name}: HTTP ${showRes.status}`);
        return;
      }
      const details = await showRes.json();

      console.log(`Model:          ${model.name}`);
      console.log(`Size:           ${(model.size / 1e9).toFixed(2)} GB`);
      console.log(`Parameter Size: ${details.details?.parameter_size ?? "N/A"}`);
      console.log(`Quantization:   ${details.details?.quantization_level ?? "N/A"}`);
      console.log(`Family:         ${details.details?.family ?? "N/A"}`);
      console.log(`Format:         ${details.details?.format ?? "N/A"}`);
      console.log("---");
    })
  );
}

auditModels().catch(console.error);

This produces a migration checklist: each model, its size, and its quantization level. Use this output to find corresponding HuggingFace model IDs and compatible quantization formats for vLLM.

Setting Up vLLM Alongside Ollama

Installing vLLM via Docker

The key principle is coexistence. Run vLLM on a separate port (8000 by default) while Ollama continues on 11434. This allows parallel operation during the entire migration window.

Single-GPU hosts: On a machine with only one GPU, you cannot run both Ollama and vLLM simultaneously with the configuration below — they will both attempt to claim the same GPU and one or both will fail with an out-of-memory error. On a single-GPU host, stop Ollama before starting vLLM (docker compose stop ollama), or use Compose profiles to manage which service is active. The configuration below assumes a multi-GPU setup with separate devices assigned to each service.

Security note: HF_TOKEN is passed as a plain environment variable (${HF_TOKEN}). Its value will be visible in docker inspect output and the container environment. For production use, consider Docker secrets or a .env file excluded from version control (add .env to .gitignore).

# docker-compose.yml
# Pin versions to ensure reproducibility. Check release notes before upgrading.

services:
  ollama:
    image: ollama/ollama:0.3.6
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  vllm:
    image: vllm/vllm-openai:v0.6.3
    ports:
      - "8000:8000"
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --max-model-len 8192
      --gpu-memory-utilization 0.85
      --port 8000
      --dtype auto
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/v1/models"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s

volumes:
  ollama_data:

Note the start_period of 120 seconds on the health check. vLLM takes significantly longer than Ollama to start because it loads the full model weights and compiles CUDA kernels on first launch. Increase to 300 seconds for models larger than 13B or on slow storage. The --gpu-memory-utilization 0.85 flag tells vLLM to reserve 85% of available VRAM for KV cache and model weights. On a 24 GB card this leaves approximately 3.6 GB; CUDA context typically requires 0.5-1 GB of that headroom.

The --dtype auto flag selects bfloat16 on Ampere+ GPUs (A10G, A100, RTX 40xx) and float16 on older architectures. On pre-Ampere GPUs (T4, V100, RTX 30xx), replace --dtype auto with --dtype float16 to prevent a potential fallback to float32, which requires approximately 2x the VRAM and will cause OOM on 24 GB cards for 8B models.

The --max-model-len 8192 matches Llama 3 8B Instruct's native context window. Setting a lower value silently truncates long conversations. Reduce this if VRAM is constrained, understanding the tradeoff.

Verifying the vLLM Endpoint

After the container reports healthy, verify with two quick requests: curl http://localhost:8000/v1/models should return the loaded model name, and a test completion request against /v1/chat/completions with a trivial prompt confirms inference is functional.

Key Configuration Differences

Model format is the most significant divergence. Ollama uses GGUF files from its own registry, while vLLM expects HuggingFace-format weights in safetensors. For quantized models, vLLM does not support GGUF quantization levels (Q4_0, Q5_K_M, etc.) directly. Instead, use --quantization awq or --quantization gptq flags with AWQ or GPTQ-quantized HuggingFace checkpoints. For multi-GPU setups, --tensor-parallel-size 2 splits the model across two GPUs. The --max-num-seqs flag controls maximum concurrent batched requests and should be tuned based on measured concurrency.

Building the Migration Proxy in Node.js

Why a Proxy Layer Matters

A proxy between the application frontend and the inference backend serves three purposes. It enables gradual traffic shifting (canary migration) without changing any client code. It provides a single stable endpoint for the React frontend regardless of which engine is active. And it enables instant rollback by flipping a single environment variable and restarting the proxy process.

Implementing the Express Proxy

// proxy.js — Migration proxy with canary routing, shadow mode, and streaming
// Requires: npm install express
// Requires: "type": "module" in package.json OR rename to proxy.mjs
import express from "express";
import { translateOllamaToOpenAI } from "./translate.js";

const app = express();
app.use(express.json());

const OLLAMA_URL = process.env.OLLAMA_URL || "http://localhost:11434";
const VLLM_URL = process.env.VLLM_URL || "http://localhost:8000";
// Note: BACKEND and CANARY_PERCENT are read at startup.
// Changing these environment variables requires restarting the proxy process.
const BACKEND = process.env.BACKEND || "ollama"; // "ollama" | "vllm" | "canary" | "shadow"
const CANARY_PERCENT = parseInt(process.env.CANARY_PERCENT || "10", 10);
const MAX_SHADOW_INFLIGHT = 10;
let shadowInflight = 0;

function chooseBackend() {
  if (BACKEND === "shadow") {
    return "shadow";
  }
  if (BACKEND === "canary") {
    return Math.random() * 100 < CANARY_PERCENT ? "vllm" : "ollama";
  }
  return BACKEND;
}

async function forwardToOllama(body) {
  const upstream = await fetch(`${OLLAMA_URL}/api/chat`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(body),
    signal: AbortSignal.timeout(120_000),
  });
  return upstream;
}

async function forwardToVllm(body) {
  const payload = translateOllamaToOpenAI(body);
  const upstream = await fetch(`${VLLM_URL}/v1/chat/completions`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(payload),
    signal: AbortSignal.timeout(120_000),
  });
  return upstream;
}

// Shadow mode: fire-and-forget request to vLLM for latency/error logging
// Bounded by MAX_SHADOW_INFLIGHT to prevent resource exhaustion under load.
function shadowFetch(body) {
  if (shadowInflight >= MAX_SHADOW_INFLIGHT) {
    console.warn("[shadow:vllm] dropped — max in-flight reached");
    return;
  }
  shadowInflight++;
  const payload = translateOllamaToOpenAI(body);
  const start = Date.now();
  fetch(`${VLLM_URL}/v1/chat/completions`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(payload),
    signal: AbortSignal.timeout(120_000),
  })
    .then(async (res) => {
      await res.text();
      console.log(`[shadow:vllm] ${Date.now() - start}ms status=${res.status}`);
    })
    .catch((err) => {
      console.error(`[shadow:vllm] ${Date.now() - start}ms error=${err.message}`);
    })
    .finally(() => {
      shadowInflight--;
    });
}

app.post("/api/chat", async (req, res) => {
  const target = chooseBackend();
  const start = Date.now();
  let upstreamStatus;

  try {
    let upstream;
    let servedBy = "unknown";

    if (target === "shadow") {
      // Shadow mode: serve from Ollama, fire-and-forget to vLLM
      shadowFetch(req.body);
      upstream = await forwardToOllama(req.body);
      servedBy = "ollama";
    } else if (target === "vllm") {
      upstream = await forwardToVllm(req.body);
      servedBy = "vllm";
    } else {
      upstream = await forwardToOllama(req.body);
      servedBy = "ollama";
    }

    upstreamStatus = upstream.status;

    if (!upstream.body) {
      res.status(502).json({ error: "No response body from upstream" });
      return;
    }

    if (req.body.stream) {
      const contentType = servedBy === "vllm"
        ? "text/event-stream"
        : "application/x-ndjson";
      res.setHeader("Content-Type", contentType);
      res.setHeader("Cache-Control", "no-cache");
      res.setHeader("Connection", "keep-alive");
      res.setHeader("X-Inference-Backend", servedBy);

      if (!upstream.ok) {
        const errBody = await upstream.text();
        console.error(`[${servedBy}] upstream error ${upstream.status}: ${errBody}`);
        res.status(upstream.status).end(errBody);
        return;
      }

      const reader = upstream.body.getReader();
      const decoder = new TextDecoder();

      const pump = async () => {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          res.write(decoder.decode(value, { stream: true }));
        }
        res.end();
      };
      await pump();
    } else {
      if (!upstream.ok) {
        const errBody = await upstream.text();
        console.error(`[${servedBy}] upstream error ${upstream.status}: ${errBody}`);
        res.status(upstream.status).json({
          error: "Upstream inference error",
          upstream_status: upstream.status,
          detail: errBody,
        });
        return;
      }
      const data = await upstream.json();
      res.setHeader("X-Inference-Backend", servedBy);
      res.status(upstream.status).json(data);
    }
  } catch (err) {
    console.error(`[${target}] Error: ${err.message}`);
    if (!res.headersSent) {
      res.status(502).json({ error: "Inference backend unavailable" });
    }
  } finally {
    console.log(`[${target}] ${Date.now() - start}ms upstream_status=${upstreamStatus ?? "N/A"}`);
  }
});

app.listen(3001, () =>
  console.log(`Proxy running on :3001 | backend=${BACKEND}`)
);

The X-Inference-Backend response header lets monitoring systems attribute latency to the correct engine. The finally block logs per-request latency and upstream status for both backends, which feeds directly into the canary comparison process.

Important: This proxy passes through the raw stream from whichever backend is active. Ollama streams NDJSON and vLLM streams SSE — these are different wire formats. The frontend hook below handles both formats by detecting data: prefixed lines (SSE from vLLM) versus raw JSON lines (NDJSON from Ollama). If you prefer proxy-side normalization, transform both formats to a single wire format before forwarding to the client.

Adapting Your React Frontend

The React frontend targets the proxy endpoint. The following hook handles both Ollama's NDJSON streaming format and vLLM's SSE format (data: {"choices":[...]}) by detecting and parsing each line format appropriately.

// useChat.js — React hook for streaming chat via the proxy
// Handles both Ollama NDJSON and vLLM SSE stream formats
import { useState, useCallback, useRef } from "react";

const MODEL_NAME = process.env.REACT_APP_MODEL_NAME || "llama3";

export function useChat() {
  const [messages, setMessages] = useState([]);
  const [isLoading, setIsLoading] = useState(false);
  // Use ref so sendMessage does not need messages in its dependency array
  const messagesRef = useRef(messages);
  messagesRef.current = messages;

  const sendMessage = useCallback(async (userMessage) => {
    const snapshot = messagesRef.current;
    const updatedMessages = [
      ...snapshot,
      { role: "user", content: userMessage },
    ];
    setMessages(updatedMessages);
    setIsLoading(true);

    let assistantContent = "";

    try {
      const response = await fetch("/api/chat", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          model: MODEL_NAME,
          messages: updatedMessages,
          stream: true,
          options: { temperature: 0.7 },
        }),
      });

      if (!response.ok) {
        throw new Error(`Upstream error: HTTP ${response.status}`);
      }

      const reader = response.body.getReader();
      const decoder = new TextDecoder();
      let buffer = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split("
");
        // Keep the last (potentially incomplete) line in the buffer
        buffer = lines.pop() ?? "";

        for (const line of lines) {
          const trimmed = line.trim();
          if (!trimmed) continue;

          // vLLM SSE format: "data: {...}" or "data: [DONE]"
          if (trimmed.startsWith("data: ")) {
            const payload = trimmed.slice(6);
            if (payload === "[DONE]") continue;
            try {
              const parsed = JSON.parse(payload);
              const token = parsed.choices?.[0]?.delta?.content ?? "";
              assistantContent += token;
            } catch {
              // Malformed SSE chunk — skip
            }
          } else {
            // Ollama NDJSON format: raw JSON object per line
            try {
              const parsed = JSON.parse(trimmed);
              if (parsed.message?.content) {
                assistantContent += parsed.message.content;
              }
            } catch {
              // Malformed NDJSON chunk — skip
            }
          }

          setMessages([
            ...updatedMessages,
            { role: "assistant", content: assistantContent },
          ]);
        }
      }
    } catch (err) {
      console.error("Chat error:", err);
      setMessages([
        ...updatedMessages,
        { role: "assistant", content: `[Error: ${err.message}]` },
      ]);
    } finally {
      setIsLoading(false);
    }
  }, []); // No dependency on messages — uses ref instead

  return { messages, sendMessage, isLoading };
}

The model value is read from the REACT_APP_MODEL_NAME environment variable (defaulting to "llama3"). When the proxy routes to vLLM, the translateOllamaToOpenAI function maps the model name to the full HuggingFace identifier. Alternatively, configure vLLM with --served-model-name llama3 to accept the short alias directly.

Migrating Request and Response Formats

Request Payload Translation

The translation between Ollama and OpenAI-compatible formats is mechanical but has edge cases that silently break inference if missed.

// translate.js — Bidirectional payload translation
// Requires: "type": "module" in package.json OR rename to translate.mjs

const MODEL_MAP = {
  "llama3": "meta-llama/Meta-Llama-3-8B-Instruct",
};

export function translateOllamaToOpenAI(ollamaPayload) {
  const openAIPayload = {
    model: MODEL_MAP[ollamaPayload.model] ?? ollamaPayload.model,
    messages: ollamaPayload.messages || [],
    stream: ollamaPayload.stream ?? false,
    temperature: ollamaPayload.options?.temperature ?? 0.7,
    // num_predict is Ollama's max-new-tokens parameter.
    // Do NOT use num_ctx here — that is the total context window size.
    max_tokens: ollamaPayload.options?.num_predict ?? 512,
    top_p: ollamaPayload.options?.top_p ?? 1.0,
  };

  // Only include repetition_penalty when explicitly provided by the caller.
  // vLLM accepts repetition_penalty directly, which matches Ollama's
  // repeat_penalty semantics (multiplicative logit penalty, default 1.0).
  // Do NOT map to frequency_penalty — it has different mathematical behavior.
  if (ollamaPayload.options?.repeat_penalty !== undefined) {
    openAIPayload.repetition_penalty = ollamaPayload.options.repeat_penalty;
  }

  if (ollamaPayload.format === "json") {
    openAIPayload.response_format = { type: "json_object" };
  }

  if (
    ollamaPayload.system &&
    openAIPayload.messages[0]?.role !== "system"
  ) {
    openAIPayload.messages = [
      { role: "system", content: ollamaPayload.system },
      ...openAIPayload.messages,
    ];
  }

  return openAIPayload;
}

export function translateOpenAIToOllama(openAIPayload) {
  return {
    model: openAIPayload.model,
    messages: openAIPayload.messages,
    stream: openAIPayload.stream ?? false,
    options: {
      temperature: openAIPayload.temperature ?? 0.7,
      num_predict: openAIPayload.max_tokens ?? 512,
      top_p: openAIPayload.top_p ?? 1.0,
    },
  };
}

Two parameter mappings deserve attention. First, num_predict (not num_ctx) is Ollama's parameter for the maximum number of new tokens to generate. num_ctx sets the total context window (input plus output) and should not be mapped to max_tokens. Second, Ollama's repeat_penalty is a multiplicative penalty on logits (default 1.0, meaning no penalty), while OpenAI's frequency_penalty is an additive penalty on log-probabilities with entirely different behavior. vLLM supports repetition_penalty as a direct parameter that matches Ollama's semantics, so use that instead. The system field handling guards against duplicates: a system message is only prepended when the caller provides a top-level system string and no system message already exists at the start of the messages array.

Ollama's repeat_penalty is a multiplicative penalty on logits (default 1.0, meaning no penalty), while OpenAI's frequency_penalty is an additive penalty on log-probabilities with entirely different behavior. vLLM supports repetition_penalty as a direct parameter that matches Ollama's semantics, so use that instead.

Response Format Differences

Ollama streams NDJSON where each line is a complete JSON object: {"message": {"content": "..."}, "done": false}. vLLM follows the OpenAI SSE specification, prefixing each chunk with data: and terminating the stream with data: [DONE]. The internal structure differs too: vLLM nests content under choices[0].delta.content during streaming. The proxy sets the appropriate Content-Type header based on the active backend (application/x-ndjson for Ollama, text/event-stream for vLLM), and the frontend hook parses both formats by detecting data: prefixed lines.

Running the Canary Migration

Phase 1: Shadow Mode (0% Live Traffic)

Shadow mode requires a code change to the proxy. Set BACKEND=shadow to activate the shadow mode path, which always serves responses from Ollama while also firing a fire-and-forget fetch to vLLM that logs latency and errors without affecting the client response. The shadow mode implementation is included in the proxy code above. Shadow requests are bounded by a concurrency limit (MAX_SHADOW_INFLIGHT) to prevent resource exhaustion under load.

Run this for 24 to 48 hours. This catches model loading failures, out-of-memory errors, and request format mismatches without affecting any user-facing behavior. Confirm all responses carry X-Inference-Backend: ollama and that vLLM latency appears in the proxy logs.

Phase 2: 10% Traffic to vLLM

Once shadow-mode logs show stable vLLM responses with no errors, set BACKEND=canary and CANARY_PERCENT=10, then restart the proxy. Monitor error rates, p50 and p95 latency, and output quality. Output may differ between backends because of quantization format differences: Ollama's GGUF quantization and vLLM's AWQ or GPTQ quantization produce numerically different results even for the same base model. Compare outputs for regressions but expect some divergence. This is a quality check, not a byte-for-byte match.

Phase 3: 50% to 100% Cutover

Increase CANARY_PERCENT to 50 and restart the proxy, then set BACKEND=vllm and restart again for full cutover. Environment variable changes require a proxy restart to take effect. Keep the Ollama service running for 48 hours as a rollback safety net. If latency or error rates spike, reverting to BACKEND=ollama and restarting the proxy provides immediate rollback. After the bake period, decommission the Ollama service.

Post-Migration Optimization

Tuning vLLM for Your Team's Workload

Start --gpu-memory-utilization at 0.85 and increase cautiously. Values above 0.90 risk OOM errors from CUDA context overhead; monitor nvidia-smi closely and set --max-num-seqs conservatively before increasing utilization. Higher values give vLLM more room for KV cache, which directly increases maximum concurrent sequences. Set --max-num-seqs based on measured concurrency from the proxy logs. If the team rarely exceeds 10 concurrent requests, setting this to 16 provides headroom without over-allocating. For latency-sensitive endpoints where time-to-first-token matters more than throughput, speculative decoding uses a smaller draft model to propose token candidates; the main model verifies them in a single forward pass. This reduces latency only when the draft model's acceptance rate is high (aim for above 70% before enabling; per vLLM documentation, lower rates increase latency). Benchmark before enabling in production, as it can increase latency for highly variable outputs.

Monitoring in Production

Track four metrics: time-to-first-token (user-perceived responsiveness), tokens per second (throughput), GPU memory utilization (capacity headroom), and queue depth (saturation indicator). When time-to-first-token degrades, check queue depth first; rising queue depth with flat throughput means you need to increase --max-num-seqs or add GPU capacity. vLLM exposes a Prometheus-compatible /metrics endpoint natively. Pairing this with Grafana provides dashboards without additional instrumentation. The proxy's per-request latency logs serve as an application-level complement to vLLM's engine-level metrics.

When time-to-first-token degrades, check queue depth first; rising queue depth with flat throughput means you need to increase --max-num-seqs or add GPU capacity.

Complete Migration Checklist

1. ☐ Inventory current Ollama models and map to HuggingFace equivalents
2. ☐ Verify GPU meets vLLM minimum requirements (16 GB+ VRAM for 7B-8B models)
3. ☐ Install nvidia-container-toolkit and configure Docker daemon
4. ☐ Accept HuggingFace model license and generate access token
5. ☐ Deploy vLLM via Docker on separate port (8000) with pinned image version
6. ☐ Validate vLLM endpoint with curl against /v1/models
7. ☐ Deploy Node.js proxy with "type": "module" in package.json and Ollama as default backend
8. ☐ Implement request/response translation layer
9. ☐ Run shadow mode (BACKEND=shadow) for 24 to 48 hours, compare latency logs
10. ☐ Canary 10% traffic (BACKEND=canary), monitor for errors and quality regressions
11. ☐ Scale to 50%, then 100% (restart proxy after each environment variable change)
12. ☐ Remove Ollama service after 48-hour bake period
13. ☐ Tune vLLM parameters based on observed workload
14. ☐ Set up monitoring dashboards (Prometheus + Grafana)

What Changes and What Doesn't

The React frontend and application logic stay the same. The frontend hook handles both NDJSON (Ollama) and SSE (vLLM) stream formats, so switching backends does not require frontend changes. The Node.js backend gains a translation layer that can target either inference engine. The proxy pattern means the migration is reversible at every stage. The real performance shift comes from moving away from sequential request processing to continuous batching, which on typical multi-user workloads delivers 3-10x higher throughput for the same hardware (actual gains depend on batch size, model size, and request concurrency). Ollama remains an excellent tool for individual local development. This migration applies specifically to shared, staging, and production environments where concurrent access defines the workload. The vLLM documentation and Ollama documentation provide engine-specific reference for further tuning.