Ollama vs vLLM: A Migration Guide for Scaling Teams

The pattern is familiar: a team spins up Ollama on a developer workstation, builds a prototype, and ships it internally. Weeks later, adoption grows and the logs fill with timeouts — and the conversation about Ollama vs vLLM starts as an operational necessity.

Table of Contents

• Why Scaling Teams Hit the Ollama Ceiling
• When to Migrate: Identifying the Bottlenecks
• Architecture Comparison: Ollama vs vLLM Under the Hood
• Migration Steps: From Ollama to vLLM
• API Compatibility: Swapping Endpoints with Minimal Code Changes
• Benchmarking: Real-World Load Tests
• Post-Migration Checklist
• When to Keep Ollama

Why Scaling Teams Hit the Ollama Ceiling

The pattern is familiar: a team spins up Ollama on a developer workstation, builds a prototype chatbot or document-processing pipeline, and ships it internally. Weeks later, adoption grows from five users to fifty, and the logs fill with timeouts. The conversation about ollama vs vllm starts not as an academic comparison but as an operational necessity driven by llm inference scaling requirements that have outgrown the original tool.

This article is a migration guide, not a "which is better" debate. Ollama is a local LLM runner that packages model management and inference behind a single binary and a convenient CLI. vLLM is a high-throughput inference engine built around GPU-optimized serving primitives designed for production workloads. Both expose OpenAI-compatible APIs, which makes migration tractable.

When to Migrate: Identifying the Bottlenecks

Concurrency Limits and Request Queuing

Ollama's default behavior funnels inference through a single-model, sequential-request pipeline. Under the hood, it wraps llama.cpp in a Go binary that processes requests sequentially per loaded model by default. Ollama exposes parallel inference via OLLAMA_NUM_PARALLEL, but this lacks the scheduling efficiency of continuous batching. When a second request arrives while the first is still generating tokens, it enters a queue unless the operator has explicitly configured parallelism. At low concurrency this is invisible. At ten or fifteen simultaneous users, queue depth climbs and downstream services start logging connection timeouts, especially in Node.js applications that set aggressive fetch deadlines.

The symptom is straightforward: application logs show a growing gap between request initiation and first-token arrival, with no corresponding increase in token generation speed. The bottleneck is not the GPU or the model; it is the serving layer's inability to efficiently batch and schedule concurrent requests.

The bottleneck is not the GPU or the model; it is the serving layer's inability to efficiently batch and schedule concurrent requests.

Latency Spikes Under Load

A related but distinct problem emerges when teams serve multiple models from a single Ollama instance. Ollama loads and unloads GGUF model weights in response to incoming requests, caching them in memory for a configurable keep-alive period (default 5 minutes). The first request to a model returns within the model's single-request baseline latency because the weights are already resident in memory. A subsequent request targeting a different model can force an unload/reload cycle that introduces 3-to-15-second latency spikes depending on model size. Even when serving a single model, memory pressure from context allocation on consumer-grade GPUs compounds the "first request quick, tenth request slow" pattern under sustained load.

Decision Checklist: Should You Migrate Now?

Use this checklist to determine whether migration is warranted:

• Sustained more than 5 concurrent inference requests hitting the serving layer (even with OLLAMA_NUM_PARALLEL enabled)
• If your P95 latency exceeds acceptable thresholds for the application's user experience (e.g., >2 s for interactive chat, >10 s for batch pipelines), the serving layer is the likely constraint
• The deployment requires continuous batching or multi-GPU serving across multiple cards
• Deploying behind a load balancer, Kubernetes ingress, or other orchestration layer
• Guaranteed throughput SLAs are required for internal or external consumers

If fewer than two of those conditions apply, Ollama may still be the right choice, especially with its parallel inference options tuned.

Architecture Comparison: Ollama vs vLLM Under the Hood

Ollama's Monolithic Approach

Ollama ships as a single Go binary that embeds llama.cpp as its inference backend. Model configuration lives in a Modelfile, a Dockerfile-like abstraction (syntax resembles Dockerfiles but is Ollama-specific and not interchangeable) that bundles the base model reference, system prompts, sampling parameters, and optional LoRA adapter paths into one declarative file. The format is convenient and opaque: it hides quantization details, memory mapping behavior, and batching configuration behind sensible defaults.

The native quantization format is GGUF, designed for CPU inference and single-GPU consumer hardware. GGUF's strength is broad hardware compatibility. Its weakness: the GPU kernels target low-latency single-user inference rather than the high-throughput batched execution that datacenter workloads demand.

vLLM's Throughput-First Architecture

vLLM's core innovation is PagedAttention, a memory management technique that treats the key-value cache the way an operating system treats virtual memory. Instead of pre-allocating a contiguous block of GPU memory for each request's KV cache, PagedAttention allocates small, non-contiguous blocks on demand. This eliminates the memory waste that forces other engines to cap batch sizes prematurely.

On top of PagedAttention, vLLM implements continuous batching: new requests enter an active batch as soon as a slot opens, rather than waiting for the entire batch to complete. The result is dramatically higher GPU utilization under concurrent load.

For multi-GPU deployments, vLLM supports tensor parallelism, sharding model layers across cards so that models too large for a single GPU's VRAM can be served without offloading to CPU. The preferred quantization formats are AWQ and GPTQ, both built for GPU kernel execution paths and delivering better throughput per watt than GGUF on NVIDIA hardware.

Instead of pre-allocating a contiguous block of GPU memory for each request's KV cache, PagedAttention allocates small, non-contiguous blocks on demand. This eliminates the memory waste that forces other engines to cap batch sizes prematurely.

Side-by-Side Architecture Table

Migration Steps: From Ollama to vLLM

Step 1: Audit Your Current Ollama Setup

Before touching the new stack, document everything the current Ollama deployment depends on: which models are served, what Modelfile customizations exist (system prompts, temperature overrides, stop tokens, adapter layers), and which API endpoints the application code calls.

A typical Ollama Modelfile looks like this:

FROM llama3:8b-q4_K_M

SYSTEM """You are a helpful customer support assistant for Acme Corp. 
Answer questions about orders, returns, and product specifications. 
Never discuss competitor products."""

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER stop "<|eot_id|>"
PARAMETER stop "</s>"
PARAMETER num_ctx 4096

ADAPTER ./fine-tuned-support-lora.gguf

This file encodes five categories of configuration: the base model and its quantization level, a system prompt, sampling parameters, stop sequences, and a LoRA adapter. Each must be accounted for during migration.

Step 2: Select the Equivalent Model and Quantization Format

Why not just keep GGUF? While vLLM ≥0.4.x supports --quantization gguf experimentally, AWQ and GPTQ remain the production-recommended formats for optimal GPU throughput. Map each model to an equivalent Hugging Face checkpoint in AWQ or GPTQ format. The vLLM team recommends AWQ as the default quantization because it delivers perplexity within 0.1-0.3 points of GPTQ on standard benchmarks (consult the AutoAWQ benchmark repository for model-specific numbers), owing to its activation-aware weight quantization approach that preserves salient weights.

Common mappings:

If no community AWQ checkpoint exists, teams can quantize from FP16 using AutoAWQ, though this adds a preparation step.

Step 3: Set Up vLLM with Docker Compose

Prerequisites: Docker Engine ≥20.10, Docker Compose v2, and NVIDIA Container Toolkit installed on the host (sudo apt install nvidia-container-toolkit). Verify GPU access with:

docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

If this command fails with "could not select device driver," the NVIDIA Container Toolkit is not installed or configured correctly. GPU passthrough in Docker is supported on Linux; macOS and Windows users require WSL2.

The following docker-compose.yml deploys vLLM with an AWQ model, GPU passthrough, and production-relevant configuration flags:

services:
  vllm:
    image: vllm/vllm-openai:v0.6.6  # Pin to current stable; see https://github.com/vllm-project/vllm/releases
    ports:
      - "8000:8000"
    volumes:
      - huggingface-cache:/root/.cache/huggingface
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}  # For production, use Docker secrets or a secrets manager rather than environment variables
      - GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.90}
    # device_requests is the correct Compose v2 standalone key for GPU access.
    # deploy.resources.reservations.devices is Swarm-only and is silently ignored by plain docker compose.
    device_requests:
      - driver: nvidia
        count: all
        capabilities: [gpu]
    command: >
      --model casperhansen/llama-3-8b-instruct-awq
      --quantization awq
      --tensor-parallel-size 1
      --max-model-len 4096
      --gpu-memory-utilization ${GPU_MEM_UTIL:-0.90}
      --port 8000

volumes:
  huggingface-cache:

Key flags explained: --quantization awq tells the engine to expect AWQ-format weights. --tensor-parallel-size controls how many GPUs to shard across (set to 2 or more for multi-GPU nodes). --max-model-len caps the maximum sequence length to control memory allocation. --gpu-memory-utilization 0.90 reserves 90% of VRAM for the model and KV cache, leaving headroom for the CUDA runtime. On shared GPU nodes where other processes also require VRAM, reduce this to 0.75-0.80 and monitor with nvidia-smi.

Note that for gated models (e.g., Llama 3), the HF_TOKEN environment variable must contain a valid Hugging Face token with accepted model access. Model downloads happen at container startup and can take several minutes for large checkpoints (4-8 GB for AWQ models); the container may appear unresponsive during this time.

Step 4: Replicate Modelfile Customizations in vLLM

vLLM has no direct Modelfile equivalent, but server-level flags (--chat-template, --served-model-name) can handle chat format and model aliasing. System prompts and sampling defaults must move to the application layer, embedded in the request body sent to vLLM's OpenAI-compatible API.

const response = await fetch("http://localhost:8000/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "casperhansen/llama-3-8b-instruct-awq",
    messages: [
      {
        role: "system",
        content:
          "You are a helpful customer support assistant for Acme Corp. Answer questions about orders, returns, and product specifications. Never discuss competitor products.",
      },
      {
        role: "user",
        content: "What is the return policy for electronics?",
      },
    ],
    temperature: 0.7,
    top_p: 0.9,
    stop: ["<|eot_id|>", "</s>"],
    max_tokens: 1024,
  }),
});

const data = await response.json();

if (!response.ok) {
  throw new Error(`vLLM error ${response.status}: ${JSON.stringify(data)}`);
}

const content = data.choices?.[0]?.message?.content;
if (content == null) {
  throw new Error("Unexpected response shape: " + JSON.stringify(data));
}

console.log(content);

This pattern preserves the exact behavior encoded in the original Modelfile. For LoRA adapters, vLLM supports runtime loading via the --enable-lora and --lora-modules flags, though the adapter must be in Hugging Face PEFT format (.safetensors + adapter_config.json) rather than GGUF. To convert a GGUF LoRA to PEFT format, the original fine-tuning must be reproduced using a PEFT-compatible trainer (e.g., trl, axolotl). There is no lossless GGUF→PEFT conversion utility. Teams with GGUF-only adapters should retain Ollama for that model or re-fine-tune.

Step 5: Validate Outputs Before Switching Traffic

Send an identical set of representative prompts to both Ollama and the new vLLM instance. Compare outputs manually or with an automated similarity check. Different quantization formats (GGUF Q4_K_M vs. AWQ 4-bit) produce different token sequences at the token level while preserving semantic equivalence. Expect this; it does not indicate misconfiguration. The goal is to verify that the model follows instructions correctly, respects stop tokens, and generates coherent responses within the same quality band.

API Compatibility: Swapping Endpoints with Minimal Code Changes

The OpenAI-Compatible API Surface

Both Ollama and vLLM expose /v1/chat/completions and /v1/completions endpoints that conform to the OpenAI API schema. For Node.js applications that use the official OpenAI SDK or raw fetch calls targeting these endpoints, the vllm migration requires changing a single configuration value: the base URL.

// Before: pointing at Ollama
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama", // Ollama ignores this but the SDK requires it
});

// After: pointing at vLLM
const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: process.env.VLLM_API_KEY ?? "unused", // Set VLLM_API_KEY when --api-key is configured on the server
});

Every subsequent client.chat.completions.create() call works unchanged.

Security note: vLLM does not enable authentication by default. Before exposing the vLLM endpoint beyond localhost, add --api-key <secret> to the server command to require a bearer token, or place the service behind an authenticated reverse proxy (e.g., nginx). An unauthenticated inference endpoint exposed to a network is a significant security risk. When --api-key is configured on the server, set the VLLM_API_KEY environment variable in the client to match; otherwise requests will fail with 401 Unauthorized.

Edge Cases and API Gaps

Ollama exposes several proprietary endpoints that have no vLLM equivalent: /api/generate (a non-OpenAI completion interface), /api/tags (listing locally available models), and /api/pull (downloading models from the Ollama registry). Refactor any application code that calls these endpoints. Model downloads in the vLLM workflow happen via the Hugging Face CLI or by baking weights into the Docker image at build time. Install the CLI via pip install huggingface_hub, then authenticate with huggingface-cli login.

Both tools stream via Server-Sent Events (SSE), but their token-level chunking granularity differs. Validate that your SSE parser handles both formats during the validation step.

Benchmarking: Real-World Load Tests

Test Setup and Methodology

To quantify the throughput difference, a load test can be executed on a single NVIDIA A10G (24 GB VRAM) running Llama 3 8B in both configurations: GGUF Q4_K_M under Ollama and AWQ under vLLM.

Requirements: Node.js ≥18 for built-in fetch and AbortSignal.timeout support. Save the script below as bench.mjs and run with node bench.mjs. On Node 16, fetch and AbortSignal.timeout are not available; install node-fetch and use a manual AbortController with setTimeout instead.

The following script fires N concurrent chat completion requests and measures aggregate throughput and latency percentiles. Note that all N requests are dispatched simultaneously via Promise.all, which measures peak burst behavior rather than sustained steady-state load. For sustained load testing, use a dedicated tool such as k6 or vegeta with a controlled request rate.

const BASE_URL = process.env.BASE_URL || "http://localhost:8000/v1";
const MODEL = process.env.MODEL || "casperhansen/llama-3-8b-instruct-awq";
const CONCURRENCY = parseInt(process.env.CONCURRENCY || "10", 10);

if (!Number.isFinite(CONCURRENCY) || CONCURRENCY < 1) {
  console.error(
    `Invalid CONCURRENCY value: "${process.env.CONCURRENCY}". Must be a positive integer.`
  );
  process.exit(1);
}

// Percentile helper — standard nearest-rank method (1-indexed)
function percentile(sortedArr, p) {
  if (sortedArr.length === 0) return 0;
  const index = Math.ceil((p / 100) * sortedArr.length) - 1;
  return sortedArr[Math.max(0, index)];
}

async function sendRequest(id) {
  const start = performance.now();

  let res;
  try {
    res = await fetch(`${BASE_URL}/chat/completions`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        model: MODEL,
        messages: [
          {
            role: "user",
            content: "Explain the concept of database indexing in two paragraphs.",
          },
        ],
        max_tokens: 256,
        temperature: 0.7,
      }),
      signal: AbortSignal.timeout(120_000), // 2-minute hard timeout per request
    });
  } catch (err) {
    console.error(`Request ${id} failed (network/timeout): ${err.message}`);
    return { id, elapsed: performance.now() - start, tokens: 0, error: true };
  }

  const elapsed = performance.now() - start;

  if (!res.ok) {
    const body = await res.text().catch(() => "(unreadable body)");
    console.error(`Request ${id} HTTP ${res.status}: ${body}`);
    return { id, elapsed, tokens: 0, error: true };
  }

  const data = await res.json();
  const tokens = data.usage?.completion_tokens ?? 0;
  return { id, elapsed, tokens, error: false };
}

async function runBenchmark() {
  console.log(`Running ${CONCURRENCY} concurrent requests against ${BASE_URL}`);
  const benchStart = performance.now();
  const promises = Array.from({ length: CONCURRENCY }, (_, i) => sendRequest(i));
  const results = await Promise.all(promises);
  const wallTime = performance.now() - benchStart;

  const errors = results.filter((r) => r.error).length;
  if (errors > 0) {
    console.warn(`${errors}/${results.length} requests failed — throughput figures are incomplete.`);
  }

  const latencies = results.map((r) => r.elapsed).sort((a, b) => a - b);
  const totalTokens = results.reduce((sum, r) => sum + r.tokens, 0);

  console.log(`Throughput: ${(totalTokens / (wallTime / 1000)).toFixed(1)} tokens/sec`);
  console.log(`P50 latency: ${percentile(latencies, 50).toFixed(0)} ms`);
  console.log(`P95 latency: ${percentile(latencies, 95).toFixed(0)} ms`);
}

runBenchmark();

Run with BASE_URL=http://localhost:11434/v1 MODEL=llama3:8b-q4_K_M CONCURRENCY=10 node bench.mjs for Ollama, then swap the environment variables for vLLM.

Results and Analysis

The following figures are approximate estimates based on the described hardware configuration (single NVIDIA A10G, 24 GB VRAM). Actual results will vary with vLLM version, CUDA/driver versions, prompt length, and system configuration. Teams should run the benchmark script above on their own infrastructure to obtain authoritative numbers.

Approximate results with Llama 3 8B (256 max output tokens per request):

~ denotes approximate values. See methodology note above for hardware and software assumptions.

At a concurrency of 1, the two engines perform within approximately 10% of each other. The divergence becomes stark at 10 concurrent requests: vLLM's continuous batching delivers roughly 6x the throughput while holding P95 latency to under 11 seconds, compared to Ollama's sequential queue pushing P95 past a minute (with default OLLAMA_NUM_PARALLEL=1). At 50 concurrent requests, Ollama begins timing out entirely on default configurations, while vLLM continues to scale. Hardware, model size, prompt length, and Ollama/vLLM version shift these numbers, but the scaling curve is consistent.

The divergence becomes stark at 10 concurrent requests: vLLM's continuous batching delivers roughly 6x the throughput while holding P95 latency to under 11 seconds, compared to Ollama's sequential queue pushing P95 past a minute.

Post-Migration Checklist

Preparation

1. Audit existing Ollama Modelfiles and document all customizations (system prompts, parameters, adapters).
2. Map each GGUF model to an AWQ or GPTQ equivalent on Hugging Face.
3. Install NVIDIA Container Toolkit and verify GPU passthrough with docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi.

Deployment

4. Deploy vLLM via Docker Compose with a pinned image version and NVIDIA GPU passthrough.
5. Move system prompts and sampling parameters to the application request layer; configure chat templates and model aliases via vLLM server flags where applicable.
6. Update baseURL in OpenAI SDK or fetch calls to point at the vLLM endpoint.
7. Add authentication to the vLLM endpoint (--api-key or reverse proxy) before exposing beyond localhost.
8. Refactor any code calling Ollama-specific APIs (/api/pull, /api/tags, /api/generate).

Validation

9. Run side-by-side output validation with representative prompts.
10. Execute load tests at target concurrency levels and record throughput and P95 latency.

Cleanup

11. Monitor P95 latency and throughput continuously in production after cutover.
12. Remove Ollama from production infrastructure once confidence is established.

When to Keep Ollama

Ollama remains an excellent tool for local development and rapid prototyping. It delivers zero-config convenience that no production engine matches, and features like OLLAMA_NUM_PARALLEL extend its useful range. But when production workloads hit concurrency walls that parallel request handling cannot resolve, vLLM provides the throughput-first architecture that continuous batching and PagedAttention were designed to deliver.

The migration is incremental. Run both engines in parallel, route a percentage of traffic to vLLM, and validate output quality and latency under real conditions. Nothing in this process requires rewriting application logic beyond the endpoint swap and the relocation of Modelfile parameters.