Quantization Explained: Q4_K_M vs AWQ vs FP16 for Local LLMs

Q4_K_M vs AWQ vs FP16 Comparison

Running large language models locally means confronting a hard constraint: GPU memory. Quantization reduces the bit-width of model weights, and that reduction is the primary technique that makes local LLM inference feasible on consumer hardware. Understanding the trade-offs between formats like Q4_K_M, AWQ, and FP16 determines whether a given model runs at all, and whether the output quality justifies the compression.

Table of Contents

• What Is Model Quantization and Why Does It Matter for Local LLMs?
• Understanding Precision Formats: FP16, INT8, INT4
• Q4_K_M Explained: GGUF and the llama.cpp Ecosystem
• AWQ Explained: Activation-Aware Weight Quantization
• FP16: The Uncompressed Baseline
• Head-to-Head Comparison: Q4_K_M vs AWQ vs FP16
• Choosing the Right Format for Your Hardware
• Practical Implementation: Loading Each Format in a Node.js Stack
• Implementation Checklist
• Common Pitfalls and Troubleshooting
• Key Takeaways

What Is Model Quantization and Why Does It Matter for Local LLMs?

The Memory Problem with Full-Precision Models

The memory arithmetic is straightforward. Each parameter in a model stored at FP16 (half-precision floating point) occupies 2 bytes. A 7-billion-parameter model at FP16 therefore requires approximately 14GB of VRAM just for the weights. A 13B model needs roughly 26GB, and a 70B model demands around 140GB, pushing well past any single consumer GPU. That 14GB figure for a 7B model already exceeds what most consumer GPUs offer. An RTX 4070 with 12GB of VRAM cannot even load the weights, let alone allocate space for the KV cache, activations, and operating system overhead. CPU-only setups face analogous pressure on system RAM, compounded by dramatically slower inference speeds.

How Quantization Solves It

Quantization reduces the number of bits used to represent each weight, compressing the model's memory footprint while attempting to preserve output quality. Think of audio bit-depth reduction: fewer bits per sample lowers fidelity, but the differences are often indistinguishable in blind comparisons for conversational tasks. The core trade-off forms a triangle between quality, speed, and memory. Aggressive quantization shrinks memory requirements and can improve throughput (fewer bytes to move through memory buses), but risks degrading the model's ability to reason, follow instructions, or produce coherent output. The key question is always how much quality survives the compression.

The core trade-off forms a triangle between quality, speed, and memory. Aggressive quantization shrinks memory requirements and can improve throughput (fewer bytes to move through memory buses), but risks degrading the model's ability to reason, follow instructions, or produce coherent output.

Understanding Precision Formats: FP16, INT8, INT4

FP16 (Half-Precision Floating Point)

FP16 uses 16 bits per weight and serves as the practical baseline for model quality. It preserves the full fidelity of a model's trained weights (most models are trained in BF16 or FP32; post-2022 models are predominantly distributed in BF16, which shares the 16-bit width but offers a wider dynamic range than FP16). The VRAM formula is simple: parameters × 2 bytes. FP16 makes sense when maximum output quality is non-negotiable and the available hardware can accommodate it, typically systems with 24GB or more of VRAM such as the NVIDIA RTX 3090, RTX 4090, or data-center GPUs like the A100. The 24GB recommendation accounts for ~14GB of weights plus ~2-4GB for KV cache at 2048 context length and additional runtime overhead.

INT8 and INT4: The Quantized Tiers

Integer quantization maps the continuous floating-point range of each weight into a smaller set of discrete integer values. INT8 uses 8 bits per weight (1 byte), cutting memory roughly in half compared to FP16. INT4 uses 4 bits (half a byte), achieving approximately a 4x reduction. The mapping process matters enormously. Per-channel quantization applies a single scale factor to an entire output channel of a weight matrix, which can distort outlier values. Per-group quantization divides weights into smaller groups (commonly 128 elements) and computes separate scale factors for each, preserving more of the original distribution. This granularity distinction is why INT4 does not mean half the quality of INT8. A well-implemented 4-bit scheme with per-group scaling can retain nearly all of the model's capability, while a naive per-channel INT4 approach would produce noticeably degraded outputs.

Q4_K_M Explained: GGUF and the llama.cpp Ecosystem

What the Name Means (Decoding Q4_K_M)

The naming convention packs several pieces of information. Q4 indicates 4-bit quantization. K refers to the k-quant method, an importance-aware mixed-precision quantization scheme developed within the llama.cpp project. M denotes the medium size variant, positioned between S (small, more aggressive compression) and L (large, less aggressive). The llama.cpp ecosystem distributes these models as GGUF files, the successor to the older GGML format, designed specifically for the llama.cpp inference engine and its ecosystem of tools.

How K-Quants Preserve Quality

K-quant quantization is not uniform 4-bit. It assigns different precision levels to different layers based on their sensitivity. Specifically, Q4_K_M quantizes attention output (wv, wo) tensors at Q6_K and remaining tensors at Q4_K. This mixed-precision strategy is what makes Q4_K_M the widely recommended "sweet spot" in the llama.cpp community. On perplexity benchmarks, Q4_K_M retains 97 to 99 percent of FP16 quality. These figures are illustrative estimates based on community benchmarks for 7B-class instruction-tuned models on Wikitext-2 perplexity. Results vary by model family, quantization tooling version, and evaluation dataset. See the llama.cpp quantization comparison wiki for model-specific measurements. For most conversational and instructional tasks, this range produces no perceptible difference in output.

Best Use Cases for Q4_K_M

Q4_K_M excels in CPU inference and CPU-plus-GPU hybrid configurations where partial layer offloading splits the model between system RAM and VRAM. It is the natural choice for systems with 8 to 16GB of RAM or VRAM, and the format is natively supported by llama.cpp, Ollama, and LM Studio, making setup minimal.

// load-gguf.js — Load a Q4_K_M GGUF model with node-llama-cpp
import { getLlama, LlamaChatSession } from "node-llama-cpp";

// Download from: https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF
// Verify exact filename on the model card before setting MODEL_PATH
const MODEL_PATH = "./models/mistral-7b-instruct-v0.2.Q4_K_M.gguf";

async function runGGUFInference() {
  const llama = await getLlama();
  const model = await llama.loadModel({ modelPath: MODEL_PATH });
  const context = await model.createContext({ contextSize: 2048 });

  try {
    const session = new LlamaChatSession({ contextSequence: context.getSequence() });

    const startTime = performance.now();
    const response = await session.prompt("Explain quantization in one paragraph.");
    const elapsed = performance.now() - startTime;

    // Word count underestimates true token count by ~20–40%.
    // For accurate measurement, use the usage stats returned by node-llama-cpp's session API.
    // This figure is an approximation for rough throughput estimation only.
    const wordCount = response.split(/\s+/).length;

    console.log("Response:", response);
    console.log(`Time: ${(elapsed / 1000).toFixed(2)}s`);
    console.log(`Approx words/sec (not tokens): ${(wordCount / (elapsed / 1000)).toFixed(1)}`);

    // VRAM metrics: use (await getLlama()).getVramState() — model.gpuVramUsage is not a documented v3 property
    console.log(`Process RSS (RAM, not GPU VRAM): ${(process.memoryUsage().rss / 1024 / 1024).toFixed(0)} MB`);
  } finally {
    await context.dispose();
  }
}

runGGUFInference().catch(console.error);

AWQ Explained: Activation-Aware Weight Quantization

How AWQ Differs from Naive Quantization

AWQ, introduced by researchers at MIT, starts from a key insight: some weight channels affect activations far more than others. A small fraction of these channels disproportionately shape the model's output, and preserving them matters far more than distributing precision uniformly. AWQ identifies these salient channels by analyzing activation distributions from a small calibration dataset, then applies per-channel scaling factors before quantization. This pre-scaling step ensures that the most critical pathways through the network retain higher effective precision even when stored at 4 bits.

AWQ vs. GPTQ: A Quick Distinction

GPTQ performs layer-by-layer post-training quantization, using calibration data to minimize reconstruction error for each layer independently. AWQ scales activations differently: it optimizes for the weights that matter most to downstream activations rather than minimizing per-layer error. In practice, AWQ generally achieves faster inference than naive GPTQ; however, GPTQ with Marlin kernels (vLLM ≥0.4.0) closes much of this gap. Benchmark both formats for your specific deployment. Both formats target GPU-based inference through frameworks like vLLM, Hugging Face Transformers, and Text Generation Inference (TGI).

Best Use Cases for AWQ

AWQ is designed for dedicated NVIDIA GPU inference with 8GB or more of VRAM (for 7B models at batch size 1; larger models require proportionally more). It fits serving and API scenarios that target sub-200ms first-token latency or sustained throughput above 40 tok/s, and it integrates naturally into Python-based ML stacks. The CUDA-optimized inference kernels that make AWQ performant are specific to NVIDIA hardware.

// call-awq.js — Query a local AWQ model served via vLLM from Node.js
// Requires Node.js >=18 (global fetch)
const VLLM_ENDPOINT = process.env.VLLM_ENDPOINT ?? "http://localhost:8000/v1/completions";
const REQUEST_TIMEOUT_MS = 30_000;

async function runAWQInference() {
  const prompt = "Explain quantization in one paragraph.";
  const startTime = performance.now();

  const response = await fetch(VLLM_ENDPOINT, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
    body: JSON.stringify({
      model: "TheBloke/Mistral-7B-Instruct-v0.2-AWQ",
      prompt,
      max_tokens: 200,
      temperature: 0.7,
    }),
  });

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`vLLM request failed: ${response.status} ${response.statusText} — ${body}`);
  }

  const data = await response.json();

  if (!data.choices?.[0]?.text) {
    throw new Error(`Unexpected vLLM response shape: ${JSON.stringify(data)}`);
  }

  const elapsed = performance.now() - startTime;
  const text = data.choices[0].text;
  const tokensUsed = data.usage?.completion_tokens ?? null;

  console.log("Response:", text);
  console.log(`Latency: ${(elapsed / 1000).toFixed(2)}s`);

  if (tokensUsed !== null) {
    console.log(`Tokens generated: ${tokensUsed}`);
    console.log(`Tokens/sec: ${(tokensUsed / (elapsed / 1000)).toFixed(1)}`);
  }
}

runAWQInference().catch(console.error);

The vLLM server would be started separately:

vllm serve TheBloke/Mistral-7B-Instruct-v0.2-AWQ --quantization awq --host 127.0.0.1

For vLLM <0.4.0, use the legacy entrypoint: python -m vllm.entrypoints.openai.api_server --model TheBloke/Mistral-7B-Instruct-v0.2-AWQ --quantization awq.

Warning: vLLM's server may bind to 0.0.0.0 by default, making it accessible across the network. To restrict to localhost only, add --host 127.0.0.1. For production deployments, add --api-key <secret> and place behind a reverse proxy.

FP16: The Uncompressed Baseline

When Full Precision Is Worth the Cost

FP16 is the reference point against which all quantized formats are measured. It is the appropriate choice for research and fine-tuning workflows where weight fidelity directly affects training outcomes, and for quality-critical production deployments where even marginal degradation is unacceptable. FP16 demands 24GB+ VRAM for a 7B model (~14GB for weights plus ~2-4GB for KV cache at 2048 context and runtime overhead), ruling out every consumer GPU below the RTX 3090. Beyond that tier, data-center accelerators like the A100 (40GB or 80GB variants) provide the headroom for 13B+ models.

VRAM Requirements by Model Size (FP16)

These figures cover weights only. System RAM overhead for the inference runtime, KV cache at longer context lengths, and OS-level VRAM reservation add meaningfully to the total.

Head-to-Head Comparison: Q4_K_M vs AWQ vs FP16

Quality (Perplexity and Human-Eval Benchmarks)

Using FP16 as the 100% baseline, AWQ at 4-bit retains 98 to 99 percent of quality on perplexity benchmarks, and Q4_K_M retains 97 to 99 percent. These figures are illustrative estimates based on community benchmarks for 7B-class instruction-tuned models on Wikitext-2 perplexity. Results vary by model family, quantization tooling version, and evaluation dataset. The overlap in those ranges is significant and intentional: the gap between the two quantized formats is narrow and task-dependent. Coding and mathematical reasoning tasks show the most sensitivity to quantization. Creative writing and summarization tasks show the least. Factual question answering falls in between.

Speed (Tokens per Second)

At batch size 1 (single-user inference), FP16 on a GPU is memory-bandwidth-bound, meaning the GPU's compute units are often underutilized while waiting for weight data. AWQ on GPU achieves the highest 4-bit inference throughput due to purpose-built CUDA kernels that exploit the reduced data size. Q4_K_M on CPU typically produces 5-15 tok/s on a modern 8-core processor; when offloaded to GPU via llama.cpp, it reaches within roughly 10-20% of AWQ throughput in community benchmarks, though it trails AWQ due to less specialized kernel optimization.

Memory Footprint (7B Model Example)

For a 7B-parameter model, FP16 requires approximately 14GB of VRAM. AWQ at 4-bit compresses this to roughly 3.5-4.2GB of VRAM (raw 4-bit weights are ~3.5GB; scale/zero-point metadata and non-quantized layers add ~0.3-0.7GB depending on architecture). Q4_K_M similarly requires approximately 4-4.5GB, but with the flexibility to place that footprint in system RAM, VRAM, or a mixture of both through partial offloading.

*Quality retention figures are illustrative estimates for 7B-class models on Wikitext-2 perplexity; actual results vary by model and evaluation dataset.

Coding and mathematical reasoning tasks show the most sensitivity to quantization. Creative writing and summarization tasks show the least. Factual question answering falls in between.

Choosing the Right Format for Your Hardware

Decision Tree by Hardware Tier

CPU Only (16GB+ System RAM)

Q4_K_M via Ollama or llama.cpp is the only practical option. AWQ and FP16 inference on CPU is technically possible but produces 1-5 tok/s for 7B models, meaning a 100-token reply takes 20-100 seconds. That rules out interactive chat entirely.

Consumer GPU (8 to 12GB VRAM)

Both Q4_K_M with GPU offloading and AWQ fit within this VRAM budget for 7B models. The tiebreaker: AWQ is preferable when staying within the Python/CUDA GPU ecosystem and prioritizing throughput. Q4_K_M offers more flexibility across platforms and runtimes.

For 13B+ models, start here: High-End GPU (24GB+ VRAM)

FP16 is viable for 7B models with ample headroom. For 13B or larger models, AWQ remains the efficient choice. Q4_K_M still has a role here: running 70B-class models on a single 24GB card, where even AWQ's 4-bit footprint for 70B (~35GB) exceeds the VRAM budget, but Q4_K_M's partial offloading can split the load across VRAM and system RAM.

Apple Silicon (M1/M2/M3 with Unified Memory)

CUDA kernels do not exist on Apple hardware, so AWQ's primary advantage disappears. Q4_K_M via llama.cpp or MLX is the recommended path, giving full access to unified memory without the kernel mismatch penalty.

Practical Implementation: Loading Each Format in a Node.js Stack

Prerequisites

You need Node.js >=18.0.0 (for ESM import, global performance.now(), and fetch) and npm >=8.0.0. The AWQ serving path requires Python >=3.9 and the CUDA toolkit. For hardware, AWQ needs an NVIDIA GPU with >=8GB VRAM; Q4_K_M runs on any system with >=8GB RAM. Download the GGUF model file to ./models/ and confirm the exact filename matches MODEL_PATH. The React dashboard component requires a build environment (e.g., Vite: npm create vite@latest) since .jsx files cannot be executed directly with Node.

Project Setup and Dependencies

{
  "name": "llm-quantization-comparison",
  "type": "module",
  "version": "1.0.0",
  "engines": {
    "node": ">=18.0.0"
  },
  "scripts": {
    "gguf": "node load-gguf.js",
    "awq": "node call-awq.js",
    "server": "node server.js"
  },
  "dependencies": {
    "node-llama-cpp": "3.1.1",
    "express": "4.18.0",
    "cors": "2.8.5"
  }
}

Install with npm install. The node-llama-cpp version is pinned to 3.1.1; API surface may differ on other 3.x versions. Express and cors are also pinned to exact versions for reproducibility in this hardware-sensitive inference stack. The GGUF model file should be downloaded from Hugging Face and placed in a ./models/ directory. AWQ and FP16 models are served externally via vLLM or a comparable Python inference server.

Unified Inference Wrapper

// inference.js — Unified wrapper across GGUF and API-served models
import { getLlama, LlamaChatSession } from "node-llama-cpp";

// Download from: https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF
// Verify exact filename on the model card before setting GGUF_MODEL_PATH
const GGUF_MODEL_PATH = process.env.GGUF_MODEL_PATH
  ?? "./models/mistral-7b-instruct-v0.2.Q4_K_M.gguf";
const API_BASE = process.env.VLLM_ENDPOINT ?? "http://localhost:8000/v1/completions";
const REQUEST_TIMEOUT_MS = 30_000;

const VALID_MODEL_TYPES = new Set(["gguf", "awq", "fp16"]);

// Singleton promise lock — prevents concurrent double-init
let ggufInitPromise = null;
let llamaInstance = null;
let ggufModel = null;

async function initGGUF() {
  if (ggufModel) return ggufModel;
  if (!ggufInitPromise) {
    ggufInitPromise = (async () => {
      llamaInstance = await getLlama();
      ggufModel = await llamaInstance.loadModel({ modelPath: GGUF_MODEL_PATH });
    })();
  }
  await ggufInitPromise;
  return ggufModel;
}

async function generateResponse(prompt, modelType = "gguf") {
  if (!VALID_MODEL_TYPES.has(modelType)) {
    throw new Error(
      `Invalid modelType "${modelType}". Must be one of: ${[...VALID_MODEL_TYPES].join(", ")}`
    );
  }
  if (!prompt || typeof prompt !== "string") {
    throw new Error("prompt must be a non-empty string");
  }

  const start = performance.now();

  if (modelType === "gguf") {
    const model = await initGGUF();
    const context = await model.createContext({ contextSize: 2048 });

    try {
      const session = new LlamaChatSession({
        contextSequence: context.getSequence(),
      });
      const text = await session.prompt(prompt);
      const elapsed = performance.now() - start;

      // VRAM metrics: use llamaInstance.getVramState() — model.gpuVramUsage is not a documented v3 property
      return {
        text,
        latencyMs: elapsed,
        modelType: "Q4_K_M (GGUF)",
      };
    } finally {
      await context.dispose();
    }
  }

  // AWQ or FP16 via vLLM-compatible API
  const modelName =
    modelType === "awq"
      ? "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"
      : "mistralai/Mistral-7B-Instruct-v0.2";

  const res = await fetch(API_BASE, {
    method: "POST",
    signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: modelName,
      prompt,
      max_tokens: 200,
      temperature: 0.7,
    }),
  });

  if (!res.ok) {
    const body = await res.text();
    throw new Error(
      `API request failed for modelType "${modelType}": ${res.status} ${res.statusText} — ${body}`
    );
  }

  const data = await res.json();

  if (!data.choices?.[0]?.text) {
    throw new Error(
      `Unexpected API response shape for modelType "${modelType}": ${JSON.stringify(data)}`
    );
  }

  const elapsed = performance.now() - start;

  return {
    text: data.choices[0].text,
    latencyMs: elapsed,
    modelType: modelType === "awq" ? "AWQ (4-bit)" : "FP16",
    tokens: data.usage?.completion_tokens ?? null,
  };
}

export { generateResponse };

React Comparison Dashboard Component

This component requires a React build environment (e.g., Vite: npm create vite@latest) and cannot be executed directly with Node.

// ComparisonDashboard.jsx — Side-by-side quantization comparison
import { useState } from "react";

const MODEL_CONFIG = {
  gguf: { label: "Q4_K_M (GGUF)" },
  awq:  { label: "AWQ (4-bit)"   },
  fp16: { label: "FP16"          },
};

const MODEL_TYPES = Object.keys(MODEL_CONFIG);
const API_URL = import.meta.env.VITE_API_URL
  ? `${import.meta.env.VITE_API_URL}/api/generate`
  : "http://localhost:3001/api/generate";

// Status: "idle" | "loading" | "done"
export default function ComparisonDashboard() {
  const [prompt, setPrompt] = useState(
    "Explain the difference between compiled and interpreted languages."
  );
  const [results, setResults]   = useState({});
  const [status,  setStatus]    = useState("idle");

  const runComparison = async () => {
    setStatus("loading");
    setResults({});

    const settled = await Promise.allSettled(
      MODEL_TYPES.map(async (modelType) => {
        const res = await fetch(API_URL, {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          signal: AbortSignal.timeout(60_000),
          body: JSON.stringify({ prompt, modelType }),
        });

        if (!res.ok) {
          const body = await res.text();
          throw new Error(`${res.status} ${res.statusText}: ${body}`);
        }

        return { modelType, data: await res.json() };
      })
    );

    const newResults = {};

    for (let i = 0; i < settled.length; i++) {
      const outcome = settled[i];
      if (outcome.status === "fulfilled") {
        newResults[outcome.value.modelType] = outcome.value.data;
      } else {
        newResults[MODEL_TYPES[i]] = { error: outcome.reason.message };
      }
    }

    setResults(newResults);
    setStatus("done");
  };

  return (
    <div style={{ maxWidth: 1200, margin: "0 auto", padding: 24 }}>
      <h1>LLM Quantization Comparison</h1>

      <textarea
        rows={3}
        style={{ width: "100%", fontSize: 16, padding: 8 }}
        value={prompt}
        onChange={(e) => setPrompt(e.target.value)}
      />

      <button
        onClick={runComparison}
        disabled={status === "loading"}
        style={{ marginTop: 12, padding: "8px 24px" }}
      >
        {status === "loading" ? "Running..." : "Compare All Formats"}
      </button>

      <div
        style={{
          display: "grid",
          gridTemplateColumns: "repeat(3, 1fr)",
          gap: 16,
          marginTop: 24,
        }}
      >
        {MODEL_TYPES.map((type) => (
          <div
            key={type}
            style={{ border: "1px solid #ccc", borderRadius: 8, padding: 16 }}
          >
            <h3>{MODEL_CONFIG[type].label}</h3>

            {status === "idle" && (
              <p style={{ color: "#999" }}>Awaiting run...</p>
            )}

            {status === "loading" && !results[type] && (
              <p style={{ color: "#999" }}>Loading...</p>
            )}

            {results[type]?.error && (
              <p style={{ color: "red" }}>Error: {results[type].error}</p>
            )}

            {results[type]?.text && (
              <>
                <p style={{ fontSize: 14, lineHeight: 1.6 }}>{results[type].text}</p>
                <hr />
                <p>
                  <strong>Latency:</strong>{" "}
                  {(results[type].latencyMs / 1000).toFixed(2)}s
                </p>
                {results[type].vramMB != null && (
                  <p><strong>VRAM:</strong> {results[type].vramMB} MB</p>
                )}
                {results[type].tokens != null && (
                  <p><strong>Tokens:</strong> {results[type].tokens}</p>
                )}
              </>
            )}
          </div>
        ))}
      </div>
    </div>
  );
}

This component issues requests to an Express backend that wraps the generateResponse function, rendering latency, VRAM, and response text in a three-column layout. All model requests run in parallel via Promise.allSettled for minimal wall-clock wait time. Note: server.js is not shown in this article; it must expose POST /api/generate accepting { prompt, modelType } and returning { text, latencyMs, modelType, vramMB?, tokens? }.

Implementation Checklist

1. Inventory your hardware (GPU model, VRAM, system RAM)
2. Determine your model size target (7B, 13B, 70B)
3. Calculate VRAM budget using the formula: parameters × bytes-per-weight (weights only; add KV cache estimate: 2 × context_length × num_layers × head_dim × num_heads × bytes-per-element)
4. Choose quantization format using the decision tree above
5. Download GGUF files tagged specifically as Q4_K_M in the filename (e.g., *.Q4_K_M.gguf) from the model's GGUF repository, AWQ-tagged for AWQ, original safetensors for FP16
6. Install the appropriate runtime (Ollama or llama.cpp for GGUF, vLLM or Transformers for AWQ)
7. Benchmark and stress-test: run identical prompts across formats comparing quality, speed, and memory; monitor VRAM during inference to confirm headroom; test long context windows, multi-turn conversations, and code generation. These three steps catch most deployment surprises before they reach users.
8. Document your configuration for reproducibility

Common Pitfalls and Troubleshooting

"Out of Memory" Despite Fitting on Paper

Reduce context length or increase GPU offloading granularity as a first response. The weight footprint is not the total VRAM cost: the KV cache grows with context length and can add several gigabytes at 4096 or 8192 token contexts. Operating systems reserve a portion of VRAM for display and system processes, typically 500MB to 1GB. A model that fits in theory will crash in practice if you ignore these hidden costs.

Quality Degradation in Specific Tasks

Quantization degrades some capabilities more than others. Mathematical reasoning, formal logic, and code generation are more sensitive to precision loss than summarization or open-ended conversation. If quality drops are noticeable on reasoning-heavy tasks, Q5_K_M or Q6_K offer middle-ground alternatives with modestly larger file sizes and improved precision for critical layers.

Mixing Up File Formats

GGUF, GPTQ, and AWQ are not interchangeable. A GGUF file cannot be loaded by vLLM, and an AWQ model cannot be loaded by llama.cpp. Always verify the model card metadata on Hugging Face to confirm the format and compatible inference engine before downloading.

GGUF, GPTQ, and AWQ are not interchangeable. A GGUF file cannot be loaded by vLLM, and an AWQ model cannot be loaded by llama.cpp. Always verify the model card metadata on Hugging Face to confirm the format and compatible inference engine before downloading.

Key Takeaways

FP16 remains the gold standard for output quality but demands hardware that most local setups cannot provide. AWQ delivers the best 4-bit inference performance on NVIDIA GPUs, with purpose-built kernels that maximize throughput. Q4_K_M is the most versatile format: it runs on CPUs, GPUs, Apple Silicon, and hybrid configurations with minimal setup, and the quality gap between it and AWQ at 4-bit is marginal for most practical tasks. Perplexity scores are a useful starting point. They do not, however, capture task-specific degradation patterns. Benchmark against your actual workloads, not generic leaderboards.