Beyond Chat: How to Set Up Local LLM Code Completion in VS Code with Ollama

Open-source models have caught up, and pair that with Ollama for local model serving and the Continue extension for VS Code integration, you can build a fully offline code completion pipeline in under 30 minutes. This tutorial walks you through every step.

Table of Contents

• Why Local Code Completion Matters Now
• How Cloud-Based Code Completion Creates Risk
• The Local LLM Stack: Architecture Overview
• Prerequisites and Hardware Requirements
• Step 1: Installing and Configuring Ollama
• Step 2: Installing and Configuring the Continue Extension
• Step 3: Testing and Optimizing the Experience
• Step 4: Adding Chat and Contextual Q&A
• Model Comparison: Which Local Model Should You Choose?
• Real-World Performance: How Does It Stack Up Against Copilot?
• Troubleshooting Common Issues
• Your Code, Your Machine, Your Rules

Why Local Code Completion Matters Now

GitHub Copilot has become table stakes for millions of developers. Microsoft reported more than 77,000 organizational customers during its Q3 FY2025 earnings call. It dominates the AI coding assistant market. But here's the tension: the same organizations adopting AI coding tools are also tightening the rules around them. Setting up local LLM code completion in VS Code gives you the productivity boost without the compliance headache.

Samsung banned employee use of ChatGPT and similar generative AI tools after engineers uploaded confidential source code and internal meeting notes to external servers. Finance, defense, and healthcare companies followed with their own restrictions. The worry is straightforward: intellectual property exposure and regulatory obligations. When your code leaves your machine and hits a third-party API, you lose control over how it gets stored, processed, or fed into model training.

When your code leaves your machine and hits a third-party API, you lose control over how it gets stored, processed, or fed into model training.

The good news: open-source models have caught up. Qwen2.5-Coder-7B scores competitively on HumanEval, putting it in the same ballpark as earlier GPT-3.5-level performance. Pair that with Ollama for local model serving and the Continue extension for VS Code integration, and you can build a fully offline code completion pipeline in under 30 minutes. This tutorial walks you through every step.

How Cloud-Based Code Completion Creates Risk

The Data Leakage Problem

When you use a cloud-based code assistant, your editor ships code context to a remote server for inference. That context can include proprietary business logic, API keys accidentally left in source files, internal architecture patterns, and unreleased feature implementations. The Samsung incident makes this concrete: employees pasted sensitive semiconductor source code and internal meeting transcripts into ChatGPT, effectively handing trade secrets to an external service. Once data hits a third-party server, your organization's control over it depends entirely on the vendor's retention policies and security posture.

Compliance and Regulatory Constraints

If your organization operates under HIPAA, SOC 2, ITAR, or FedRAMP requirements, sending code context to external APIs introduces audit complexity at best and outright violations at worst. Air-gapped environments, common in defense and critical infrastructure, make cloud-based tools physically impossible to use.

Local inference sidesteps all of this. The model runs on your hardware. The data stays in your process. There's no network request to audit or justify.

The Local LLM Stack: Architecture Overview

The architecture is simple. VS Code runs the Continue extension, which acts as the client layer for both tab completions and chat interactions. Continue sends requests over HTTP to Ollama, which runs as a local server on your machine. Ollama loads the model into memory, runs inference, and streams results back to Continue, which renders them as ghost text in your editor or as chat responses in the sidebar.

The flow:

VS Code Editor → Continue Extension → HTTP (localhost:11434) → Ollama → Loaded Model → Response

For tab completions, Continue uses fill-in-the-middle (FIM) prompting. It sends the code before and after your cursor as context and asks the model to predict what goes in between. For chat, it uses standard instruct-style prompting. Ollama handles model management, quantization, and GPU offloading, so you never touch raw model weights directly.

Prerequisites and Hardware Requirements

Minimum and Recommended Specs

You don't need a monster rig for this, but more muscle helps.

• 16 GB system RAM minimum, a modern multi-core CPU. No dedicated GPU required for 7B parameter models at Q4 quantization, though inference will be noticeably slower on CPU alone.
• For a better experience: Apple Silicon M1 Pro or newer (unified memory makes 7B models very fast), or an NVIDIA RTX 3060 or better with at least 12 GB VRAM. 32 GB system RAM gives you headroom for larger models.
• macOS, Linux, and Windows (via native install or WSL2) all work.

Software You'll Need

• VS Code (latest stable release)
• Ollama (latest release from ollama.com)
• Continue extension (from the VS Code Marketplace)

That's it. No Docker, no Python environment, no CUDA toolkit wrangling.

Step 1: Installing and Configuring Ollama

Installation

Install Ollama with a single command:

# macOS (via Homebrew) or download from ollama.com brew install ollama # Linux curl -fsSL https://ollama.com/install.sh | sh # Windows # Download the installer from https://ollama.com/download

After installation, start the Ollama service:

ollama serve

On macOS and Windows, the desktop app starts the service automatically.

Pulling a Code-Optimized Model

Qwen2.5-Coder at 7B parameters is a strong choice for local code completion. It supports a 128K context window natively (though you'll typically constrain this in practice for performance) and runs comfortably on 16 GB of RAM at Q4 quantization. Pull it:

ollama pull qwen2.5-coder:7b

Run a quick smoke test:

ollama run qwen2.5-coder:7b "Write a Python function that checks if a string is a palindrome"

You should see a complete, syntactically valid function streamed to your terminal. If you get garbage, something went wrong with the pull. Try again.

Verifying the Local API

Ollama exposes a REST API on http://localhost:11434 by default. Verify it:

curl http://localhost:11434/api/generate -d '{ "model": "qwen2.5-coder:7b", "prompt": "def fibonacci(n):\n ", "stream": false }'

You should get back a JSON response with a response field containing the model's completion. If this works, your local inference backend is ready.

Step 2: Installing and Configuring the Continue Extension

Why Continue Over the Alternatives

Several extensions connect VS Code to local models: Twinny, llama-coder, CodeGPT, among others. I keep coming back to Continue for a few reasons. It supports both tab autocomplete and sidebar chat in a single extension. It lets you assign different models to different roles, so you can run a small, fast model for completions and a bigger one for chat. The maintainers ship frequent releases. And it's open source.

Installation in VS Code

Open the Extensions panel, search for "Continue," click Install. Done.

For air-gapped environments, download the .vsix from the Continue GitHub releases page and install manually via Extensions: Install from VSIX in the command palette.

The Config File: Your Control Center

Continue stores its configuration in config.json (the exact location depends on your Continue version; newer versions use ~/.continue/config.json, though the project has been migrating toward a config.yaml format). This file defines which models to use, how to reach them, and how completions behave. Here's a complete, annotated configuration you can drop in and start using:

{ "models": [ { "title": "Qwen2.5-Coder 7B (Chat)", "provider": "ollama", "model": "qwen2.5-coder:7b", "apiBase": "http://localhost:11434", "contextLength": 32768 } ], "tabAutocompleteModel": { "title": "Qwen2.5-Coder 7B (Autocomplete)", "provider": "ollama", "model": "qwen2.5-coder:7b", "apiBase": "http://localhost:11434", "contextLength": 4096 }, "tabAutocompleteOptions": { "debounceDelay": 500, "maxPromptTokens": 2048, "multilineCompletions": "always", "useCache": true }, "requestOptions": { "timeout": 30000 }, "allowAnonymousTelemetry": false }

Let me walk through what matters here.

tabAutocompleteModel is the model that powers inline ghost-text completions. I set contextLength to 4096 here because shorter context keeps inference fast. You're not writing a novel; you're completing a function.

models defines what's available in the chat sidebar. Higher contextLength makes sense here because chat interactions are less latency-sensitive than tab completions.

debounceDelay controls how many milliseconds Continue waits after you stop typing before firing a completion request. 500ms works well on most hardware. Drop it to 300ms if you're on an M3 Max and feeling impatient.

maxPromptTokens controls how much surrounding code Continue sends as context. More context means better suggestions but slower inference. 2048 is a solid default.

multilineCompletions set to "always" gives you full function body completions, not just single-line fills. This is where local models really start to feel useful.

allowAnonymousTelemetry set to false keeps everything local. The whole point of this setup, right?

You're not writing a novel; you're completing a function.

Step 3: Testing and Optimizing the Experience

Your First Local Tab Completion

Open any code file in VS Code. Type a function signature and pause:

# You type this def calculate_moving_average(data: list[float], window: int) -> list[float]: pass # Continue shows ghost text like this, which you accept with Tab def calculate_moving_average(data: list[float], window: int) -> list[float]: if len(data) < window: return [] result = [] for i in range(len(data) - window + 1): window_slice = data[i:i + window] result.append(sum(window_slice) / window) return result

Press Tab to accept. Escape to dismiss.

Tuning for Speed

If completions feel sluggish, you've got several levers to pull.

Cut contextLength in tabAutocompleteModel to 2048 or even 1024. Pull a Q4 quantization variant if you haven't already. Bump debounceDelay to 750ms to reduce inference calls while you're still mid-thought.

You can also create a custom Modelfile for Ollama with tuned parameters:

FROM qwen2.5-coder:7b PARAMETER num_ctx 2048 PARAMETER temperature 0.2 PARAMETER top_p 0.9

Save that as Modelfile and create your custom model:

ollama create qwen-fast -f Modelfile

Then update config.json to reference qwen-fast as the model name.

Tuning for Quality

Want better suggestions? Increase contextLength, bump temperature to 0.4 for more varied completions, or step up to a 14B or 32B model if your hardware can handle it. The tradeoff is always speed versus quality, and the right balance depends on your machine and your patience.

Step 4: Adding Chat and Contextual Q&A

Setting Up the Chat Model

You can use the same model for both autocomplete and chat, or dedicate a larger model to chat. To add a separate chat model, update the models array in your config:

{ "models": [ { "title": "Qwen2.5-Coder 14B (Chat)", "provider": "ollama", "model": "qwen2.5-coder:14b", "apiBase": "http://localhost:11434", "contextLength": 32768 } ] }

The chat sidebar in Continue supports inline edit commands, code explanation, and refactoring workflows, all running against your local model.

Using @-Mentions for Context

Continue supports context providers that let you reference parts of your workspace in chat. Type @file to include a specific file's contents, @codebase to let Continue search your project for relevant code, or @terminal to include recent terminal output. This gives the model grounded context without any data leaving your machine.

I use @codebase constantly. It's the closest thing to Copilot's multi-file awareness you'll get locally.

Model Comparison: Which Local Model Should You Choose?

A note on those numbers: HumanEval pass@1 scores vary significantly depending on evaluation methodology, prompting strategy, and quantization. The figures above are approximate, based on published benchmarks from model authors. Your results with quantized local inference will differ. I've seen Qwen2.5-Coder produce noticeably worse completions at Q4 than its reported benchmarks suggest. Still good. Just not magic.

For web development in TypeScript or Python, Qwen2.5-Coder-7B offers the best quality-to-size ratio. I've tried the others and keep coming back to it.

If you're working on enterprise Java or C++ with large files, DeepSeek-Coder-V2-Lite handles longer context better, though note it uses a Mixture-of-Experts architecture with 2.4B active parameters out of 16B total.

Running on 8 GB of RAM? Drop to qwen2.5-coder:3b. The quality takes a hit, but it's still useful for boilerplate and simple completions.

Real-World Performance: How Does It Stack Up Against Copilot?

Where Local Wins

Local inference has zero latency variance. No network round-trip. No server-side queuing. No outage risk. It works on planes, in classified facilities, behind any firewall. There's no telemetry to audit, no data retention policy to negotiate with legal.

I ran both setups side-by-side for two weeks. For single-file Python and TypeScript work, the local setup felt faster than Copilot on several occasions, simply because there was no network jitter.

Where Cloud Still Leads

Copilot benefits from frontier-class models with massive context windows and multi-file awareness powered by cloud-scale compute. For complex cross-file refactoring, cloud models still produce better results. And there's zero hardware overhead on your local machine.

The Honest Assessment

Local 7B models cover the majority of what most developers actually use Copilot for day-to-day: single-file completions, boilerplate generation, short function implementations. Stepping up to 14B or 32B for chat narrows the gap further. For developers in restricted environments, this is more than enough.

It won't blow your mind on a complex multi-file refactor across 15 modules. Be realistic about that.

Local 7B models cover the majority of what most developers actually use Copilot for day-to-day: single-file completions, boilerplate generation, short function implementations.

Troubleshooting Common Issues

If completions aren't appearing, verify Ollama is running with ollama list (it should show your pulled models). Check that the model name in config.json exactly matches the Ollama model tag. Open the Continue output panel in VS Code (Output → Continue) to see error messages. Nine times out of ten, it's a model name mismatch.

Slow or laggy completions usually mean you're sending too much context. Reduce contextLength and maxPromptTokens in your autocomplete config. Run ollama ps to confirm the model is loaded and check whether GPU layers are being offloaded. If you see 0 GPU layers, that's your bottleneck.

Getting nonsensical suggestions? Make sure the model you're using supports FIM (fill-in-the-middle) prompting. Lower temperature to 0.1 or 0.2 for more deterministic output. Cut maxTokens to stop the model from rambling past the useful completion.

Your Code, Your Machine, Your Rules

The local code completion stack has crossed from "interesting experiment" to "daily driver." Ollama makes model management painless, Continue provides a polished editor experience, and 7B-class code models deliver genuinely useful suggestions. This isn't about replacing Copilot for every developer. It's about having a real alternative when your code can't leave your machine.

This isn't about replacing Copilot for every developer. It's about having a real alternative when your code can't leave your machine.

Copy the config JSON from this article, pull qwen2.5-coder:7b, and run it for a week alongside your current workflow. The Continue and Ollama communities on Discord and GitHub are active and helpful when you hit snags.

Your code stays on your hardware. That's the whole point.