Skip to content
BranchesTags

Repository files navigation

vllm-local-developer-stack

License: MIT pre-commit Docker Compose Python ShellCheck NVIDIA CUDA vLLM

Production-grade automation for self-hosting Qwen3-Coder-30B-A3B-Instruct-AWQ on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.

Component Specification
GPUs 2× NVIDIA RTX 3060 12GB
Total VRAM 24 GB
Parallelism Tensor Parallel (size=2)
Host OS Ubuntu 22.04 LTS
Model QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ (MoE, 30B total / ~3B active)
Quantization AWQ (4-bit activation-aware)
Context 16,384 tokens — measured, not estimated; see Display Server Impact before raising this

Deployment Scope

This repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.

The intended use case is:

  • Run vLLM on one GPU-equipped Ubuntu host
  • Expose the vLLM API on the private LAN
  • Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider
  • Optionally connect a local web UI for direct chat interaction

This repo does not currently target:

  • Multi-node vLLM
  • Ray-based distributed inference
  • Kubernetes
  • Public internet exposure
  • Production authentication or TLS

Table of Contents

Repository Structure

vllm-local-developer-stack/
├── .gitignore                          # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results
├── .pre-commit-config.yaml             # Pre-commit framework configuration
├── .secrets.baseline                   # detect-secrets baseline — known false positives
├── README.md
├── deploy-artifacts/
│   ├── docker-compose.yml              # vLLM service definition (image, GPU reservation, healthcheck)
│   ├── docker-compose.open-webui.yml  # Optional WebUI service and volume definition
│   ├── docker-compose.litellm.yml     # Optional LiteLLM proxy service definition
│   ├── litellm-config.yaml.example    # Annotated reference for the auto-generated LiteLLM model_list
│   └── docker-compose.override.yml     # Auto-generated by deploy.sh — gitignored, do not edit
├── git-hooks/
│   └── check-commit-msg-secrets.py     # commit-msg hook: scans commit messages for secrets
└── scripts/
    ├── prereqs/
    │   └── install-prereqs.sh          # Idempotent dependency installer (drivers, Docker, toolkit)
    ├── deploy/
    │   ├── deploy.sh                   # Single entry point — orchestrates the full deployment
    │   ├── .env.example                # Annotated parameter reference — copy to .env
    │   ├── validate-system.sh          # Pre-flight hardware validation
    │   ├── validate-vram.sh            # Live startup telemetry monitor
    │   ├── setup-zed.sh                # Zed IDE integration hook (primary supported IDE)
    │   ├── setup-continue.sh           # Continue extension hook (VS Code & JetBrains)
    │   ├── setup-aider.sh              # Aider CLI integration hook
    │   ├── setup-opencode.sh           # OpenCode CLI integration hook (via LiteLLM proxy)
    │   ├── smoke-test.sh               # Smoke test verification for endpoints
    │   └── teardown.sh                 # Graceful server teardown wrapper (stops and deletes containers)
    └── tuning/
        ├── tune-inference.sh           # Hardware-sensing config generator
        ├── check-bottlenecks.sh        # Hardware & OS performance advisor (--json for machine-readable output)
        ├── snapshot-diagnostics.sh     # Read-only GPU state + log capture (run before teardown.sh when debugging)
        ├── benchmark.sh                # Single-stream token throughput evaluator
        ├── load-test.sh                # Concurrent-request load tester (req/s, latency percentiles)
        └── compare-benchmarks.sh       # Regression detector across benchmark-results/ history

benchmark.sh and load-test.sh each write a timestamped JSON record to benchmark-results/ (created on first run) so tuning changes can be compared over time with compare-benchmarks.sh.

Deploying

1. Configure scripts/deploy/.env

cp scripts/deploy/.env.example scripts/deploy/.env
$EDITOR scripts/deploy/.env          # set BIND_HOST, HF_CACHE_DIR at minimum

See scripts/deploy/.env.example for every available option and its explanation.

2. Run the deploy script

sudo bash scripts/deploy/deploy.sh

deploy.sh is the single entry point for standing up the stack. It is not something you run after validating and tuning the system — it runs those steps for you, in this order. Root is required end-to-end (package installation and enabling the Docker service both need it):

Step What happens Blocking?
1 Load and validate scripts/deploy/.env ✅ Aborts if .env is missing
2 Resolve BIND_HOST (auto-detects your LAN IP if unset) —
3 If BIND_HOST is a non-loopback address (e.g. 10.1.10.17, 192.168.0.x), check for an active firewall (ufw/firewalld) and open the API port if it isn't already allowed; if no firewall is active, or BIND_HOST is 127.0.0.1, skip — nothing to do —
4 Run install-prereqs.sh — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) ✅
5 Run validate-system.sh — GPU ↔ Docker connectivity, PCIe link quality ✅
6 Run check-bottlenecks.sh — performance advisory ⚠️ Never blocks
7 Run tune-inference.sh — updates only the GPU-tuned keys in scripts/deploy/.env in place, diffing against any existing values first ✅
8 Re-apply your user-set values on top (your settings always win over auto-tuning) —
9 Generate docker-compose.override.yml with the fully-resolved vLLM command —
10 Ensure docker.service is enabled at boot, so the container (restart: unless-stopped) comes back up after a host reboot, not just a plain restart —
11 docker compose up -d —
12 Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected —

A single successful run leaves you with a running, health-checked server at http://<BIND_HOST>:<PORT>/v1. If you want to run any of these steps individually — for debugging, or because you want finer control — see Manual Step-by-Step Setup below.

⚠ If NVIDIA drivers were just installed by install-prereqs.sh for the first time, reboot, then re-run deploy.sh.

3. Verify performance

Benchmark — single-stream throughput, using a multi-step Rust/Tokio programming prompt:

bash scripts/tuning/benchmark.sh
  Run  Elapsed (s)   Prompt tok    Comp tok    Tok/s      Finish
  ────────────────────────────────────────────────────────────────
  1    42.31          387           1024        24.20      length
  2    41.87          387           1024        24.46      length
  3    43.02          387           1024        23.80      length

  ════════════════════════════════════════════════════════════════
  SUMMARY (n=3 runs)
  ════════════════════════════════════════════════════════════════
  Avg prompt tokens                   387
  Avg completion tokens               1024
  Avg generation time                 42.40 s
  Avg throughput                      24.15 tok/s
  Peak throughput                     24.46 tok/s
  Min throughput                      23.80 tok/s
  ════════════════════════════════════════════════════════════════
  Performance tier: ✓ Good (15–30 tok/s)

  Results saved to: benchmark-results/benchmark_20260702T044144Z.json

Load test (optional) — benchmark.sh measures one request at a time; real usage (multiple editor sessions, multiple users) is concurrent. load-test.sh fires many small requests from several simultaneous workers and reports aggregate requests/sec, aggregate tokens/sec, and latency percentiles (p50/p95/p99):

bash scripts/tuning/load-test.sh [concurrency] [duration_seconds]   # defaults: 4, 30
bash scripts/tuning/load-test.sh 8 60                                # 8 concurrent clients, 60s

It also samples nvidia-smi's per-GPU throttle-reason flags (power cap, HW/SW thermal slowdown, power brake) once per second for the duration of the run and reports any that go active — distinct from check-bottlenecks.sh's point-in-time power-cap check, this catches actual throttling as it happens under sustained concurrent load (e.g. a card that's fine at idle but thermal-throttles a minute into real traffic).

Tracking changes over time — every benchmark.sh and load-test.sh run is saved as a timestamped JSON record in benchmark-results/. After changing scripts/deploy/.env (via tune-inference.sh) or host tuning (via check-bottlenecks.sh's recommendations), re-run the same script and diff against the previous result:

bash scripts/tuning/compare-benchmarks.sh              # latest 2 benchmark.sh runs
bash scripts/tuning/compare-benchmarks.sh --load-test   # latest 2 load-test.sh runs

It prints a per-metric delta table and exits non-zero if any metric regressed beyond its threshold (throughput: 10%, latency: 15–20% depending on percentile) — safe to drop into a personal tuning script or CI job for GPU tuning iterations.

benchmark-results/ grows one JSON file per run and is never pruned automatically. Once it has real history, trim it with:

bash scripts/tuning/compare-benchmarks.sh --prune              # dry run — lists what would be deleted (keeps last 20 of each type)
bash scripts/tuning/compare-benchmarks.sh --prune --keep 10     # dry run with a custom retention count
bash scripts/tuning/compare-benchmarks.sh --prune --force       # actually delete

Dry run is the default since deleting benchmark history is irreversible — nothing is removed until you pass --force.

Manual Step-by-Step Setup (Advanced)

deploy.sh covers the steps below automatically. Run them individually only if you want to re-run a single step in isolation (e.g. re-validate after a hardware change) or need to debug a specific stage.

Expand manual steps

Step 1 — Install prerequisites

sudo bash scripts/prereqs/install-prereqs.sh

Installs NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit, and system tools. Fully idempotent — safe to re-run.

Anything already installed (driver, Docker, toolkit, NVIDIA runtime registration in daemon.json) is left untouched — never updated, upgraded, or reconfigured. For anything missing, it prompts [y/N] before installing each component. Pass -y/--yes to auto-confirm every prompt for unattended/automated runs (e.g. deploy.sh still runs it in the foreground, so prompts surface normally unless you pass -y).

Step 2 — Validate system hardware

bash scripts/deploy/validate-system.sh

Checks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and display server VRAM impact on GPU 0.

Check Pass Condition
Docker GPU access Container sees all GPUs via nvidia runtime
PCIe Gen current == max for each GPU
PCIe Width current == max for each GPU
Idle VRAM (GPU 0) < 800 MiB (no display server consuming budget)

If a display server is detected on GPU 0, switch to headless mode before deploying (saves ~600–1500 MiB on GPU 0):

sudo systemctl isolate multi-user.target

Step 3 — Generate tuned configuration

bash scripts/tuning/tune-inference.sh

Queries your GPU topology dynamically and writes a hardware-appropriate scripts/deploy/.env. If scripts/deploy/.env doesn't exist yet, it's created from scratch. If it already exists, only the hardware-tuned keys below are updated in place — any change is printed as an old -> new diff before being applied. Everything else in the file (MODEL, BIND_HOST, PORT, HF_CACHE_DIR, HF_TOKEN, optional feature flags, comments) is left exactly as it is on disk.

Parameter Value (24 GiB setup) Rationale
TENSOR_PARALLEL_SIZE 2 One shard per GPU
GPU_MEMORY_UTILIZATION 0.90 Recommended value for 12 GiB RTX 3060 cards (see MEMORY & CONTEXT in .env.example) — needed for real headroom with this model; see Display Server Impact
MAX_MODEL_LEN 16384 Measured, not estimated — see Display Server Impact for the full story (an earlier 8192/0.85-util config booted but was too small for real agentic traffic) before raising it further
SWAP_SPACE 4 GiB CPU offload buffer for burst traffic

Review scripts/deploy/.env before continuing — you may manually adjust any value.

Step 4 — Start the server and monitor initialization

bash scripts/deploy/validate-vram.sh

Launches the container (docker compose up -d against the plain docker-compose.yml, no override file) and monitors VRAM allocation in real time during the KV cache loading phase, every 5s for up to 150s.

Signal Action
Uvicorn running on... ✅ Exit 0 — server ready
CUDA out of memory ❌ Exit 1 — prints recovery instructions
Timeout (150s) ⚠ Model still downloading — check docker logs vllm-coder-server --follow

Or bypass the monitor entirely:

docker compose -f deploy-artifacts/docker-compose.yml up -d
docker logs vllm-coder-server --follow

From here, continue with Verify performance above, or Client & Editor Integrations below.

⚠ Going this manual route skips the boot-persistence check deploy.sh does automatically. docker-compose.yml sets restart: unless-stopped, so the container itself restarts once the Docker daemon is up — but only if docker.service is enabled to start at boot:

sudo systemctl enable docker

API Usage

Once the server is running, it exposes a fully OpenAI-compatible API:

# Chat Completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder-30b-a3b-awq",
    "messages": [{"role": "user", "content": "Write a Python async HTTP client"}],
    "max_tokens": 512,
    "temperature": 0.1
  }'

# List loaded models
curl http://localhost:8000/v1/models

# Health check
curl http://localhost:8000/health

OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy",  # vLLM does not enforce API keys by default
)

response = client.chat.completions.create(
    model="qwen3-coder-30b-a3b-awq",
    messages=[{"role": "user", "content": "Implement a binary search tree in Go"}],
    max_tokens=1024,
    temperature=0.1,
)
print(response.choices[0].message.content)

Client & Editor Integrations

These are client-side setup steps, separate from deploying, tuning, or benchmarking the server. Each script below configures one editor/tool to point at your running vLLM endpoint — run interactively (it prompts, or reads BIND_HOST:PORT from .env if present) or with the host passed directly as an argument. Run a given script on whichever workstation has that tool installed — it does not need to be the machine hosting vLLM.

Zed

Zed is the primary supported editor for this stack's AI assistant integration.

Step 1 — Run the configuration script:

bash scripts/deploy/setup-zed.sh                    # reads host/port from .env or prompts
bash scripts/deploy/setup-zed.sh 192.168.1.50:8000  # or pass the host directly

This configures ~/.config/zed/settings.json, injecting the vLLM endpoint as a custom OpenAI-compatible provider. It resolves the live model ID and context window from GET /v1/models if the server is reachable, falling back to scripts/deploy/.env's SERVED_MODEL_NAME/MAX_MODEL_LEN otherwise. Any existing settings.json is backed up first (settings.json.bak.<timestamp>), so it's safe to re-run whenever the server's model or context length changes.

Step 2 — Set the API key placeholder:

vLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to be present:

  1. In Zed, open the Agent panel and click the model selector in the bottom-right corner → Configure... (or run the agent: settings command via Ctrl+Shift+A).
  2. Under the OpenAI provider section, enter dummy as the API key.
Setting Value
Provider openai (custom endpoint via api_url)
API Base http://<host>:<port>/v1
Model Resolved live from GET /v1/models if reachable (typically qwen3-coder-30b-a3b-awq) — falls back to .env's SERVED_MODEL_NAME
API Key Not enforced by vLLM — use dummy in Zed's provider settings

Tip

Zed's Agent panel (Ctrl+Enter) is a general chat window — it only prints copy-paste code blocks, it doesn't touch your files. To have the model edit the active file directly, highlight the target lines and invoke the Inline Assistant with Ctrl+I instead.

Continue (VS Code & JetBrains)

Install the Continue extension/plugin (VS Code Marketplace or JetBrains Marketplace, for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:

bash scripts/deploy/setup-continue.sh
# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000

Supported IDEs

  • VS Code — run the script, then reload the window (Ctrl+Shift+P → Developer: Reload Window).
  • JetBrains IDEs (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same ~/.continue/config.json global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.

How it Works / Custom Configurations

The setup script injects the vLLM endpoint into ~/.continue/config.yaml on the machine you run it on, setting it as both the chat model and the tab-autocomplete model. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.

Setting Value
Provider openai (OpenAI-compatible)
API Base http://<host>:<port>/v1
Model Resolved live from GET /v1/models if the server is reachable (typically qwen3-coder-30b-a3b-awq, matching --served-model-name) — falls back to MODEL= from scripts/deploy/.env with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place scripts/deploy/.env exists)
Autocomplete Same resolved model, max_tokens=512, temperature=0.05

If the server isn't reachable yet when you run this from a remote workstation, it falls back to the default HuggingFace model ID — re-run it once the server is up for an accurate config.

Aider

You can also use Aider as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.

bash scripts/deploy/setup-aider.sh
# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000

This script:

  1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via pipx or pip).
  2. Resolves the vLLM server address (interactively prompting for IP and port, reading from scripts/deploy/.env, or using the command-line argument).
  3. Safely updates or creates Aider configuration files (.aider.conf.yml at the project root or ~/.aider.conf.yml in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and edit-format: diff.
  4. Generates or updates .aider.model.metadata.json alongside your Aider config to register the correct context window size (based on the server's MAX_MODEL_LEN) and token cost structures, suppressing any "Unknown context window size and costs" warnings.

Important

The edit-format: diff setting is required, not optional. Aider doesn't recognize local/unregistered models in its built-in model list, so without it Aider silently falls back to read-only chat mode — it'll print code suggestions but never actually write them to disk. Forcing diff format makes the model emit unified diffs that Aider's parser can apply directly.

Once configured, simply run:

aider

OpenCode

OpenCode is a terminal-native AI coding agent. Unlike Zed/Continue/Aider above, it doesn't connect to vLLM directly — it connects to the LiteLLM proxy, which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind one OpenAI-compatible endpoint. LiteLLM must be deployed first (ENABLE_LITELLM=true in scripts/deploy/.env, then deploy.sh).

Run this on whichever workstation has (or should have) OpenCode installed — it does not need to be the machine hosting vLLM/LiteLLM:

bash scripts/deploy/setup-opencode.sh                    # reads host/port from .env or prompts
bash scripts/deploy/setup-opencode.sh 192.168.1.50:4000   # or pass the LiteLLM host directly

This script:

  1. Installs OpenCode if it isn't already on PATH (via the official curl -fsSL https://opencode.ai/install | bash installer, after confirmation).
  2. Resolves the LiteLLM proxy address (command-line argument, or scripts/deploy/.env's LITELLM_HOST/LITELLM_PORT when run on the LiteLLM host itself, or an interactive prompt).
  3. If the proxy has LITELLM_MASTER_KEY set, resolves it automatically (environment variable, or a local scripts/deploy/.env when run on the LiteLLM host itself) or prompts for it interactively if it detects a 401 while probing the proxy.
  4. Writes ~/.config/opencode/opencode.json, adding the opencode-plugin-litellm plugin and a provider.litellm block pointed at the proxy's LAN address. Any existing config is backed up first (opencode.json.bak.<timestamp>), so it's safe to re-run.
Setting Value
Provider litellm (@ai-sdk/openai-compatible, custom baseURL)
API Base http://<litellm-host>:<litellm-port>/v1
Models Not hand-listed. opencode-plugin-litellm queries GET /v1/models at every OpenCode startup and populates the model picker automatically — whatever is in LiteLLM's model_list (vLLM, DeepSeek, or anything else added later) shows up with zero config edits on the client side.
API Key The resolved LITELLM_MASTER_KEY if the proxy has auth enabled, otherwise the literal string dummy (unenforced, just satisfies the field)

Once configured, run opencode and select a model from /models — no manual config editing required, even after the server-side model_list changes (re-run deploy.sh on the server, then just restart opencode).

Tuning Reference

OOM Recovery

If vLLM exits with a CUDA OOM error during initialization:

# Edit scripts/deploy/.env
MAX_MODEL_LEN=8192            # Halve the context window (current default: 16384)

# Restart
docker compose -f deploy-artifacts/docker-compose.yml down
sudo bash scripts/deploy/deploy.sh

Already at GPU_MEMORY_UTILIZATION=0.90 (the recommended value for these cards) — don't raise it further as a fix for OOM; halving MAX_MODEL_LEN is the lever to pull. vLLM's own startup error reports the exact maximum context length that would have worked (e.g. "estimated maximum model length is 14208") — faster than guessing a halved value.

PCIe Bandwidth Notes

The RTX 3060 does not support NVLink. All inter-GPU communication for Tensor Parallelism goes over PCIe. A secondary slot running at x4 instead of x16 will reduce NCCL all-reduce bandwidth and may increase latency by 10–25% on large token batches.

To diagnose: run bash scripts/deploy/validate-system.sh and review the PCIe table (flags a GPU running below its own rated Gen/Width spec), or bash scripts/tuning/check-bottlenecks.sh for the more directly relevant check — it computes actual effective GB/s per link and flags anything below the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce needs, which a card can fail even while running at its own full rated spec (e.g. a Gen2×16 slot).

Display Server Impact

A desktop environment competing for GPU 0's VRAM can push KV cache allocation into an OOM at boot — this is why GPU_MEMORY_UTILIZATION=0.90 (not higher) and MAX_MODEL_LEN=16384 are the shipped defaults rather than more aggressive values the hardware can otherwise support. If GPU 0 shows

800 MiB idle usage, free it before deploying:

# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)
sudo systemctl isolate multi-user.target

# Re-enable desktop when done
sudo systemctl isolate graphical.target

Qwen3-Coder-30B-A3B-Instruct-AWQ genuinely has the tightest KV-cache headroom of any model this repo has shipped — measured, not estimated, and it took two rounds of tuning to land on values that actually work in practice, not just at server boot:

  1. GPU_MEMORY_UTILIZATION=0.85, MAX_MODEL_LEN=16384 — failed outright at vLLM startup: 
    ValueError: To serve at least one request with the model's max seq len (16384),
0.75 GiB KV cache is needed, which is larger than the available KV cache
memory (0.65 GiB). Based on the available memory, the estimated maximum
model length is 14208.
  2. Dropped to MAX_MODEL_LEN=8192 at the same 0.85 utilization — booted fine (vLLM reported a 14,208-token KV cache pool, 1.73x concurrency), but broke in real use: MAX_MODEL_LEN is a combined prompt+completion budget, and OpenCode requests its full declared max_output_tokens (4096) on essentially every call — any prompt over ~4K tokens (routine for agentic coding: tool schemas, file contents, conversation history) blew the 8192 ceiling, producing a ContextWindowExceededError on nearly every real request.
  3. Raised GPU_MEMORY_UTILIZATION to 0.90 — the value this repo's own .env.example already recommends for 12 GiB RTX 3060 cards; 0.85 had been a leftover conservative choice from tuning earlier, smaller-headroom models. The KV cache pool nearly doubled (1.23 GiB/GPU available, up from 0.65). Restored MAX_MODEL_LEN=16384 on top of that: now boots with real margin — GPU KV cache size: 26,928 tokens, 1.64x concurrency at 16384 tokens/request.

Verified live at the final config: tool calling with a real max_tokens=4096 request (OpenCode's actual pattern) succeeds; requests over the ceiling still get a clean 400, not a crash.

If GPU 0 is already headless (no idle usage), there may be a little more room to raise MAX_MODEL_LEN further — but given how little headroom this model leaves even at 0.90, don't guess. Run bash scripts/deploy/validate-vram.sh after any change and watch actual VRAM usage before committing to a higher value; if vLLM fails to start, its own error message reports the exact maximum context length that would have worked, which is the fastest way to find the real ceiling.

Server Management

# Stop the server
docker compose -f deploy-artifacts/docker-compose.yml down
# or: bash scripts/deploy/teardown.sh   (same thing, plus a post-stop VRAM confirmation table)

# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —
# useful when debugging a crash, OOM, or silent hang. Read-only, never
# modifies GPU/container state.
bash scripts/tuning/snapshot-diagnostics.sh [log_lines]   # default: last 50 log lines
# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt

# View live logs
docker compose -f deploy-artifacts/docker-compose.yml logs -f

# Restart after a config change
docker compose -f deploy-artifacts/docker-compose.yml down
sudo bash scripts/deploy/deploy.sh   # Re-tunes and regenerates the override file

# Check container health
docker inspect --format='{{.State.Health.Status}}' vllm-coder-server

Open WebUI Support (Optional)

This repository includes optional support for Open WebUI, allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.

Key Features

  • Browser Access: Use the model from any device (phone, laptop, tablet) on your local network.
  • Optional & Disabled by Default: Kept disabled by default (ENABLE_OPEN_WEBUI=false) to preserve the core vLLM-only focus.
  • Data Persistence: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal teardown.sh operations.
  • Auto-Boot: Starts automatically at host reboot alongside vLLM when enabled.

Configuration

All Open WebUI settings live in your scripts/deploy/.env file. Copy these values from scripts/deploy/.env.example if they are not already in your configuration:

# Enable Open WebUI deployment (true/false)
ENABLE_OPEN_WEBUI=true

# Port on the host network where Open WebUI will listen
OPEN_WEBUI_PORT=3000

# Subnet CIDR of your private LAN to restrict firewall access (optional)
# Example: LAN_CIDR=10.1.10.0/24
LAN_CIDR=10.1.10.0/24

Deployment

Simply set ENABLE_OPEN_WEBUI=true in scripts/deploy/.env and run the deployment script:

sudo bash scripts/deploy/deploy.sh

The script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to LAN_CIDR if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.

After successful deployment, the script outputs the connection URLs:

vLLM API      : http://10.1.10.17:8000/v1
Open WebUI    : http://10.1.10.17:3000

Note

On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.

Verification & Smoke Testing

To verify both services are running and accessible from either the host itself or another LAN client:

# Run the validation smoke test
bash scripts/deploy/smoke-test.sh

You can also pass overrides to verify connectivity from another machine on your LAN:

# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]
bash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true

The smoke test validates:

  1. vLLM /v1/models endpoint responds successfully.
  2. Open WebUI HTTP endpoint responds on the configured port.
  3. Both containers are configured with unless-stopped (or your configured) restart policies.

Teardown and Data Preservation

To stop the services and release GPU VRAM:

bash scripts/deploy/teardown.sh

This stops both vLLM and Open WebUI containers. Your Open WebUI chat history, user accounts, and settings are preserved.

To perform a deep-clean and delete all Open WebUI data/volumes, pass the --purge flag:

# WARNING: This deletes the Open WebUI database/volume permanently!
bash scripts/deploy/teardown.sh --purge

Troubleshooting

  • Container fails to start or port already in use: The deployment script checks port availability and fails-fast. If the port is in use, verify with ss -tlnp (as root) or configure a different OPEN_WEBUI_PORT in scripts/deploy/.env.
  • Cannot reach Open WebUI from another LAN host: Ensure OPEN_WEBUI_HOST is set to 0.0.0.0 (all interfaces) in scripts/deploy/.env. Verify that the firewall (UFW/firewalld) is allowing the port and that LAN_CIDR matches your client's subnet.
  • Open WebUI cannot reach vLLM: Open WebUI connects to vLLM inside the Docker network. Ensure OPEN_WEBUI_OPENAI_API_BASE_URL in scripts/deploy/.env points to http://vllm:8000/v1 (using the container service name vllm rather than localhost).
  • Services not starting after reboot: Check if the Docker service is enabled to start at boot (systemctl is-enabled docker). Verify the restart policies in scripts/deploy/.env (VLLM_RESTART_POLICY and OPEN_WEBUI_RESTART_POLICY) are set to unless-stopped or always.

LiteLLM Proxy Support (Optional)

This repository includes optional support for LiteLLM, which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind a single OpenAI-compatible endpoint on the LAN. This is what enables OpenCode's automatic multi-model discovery — clients query LiteLLM's /v1/models instead of each backend individually, and don't need a hand-maintained model list.

This section covers only what this repo automates: enabling the proxy, its model_list, and its optional master-key auth. LiteLLM is a large project with its own extensive feature set (routing/load-balancing strategies, per-key budgets and rate limits, spend tracking, caching, guardrails, a Postgres-backed virtual-key store, Prometheus metrics, and more) — none of that is wired up here. For anything beyond basic install/config, see the official LiteLLM documentation rather than this README; deploy-artifacts/litellm-config.yaml.example links back to the relevant docs pages inline where it matters.

Key Features

  • One endpoint, multiple models: GET /v1/models on the LiteLLM proxy lists every model in its model_list — the local vLLM model, plus DeepSeek when DEEPSEEK_API_KEY is configured.
  • Optional & Disabled by Default: Kept disabled by default (ENABLE_LITELLM=false) to preserve the core vLLM-only focus, same as Open WebUI.
  • Reuses vLLM's healthcheck: The proxy container's depends_on: vllm: condition: service_healthy means LiteLLM only starts once vLLM's own /health check passes — no separate readiness mechanism to keep in sync.
  • Auto-Boot: Starts automatically at host reboot alongside vLLM when enabled (restart: unless-stopped, same as every other service in this stack).
  • Secrets never touch a generated file: DEEPSEEK_API_KEY and LITELLM_MASTER_KEY are read by LiteLLM directly from its container environment (os.environ/... references in the generated config) — neither is ever written into deploy-artifacts/litellm-config.yaml itself, same secrets pattern as HF_TOKEN for vLLM.
  • Optional auth: no authentication by default (matches vLLM/Open WebUI), or set one master key to require Authorization: Bearer <key> on every request — see Authentication below.
  • Always includes a Postgres database: not independently toggleable — enables the Admin UI and virtual keys with no extra config; see Database below.

Configuration

All LiteLLM settings live in your scripts/deploy/.env file. Copy these values from scripts/deploy/.env.example if they are not already in your configuration:

# Enable the LiteLLM proxy deployment (true/false)
ENABLE_LITELLM=true

# Port on the host network where the LiteLLM proxy will listen
LITELLM_PORT=4000

# DeepSeek API key — omit to run a vLLM-only proxy
DEEPSEEK_API_KEY=sk-your_key_here

# Optional: require auth on every proxy request — see Authentication below
LITELLM_MASTER_KEY=your-password-here

deploy-artifacts/litellm-config.yaml (the file LiteLLM actually loads) is auto-generated by deploy.sh from these values every run — see deploy-artifacts/litellm-config.yaml.example for the annotated reference. It's gitignored; don't hand-edit it.

Authentication (Optional)

By default the proxy has no authentication — same trusted-home-LAN posture as vLLM and Open WebUI. Set LITELLM_MASTER_KEY in scripts/deploy/.env to require Authorization: Bearer <key> on every proxy request, including GET /v1/models.

  • Any string works as the key. LiteLLM does a plain comparison — there's no required sk- prefix or format, despite that being a common convention in examples (including its own docs).
  • /health and /health/liveliness stay exempt — Docker's healthcheck doesn't send credentials, so enabling auth doesn't break container health status.
  • Takes effect on the next deploy.sh run — the key is baked into the container's environment and the generated config's general_settings.master_key at deploy time, not read live.
  • Log into the LiteLLM Admin UI at http://<host>:<port>/ui with any username and the master key as the password — this requires the database below to actually be connected; without it, /ui login fails outright with Authentication Error, Not connected to DB! (not just "no key set"). This repo always deploys the database alongside LiteLLM, so this works out of the box. See the official Admin UI docs for what's in there.
  • scripts/deploy/setup-opencode.sh resolves it automatically — from the environment, from a local scripts/deploy/.env when run on the proxy host itself, or by prompting interactively if it detects a 401 while probing the proxy.
LITELLM_MASTER_KEY=your-password-here

Verify it's working:

# Unauthenticated — should now fail with 401
curl -o /dev/null -w '%{http_code}\n' http://10.1.10.17:4000/v1/models

# Authenticated — should succeed
curl -H "Authorization: Bearer your-password-here" http://10.1.10.17:4000/v1/models

This repo only wires up the single master key. LiteLLM's virtual keys (per-user/per-app keys with their own budgets, rate limits, and model access lists, issued via the master key) are a real LiteLLM feature but not something deploy.sh generates or manages — see the virtual keys docs if you need that.

Database

LiteLLM is always deployed with a Postgres database (litellm-db, postgres:16-alpine) — it's not independently toggleable, only ENABLE_LITELLM gates the whole feature. Without it, LiteLLM runs in "DB-less mode": model proxying and LITELLM_MASTER_KEY auth on API requests still work fine, but /ui login fails and there's no way to create virtual keys.

  • Not exposed on any host port. Only the litellm container can reach it, over the Docker Compose network (litellm-db:5432) — there's no LITELLM_DB_PORT to open a firewall rule for.
  • LITELLM_DB_PASSWORD is auto-generated by deploy.sh on first run and persisted to scripts/deploy/.env — an internal container-to-container credential only, never typed by a human. Generated once and kept stable across redeploys deliberately: the password is baked into the Postgres volume's own user table on first init, so a fresh random value on a later run would lock LiteLLM out of its own already-provisioned database.
  • Persistent volume (LITELLM_DB_DATA_VOLUME, default litellm-db-data) — survives container restarts, redeployments, and plain teardown.sh. Only removed by teardown.sh --purge (same flag that also removes Open WebUI's data — there's no way to purge just one).
  • First boot runs LiteLLM's own Prisma migrations against the fresh database before the proxy starts serving — this can take significantly longer than a normal restart (observed ~40s on top of regular startup on this deployment's hardware). deploy.sh's post-deployment validation retries for up to 90s to account for this; subsequent redeploys against an already-migrated database are fast.
  • Verify it's actually connected, not just that the container is healthy (container health only checks Postgres itself, not that LiteLLM successfully attached to it) — try a DB-backed feature like minting a virtual key: 
    curl -X POST http://10.1.10.17:4000/key/generate \
  -H "Authorization: Bearer your-master-key" \
  -H "Content-Type: application/json" -d '{}'
    A JSON key object back means the DB is genuinely wired up; Authentication Error, Not connected to DB! means it isn't.

Beyond standing the database up, this repo doesn't configure anything that uses it — budgets, rate limits, teams/orgs, spend dashboards, and everything else backed by it are managed through the Admin UI or LiteLLM's own API once you're logged in; see the official LiteLLM proxy docs for all of that.

Deployment

Simply set ENABLE_LITELLM=true in scripts/deploy/.env and run the deployment script:

sudo bash scripts/deploy/deploy.sh

The script automatically detects that LiteLLM is enabled, generates litellm-config.yaml from MODEL/SERVED_MODEL_NAME/DEEPSEEK_API_KEY, performs port availability checks, opens UFW/firewalld rules (restricted to LAN_CIDR if set), launches the container stack in dependency order (vLLM → litellm-db → LiteLLM), and validates that all of them are running, correctly configured with their restart policies, and that /v1/models lists the expected model(s).

After successful deployment, the script outputs the connection URLs alongside vLLM's and Open WebUI's:

vLLM API      : http://10.1.10.17:8000/v1
LiteLLM Proxy : http://10.1.10.17:4000/v1
LiteLLM UI    : http://10.1.10.17:4000/ui

From there, run setup-opencode.sh on any workstation to point OpenCode at it.

Troubleshooting

  • /v1/models doesn't list the vLLM model: Check deploy-artifacts/litellm-config.yaml (auto-generated — confirm SERVED_MODEL_NAME in scripts/deploy/.env matches) and docker logs litellm-proxy.
  • /v1/models doesn't list DeepSeek: DEEPSEEK_API_KEY is unset or invalid — deploy.sh prints a warning and omits the DeepSeek entry entirely rather than shipping a broken one.
  • Cannot reach LiteLLM from another LAN host: Ensure LITELLM_HOST is set to 0.0.0.0 in scripts/deploy/.env, and that the firewall (UFW/firewalld) is allowing LITELLM_PORT.
  • LiteLLM cannot reach vLLM: LiteLLM connects to vLLM inside the Docker network via the vllm service name (http://vllm:8000/v1), same as Open WebUI — this is hardcoded into the generated config and isn't user-configurable.
  • 401 Unauthorized on /v1/models or chat completions: LITELLM_MASTER_KEY is set — every request needs Authorization: Bearer <key>. See Authentication. If you didn't mean to enable auth, remove LITELLM_MASTER_KEY from scripts/deploy/.env and re-run deploy.sh.
  • /ui login fails with Authentication Error, Not connected to DB!: the litellm-db container isn't healthy, or deploy.sh hasn't run since it was added. Check docker inspect --format='{{.State.Health.Status}}' litellm-db and docker logs litellm-db; see Database for the virtual-key verification command that confirms the connection independently of container health status.
  • Deploy seems to hang for ~30–60s longer than usual on litellm-proxy: expected on the first deploy after enabling LiteLLM (or after a fresh litellm-db volume) — LiteLLM runs its own Prisma migrations against the database before it starts serving. Subsequent redeploys are fast; deploy.sh already budgets for this (retries its /v1/models check for up to 90s).

Security Notes

  • The server binds to 0.0.0.0 on all interfaces by default. If running on a network-accessible machine, set BIND_HOST=127.0.0.1 in scripts/deploy/.env to restrict it to localhost, or add a firewall rule.
  • If BIND_HOST is set to a LAN address, deploy.sh opens the API port on ufw/firewalld automatically (only if one of them is active — it never installs or enables a firewall for you). It only ever opens the single PORT from scripts/deploy/.env, never a broad range.
  • vLLM does not enforce API key authentication by default. Add --api-key <secret> to the command in docker-compose.yml (or via docker-compose.override.yml) to enable it.
  • LiteLLM (when enabled) also has no auth configured by default — same trusted-home-LAN assumption as vLLM and Open WebUI. It does, however, hold your DEEPSEEK_API_KEY in its container environment; anyone who can reach LITELLM_PORT on the LAN can spend against that key through the proxy. Restrict LAN_CIDR in scripts/deploy/.env, and/or set LITELLM_MASTER_KEY to require Authorization: Bearer <key> on every request — see Authentication.
  • The HuggingFace cache is mounted from the host via HF_CACHE_DIR. Ensure the model cache directory has appropriate permissions.

Development & Code Quality

This repository uses pre-commit to automate code validation and enforce security best practices before any changes are committed.

Installed Hooks

Syntax linting & formatting

  • trailing-whitespace — trims trailing whitespace from files
  • end-of-file-fixer — ensures files end with a newline
  • check-yaml — validates YAML syntax (e.g. deploy-artifacts/docker-compose.yml, .pre-commit-config.yaml)
  • check-json — validates JSON syntax
  • check-added-large-files — blocks accidentally committing large files (e.g. model weights, cached tensors)
  • shellcheck — runs ShellCheck on all shell scripts in scripts/

Security & secret detection

  • detect-private-key — checks for the presence of private keys
  • detect-secrets — scans staged changes for hardcoded secrets, API keys, or credentials using detect-secrets (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder api_key="dummy" used since vLLM doesn't enforce API keys by default) are tracked in .secrets.baseline — if you intentionally add a new one, regenerate it with detect-secrets scan > .secrets.baseline and mark it as a false positive.
  • Custom commit message scanner — git-hooks/check-commit-msg-secrets.py scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the commit-msg hook phase

Manual Verification

Run every pre-commit check against all files at any time:

pre-commit run --all-files

License

MIT

About

Containerized local LLM development stack. Features automated multi-GPU vLLM hosting (optimized for Qwen-Coder), | Tunables & Benchmarks │ One-click integration configurations for Zed IDE, VS Code Continue, and Aider CLI.

Topics

ai-assistantaidercontinue-devdocker-composedual-gpugpu-accelerationlocal-llmqwenqwen-coderself-hostedvllmzed-editor

Resources

Readme
Apache-2.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages