How to Run a 27B LLM on a 32GB AMD GPU (ROCm, No CUDA)

A 27B model fits comfortably on a 32GB AMD card if you pick the right quant and KV-cache settings. Here's the exact recipe used in ai-stack.

The VRAM math

A 27B model at Q6_K weighs about 21 GB on disk and loads into roughly the same amount of VRAM. On a 32 GB card that leaves ~11 GB for the KV cache and compute buffers — plenty, if you quantize the cache.

Component	~VRAM
Weights (Q6_K, 27B)	~21 GB
KV cache @ 128K ctx, q8_0 K/V	~6–8 GB
Compute / overhead	~1–2 GB

The llama.cpp command

Offload all layers to the GPU (-ngl 99), enable Flash Attention, and quantize the KV cache to q8_0 so a 128K context still fits:

llama-server --host 127.0.0.1 --port 8078 \
  -m /models/your-27b-Q6_K.gguf \
  -ngl 99 -fa on --jinja \
  -c 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 1

Tip: if you're VRAM-tight, drop context to -c 32768 or move to Q5_K_M (~19.6 GB) before touching -ngl — keeping all layers on the GPU matters most for speed.

Speed: speculative decoding (MTP)

Some 27B GGUF builds ship an embedded multi-token-prediction (MTP) draft head. llama.cpp can use it for speculative decoding, which roughly doubles tokens/sec for free:

  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.65

Running it hands-free

In ai-stack, this model is managed by llama-swap: it loads on the first chat request and unloads after an idle timeout so the GPU frees up for image generation. See the companion guide on sharing one GPU between an LLM and image models.

FAQ

Can a 27B model really run on a 32GB AMD GPU?

Yes. At Q6_K the weights are ~21 GB; with an 8-bit KV cache even a 128K context fits inside 32 GB on a single card.

Do I need CUDA or an NVIDIA GPU?

No. This runs on AMD ROCm with llama.cpp. No CUDA is involved.

What quantization should I use?

Q6_K is a strong quality/size balance for 27B on 32 GB. Drop to Q5_K_M if you need more headroom for context or image models.

Get the full stack on GitHub →