AMD Radeon AI PRO R9700 LLM Benchmarks (27B, ROCm)
Measured on the actual hardware, not estimated. A 27B model at Q6_K generates about 33 tokens/sec on the R9700 — comfortably faster than you can read.
Results
Single representative run, steady-state (model already loaded), via llama.cpp's own timings:
| Metric | Measured |
|---|---|
| Generation (decode) | 33.2 tokens/sec |
| Prompt eval (prefill) | 110.0 tokens/sec |
| MTP speculative-decode acceptance | 145 / 170 drafts (~85%) |
| Output length | 300 tokens |
What was tested
| GPU | AMD Radeon AI PRO R9700, 32 GB, RDNA4 (gfx1201) |
| Runtime | llama.cpp on ROCm, via llama-swap |
| Model | Qwen3.8-27B (Q6_K, ~21 GB) |
| Context / KV cache | 131072 ctx, q8_0 K and V |
| Features | Flash attention, embedded-MTP speculative decoding |
Key flags: -ngl 99 -fa on -c 131072 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp.
Reproduce it
curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"<your-model>","messages":[{"role":"user","content":"Write ~200 words about ROCm."}],"max_tokens":300,"stream":false}' \
| python3 -c 'import sys,json;t=json.load(sys.stdin)["timings"];print(t["predicted_per_second"],"tok/s decode;",t["prompt_per_second"],"tok/s prefill")'Technical FAQ
How many tokens per second does a 27B LLM get on the AMD R9700?
About 33 tokens/sec for generation and ~110 tokens/sec for prompt eval, measured on a 27B Q6_K model with llama.cpp on ROCm (128K context, q8_0 KV cache, MTP speculative decoding).
Is the Radeon AI PRO R9700 fast enough for local LLM chat?
Yes. ~33 tokens/sec on a 27B model is well above human reading speed, so chat feels responsive. Smaller models run much faster.
Does speculative decoding (MTP) help on AMD ROCm?
Yes — in this run the MTP draft head achieved ~85% acceptance (145 of 170 drafted tokens), which raises effective decode throughput at no quality cost.
What changes LLM throughput on the R9700?
Quantization (Q6_K vs Q5/Q4), context length, KV-cache type (q8_0 vs f16), whether speculative decoding is on, and competing GPU load (e.g. an image model) all affect tokens/sec.
How do I benchmark my own AMD GPU?
Send a non-streaming chat completion to your llama.cpp server and read the timings field it returns — predicted_per_second is decode tok/s, prompt_per_second is prefill. The exact command is above.