Running llama.cpp on AMD ROCm (RDNA4 / gfx1201)

Getting llama.cpp talking to an AMD RDNA4 card under rootless Podman comes down to a few device and permission details. Here they are.

GPU passthrough into the container

ROCm needs two device nodes and membership in the render and video groups. In a Podman/compose service:

devices: ["/dev/kfd", "/dev/dri"]
group_add: ["<render-gid>", "<video-gid>"]   # getent group render / video
security_opt: ["seccomp=unconfined"]
environment: ["HIP_VISIBLE_DEVICES=0"]

HIP_VISIBLE_DEVICES=0 pins to the discrete GPU (check rocm-smi if an iGPU also appears). RDNA4 cards report as gfx1201; build or pull a ROCm llama.cpp image targeting that arch.

Verify ROCm sees the card

rocm-smi                 # on the host
podman exec <llm> rocminfo | grep gfx   # inside the container -> gfx1201

On-demand model swapping with llama-swap

llama-swap sits in front of llama.cpp and exposes an OpenAI-compatible API, loading each model on first request and unloading it after a TTL. That's what lets a chat model and a coder model coexist, and frees VRAM when idle:

models:
  "chat-27b":
    cmd: llama-server -m /models/chat.gguf -ngl 99 -fa on -c 131072 ...
    ttl: 600          # unload after 10 min idle
  "coder-1.5b":
    cmd: llama-server -m /models/coder.gguf -ngl 99 -fa on -c 16384 ...
    ttl: 1800

Point any OpenAI-compatible client (like Open WebUI) at http://localhost:8080/v1 — llama-swap handles the rest.

FAQ

What GPU architecture is RDNA4 for ROCm?

RDNA4 Radeon cards (e.g. the AI PRO R9700) report as gfx1201. Use a ROCm/llama.cpp build targeting gfx1201.

Why do I need the render and video groups?

ROCm accesses /dev/kfd and /dev/dri; the container user must be in the host's render and video groups to use them.

Can I run multiple models without reloading constantly?

Yes — llama-swap keeps lightweight models loaded and swaps heavy ones on demand, with per-model idle TTLs.

Get the full stack on GitHub →