Open WebUI with a Local AMD ROCm LLM: Setup & Troubleshooting

Q: How do I connect Open WebUI to a local llama.cpp server?

Set OPENAI_API_BASE_URLS to your llama.cpp/llama-swap endpoint (e.g. http://llm:8080/v1 ) and put any non-empty string in OPENAI_API_KEYS . Open WebUI treats it as a standard OpenAI-compatible backend.

Q: My model list in Open WebUI is empty — why?

Open WebUI lists whatever /v1/models returns from your backend. If it's empty, llama-swap isn't reachable at the base URL, or no models are defined in its config. Check curl http://localhost:8080/v1/models .

Q: How do I set a default model for new chats?

Set the DEFAULT_MODELS environment variable to a model id that your backend serves. llama-swap has no default-model concept, so the default lives in Open WebUI.

Q: How do I add web search to Open WebUI without a paid API?

Run a local SearXNG instance and point Open WebUI at it: ENABLE_WEB_SEARCH=True , WEB_SEARCH_ENGINE=searxng , SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query> . No external search API or key needed.

Q: How do I keep everything private / loopback-only?

Bind each published port to 127.0.0.1 in your compose file (e.g. 127.0.0.1:3000:8080 ). Containers talk to each other over the internal network by service name; nothing is exposed to your LAN.

Open WebUI is the chat front-end in ai-stack. It talks to a local, OpenAI-compatible llama.cpp server over ROCm — here's how to wire it up and fix the usual snags.

The core connection

Open WebUI needs one setting to use your local AMD LLM — the OpenAI-compatible base URL of llama-swap:

OPENAI_API_BASE_URLS=http://llm:8080/v1
OPENAI_API_KEYS=sk-none        # any non-empty placeholder
WEBUI_AUTH=False               # single-user homelab
DEFAULT_MODELS=your-model-id   # pre-select a model for new chats

That's it — no NVIDIA, no cloud keys. The GPU work happens in the llama.cpp container; Open WebUI just renders chat and orchestrates tools.

Technical FAQ

How do I connect Open WebUI to a local llama.cpp server?

Set OPENAI_API_BASE_URLS to your llama.cpp/llama-swap endpoint (e.g. http://llm:8080/v1) and put any non-empty string in OPENAI_API_KEYS. Open WebUI treats it as a standard OpenAI-compatible backend.

My model list in Open WebUI is empty — why?

Open WebUI lists whatever /v1/models returns from your backend. If it's empty, llama-swap isn't reachable at the base URL, or no models are defined in its config. Check curl http://localhost:8080/v1/models.

How do I set a default model for new chats?

Set the DEFAULT_MODELS environment variable to a model id that your backend serves. llama-swap has no default-model concept, so the default lives in Open WebUI.

How do I add web search to Open WebUI without a paid API?

Run a local SearXNG instance and point Open WebUI at it: ENABLE_WEB_SEARCH=True, WEB_SEARCH_ENGINE=searxng, SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>. No external search API or key needed.

Can Open WebUI call external tools (MCP)?

Yes. Open WebUI supports MCP/tool servers: register a Streamable-HTTP MCP endpoint (e.g. http://mcp-search:8000/mcp) as a tool server and the model can call it during chat.

Does Open WebUI run on the GPU?

No — Open WebUI is just the UI/orchestration layer. The GPU is used by the llama.cpp (LLM) and ComfyUI (image) containers. Open WebUI runs fine on CPU.

How do I keep everything private / loopback-only?

Bind each published port to 127.0.0.1 in your compose file (e.g. 127.0.0.1:3000:8080). Containers talk to each other over the internal network by service name; nothing is exposed to your LAN.

Get the full stack on GitHub →