Run an LLM and Image Generation on One GPU Without Running Out of VRAM
One 32GB GPU can't hold a 27B LLM and a diffusion pipeline at once. The fix is to treat VRAM as a shared resource and hand it off automatically.
The problem
A 27B LLM needs ~28 GB with its KV cache; a modern image pipeline needs several GB more. On a 32 GB card they can't coexist. Loading both means an out-of-memory crash.
The solution: automatic handoff
- Idle unload. llama-swap unloads the chat model after an idle timeout (
ttl: 600), freeing the GPU when you stop typing. - Unload-before-generate. The image tool explicitly unloads the LLM before submitting a ComfyUI job.
- Free-after-generate. After the image is produced, ComfyUI's VRAM is freed (
--cache-noneplus a free call) so chat resumes instantly.
Why it matters on AMD
AMD cards give you more VRAM per dollar, but you still want to use every GB well. This handoff pattern is what makes a single mid-range AMD workstation card behave like a much more expensive setup — see running a 27B LLM on 32GB and ComfyUI on ROCm.
FAQ
Can one GPU run both an LLM and Stable-Diffusion-style image generation?
Not simultaneously on a 32GB card, but yes sequentially — unload the LLM to free VRAM before image generation, then reload it after.
How do I avoid CUDA/ROCm out-of-memory errors when switching tasks?
Use idle-TTL unloading and an explicit unload-before-generate step so only one large model is resident at a time.
Does this need manual intervention?
No — llama-swap and the image tool handle the handoff automatically.