vLLM on Strix Halo — status, failures, and working alternatives

Not recommended hardware. Tokios does not recommend AMD Strix Halo (Ryzen AI Max+ 395) as a primary inference platform. The hardware is community-explored, not production-ready, and driver/ROCm support remains inconsistent. For reliable inference, use a Mac Studio, DGX Spark, or RTX workstation instead. This page is for users who already own Strix Halo hardware and want to understand what works.

Quick verdict: vLLM does not work reliably on Strix Halo

vLLM combined with PyTorch Flash Attention causes GPU hangs on the Radeon 8060S (gfx1151). The TheRock PyTorch builds that target this GPU are incompatible with vLLM’s CUDA-origin code paths, and the eager-mode workaround breaks tool calling. Community reports consistently confirm this across multiple kernel and ROCm versions. Use one of these alternatives instead:

Alternative	Backend	Status	Best for
llama.cpp	Vulkan	Works	Best performance, most control
Ollama 0.17.7	Vulkan/ROCm	Works (pinned)	Easiest setup
vLLM eager mode	ROCm	Partial — no tool calls	Not recommended

Why vLLM fails

Strix Halo’s iGPU is gfx1151 (RDNA 3.5). vLLM was designed for NVIDIA CUDA GPUs and later gained ROCm support for datacenter AMD GPUs (MI250, MI300X). The consumer iGPU in Strix Halo sits outside that supported matrix. The failure chain:

PyTorch TheRock builds — AMD ships TheRock builds for gfx1151, but these builds interact poorly with vLLM’s Flash Attention implementation.
Flash Attention + vLLM = GPU hang — when vLLM enables Flash Attention on gfx1151, the GPU hangs and the system becomes unresponsive, requiring a hard power cycle.
Eager mode workaround — disabling Flash Attention avoids the hang, but throughput drops significantly and tool calling breaks entirely.
Tool calls broken on AMD — even in eager mode, vLLM’s tool calling parser fails on AMD ROCm, which rules out coding agent use.

Recommended alternative: llama.cpp with Vulkan

llama.cpp compiled with the Vulkan backend is the most reliable and fastest option on Strix Halo. Vulkan delivers 21% faster token generation than ROCm on this hardware and works on both Linux and Windows.

Prerequisites

A Strix Halo system with 128GB unified memory
Vulkan drivers installed (included in Mesa on Linux, Adrenalin on Windows)
cmake, gcc/clang, and git

Build llama.cpp with Vulkan

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

On Windows, the same commands work in PowerShell (with Visual Studio or MinGW toolchain).

Launch the server

./build/bin/llama-server \
  -m /path/to/model-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999

The -ngl 999 flag offloads all layers to the GPU. Verify the server responds:

curl http://127.0.0.1:8080/v1/models

On Windows, set the AMD Adrenalin VRAM slider to 96GB before launching. Without this, the iGPU only sees ~15.5GB and large models will fail to load. See Strix Halo model compatibility for the full memory allocation guide.

Alternative: Ollama (pin to 0.17.7)

Ollama is the easiest path if you want ollama run simplicity. You must pin version 0.17.7 — all 0.18.x versions have a GPU detection regression on Strix Halo that causes Ollama to fall back to CPU-only inference.

Install and pin Ollama 0.17.7

# Download the specific release
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.17.7 sh

# Verify the version
ollama --version
# Expected: ollama version is 0.17.7

# Prevent auto-updates
sudo systemctl disable ollama-update.timer 2>/dev/null || true

Run a model

ollama run qwen3-coder:30b

Do not run ollama update or let a package manager upgrade Ollama past 0.17.7. The 0.18.x GPU detection regression is unresolved. If you accidentally upgrade, reinstall 0.17.7 from the GitHub releases page.

Workaround: vLLM eager mode (partial, not recommended)

If you have a specific reason to need vLLM’s API shape exactly, you can try eager mode. Expect degraded throughput and broken tool calling.

export HSA_OVERRIDE_GFX_VERSION=11.5.1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export VLLM_USE_FLASH_ATTN_TRITON=0

vllm serve your-model \
  --dtype float16 \
  --enforce-eager \
  --max-model-len 4096 \
  --trust-remote-code

Known limitations:

No Flash Attention — throughput is significantly lower than working setups
Tool calls broken — the structured output parser fails on AMD
Context length limited — memory pressure builds fast without paged attention
GPU hangs still possible — long contexts can still trigger hangs

Backend comparison summary

Factor	llama.cpp + Vulkan	Ollama 0.17.7	vLLM eager
Token generation speed	Fastest (21% over ROCm)	Good	Slowest
Setup difficulty	Build from source	1 command + pin	Complex env vars
Tool calling	Works	Works	Broken
Streaming	Works	Works	Works
OpenAI-compatible API	`/v1` on port 8080	`/v1` on port 11434	`/v1` on port 8000
Windows support	Yes (Vulkan)	Yes (Vulkan)	No (Linux only)
Stability	Good	Good (pinned)	Poor

Expose your model via Tokios

Once you have a working runtime (llama.cpp or Ollama), run the Tokios connector next to it. The connector dials out to https://api.tokios.com — no inbound ports on your box — and gives you a single authenticated endpoint.

# Start llama.cpp
./build/bin/llama-server -m model.gguf --port 8080

# Install and pair the Tokios connector
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

See the Strix Halo use-case page for the full connector pairing walkthrough.

If you are buying hardware today

Strix Halo’s 128GB unified memory sounds compelling on paper, but the software ecosystem is fragile. For reliable local LLM inference:

Platform	Memory	Bandwidth	Ecosystem
DGX Spark	128GB unified	273 GB/s	NVIDIA CUDA — mature, well-supported
Mac Studio M4 Ultra	192GB unified	819 GB/s	Metal — excellent, polished
RTX 4090 workstation	24GB VRAM	1008 GB/s	CUDA — fastest per-token, limited memory

Any of these will give you a smoother experience than Strix Halo for the same or less effort.

Next steps

Strix Halo model compatibility

Which models fit in 128GB and at what quantization.

Strix Halo use case

Full connector pairing walkthrough for Strix Halo.

DGX Spark use case

The recommended alternative with 128GB unified memory.

vLLM remote access

Expose vLLM on supported NVIDIA hardware securely.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

vLLM on Strix Halo — status, failures, and working alternatives

Quick verdict: vLLM does not work reliably on Strix Halo

Why vLLM fails

Recommended alternative: llama.cpp with Vulkan

Prerequisites

Build llama.cpp with Vulkan

Launch the server

Alternative: Ollama (pin to 0.17.7)

Install and pin Ollama 0.17.7

Run a model

Workaround: vLLM eager mode (partial, not recommended)

Backend comparison summary

Expose your model via Tokios

If you are buying hardware today

Next steps

Strix Halo model compatibility

Strix Halo use case

DGX Spark use case

vLLM remote access

​Quick verdict: vLLM does not work reliably on Strix Halo

​Why vLLM fails

​Recommended alternative: llama.cpp with Vulkan

​Prerequisites

​Build llama.cpp with Vulkan

​Launch the server

​Alternative: Ollama (pin to 0.17.7)

​Install and pin Ollama 0.17.7

​Run a model

​Workaround: vLLM eager mode (partial, not recommended)

​Backend comparison summary

​Expose your model via Tokios

​If you are buying hardware today

​Next steps

Strix Halo model compatibility

Strix Halo use case

DGX Spark use case

vLLM remote access

Quick verdict: vLLM does not work reliably on Strix Halo

Why vLLM fails

Recommended alternative: llama.cpp with Vulkan

Prerequisites

Build llama.cpp with Vulkan

Launch the server

Alternative: Ollama (pin to 0.17.7)

Install and pin Ollama 0.17.7

Run a model

Workaround: vLLM eager mode (partial, not recommended)

Backend comparison summary

Expose your model via Tokios

If you are buying hardware today

Next steps