This project provides pre-built containers (“toolboxes”) for running LLMs on AMD Ryzen AI Max “Strix Halo” integrated GPUs. Toolbx is the standard developer container system in Fedora (and now works on Ubuntu, openSUSE, Arch, etc).

📺 Video Demo

✅ Stable Configuration

OS: Fedora 42/43
Linux Kernel: 6.18.6-200
Linux Firmware: 20260110

This is currently the most stable setup. Kernels older than 6.18.4 have a bug that causes stability issues on gfx1151 and should be avoided. Also, do NOT use linux-firmware-20251125. It breaks ROCm support on Strix Halo (instability/crashes).

✅ ROCm 7 Performance Regression Workaround Applied — 2026-02-04

The performance regression previously observed in ROCm 7+ builds (compared to ROCm 6.4.4) has been resolved in the toolboxes via a workaround.

The issue was caused by a compiler regression (llvm/llvm-project#147700) affecting loop unrolling thresholds. We have applied the workaround (-mllvm --amdgpu-unroll-threshold-local=600) in the latest toolbox builds, restoring full performance.

This workaround will be removed once the upstream fix lands. For details, see the issue: kyuz0/amd-strix-halo-toolboxes#45

📦 Supported Toolboxes

You can check the containers on DockerHub: kyuz0/amd-strix-halo-toolboxes.

Container Tag	Backend/Stack	Purpose / Notes
`vulkan-amdvlk`	Vulkan (AMDVLK)	Fastest backend—AMD open-source driver. ≤2 GiB single buffer allocation limit, some large models won't load.
`vulkan-radv`	Vulkan (Mesa RADV)	Most stable and compatible. Recommended for most users and all models.
`rocm-6.4.4`	ROCm 6.4.4 (Fedora 43)	Latest stable 6.x build. Uses Fedora 43 packages with backported patch for kernel 6.18.4+ support.
`rocm-7.2`	ROCm 7.2	Latest stable 7.x build. Includes patch for kernel 6.18.4+ support.
`rocm7-nightlies`	ROCm 7 Nightly	Tracks nightly builds. Includes patch for kernel 6.18.4+ support.

These containers are automatically rebuilt whenever the Llama.cpp master branch is updated. Legacy images (rocm-6.4.2, rocm-6.4.3, rocm-7.1.1) are excluded from this list.

🚀 Quick Start

1. Create & Enter Toolbox

Option A: Vulkan (RADV/AMDVLK) - best for compatibility

toolbox create llama-vulkan-radv \
  --image docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv \
  -- --device /dev/dri --group-add video --security-opt seccomp=unconfined

toolbox enter llama-vulkan-radv

Option B: ROCm (Recommended for Performance)

toolbox create llama-rocm-7.2 \
  --image docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-7.2 \
  -- --device /dev/dri --device /dev/kfd \
  --group-add video --group-add render --group-add sudo --security-opt seccomp=unconfined

toolbox enter llama-rocm-7.2

(Ubuntu users: use Distrobox as toolbox may break GPU access).

2. Check GPU Access

Inside the toolbox:

llama-cli --list-devices

3. Download Model

Example: Qwen3 Coder 30B (BF16)

HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
  BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  --local-dir models/qwen3-coder-30B-A3B/

4. Run Inference

⚠️ IMPORTANT: Always use flash attention (-fa 1) and no-mmap (--no-mmap) on Strix Halo to avoid crashes/slowdowns.

Server Mode (API):

llama-server -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  -c 8192 -ngl 999 -fa 1 --no-mmap

CLI Mode:

llama-cli --no-mmap -ngl 999 -fa 1 \
  -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  -p "Write a Strix Halo toolkit haiku."

5. Keep Updated

Refresh your authenticated toolboxes to the latest nightly/stable builds:

./refresh-toolboxes.sh all

📊 Performance Benchmarks

🌐 Interactive Viewer: https://kyuz0.github.io/amd-strix-halo-toolboxes/

See docs/benchmarks.md for full logs.

💾 Memory Planning & VRAM Estimator

Strix Halo uses unified memory. To estimate VRAM requirements for models (including context overhead), use the included tool:

gguf-vram-estimator.py models/my-model.gguf --contexts 32768

See docs/vram-estimator.md for details.

🛠️ Building Locally

You can build the containers yourself to customize packages or llama.cpp versions. Instructions: docs/building.md.

⚙️ Host Configuration

Test Machine: Framework Desktop / Mini Server

CPU: Ryzen AI MAX+ 395 "Strix Halo"
RAM: 128 GB
OS: Fedora 43 (Kernel 6.18-rc6)

Kernel Parameters

Add these to GRUB_CMDLINE_LINUX to manage unified memory (reserved 4GB for OS): amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856

Apply with:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

Ubuntu 24.04

See TechnigmaAI's Guide.

🌩️ Distributed Inference

Run models across a cluster of Strix Halo machines using run_distributed_llama.py.

Setup SSH keys between nodes.
Run python3 run_distributed_llama.py on the main node.
Follow the TUI to launch the cluster.

README.md

AMD Strix Halo Llama.cpp Toolboxes