Fedora does not currently have prebuilt packages for the brand-spanking-new (like, within the last week) completed kernel + userspace NPU stack for Linux. It revolves around the xdna-driver for the kernel driver, XRT for the userspace libraries, FastFlowLM for the userspace program that does the actual running of LLMs, and finally Lemonade for the orchestrator and frontend which can serve ollama and OpenAI compatible endpoints for every AMD backend out there. (This also makes running models on the GPU with a ROCm backend very easy, a super welcome move from AMDs… Troubled previous years.)

As far as I know, I’m the first person to get everything built from source, all integrated together and working on Fedora. I’m running this on my 128GB Framework Desktop, which, amusingly, has appreciated $700 and counting since I got my first-batch preorder in August.

My feelings about LLMs are ambivalent. Personally, I suspect I won’t use this very much; it’s more useful for the researchers who I’m serving LLMs to for their NLP analysis. Hopefully, the extra compute will speed up parallel processing on their datasets on top of the GPU working as hard as it can.

Also, I wanted this all set up and working because the NPU is a fascinating piece of computer architecture, and I’m hoping to do some very silly things with it soon.

Considering how brand new all of this is, I’m mostly posting this for posterity, as well as hopefully a guide someone smarter than me can use to get everything packaged up with a nice bow tie for end users. I’ll update the post when that day arrives; until then, good luck, and have fun! It shouldn’t take more than half an hour now that I’ve laid out all the steps for you.

git clone --recursive https://github.com/amd/xdna-driver.git
git clone --recursive https://github.com/FastFlowLM/FastFlowLM.git

Open this file:
xdna-driver/xrt/src/runtime_src/tools/scripgs/xrtdeps.sh
Find the FD_LIST variable. Install all of those packages. DO NOT execute the script, it doesn't work.

if you need rocm… There's a collision between the OpenCL package it needs and the one XRT needs.. I just gave up and removed it.

package OpenCL-ICD-Loader-3.0.6-4.20241023git5907ac1.fc43.x86_64 from fedora conflicts with ocl-icd provided by ocl-icd-2.3.4-1.fc43.x86_64 from fedora

Build the kernel driver:
cd xdna-driver/build
./build.sh -release -j 32

Build the userspace libs:
cd ../xrt/build/
./build.sh -npu -opt -noctest -j 32

The following will install all the userspace libs you just built, as well as the kernel driver you built a second ago:
sudo dnf install Release/*.rpm ../../build/Release/xrt_plugin.2.23.0_43-x86_64-amdxdna.rpm

Get the dependencies for FastFlowLM:
(Note that you can also use ffmpeg-devel with RPMFusion enabled due to Fedora's silly patent issues. Also, you should probably install Rust and Cargo through Rustup instead.)
sudo dnf install ninja-build ffmpeg-free-devel fftw-devel rust cargo

Configure, build, and install as described in the README: https://github.com/FastFlowLM/FastFlowLM?tab=readme-ov-file

Set memory limits for the NPU device:

sudo tee /etc/security/limits.d/99-amdxdna.conf > /dev/null << 'EOF'

soft memlock unlimited

hard memlock unlimited
EOF

Make sure everything can see the XRT libraries:

sudo tee /etc/ld.so.conf.d/flm.conf > /dev/null << 'EOF'
/opt/fastflowlm/lib
/opt/xilinx/xrt/lib64
EOF

sudo ldconfig

Reboot

Run this to make sure everything works:

flm validate

Install lemonade per here: https://lemonade-server.ai/install_options.html#linux
Lemonade should pick up on the presence of flm and make NPU models available to download and run. Enjoy!