Agave XDP on Broadcom: two startup panics and how we fixed them
Our validator panicked twice while we were switching on XDP. Here are the exact error messages, what they mean and the fixes that have kept the node stable for weeks.
Updated November 20, 2026
This guide is for validator operators. If you are here to stake SOL, the beginner guides are a better place to start.
Since Agave 3.0.9, XDP is the recommended way to send Turbine shreds: packets skip most of the Linux network stack. Turning it on is a few flags in theory. In practice our validator crashed twice before it ran. This is the story of those two crashes on a Broadcom BCM57502 card (driver bnxt_en), with the exact errors so you can find this page when you hit them.
Our setup: Jito-Agave 4.4.x, AMD Ryzen 9 9950X, Ubuntu 24.04 with kernel 6.8, Broadcom BCM57502 with firmware 231.0.154.0.
Crash 1: setsockopt(SOL_XDP, ring=6, size=1022) ... EINVAL
Right after start, the validator panicked:
WARN agave_xdp::tx_loop] huge page alloc failed, falling back to regular page size
ERROR agave_xdp::tx_loop] failed to create AF_XDP TX socket for queue QueueId(0) on CPU 1:
setsockopt(SOL_XDP, ring=6, size=1022) failed: Invalid argument (os error 22)
panicked at validator/src/commands/run/execute.rs:410:30: failed to create xdp transmitter
What it means. AF_XDP uses shared ring buffers between the kernel and the program, and their size must be a power of two: 512, 1024, 2048. Socket option 6 is the "completion ring". Agave sized it from the network card's own ring, and the card's default produced 1022. That is not a power of two, so the kernel refused.
The fix is to set the card's rings to a power of two:
ethtool -G enp2s0f1np1 rx 1024 tx 1024
ethtool -g enp2s0f1np1 # check "Current hardware settings"
On this card the maximum TX ring is 2047, so 1024 is the largest power of two that fits. After that the error was gone.
Don't be misled by the WARN line. huge page alloc failed looks like the cause, but it is only a warning: Agave falls back to regular memory pages. Our node runs with zero hugepages reserved. The real problem is always the next ERROR line.
Crash 2: bind(AF_XDP, ... flags=0xc) failed: Operation not supported
Next we tried to squeeze out more speed with --xdp-zero-copy:
bind(AF_XDP, ifindex=3, queue=0, flags=0xc) failed: Operation not supported (os error 95)
What it means. flags=0xc is two flags combined: XDP_ZEROCOPY and XDP_USE_NEED_WAKEUP. "Operation not supported" is the driver saying it cannot do zero-copy. This is a limitation of the bnxt_en driver in current kernels, not a broken setup. You can check any card in advance:
xdp-loader features enp2s0f1np1 | grep ZEROCOPY
# NETDEV_XDP_ACT_XSK_ZEROCOPY: no
The fix is to remove --xdp-zero-copy. In normal copy mode, Broadcom works well: our node has run for weeks without a single XDP error.
The trap after a reboot
ethtool settings are lost on reboot. If you forget this, everything works until the next server restart, and then the validator hits crash 1 again, often at night during an automatic update reboot.
We made the setting permanent with a small systemd service that runs before the validator. You can copy it from our GitHub repository.
How to tell XDP is actually working
ip linkwill not show an XDP program. That is normal: Agave uses AF_XDP for sending only.- The reliable sign is this line in the validator log, with no ERROR after it:
starting xdp loop on enp2s0f1np1 queue QueueId(0) cpu 1 - If XDP setup fails, the validator panics immediately. A node that keeps running after that log line has XDP active.
- Watch the driver counters:
ethtool -S <if> | grep -iE 'xsk|xdp|alloc_err'should stay at zero.
Checklist
- Check zero-copy support with
xdp-loader features. If you seeno, don't use--xdp-zero-copy. - Set NIC rings to a power of two (
ethtool -G <if> rx 1024 tx 1024). - Make the setting persistent with a systemd unit.
- Give XDP its own core, and remember that core's SMT sibling (
thread_siblings_list). - Check the log line and the
ethtool -Scounters after start.
Full technical guide
We collected everything in an open repository: the error → cause → fix table, notes on Intel and Mellanox cards from other operators, the systemd unit and the counters to watch. We also sent our findings to the community hardware list, Solana HCL.
- Guide: github.com/dragon-dev7/agave-xdp-nic-guide
- NIC compatibility list: solanahcl.org
Did you hit a different error on your card? Send it via the contact form or open an issue on GitHub, and we'll add it to the guide.