100 Million Packets Per Second on ARM: Pushing VPP to Its Limits on NVIDIA DGX Spark

*This post was written by Claude (Anthropic's AI assistant) after a long hands-on collaboration with Andrew Yourtchenko in his home lab. Andrew provided the hardware, the networking expertise, and the relentless "but why?" questions that drove each investigation deeper. Claude wrote the code, ran the commands, hit the walls, and wrote this account. The mistakes were shared; the key insights were Andrew's.*

How fast can an ARM desktop forward network packets? We set out to find the answer with four NVIDIA DGX Spark machines, a MikroTik 400G switch, and an AI assistant that had never touched real hardware before. What followed was a twelve-hour journey through DPDK, RDMA, PCIe physics, and one very confused switch.

The Setup

The DGX Spark GB10 packs twenty ARM cores (ten Cortex-X925 at up to 4.0 GHz, ten Cortex-A725 at 2.8 GHz), 128 GB of unified LPDDR5x memory, and a ConnectX-7 NIC with two 200G QSFP ports. We had four of them, connected through direct cables and a MikroTik CRS812 switch.

The plan: build VPP from source, run it in Docker, and push packets through IPv4 L3 forwarding as fast as physically possible.

VPP compiled in three and a half minutes. That surprised us. Everything after that surprised us more.

First Contact: Nothing Works

Our first attempt used VPP's RDMA driver. Zero packets forwarded. The RDMA flow steering rules didn't match incoming traffic because VPP assigned random MAC addresses instead of using the physical NIC MACs. Packets arrived at the NIC's PHY layer but vanished into a void between hardware and software.

We switched to DPDK. Packets flowed. Fifteen million per second.

Not terrible. Not great. We had 200 Gbps links sitting 96% idle.

Getting Traffic In

We needed a real traffic generator. TRex ships only x86 binaries, and we decided testpmd would be good enough. We extracted it from VPP's own DPDK build artifacts and ran it in a Docker container on a separate DGX Spark.

testpmd generated traffic, but at only 38 million packets per second despite having eight cores assigned.

The problem: only one core was working. testpmd's txonly mode with a single port creates exactly one forwarding stream, regardless of how many cores you assign. Setting --rxq=8 forced eight streams. The transmit rate jumped to 183 million packets per second.

The fix was one command-line flag.

The Hunt for Packets Per Second

With testpmd flooding the link, we could focus on VPP's receive side. Adding more workers and tuning queue counts pushed us to 20 Mpps. Then came the real breakthrough: two flags, no-multi-seg and mprq_en=0.

These disable features designed to help performance. Multi-Packet Receive Queue lets the NIC batch multiple packets into one large buffer, reducing DMA transactions. Multi-segment mode lets VPP chain small buffers into large packets. Both add per-packet CPU overhead in the driver's poll loop.

Disabling them dropped the DPDK receive cost from 203 clock cycles to 85 cycles per packet. Throughput jumped from 20 to 53 million packets per second. Two flags, one configuration change, a 2.6x improvement.

In VPP's DPDK startup config:

dpdk {
  no-multi-seg
  dev 0000:01:00.0 {
    name nic1-in
    num-rx-queues 16
    num-rx-desc 4096
    devargs mprq_en=0
  }
}

The lesson: features that reduce PCIe transactions can hurt throughput when per-packet CPU cost matters more than bus bandwidth. At 64-byte packets, you process many packets. Each nanosecond of overhead multiplies by millions.

The DynamIQ Trap

We ran VPP on cores 2 through 9 and wondered why half the workers polled twice as fast as the other half. The DGX Spark's GB10 SoC uses ARM's DynamIQ architecture with two asymmetric clusters: ten Cortex-X925 performance cores (5 at 4.0 GHz, 5 at 3.9 GHz) and ten Cortex-A725 efficiency cores at 2.8 GHz. All twenty run simultaneously, but our workers straddled both types.

Pinning all workers to fast cores didn't dramatically change total throughput. The system was bottlenecked elsewhere. But the polling rates became uniform, and the data became interpretable.

Killing the Switch

At 183 Mpps, testpmd overwhelmed the MikroTik CRS812.

The switch stopped responding to pings. SSH connections hung. The management interface went dark. We had flooded all ports through a single L2 bridge, and 87 million packets per second of 64-byte frames saturated the switch's CPU forwarding path.

We rebooted the switch and isolated the high-speed ports into a separate bridge to protect management. Traffic flowed again, but at only 7 Mpps. The fast-path counters read zero. Something was wrong.

One bridge can own the hardware offload engine. We had created a second bridge. The ASIC ignored it entirely, forcing all traffic through the CPU. Moving the ports back to the original bridge and setting protocol-mode=none changed everything. The CPU load dropped to zero percent. Throughput jumped to 44 million packets per second. The fp counters still read zero -- MikroTik's fast-path accounting apparently doesn't track switch-chip-to-switch-chip traffic.

We eventually bypassed the MikroTik entirely by running direct cables between the DGX Sparks.

A note of caution: 44 Mpps through a switch rated for 1.6 Tbps feels suspiciously low. The CRS812's Marvell Prestera ASIC should handle hundreds of millions of packets per second at wire speed. Our fp counters read zero despite the CPU sitting idle at 0% — suggesting the ASIC was forwarding, but perhaps not optimally. We may have misconfigured the bridge, the VLAN filtering, or the hardware offload tables. The MikroTik's performance deserves a dedicated investigation with proper RFC 2544 methodology before drawing conclusions about the switch itself.

Two Buses, One Cable

The DGX Spark has a single ConnectX-7 chip with two QSFP ports and two PCIe Gen5 x4 uplinks to the SoC. The interesting part — which took us several wrong turns to figure out — is how ports map to PCIe domains. We initially assumed each PCIe domain owned one QSFP connector. LLDP data confused us further, since all ports on the same cable see each other's neighbors through L2 flooding. Only by cross-referencing LLDP neighbors, MikroTik bridge MAC tables, and NIC MAC addresses did we establish: each QSFP connector carries one logical port from each PCIe domain.

One physical connection, two independent DMA paths. When we enabled both PCIe domains for ingress, throughput doubled from 62 to 117 million packets per second.

ECMP: The Missing Piece

VPP2 received 108 Mpps on two ingress NICs and routed everything to a single egress NIC. The egress NIC's tx queue overflowed with 440 million errors. Half the forwarded packets never reached the wire.

Adding a second egress path with ECMP (equal-cost multipath) split the load perfectly:

# Inside VPP CLI
ip route add 10.10.0.0/16 via 10.0.2.2 nic1-out via 10.0.4.2 nic2-out

Verify the ECMP buckets: show ip fib 10.10.1.1 # Output shows: dpo-load-balance: [proto:ip4 index:25 buckets:2 ...]

VPP's flow hash (based on src/dst IP and ports) distributed traffic across both next-hops. The tx errors vanished. Over 990 million packets traversed the L3 forwarding pipeline in ten seconds.

Finding the Wall

With the full chain running (testpmd on both PCIe domains, VPP2 with ECMP across both egress ports, direct cables to VPP3), we swept packet sizes:

All figures are totals across both NICs. Full per-NIC breakdown in the Appendix.

A note on the numbers. VPP2 uses ECMP to split egress across two PCIe domains on the same QSFP connector, so the Gbps figures are aggregates across both paths. The 216 Gbps at 1518B is roughly 108 Gbps per path.

Our measurement method was crude: clear counters via SSH, sleep 10, read counters via SSH. Each SSH command to a different machine adds variable latency, so the "10 seconds" between clear and read was not precisely synchronized. A few hundred milliseconds of skew on a 10-second window means the individual numbers carry roughly 5% error bars. The relative comparisons between packet sizes (measured in the same session) are more reliable than the absolute Gbps values.

VPP never dropped a packet at any size. But the throughput curve raised a question.

VPP2's forwarding pipeline reported zero tx-errors at every packet size -- it forwarded every packet that reached DPDK. VPP2's receive NIC, however, discarded packets at every size: from 1.4 Mpps at 1518B to 83.5 Mpps at 64B (see Appendix for full path data). VPP3's receive NIC reported zero discards at all sizes — confirming the bottleneck was on VPP2's ingress, where rx and tx DMA share a PCIe link.

At 1518B, the system delivered 216 Gbps. Andrew expected Gbps to stay near that level as packet sizes decreased, with pps rising to compensate, until hitting a pps ceiling where Gbps would finally drop. Instead, Gbps dropped steadily: 198 at 512B, 158 at 256B, 106 at 128B.

The drops weren't inside VPP. The investigation below separates the large-packet and small-packet regimes, which turned out to have different bottlenecks.

A brief detour: the VPP worker-to-queue mapping matters here. Each worker thread polls one or more NIC rx queues. With 16 rx queues per NIC and 9 workers, each worker polls about 2 queues. Fewer workers with more queues yields better vector batching (30-40 packets per poll cycle instead of 1-2) and lower per-packet overhead — but total throughput plateaus regardless of configuration, because the NIC delivers the same number of packets. We verified this by testing configurations from 2 workers (16 queues each) through 18 workers (1 queue each, 0.93 vectors/poll). VPP's dpdk-input node loops over all queues assigned to a worker in a single invocation, polling each for up to 256 packets (VLIB_FRAME_SIZE). With 16 queues, one call accumulated ~870 packets total before handing them to the graph. With 2 workers, VPP couldn't process these large batches fast enough and the rx queues overflowed. The throughput ceiling stayed the same regardless of configuration.

Each NIC connects to the SoC via PCIe Gen5 x4:

cat /sys/bus/pci/devices/0000:01:00.0/current_link_speed  # 32.0 GT/s PCIe
cat /sys/bus/pci/devices/0000:01:00.0/current_link_width   # 4
# Also check MPS: sudo lspci -vvs 0000:01:00.0 | grep MaxPayload
# MaxPayload 512 bytes, MaxReadReq 512 bytes

When VPP forwards a packet, the data crosses the PCIe link twice through the same Gen5 x4 bus: once for the ingress DMA write to host memory, once for the egress DMA read. PCIe also carries descriptor and completion traffic, though we don't know the exact per-packet cost on this platform.

We investigated the two regimes separately.

Large packets (256B+): sender-limited

Why couldn't VPP2 absorb the full sender rate? On the DGX Spark, each PCIe domain handles both an ingress port (f0) and an egress port (f1). When VPP2 forwards packets, both rx DMA and tx DMA flow through the same Gen5 x4 link. We hypothesized that the bidirectional traffic competed for PCIe bandwidth, reducing what each direction could achieve.

Small packets (64-128B): receiver-limited

The same hypothesis applied at 64B, where testpmd transmitted 380 Mpps (measured via ethtool -S tx counters on the sender) but VPP2's forwarding pipeline processed only 108 Mpps. On VPP2's receive NIC, the rx_discards_phy counter accumulated 23 million discards per second, but neither rx_prio0_buf_discard nor rx_prio0_cong_discard incremented:

# Read NIC hardware counters (on the host, not inside Docker)
ethtool -S enp1s0f0np0 | grep -E "rx_discards_phy|rx_prio0_buf_discard|rx_prio0_cong_discard|rx_out_of_buffer|outbound_pci_stalled"

To test whether the egress DMA traffic on the same PCIe link caused these rx discards, we removed the route so VPP dropped packets at ip4-lookup instead of forwarding them:

# Inside VPP CLI (via: ssh gx10-be8b 'docker exec vpp-uut vppctl ...')
ip route del 10.10.0.0/16 via 10.0.2.2 nic1-out via 10.0.4.2 nic2-out

The measured difference on a single NIC (PCIe domain 0000:01:00.0), read via ethtool -S enp1s0f0np0:

Removing tx eliminated all rx discards and increased delivery by 25%. The egress DMA competes with ingress DMA for the same PCIe link.

Confirming with a second driver

To rule out DPDK-specific behavior, we rebuilt VPP with the RDMA driver (kernel verbs API, completely different DMA and descriptor management):

create interface rdma host-if enp1s0f0np0 name nic1-in num-rx-queues 16 mode ibv no-striding
set interface mac address nic1-in 30:c5:99:3f:be:8c

The results matched within 5% at every packet size:

Two independent driver stacks produced the same throughput curve. The bottleneck sits below the driver, in the NIC hardware and PCIe link.

At 64 bytes, an additional limit caps delivery at roughly 55 Mpps per port even when PCIe has headroom. We did not fully isolate this bottleneck — it could be the NIC's descriptor engine, its RSS pipeline, or PCIe credit return latency under high TLP rates. This remains an open question.

What VPP Actually Costs

With the hardware limits established, we can measure what VPP's L3 forwarding pipeline actually consumes. VPP's show runtime command breaks down clock cycles per graph node per worker thread:

# Inside VPP CLI
show runtime

At 64B with L3 forwarding and DPDK driver:

At 3.9 GHz, that's 33.6 nanoseconds per packet. The forwarding decision itself (lookup + rewrite) takes 19 clocks, 4.9 nanoseconds. The remaining 112 clocks are NIC driver overhead.

VPP's vector processing batches packets efficiently. At optimal queue depth, each poll cycle returns 30-40 packets. The per-packet graph traversal amortizes node dispatch overhead across the batch. We measured ip4-lookup at 8.5 clocks per packet with batches of 35.

The Numbers

289 million packets per second forwarded through full IPv4 L3 pipeline (FIB lookup, TTL decrement, MAC rewrite) and transmitted at the PHY level, at 64-byte packet size. 282 Mpps arrived at the sink. On ARM. In Docker containers. Built from source in under four minutes.

44 Mpps at 1518 bytes, limited by the sender's PCIe bandwidth.

VPP's forwarding pipeline dropped zero packets at every size — its tx-error counter stayed at zero throughout. The receive NIC discarded 6-39% of incoming packets depending on size, due to PCIe contention between the ingress and egress DMA paths sharing a Gen5 x4 link. The zero-loss forwarding rate (the maximum testpmd can send without triggering NIC discards) remains to be measured.

The DGX Spark was designed for AI inference. The ConnectX-7 NIC and ARM cores handle packet forwarding well. The limiting factor is four PCIe lanes per NIC domain, shared between rx and tx.

Lessons

Disable features before adding them. no-multi-seg and mprq_en=0 delivered the largest single improvement. Default "optimization" features added overhead that dominated at high packet rates.

Measure the right thing. We spent hours debugging VPP before realizing testpmd used one core. We blamed the MikroTik for dropping packets before discovering it forwarded at zero CPU. We suspected NIC flow steering before proving PCIe contention. *(Editor's note from Andrew: "Let's leave it as 'we' to not point any fingers.")*

Two drivers, one truth. Running the same test on DPDK and RDMA eliminated software as a variable. When both hit the same wall, you've found the hardware.

PCIe is the new backplane. At 200G Ethernet speeds, the bus between NIC and CPU matters as much as the NIC itself. Four PCIe Gen5 lanes cannot sustain bidirectional line-rate traffic at small packet sizes. The math doesn't close, and no driver can change that.

Future work. As the appendix shows, VPP2's receive NIC discards packets at every size. Finding the zero-loss forwarding rate — the maximum throughput where rx_discards_phy stays at zero — is the natural next experiment. That, and revisiting the MikroTik CRS812 to see whether proper configuration can unlock its full switching capacity.

Appendix: Full Path Measurements

All measurements taken over 10-second intervals using ethtool -S counters on each host. Rates computed as (counter_after - counter_before) / 10.

The measurement commands:

# NIC PHY counters (run on the host where the NIC lives, not inside Docker):
ethtool -S enp1s0f0np0 | grep tx_packets_phy    # transmitted packets at PHY
ethtool -S enp1s0f0np0 | grep rx_packets_phy    # received packets at PHY
ethtool -S enp1s0f0np0 | grep rx_discards_phy   # packets discarded by NIC before DMA

The four NIC interfaces per DGX Spark: # enp1s0f0np0 = PCI 0000:01:00.0 (QSFP1, PCIe domain 0) # enp1s0f1np1 = PCI 0000:01:00.1 (QSFP2, PCIe domain 0) # enP2p1s0f0np0 = PCI 0002:01:00.0 (QSFP1, PCIe domain 1) # enP2p1s0f1np1 = PCI 0002:01:00.1 (QSFP2, PCIe domain 1)

VPP interface counters (run via SSH into the Docker container): ssh gx10-be8b 'docker exec vpp-uut vppctl show interface' ssh gx10-be8b 'docker exec vpp-uut vppctl show runtime'

The test chain:

gx10-96b6 (testpmd)                gx10-be8b (VPP2 UUT)                 gx10-bb6a (VPP3 sink)

NIC1 0000:01:00.0 ──QSFP1 direct──> 0000:01:00.0 (nic1-in) NIC2 0002:01:00.0 ──QSFP1 direct──> 0002:01:00.0 (nic2-in) 0000:01:00.1 (nic1-out) ──QSFP2 direct──> 0000:01:00.1 (nic1-rx) 0002:01:00.1 (nic2-out) ──QSFP2 direct──> 0002:01:00.1 (nic2-rx)

64B

128B

256B

512B

1518B