aarch64: softirq boot-test flake under host starvation (~1%, pre-existing)

[ryanbreen]

Summary

A ~1% aarch64 boot-test flake (softirq_aarch64) was observed during PR-3 (#470 F4 x86 free-path
follow-up) starved-gate stress testing (14 host yes hogs, 100-run reproof). Structural and
historical analysis shows this is a pre-existing, branch-independent race in the softirqd
subsystem, not a regression. It joins a documented family of pre-existing aarch64 EarlyBoot-suite
observational flakes (#516, #519).

The failure

Branch under test: fix/470-pr3-x86-free-path @ b5ca64d4, run starved-100-reproof/fail_077 (1 of
100 starved runs).

[TEST:interrupts:softirq_aarch64:START]
[TEST:interrupts:softirq_aarch64:FAIL:raise_softirq did not set pending bit on ARM64]
...
[SUBSYSTEM:interrupts:early:COMPLETE:7/8]

Boot itself otherwise completed cleanly: 4/4 CPUs online, userspace ran, and every #470 PR-3
custody / retire / reclaim oracle in the same run PASSED. This was the only failure in the run.

Test source: kernel/src/test_framework/registry.rs:3974 (test_softirq_aarch64), registered at
registry.rs:5400 as TestStage::EarlyBoot, i.e. it runs in one of the test executor's parallel
kthreads with interrupts unmasked. The failing assertion is exactly:

const TEST_SOFTIRQ: u32 = 5;
per_cpu_aarch64::raise_softirq(TEST_SOFTIRQ);           // set bit 5 in this CPU's pending mask
let after_raise = per_cpu_aarch64::softirq_pending();   // read the mask back
if (after_raise & (1 << TEST_SOFTIRQ)) == 0 {
    return TestResult::Fail("raise_softirq did not set pending bit on ARM64");
}

Mechanism — the assertion is not an invariant

Between the raise and the read the kthread is fully preemptible and interruptible. Two independent,
pre-existing mechanisms can legitimately clear or hide bit 5 in that window:

(a) IRQ-exit softirq drain (primary). Every aarch64 IRQ return runs
crate::task::softirqd::do_softirq() (kernel/src/arch_impl/aarch64/exception.rs:1954). do_softirq
(kernel/src/task/softirqd.rs:169-210) takes the whole pending mask and, for each set bit, calls
per_cpu::clear_softirq(nr) before looking for a handler — an unregistered softirq number is
cleared just the same. NR_SOFTIRQS = 10 (softirqd.rs:40), so the test's bit 5 is inside the
drained range. One timer tick, virtio-blk completion, or net IRQ landing in the two-instruction
window erases the bit the test is about to read. The failing serial shows exactly that concurrent
traffic: virtio_blk_sequential_read in flight, timer_delay running (irqs=4 slices=57), and a
NetRx softirq already pre-primed at boot (NET: pre-primed NetRx softirq for bootstrap callback re-enable), which guarantees do_softirq finds pending != 0 and enters the drain loop.

(b) Cross-CPU migration (secondary). softirq_pending reads this CPU's per-CPU data. With 4
CPUs online and a preemptible kthread, a raise on CPU A followed by a reschedule and a read on CPU B
observes B's mask — no bit 5.

Why starvation, and why ~1%. Under host yes hogs, QEMU's vCPU thread can be descheduled inside
the window for milliseconds. The guest instruction stream stops after raise_softirq while GIC
interrupts accumulate; on resume the CPU takes the pending IRQ(s) before executing the read. Host
descheduling stretches an otherwise ~10 ns window by 5-6 orders of magnitude — exactly the observed
profile (100/100 clean, 1/100 starved). The stretch factor is a property of the host scheduler, not
the guest's code.

History — both the test and the racing path predate the branch by ~6 months

Element	Introduced	Commit
test_softirq_aarch64 (this exact assertion + failure string)	2026-01-28	80da9d97 arm64: complete remaining parity tasks (#33-37)
softirqd subsystem incl. clear-before-handler drain	2026-01-21	edf9186f feat(softirq): add Linux-style softirq subsystem
do_softirq() on the aarch64 IRQ-exit path	2026-02-06	0668208d feat(arm64): interrupt-driven networking with softirq processing

The last edit to the test body was 6f3ebd2e (2026-01-29), a warning fix. Nothing in the #470 PR-3
campaign touched it.

Branch-independence — every implicated path is byte-identical to main

git diff --stat main...b5ca64d4 -- \
  kernel/src/arch_impl kernel/src/per_cpu_aarch64.rs kernel/src/task/softirqd.rs \
  kernel/src/net kernel/src/drivers/virtio/net_mmio.rs kernel/src/drivers/virtio/net_pci.rs \
  kernel/src/task/kthread.rs kernel/src/test_framework/executor.rs
   -> (empty)

No file under kernel/src/arch_impl/ changed at all: the aarch64 exception vector, GIC handling, IRQ
enter/exit, timer interrupt, per-CPU softirq storage, softirqd, the NAPI/net RX path, the kthread
implementation, and the test executor are all identical to main. A full-diff grep for
softirq|gic|napi|irq_exit|irq_enter|daif|tick returns no hit in any aarch64-reachable production
path touched by PR-3. Every branch change reachable at all during the EarlyBoot softirq test (frame
allocator custody bookkeeping, virtio ring contiguous allocation, x86-only scheduler/registry changes
compiled out on aarch64, procfs custody-summary emitted after BOOT_TESTS_COMPLETE) was individually
ruled out as a mechanism — none changes IRQ rate, IRQ semantics, or the softirq mask.

Prior sightings — part of a documented flake family

No prior sighting of this exact signature exists in preserved logs from earlier #470 campaign phases
(PR-1a, PR-2 preserved serials all show softirq_aarch64:PASS), but the same aarch64 EarlyBoot suite
has a documented population of pre-existing observational flakes of exactly this class:

• Flaky (pre-existing): aarch64 timer_quantum_reset_aarch64 / arm64_socket_reset_quantum ~4% at zero load #516 (CLOSED) — timer_quantum_reset_aarch64 / arm64_socket_reset_quantum, ~4/100 at zero
  load, explicitly "pre-existing on main"; resolution guidance was "prefer fixing the test assertion
  robustness".
• Flaky (boot-infra): rare PSCI CPU_ON 3-of-4 CPU bring-up miss fails exit_kick_protocol_gate (~1% zero-load) #519 (CLOSED) — PSCI CPU_ON bring-up miss, ~1% at zero load.
• The same PR-3 branch's earlier starved failure (current_thread_exists) was structurally proven to
  assert an invariant the aarch64 boot path never establishes; its oracle was rewritten in-branch.

softirq_aarch64 is the fourth member of the same family.

Statistical context

1/100 alone establishes little about attribution (95% CI 0.03%-5.4%). The main comparator's 0/100
starved run (same host, same 14 hogs, same script) only bounds the true rate at < 3% by the rule of
three — a genuine ~1% flake is entirely consistent with a zero-failure sample. Fisher exact tests on
the observed counts (branch 1/100 vs main 0/100, p = 1.0; pooled branch 2/200 vs comparators 0/200,
p = 0.50) show no statistically significant difference. Attribution therefore rests on the structural
and historical evidence above, which is unambiguous: the asserted property is not an invariant, the
code that breaks it is byte-identical to main, and it has been breakable since February 2026.

Recommendation

Do not gate merges on this. Fix the test, not the kernel — test_softirq_aarch64 should either run
its raise/read sequence with interrupts masked and preemption disabled (closing both the drain and
cross-CPU-migration mechanisms), or assert a property it can actually own (e.g. "bit set OR a drain
observably ran"). No aarch64 kernel change is warranted; the softirq clear-before-handler drain
behaviour is correct Linux-style semantics.

Evidence

• Test body: kernel/src/test_framework/registry.rs lines 3974-4035
• Drain: kernel/src/task/softirqd.rs lines 169-229
• IRQ-exit call site: kernel/src/arch_impl/aarch64/exception.rs line 1954