investigate(rp2040): readable BOOTSEL volume stalls on first write; make watchdog cleanup handle-safe

[zackees]

Context

Two RP2040 boards have now refused programming. The current failure has a specific, repeatable state:

• The RP2040 enters BOOTSEL and enumerates as USB\VID_2E8A&PID_0003\E0C9125B0D9B.
• Its mass-storage interface (MI_00) is healthy.
• Windows mounts RPI-RP2 on G: and reports the volume healthy.
• INFO_UF2.TXT and other reads succeed.
• Writes do not complete:
  • fbuild's NEW.UF2 write stalls;
  • a normal Windows file copy stalls;
  • a paced write-through test using 8 KiB chunks stalled before its first completed chunk and recorded zero progress.
• Windows logged Disk events 51 and 153 during the attempts, including retries at LBAs 0x134 and 0x14c.
• Restart Manager reported no user-mode holder, and an exclusive volume open succeeded between attempts.

The last point does not prove that the failure is device-side. Kernel storage filters, filesystem behavior, USB transport faults, QSPI programming behavior, and a host/controller problem can all exist without a user-mode volume handle. A USB trace is required before claiming that the RP2040 bulk-OUT endpoint failed to acknowledge the transfer.

The separate MI_01 PICOBOOT interface currently has no Windows driver. That does not explain why the healthy MI_00 mass-storage interface can be read but not written.

fbuild regression question

RP2040 deploy hardening was merged in commit debc9963 through PR #1083.

The relevant commit boundary is:

1. d5144e0111d04b390f431e9700211a65cfee2f9d — immediately before the hardening
2. a3d8c58b94111a4e055f66d67d4ff8ad702fd730 — fresh-enumeration retries
3. c472f56f5989a4507321e862fdb62d7c1bcf309f — watchdog and broader hardening
4. debc9963527e0a5a09f1224883b02d63ba174425 — merged hardening
5. 16bb9ffae6aeab38957176357425110b002d98b1 — current main during this reproduction

Static comparison shows that the low-level first-write shape was not materially changed by #1083: both sides open/truncate NEW.UF2 and make a whole-buffer write_all call.

The hardening did introduce a definite recovery hazard. run_with_watchdog deliberately abandons a timed-out worker thread:

fbuild/crates/fbuild-deploy/src/rp2040.rs

Lines 544 to 560 in 16bb9ff

	/// Run `work` on a dedicated thread and give up after `budget`. A storport
	/// retry storm behind a sick hub can block the NEW.UF2 `write_all` for
	/// minutes with no output; this turns that hang into an actionable failure
	/// that feeds the fresh-enumeration retry gate. On timeout the worker thread
	/// is deliberately abandoned, not joined: it only touches its own locals and
	/// its channel send fails silently once this receiver is dropped, so the
	/// wedged kernel write can unblock (or not) without holding up the deploy.
	fn run_with_watchdog<T: Send + 'static>(
	budget: Duration,
	label: &str,
	work: impl FnOnce() -> Result<T> + Send + 'static,
	) -> Result<T> {
	let (sender, receiver) = std::sync::mpsc::channel();
	std::thread::spawn(move || {
	let _ = sender.send(work());
	});
	match receiver.recv_timeout(budget) {

That worker owns the open NEW.UF2 file. If the kernel write remains blocked, the thread and handle can remain alive inside the daemon after fbuild reports a timeout. This can contaminate retries and later deploy attempts even if the watchdog did not cause the original first-write stall.

The existing watchdog test verifies that the caller returns quickly by leaving a five-second worker running. It does not verify worker termination, handle release, or whether an immediate second write can open the destination.

There is also a local version-isolation risk:

• Global executable: C:\tools\python13\Scripts\fbuild.exe, reporting 2.5.2
• FastLED virtual-environment executable: reporting 2.5.3
• Installed Python metadata in that environment: reporting 2.5.2

Every hardware result therefore needs to record the exact executable, executable hash, source commit, daemon port, and daemon binary used.

Pinned artifact evidence

The current artifact is:

• Size: 882,688 bytes
• SHA-256: 1C770999678BB7563A81D36C696DC44C074A712E25F0DE68A8A83F859E8AF0E0
• 1,724 UF2 blocks of 512 bytes
• RP2040 family ID: 0xE48BFF56
• Target range: 0x10000000 through 0x1006BB00
• Block magic, block numbering, payload lengths, family ID, and declared block count are structurally valid

An earlier deploy transcript reported a different artifact size. Commit comparisons must therefore use one pinned artifact without rebuilding between runs.

Proposal

1. Produce a deterministic reproduction record

Before every attempt, record:

• physical board serial and VID:PID;
• PnP health for both BOOTSEL interfaces;
• drive letter and USB hub/root-port topology;
• exact fbuild executable path, version, hash, source commit, and daemon binary;
• a unique five-digit FBUILD_DAEMON_PORT;
• artifact path, size, and SHA-256;
• relevant Windows Disk 51/153 events before and after the attempt;
• whether any fbuild worker/process/volume handle remains after timeout.

Use an outer killable process boundary for any experiment that may block. Do not allow one timed-out attempt to contaminate the next case.

2. Run a transport/artifact matrix

On the same board, cable, direct motherboard port, and clean BOOTSEL enumeration, compare:

1. the previously proven RAM-only UF2;
2. a small official known-good RP2040 flash Blink UF2;
3. the pinned fbuild-generated flash UF2.

Interpretation:

• RAM succeeds but both flash images fail: investigate the flash-target/QSPI path.
• Official flash succeeds but fbuild flash fails: investigate artifact generation or layout.
• All three fail on the first storage request: investigate the Windows/USB/volume path.
• The result changes with fbuild commit while the pinned artifact and physical setup remain identical: investigate an fbuild deploy regression.

Also repeat the decisive case using:

• a known-good data cable;
• a direct USB 2.0-capable motherboard port;
• a second clean host, preferably Linux.

On Linux, a guarded comparison between a normal mounted-volume copy and an unmounted raw block-device write may distinguish filesystem integration from USB/device behavior. The exact block device must be verified before any raw write.

3. Capture the failing layer

Capture one failure with:

• USBPcap/Wireshark, to determine whether the first mass-storage bulk-OUT transaction leaves the host and what response/reset sequence follows;
• Process Monitor filtered to the selected volume and fbuild process;
• Windows Disk, StorPort, Kernel-PnP, and USB events;
• read-only fltmc/PnP inspection when elevated access is available.

The trace should classify the failure without assuming that readable media plus a missing user-mode handle proves a board fault.

4. Perform an exact hardening A/B

Build and run isolated versions at the commit boundary listed above:

• use separate worktrees/installations;
• use separate daemon ports and state directories;
• verify both CLI and daemon provenance;
• use the same pinned UF2;
• start with the pre-hardening build after a clean reboot;
• do not run an older build after a watchdog failure unless every newer worker/process/handle has been proven gone.

Decision criteria:

• d5144e01 repeatedly succeeds while c472f56f fails in the clean identical setup: hardening regression.
• Both fail on the same first storage operation: the initial stall predates or is independent of the hardening.
• Both fail initially, but only watchdog-enabled builds retain a worker/handle or poison the next attempt: underlying transport failure plus a separate fbuild recovery regression.

5. Make timeout cleanup handle-safe

If the watchdog path is retained, it must not abandon a thread that owns a removable-volume file handle.

Use either:

• a killable helper process whose termination closes the handle; or
• cancellable Windows overlapped I/O using CancelIoEx, followed by bounded completion and thread joining.

After timeout:

• the worker must be terminated or joined;
• the NEW.UF2 handle must be closed;
• stale volume paths must be discarded;
• retries may occur only after a verified fresh BOOTSEL enumeration;
• an immediate second open/write probe must not fail because of the prior attempt.

Add a deterministic regression test covering timeout followed by immediate handle reacquisition and a second write.

6. Use PICOBOOT only as a controlled comparison

PICOBOOT/picotool can be used as a transport control on another host, or where the interface already has a compatible driver. Do not install or rebind drivers on the current machine as part of this issue.

Acceptance criteria

• A reproducible log captures exact board, artifact, executable, daemon, USB topology, and Windows event identities.
• The RAM-only, official flash, and fbuild flash artifact matrix is completed without rebuilding between cases.
• At least one USB trace identifies whether the first write leaves the host and what fails next.
• The pre-hardening through current-main A/B identifies whether harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets #1083 changed the initial failure.
• The initial transport failure and any post-timeout contamination are reported as separate findings.
• A watchdog timeout leaves no live worker, helper, or NEW.UF2 handle.
• A deterministic test proves that a second destination open/write succeeds immediately after timeout cleanup.
• Hardware closeout runs bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lint with a unique daemon port and records the exact decisive output.
• Conclusions do not label either RP2040 board defective without the artifact, host, and transport controls above.

Fixed environmental constraint / non-goals

• Acronis must remain installed, enabled, and unmodified.
• Do not stop, disable, uninstall, upgrade, reconfigure, detach, or mutate any Acronis service, driver, filter, or installation.
• Read-only observation of the existing storage stack is allowed; changing it is not.
• Do not run chkdsk or attempt filesystem repair on the synthetic BOOTSEL FAT volume.
• Do not treat “Restart Manager found no process” as proof that no kernel component participated.
• Do not require a BOOTSEL button sequence as the final fbuild recovery design.
• Do not install Zadig or rebind the current machine's RP2040 interfaces as part of this investigation.

External reports

Similar symptoms exist outside fbuild:

• Raspberry Pi forum: readable BOOTSEL files but writes disconnect before transferring bytes. A Linux unmount/raw-write comparison worked in that report, suggesting the mounted filesystem path can matter.
• Adafruit: Windows UF2 copy hangs at 0%. Their documented case involved host disk-management software, showing that this surface symptom is not sufficient to diagnose a board fault.
• Raspberry Pi: the BOOTSEL drive is a synthetic virtual volume.
• UF2 specification.

These are analogous reports, not proof of the same root cause.

Related issues

• investigate(rp2040): harden hub-path BOOTSEL mass-storage diagnostics #1081 — hub-path BOOTSEL mass-storage diagnostics
• harden(rp2040): deploy robustness across host setups (hub chains, drive-scan stalls, timeout policy) #1082 — deploy hardening and the original open question about abandoning blocked writer threads
• feat(windows): recover RP2040 phantom/BOOTSEL USB state through scoped admin PnP flow #1152 — Windows phantom/BOOTSEL recovery; distinct because the present volume is healthy and readable
• feat(rp2040): zero-touch deployment to an as-received Pico via ROM UF2 #1040 — original zero-touch RP2040 deployment and RAM-versus-flash evidence
• feat(rp2040): deploy parity with Arduino-Pico, PlatformIO, and official picotool #1049 — canonical RP2040 UF2 artifact and transfer semantics

[zackees]

Pre-hardening rollback map and live d5144e01 result (2026-07-23)

Bottom line

I rebuilt the exact pre-hardening boundary at d5144e01, installed that build into an isolated venv, verified that uv run --no-project fbuild resolved to the rebuilt executable, and used it to deploy the pinned 882,688-byte UF2.

The pre-hardening deploy also failed on this host/board. It returned a mass-storage error 1006 while copying to G:\NEW.UF2, then the managed-picotool fallback could see the PICOBOOT interface but could not open it because MI_01 has no compatible driver. Windows logged Disk 153 at LBA 0x134 plus thirteen Disk 51 events during this attempt.

That is evidence that the initial storage-path failure exists before #1083. It is not evidence that the watchdog cleanup is safe: the post-timeout abandoned-worker/handle hazard introduced by the hardening remains a separate issue.

Which commit to roll back to

Commit	What it represents	Use
0bf8feb0 (v2.5.2)	Last published tag before #1083. It predates b82948e5 and d5144e01.	Easiest released rollback, but slightly earlier than the exact boundary.
b82948e5	Last RP2040 behavior commit before hardening: #1077, “avoid inferring QSPI fault from eject timeout.”	Best “last RP2040 code before hardening” point.
d5144e01	Exact main commit immediately before squash merge debc9963. Its only diff from b82948e5 is six lines in CLAUDE.md; there is no fbuild-deploy diff.	Recommended clean A/B baseline. This is what I built and ran below.
a3d8c58b	PR-only phase with fresh-enumeration transfer retries. It changes only rp2040.rs and does not contain the later watchdog/full hardening. It was never on main.	Useful surgical bisect point after d5144e01; not a release rollback.
62d86f2a	Formatting commit on the PR branch.	No useful behavior boundary.
c472f56f	Full PR head: watchdog, topology diagnostics, tunable budgets, fallback/policy changes.	Hardened comparison point, not a rollback.
debc9963	Squash merge of #1083; first main commit containing the complete hardening.	First mainline “after” point.
d5eb1524 (v2.5.3)	Release tag that contains debc9963 plus subsequent changes.	Not a pre-hardening build.

The intermediate SHAs a3d8c58b and c472f56f initially appear missing from a normal local main checkout because #1083 was squash-merged. Fetch the PR head before using either:

git fetch origin pull/1083/head:refs/remotes/origin/pr/1083

Rebuild and run the exact pre-hardening CLI

This keeps the old CLI/daemon out of the global install and makes uv run --no-project fbuild resolve to the rebuilt executable:

git worktree add --detach .claude/worktrees/issue-1161-d5144e01 `
  d5144e0111d04b390f431e9700211a65cfee2f9d
Set-Location .claude/worktrees/issue-1161-d5144e01

uv venv .venv
uv pip install --python .venv\Scripts\python.exe "setuptools>=64"

$env:VIRTUAL_ENV = (Resolve-Path .venv).Path
$env:PATH = "$env:VIRTUAL_ENV\Scripts;$env:PATH"
$env:FBUILD_BUILD_RELEASE = "1"

# On this host, plain `uv build --wheel` failed to provision setuptools in
# the historical isolated build env. This explicit fallback succeeded.
uv build --wheel --no-build-isolation --out-dir dist-issue1161-d514
uv pip install --python .venv\Scripts\python.exe --force-reinstall `
  .\dist-issue1161-d514\fbuild-2.5.2-cp313-cp313-win_amd64.whl

uv run --no-project fbuild --version
# fbuild 2.5.2

Provenance from this build:

• source commit: d5144e0111d04b390f431e9700211a65cfee2f9d
• wheel: 26,199,746 bytes, SHA-256 B96AF74A035DE35925D931CC9A775185F71371D5EF33A48F85E2610EF8D140DC
• CLI: 35,405,824 bytes, SHA-256 36CDCC27AAE44740956BABFA2B7F56A01E7CC75B6680B9076AB4D447D43A79B5
• daemon: 37,073,408 bytes, SHA-256 F78A756C736D44D72D1089B28590182210AA7D4E55761D6CC6E6C1D4BCE412D3
• where.exe fbuild under uv run --no-project listed the isolated .venv\Scripts\fbuild.exe first.

For isolation, I used:

$env:FBUILD_DAEMON_PORT = "52161"
$env:FBUILD_DEV_MODE = "1"
$env:FBUILD_CACHE_DIR = "<worktree>\.claude\tmp\issue1161-runtime\cache"
$env:FBUILD_BUILD_DIR = "<worktree>\.claude\tmp\issue1161-runtime\build"

Artifact-controlled deploy command

The pinned artifact from this issue was recovered at:

C:\Users\niteris\dev\fastled\.build\pio\rp2040\.fbuild\build\release\firmware.uf2

Verified values:

• firmware.uf2: 882,688 bytes, SHA-256 1C770999678BB7563A81D36C696DC44C074A712E25F0DE68A8A83F859E8AF0E0
• firmware.bin: 441,316 bytes, SHA-256 F29910E24324F71ADB58D40DE97D6C1746F1F688BE111615858CD01F70B91BC7

I copied the existing firmware.bin, .uf2, .elf, .map, and build_info.json byte-for-byte into the isolated $env:FBUILD_BUILD_DIR\release and rehashed them. The deploy used --skip-build, so the artifact could not change:

uv run --no-project fbuild `
  C:\Users\niteris\dev\fastled\.build\pio\rp2040 `
  deploy -e rp2040 --skip-build -v

The command ran behind a 120-second outer process boundary. Before starting it there were no fbuild.exe or fbuild-daemon.exe processes. The boundary did not fire; the historical deploy returned exit 1 itself.

Decisive output:

deploy failed: failed to copy RP2040 UF2 ...\firmware.uf2 to G:\NEW.UF2:
The volume for a file has been externally altered so that the opened file
is no longer valid. (os error 1006)

Managed picotool error: ... RP2040 device in BOOTSEL mode, but picotool was
unable to connect. You may need to install a driver via Zadig.

No driver was installed or rebound, and Acronis was not stopped, modified, or inspected beyond the allowed read-only host observations.

Hardware and Windows evidence

Before the attempt:

• BOOTSEL composite: USB\VID_2E8A&PID_0003\E0C9125B0D9B
• mass storage MI_00: present/OK
• PICOBOOT MI_01: present, problem code 28/no driver
• RPI-RP2: G:, FAT, Healthy/OK
• runtime identity from the prior enumeration: 2E8A:000A, serial 5303284720C4641C, COM12 (not present while in BOOTSEL)

During the pre-hardening attempt (start 2026-07-23T12:55:09-07:00):

• Disk 153: one retry at LBA 0x134 on Disk 2
• Disk 51: thirteen paging-operation errors on \Device\Harddisk2\DR25
• G:\NEW.UF2 was not present after the failure

Immediately after the command, BOOTSEL MI_00/MI_01 and G: were still present, G: still reported Healthy/OK, and no fbuild CLI/daemon process remained. Thus this particular old-build failure did not leave a live process-level handle after it returned.

Later G: disappeared. After the board was physically replugged, Windows did not enumerate either 2E8A endpoint. The only new present node was:

USB\VID_0000&PID_0002\6&3AF0F9CE&0&14
Unknown USB Device (Device Descriptor Request Failed)
problem code 43
Port_#0014.Hub_#0001
ACPI(...RHUB)#ACPI(PO14)   # direct root port

All prior 2E8A nodes and COM12 were non-present; the current scanner showed only the phantom COM12 record. This is a post-replug descriptor-level enumeration failure. It is recorded here without asserting that the deploy caused it.

Interpretation and next bisect

1. d5144e01 failed on the same Windows mass-storage surface with the same pinned artifact. Therefore harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets #1083 is not required to produce the initial failure.
2. This run did not exercise run_with_watchdog, because it predates that code. It cannot clear the abandoned-worker/handle concern in current builds.
3. The sharp next code boundary is a3d8c58b: it has fresh-enumeration retries but no watchdog. After host/USB enumeration is restored, run that against the same pinned files and an isolated port/cache.
4. Then compare c472f56f or debc9963 with the same process boundary. If only the watchdog-enabled case leaves a live daemon worker/handle after its caller returns, report that separately from the underlying Disk 51/153 transport failure.
5. bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lint could not be completed because this deploy did not produce an application CDC endpoint.

One additional limitation: this was started with a clean fbuild process audit, but not after a full host reboot. It is a useful pre-hardening reproduction, not the complete clean-reboot A/B matrix requested by the issue.

[zackees]

Open-source uploader comparison: make PICOBOOT a first-class RP2040 transport

The Arduino/PlatformIO source comparison changes the recommended direction here.

What the other toolchains actually do

Arduino IDE itself does not own one upload implementation; the selected board core supplies the recipe.

• The Earle Philhower Arduino-Pico core used by fbuild defaults to its UF2-volume path: touch the application serial port, find a BOOTSEL volume, then write NEW.UF2. Source: uf2conv.py.
• That same core also exposes a Picotool upload method which runs picotool load <firmware.uf2> -f -x, bypassing the Windows filesystem. Source: platform.txt.
• Arduino's official rp2040tools uses rp2040load to orchestrate elf2uf2, picotool info, picotool load, and picotool reboot. Its sequence is visible directly in main.go.
• PlatformIO's default RP2040 protocol is picotool. Its builder performs the 1200-baud touch and then launches rp2040load -v -D <firmware.elf>; this is direct PICOBOOT transport, not an RPI-RP2 disk copy. Source: platform-raspberrypi/builder/main.py.

So the useful comparison is:

Path	USB interface	Transfer
Arduino-Pico default / current fbuild primary	BOOTSEL MI_00 mass storage	Write NEW.UF2 through Windows filesystem/storage stack
PlatformIO default / Arduino Picotool option	BOOTSEL MI_01 PICOBOOT	picotool load directly over USB

Recommended fbuild direction

fbuild should make the direct PICOBOOT/picotool path a first-class, explicitly selectable transport and preferably the primary automatic path, rather than reaching it only after the mass-storage copy has already failed.

The model already proven by PlatformIO/Arduino is approximately:

1. If application CDC is present, perform the 1200-baud reset/touch.
2. Wait for the 2E8A:0003 BOOTSEL/PICOBOOT interface.
3. Probe with picotool info.
4. Load the generated UF2/ELF using picotool load.
5. Reboot/execute and verify application re-enumeration.
6. Retain RPI-RP2/NEW.UF2 as an explicit alternative or fallback transport.

fbuild already has a managed-picotool fallback, so this should mostly be a matter of transport selection, ordering, error classification, and testing—not implementing PICOBOOT from scratch.

This also avoids the exact Windows surface implicated in this reproduction: the pre-hardening failure was in the G:\NEW.UF2 filesystem write and produced Disk 51/153 events. A successful direct PICOBOOT load would never open that drive or send the firmware through the Windows mass-storage filesystem path.

Important limitation from this host

The historical fbuild run did reach managed picotool, but Windows currently reports PICOBOOT MI_01 with problem code 28/no compatible driver, so picotool could see BOOTSEL but could not open it. That is a separate connection/driver prerequisite and does not make the mass-storage route equivalent to PICOBOOT. No driver was installed or rebound during this investigation.

Once USB enumeration is restored, the next controlled test should be PICOBOOT-only against the already-pinned firmware artifact. If that succeeds without touching G: and without new Disk 51/153 events, it cleanly isolates the observed failure to the mass-storage deployment path and provides the implementation direction for fbuild.