Resume the Bitcoin fee walk from a persisted cursor - #7
Open
droplister wants to merge 1 commit into
Open
Conversation
The fee runner restarted at the top of the missing-fee index every tick. The top ~930 candidates are 2019–2021 P2SH-segwit rows whose Counterparty tx_hash is a witness hash, so Electrs, mempool.space and blockstream all return 404 for them forever. Each run spent its whole 1,000-request budget on those same rows (logged as requested: 1000, updated: 0 after 128 s) and never reached the 1.08M pre-segwit rows below that Electrs can serve. Counterparty's own fee field was checked as an alternative source and rejected: against 5,000 stored Bitcoin-authoritative fees it disagreed on 7–44% of rows (mostly 0, some null, some different values), which is why the replayer stopped ingesting it on 2026-07-16. Now each run fetches transactions newer than a tip watermark first, then resumes the historical walk from core_state.bitcoin_fees_walk_cursor, so an unresolvable row costs one request per full cycle. Requests are smoothed to the provider's documented budget (3 per second) instead of six concurrent bursts that drew 429s; 300 candidates per two-minute tick clears the backlog in about five days. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016bNJBT1Ugmg2VG25BrdsJz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
reconcileStagedBitcoinFeesloggedrequested: 1000, updated: 0(128 s) every tick andbitcoin_fees_remainingsat at 1,080,033. The walk restarted at the top oftx_index DESCeach run, and the top ~930 candidates are 2019–2021 P2SH-segwit rows whose Counterpartytx_hashis a witness hash — Electrs, mempool.space and blockstream all 404 on them forever. The runner never reached the 1.08M pre-segwit rows (blocks 250k–550k) that Electrs can serve.Why not Counterparty's
feeChecked 5,000 transactions against stored Bitcoin-authoritative fees: Counterparty's value disagreed on 7–44% per 1,000-row page (mostly
0, somenull, some different numbers). That is why32900bestopped ingesting it. The earlier handoff's "returns their exact fees" was not validated.Change
core_state.bitcoin_fees_tip_watermarkfirst, so new transactions get fees within a tick.core_state.bitcoin_fees_walk_cursor; resets to the top when a cycle completes. Unresolvable rows cost one request per cycle instead of every run.ELECTRS_REQUEST_BATCH_SIZE/INTERVAL(3/s) instead of 6 concurrent bursts that drew 429s. 300 candidates per tick ≈ 100 s; backlog clears in ~5 days.UPDATEs, 2core_stateupserts.Tests: SQLite-backed D1 shim covers cursor advance past 404s, resume across runs, cycle reset, and the tip pass.
npm run checkandnpm test -w xcp-api(397) pass.🤖 Generated with Claude Code
https://claude.ai/code/session_016bNJBT1Ugmg2VG25BrdsJz