Skip to content

[codex] add evm-only executor load test harness#3658

Draft
codchen wants to merge 14 commits into
codex/evmonly-staking-dynamic-gasfrom
codex/evm-only-executor-loadtest
Draft

[codex] add evm-only executor load test harness#3658
codchen wants to merge 14 commits into
codex/evmonly-staking-dynamic-gasfrom
codex/evm-only-executor-loadtest

Conversation

@codchen

@codchen codchen commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a standalone evmonly-loadtest command that feeds generated EVM-only blocks into the EVM-only executor with generated genesis state, configurable result sinks, and Prometheus/stdout throughput metrics.

The executor owns the result-sink boundary. giga/evmonly exposes ResultSink and WithResultSink, and executor completion invokes the sink for the produced StateChangeSet and receipts before returning. The loadtest harness implements discard/file sink modes through that interface.

This branch also pipelines stateless sender recovery ahead of execution:

  • Executor.PrepareBlock decodes raw tx RLP and recovers senders into PreparedBlock.
  • Executor.ExecutePreparedBlock executes an already prepared block.
  • ExecuteBlock remains the convenience prepare-then-execute path.
  • evmonly-loadtest now has --prepare-workers; raw blocks flow through parallel prepare workers into an ordered prepared-block queue consumed by the single external executor worker.

The harness supports optimistic no-overlap native transfers and ERC20 transfers. It defaults to unique senders and recipients, supports --prebuild-blocks, and includes a lightweight async file sink via --result-sink=file --persist-dir=<dir>. Persistence records are append-only RLP records for changesets and receipts, and the sink removes files on normal completion, execution errors, and SIGINT/SIGTERM.

Metrics include block input/prepared/finished throughput, prepared tx/s, executed tx/s, gas/s, OCC attempts/fallbacks/conflicts, result-sink queue depth, enqueue wait, write time, bytes written, and record counts. sink_enqueue_wait is the backpressure signal: if it rises, executor workers are blocked on persistence queue capacity.

Validation

  • go test ./giga/evmonly/...
  • Native smoke:
go run ./giga/evmonly/cmd/evmonly-loadtest \
  --metrics-addr= \
  --report-interval=0 \
  --prebuild-blocks \
  --blocks=30 \
  --txs-per-block=1000 \
  --builders=8 \
  --prepare-workers=8 \
  --workers=1 \
  --executor-workers=12 \
  --gas-price-wei=0 \
  --min-gas-price-wei=0 \
  --queue-size=64

Observed locally: 197,429 TPS, prepare_errors=0, errors=0.

  • ERC20 smoke:
go run ./giga/evmonly/cmd/evmonly-loadtest \
  --metrics-addr= \
  --report-interval=0 \
  --prebuild-blocks \
  --blocks=30 \
  --txs-per-block=1000 \
  --builders=8 \
  --prepare-workers=8 \
  --workers=1 \
  --executor-workers=12 \
  --workload=erc20-transfer \
  --gas-price-wei=0 \
  --min-gas-price-wei=0 \
  --queue-size=64

Observed locally: 189,483 TPS, prepare_errors=0, errors=0.

EC2 Persistent Runs

Run on commit e312a845b8439fe28824e4bc3aef85cd298ecdd2 using a temporary c8i.48xlarge in us-east-1a, SMT disabled with CoreCount=96,ThreadsPerCore=1, Go 1.25.6, GOMAXPROCS=96, GOGC=400, one external executor worker, prebuilt raw blocks, unique senders/recipients, zero gas price/min gas price, and --result-sink=file. The temporary instance and security group were deleted after collecting logs.

Native transfer tuned command:

GOMAXPROCS=96 GOGC=400 /tmp/evmonly-loadtest \
  --metrics-addr= \
  --blocks=1000 \
  --txs-per-block=5000 \
  --prebuild-blocks \
  --builders=96 \
  --prepare-workers=24 \
  --workers=1 \
  --executor-workers=56 \
  --gas-price-wei=0 \
  --min-gas-price-wei=0 \
  --report-interval=5s \
  --queue-size=64 \
  --result-sink=file \
  --persist-dir="$DIR"

Observed:

prebuild complete elapsed=8.965s blocks=1000 txs=5000000 build_blocks/s=111.55 build_tx/s=557745.01
complete elapsed=27.043s input_blocks=1000 prepared_blocks=1000 prepared_txs=5000000 finished_blocks=1000 txs=5000000 gas=105000000000 prepare_errors=0 errors=0 occ_attempts=1000 occ_fallbacks=0 occ_conflicts=0 sink_queue=1 sink_enqueued=2000 sink_written=1998 sink_bytes=1727519751 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=7.209933s avg_input_blocks/s=36.98 avg_prepared_blocks/s=36.98 avg_prepared_tx/s=184893.42 avg_finished_blocks/s=36.98 avg_tx/s=184893.42 avg_gas/s=3882761797.16
result sink close elapsed=97ms sink_queue=0 sink_enqueued=2000 sink_written=2000 sink_bytes=1729249000 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=7.212607s

ERC20 transfer tuned command:

GOMAXPROCS=96 GOGC=400 /tmp/evmonly-loadtest \
  --metrics-addr= \
  --blocks=1000 \
  --txs-per-block=5000 \
  --prebuild-blocks \
  --builders=96 \
  --prepare-workers=24 \
  --workers=1 \
  --executor-workers=64 \
  --workload=erc20-transfer \
  --gas-price-wei=0 \
  --min-gas-price-wei=0 \
  --report-interval=5s \
  --queue-size=64 \
  --result-sink=file \
  --persist-dir="$DIR"

Observed:

prebuild complete elapsed=19.506s blocks=1000 txs=5000000 build_blocks/s=51.27 build_tx/s=256327.90
complete elapsed=29.693s input_blocks=1000 prepared_blocks=1000 prepared_txs=5000000 finished_blocks=1000 txs=5000000 gas=229960303044 prepare_errors=0 errors=0 occ_attempts=1000 occ_fallbacks=0 occ_conflicts=0 sink_queue=1 sink_enqueued=2000 sink_written=1998 sink_bytes=3146535315 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=10.447174s avg_input_blocks/s=33.68 avg_prepared_blocks/s=33.68 avg_prepared_tx/s=168389.38 avg_finished_blocks/s=33.68 avg_tx/s=168389.38 avg_gas/s=7744574591.71
result sink close elapsed=167ms sink_queue=0 sink_enqueued=2000 sink_written=2000 sink_bytes=3149685000 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=10.451667s

Tuning notes from the same EC2 instance:

  • --queue-size=64 beat 512 by reducing heap/RSS; native improved from 170.2k to 178.1k TPS before worker tuning.
  • Lowering --prepare-workers from 96 to 24 helped because recovery is no longer the steady-state bottleneck and otherwise competes with execution.
  • Native short sweep best: --prepare-workers=24 --executor-workers=56, 194.0k TPS over 2.5M tx.
  • ERC20 short sweep best: --prepare-workers=24 --executor-workers=64, 170.5k TPS over 2.5M tx.
  • No persistent sink backpressure was observed in the tuned full runs: sink_enqueue_wait=0s.

Historical Non-Persistent 199.3k Repro

The earlier 199.3k EC2 benchmark was run before persistent sink and prepared pipeline changes, from commit 4ec8da52c, on a c8i.48xlarge in us-east-1a with SMT disabled, Go 1.25.6, one external executor worker, prebuilt blocks, unique senders/recipients, and zero gas price/min gas price.

GOMAXPROCS=96 GOGC=200 /tmp/evmonly-loadtest \
  --metrics-addr= \
  --blocks=1000 \
  --txs-per-block=5000 \
  --prebuild-blocks \
  --builders=96 \
  --workers=1 \
  --executor-workers=96 \
  --gas-price-wei=0 \
  --min-gas-price-wei=0 \
  --report-interval=5s

Observed:

prebuild complete elapsed=9.366s blocks=1000 txs=5000000 build_blocks/s=106.77 build_tx/s=533873.27
complete elapsed=25.09s input_blocks=1000 finished_blocks=1000 txs=5000000 gas=105000000000 errors=0 avg_input_blocks/s=39.86 avg_finished_blocks/s=39.86 avg_tx/s=199284.61 avg_gas/s=4184976805.58

EC2 Worker Pool / Pinning Follow-up

Commit 63b8662b4bc90936175181b1fff6347c502fb03f adds a persistent OCC worker pool and Linux worker pinning controls. Re-tested on a temporary c8i.48xlarge in us-east-1a, SMT disabled with CoreCount=96,ThreadsPerCore=1, Go 1.25.6, GOMAXPROCS=96, GOGC=400, prebuilt raw blocks, async file persistence, and the same 5M-tx shapes.

Native transfer, persistent OCC pool, unpinned:

complete elapsed=26.794s input_blocks=1000 prepared_blocks=1000 prepared_txs=5000000 finished_blocks=1000 txs=5000000 gas=105000000000 prepare_errors=0 errors=0 occ_attempts=1000 occ_fallbacks=0 occ_conflicts=0 sink_queue=1 sink_enqueued=2000 sink_written=1998 sink_bytes=1727519751 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=7.416901s avg_input_blocks/s=37.32 avg_prepared_blocks/s=37.32 avg_prepared_tx/s=186607.41 avg_finished_blocks/s=37.32 avg_tx/s=186607.41 avg_gas/s=3918755519.49
Percent of CPU: 2404%

Native transfer, persistent OCC pool, --pin-workers --prepare-cpu-offset=0 --executor-cpu-offset=24:

complete elapsed=26.783s input_blocks=1000 prepared_blocks=1000 prepared_txs=5000000 finished_blocks=1000 txs=5000000 gas=105000000000 prepare_errors=0 errors=0 occ_attempts=1000 occ_fallbacks=0 occ_conflicts=0 sink_queue=1 sink_enqueued=2000 sink_written=1998 sink_bytes=1727519751 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=7.68812s avg_input_blocks/s=37.34 avg_prepared_blocks/s=37.34 avg_prepared_tx/s=186682.92 avg_finished_blocks/s=37.34 avg_tx/s=186682.92 avg_gas/s=3920341280.53
Percent of CPU: 2283%

ERC20 transfer, persistent OCC pool, unpinned:

complete elapsed=29.325s input_blocks=1000 prepared_blocks=1000 prepared_txs=5000000 finished_blocks=1000 txs=5000000 gas=229960303044 prepare_errors=0 errors=0 occ_attempts=1000 occ_fallbacks=0 occ_conflicts=0 sink_queue=1 sink_enqueued=2000 sink_written=1998 sink_bytes=3146535315 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=10.314016s avg_input_blocks/s=34.10 avg_prepared_blocks/s=34.10 avg_prepared_tx/s=170501.05 avg_finished_blocks/s=34.10 avg_tx/s=170501.05 avg_gas/s=7841694425.87
Percent of CPU: 2003%

ERC20 transfer, persistent OCC pool, pinned:

complete elapsed=30.25s input_blocks=1000 prepared_blocks=1000 prepared_txs=5000000 finished_blocks=1000 txs=5000000 gas=229960303044 prepare_errors=0 errors=0 occ_attempts=1000 occ_fallbacks=0 occ_conflicts=0 sink_queue=1 sink_enqueued=2000 sink_written=1998 sink_bytes=3146535315 sink_enqueue_wait=0s sink_enqueue_wait_events=0 sink_write=11.436487s avg_input_blocks/s=33.06 avg_prepared_blocks/s=33.06 avg_prepared_tx/s=165290.26 avg_finished_blocks/s=33.06 avg_tx/s=165290.26 avg_gas/s=7602039743.89
Percent of CPU: 2009%

Conclusion: the persistent OCC pool removes per-block worker creation overhead but does not materially increase core utilization. Native moved from ~184.9k to ~186.6k TPS, while CPU remained around 20-24 effective cores on a 96-core instance. Thread pinning did not help on this workload and slightly reduced CPU utilization in the native case.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJul 2, 2026, 9:54 AM

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.55463% with 527 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.35%. Comparing base (9cb00d6) to head (63b8662).

Files with missing lines Patch % Lines
giga/evmonly/cmd/evmonly-loadtest/main.go 63.40% 402 Missing and 91 partials ⚠️
...only/cmd/evmonly-loadtest/worker_affinity_linux.go 0.00% 17 Missing ⚠️
giga/evmonly/occ.go 79.26% 16 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@                          Coverage Diff                          @@
##           codex/evmonly-staking-dynamic-gas    #3658      +/-   ##
=====================================================================
+ Coverage                              58.15%   58.35%   +0.19%     
=====================================================================
  Files                                   2187     2182       -5     
  Lines                                 178718   178712       -6     
=====================================================================
+ Hits                                  103939   104291     +352     
+ Misses                                 65454    65234     -220     
+ Partials                                9325     9187     -138     
Flag Coverage Δ
sei-chain-pr 69.07% <63.55%> (+12.21%) ⬆️
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...only/cmd/evmonly-loadtest/worker_affinity_linux.go 0.00% <0.00%> (ø)
giga/evmonly/occ.go 78.94% <79.26%> (+6.26%) ⬆️
giga/evmonly/cmd/evmonly-loadtest/main.go 63.40% <63.40%> (ø)

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codchen codchen force-pushed the codex/evmonly-staking-dynamic-gas branch from bd57e4c to 7c6c44c Compare June 30, 2026 06:11
@codchen codchen force-pushed the codex/evm-only-executor-loadtest branch from 8e28929 to 4ec8da5 Compare June 30, 2026 06:11
@codchen codchen force-pushed the codex/evmonly-staking-dynamic-gas branch from 7c6c44c to 6f9547a Compare July 2, 2026 05:53
@codchen codchen force-pushed the codex/evm-only-executor-loadtest branch from 8cc2094 to a65459e Compare July 2, 2026 05:55
@codchen codchen force-pushed the codex/evmonly-staking-dynamic-gas branch from 6f9547a to 31f16a8 Compare July 2, 2026 08:39
@codchen codchen force-pushed the codex/evm-only-executor-loadtest branch from fc79ac5 to e312a84 Compare July 2, 2026 08:40
@codchen codchen force-pushed the codex/evmonly-staking-dynamic-gas branch from 31f16a8 to 9cb00d6 Compare July 2, 2026 09:51
@codchen codchen force-pushed the codex/evm-only-executor-loadtest branch from e312a84 to 63b8662 Compare July 2, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant