Engine: SEE-pruned quiescence (static exchange evaluation) (closes #212) - #219
Engine: SEE-pruned quiescence (static exchange evaluation) (closes #212)#219testtest126 wants to merge 3 commits into
Conversation
Adds Board.see(_:) — the standard "swap algorithm" Static Exchange Evaluation over the mailbox board: net material result of the full capture/recapture sequence on a square, least-valuable-attacker order, with x-ray/battery attackers picked up automatically (attackers are recomputed from scratch against the reduced occupancy at each step, so a piece revealed behind a captured blocker is naturally found — no separate x-ray bookkeeping needed). Unit-tested against hand-verified cases: a clean win, a losing queen-takes-defended-pawn, an equal rook trade, a battery/x-ray position specifically chosen to discriminate a correct implementation from one that only looks at pre-capture attackers, en passant (whose victim isn't on the destination square), and a no-recapture edge case. Wires it into Search.quiesce (NegamaxEngine.swift): out of check, a capture with SEE < 0 is skipped rather than searched — losing the exchange can't beat standing pat, so confirming that by search is wasted work. Promotions are exempt (SEE only prices the destination-square exchange, not the new queen). Never applied while in check. Adds a seePruning: Bool toggle (default true) to NegamaxEngine/ PersistentNegamaxEngine/EngineConfig, and --baseline-see/--candidate-see/ --candidate-evaluator default flags to EngineLab's abtest CLI, mirroring the lateMoveReductions seam from #218/PR feature/lmr-search — this is what let the heuristic be isolated and SPRT-measured. Re-pins the EngineLabTests bench determinism signature. SPRT (abtest, DefaultEvaluator fixed both sides, 15,000-node budget both sides, 400 openings sampled from testtest126/books' noob_3moves.epd, seed 1, elo0=0/elo1=10): full 800-game schedule played without resolving. SEE +179 =440 -181 (49.9%), Elo -1 ± 16 (95%), llr -0.71 vs bounds -2.94..2.94 -> inconclusive. This is an honest non-result, not a win: unlike LMR (#218), SEE-pruned quiescence shows no measurable Elo effect at this node budget/depth regime, despite implementing correctly (per the unit tests) and reducing node count meaningfully in tactical positions (kiwipete: 1.7-2.1x fewer nodes across depths 3-6) — but inconsistently (middlegame fixture: 1.27- 1.51x fewer at depths 3-5, briefly *more* nodes at depth 6, 0.76x, from pruning interacting with move ordering/alpha-beta cutoffs elsewhere in the tree) and only mildly on quieter positions (startpos: ~1.00-1.05x). Plausible explanation: quiescence is a much smaller, shallower slice of total search cost than the whole main tree LMR prunes recursively, so even a real per-node speedup there reinvests into comparatively little extra useful depth. Not recommending this merge as a strength improvement as-is; flagging for the orchestrator/owner to decide whether to shelve or take a different angle (SEE-based capture *ordering* rather than pruning, a wider EPD sample, or re-testing once combined with #218). Closes #212.
|
Merge request: PR #219 @ 863398f SPRT (abtest, DefaultEvaluator fixed both sides, equal 15k-node budget both sides, 400 openings sampled from testtest126/books' noob_3moves.epd, seed 1, elo0=0/elo1=10): full 800-game schedule played, SEE +179 =440 -181 (49.9%), Elo -1 ± 16 (95%), llr -0.71 vs bounds -2.94..2.94 -> inconclusive. Flagging explicitly: this is NOT a demonstrated strength win, unlike #218. Implementation is correct (SEETests pass, including an x-ray-discriminating case) and reduces quiescence node count meaningfully in tactical positions (kiwipete: 1.7-2.1x fewer nodes), but that didn't translate to a measurable Elo gain at this node-budget regime. Recommend the owner decide explicitly whether to merge anyway (correctness/infra value), hold for a different angle (SEE-based move ordering instead of pruning, wider sample), or shelve — not a routine green-light merge like #218. (Posted as a PR comment per the open-pr skill's fallback — same as PR #218, couldn't confidently identify the orchestrator session from list_sessions.) |
1 participant
Closes #212.
Honest result up front: this is NOT a demonstrated strength improvement
A properly-powered SPRT (800 games, full schedule, did not resolve either way) found no measurable Elo effect: -1 ± 16 Elo, essentially a coin flip. Unlike LMR (#218), which resolved cleanly positive, this one genuinely didn't move the needle. I'm opening the PR as requested (correct, tested implementation with real infra value), but not recommending it merge as a strength win — that call is the orchestrator's/owner's, with this data in front of them. Details and a plausible explanation below.
What
Adds
Board.see(_:)(ChessKit/Sources/ChessKit/Board.swift) — the standard "swap algorithm" Static Exchange Evaluation over the mailbox board: net material result of the full capture/recapture sequence on a square, least-valuable-attacker order each step. X-ray/battery attackers are picked up for free —attackers(to:by:excluding:)recomputes sliding attacks from scratch against the reduced occupancy at every step of the exchange, so a piece revealed behind a captured blocker is naturally found without separate bookkeeping.Wires it into
Search.quiesce(ChessKit/Sources/ChessProtocol/NegamaxEngine.swift, ~:493-530): out of check, a capture withsee(move) < 0is skipped rather than searched. Promotions are exempt (SEE only prices the destination-square material exchange, not the new queen). Never applied while in check — the existing "search every evasion" branch is untouched.Adds a
seePruning: Booltoggle (defaulttrue) toNegamaxEngine/PersistentNegamaxEngine/EngineConfig, and--baseline-see/--candidate-see/--candidate-evaluator defaultflags toEngineLab'sabtestCLI, mirroring thelateMoveReductionsseam from #218 — this is what let the heuristic be isolated and SPRT-measured against an otherwise-identical baseline.Re-pins the
EngineLabTestsbench determinism signature.SEE correctness —
SEETests.swiftSix hand-verified cases: a clean undefended win (+100), a losing queen-takes-pawn-defended-by-pawn (-800), an equal rook trade (0), en passant (whose victim isn't on
move.to— a naive implementation would score it as a free capture), a no-recapture edge case, and a battery/x-ray position specifically constructed to discriminate a correct implementation from a naive one: two White rooks doubled on a file, only the front one initially able to reach the target square, with Black holding one defending rook. Correct answer (x-ray-aware): +500. A SEE that only looked at attackers before any capture (missing the rear rook, since it starts blocked) would return 0 instead — this is the actual bug the test catches, not just a smoke check. All 6 pass.SPRT result
abtest,DefaultEvaluatorheld fixed both sides (--candidate-evaluator default), 15,000-node budget both sides (equal-budget, not equal-depth — matches the LMR methodology), 400 openings sampled (seed 1) from testtest126/books'noob_3moves.epd,--elo0 0 --elo1 10:Node-count effect — real, but inconsistent (helps explain the neutral SPRT)
Measured via a local scratch probe against the built library (not committed), SEE-on vs SEE-off at equal depth:
SEE pruning genuinely cuts nodes in capture-rich positions (kiwipete: consistently 1.7-2.1x fewer) and barely matters in quiet ones (startpos), as expected. But it's not monotonic — at middlegame/depth 6 it used more nodes than the unpruned search, from pruning interacting with move ordering and alpha-beta cutoffs elsewhere in the tree (a real, known effect, not a bug — confirmed by
SEEQuiescenceTestspassing, including a dedicatedtestSEEPruningReducesNodeson a different fixture where the reduction does hold).Plausible explanation for the neutral SPRT: quiescence is a much smaller, shallower slice of total search cost than the whole main tree LMR (#218) prunes recursively at every ply — LMR's node savings compound across the tree, SEE's don't. The captures SEE prunes are also often ones MVV-LVA ordering already pushes late and alpha-beta already cuts off cheaply, so the marginal compute actually freed may be smaller than the raw "captures skipped" count suggests — leaving little left over to reinvest into useful extra depth at this node-budget regime.
Verified locally at 863398f (this PR's exact head), in a detached scratch worktree
ChessKit:swift test(release, and separately confirmed in debug for the bench signature + both new test files) — EXIT:0chess-server:swift test— EXIT:0 (99 tests; unaffected by this change, verified for the record)ios-chess-client:xcodebuild build(iPhone 15 Simulator, iOS 17.5) — EXIT:0Baseline note
This was tested against current
main(f80248c), which does not yet include #218 (LMR, still unmerged as of this writing). If/when both merge, the combined effect should be re-validated — LMR and SEE-pruned quiescence touch different parts of the same search and aren't guaranteed to be perfectly additive (LMR changes which nodes get reduced-depth looks in the main tree; SEE changes which captures quiescence bothers searching at all — plausible interactions either way).Scope / classification
ChessOnline, orChessKit/Sources/ChessOnline/Messages.swift.Board.swift,NegamaxEngine.swift,PersistentNegamaxEngine.swift,EngineLab/CLI.swift,EngineLab/SelfPlay.swift,EngineLabTests/BenchTests.swift(re-pin), and the two new test files. Does not touch PR Server: move the auth rate limiter to a shared sliding-window store (closes #79) #116 or the LMR branch (feature/lmr-search) — separate worktree, separate branch offorigin/main.Per repo convention, requesting the orchestrator review this — not merging it myself, and given the inconclusive result, it may be one for the owner to explicitly decide on rather than a routine merge.