Skip to content

Validate out-of-order tables incrementally while parsing#509

Open
gaoflow wants to merge 1 commit into
python-poetry:masterfrom
gaoflow:fix-479-quadratic-validation
Open

Validate out-of-order tables incrementally while parsing#509
gaoflow wants to merge 1 commit into
python-poetry:masterfrom
gaoflow:fix-479-quadratic-validation

Conversation

@gaoflow

@gaoflow gaoflow commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Fixes #479.

Parsing n dotted keys (or table headers) that share a prefix is super-cubic: every append to an out-of-order key triggers OutOfOrderTableProxy.validate, which re-merges — and deep-copies — every earlier fragment into a fresh temp container, and the membership probe in append() rebuilds the full proxy each time as well. Profiling the issue's payload shows ~13M deepcopy calls for 60 keys.

Three changes, all confined to parse time (_parsed containers):

  • _validate_out_of_order_table keeps a per-key cache of (fragments validated so far, temp container) and resumes from there, so each fragment is merged and deep-copied exactly once. Fragments are only appended during parsing; any other mutation (remove, _remove_at, _replace_at) clears the cache, and a validation failure drops the entry so a caught-and-retried error can't resume from a partially merged container.
  • __contains__ skips rebuilding the proxy when the cache shows exactly the current fragments already validated clean.
  • The no-key validation sweep iterates a maintained set of out-of-order keys instead of every key in the map (the per-key path still re-checks isinstance(idx, tuple), so stale entries are harmless).

Benchmark (the issue's repro):

keys before after
40 1.3 s 5.5 ms
80 15.6 s 10 ms
160 219 s 19 ms
1000 0.15 s

Error behavior is unchanged: collisions across fragments still raise at parse time (added tests), and the existing suite passes unmodified.

@gaoflow gaoflow force-pushed the fix-479-quadratic-validation branch from 191e8d2 to eb798da Compare June 12, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Security] Quadratic complexity DoS when parsing dotted keys with shared prefix (CWE-400)

1 participant