Skip to content

[#860] Serve JDBC cursor repositioning from the fetched buffer and grow batches adaptively - #863

Merged
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/860-jdbc-cursor-refetch
Aug 12, 2026
Merged

[#860] Serve JDBC cursor repositioning from the fetched buffer and grow batches adaptively#863
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/860-jdbc-cursor-refetch

Conversation

@vharseko

Copy link
Copy Markdown
Member

Fixes #860 (reported in discussion #859: ~52 MB/s of database network traffic on a ~13k-entry directory).

Problem

JDBCStorage.CursorImpl fetched eager batches of 1000 (k,v) rows and every positionToKeyOrNext() discarded the buffer and re-fetched a fresh batch — even when the requested key was already buffered. DN2ID.ChildrenCursor repositions once per child, so a one-level search enumerating N children transferred N × 1000 rows instead of ~N. hasSubordinates() and subtree delete hit the same pattern.

Changes

  • positionToKeyOrNext() is served from the already-fetched buffer: buffered rows are the contiguous sorted rows following the current one (byte order matches the database binary collation), so forward repositioning within the fetched range runs no SQL at all. The ChildrenCursor pattern collapses from N × 1000 rows to ~N rows total.
  • Adaptive batch sizing: cursors start at org.openidentityplatform.opendj.jdbc.fetchsize.initial (default 32) and grow geometrically to fetchsize (default 1000) while reads stay sequential; a jump outside the buffered range resets the size. Most cursors read only a handful of rows (hasSubordinates reads 1–2), so eager 1000-row fetches were pure overhead.
  • Oracle: create the cursor index on (k) in openTree — previously only PostgreSQL and MySQL got it, leaving Oracle keyset-pagination queries (where k>? order by k) unserved by the (h,k) primary key. MSSQL keeps no index: varbinary(max) cannot be an index key column (documented in a comment).
  • isExistsIndex now calls getIndexInfo with approximate=true — with false the Oracle driver runs ANALYZE on every metadata check.

Expected effect

Scenario Before After
One-level search, N children (flat tree) N × 1000 rows ~N rows
One-level search, deep tree (miss per child) N × 1000 rows N × 32 rows
hasSubordinates per returned entry 1000 rows 32 rows
Subtree scope set (~4000 rows) 5 fetches ~7 fetches, same volume

Tests

  • testPositionToKeyOrNextServedFromBuffer — asserts via a fetch counter that repositioning within the buffer runs no SQL, batch growth follows the 2→8 schedule, and a ChildrenCursor-style sibling scan over 39 rows takes ≤8 fetches (38 before the fix).
  • testCursorKeyOrderIsUnsigned{0x7F} vs {0x80,0x01} keys verify the database collates keys in unsigned byte order, which the buffer-serving logic relies on.
  • Ran PgSqlTestCase and MySqlTestCase locally: 37 tests each, 0 failures. Oracle/MSSQL are covered by CI.

…etched buffer and grow batches adaptively

Every positionToKeyOrNext() discarded the buffer and re-fetched a full
"fetchsize" batch (1000 rows), so DN2ID.ChildrenCursor transferred
N x 1000 rows to enumerate N children. Forward repositioning within the
already-fetched range is now served from the buffer without SQL, and
batches start at "fetchsize.initial" (32) growing geometrically to
"fetchsize" on sequential reads. Also create the cursor index on (k)
for Oracle and stop the Oracle driver running ANALYZE in metadata checks.
@vharseko
vharseko requested a review from maximthomas August 12, 2026 06:56
@vharseko vharseko added jdbc performance Performance / concurrency / lock-contention work bug tests Test suites: fixing, enabling, un-disabling labels Aug 12, 2026
@vharseko
vharseko merged commit ea44821 into OpenIdentityPlatform:master Aug 12, 2026
18 checks passed
@vharseko
vharseko deleted the fix/860-jdbc-cursor-refetch branch August 12, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug jdbc performance Performance / concurrency / lock-contention work tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JDBC backend: cursor repositioning re-fetches full 1000-row batches, causing excessive database network traffic

2 participants