[#860] Serve JDBC cursor repositioning from the fetched buffer and grow batches adaptively - #863
Merged
vharseko merged 1 commit intoAug 12, 2026
Conversation
…etched buffer and grow batches adaptively Every positionToKeyOrNext() discarded the buffer and re-fetched a full "fetchsize" batch (1000 rows), so DN2ID.ChildrenCursor transferred N x 1000 rows to enumerate N children. Forward repositioning within the already-fetched range is now served from the buffer without SQL, and batches start at "fetchsize.initial" (32) growing geometrically to "fetchsize" on sequential reads. Also create the cursor index on (k) for Oracle and stop the Oracle driver running ANALYZE in metadata checks.
maximthomas
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #860 (reported in discussion #859: ~52 MB/s of database network traffic on a ~13k-entry directory).
Problem
JDBCStorage.CursorImplfetched eager batches of 1000(k,v)rows and everypositionToKeyOrNext()discarded the buffer and re-fetched a fresh batch — even when the requested key was already buffered.DN2ID.ChildrenCursorrepositions once per child, so a one-level search enumerating N children transferred N × 1000 rows instead of ~N.hasSubordinates()and subtree delete hit the same pattern.Changes
positionToKeyOrNext()is served from the already-fetched buffer: buffered rows are the contiguous sorted rows following the current one (byte order matches the database binary collation), so forward repositioning within the fetched range runs no SQL at all. TheChildrenCursorpattern collapses from N × 1000 rows to ~N rows total.org.openidentityplatform.opendj.jdbc.fetchsize.initial(default 32) and grow geometrically tofetchsize(default 1000) while reads stay sequential; a jump outside the buffered range resets the size. Most cursors read only a handful of rows (hasSubordinatesreads 1–2), so eager 1000-row fetches were pure overhead.(k)inopenTree— previously only PostgreSQL and MySQL got it, leaving Oracle keyset-pagination queries (where k>? order by k) unserved by the(h,k)primary key. MSSQL keeps no index:varbinary(max)cannot be an index key column (documented in a comment).isExistsIndexnow callsgetIndexInfowithapproximate=true— withfalsethe Oracle driver runsANALYZEon every metadata check.Expected effect
hasSubordinatesper returned entryTests
testPositionToKeyOrNextServedFromBuffer— asserts via a fetch counter that repositioning within the buffer runs no SQL, batch growth follows the 2→8 schedule, and aChildrenCursor-style sibling scan over 39 rows takes ≤8 fetches (38 before the fix).testCursorKeyOrderIsUnsigned—{0x7F}vs{0x80,0x01}keys verify the database collates keys in unsigned byte order, which the buffer-serving logic relies on.PgSqlTestCaseandMySqlTestCaselocally: 37 tests each, 0 failures. Oracle/MSSQL are covered by CI.