Skip to content

fts: fix stopword maintenance and transactional statistics - #42

Open
jkumz wants to merge 3 commits into
LadybugDB:mainfrom
jkumz:fts-bugs
Open

fts: fix stopword maintenance and transactional statistics#42
jkumz wants to merge 3 commits into
LadybugDB:mainfrom
jkumz:fts-bugs

Conversation

@jkumz

@jkumz jkumz commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes three FTS consistency bugs:

  • Index maintenance skipped configured stopword filtering when stemming was disabled.
  • A stopword-only document stopped the remainder of an insertion batch from being indexed.
  • FTS document statistics were retained after insert or delete transactions rolled back.

Changes

Apply consistent stopword filtering

Apply configured stopword filtering to conjunctive index-maintenance paths when stemmer := 'none'.

This keeps initial index construction, runtime insertion, and deletion consistent while preserving the existing no-stemming behavior for non-conjunctive query paths.

Preserve batch processing

Skip documents that produce no indexed terms without terminating the remaining batch.

The insertion and deletion paths now:

  • Count only documents that produce indexed terms.
  • Calculate average document length from indexed documents only.
  • Resolve indexed property vectors using their logical batch positions and selection vectors.
  • Continue processing documents after a stopword-only entry.

Make FTS statistics transactional

Accumulate document-count, total-document-length, and checkpoint-watermark changes per transaction.

  • Commit callbacks apply pending deltas to the committed aggregate.
  • Rollback callbacks discard pending deltas.
  • Queries read committed statistics plus their own transaction’s pending changes.
  • Checkpoint serialization snapshots committed statistics under synchronization.

Testing

  • Rebuilt lbug_fts_extension.
  • Passed focused default and custom stopword tests.
  • Passed COPY and UNWIND stopword-only batch tests.
  • Passed live and close/reopen deletion tests.
  • Passed insert and delete rollback metadata tests.
  • Passed focused update and recovery regressions.
  • Passed the FTS extension CTest targets: 2/2.
  • Passed clang-format --dry-run --Werror.
  • Passed git diff --check.

Fixes #39 #40 #41

jkumz added 3 commits August 1, 2026 17:07
- Filter configured stopwords during conjunctive no-stemmer maintenance paths
- Preserve the zero-work fast path for non-conjunctive no-stemmer queries
- Cover create, insert, mixed lifecycle, custom stopword, and reload regressions
- Continue indexing after stopword-only documents and count only indexed documents in FTS statistics
- Resolve indexed property vectors by logical batch position during term reconstruction
- Cover COPY and UNWIND stopword regressions with score stability after deletion
- Accumulate document counts, lengths, and checkpoint watermarks per transaction
- Apply metadata on commit and discard it safely on rollback
- Preserve read-your-own-writes scoring and repeated batch finalization
- Cover multi-insert and mixed delete/insert rollback regressions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: FTS maintenance skips stopword filtering when stemming is disabled

1 participant