OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms - #1167
Draft
krickert wants to merge 14 commits into
Draft
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167krickert wants to merge 14 commits into
krickert wants to merge 14 commits into
Conversation
krickert
changed the title
krickert
changed the title
krickert
force-pushed
the
wordnet-expansion
branch
6 times, most recently
from
July 17, 2026 20:45
5db6a8a to
917e4be
Compare
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 19, 2026 11:41
1b90059 to
f4cc90c
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 24, 2026 19:27
ff066b0 to
b1aef66
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
The clean-room CFSA2/FSA5 readers and the PoliMorf lemmatizer had no node because until today they had no branch: they were written on the OPENNLP-1887 line under formats: and lemmatizer: titles and were only visible as a sentence in that pull request's notes. They now sit on their own branch off current main and get a node and a row like every other staged feature. The note on apache#1167 says what is actually true: the commits were copied out, and dropping them from that branch is a separate rebase that has not happened.
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 31, 2026 10:38
89a5a27 to
840d4fd
Compare
…s and a Morphy lemmatizer Adds the LexicalKnowledgeBase contract in opennlp.tools.wordnet and the opennlp-wordnet module implementing it twice: WnLmfReader for WN-LMF XML and WndbReader for the legacy WNDB database files, with reader-equivalence coverage over miniature fixtures of both formats. The WN-LMF reader skips DOCTYPE declarations unresolved with DTD support off, so Open English WordNet releases parse unmodified while entity expansion stays closed; the WNDB fixtures are pinned to LF so their embedded byte offsets survive Windows checkout. The Morphy lemmatizer resolves inflected forms through suffix rules and the format's exception lists. Null arguments fail loudly with IllegalArgumentException, malformed data raises the checked InvalidFormatException, and the public seam carries no brand name: WordNet stays in the names of the classes that actually read WordNet formats.
Add docbkx/wordnet.xml, wire it into the manual, and add WordNetUsageExampleTest asserting the lookups the chapter prints.
…ment with the review conventions
…undary, pin WNDB rejections - Narrow the LexicalKnowledgeBase javadoc so the interface stops prescribing what only an implementation can promise: lemma matching semantics and thread safety are now stated as implementation specific and documented where they hold, on InMemoryWordNetLexicon, which already carries @threadsafe and describes the folding it applies. - Reword the contains() javadoc to say plainly that the default implementation delegates to lookup(), instead of speculating about cheaper overrides. - Move the null-element checks in MorphyLemmatizer up to the public lemmatize() overloads, both the array form and the list form, so validation happens once at the boundary the caller sees; the private lemmasOf() no longer repeats them and now documents that its arguments are validated by the caller. - Reject a null argument in LemmaFolding.splitOnSpaces() rather than letting it fail later as a NullPointerException, and capitalize the fold() message so it matches the wording the other validators use. - Document the throws clauses that the explicit validation adds, on LemmaKey.of() and on splitOnSpaces(). - Extract the repeated WN-LMF attribute names into ID_ATTRIBUTE, PART_OF_SPEECH_ATTRIBUTE, REL_TYPE_ATTRIBUTE, and TARGET_ATTRIBUTE, and the shared error opening into MALFORMED_PREFIX, so the element handlers stop repeating string literals. - Extract the WNDB offset failure detail into MALFORMED_OFFSET, shared by the length check and the digit check. - Fold the duplicated WNDB message construction into malformedMessage(), so the tokenizer builds the text directly instead of constructing an InvalidFormatException only to read getMessage() back off it. - Drop the redundant fileName parameter from WndbReader.readAll(), which already names the full path it failed to open. - Reduce the visibility of the Parser helpers in WnLmfReader: malformed() and line() are now private instance methods like every other helper in that class. - Make MorphyLemmatizer.rulesFor() an instance method for the same reason, so the lemmatizer's private helpers are consistent. - Add the missing javadoc on the RELATION_NAMES and POINTER_SYMBOLS lookup tables and on both RawSynset holders, the last undocumented members in the readers. - Correct two stale comments: the build() comment now points at memberLemmas(), where the synset and member part-of-speech agreement is really checked, and the mutate() comment in the tests states the actual constraint, that an edit which changes a line's length is only safe when the reader is expected to fail on that line before it reads the ones after it. - Add a parameterized WNDB test pinning eight field-level rejections that had no coverage: the offset length and digit checks, the synset and index part of speech mismatches, the base-16 word count field, the minimum word count, the pointer pos, the gloss separator, and the syntactic marker. - Add pinning tests for the newly explicit validation: splitOnSpaces() on null, and the list lemmatize() overload with a null token and with a null tag. - Share the fixtures instead of duplicating them: WndbReaderTest now exposes DOG_ID, CANID_ID, and its fixtureDirectory(), WnLmfReaderTest exposes fixture(), and LexiconConcurrencyTest and WordNetUsageExampleTest use those instead of their own loader copies and hardcoded ids. - Document the package-private test fixture helpers and switch WordNetUsageExampleTest to static assertion imports, matching the other tests in the module.
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
August 9, 2026 13:09
af19962 to
be24524
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 9, 2026
Adds the symbol-joiner and dehyphenation rows, marks the apache#1167 restack and the morfologik reconciliation, and moves the state line to fc9824a.
…reader-test gaps Add a Navigating relations section to the WordNet chapter and mirror it, pin the chapter's Path-based loading listing with a usage-test method, give the exception fixture a two-base-form entry (axes to axis, ax) so the list-form lemmatizer path is exercised with multiple candidates, and fold the copy-paste WN-LMF rejection tests into one parameterized ladder with identical documents and message pins.
… definition A blank check under the toolkit's whitespace definition, which unlike String.isBlank covers the no-break spaces, so annotators validating labels and identifiers share one predicate instead of each carrying a private copy. Reads whole code points; tests pin the no-break and figure spaces, the empty string, and a supplementary-plane letter.
…nsion over the lexical knowledge base Expands a term into the synonyms sharing its synsets, the lemmas of its hypernym ancestors up to a configured depth (following both the direct and the instance relation), and optionally its direct hyponyms. Each expansion carries a deterministic heuristic weight: sense rank and every relation step multiply configurable decays, so consumers can discount looser expansions instead of treating them as the original term. Results exclude the input, deduplicate case-insensitively keeping the highest weight, and order stably by weight, kind, and term. Inflected input works through an optional Lemmatizer invoked with the WordNetPos name as the tag, which the Morphy lemmatizer understands: dogs expands through dog, mice through the exception list to mouse. Hypernym walks are visited-checked so malformed cyclic data terminates. Tests cover the graph behavior on a hand-built lexicon (sense ranking, decay arithmetic, dedup, cycle termination, validation) and run the whole stack over the miniature WN-LMF and WNDB fixtures, asserting both readers produce identical expansions.
…yping SynsetSimilarity scores noun synset pairs with the path, Wu-Palmer, and Leacock-Chodorow measures over the knowledge base seam. HypernymTyper labels a word by walking its senses' hypernym and instance-hypernym chains to the nearest caller-registered anchor concept, so a knowledge base can type names as person, organization, or location without a model. Blank checks follow the toolkit whitespace definition.
…examples Add the lexical expansion section to docbkx/wordnet.xml and LexicalExpansionUsageExampleTest asserting the expansion values the chapter prints; carry the WordNet usage example alongside it.
…ment with the review conventions
…and default behavior Show the expansion and similarity taxonomies in the chapter's listing comments so every synset id and word in the listings exists in the shown lexicon, add a hypernym-anchored typing section, and extend the mirror test to build exactly those taxonomies and assert every printed value. Pin the formula-driven numeric regimes (Wu-Palmer zero at the root pair, negative Leacock-Chodorow under an understated depth budget) and the expander builder's accepting boundaries and defaults (senseDecay 1.0, hypernymDepth 0, maxSenses 3, maxExpansions 20) against fixture lemmas that exceed the defaults.
krickert
force-pushed
the
wordnet-expansion
branch
from
August 10, 2026 10:24
be24524 to
3dddfed
Compare
…m no ancestry city (n8) and company (n10) share only the taxonomy root, yet wuPalmer returns 0.0 for them, the same score as the disconnected chemist/abstract pair. The documented contract reserves 0 for pairs with no shared ancestor. Red: testWuPalmerDistinguishesRootOnlyAncestryFromNoAncestry fails with 'city and company share entity; expected positive Wu-Palmer, got 0.0'.
…ores positive The edge-counted formula gave the taxonomy root depth zero, so pairs whose only shared ancestor is the root collapsed to 0.0 and were indistinguishable from pairs sharing no ancestor, violating the documented contract that 0 is reserved for no shared ancestor. Wu and Palmer's node counting places the root at depth one; the formula 2 * depth(lcs) / (depth(a) + depth(b)) keeps its shape, the zero denominator guard becomes unreachable, and (root, root) self-similarity rises from the pinned 0.0 to a full 1.0. Under the corrected convention wuPalmer(n5, n6) moves from 8/9 to 10/11; the pinned tests, the usage example mirror test, and the manual's similarity listing and prose are updated to match.
krickert
added a commit
that referenced
this pull request
Aug 15, 2026
# Conflicts: # opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java # opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #1155 (wordnet-api): this branch builds on that PR's LexicalKnowledgeBase seam, and since that head lives on a fork it cannot be this PR's base ref. Until #1155 merges, the diff here shows its commits too; only the commits from 1404a50 onward are this PR. One of them (082592d) is a carried copy of the OPENNLP-1888 StringUtil.isBlank utility this code uses; it drops out automatically once that lands. After #1155 lands, the branch rebases onto main and the diff collapses to this change alone.
Adds LexicalExpander to the opennlp-wordnet module: given a term, it produces weighted related terms from a LexicalKnowledgeBase, the synonyms sharing its synsets, the lemmas of hypernym ancestors up to a configured depth (following both the direct and the instance relation, visited-checked so cyclic data terminates), and optionally direct hyponyms. Each expansion carries a deterministic heuristic weight (sense-rank decay times depth decay, both configurable), results are deduplicated case-insensitively keeping the highest weight, the input term is never returned, and ordering is stable. An optional Lemmatizer fallback expands inflected input; the Morphy lemmatizer of this module plugs in directly, so dogs expands via dog and mice via mouse.
Tests cover the behavior over a hand-built graph with a controlled shape (sense ranking, depth decay, dedupe, cycle termination, hyponym opt-in, configuration validation) plus integration over the miniature WN-LMF and WNDB fixtures asserting both readers expand identically.