Skip to content

replace InCB_Extend_table search with simple predicates - #178

Merged
Manishearth merged 1 commit into
unicode-rs:masterfrom
cometkim:drop-incb-table
Jul 31, 2026
Merged

replace InCB_Extend_table search with simple predicates#178
Manishearth merged 1 commit into
unicode-rs:masterfrom
cometkim:drop-incb-table

Conversation

@cometkim

Copy link
Copy Markdown
Contributor

Context: #177

(I wrote it without using the strategy of folding InCB=Consonant into the cats enum to make changes small and isolated)


According to UCD, the property InCB=Extend is defined as:

\p{InCB = Extend} ≔ [
\p{gcb=Extend}
\p{gcb=ZWJ}
- \p{InCB=Linker}
- \p{InCB=Consonant}
- [\u200C]
]

And no InCB=Consonant is gcb=Extend or gcb=ZWJ. So in can be determined with the grapheme category the caller already has and two equality.

scripts/unicode.py now checks that derivation against the UCD. A future Unicode version that breaks either fact fails the generator instead of silently mis-segmenting Indic text.

Also one fact that is InCB=Linker and InCB=Extend are subset of gcb=Extend, any other category rules out both without inspecting the codepoint.

This change improves the overall performance of the grapheme APIs by skipping unnecessary category checks and table searches in the hot loop. On my machine, it measures approximately -15 to -20%.

According to [UCD](https://www.unicode.org/reports/tr44/),
the property `InCB=Extend` is defined as:
```
\p{InCB = Extend} ≔ [
\p{gcb=Extend}
\p{gcb=ZWJ}
- \p{InCB=Linker}
- \p{InCB=Consonant}
- [\u200C]
]
```

And no `InCB=Consonant` is `gcb=Extend` or `gcb=ZWJ`.
So in can be determined with the grapheme category the caller already has and two equality.

`scripts/unicode.py` now checks that derivation against the UCD.
A future Unicode version that breaks either fact fails the generator
instead of silently mis-segmenting Indic text.

Also one fact that is `InCB=Linker` and `InCB=Extend` are subset of
`gcb=Extend`, any other category rules out both without inspecting the codepoint.

This change improves the overall performance of the grapheme APIs
by skipping unnecessary category checks and table searches in the hot loop.
On my machine, it measures approximately -15 to -20%.

@Manishearth Manishearth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks! These properties do change often, but the python code's preemptive check is good and also pretty clear about how to change things.

@Manishearth
Manishearth merged commit 091fb72 into unicode-rs:master Jul 31, 2026
4 checks passed
@cometkim

Copy link
Copy Markdown
Contributor Author

Thanks for the review! next up will be also similar approach and size.

@cometkim
cometkim deleted the drop-incb-table branch July 31, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants