Skip to content

Convert interior positions at the byte column under the negotiated encoding - #370

Merged
Firehed merged 12 commits into
mainfrom
slice/S1.5
Jul 24, 2026
Merged

Convert interior positions at the byte column under the negotiated encoding#370
Firehed merged 12 commits into
mainfrom
slice/S1.5

Conversation

@Firehed

@Firehed Firehed commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Slice

  • Slice: S1.5 — Position round-trip corpus (regression net)
  • Plan step: Execution Plan 0002, Step 1 (capability negotiation + encoding edge)
  • RFC: 0001-foundational-architecture.md §4.9 (Position Encoding); §8.1 invariant 9

Related

What and why

S1.2 (#366) fixed the conversion at the document boundary (TextDocument::offsetAt /
positionAt) but deferred the interior sites that still treated the wire
character as a byte length — a §4.9 violation ("interior components MUST operate
only on [the negotiated] representation"). This slice drives those out and adds the
regression net Step P deliberately excludes for the positional surface.

Inbound reads (text before the cursor) now slice at the byte column via a shared
TextDocument::textBeforeCursor() helper instead of substr($lineText, 0, $character):

  • CompletionHandler (classification / call-context text)
  • TextFallbackHelper::getMemberAccessContext
  • SymbolResolver::resolveVariableAccessWithAst

Outbound completion Range columns now size the typed prefix in the negotiated
encoding via a new Range::forPrefix(line, character, prefix, encoding) factory —
the one place the conversion lives — instead of $character - strlen($prefix):

  • ClassCandidates and NamespaceCandidates, which now receive the negotiated
    encoding through SessionCapabilitiesProvider (S1.2 kept it confined to the
    document boundary).

Why not native mb_ functions

Flagging this so it is not "simplified" later (and it is documented in
PositionEncoding): an LSP character is a UTF-16 code unit count, but the
native mb_ functions count codepoints. An astral codepoint (e.g. 😀,
U+1F389, four UTF-8 bytes) is one mb_ character but two UTF-16 code units (a
surrogate pair). So mb_strlen over-shortens a length and mb_substr mis-slices
whenever an astral codepoint is present — the exact defect this slice fixes. The
surrogate accounting in PositionEncoding::utf16Units() is what mb_ does not
provide; mb_str_split (codepoint split) is the one mb_ primitive that fits and
is already used. mb_convert_encoding(..., 'UTF-16LE', ...) could shorten the
length-only case but not the character → byte direction, so it would fork the
single shared mechanism for no net gain.

Acceptance criteria

  • Interior inbound reads no longer treat the wire column as a byte length; the
    repeated pattern is a shared TextDocument helper (byte column from the
    document's own encoding, no encoding threading).
  • Outbound completion Range start columns are measured in the negotiated
    encoding's code units, with PositionEncoding reachable at the completion
    sources.
  • A position round-trip corpus proves the boundary is a bijection across a file
    of mixed-width codepoints, and re-runs existing cursor-marker fixtures under
    the negotiated encoding.
  • Each corrected site is covered by a dedicated multibyte fixture/test.
  • composer test green (PHPStan + suite + PHPCS).

§4.9 enforcement

Per §8.1, invariant 9's mechanism is a test (multibyte round-trip at the boundary)
plus review of interior offset use — no new static rule. This slice supplies the
tests: PositionRoundTripCorpusTest, plus per-site multibyte cases across
TextDocumentTest, PositionEncodingTest, RangeTest, CompletionHandlerTest,
TextFallbackHelperTest, SymbolResolverTest, ClassCandidatesTest, and
NamespaceCandidatesTest.

Candidate closes

None attributed to S1.5 in the build manifest. See Related above for #192, which
this slice advances (its Closes sits with S1.2 in the manifest); the reviewer
decides whether it is now fully resolved.

Notes for review

SymbolResolver::resolveVariableAccessWithAst is a deep fallback: for the multibyte
scenarios tried, the AST path (which already uses the correct offsetAt) serves the
query, so its substr is corrected for consistency and kept covered by the existing
tests (16×) rather than by a contrived red case. The shared text-fallback prefix
behavior is proven red→green in TextFallbackHelperTest.

@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.61%. Comparing base (49e2797) to head (d81cb9c).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #370   +/-   ##
=========================================
  Coverage     97.60%   97.61%           
- Complexity     1692     1696    +4     
=========================================
  Files           105      105           
  Lines          4256     4271   +15     
=========================================
+ Hits           4154     4169   +15     
  Misses          102      102           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Firehed and others added 2 commits July 24, 2026 12:49
The corpus round-trip test only asserts a symmetric identity, so a
wrong-but-consistent per-codepoint unit count survives it. The 2- and
4-byte widths were pinned by concrete goldens; the 3-byte width was not,
so treating a 3-byte BMP codepoint as a surrogate pair passed the whole
suite. Pin it the same way.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The existing multibyte variable-access test resolves through the AST
node path, so it never exercises resolveVariableAccessWithAst's own
text slice; reverting that slice to the raw wire column left the test
green. Add a typed-parameter access inside an incomplete while() — a
shape that denies a clean AST node and routes through the fallback — on
a line carrying a multibyte character, so the byte-column slice is now
covered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Firehed
Firehed merged commit 46cacbc into main Jul 24, 2026
7 checks passed
@Firehed
Firehed deleted the slice/S1.5 branch July 24, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant