Skip to content

Normalize undefined-identifier eval error message - #3828

Draft
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixeval-stop-eval-error-messages-df277d
Draft

Normalize undefined-identifier eval error message#3828
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixeval-stop-eval-error-messages-df277d

Conversation

@posthog

@posthog posthog Bot commented Aug 11, 2026

Copy link
Copy Markdown

Problem

  • The eval team loses visibility: one class of user mistake mints a fresh error tracking issue every time, so a real regression in this code path is easy to miss.
  • When circuit code references a part number that was never imported (A_12401610E4_2A, TXB0104RGYR), the engine throws a ReferenceError and the catch at lib/eval/import-local-file.ts splices the identifier straight into the thrown message.
  • Part numbers are unique per project, so every project that makes the same mistake gets its own fingerprint and its own issue.
  • The browser splits it again: WebKit says Can't find variable: X while V8 and Gecko say X is not defined.
  • Nothing is broken for users — the eval runner catches genuine user-code errors correctly (mechanism.handled is true). The cost is observability noise, and it grows with every new project.

Changes

  • Normalize the message before it leaves the catch: keep a stable prefix and move the identifier into a trailing suffix.
  • Match both engine wordings so the WebKit and V8/Gecko variants land together.
- Error evaluating "entrypoint.tsx": A_12401610E4_2A is not defined
- Error evaluating "entrypoint.tsx": Can't find variable: TXB0104RGYR
+ Error evaluating "entrypoint.tsx": undefined identifier "A_12401610E4_2A"
+ Error evaluating "entrypoint.tsx": undefined identifier "TXB0104RGYR"
  • The identifier stays visible in the suffix for the person debugging.
  • Other eval-wrapper errors (for example "not exported by") keep their exact message.
  • Add tests/examples/example10-undefined-identifier.test.tsx.

Agent context

  • The wording changed once before in fe347fa, so both old and new phrasings appear in the data today. This change targets only the undefined-identifier family.
  • example09-not-defined-component.test.tsx asserts a "not exported by" message, which this change leaves unchanged; that test still passes.

Created with PostHog Desktop from this inbox report.

Undefined-identifier ReferenceErrors from user circuit code splice a project-unique identifier (often a part number) and a per-engine wording into the thrown message. Both make error tracking mint a fresh issue per project.

Normalize the message to a stable prefix "Error evaluating \"...\": undefined identifier" and keep the identifier in a trailing suffix so the family groups as one issue. Cover V8/Gecko ("X is not defined") and WebKit ("Can't find variable: X") wordings.

Add tests/examples/example10-undefined-identifier.test.tsx.

Generated-By: PostHog Code
Task-Id: 71e6ec3b-2ec6-42d8-aef0-acc23e44d3e5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants