Remove extractor logic from CLI package and use the extractor from @dittowords/text-extract library

[jholiga]

Overview

This PR removes the extractor code from the CLI. The CLI now calls the @dittowords/text-extract package instead.

Changes:

• Add @dittowords/text-extract@^0.2.0 as a dependency.
• Delete lib/src/scan/extract.ts, walk.ts, rules.ts and types.ts.
• Delete the lib/src/scan/lang/ directory containing all the extractors.
• Delete the test files that cover the deleted code (the tests were moved to the package too).
• Import runExtract, DittoScanCandidate and DittoScanExtractSummary from
  the package.
• Re-export the scan types from the package in lib/ditto.ts.
• Add lib/src/scan/extract.test.ts. This test confirms that the CLI resolves
  the package.
• Remove extract/scan-related dependencies that the CLI no longer uses. See the table below.

Removed dependencies

Dependency	Reason
@ast-grep/napi	The package owns it now.
@ast-grep/lang-kotlin	The package owns it now.
@ast-grep/lang-swift	The package owns it now.
globby	The package owns it now.
yaml	The package owns it now.
@google/genai	Already unused on master.

Logging change

Previously, runExtract wrote each failure to stderr at the moment of the failure.

The package returns summary.filesFailed and summary.failures instead, and logExtractSummary now prints them.

Two limits apply to the new report:

• The filesFailed count stays exact.
• logExtractSummary runs at the end of the command. On the remote path it runs only after initiateClassify succeeds. A classify failure hides the report.

This is the only behavior change. The candidate output does not change.

Context

The extractor code was originally in the CLI repo, but we ported it to a standalone package so that the Ditto app can more easily reuse it for work on local flows. A future GitHub scan originating from the main app's UI can re-use the same code.

Testing

I scanned the v0-demo repository twice, first against current master and then again on this branch. v0-demo covers a good basis of extractor functionality (at least as far as TS goes). I also ran the same check against a couple other smaller test repos.

The two scans resulted in the exact same 182 candidatese.

Check	Result
Candidate count	182 = 182
Exact payload match, every field	YES. 0 only in before, 0 only in after.
Candidate id set	YES
Tally by detection_kind	MATCH. other 75, resource_value 60, markup_attr 31, markup_text 16.
Tally by language	MATCH. tsx 111, json_i18n 60, javascript 4, html 4, typescript 3.
Tally by locale_key	MATCH. null 122, en 30, es 30.
Tally by location.file	MATCH across all 19 files
stderr summary, timings stripped	IDENTICAL
--list-directories breakdown	IDENTICAL
Duplicate ids	0 in both

Screenshots

Test Plan

Setup

• Check out this branch.
• Run yarn install. Confirm that yarn installs
  @dittowords/text-extract.

Unit tests

• Run yarn build and confirm build passes.
• Run yarn test. Confirm all unit tests pass.

Scan a repository

• Run yarn scan <path-to-a-frontend-repo> --local --out-dir /tmp/scan-app.
• Navigate to /tmp/scan-app and open summary.ndjson. Confirm that the summary reports the correct framework tokens, for example react, next.
• Confirm that the command emits candidates (candidates.ndjson) and reports no failed files.

Compare same repo against master

• Check out master. Run yarn install and yarn scan <path-to-a-frontend-repo> --local --out-dir /tmp/scan-master.
• Run sort /tmp/scan-master/candidates.ndjson > /tmp/a.
• Run sort /tmp/scan-v0/candidates.ndjson > /tmp/b. (need to do these sorts because the globby package's file traversal is non-deterministic due to parallelization)
• Run diff /tmp/a /tmp/b. Confirm that the command reports no difference.

[laurakoye]

All the testing steps were good except there is no summary.ndjson specifically, but I did see: [ditto-cli scan][extract] framework: react output in stderr

[laurakoye]

All the testing steps were good except there is no summary.ndjson specifically, but I did see: [ditto-cli scan][extract] framework: react output in stderr