Skip to content

Replace scan extraction with @dittowords/text-extract - #150

Closed
jholiga wants to merge 1 commit into
masterfrom
feat/use-text-extract-library
Closed

Replace scan extraction with @dittowords/text-extract#150
jholiga wants to merge 1 commit into
masterfrom
feat/use-text-extract-library

Conversation

@jholiga

@jholiga jholiga commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Overview

The CLI held its own copy of the extraction pipeline. That code now ships as @dittowords/text-extract. This PR imports the library and deletes the local copy.

  • Add @dittowords/text-extract@^0.2.0 as a dependency.
  • Delete lib/src/scan/{extract,walk,rules,types}.ts and all of lib/src/scan/lang/.
  • Delete the 19 test files for the deleted code. The library tests it.
  • Import DittoScanCandidate, DittoScanExtractSummary and runExtract from the library in commands/scan.ts, http/scan.ts and ditto.ts.
  • Remove @ast-grep/napi, @ast-grep/lang-kotlin, @ast-grep/lang-swift, globby and yaml from package.json. The library owns them now.
  • Log the library's new summary.filesFailed count and the failure detail. A failed file drops out of the results, so a non-zero count means strings are missing.
  • Add lib/src/scan/extract.test.ts. It checks that the CLI resolves the library and gets usable candidates from testfiles/.

lib/src/scan/analyzeDirectories.ts stays. It is CLI-only code and the library does not have it.

Net: -5,318 lines.

Context

The library is a superset of the deleted code. It adds a single-file entry point (extractFile) and a failure count in the summary. The CLI only uses runExtract, and that behaves the same.

The .d.ts bundle now re-exports the scan types from the library instead of inlining them. @dittowords/text-extract is a declared runtime dependency of the published package, so consumers resolve it.

Screenshots

No visual changes.

Test Plan

Testing successfully completed locally via:

  • yarn test — 39 suites, 285 tests pass.
  • npx tsc --noEmit --module preserve — no errors. (Plain npx tsc --noEmit fails on a pre-existing tsconfig.json module/moduleResolution mismatch, on master too.)
  • node esbuild.mjs — the bundle and bin/ditto.d.ts build.
  • ditto scan testfiles --local — the candidate set is byte-identical to master's, after sorting.
  • ditto scan lib --local — the candidate set was byte-identical to master's before the source files were deleted.

Out of scope

ignore and @google/genai are unused dependencies on master already. This PR leaves them.

🤖 Generated with Claude Code

The CLI held its own copy of the extraction pipeline. That code now
ships as @dittowords/text-extract. Import the library and delete the
local copy.

Changes:

- Add @dittowords/text-extract as a dependency.
- Delete lib/src/scan/{extract,walk,rules,types}.ts and lib/src/scan/lang.
- Delete the tests for the deleted code. The library tests it.
- Import DittoScanCandidate, DittoScanExtractSummary and runExtract from
  the library.
- Re-export the scan types from the library in lib/ditto.ts.
- Log the new summary.filesFailed count and the failure detail. A failed
  file drops out of the results, so its strings are missing.
- Remove @ast-grep/napi, @ast-grep/lang-kotlin, @ast-grep/lang-swift,
  globby and yaml. The library owns them now.
- Add lib/src/scan/extract.test.ts. It checks that the CLI resolves the
  library and gets usable candidates from the test fixtures.

lib/src/scan/analyzeDirectories.ts stays. It is CLI-only code.

The output does not change. A scan of testfiles/ and of lib/ gives
candidates that are identical to the candidates from master.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jholiga jholiga closed this Aug 14, 2026
@jholiga
jholiga deleted the feat/use-text-extract-library branch August 14, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant