Replace scan extraction with @dittowords/text-extract - #150
Closed
jholiga wants to merge 1 commit into
Closed
Conversation
The CLI held its own copy of the extraction pipeline. That code now
ships as @dittowords/text-extract. Import the library and delete the
local copy.
Changes:
- Add @dittowords/text-extract as a dependency.
- Delete lib/src/scan/{extract,walk,rules,types}.ts and lib/src/scan/lang.
- Delete the tests for the deleted code. The library tests it.
- Import DittoScanCandidate, DittoScanExtractSummary and runExtract from
the library.
- Re-export the scan types from the library in lib/ditto.ts.
- Log the new summary.filesFailed count and the failure detail. A failed
file drops out of the results, so its strings are missing.
- Remove @ast-grep/napi, @ast-grep/lang-kotlin, @ast-grep/lang-swift,
globby and yaml. The library owns them now.
- Add lib/src/scan/extract.test.ts. It checks that the CLI resolves the
library and gets usable candidates from the test fixtures.
lib/src/scan/analyzeDirectories.ts stays. It is CLI-only code.
The output does not change. A scan of testfiles/ and of lib/ gives
candidates that are identical to the candidates from master.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The CLI held its own copy of the extraction pipeline. That code now ships as
@dittowords/text-extract. This PR imports the library and deletes the local copy.@dittowords/text-extract@^0.2.0as a dependency.lib/src/scan/{extract,walk,rules,types}.tsand all oflib/src/scan/lang/.DittoScanCandidate,DittoScanExtractSummaryandrunExtractfrom the library incommands/scan.ts,http/scan.tsandditto.ts.@ast-grep/napi,@ast-grep/lang-kotlin,@ast-grep/lang-swift,globbyandyamlfrompackage.json. The library owns them now.summary.filesFailedcount and the failure detail. A failed file drops out of the results, so a non-zero count means strings are missing.lib/src/scan/extract.test.ts. It checks that the CLI resolves the library and gets usable candidates fromtestfiles/.lib/src/scan/analyzeDirectories.tsstays. It is CLI-only code and the library does not have it.Net: -5,318 lines.
Context
The library is a superset of the deleted code. It adds a single-file entry point (
extractFile) and a failure count in the summary. The CLI only usesrunExtract, and that behaves the same.The
.d.tsbundle now re-exports the scan types from the library instead of inlining them.@dittowords/text-extractis a declared runtime dependency of the published package, so consumers resolve it.Screenshots
No visual changes.
Test Plan
Testing successfully completed locally via:
yarn test— 39 suites, 285 tests pass.npx tsc --noEmit --module preserve— no errors. (Plainnpx tsc --noEmitfails on a pre-existingtsconfig.jsonmodule/moduleResolutionmismatch, on master too.)node esbuild.mjs— the bundle andbin/ditto.d.tsbuild.ditto scan testfiles --local— the candidate set is byte-identical to master's, after sorting.ditto scan lib --local— the candidate set was byte-identical to master's before the source files were deleted.Out of scope
ignoreand@google/genaiare unused dependencies on master already. This PR leaves them.🤖 Generated with Claude Code