feat: store duplicated file contents only once when packing

[MarshallOfSound]

What

Files whose contents hash identically are now written into the archive a single time. Each copy still gets its own header entry with its own size, integrity, and executable bit — only the offset is shared, so the redundant bytes are never written.

before                                 after
offset=     0 /LICENSE                 offset=     0 /LICENSE
offset= 28283 /vendor/pkg-a/LICENSE    offset=     0 /vendor/pkg-a/LICENSE
offset= 48362 /vendor/pkg-b/LICENSE    offset=     0 /vendor/pkg-b/LICENSE
offset= 20032 /lib/run-copy.sh         offset= 20032 /lib/run-copy.sh
offset= 28264 /run.sh (executable)     offset= 20032 /run.sh (executable)

How

Filesystem.storeFileEntry keeps a Map<sha256, offset> keyed on the hash that is already computed for every entry's integrity field, so detection costs no extra read and no extra hashing pass. On a hit the entry takes the stored offset and the archive does not grow; insertFile reports the duplicate, and that rides alongside the existing unpack flag through the file lists so both writers skip the payload. Small files also release their cached buffer immediately instead of holding it until the write pass, and the transform path deletes a duplicate's temp file instead of leaving it behind.

Unpacked files (unpack / unpackDir) are excluded — they are copied out as standalone files next to the archive, so every copy has to exist on disk.

Compatibility

Nothing in the format required offsets to be unique; readers consume offset + size per entry either way. Archives whose inputs contain no duplicates are byte-for-byte unchanged, and packing stays deterministic. An archive packed by this branch extracts correctly with the CLI built from main.

No new option or flag: it applies to every pack.

Benchmarks

2000 files × 4 KB, varying how much of the input is duplicated (before = main, after = this branch):

duplicate share	archive before	archive after	Δ size
0%	8.29 MB	8.29 MB	0.0%
10%	8.29 MB	7.51 MB	-9.4%
25%	8.29 MB	6.34 MB	-23.5%
50%	8.29 MB	4.39 MB	-47.1%
75%	8.29 MB	2.43 MB	-70.6%
90%	8.29 MB	1.26 MB	-84.8%

Savings track the duplicated share of bytes, which is the ceiling. A hoisted node_modules (3910 files / 181 MB) only shrinks 0.4%, since the package manager already deduplicates; nested installs and repeated assets are where the win lands.

Pack time, interleaved A/B with a before-vs-before control run each iteration to establish the noise floor:

fixture	pack before	pack after	Δ pack	noise floor
duplicate-heavy (3000 files, 50% dup)	168.0 ms	167.9 ms	-0.0%	±4.2%
medium (500 files)	28.8 ms	28.1 ms	-2.5%	±4.1%
many-small-files (10000 files)	426.8 ms	433.8 ms	+1.6%	±3.7%
few-large-files (20 files, 20 MB)	50.8 ms	50.5 ms	-0.7%	±1.1%
large (5000 files, 39 MB)	350.1 ms	350.0 ms	-0.0%	±1.4%

Every delta sits inside the noise floor. On the worst case for this change — zero duplicates, pure map overhead — three repeat runs gave +3.5%, -2.6%, -1.9% against a ±2.7% floor, so the overhead is not measurable. Pack time does not improve even at 50% duplication: every file still has to be hashed to know it is a duplicate, and skipped writes land in the page cache anyway. The win is bytes on disk and the reads of them, not wall clock.

benchmark/generate-fixtures.ts gains a duplicateRatio knob and a duplicate-heavy fixture so this is reproducible.

Testing

10 new tests: shared offsets, files larger than the 2 MB buffer-hash threshold (streaming path), same-size-different-content, executable-bit divergence between copies, unpacked copies still all written, dedup computed on transformed bytes, the streams API, and empty files.

Also driven by hand through the CLI: pack/extract/diff -r round trips with --unpack, --unpack-dir, --ordering, symlinks pointing at deduped files, and 3 × 3 MB identical files (9 MB in → 3146492 bytes out).

---

Generated by Claude Code

[electron-npm-package-publisher]

🎉 This PR is included in version 4.3.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀