Skip to content

Replace prebuild/prebuild-install with prebuildify + node-gyp-build#44

Open
arv wants to merge 3 commits into
mainfrom
arv/prebuildify
Open

Replace prebuild/prebuild-install with prebuildify + node-gyp-build#44
arv wants to merge 3 commits into
mainfrom
arv/prebuildify

Conversation

@arv

@arv arv commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Prompted by WiseLibs/better-sqlite3#655: prebuild-install drags in ~40 transitive packages, most of them deprecated (npmlog, gauge, rc, request, tar-fs@2, …), and downloads binaries from GitHub releases at install time. This switches to the recommended prebuildify + node-gyp-build pair.

What changes

  • Runtime deps: bindings + prebuild-install → just node-gyp-build (zero transitive deps). A fresh install (including all devDeps) drops from 461 to 282 packages.
  • Install: prebuilt binaries ship inside the npm package under prebuilds/<platform>-<arch>/; node-gyp-build picks the right one at require time. No network access, no GitHub-release download step. The install script keeps the exact same fallback chain as before (matching prebuild → existing build/Releasenode-gyp rebuild).
  • Per-ABI naming: we build with --no-napi --name node, producing node.abi<N>.node files (better-sqlite3 uses V8 APIs directly, so binaries are ABI-specific). Linux builds add --tag-libc so glibc and musl coexist in the same prebuilds/linux-<arch>/ dir; node-gyp-build matches the libc tag at runtime.
  • Shell: prebuild tarballs used to include the zero_sqlite3 shell via --include-regex. prebuildify only ships .node files, so deps/copy-shell.js copies the shell into prebuilds/ and shell.js falls back to that copy (skipped on musl, where the shell target isn't built — same as today).
  • CI: prebuild jobs upload prebuilds/ as artifacts (actions pinned to SHAs); the publish job merges all of them into the package before npm publish. The jobs no longer need contents: write.
  • node-gyp pinned as devDependency: removing prebuild also removed its modern node-gyp from node_modules/.bin, leaving the ancient node-gyp 8.4.1 hoisted from the sqlite3 benchmark devDep — which emits broken llvm-lib /LTCG flags with VS2022 on Node 24. Depending on node-gyp ^11 directly makes the build toolchain deterministic.

Verified locally (linux-x64, Node 22)

  • Full source build + all 338 tests pass.
  • prebuildify emits prebuilds/linux-x64/node.abi127.glibc.node; with build/ removed, the addon, the shell wrapper, and the node-gyp-build install script all resolve from prebuilds/ alone.
  • overrides: { prebuildify: { node-abi: ^4.25.0 } } gives the same ABI set as the old prebuild flow (115/127/131/137/141/147 for Node 20/22/23/24/25/26).

Trade-offs / notes

  1. Package size. All platforms ship in one tarball. Base package is ~11.5 MB unpacked; each stripped addon binary adds ~2.2 MB. The full matrix (~6 ABIs × 9 platform/arch/libc combos + 7 shell binaries) lands around ~140 MB unpacked, likely ~50 MB compressed. The old flow downloaded only the needed binary. If that's too much, the usual fix is per-platform optionalDependencies packages (esbuild-style).
  2. Upstream v13 would shrink this a lot. WiseLibs/better-sqlite3 v13.0.0 (2026-07-21) refactored to node-addon-api (N-API). After merging that (separate PR — upstream is 18 commits ahead of our last sync in Merge upstream (gh/WiseLibs/better-sqlite3) changes #42, including SQLite 3.53.3, Statement#toString(), Database#explain()), we could drop --no-napi and ship one ABI-stable binary per platform (~9 binaries, no per-Node-version builds, no GCC bullseye/bookworm split), bringing the tarball back to roughly today's size.
  3. Bun. Bun 1.3.11 fails to load node-targeted prebuilds (undefined symbol: v8::FunctionTemplate::InstanceTemplate) — its V8 shim doesn't cover the symbols our per-ABI binaries reference. This is not a regression: the current prebuild-install binaries are compiled identically and hit the same wall; Bun effectively requires a source build either way (which is what the test-bun CI job does). Upstream's N-API refactor would fix this properly.
  4. The release-only prebuild jobs can't be exercised by PR CI (they run on release events), so the artifact merge flow is untested until a release dry-run.

@socket-security

socket-security Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednode-gyp-build@​4.8.4991008782100

View full report

@arv
arv force-pushed the arv/prebuildify branch from f65ec68 to 120fdda Compare July 22, 2026 09:39
@arv
arv marked this pull request as ready for review July 22, 2026 09:40
@arv
arv requested review from Copilot and tantaman July 22, 2026 09:40
@arv arv changed the title WIP: Replace prebuild/prebuild-install with prebuildify + node-gyp-build Replace prebuild/prebuild-install with prebuildify + node-gyp-build Jul 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the native addon packaging and installation flow away from prebuild-install (and its large/deprecated dependency tree) to the recommended prebuildify + node-gyp-build approach, and updates the shell wrapper + CI to ship and publish prebuilds inside the npm tarball.

Changes:

  • Replace runtime native binding resolution (bindings) with node-gyp-build in library code and tests.
  • Add a prebuilds/** package payload and update shell.js to fall back to a packaged shell binary when present.
  • Rework the release CI to build prebuilds/, upload them as artifacts, and merge them into the publish job (instead of uploading/downloading release assets).

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
test/10.database.open.js Updates the nativeBinding test to use node-gyp-build’s resolved binding path.
shell.js Adds a prebuilds-based fallback lookup for the packaged shell binary.
package.json Switches deps/scripts to node-gyp-build + prebuildify and includes prebuilds/** in published files.
lib/database.js Loads the addon via node-gyp-build instead of bindings.
deps/copy-shell.js New helper to copy the built shell executable into prebuilds/<platform>-<arch>/.
.gitignore Ignores generated prebuilds/ output.
.github/workflows/build.yml Updates prebuild commands and publishes merged prebuilds/ artifacts on release.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread deps/copy-shell.js Outdated
arv added 3 commits July 22, 2026 11:50
Prompted by WiseLibs/better-sqlite3#655: prebuild-install pulls in a large
tree of deprecated transitive dependencies (npmlog, gauge, rc, request,
tar-fs 2, ...) and downloads binaries from GitHub releases at install time.

With prebuildify, all prebuilt binaries ship inside the npm package under
prebuilds/<platform>-<arch>/ and node-gyp-build (zero dependencies) resolves
the right one at runtime, so installs need no network access and no install
scripts on supported platforms.

- runtime deps drop from bindings + prebuild-install (~40 packages) to just
  node-gyp-build (0 transitive deps)
- install script is now plain node-gyp-build: use prebuild if compatible,
  else reuse existing build/Release, else compile (same fallback chain)
- binaries stay per-ABI (--no-napi) since better-sqlite3 uses V8 APIs
  directly; --name node emits canonical node.abi<N>[.<libc>].node names
- linux builds pass --tag-libc so glibc and musl coexist per arch
- the zero_sqlite3 shell (previously included in prebuild tarballs via
  --include-regex) is copied into prebuilds/ by deps/copy-shell.js, and
  shell.js falls back to the prebuilt copy
- CI prebuild jobs upload prebuilds/ artifacts instead of release assets;
  the publish job merges them into the package before npm publish
Removing prebuild dropped its modern node-gyp from node_modules/.bin,
leaving the ancient node-gyp 8.4.1 hoisted from the sqlite3 benchmark
devDependency. That version emits broken llvm-lib /LTCG flags with
VS2022 on Node 24 (CI: Testing Node 24 on windows-2022). Depend on
node-gyp ^11 explicitly so builds use a deterministic, current version.
- Pin actions/upload-artifact (v4.6.2) and actions/download-artifact
  (v4.3.0) to commit SHAs, consistent with the other actions in the
  workflow.
- Fail fast in deps/copy-shell.js when --arch is passed without a value
  instead of silently writing to prebuilds/<platform>-undefined/.
@arv
arv force-pushed the arv/prebuildify branch from 3d1b7e1 to cffb822 Compare July 22, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants