feat: pure-Go SQLCipher encryption (zero modernc dep)

README.md

@@ -4,35 +4,33 @@
 
 Dual-backend SQLite driver for the Hanzo ecosystem. Registers the
 `database/sql` driver name **`sqlite`** under both build configurations and
-exposes the same API either way:
+exposes the same API either way. **Both backends encrypt at rest, in the same
+SQLCipher-4 format** — a database written by one opens under the other.
 
 | Build | Backend | Encryption | Use |
 |-------|---------|------------|-----|
-| `CGO_ENABLED=1` + `-tags libsqlite3` + libsqlcipher | mattn/go-sqlite3 → SQLCipher | AES-256 page-level, at rest | **production** |
-| `CGO_ENABLED=0` | modernc.org/sqlite (pure Go) | none | CI tests / lint / local dev |
+| `CGO_ENABLED=0` (default) | vendored pure-Go engine + **hanzoai/sqlcipher** codec VFS | AES-256 page-level, at rest | **default** — CI, tests, pure-Go deploys |
+| `CGO_ENABLED=1` + `-tags libsqlite3` + libsqlcipher | **hanzoai/csqlite** → SQLCipher | AES-256 page-level, at rest | the C engine, for speed |
 
-One import, one driver name, two backends:
+One import, one driver name, two backends, one format:
 
 ```go
 import _ "github.com/hanzoai/sqlite" // registers "sqlite" under both tags
 
-db, _ := sql.Open("sqlite", dsn)     // mattn+SQLCipher (cgo) or modernc (!cgo)
+db, _ := sql.Open("sqlite", dsn)     // pure-Go codec VFS (!cgo) or csqlite+SQLCipher (cgo)
 ```
 
-The pure-Go backend **cannot encrypt**. Demanding a key on it
-(`Open(path, WithKey(...))`, `OpenDB(path, key)`) returns
-`ErrEncryptionUnavailable` and writes nothing — it never silently persists
-plaintext.
+The pure-Go backend **always encrypts** — no cgo, no external C library, nothing
+to link, and **`go list -m all` shows zero `modernc.org/*`** (the engine is
+vendored in-tree). A keyed database is single-writer, single-process (WAL on an
+in-process wal-index).
 
-## Building the encrypted (production) backend — READ THIS
+## Building the CGO backend (optional, for the C engine's speed)
 
-mainline `mattn/go-sqlite3` has **no `sqlcipher` build tag** and no
-`sqlite3_key()` binding. SQLCipher works only when you:
-
-1. link the **system** sqlite (the `libsqlite3` tag) against **libsqlcipher**, and
-2. enable the codec + URI keying via CGO flags, and
-3. supply the key as SQLCipher's **native URI `key` parameter** so it is applied
-   inside `sqlite3_open_v2` — *before* mattn's pragma battery runs.
+The default `CGO_ENABLED=0` build already encrypts and needs no flags. The CGO
+backend uses hanzoai/csqlite linked against libsqlcipher, with the key supplied as
+SQLCipher's native URI `key` parameter (applied inside `sqlite3_open_v2`, so create
+and reopen both work):
 
 ```sh
 CGO_ENABLED=1 \
@@ -41,25 +39,13 @@ CGO_LDFLAGS="-L<sqlcipher>/lib -lsqlcipher" \
 go build -tags "libsqlite3 sqlite_fts5" ./...
 ```
 
-Alpine: `apk add gcc musl-dev sqlcipher-dev pkgconfig`.
-
-### Why not `-tags sqlcipher` + `PRAGMA key` in a ConnectHook?
-
-Both are traps that ship **plaintext**:
-
-- `-tags sqlcipher` is inert in mainline mattn (no such tag) → links plain
-  sqlite → `PRAGMA key` is a silent no-op.
-- mattn runs `PRAGMA busy_timeout/journal_mode/foreign_keys/...` via
-  `sqlite3_exec` **before** the ConnectHook fires. On an existing encrypted file
-  that touches the header before the key is set → `file is not a database` on
-  reopen. So a ConnectHook can *create* but never *reopen* a SQLCipher DB.
-
-The URI `key` parameter sidesteps both: SQLCipher's VFS keys the connection at
-open time. `TestEncryptionProof` asserts real ciphertext on disk and a working
-keyed reopen, so a mis-linked build **fails CI** instead of shipping plaintext.
+Alpine: `apk add gcc musl-dev sqlcipher-dev pkgconfig`. A cgo build that forgets
+to link libsqlcipher silently writes plaintext; `CodecLinked()` proves the codec
+at runtime and `TestEncryptionProof` fails such a build.
 
-> The key rides the DSN (`file:PATH?...&key=x'HEX'`). **Never log the DSN.**
-> IAM keeps `showSql=false` and does not log it.
+> On the CGO backend the key rides the DSN (`file:PATH?...&key=x'HEX'`). **Never
+> log the DSN.** On the pure-Go backend the key never touches the DSN — it binds
+> to the codec VFS.
 
 ## Encryption
 

checkpoint_concurrent_test.go

@@ -0,0 +1,161 @@
+package sqlite
+
+import (
+	"database/sql"
+	"path/filepath"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// TestConcurrentCheckpointUnderLoad is the verification bar for concurrent-
+// checkpoint locking — the property WAL-shipping replication (hanzoai/replicate)
+// depends on. It runs a TRUNCATE-checkpoint loop on one connection while several
+// writers hammer INSERTs on another, then asserts:
+//
+//   - no wal_checkpoint(TRUNCATE) ever fails with a hard SQLITE_BUSY statement
+//     error (busy_timeout must make the checkpoint BLOCK for the writer lock, not
+//     bail — this is exactly what a broken/regressed blocking-lock path breaks);
+//   - the checkpoint actually completes (busy=0) at least once under load;
+//   - every committed row survives (TRUNCATE never truncates un-checkpointed
+//     frames out from under a writer — no data loss);
+//   - reopening the file reads every row back (durable, uncorrupted).
+//
+// It is backend-neutral: OpenPragma applies busy_timeout + wal_autocheckpoint=0
+// on every pooled connection under both the pure-Go and CGO backends, so the
+// same bar guards both. busy_timeout is generous (10s) because the correctness
+// claim is "the checkpoint blocks and then succeeds", not "it succeeds within an
+// arbitrarily tight deadline".
+func TestConcurrentCheckpointUnderLoad(t *testing.T) {
+	const (
+		writers      = 4
+		perWriter    = 150
+		checkpoints  = 40
+		blobSize     = 4000
+		busyTimeout  = "10000"
+		checkpointBT = "10000"
+	)
+	dir := t.TempDir()
+	path := filepath.Join(dir, "ckpt.db")
+
+	// Writers pool: busy_timeout so a writer BLOCKS on the checkpoint's lock
+	// instead of erroring; WAL so the checkpoint has something to reclaim.
+	w, err := OpenPragma("file:"+path, []Pragma{
+		{Name: "busy_timeout", Value: busyTimeout},
+		{Name: "journal_mode", Value: "WAL"},
+		{Name: "synchronous", Value: "NORMAL"},
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer w.Close()
+	w.SetMaxOpenConns(writers)
+	if _, err := w.Exec(`CREATE TABLE t (id INTEGER PRIMARY KEY, v BLOB)`); err != nil {
+		t.Fatal(err)
+	}
+
+	// Checkpointer: single connection, busy_timeout, auto-checkpoint OFF so the
+	// only checkpoints are the explicit TRUNCATEs below (mirrors replicate).
+	c, err := OpenPragma("file:"+path, []Pragma{
+		{Name: "busy_timeout", Value: checkpointBT},
+		{Name: "wal_autocheckpoint", Value: "0"},
+		{Name: "journal_mode", Value: "WAL"},
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer c.Close()
+	c.SetMaxOpenConns(1)
+
+	// Fire the writers.
+	var wrote int64
+	var firstWrite sync.Once
+	started := make(chan struct{})
+	var wg sync.WaitGroup
+	blob := make([]byte, blobSize)
+	for g := 0; g < writers; g++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for i := 0; i < perWriter; i++ {
+				for {
+					_, err := w.Exec(`INSERT INTO t (v) VALUES (?)`, blob)
+					if err == nil {
+						break
+					}
+					if isBusy(err) { // a well-behaved writer retries; must never see this with busy_timeout, but be safe
+						time.Sleep(time.Millisecond)
+						continue
+					}
+					t.Errorf("writer insert: %v", err)
+					return
+				}
+				atomic.AddInt64(&wrote, 1)
+				firstWrite.Do(func() { close(started) })
+			}
+		}()
+	}
+	<-started // don't checkpoint an empty WAL
+
+	// Hammer TRUNCATE checkpoints concurrently with the writers.
+	var completed, incomplete int
+	for k := 0; k < checkpoints; k++ {
+		var busy, logN, ckN int
+		if err := c.QueryRow(`PRAGMA wal_checkpoint(TRUNCATE)`).Scan(&busy, &logN, &ckN); err != nil {
+			// The regression signature: TRUNCATE returns SQLITE_BUSY as a hard
+			// statement error instead of blocking on busy_timeout.
+			t.Fatalf("checkpoint %d hard-failed (blocking-lock regression): %v", k, err)
+		}
+		if busy == 0 {
+			completed++
+		} else {
+			incomplete++
+		}
+		time.Sleep(time.Millisecond)
+	}
+	wg.Wait()
+
+	if completed == 0 {
+		t.Fatalf("no TRUNCATE checkpoint completed under load (%d incomplete) — checkpoint starved", incomplete)
+	}
+	t.Logf("checkpoints: %d completed, %d busy; rows written: %d", completed, incomplete, atomic.LoadInt64(&wrote))
+
+	// No data loss: every committed row is present.
+	want := int64(writers * perWriter)
+	if got := atomic.LoadInt64(&wrote); got != want {
+		t.Fatalf("writers wrote %d rows, want %d", got, want)
+	}
+	var n int64
+	if err := c.QueryRow(`SELECT count(*) FROM t`).Scan(&n); err != nil {
+		t.Fatal(err)
+	}
+	if n != want {
+		t.Fatalf("row count %d after concurrent checkpoint, want %d (frames lost)", n, want)
+	}
+
+	// Durable + uncorrupted: a fresh open reads every row back.
+	w.Close()
+	c.Close()
+	re, err := sql.Open("sqlite", "file:"+path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer re.Close()
+	var n2 int64
+	if err := re.QueryRow(`SELECT count(*) FROM t`).Scan(&n2); err != nil {
+		t.Fatal(err)
+	}
+	if n2 != want {
+		t.Fatalf("reopened DB has %d rows, want %d", n2, want)
+	}
+}
+
+func isBusy(err error) bool {
+	if err == nil {
+		return false
+	}
+	s := err.Error()
+	return strings.Contains(s, "database is locked") || strings.Contains(s, "SQLITE_BUSY") || strings.Contains(s, "database table is locked")
+}

connector.go

@@ -0,0 +1,95 @@
+package sqlite
+
+import (
+	"context"
+	"database/sql"
+	"database/sql/driver"
+	"fmt"
+	"sync"
+)
+
+// driverName is the database/sql driver name registered by BOTH backends
+// (driver_nocgo.go registers the pure-Go engine under it; driver_cgo.go
+// registers csqlite under it). Tag-neutral single source of truth.
+const driverName = "sqlite"
+
+// OpenPragma opens a *sql.DB on the registered "sqlite" backend where EVERY
+// pooled connection has the given pragmas applied, in order, via PRAGMA
+// statements run at connect time.
+//
+// Use it for pragmas that must hold on every connection but that the two
+// backends do NOT accept uniformly in the DSN. The load-bearing case is
+// wal_autocheckpoint: the pure-Go engine honors `?_pragma=wal_autocheckpoint(0)`
+// in the DSN, but csqlite silently DROPS it (its DSN grammar has no such param),
+// so a binary built CGO=1 would re-enable auto-checkpoint and truncate the WAL
+// out from under a WAL-shipping replicator — losing committed frames. Running
+// the pragma per connection via ExecContext is backend-neutral (both conns
+// implement driver.ExecerContext), so it takes effect either way.
+//
+// Order matters: put busy_timeout first so a connection blocks on a busy
+// database before journal_mode=WAL is attempted (WAL can't be set while another
+// connection holds the db).
+//
+// dsn is a plain (unencrypted) DSN — typically "file:"+path. Encrypted opens go
+// through OpenDB (the key binds to the codec VFS, not the DSN). Do NOT also put
+// these pragmas in the DSN; they're applied here.
+func OpenPragma(dsn string, pragmas []Pragma) (*sql.DB, error) {
+	drv := backendDriver()
+	if drv == nil {
+		return nil, fmt.Errorf("sqlite: OpenPragma: no %q driver registered", driverName)
+	}
+	return sql.OpenDB(&pragmaConnector{driver: drv, dsn: dsn, pragmas: pragmas}), nil
+}
+
+// pragmaConnector is a database/sql Connector that opens a backend connection
+// and applies pragmas to it before handing it to the pool. Because the pool
+// calls Connect for every new connection, the pragmas hold pool-wide.
+type pragmaConnector struct {
+	driver  driver.Driver
+	dsn     string
+	pragmas []Pragma
+}
+
+func (pc *pragmaConnector) Connect(ctx context.Context) (driver.Conn, error) {
+	conn, err := pc.driver.Open(pc.dsn)
+	if err != nil {
+		return nil, err
+	}
+	execer, ok := conn.(driver.ExecerContext)
+	if !ok {
+		_ = conn.Close()
+		return nil, fmt.Errorf("sqlite: OpenPragma: driver conn %T is not a driver.ExecerContext", conn)
+	}
+	for _, p := range pc.pragmas {
+		stmt := "PRAGMA " + p.Name + " = " + p.Value
+		if _, err := execer.ExecContext(ctx, stmt, nil); err != nil {
+			_ = conn.Close()
+			return nil, fmt.Errorf("sqlite: OpenPragma: apply %q: %w", stmt, err)
+		}
+	}
+	return conn, nil
+}
+
+func (pc *pragmaConnector) Driver() driver.Driver { return pc.driver }
+
+var (
+	backendDriverOnce sync.Once
+	backendDriverInst driver.Driver
+)
+
+// backendDriver returns the driver.Driver registered under the name "sqlite" by
+// this package (csqlite under cgo, the pure-Go engine otherwise). It is resolved
+// once, via a throwaway in-memory sql.DB, so the connector stays tag-neutral —
+// it needs no per-backend driver symbol and works whichever backend init()
+// registered.
+func backendDriver() driver.Driver {
+	backendDriverOnce.Do(func() {
+		db, err := sql.Open(driverName, "file::memory:")
+		if err != nil {
+			return
+		}
+		backendDriverInst = db.Driver()
+		_ = db.Close()
+	})
+	return backendDriverInst
+}

connhook_nocgo.go

@@ -0,0 +1,34 @@
+//go:build !cgo
+
+package sqlite
+
+import engine "github.com/hanzoai/sqlite/internal/engine"
+
+// Connection-hook surface — re-exported from the vendored pure-Go engine so
+// external HA/replication libraries (e.g. litesql/go-sqlite-ha, used by base-ha)
+// can migrate off modernc.org/sqlite WITHOUT importing internal packages. These
+// are the pure-Go engine's connection-hook feature; there is no CGO analogue
+// (csqlite has a different hook model), so they exist only under !cgo — the
+// backend base-ha and the HA layer run on.
+
+// Driver is the pure-Go SQLite driver type. A zero value is usable: register
+// connection hooks on it, then hand &Driver to a database/sql connector. This is
+// the same driver type registered under the "sqlite" name; go-sqlite-ha does
+// `var drv sqlite.Driver; drv.RegisterConnectionHook(fn)` then opens through it.
+type Driver = engine.Driver
+
+// ExecQuerierContext is the exec+query surface a ConnectionHookFn receives: the
+// minimal handle a hook uses to run setup SQL on each new connection before it
+// enters the pool.
+type ExecQuerierContext = engine.ExecQuerierContext
+
+// ConnectionHookFn runs on every new connection the driver opens, after setup
+// and before the connection enters the pool — the hook point HA/replication
+// layers install their per-connection wiring on.
+type ConnectionHookFn = engine.ConnectionHookFn
+
+// RegisterConnectionHook installs fn on the driver registered under the "sqlite"
+// name; fn then runs for every new connection opened via sql.Open("sqlite", …).
+// (A private Driver instance registers hooks via its own RegisterConnectionHook
+// method — see Driver.)
+func RegisterConnectionHook(fn ConnectionHookFn) { engine.RegisterConnectionHook(fn) }