Implement copy-on-write overlay filesystem and three-way merge#7
Merged
Conversation
Run the test suite on ubuntu-24.04 alongside macOS. Guard the Darwin-only surfaces so the package builds on Linux: - SecurityScopedFilesystem (security scopes and URL bookmarks) is compiled only where Darwin is available. - SandboxFilesystem app-group resolution throws unsupported on platforms without container APIs. - OverlayFilesystem import classifies entries via FileManager attributes instead of URL resource values. The workflow also runs on pushes to core-improvements so changes are verified before a PR exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
…tency - CI: run the Linux job inside the official swift:6.2-noble container; setup-swift's GPG verification fails on ubuntu runners. - WorkspacePath.globToRegex: * and ? no longer cross path separators, ** matches recursively, and [!abc] negation is translated correctly. Previously /docs/*.txt also matched /docs/sub/deep.txt. - ReadWriteFilesystem: entry-level operations (stat, exists, remove, move source, readSymlink) now use lstat semantics behind the same parent containment check. Symlinks whose targets lie outside the root were previously unstattable and undeletable, and exists() bypassed the jail entirely, following links and acting as an existence oracle for arbitrary external paths. Content operations still refuse to dereference an escaping link. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
FileSystem gains four operations, all with default implementations so existing conformances keep compiling: - readFile(path:offset:length:) reads a byte range; ReadWriteFilesystem overrides it with a seeking file handle so large files are not loaded fully into memory. - readFileChunks(path:chunkSize:) streams contents as AsyncThrowingStream chunks; the disk backend reads incrementally. - createFile(path:data:) creates exclusively and fails with EEXIST; the disk backend uses an atomic no-overwrite write and the in-memory backend checks-and-writes inside its actor. - capabilities() advertises optional features (symlinks, hard links, permissions, real-path resolution) so callers can branch instead of probing for unsupported errors. Wrappers (Permissioned, Mountable, Overlay, Sandbox, SecurityScoped) forward the new operations; PermissionedFileSystem maps them onto the existing readFile/writeFile permission operations. Workspace exposes readData(from:offset:length:). Also updates the three glob tests that asserted the old crossing-separator wildcard behavior, and marks the checkout safe for git inside the Linux CI container so the test reporter can run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
The previous implementation eagerly copied the entire source tree, contents included, into memory at configure time, so overlaying a real project cost memory and startup time proportional to the whole tree. The overlay is now a merged view of two layers: reads pass through lazily to the source directory, mutations land in an in-memory upper layer with on-demand copy-up (append copies the file up, whole-tree moves and copies materialize the affected subtree), and deletions are whiteout cuts that hide lower entries without touching the disk. Directory listings merge both layers with upper entries shadowing lower ones, re-created directories do not resurrect hidden lower children, symlinks resolve across layers at the final component, and stat passes through source permissions and modification dates (previously mtimes were lost on import). reload() and configure() drop the upper layer and whiteouts in O(1) instead of re-importing. Behavioral note: files that change on disk are now visible through the overlay immediately, since reads are pass-through rather than a configure-time snapshot. Also fixes the escaped-separator expectation in the globToRegex test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
FileCheckpointStore now persists snapshots as manifests plus content-addressed blobs: - File bytes live in per-workspace blobs/<sha256> files, written once per unique content, so consecutive checkpoints share unchanged files instead of re-serializing the whole tree, and contents are stored raw instead of base64-inflated inside pretty-printed JSON. - Snapshot manifests carry the tree structure, permissions, and content hashes. Legacy inline-snapshot JSON files still load. - Hashing uses a small dependency-free SHA-256 (FIPS 180-4) with test vectors, keeping the package free of external dependencies. - A missing blob surfaces as the new WorkspaceError.storageCorrupted. Store robustness: - rollback(to:) now records the tree changes it performs as a new `rollback` mutation kind, closing the gap where the mutation log missed exactly the changes made by rollbacks. - CheckpointStore gains pruneMutationRecords(workspaceId:throughSequence:) and Workspace gains pruneMutationHistory(throughSequence:), which bound the mutation log while always retaining the newest record so sequence numbers stay monotonic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
…anning Workspace gains a TrackingPolicy controlling how much detail tracked writes record: - .full (default) keeps today's behavior: full mutation records with per-file line diffs. - .fullDiffs(maxDiffBytes:) caps the content size for which diffs are computed, so huge files no longer trigger whole-content diffing. - .pathsOnly records touched paths and effects without diffs. - .disabled records no history; change events still fire, and the policy is inherited by branches. Merge's uncheckpointed-changes guard cannot see edits made in this mode (documented). MutationRecord and mutationRecords() are now public, so consumers can actually read the history the workspace records. Event planning for deletions and moves now walks structure only (stats and listings) instead of capturing full file contents, removing a whole-subtree content read from every removeItem/moveItem/copyItem. Replacement change events are driven by replacement counts instead of diff presence so they keep firing when diffs are suppressed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
mergeThreeWay(_:label:) resolves each path against the branch's base checkpoint snapshot: the side that changed relative to the base wins, identical changes on both sides merge cleanly, and branch deletions propagate. Unlike merge(_:), the parent may have advanced -- with or without checkpoints -- since the branch was created. When both sides changed the same path differently, nothing is applied and the result carries structured MergeConflict values (bothModified, modifiedAndDeleted, bothCreated) with base-to-ours and base-to-theirs text diffs, so callers can present or resolve conflicts instead of handling a thrown error. A clean merge restores the merged tree, records a mergeWorkspace mutation, and creates a checkpoint carrying the mergedFrom lineage fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a complete redesign of the overlay filesystem and adds three-way merge support for workspaces.
Summary
The
OverlayFilesystemis reimplemented as a true copy-on-write (CoW) system with lazy pass-through reads to a lower disk layer and mutations recorded in an upper in-memory layer. Additionally, workspaces now support three-way merging against a branch's base checkpoint, enabling merges even when the parent workspace has advanced since branching.Key Changes
Overlay Filesystem Redesign
OverlayFilesystemfrom eager snapshot-based to lazy copy-on-write architecturereload()discards the upper layer and whiteouts to reflect the current disk stateglob(),readSymlink(),resolveRealPath(), and other missing filesystem operationsThree-Way Merge
Workspace.mergeThreeWay(_:label:)for per-path three-way resolution against a branch's base checkpointThreeWayMergeResultwith either a checkpoint or a list ofMergeConflictentriesmerge(_:label:))Tracking Policy
Workspace.TrackingPolicyto control mutation history detail:.fullDiffs(maxDiffBytes:)records full diffs with optional size cap (default).pathsOnlyrecords touched paths and effects without diffs.disableddisables mutation history entirelySnapshot Storage Optimization
FileCheckpointStoreto deduplicate snapshot content into content-addressed blobsSnapshotManifestwith file contents replaced by SHA-256 hashesblobs/<sha256>filesSHA256utility for content addressing (FIPS 180-4 compliant)Filesystem Improvements
readFile(path:offset:length:)for ranged reads without loading entire filesreadFileChunks(path:chunkSize:)for streaming large files incrementallyFileSystemCapabilitiesto advertise optional features (symlinks, hardlinks, permissions, real path resolution)lstatsemantics inReadWriteFilesystem.stat()to handle symlinks without following themMutation Record Improvements
MutationRecordpublic with improved documentationCheckpointStore.pruneMutationRecords()to clean up old mutation history while retaining the latest recordrollbackmutation kind to record rollback operations in the mutation logPlatform Support
Notable Implementation Details
Set<WorkspacePath>to track "cuts" (deleted entries) efficientlyReadWriteFilesystemvalidates symlink targets to prevent escaping the configured root directory*(single component),?(single character), and**(any characters including/)https://claude.ai/code/session_01G29SXqzHqDycqXGa4dRr4x