Save/load snapshot from disk (oci format) by ludfjig · Pull Request #1465 · hyperlight-dev/hyperlight

ludfjig · 2026-05-20T17:31:10Z

Adds save/load of Snapshot to disk as an OCI Image Layout. Public API:

Snapshot::to_oci(path, tag)
Snapshot::from_oci(path, tag) (verifies sha256 on every blob)
Snapshot::from_oci_unchecked(path, tag) (unsafe because skips digest checks, not unsafe in rust UB sense)

Known limitations

Core dumps from a snapshot-loaded sandbox lack binary_path and AT_ENTRY for
Call snapshots. mem_profile lacks accurate traces. Fixing either requires
extending the on-disk format.
max_guest_log_level is not plumbed through snapshot load. It is also
intrinsically ineffective for Call snapshots and should be addressed
separately.
The backing OCI directory must not be modified, truncated, renamed over, or
deleted for the lifetime of a loaded Snapshot or any MultiUseSandbox
built from it. On Linux this is unenforced. On Windows the OS refuses the
operation with ERROR_USER_MAPPED_FILE (1224). Firecracker has the same
constraint:

The memory file (pointed by backend_path when using File backend type,
or pointed by mem_file_path) must be considered immutable from
Firecracker and host point of view. It backs the guest OS memory for read
access through the page cache. External modification to this file corrupts
the guest memory and leads to undefined behavior.

firecracker docs

Future work

Typed error variants. TSC and rand seeding capture. Fuzz target for from_oci.
CoW overlay layers. Cross-hypervisor portability via sregs normalisation. Huge
page support (MAP_HUGETLB)

Copilot

Pull request overview

Adds an OCI Image Layout–backed on-disk format for Snapshot, enabling saving a sandbox snapshot to disk and loading it back (with optional digest verification) as a building block for snapshot-driven sandbox creation (per the dependency on #1459).

Changes:

Implement Snapshot::to_oci, Snapshot::from_oci, and Snapshot::from_oci_unchecked using OCI Image Layout + sha256-addressed blobs.
Add serde-encoded snapshot config schema (arch/hypervisor/ABI gating, entrypoint/sregs, layout, host function signatures) plus extensive validation + tests.
Extend benchmarks and docs to cover snapshot file save/load paths and usage.

Reviewed changes

Copilot reviewed 12 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/hyperlight_host/src/sandbox/snapshot/mod.rs	Wires in the new snapshot OCI file backend modules.
src/hyperlight_host/src/sandbox/snapshot/file/mod.rs	Implements OCI layout writer/loader for snapshots (manifest/config/blob handling).
src/hyperlight_host/src/sandbox/snapshot/file/config.rs	Defines and validates the JSON config schema for OCI snapshot artifacts.
src/hyperlight_host/src/sandbox/snapshot/file/digest.rs	Adds sha256 digest helpers and verification for blobs.
src/hyperlight_host/src/sandbox/snapshot/file/fsutil.rs	Adds bounded file read + atomic write + content-addressed blob write helpers.
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs	Defines versioned media types and ABI version constant for snapshot artifacts.
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs	Adds comprehensive unit/integration tests for the on-disk OCI snapshot format.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Updates docs to show creating a sandbox from an on-disk-loaded snapshot.
src/hyperlight_host/src/mem/shared_mem.rs	Adjusts Linux file-backed mapping protection flags for MSHV requirements.
src/hyperlight_host/Cargo.toml	Adds dependencies needed for OCI + hashing + serde config.
src/hyperlight_host/benches/benchmarks.rs	Adds benchmarks for sandbox-from-snapshot and snapshot file save/load/cold-start paths.
docs/snapshot-oci-format.md	Documents the OCI Image Layout snapshot on-disk format and semantics.
CHANGELOG.md	Notes the new snapshot persistence APIs.
Cargo.lock	Locks newly added transitive dependencies.

ludfjig · 2026-06-09T17:06:55Z

+use crate::hypervisor::regs::CommonSpecialRegisters;
+use crate::mem::layout::SandboxMemoryLayout;
+use crate::mem::memory_region::MemoryRegionFlags;
+use crate::mem::shared_mem::{ReadonlySharedMemory, SharedMemory};


this is bull

ludfjig · 2026-06-09T17:07:08Z

        // 2. Overlay the file content on the middle slot with
        //    `MAP_FIXED`. The guest maps these pages READ|EXECUTE,
        //    so the host VMA is read-only. `MAP_PRIVATE` keeps the
        //    mapping detached from the underlying file.


There's a comment right there

ludfjig · 2026-06-09T17:08:03Z

+serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 elfcore = { version = "2.0", optional = true }
 uuid = { version = "1.23.2", features = ["v4"] }
+oci-spec = { version = "0.8", default-features = false, features = ["image"] }


Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

andreiltd · 2026-06-12T12:15:05Z

+    bytes: &[u8],
+) -> crate::Result<()> {
+    let target = blobs_dir.join(&digest.hex);
+    if let Ok(meta) = std::fs::metadata(&target)


Do we want to check for symlinks as well? I think if blob is symlink it should be rejected.

andreiltd · 2026-06-12T12:19:55Z

+#[serde(rename_all = "lowercase")]
+pub(super) enum Arch {
+    X86_64,
+    Aarch64,


Does this require synchronization with aarch64 support PR? It looks like registers for arm are placeholders only now.

andreiltd · 2026-06-12T12:33:46Z

+        //    of the snapshot region avoids overlap with
+        //    `map_file_cow` regions installed immediately after the
+        //    snapshot in guest PA space.
+        let memory = ReadonlySharedMemory::from_file(&snap_file, layout.snapshot_size)?;


Should we check memory size here again against cfg.memory_size?

if memory.mem_size() as u64 != cfg.memory_size { return Err(...); }

Is it possible that file will be modified between verification and mapping or is file immutability enforced somehow? If it can be modified, I think that without this check later restore can explode, no?

let snapshot_pt_end = self.shared_mem.mem_size(); let snapshot_pt_start = snapshot_pt_end - snapshot_pt_siz

andreiltd · 2026-06-12T12:38:40Z

+}
+
+impl OciSnapshotConfig {
+    pub(super) fn validate_for_load(&self) -> crate::Result<()> {


I think we are missing stack_top_gva validation. It looks like it's loaded but not validated.

andreiltd · 2026-06-12T12:44:17Z

+        }
+
+        // 6. Reconstruct layout.
+        let mut sbox_cfg = crate::sandbox::SandboxConfiguration::default();


I think all those config fields that are used to calculate layout arithmetic should be verified/bounded before creating layout?

andreiltd · 2026-06-12T13:03:19Z

+    pub(super) init_data_permissions: Option<u32>,
+    pub(super) scratch_size: usize,
+    pub(super) snapshot_size: usize,
+    pub(super) pt_size: Option<usize>,


Why this is optional? Can we populate 0 size pt?

https://github.com/hyperlight-dev/hyperlight/blob/main/src/hyperlight_host/src/mem/mgr.rs#L624

ludfjig force-pushed the disk_snapshot_oci branch 10 times, most recently from 84d17dd to c4aec72 Compare June 3, 2026 22:27

ludfjig force-pushed the disk_snapshot_oci branch from c4aec72 to 012bd11 Compare June 8, 2026 17:52

simongdavies mentioned this pull request Jun 8, 2026

Enable guest time #1422

Open

ludfjig force-pushed the disk_snapshot_oci branch 3 times, most recently from 4cb8124 to 7c8175b Compare June 9, 2026 16:37

ludfjig marked this pull request as ready for review June 9, 2026 16:38

Copilot AI review requested due to automatic review settings June 9, 2026 16:38

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners June 9, 2026 16:38

Copilot started reviewing on behalf of ludfjig June 9, 2026 16:38 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Jun 9, 2026

ludfjig force-pushed the disk_snapshot_oci branch from 7c8175b to a37f54e Compare June 9, 2026 17:32

Map snapshot pages writable on MSHV

be93bab

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

ludfjig force-pushed the disk_snapshot_oci branch 2 times, most recently from 517f50d to 54107fb Compare June 9, 2026 18:40

ludfjig added 2 commits June 9, 2026 12:05

Add OCI snapshot persistence

2819ff1

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Add tests for OCI snapshot persistence

79f3823

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

ludfjig force-pushed the disk_snapshot_oci branch from 54107fb to f270a7e Compare June 9, 2026 19:05

Add OCI snapshot benchmarks

8a542b8

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

ludfjig force-pushed the disk_snapshot_oci branch from f270a7e to 8a542b8 Compare June 9, 2026 20:05

andreiltd reviewed Jun 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Save/load snapshot from disk (oci format)#1465

Save/load snapshot from disk (oci format)#1465
ludfjig wants to merge 4 commits into
hyperlight-dev:mainfrom
ludfjig:disk_snapshot_oci

ludfjig commented May 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

ludfjig Jun 9, 2026

Uh oh!

ludfjig Jun 9, 2026

Uh oh!

ludfjig Jun 9, 2026

Uh oh!

andreiltd Jun 12, 2026

Uh oh!

andreiltd Jun 12, 2026

Uh oh!

andreiltd Jun 12, 2026

Uh oh!

andreiltd Jun 12, 2026

Uh oh!

andreiltd Jun 12, 2026

Uh oh!

andreiltd Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ludfjig commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Known limitations

Future work

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ludfjig commented May 20, 2026 •

edited

Loading