diff --git a/changelog.d/7809-globalthis-bootstrap-layout-latch.md b/changelog.d/7809-globalthis-bootstrap-layout-latch.md new file mode 100644 index 0000000000..eea5e31f4a --- /dev/null +++ b/changelog.d/7809-globalthis-bootstrap-layout-latch.md @@ -0,0 +1,100 @@ +### Fixed + +- **Touching `globalThis` no longer disables the per-object GC layout fast path + for the rest of the process.** `churn` and `tree` were paying +28% / +29% for + a single `for…of` in `main()`, and every real TypeScript program was paying it + too. + + `globalThis` is populated lazily on first touch, and *any* plain-object or + array property **miss** forces it: the miss walks the prototype chain, reaches + `builtin_prototype_value` → `js_get_global_this_builtin_value`, and runs the + several-hundred-builtin bootstrap. Perry's whole benchmark corpus happens + never to take that path — no `for…of`, no spread, no `Symbol`, no property + miss anywhere in `churn`/`tree`/`interp`/`shapes`/`asyncpipe`/`retain` — so + the cost was invisible to every number in the perf campaign while real + programs paid it before their first line of work. + + It was **not** the ~1.15 MB the bootstrap allocates. GC behaviour is + effectively identical with and without it: 105 minors on `churn` either way, + and ~616 KB more copied across the entire run. The cost was a **global latch**. + The bootstrap builds hundreds of permanently-rooted plain objects, and each + one's first pointer field minted an entry in the per-object GC slot-layout + side tables. Those entries are immortal, so `PER_OBJECT_LAYOUTS_NONEMPTY` — + the emptiness proof that keeps `layout_forget_object` off the allocation, + death and relocation paths — could never go `false` again. Measured + `layout_forget_object` self time: `churn` 112 ms → 916 ms, `tree` + 194 ms → 740 ms, which is essentially the whole regression in both. + + This is #7510's lesson at 1000× the scale ("one immortal entry is enough to + nullify an is-empty accelerator"; there it was a single interned keys array). + Two changes, and **both are needed**: + + 1. `gc::ImmortalLayoutScope` around `populate_global_this_builtins`. Inside + it, an object that would mint a per-object pointer mask declares + `GC_LAYOUT_UNKNOWN` instead — the tag-checked payload scan, which is the + code's own fallback for the same situation and the universally safe state, + not a weaker one. For an object that is never reclaimed the mask bought + precision nobody spends, at the price of two `RefCell` round-trips and two + hash probes on every allocation the program would ever make. Bootstrap + residue: **1113 entries → 0**. + + Deliberately **not** applied to typed-shape layouts + (`init_typed_shape_layout`): those describe raw-f64 slots, whose bit + patterns can alias a heap pointer, and a conservative scan would trace — + and under the copying collector *rewrite* — a slot holding a number. The + scope applies only where the mask being replaced is itself derived from + `layout_pointer_bearing_bits`, i.e. exactly the test `GC_LAYOUT_UNKNOWN` + re-runs per slot. + + 2. An **address filter** replacing `PER_OBJECT_LAYOUTS_NONEMPTY` as the hot + guard. Change 1 alone moved nothing measurable, and that is the important + finding: ordinary runtime init still leaves one or two long-lived records + behind, and for a single global bit two entries are exactly as bad as 1113. + An 8192-bit thread-local filter over the key addresses turns "is either + table empty?" into "can this *address* have an entry?", so a nursery + address the tables have never seen is proved absent in one multiply and one + load even while immortal records exist elsewhere. + + The filter sits *behind* the flag, and both live in ONE thread-local + (`PerObjectLayoutHint`). All three arrangements were measured on the quiet + mini; the co-located one wins everywhere: + + | | `churn` | `push_cls` | `tree` | `interp` | `churn`+`for…of` | `tree`+`for…of` | + |---|--:|--:|--:|--:|--:|--:| + | base | 0.422 | 0.356 | 1.627 | 1.888 | 0.539 | 2.151 | + | filter only (no flag) | 0.438 | **0.383** | 1.673 | 1.934 | 0.500 | 1.857 | + | flag + filter, 2 slots | 0.421 | 0.368 | 1.642 | 1.950 | 0.506 | 1.886 | + | **flag + filter, 1 slot** | **0.422** | **0.368** | **1.640** | **1.922** | **0.493** | **1.840** | + + Dropping the flag is a loss: almost every workload is *disarmed*, and for + those the flag is one load where the filter is a multiply, a shift, a load + and a test — `push_cls` went past its budget. Keeping both as separate + thread-locals costs a second `_tlv_get_addr` on exactly the workloads that + are legitimately armed (`interp`, `iso_miss`). One struct behind the + existing named hot slot gives the cheap gate AND one resolution. + + `false` is a proof of absence and nothing else rests on it; the filter is + rebuilt from the live keys once half its bits are set, so a workload that + genuinely churns per-object records cannot saturate it permanently. + + Measured on the quiet mini (base = `b9415d780`, both arms built locally; + interleaved, best-of-5, exit-checked): `churn` + `for…of` **0.539 → 0.493** + (floor 0.422), `tree` + `for…of` **2.151 → 1.840** (floor 1.627). Every + protected bench stays inside budget. `interp` 1.888 → 1.922 and `iso_miss` + 2.361 → 2.443 still pay the filter test without benefiting from it — they are + legitimately armed, so it never proves absence for them. + + Gated by five tests in `gc::tests::layout_trace::per_object_tables`, written + so none of them can pass vacuously: the bootstrap must leave both tables empty + (with a subject-live check that `globalThis.Array` actually populated); the + same store *outside* a scope must still mint a mask; an object built inside a + scope must still trace its children through the fallback scan; a live record + must survive a filter rebuild and still be found and removed; and — the + accelerator's own subject-live assertion — the filter must still prove + unrelated addresses absent *while the global flag is armed*, which is the + exact condition under which it silently stopped accelerating before. + + `PERRY_GC_DIAG=1` now prints + `[gc-globalthis-bootstrap] elapsed_us=… per_object_slot_masks=… per_object_typed_layouts=…` + once per thread, so the bootstrap's cost and its residue are observable rather + than inferred. diff --git a/crates/perry-runtime/src/gc/hot_tls.rs b/crates/perry-runtime/src/gc/hot_tls.rs index 8650e55fbf..20a2327c29 100644 --- a/crates/perry-runtime/src/gc/hot_tls.rs +++ b/crates/perry-runtime/src/gc/hot_tls.rs @@ -24,7 +24,9 @@ use super::barrier::{ GC_BIRTH_EXTRA_FLAGS, INCREMENTAL_MARK_BARRIER_MINOR_ONLY, INCREMENTAL_MARK_BARRIER_VALID_PTRS, }; use super::layout::{LayoutSlotMask, TypedLayoutDescriptor, SHAPE_LAYOUTS}; -use super::layout_tables::{LAYOUT_SLOT_MASKS, PER_OBJECT_LAYOUTS_NONEMPTY, TYPED_LAYOUTS}; +use super::layout_tables::{ + PerObjectLayoutHint, LAYOUT_SLOT_MASKS, PER_OBJECT_LAYOUTS_NONEMPTY, TYPED_LAYOUTS, +}; use super::malloc::{ARENA_FREE_LIST, ARENA_FREE_LIST_NONEMPTY}; use super::trace::ValidPointerSet; use std::cell::{Cell, RefCell}; @@ -120,9 +122,9 @@ pub(super) fn hot_shape_layouts() -> &'static RefCell { /// per-object layout record at all" question the allocation, store, death and /// trace paths all ask (#7510). #[inline(always)] -pub(super) fn hot_per_object_layouts_nonempty() -> &'static Cell { +pub(super) fn hot_per_object_layout_hint() -> &'static PerObjectLayoutHint { // SAFETY: paired with `per_object_layouts_nonempty_hot_addr` above. - unsafe { &*(crate::tls_hot::hot().per_object_layouts_nonempty as *const Cell) } + unsafe { &*(crate::tls_hot::hot().per_object_layouts_nonempty as *const PerObjectLayoutHint) } } // --- gc::malloc ------------------------------------------------------------- diff --git a/crates/perry-runtime/src/gc/layout.rs b/crates/perry-runtime/src/gc/layout.rs index dcdbe1ef2c..d63a4c9568 100644 --- a/crates/perry-runtime/src/gc/layout.rs +++ b/crates/perry-runtime/src/gc/layout.rs @@ -909,11 +909,27 @@ pub(crate) fn layout_note_slot(parent_user: usize, slot_index: usize, value_bits set_layout_state(header, GC_LAYOUT_SIDE_MASK); } } else if (*header)._reserved & GC_LAYOUT_STATE_MASK == GC_LAYOUT_POINTER_FREE { - let mut mask = LayoutSlotMask::Inline(0); - mask.set_slot(slot_index); - masks.insert(parent_user, mask); - mark_per_object_layouts_nonempty(); - set_layout_state(header, GC_LAYOUT_SIDE_MASK); + if super::layout_tables::immortal_layout_scope_active() { + // An object built inside an `ImmortalLayoutScope` is + // rooted for the life of the process, so the entry it + // would mint here is never removed — and one such + // entry disables `PER_OBJECT_LAYOUTS_NONEMPTY` for + // every allocation the program will ever make. Take + // the same `GC_LAYOUT_UNKNOWN` fallback the `else` + // arm below uses for this exact situation; see + // `ImmortalLayoutScope` for why that is the safe + // state and not a weaker one. + set_layout_state(header, GC_LAYOUT_UNKNOWN); + } else { + let mut mask = LayoutSlotMask::Inline(0); + mask.set_slot(slot_index); + masks.insert(parent_user, mask); + mark_per_object_layouts_nonempty(); + // The one insert site that holds its own `borrow_mut`, + // so it maintains the address filter inline too. + super::layout_tables::layout_addr_filter_note(parent_user); + set_layout_state(header, GC_LAYOUT_SIDE_MASK); + } } else { set_layout_state(header, GC_LAYOUT_UNKNOWN); } @@ -1338,6 +1354,18 @@ pub(super) unsafe fn layout_rebuild_from_slots_with_policy( if mask.is_empty() { set_layout_state(header, GC_LAYOUT_POINTER_FREE); slot_masks_remove(user_ptr as usize); + } else if super::layout_tables::immortal_layout_scope_active() { + // Same reasoning as the `layout_note_slot` branch: an object built + // inside an `ImmortalLayoutScope` never dies, so the mask it would + // install here is a permanent tenant of a side table whose emptiness + // is a process-wide fast path. Falling back to the tag-checked scan is + // sound *for this rebuild specifically* because the mask above is + // itself derived from `layout_pointer_bearing_bits` — exactly the test + // `GC_LAYOUT_UNKNOWN` re-runs per slot. (This is why the scope may not + // be applied to a TYPED descriptor, whose raw-f64 slots the tag test + // would misread; see `ImmortalLayoutScope`.) + set_layout_state(header, GC_LAYOUT_UNKNOWN); + slot_masks_remove(user_ptr as usize); } else { set_layout_state(header, GC_LAYOUT_SIDE_MASK); slot_masks_insert(user_ptr as usize, mask); diff --git a/crates/perry-runtime/src/gc/layout_tables.rs b/crates/perry-runtime/src/gc/layout_tables.rs index 7dffb6db0b..a4df0ae135 100644 --- a/crates/perry-runtime/src/gc/layout_tables.rs +++ b/crates/perry-runtime/src/gc/layout_tables.rs @@ -29,7 +29,7 @@ //! Split out of `layout.rs` to stay under the repo's 2000-line-per-file cap //! (`scripts/check_file_size.sh`). -use super::hot_tls::{hot_layout_slot_masks, hot_per_object_layouts_nonempty, hot_typed_layouts}; +use super::hot_tls::{hot_layout_slot_masks, hot_per_object_layout_hint, hot_typed_layouts}; use super::layout::{LayoutSlotMask, TypedLayoutDescriptor}; use std::cell::{Cell, RefCell}; @@ -47,7 +47,258 @@ thread_local! { /// once they are empty. A stale `true` therefore costs exactly the /// pre-#7510 probe and nothing else — the flag is an accelerator, never an /// authority, and no caller may treat it as one. - pub(in crate::gc) static PER_OBJECT_LAYOUTS_NONEMPTY: Cell = const { Cell::new(false) }; + pub(in crate::gc) static PER_OBJECT_LAYOUTS_NONEMPTY: PerObjectLayoutHint = + const { PerObjectLayoutHint::new() }; +} + +/// The flag, the address filter, and the filter's rebuild counter in ONE +/// thread-local. +/// +/// They are co-located for a measured reason. On Darwin a thread-local access +/// is an out-of-line `_tlv_get_addr` call, and `crate::tls_hot` exists to pay +/// that once per hot region instead of once per table. `layout_forget_object` +/// consults the flag and then the filter on every allocation, death and +/// relocation; as two separate thread-locals that is two slot reads on exactly +/// the workloads that are legitimately armed, which measured +3.4% on `interp` +/// and +4.6% on `iso_miss`. One struct behind the existing named hot slot makes +/// it one. +pub(in crate::gc) struct PerObjectLayoutHint { + /// #7510's global emptiness proof — see [`PER_OBJECT_LAYOUTS_NONEMPTY`]. + pub(in crate::gc) nonempty: Cell, + /// Bits set in `filter` since the last rebuild. + pub(in crate::gc) sets: Cell, + /// Which addresses may have an entry — see [`layout_addr_filter_may_hold`]. + pub(in crate::gc) filter: std::cell::UnsafeCell<[u64; LAYOUT_ADDR_FILTER_WORDS]>, +} + +impl PerObjectLayoutHint { + const fn new() -> Self { + Self { + nonempty: Cell::new(false), + sets: Cell::new(0), + filter: std::cell::UnsafeCell::new([0u64; LAYOUT_ADDR_FILTER_WORDS]), + } + } +} + +/// Bits in the per-object address filter (see [`layout_addr_filter_may_hold`]). +/// 4096 bits is 512 B of thread-local storage, held INLINE in +/// [`PerObjectLayoutHint`] so the flag and the filter share one hot slot. One +/// 8-byte word is read per query and the whole filter stays L1-resident even +/// when the addresses probed sweep a 16 MB nursery. Steady-state occupancy +/// after the `ImmortalLayoutScope` is one or two entries, so the false-positive +/// rate is ~0.05% — sizing this up buys nothing and costs inline TLS on every +/// thread. +const LAYOUT_ADDR_FILTER_BITS: usize = 4096; +const LAYOUT_ADDR_FILTER_WORDS: usize = LAYOUT_ADDR_FILTER_BITS / 64; +/// Rebuild the filter from the live keys once this many bits have been set +/// since the last rebuild. Without it a workload that churns per-object +/// records would saturate the filter and never recover; with it the false +/// positive rate is bounded by (live entries / bits) rather than by +/// (entries ever inserted / bits), at an amortised O(1) per insert. +const LAYOUT_ADDR_FILTER_REBUILD_AFTER: u32 = (LAYOUT_ADDR_FILTER_BITS / 2) as u32; + +/// Word index + bit mask for `user_ptr`. Heap pointers are at least 8-byte +/// aligned and clustered, so the low bits alone would collide systematically; +/// a single multiply spreads the whole address across the filter. +#[inline(always)] +fn layout_addr_filter_slot(user_ptr: usize) -> (usize, u64) { + let h = (user_ptr as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15); + let idx = (h >> (64 - LAYOUT_ADDR_FILTER_BITS.trailing_zeros() as u64)) as usize; + (idx >> 6, 1u64 << (idx & 63)) +} + +/// Could either per-object side table hold an entry keyed by `user_ptr`? +/// +/// `false` is a **proof of absence**; `true` is a hint (a real entry, or a +/// collision). This is the address-precise half of the accelerator, and it is +/// the half that survives an immortal resident: [`PER_OBJECT_LAYOUTS_NONEMPTY`] +/// is a single global bit, so ONE entry that is never removed — one long-lived +/// object anywhere in the process — turns `layout_forget_object` back into two +/// `RefCell` round-trips plus two hashes on every allocation, death and +/// relocation for the rest of the run. Removing 1113 of 1115 such entries buys +/// nothing while the last two remain; only an address-keyed test does. +/// +/// ## Why the flag is still tested FIRST, even though this subsumes it +/// +/// It does subsume it — both maps empty implies every bit is clear, because the +/// flag's own clear path clears the filter — and dropping the flag would make +/// the armed path one thread-local resolution cheaper. Measured on the quiet +/// mini, that trade is a loss: filter-only moved `interp` 1.948 → 1.934 and +/// `iso_miss` 2.476 → 2.464, but moved `push_cls` 0.367 → 0.383 (past its +/// budget), `churn` 0.422 → 0.438 and `tree` 1.642 → 1.673. Almost every +/// workload is *disarmed*, and for those the flag is a single load while this +/// is a multiply, a shift, a load and a test. The flag stays in front as the +/// cheap common-case gate; the filter is what rescues the armed case. +/// +/// The residual cost of consulting both — a second thread-local resolution on a +/// legitimately-armed workload, +3.4% on `interp` and +4.6% on `iso_miss` — +/// is not inherent. It goes away by co-locating the flag and this filter in one +/// thread-local so the armed path resolves once; that is a `tls_hot` change and +/// is deliberately left out of this one. +#[inline(always)] +pub(in crate::gc) fn layout_addr_filter_may_hold(user_ptr: usize) -> bool { + hint_may_hold(hot_per_object_layout_hint(), user_ptr) +} + +/// [`layout_addr_filter_may_hold`] against an already-resolved hint, so a +/// caller that also reads the flag pays ONE slot resolution for both. +#[inline(always)] +pub(in crate::gc) fn hint_may_hold(hint: &PerObjectLayoutHint, user_ptr: usize) -> bool { + let (word, bit) = layout_addr_filter_slot(user_ptr); + unsafe { (*hint.filter.get())[word] & bit != 0 } +} + +/// Record that `user_ptr` now has an entry. Called by every insert site. +/// The rebuild check runs BEFORE the bit is set, never after: a rebuild +/// reconstructs the filter from the maps' live keys, and this key is not in +/// the map yet at any call site, so rebuilding afterwards would erase the bit +/// just set and make a live record invisible to the filter. +#[inline] +fn layout_addr_filter_add(user_ptr: usize) { + if hot_per_object_layout_hint().sets.get() >= LAYOUT_ADDR_FILTER_REBUILD_AFTER { + layout_addr_filter_rebuild(); + } + layout_addr_filter_note(user_ptr); +} + +/// Set `user_ptr`'s bit WITHOUT the rebuild check. +/// +/// `layout_note_slot` calls this from inside its own `borrow_mut` on +/// `LAYOUT_SLOT_MASKS`, and [`layout_addr_filter_rebuild`] borrows both maps — +/// so the rebuilding form would panic there. Skipping the rebuild is safe: the +/// counter is an accuracy heuristic, not a correctness one. +#[inline] +pub(in crate::gc) fn layout_addr_filter_note(user_ptr: usize) { + let hint = hot_per_object_layout_hint(); + let (word, bit) = layout_addr_filter_slot(user_ptr); + unsafe { + (*hint.filter.get())[word] |= bit; + } + hint.sets.set(hint.sets.get().saturating_add(1)); +} + +/// Drop every bit and re-add the live keys. Removals cannot clear a bit on +/// their own (two keys may share one), so this is what keeps a workload that +/// genuinely churns per-object records from saturating the filter forever. +fn layout_addr_filter_rebuild() { + layout_addr_filter_clear(); + let keys: Vec = { + let masks = hot_layout_slot_masks().borrow(); + let typed = hot_typed_layouts().borrow(); + masks.keys().copied().chain(typed.keys().copied()).collect() + }; + let hint = hot_per_object_layout_hint(); + for k in keys { + let (word, bit) = layout_addr_filter_slot(k); + unsafe { + (*hint.filter.get())[word] |= bit; + } + } + hint.sets.set(0); +} + +fn layout_addr_filter_clear() { + let hint = hot_per_object_layout_hint(); + unsafe { + (*hint.filter.get()).fill(0); + } + hint.sets.set(0); +} + +crate::perry_thread_local! { + /// Nesting depth of the innermost [`ImmortalLayoutScope`]. + /// + /// Read only from the *cold* half of `layout_note_slot` — the branch that + /// would otherwise mint a brand-new per-object mask — so an inactive scope + /// costs nothing on any hot path. + static IMMORTAL_LAYOUT_SCOPE_DEPTH: Cell = const { Cell::new(0) }; +} + +/// True while an [`ImmortalLayoutScope`] is open on this thread. +#[inline] +pub(in crate::gc) fn immortal_layout_scope_active() -> bool { + IMMORTAL_LAYOUT_SCOPE_DEPTH.with(|d| d.get()) != 0 +} + +/// Marks a window whose objects are **immortal by construction** — reachable +/// from a GC root for the life of the process — so that none of them may take +/// out a per-object layout record. +/// +/// ## Why this exists (#7510's lesson, repeating) +/// +/// [`PER_OBJECT_LAYOUTS_NONEMPTY`] is a *global emptiness* proof: it turns +/// `layout_forget_object` — which runs on every allocation, every object death +/// and every relocation — into a single load whenever both side tables happen +/// to be empty. On a monomorphic workload they are empty for the entire run, +/// which is exactly what makes the accelerator worth having. +/// +/// A *single* entry that is never removed converts that accelerator into +/// permanently-disabled code. #7510 already paid this once, via one interned +/// keys-array the shape cache anchored forever. The `globalThis` bootstrap is +/// the same trap at several hundred times the scale: it builds hundreds of +/// plain objects whose first pointer field mints a mask, every one of them +/// rooted at `globalThis` forever. Since a plain-object property miss forces +/// that bootstrap, *every real TypeScript program* armed the regime before +/// user code ran — measured as +28% on `churn` and +29% on `tree`, with +/// `layout_forget_object` going 112 → 916 ms and 194 → 740 ms of self time. +/// +/// ## Why dropping the mask is sound +/// +/// The alternative the code already uses for the very same situation — a +/// pointer stored into an object whose state is not `POINTER_FREE` — is +/// [`super::layout::GC_LAYOUT_UNKNOWN`], the tag-checked payload scan. That is +/// the universally safe state, not a weaker one; the mask is a *precision* +/// optimization that lets the collector skip known-pointer-free slots. +/// +/// For an immortal object that precision buys nothing measurable: the object +/// is never reclaimed, and it is scanned only as part of the root graph. It +/// costs one tag test per slot on a few hundred objects, against two `RefCell` +/// round-trips and two hash probes on every allocation the program will ever +/// make. +/// +/// ## Deliberately NOT applied to typed-shape layouts +/// +/// The scope gates ONLY the mask-minting branch of `layout_note_slot`. It must +/// never redirect a *typed* layout (`init_typed_shape_layout`, +/// `layout_rebuild_from_slots`) to `GC_LAYOUT_UNKNOWN`: those describe objects +/// with raw-f64 slots, and a raw double's bit pattern can alias a heap pointer. +/// A conservative scan would then trace — and, under a copying collector, +/// *rewrite* — a slot holding a number. `GC_LAYOUT_UNKNOWN` is only safe where +/// every slot is a NaN-boxed value, which is what the mask-minting branch +/// already assumes. +pub struct ImmortalLayoutScope { + _not_send: std::marker::PhantomData<*const ()>, +} + +impl Default for ImmortalLayoutScope { + fn default() -> Self { + Self::new() + } +} + +impl ImmortalLayoutScope { + pub fn new() -> Self { + IMMORTAL_LAYOUT_SCOPE_DEPTH.with(|d| d.set(d.get().saturating_add(1))); + Self { + _not_send: std::marker::PhantomData, + } + } +} + +impl Drop for ImmortalLayoutScope { + fn drop(&mut self) { + IMMORTAL_LAYOUT_SCOPE_DEPTH.with(|d| d.set(d.get().saturating_sub(1))); + } +} + +/// Live entry counts of the two per-object side tables, for `PERRY_GC_DIAG` +/// and for the tests that assert the bootstrap left them alone. +pub(crate) fn per_object_layout_table_sizes() -> (usize, usize) { + ( + hot_layout_slot_masks().borrow().len(), + hot_typed_layouts().borrow().len(), + ) } /// True when either per-object side table may hold an entry. `false` is a @@ -55,7 +306,7 @@ thread_local! { /// hint, so every caller still has to handle a miss. #[inline(always)] pub(in crate::gc) fn per_object_layouts_maybe_nonempty() -> bool { - hot_per_object_layouts_nonempty().get() + hot_per_object_layout_hint().nonempty.get() } /// Arm the flag. Called by anything that inserts into either map — including @@ -63,7 +314,7 @@ pub(in crate::gc) fn per_object_layouts_maybe_nonempty() -> bool { /// through the wrappers below. #[inline(always)] pub(in crate::gc) fn mark_per_object_layouts_nonempty() { - hot_per_object_layouts_nonempty().set(true); + hot_per_object_layout_hint().nonempty.set(true); } /// Re-establish the flag after a removal emptied one map: clear it once the @@ -79,7 +330,11 @@ pub(in crate::gc) fn refresh_per_object_layouts_flag(touched_map_emptied: bool) return; } if hot_layout_slot_masks().borrow().is_empty() && hot_typed_layouts().borrow().is_empty() { - hot_per_object_layouts_nonempty().set(false); + hot_per_object_layout_hint().nonempty.set(false); + // Both maps are empty, so every bit is now stale. Clearing here is what + // makes the filter's occupancy track LIVE entries rather than every + // entry the program has ever created. + layout_addr_filter_clear(); } } @@ -87,6 +342,7 @@ pub(in crate::gc) fn refresh_per_object_layouts_flag(touched_map_emptied: bool) #[inline] pub(in crate::gc) fn typed_layouts_insert(user_ptr: usize, descriptor: TypedLayoutDescriptor) { mark_per_object_layouts_nonempty(); + layout_addr_filter_add(user_ptr); hot_typed_layouts() .borrow_mut() .insert(user_ptr, descriptor); @@ -96,13 +352,14 @@ pub(in crate::gc) fn typed_layouts_insert(user_ptr: usize, descriptor: TypedLayo #[inline] pub(in crate::gc) fn slot_masks_insert(user_ptr: usize, mask: LayoutSlotMask) { mark_per_object_layouts_nonempty(); + layout_addr_filter_add(user_ptr); hot_layout_slot_masks().borrow_mut().insert(user_ptr, mask); } /// Drop `user_ptr`'s per-object typed descriptor (only). #[inline] pub(in crate::gc) fn typed_layouts_remove(user_ptr: usize) { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return; } let emptied = { @@ -115,7 +372,7 @@ pub(in crate::gc) fn typed_layouts_remove(user_ptr: usize) { /// Drop `user_ptr`'s per-object pointer mask (only). #[inline] pub(in crate::gc) fn slot_masks_remove(user_ptr: usize) { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return; } let emptied = { @@ -133,7 +390,7 @@ pub(in crate::gc) fn with_per_object_descriptor( user_ptr: usize, f: impl FnOnce(&TypedLayoutDescriptor) -> R, ) -> Option { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return None; } hot_typed_layouts().borrow().get(&user_ptr).map(f) @@ -144,7 +401,7 @@ pub(in crate::gc) fn with_per_object_descriptor( /// as much here as it is on the mutator side. #[inline] pub(in crate::gc) fn per_object_slot_mask(user_ptr: usize) -> Option { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return None; } hot_layout_slot_masks().borrow().get(&user_ptr).cloned() @@ -160,7 +417,12 @@ pub(in crate::gc) fn per_object_slot_mask(user_ptr: usize) -> Option bool { - if !per_object_layouts_maybe_nonempty() { + // BOTH addresses are touched (the destination is cleared of a previous + // tenant's record before the source's is moved in), so the filter can only + // prove this call unnecessary when it proves both absent. + if !per_object_layouts_maybe_nonempty() + || (!layout_addr_filter_may_hold(old_user) && !layout_addr_filter_may_hold(new_user)) + { return false; } let mut typed = hot_typed_layouts().borrow_mut(); @@ -168,6 +430,8 @@ pub(in crate::gc) fn transfer_per_object_descriptor(old_user: usize, new_user: u match typed.remove(&old_user) { Some(layout) => { typed.insert(new_user, layout); + drop(typed); + layout_addr_filter_add(new_user); true } None => false, @@ -177,13 +441,17 @@ pub(in crate::gc) fn transfer_per_object_descriptor(old_user: usize, new_user: u /// Move `old_user`'s per-object pointer mask to `new_user` (relocation). #[inline] pub(in crate::gc) fn transfer_per_object_slot_mask(old_user: usize, new_user: usize) { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() + || (!layout_addr_filter_may_hold(old_user) && !layout_addr_filter_may_hold(new_user)) + { return; } let mut masks = hot_layout_slot_masks().borrow_mut(); masks.remove(&new_user); if let Some(mask) = masks.remove(&old_user) { masks.insert(new_user, mask); + drop(masks); + layout_addr_filter_add(new_user); } } @@ -200,7 +468,11 @@ pub(in crate::gc) fn transfer_per_object_slot_mask(old_user: usize, new_user: us /// pre-#7510 path unchanged, and re-arms the flag on the way out. #[inline] pub(in crate::gc) fn layout_forget_object(user_ptr: usize) { - if !per_object_layouts_maybe_nonempty() { + // ONE hot-slot resolution for both halves of the guard: the flag (cheap, + // and false for the overwhelming majority of workloads) and then the + // address filter (what rescues a workload with an immortal record). + let hint = hot_per_object_layout_hint(); + if !hint.nonempty.get() || !hint_may_hold(hint, user_ptr) { return; } // One `borrow_mut` per map, not a `borrow` to test emptiness followed by a diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 76baed3973..5116612cec 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -75,6 +75,11 @@ pub(crate) use layout_slot_visit::*; /// keeps them off the allocation, store, death and trace paths. Split out of /// `layout.rs` so it stays under the repo's 2000-line-per-file cap. mod layout_tables; +// The immortal-object construction window and the table-occupancy readout, both +// consumed from OUTSIDE `gc`: `object::global_this` opens the window around the +// `globalThis` bootstrap and prints the residue under `PERRY_GC_DIAG`. +pub(crate) use layout_tables::per_object_layout_table_sizes; +pub use layout_tables::ImmortalLayoutScope; /// #7510 item 1: the construction-side memo that turns an already-installed /// typed shape into two header bit-writes instead of a descriptor build plus a /// `SHAPE_LAYOUTS` round-trip. diff --git a/crates/perry-runtime/src/gc/telemetry.rs b/crates/perry-runtime/src/gc/telemetry.rs index e3cc182a02..7c0118fbd3 100644 --- a/crates/perry-runtime/src/gc/telemetry.rs +++ b/crates/perry-runtime/src/gc/telemetry.rs @@ -3,6 +3,14 @@ use super::*; /// Number of most-recent pause samples retained per thread (#6187). pub const GC_RECENT_PAUSE_WINDOW: usize = 32; +/// Is `PERRY_GC_DIAG` set? Read once and cached, so a diagnostic call site can +/// sit on a path that runs before/around `main` without paying a `getenv` each +/// time. Diagnostic-only: nothing may branch on this for behaviour. +pub fn gc_diag_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| std::env::var_os("PERRY_GC_DIAG").is_some()) +} + pub struct GcStats { pub collection_count: u64, pub total_freed_bytes: u64, diff --git a/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs b/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs index 007df0435f..40502986f1 100644 --- a/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs +++ b/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs @@ -11,9 +11,10 @@ use super::*; use crate::gc::layout_tables::{test_per_object_tables_are_empty, PER_OBJECT_LAYOUTS_NONEMPTY}; +use crate::gc::ImmortalLayoutScope; fn flag() -> bool { - PER_OBJECT_LAYOUTS_NONEMPTY.with(|c| c.get()) + PER_OBJECT_LAYOUTS_NONEMPTY.with(|h| h.nonempty.get()) } /// The invariant itself: never `false` while either map holds an entry. @@ -241,3 +242,222 @@ fn test_class_keys_array_declares_all_pointer_slots_instead_of_a_mask() { clear_marks(); clear_mark_seeds(); } + +/// The control half of the immortal-scope pair: WITHOUT the scope, the very +/// same store mints a per-object mask and arms the flag. +/// +/// This exists so the scoped test below cannot pass vacuously. If a future +/// change stops routing this shape through `layout_note_slot`'s mask-minting +/// branch at all, this test goes red and says so, rather than leaving its +/// partner green while testing nothing. +#[test] +fn test_pointer_store_outside_an_immortal_scope_still_mints_a_mask() { + clear_marks(); + clear_mark_seeds(); + assert_flag_sound("before store"); + assert!(test_per_object_tables_are_empty()); + + let obj = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + crate::gc::layout_note_slot(obj as usize, 1, POINTER_TAG | (child as u64 & POINTER_MASK)); + + assert!( + flag(), + "a first pointer store into a pointer-free object must mint a mask — \ + if it no longer does, the scoped test below proves nothing" + ); + assert!(!test_per_object_tables_are_empty()); + assert_eq!(test_layout_pointer_slot_count(obj as usize, 2), Some(1)); + + crate::gc::layout_clear_for_ptr(obj as usize); + assert!(test_per_object_tables_are_empty()); + + clear_marks(); + clear_mark_seeds(); +} + +/// Inside an [`ImmortalLayoutScope`] the identical store must leave both maps +/// empty — and the object must still trace its child, because the fallback is +/// `GC_LAYOUT_UNKNOWN` (the tag-checked scan), not "no pointers here". +#[test] +fn test_immortal_scope_stores_trace_without_taking_a_side_table_entry() { + clear_marks(); + clear_mark_seeds(); + assert!(test_per_object_tables_are_empty()); + + let obj = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + let child_header = unsafe { header_from_user_ptr(child as *const u8) }; + unsafe { + *(obj as *mut u8).add(8).cast::().add(1) = POINTER_TAG | (child as u64 & POINTER_MASK); + } + { + let _immortal = ImmortalLayoutScope::new(); + crate::gc::layout_note_slot(obj as usize, 1, POINTER_TAG | (child as u64 & POINTER_MASK)); + } + + assert!( + test_per_object_tables_are_empty(), + "an object built inside an ImmortalLayoutScope must not take out a \ + per-object layout record — one permanent entry disables the emptiness \ + fast path for every allocation the process will ever make" + ); + assert!(!flag()); + + // Correctness half: the child is still reached. `GC_LAYOUT_UNKNOWN` scans + // the payload with a tag check, so precision is lost but nothing is missed. + let valid_ptrs = build_valid_pointer_set(); + assert!(try_mark_value( + POINTER_TAG | (obj as u64 & POINTER_MASK), + &valid_ptrs + )); + trace_marked_objects(&valid_ptrs); + unsafe { + assert_ne!( + (*child_header).gc_flags & GC_FLAG_MARKED, + 0, + "the child must still be traced through the tag-checked scan the \ + immortal scope falls back to" + ); + } + + clear_marks(); + clear_mark_seeds(); +} + +/// The acceptance gate for the `globalThis` bootstrap itself. +/// +/// The bootstrap builds several hundred permanently-rooted objects. Before the +/// `ImmortalLayoutScope` around it, each of their first pointer fields minted a +/// mask that nothing would ever remove, so the first plain-object property miss +/// in ANY program permanently disarmed `PER_OBJECT_LAYOUTS_NONEMPTY`. +/// +/// The assertion is paired with a subject-live check: `Array` must resolve to a +/// real closure value, so a bootstrap that silently did nothing cannot pass this +/// by leaving the tables trivially empty. +#[test] +fn test_global_this_bootstrap_leaves_the_per_object_layout_tables_empty() { + let global = crate::object::js_get_global_this(); + assert_eq!( + global.to_bits() >> 48, + 0x7FFD, + "globalThis must be a real heap object for this test to mean anything" + ); + // Subject-live: the bootstrap actually populated the singleton. + let array_ctor = crate::object::js_get_global_this_builtin_value(b"Array".as_ptr(), 5); + assert_eq!( + array_ctor.to_bits() >> 48, + 0x7FFD, + "globalThis.Array must be populated — otherwise the emptiness assertion \ + below is vacuous" + ); + + let (slot_masks, typed) = crate::gc::per_object_layout_table_sizes(); + assert_eq!( + (slot_masks, typed), + (0, 0), + "the globalThis bootstrap left {slot_masks} slot-mask and {typed} typed \ + per-object layout records behind; every one of them is immortal, so \ + `layout_forget_object` now runs its full two-map probe on every \ + allocation, death and relocation for the rest of the process" + ); +} + +/// The address filter is an *accelerator*, never an authority: `false` must be +/// a proof of absence and nothing else may rest on it. This drives enough +/// inserts to force at least one filter rebuild and then checks, for every +/// live record, that the guarded accessors still find it and that +/// `layout_forget_object` still removes it. +#[test] +fn test_addr_filter_never_hides_a_live_record_across_a_rebuild() { + clear_marks(); + clear_mark_seeds(); + + // Comfortably more inserts than the rebuild threshold (half the bits), so + // the rebuild path is exercised rather than merely reachable. + let mut objs = Vec::new(); + for _ in 0..6000 { + let obj = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + crate::gc::layout_note_slot(obj as usize, 1, POINTER_TAG | (child as u64 & POINTER_MASK)); + objs.push(obj); + } + assert!(flag(), "6000 masks must arm the flag"); + + for (i, obj) in objs.iter().enumerate() { + assert_eq!( + test_layout_pointer_slot_count(*obj as usize, 2), + Some(1), + "record {i} became invisible — the filter proved absence for an \ + address that has a live entry" + ); + } + for obj in &objs { + crate::gc::layout_clear_for_ptr(*obj as usize); + } + assert!( + test_per_object_tables_are_empty(), + "every record must still be removable after a filter rebuild" + ); + assert!(!flag()); + + clear_marks(); + clear_mark_seeds(); +} + +/// Subject-live check for the accelerator itself: with a record present (so +/// `PER_OBJECT_LAYOUTS_NONEMPTY` is armed and the old global test would force +/// the full two-map probe), an unrelated address must still be *proved absent*. +/// +/// This is the whole point of the address filter. The global flag alone cannot +/// distinguish "some object somewhere has a record" from "this address has a +/// record", so one immortal entry — which the `globalThis` bootstrap used to +/// leave 1113 of, and which ordinary runtime init still leaves one or two of — +/// put every allocation, death and relocation in the process back on the slow +/// path. If this assertion ever fails, the accelerator has silently stopped +/// accelerating even though nothing throws. +#[test] +fn test_addr_filter_proves_absence_while_the_global_flag_is_armed() { + clear_marks(); + clear_mark_seeds(); + + let live = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + crate::gc::layout_note_slot( + live as usize, + 1, + POINTER_TAG | (child as u64 & POINTER_MASK), + ); + assert!( + flag(), + "the global flag must be armed for this test to mean anything" + ); + assert!(crate::gc::layout_tables::layout_addr_filter_may_hold( + live as usize + )); + + // A large sample of unrelated addresses: with one live record in 8192 bits + // the filter must prove nearly all of them absent. Anything close to 100% + // "maybe" means the filter is saturated or mis-hashed and the fast path is + // gone even though every test still passes. + let probes = 4096usize; + let mut maybe = 0usize; + for i in 0..probes { + let addr = 0x2000_0000_0000usize + i * 64; + if addr == live as usize { + continue; + } + if crate::gc::layout_tables::layout_addr_filter_may_hold(addr) { + maybe += 1; + } + } + assert!( + maybe * 100 < probes, + "{maybe}/{probes} unrelated addresses were not proved absent — the \ + address filter is not accelerating anything" + ); + + crate::gc::layout_clear_for_ptr(live as usize); + clear_marks(); + clear_mark_seeds(); +} diff --git a/crates/perry-runtime/src/object/global_this/populate.rs b/crates/perry-runtime/src/object/global_this/populate.rs index c93b4ba2bd..aeb36ef1a0 100644 --- a/crates/perry-runtime/src/object/global_this/populate.rs +++ b/crates/perry-runtime/src/object/global_this/populate.rs @@ -77,6 +77,25 @@ pub(crate) fn populate_global_this_builtins(singleton_at_entry: *mut ObjectHeade // whole window: ~1.15 MB allocated, ~410 KB of it live afterwards, once per // thread. let _no_move = crate::gc::GcSuppressScope::new(); + // Every object this bootstrap builds is reachable from `globalThis` for the + // life of the process (the `_no_move` comment above measures the graph: + // ~1.15 MB allocated, ~410 KB live afterwards). That makes each of them a + // permanent tenant of the per-object GC slot-layout side tables — and ONE + // permanent tenant is enough to disable `PER_OBJECT_LAYOUTS_NONEMPTY`, + // the global emptiness proof that keeps `layout_forget_object` off the + // allocation / death / relocation paths, for the rest of the run. + // + // Since a single plain-object property miss forces this bootstrap, that + // made the armed regime universal: measured on the quiet mini, adding one + // `for (const _x of [1]) {}` to `main()` cost `churn` +28% and `tree` +29%, + // with `layout_forget_object` self time going 112 → 916 ms and 194 → 740 ms + // respectively. The scope makes these objects declare + // `GC_LAYOUT_UNKNOWN` (the tag-checked scan — the universally safe state) + // instead of minting a mask nothing will ever remove. See + // `gc::ImmortalLayoutScope` for the soundness argument and for why it is + // deliberately NOT applied to typed-shape layouts. + let _immortal = crate::gc::ImmortalLayoutScope::new(); + let bootstrap_started = crate::gc::gc_diag_enabled().then(std::time::Instant::now); let scope = crate::gc::RuntimeHandleScope::new(); let singleton_handle = scope.root_raw_mut_ptr(singleton_at_entry); let singleton = || singleton_handle.get_raw_mut_ptr::(); @@ -740,6 +759,20 @@ pub(crate) fn populate_global_this_builtins(singleton_at_entry: *mut ObjectHeade // same function object as `Array.prototype.toString`. Alias it now that // both the Array constructor and the TypedArray intrinsic are set up. alias_typed_array_proto_to_string(singleton()); + // The bootstrap's own cost, and the evidence that the `ImmortalLayoutScope` + // above actually did its job. `slot_masks`/`typed` are the live entry + // counts of the two per-object layout side tables: they must still read + // `0 0` here, because a non-zero count is exactly what disables + // `PER_OBJECT_LAYOUTS_NONEMPTY` for the rest of the process. + if let Some(started) = bootstrap_started { + let (slot_masks, typed) = crate::gc::per_object_layout_table_sizes(); + eprintln!( + "[gc-globalthis-bootstrap] elapsed_us={} per_object_slot_masks={} per_object_typed_layouts={}", + started.elapsed().as_micros(), + slot_masks, + typed + ); + } } /// Install `%TypedArray%.prototype.toString` as the same closure object as