Skip to content

feat(compiler): FFI and C# bindings for RVM, incl. host-await built-ins#672

Open
kusha wants to merge 11 commits into
microsoft:mainfrom
kusha:markbirger/host-await-ffi-csharp-bindings
Open

feat(compiler): FFI and C# bindings for RVM, incl. host-await built-ins#672
kusha wants to merge 11 commits into
microsoft:mainfrom
kusha:markbirger/host-await-ffi-csharp-bindings

Conversation

@kusha

@kusha kusha commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Exposes the registered host-await builtins, Program compilation, and RVM runtime accessors through the FFI layer and C# bindings. This is the companion to #667 (registered host-await builtins in the compiler/VM), making the feature usable from C# consumers.

Rebased onto the updated #667. The multi-identifier SetHostAwaitResponses API (below) supersedes the earlier single-identifier form and addresses the review note about regorus_rvm_set_host_await_responses clearing other identifiers' queues.

Motivation

The previous PR added registered host-await builtins to the Rust compiler and VM. However, the FFI boundary and C# bindings only exposed the raw __builtin_host_await path. This PR bridges the gap so C# consumers can:

  1. Register host-await builtins at compile time - pass builtin names to Program.CompileFromModules, which emits HostAwait instructions directly.
  2. Pre-load responses for run-to-completion mode - call Rvm.SetHostAwaitResponses to queue responses (for any number of identifiers) before execution.
  3. Inspect suspension state - call Rvm.GetHostAwaitIdentifier() and Rvm.GetHostAwaitArgument() to determine which builtin suspended and with what argument.

Changes

Rust VM (src/rvm/vm/machine.rs)

  • get_host_await_argument() and get_host_await_identifier() - const fn accessors that return the argument/identifier when the VM is in a HostAwait-suspended state.

FFI (bindings/ffi/src/rvm.rs)

  • RegorusHostAwaitBuiltin - #[repr(C)] struct carrying a builtin name. The argument count is fixed to 1 by the compiler, so it is not part of the FFI surface.
  • regorus_program_compile_from_modules_with_host_await - compile from modules with registered host-await builtins; the non-host-await path is treated as the empty-builtins case.
  • regorus_rvm_set_host_await_responses - pre-load responses for run-to-completion mode. Accepts an array of RegorusHostAwaitResponseSet (one per identifier), so multiple identifiers are configured in a single call.
  • regorus_rvm_get_host_await_argument / regorus_rvm_get_host_await_identifier - JSON accessors for suspension state.

C# Bindings (bindings/csharp/Regorus/)

  • HostAwaitBuiltin - readonly struct wrapping a builtin name (single-argument constructor; the arg count is fixed to 1 at the compiler level).
  • ExecutionMode - enum (RunToCompletion, Suspendable).
  • Program.CompileFromModules - overload accepting IReadOnlyList<HostAwaitBuiltin>.
  • Rvm.SetHostAwaitResponses(IReadOnlyDictionary<string, IReadOnlyList<string>>) - queue JSON responses for any number of identifiers in one call. Replaces the entire response configuration.
  • Rvm.GetHostAwaitArgument() / GetHostAwaitIdentifier() - read suspension state.
  • ModuleMarshalling - PinnedUtf8Strings, PinnedHostAwaitBuiltins, and PinnedHostAwaitResponseSets helpers for safe FFI marshalling.

Documentation

  • bindings/csharp/README.md - suspendable and run-to-completion examples with registered builtins.
  • bindings/csharp/API.md - Program, Rvm, HostAwaitBuiltin, ExecutionMode API reference.
  • docs/rvm/vm-runtime.md - the new VM accessors.

Tests (bindings/csharp/Regorus.Tests/RvmProgramTests.cs)

  • RegisteredHostAwait_Suspendable_SuspendAndResume - registers get_account, suspends, verifies identifier and argument, resumes with a response.
  • RegisteredHostAwait_RunToCompletion_WithPreloadedResponses - registers translate, pre-loads a response, verifies end-to-end execution.
  • RegisteredHostAwait_RunToCompletion_MultipleIdentifiersInSingleCall - preloads responses for two identifiers in one SetHostAwaitResponses call.
  • RegisteredHostAwait_RunToCompletion_FailsWhenResponseQueueExhausted - missing response surfaces as an error.
  • RegisteredHostAwait_GetAccessorsReturnNullWhenVmIsNotSuspended - accessors return null outside a suspension.
  • RegisteredHostAwait_CompileRejectsEmptyOrWhitespaceName / CompileRejectsDuplicateRegistration / CompileRejectsReservedName - registration validation errors surface through the bindings.

Notes

  • SetHostAwaitResponses replaces the full response set: it configures responses for all supplied identifiers in a single call and replaces any previously configured responses. This mirrors the underlying RegoVM::set_host_await_responses replace-all semantics (pre-existing upstream behavior).

  • CompileFromEngine does not yet support host-await builtins: Only CompileFromModules accepts HostAwaitBuiltin[]. This is a scoping choice to keep the PR smaller - there is no technical constraint preventing it. The engine-based path can be extended in a follow-up if needed.

@kusha
kusha force-pushed the markbirger/host-await-ffi-csharp-bindings branch from fa62435 to 54fbb82 Compare April 9, 2026 20:31
@anakrish
anakrish requested a review from Copilot April 9, 2026 21:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the RVM host-await feature end-to-end by wiring “registered host-await builtins” through the Rust compiler, VM accessors, the FFI layer, and the C# bindings, with accompanying docs and tests.

Changes:

  • Add compiler support to register host-awaitable builtin names at compile time and emit HostAwait for natural function-call syntax.
  • Expose host-await suspension inspection (identifier/argument) and run-to-completion response preloading via FFI + C# APIs.
  • Add Rust YAML regression cases plus new C# binding tests and documentation updates.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/rvm/rego/mod.rs Extends YAML harness to pass registered host-await builtins into compilation and (optionally) assert host-await arguments in suspendable mode.
tests/rvm/rego/cases/registered_host_await.yaml Adds regression coverage for registered builtins (shadowing, queueing, arg-count validation, overrides).
src/rvm/vm/machine.rs Adds VM accessors for host-await identifier/argument when suspended.
src/languages/rego/compiler/rules.rs Adds compile_from_policy_with_host_await and wires registrations into compilation.
src/languages/rego/compiler/mod.rs Adds storage + registration API for host-awaitable builtins (with validation).
src/languages/rego/compiler/function_calls.rs Updates call resolution/emission to support registered host-await builtins and emit identifier literals.
docs/rvm/vm-runtime.md Documents new VM host-await accessors.
docs/rvm/instruction-set.md Documents registered host-await builtin behavior and resolution order.
docs/rvm/architecture.md Describes explicit vs registered HostAwait emission paths.
bindings/ffi/src/rvm.rs Adds FFI structs/APIs for compile-with-builtins, response preloading, and suspension JSON accessors.
bindings/csharp/Regorus/Rvm.cs Adds C# wrappers for host-await inspection + response preloading.
bindings/csharp/Regorus/Program.cs Adds CompileFromModules overload supporting host-await builtin registrations; refactors compile paths.
bindings/csharp/Regorus/NativeMethods.cs Adds P/Invoke declarations and FFI struct for host-await builtins + new VM functions.
bindings/csharp/Regorus/ModuleMarshalling.cs Generalizes UTF-8 string pinning helper and adds host-await builtin marshalling.
bindings/csharp/Regorus/Compiler.cs Introduces public HostAwaitBuiltin struct for registration.
bindings/csharp/Regorus.Tests/RvmProgramTests.cs Adds suspendable + run-to-completion tests for registered host-await builtins.
bindings/csharp/README.md Adds usage examples for registered host-await in both execution modes.
bindings/csharp/API.md Documents new/expanded Program/Rvm/HostAwaitBuiltin APIs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread bindings/ffi/src/rvm.rs
Comment thread bindings/csharp/Regorus/Rvm.cs Outdated
Comment thread bindings/csharp/Regorus/ModuleMarshalling.cs Outdated
Comment thread tests/rvm/rego/cases/registered_host_await.yaml Outdated
@kusha
kusha force-pushed the markbirger/host-await-ffi-csharp-bindings branch from 54fbb82 to 145ee89 Compare June 29, 2026 06:55
@kusha
kusha marked this pull request as ready for review June 29, 2026 16:49
@anakrish

anakrish commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

@ review this PR using all the review skills in the repo. Use a separate agent for each skill.

@anakrish anakrish left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated multi-perspective review of the host-await FFI / C# binding work. Inline comments below, roughly highest-impact first.

Two items I couldn't attach inline because they fall outside this PR's diff, but that are worth a look while you're in here:

  • RegoVM::set_host_await_responses (machine.rs) accumulates response queues with no enforce_limit()/fallible reservation. This is pre-existing code, not something you changed, but the new FFI surface makes it reachable with caller-supplied volume, so it may be worth hardening as a follow-up.
  • The Vec::with_capacity(caller_len) panic-on-huge-length pattern I flagged in rvm.rs already exists in the older compile_from_modules paths too — a fallible-reservation pass across all of them would be a nice cleanup.

Comment thread bindings/ffi/src/rvm.rs Outdated
) -> RegorusResult {
with_unwind_guard(|| {
to_regorus_result(|| -> Result<()> {
let vm = to_ref(vm)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These new host-await entry points reach for the VM handle with to_ref(vm)?, but every other RVM method in this file (e.g. the run/step/result functions above) goes through to_shared_ref(vm as *const RegorusRvm)? and then takes the lock via try_read/try_write. That ordering isn't accidental: to_shared_ref is what lets the lock arbitrate concurrent access, whereas to_ref manufactures an exclusive &mut RegorusRvm straight from a caller-owned handle. If two foreign threads call into the same vm here, both can materialize that &mut before either lock is acquired, which is exactly the aliasing hazard the rest of the file is structured to avoid.

Could we switch these three (set_host_await_responses, get_host_await_argument, get_host_await_identifier) to to_shared_ref(...) followed by try_write/try_read, to match the established pattern?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched all three (set_host_await_responses, get_host_await_argument, get_host_await_identifier) to to_shared_ref(vm as *const RegorusRvm)? followed by try_write/try_read, so the lock arbitrates concurrent access like the rest of the file. Left regorus_rvm_drop on to_ref - it needs exclusive ownership for Box::from_raw and takes no lock.

Comment thread bindings/ffi/src/rvm.rs Outdated
return Err(anyhow!("null response_sets pointer"));
}

let mut all = Vec::with_capacity(response_sets_len);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These Vec::with_capacity(response_sets_len) / VecDeque::with_capacity(set.values_len) (and the Vec::with_capacity(len) further down) reserve straight from caller-controlled usize values. A hostile or buggy caller passing something near usize::MAX will trip a capacity-overflow panic inside the unwind guard, which for this engine means process-wide poisoning rather than a clean error return.

I'll flag that this isn't new to your PR — the existing compile_from_modules paths in this file already use with_capacity(modules_len) the same way — so I don't think it should block the PR on its own. But since we're adding several more instances of the pattern, it'd be a good moment to switch to fallible reservation (try_reserve) or validate against a sane maximum before allocating, and return an FFI error instead of risking a panic.

@kusha kusha Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converted the three new host-await sites to try_reserve, so an absurd caller length returns a clean FFI error instead of a capacity-overflow panic under the unwind guard. As you noted, the pre-existing compile_from_modules sites share the pattern - I've left those out to keep this PR scoped and tracked them as a follow-up.

Comment thread bindings/ffi/src/rvm.rs

let id_str = from_c_str(set.identifier)
.map_err(|e| anyhow!("invalid identifier in response set at index {i}: {e}"))?;
let id_value = Value::String(id_str.into());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth tracing the identifier round-trip here. On the way in we wrap the C string as Value::String(id_str.into()), but the getter below (get_host_await_identifier) hands the identifier back as to_json_str(), i.e. a JSON-encoded value. So a host that reads the current identifier and feeds it straight back gets Value::String("\"get_account\"") — the quotes become part of the string and it no longer matches.

It also means a HostAwait identifier that isn't a string (say an integer 42 baked into the policy) can never be matched from the FFI side at all, since everything coming through here is coerced to Value::String. If identifiers are intended to be string-only, it'd be clearer to document and enforce that and have the getter return the raw string; otherwise we probably want an identifier_json in and a parse back to Value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed - the getter now returns the raw identifier string instead of to_json_str(), so it feeds straight back into set_host_await_responses. Identifiers are string-only per the compiler, so I made the getter enforce that explicitly (rejecting a non-string identifier loudly) rather than the identifier_json-in route. Updated the C# docs and the round-trip test to expect the unquoted form.

Comment thread bindings/ffi/src/rvm.rs
let vm = to_ref(vm)?;
let guard = vm.try_read()?;
match guard.get_host_await_argument() {
Some(arg) => Ok(Some(arg.to_json_str()?)),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to_json_str() is lossy for the Rego-only Value variants. If a policy hands a Set to HostAwait, it serializes here indistinguishably from an Array, and when the host resumes with that JSON it comes back as an Array — so a downstream is_set(resp) flips to false. Undefined has the same problem. For most payloads this is fine, but it's a silent shape change at the boundary. Either we should reject the non-JSON-representable variants explicitly at this boundary (so the failure is loud), or expose a typed marshalling path that preserves them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right it's lossy. We chose to make the convention explicit rather than reject: the argument uses the same Value→JSON encoding as evaluation results (which already emit Set as an array), so documenting it keeps one consistent mental model — and since policies are third-party-authored, rejecting a Set argument would turn a policy-authoring choice into a runtime failure.

The sharper Undefined half is a problem, let's proceed in the thread on RvmProgramTests.cs

Comment thread bindings/ffi/src/rvm.rs
/// FFI boundary. The struct exists as a stable layout to allow future
/// expansion (e.g. an explicit `arg_count` field) without breaking ABI
/// when callers pin a fixed-size array of these.
#[repr(C)]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these #[repr(C)] structs get passed as C arrays, their size is baked into the caller's stride arithmetic. The moment we want to add a field later (an arity, a flags word, etc.) sizeof(RegorusHostAwaitBuiltin) changes and any client compiled against the old layout walks the array incorrectly — a silent ABI break rather than a link error. If we expect these to grow, it's cheap to add a struct_size/version (or a couple of reserved fields) now so future additions stay backward-compatible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an explicit element-size parameter to both array-taking entry points; the native side validates it covers the current layout, then walks the array by the caller-supplied stride - so a caller built against a different struct layout indexes correctly or fails loud rather than corrupting memory. Applied to both new structs, with unit tests for the native-size, undersized-reject, and oversized (newer-caller) cases.

pub fn compile_from_policy_with_host_await(
policy: &CompiledPolicy,
entry_points: &[&str],
host_await_builtins: &[(&str, usize)],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor design heads-up on the public signature: &[(&str, usize)] freezes the registration metadata to exactly "name + arity". The first time we want a third per-builtin attribute we're forced into either a breaking signature change or a parallel _v2 entry point. A small HostAwaitBuiltinSpec (with a constructor/builder) would let this grow additively. Not blocking, just easier to live with long-term while the API is still young.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it'll want a struct once there's a third attribute. Since this is an internal Rust signature (not the C ABI) with only a few in-tree callers, changing it later is a loud compile error rather than a silent break — so I'd rather introduce HostAwaitBuiltinSpec together with the arity work, when the real fields are known, instead of landing a 2-field wrapper now. Ok?

Comment thread bindings/csharp/Regorus/Program.cs Outdated
{
var result = API.regorus_engine_compile_program_with_entrypoints(
(RegorusEngine*)enginePtr,
var result = API.regorus_program_compile_from_modules_with_host_await(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads-up on a compatibility regression: the previous overloads used to call regorus_program_compile_from_modules, and now every path routes through regorus_program_compile_from_modules_with_host_await, including the no-builtins case. That means an existing app that updates only the managed package against an older native library will hit an EntryPointNotFoundException at runtime, even though it never asked for host-await.

Could we keep calling the original regorus_program_compile_from_modules when hostAwaitBuiltins is null/empty, and only reach for the new symbol when there's actually something to register? That keeps the old code path binary-compatible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored the dual-path routing: CompileFromModulesInner calls the original regorus_program_compile_from_modules when there are no builtins, and only reaches for _with_host_await when there's something to register. Keeps the no-host-await path binary-compatible with older native libraries.

Comment thread bindings/csharp/Regorus/Program.cs Outdated
{
if (modules is null)
return CompileFromModulesInner(dataJson, modules, entryPoints, hostAwaitBuiltins: null);
} /// <summary>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny formatting bug: the /// <summary> starts on the same line as the closing }, so the compiler doesn't treat it as a doc comment for the following method — the XML docs for that member silently disappear. A newline before /// <summary> fixes it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed - added the newline so CompileFromEngine's XML docs are recognized.

{
for (int i = 0; i < count; i++)
{
var namePinned = Utf8Marshaller.Pin(builtins[i].Name);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These Utf8Marshaller.Pin(...) calls (the builtin name here, and the response key/value pins further down) hand the string to the native side as a C string, but nothing rejects an embedded \0. Rust's CStr stops at the first NUL, so new HostAwaitBuiltin("foo\0bar") silently registers "foo". It's an edge case, but it's the kind of silent truncation that's painful to debug, so it'd be worth rejecting embedded NULs in the public C# surface (or moving these to pointer+length strings).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed for the identifiers, which is where truncation actually mis-routes: an embedded NUL in the registration name or a response key would silently truncate with no downstream check, so both now reject it (shared Utf8Marshaller.ThrowIfContainsNul). I left response values unvalidated on purpose - they're payload backstopped by from_json_str, and a raw NUL in a JSON value is the same client-responsibility contract that already applies to AddDataJson/SetInputJson and module content, so I didn't want to single out host-await. Tests cover identifier rejection plus escaped \u0000 round-trips on the preset and resume paths.

// test failure that has to be explicitly re-acknowledged.
var actual = vm.Execute();
Assert.AreEqual(
"\"<undefined>\"",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one I'd most want a second opinion on. The test codifies that a run-to-completion HostAwait with no available response yields the string "<undefined>" rather than surfacing an error. Functionally that means a missing response is swallowed as a rule-body failure and propagates as plain undefined.

In a policy engine that's a fail-open shape: if the host meant to supply a response and didn't (misconfig, dropped lookup, etc.), the rule quietly evaluates to undefined instead of telling anyone something went wrong. I'd argue HostAwaitResponseMissing should propagate as an execution error through RVM → FFI → C#, and this assertion should expect an exception. If undefined-on-missing is genuinely the desired contract, it'd be good to spell out why in a comment, because the default reading is surprising.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. To be clear on origin, the behavior is pre-existing from #667HostAwaitResponseMissing already isn't in is_fatal_vm_error (src/rvm/vm/rules.rs), so it's swallowed to undefined; this PR only added the test that pins it. It should propagate: a missing preloaded response is never a valid outcome, and the resource-limit errors are fatal precisely so not builtin(...) can't silently flip. Since fixing it is a core-RVM change across both execution paths and every binding (and it flips this test's contract), I'd prefer a separate focused PR. Would it be ok to solve in a follow-up?

kusha pushed a commit to kusha/regorus that referenced this pull request Jul 17, 2026
PR microsoft#672 review: the host-await FFI functions acquired the VM via
`to_ref`, which fabricates an exclusive `&mut RegorusRvm` before the
lock is taken. Concurrent foreign callers could each materialize that
`&mut` to the same handle, aliasing before `try_read`/`try_write`
arbitrates. Switch `set_host_await_responses`, `get_host_await_argument`,
and `get_host_await_identifier` to `to_shared_ref` + lock, matching the
other RVM entry points in this file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mark Birger and others added 11 commits July 17, 2026 17:55
Expose registered host-await builtins, Program compilation, and RVM
accessors through the FFI layer and C# bindings.

FFI (bindings/ffi/src/rvm.rs):
- compile_from_modules with host-await builtin registration
- set/get host-await responses, argument, identifier
- RegorusHostAwaitBuiltin C struct

C# (bindings/csharp/Regorus/):
- Program.CompileFromModules overloads with HostAwaitBuiltin[]
- Rvm.SetHostAwaitResponses, GetHostAwaitArgument, GetHostAwaitIdentifier
- HostAwaitBuiltin readonly struct, ExecutionMode enum
- ModuleMarshalling: PinnedUtf8Strings, PinnedHostAwaitBuiltins

Rust (src/rvm/vm/machine.rs):
- get_host_await_argument() and get_host_await_identifier() accessors

Docs: README examples, API.md reference, vm-runtime.md accessors
Tests: suspendable + run-to-completion C# scenarios
- HostAwaitBuiltin: single-arg (name) ctor; arg_count fixed to 1
- SetHostAwaitResponses: multi-identifier dictionary API
- always route CompileFromModules through host-await FFI
- dead-check/IIFE cleanup, pub(crate), expanded tests + docs
PR microsoft#672 review: the host-await FFI functions acquired the VM via
`to_ref`, which fabricates an exclusive `&mut RegorusRvm` before the
lock is taken. Concurrent foreign callers could each materialize that
`&mut` to the same handle, aliasing before `try_read`/`try_write`
arbitrates. Switch `set_host_await_responses`, `get_host_await_argument`,
and `get_host_await_identifier` to `to_shared_ref` + lock, matching the
other RVM entry points in this file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The three host-await entry points allocated Vec/VecDeque with capacities
taken directly from caller-supplied lengths (response_sets_len,
values_len, builtins len). A hostile or buggy caller passing an absurd
length would trigger a capacity-overflow panic inside with_unwind_guard,
flipping the process-global poison flag and permanently disabling every
engine instance in the process.

Switch these three sites to fallible reservation (try_reserve) so an
over-large request becomes a clean RegorusStatus error instead of a
panic. Only the three new host-await sites are converted here; the
pre-existing with_capacity sites elsewhere are left as a follow-up.

This introduces the try_reserve pattern to the repo (not previously
used); enforce_limit() is not applicable at the FFI boundary since it
bounds cumulative evaluation allocation, not a single caller-controlled
reservation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
regorus_rvm_get_host_await_identifier serialized the identifier via
to_json_str(), so a string identifier came back JSON-quoted (e.g.
"get_account" with literal surrounding quotes). But
regorus_rvm_set_host_await_responses consumes the identifier as a raw C
string and wraps it in Value::String. The core matches identifiers by
exact Value equality, so the getter's quoted output did not round-trip
back into the setter.

Return the raw identifier string instead, symmetric with the setter, and
reject non-string identifiers loudly (identifiers are always strings per
the compiler). Update the C# wrapper docs and the round-trip test to
expect the unquoted form.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The host-await argument getter serializes the policy-supplied value with
the engine's canonical Value->JSON encoding, identical to evaluation
results. Note this in vm-runtime.md so hosts know the Rego-only variants
are encoded (a Set as an array, Undefined as the string "<undefined>")
rather than preserved, since JSON cannot round-trip them back.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The host-await builtin and response-set structs are passed as C arrays, so
their size is baked into the caller's stride arithmetic. Adding a field to
either struct later (e.g. an arity) would change sizeof and make a client
compiled against the old layout mis-walk the array -- a silent ABI break.

Add an explicit element-size parameter to the two array-taking entry points
(regorus_program_compile_from_modules_with_host_await and
regorus_rvm_set_host_await_responses). The native side validates the size
covers the current layout, then walks the array by the caller-supplied
stride instead of native sizeof, so a caller built against a different
(older/newer) struct layout still indexes correctly or fails loud rather
than corrupting memory.

The C# bindings pass sizeof(struct); the cbindgen headers regenerate
automatically. Adds Rust unit tests covering the native-size path, the
undersized-stride rejection, the oversized-stride (newer caller) forward-
compat direction, and the len==0 no-op.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
register_host_await_builtin matched the as-written call path, so
registering a data.*-qualified name (e.g. "data.other.lookup") would
intercept the package-qualified call data.other.lookup(x) and silently
shadow the user-defined rule/function at that path -- breaking the
documented guarantee that package-qualified calls always resolve to the
user rule (the reliable bypass around host interception).

Reject names starting with "data." at registration. This closes the
loophole while preserving the two intended uses: bare names register
host functions, and builtin-namespace names (e.g. time.parse_duration_ns,
which are not under data.) still override standard builtins.

Adds a negative case (registered_builtin_rejects_data_qualified_name) to
the registered_host_await suite; the existing builtin-override case still
passes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Since 145ee89, every CompileFromModules path routed through
regorus_program_compile_from_modules_with_host_await, including the
no-builtins case. An app that updates only the managed package against an
older native library (which lacks that symbol) would then hit an
EntryPointNotFoundException at runtime even though it never used
host-await.

Restore the pre-refinement dual-path routing in CompileFromModulesInner:
call the original regorus_program_compile_from_modules when there are no
builtins, and only reach for the _with_host_await symbol when there is
something to register. This is the routing 145ee89 collapsed; that
refinement was pure consolidation and fixed no bug, so restoring the
branch reintroduces nothing (the with-builtins branch still passes the
struct-size argument added later).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The closing brace of CompileFromModules sat on the same line as the
following `/// <summary>`, so the compiler did not treat it as a doc
comment for CompileFromEngine and that member's XML docs silently
disappeared. Add a newline before the doc comment so it is recognized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Strings passed to native code become null-terminated C strings, and
Rust's CStr stops at the first NUL. So a host-await identifier containing
an embedded NUL (e.g. new HostAwaitBuiltin("foo\0bar")) would be silently
truncated ("foo") and mis-route, with no downstream check to catch it.

Add a shared Utf8Marshaller.ThrowIfContainsNul helper and apply it to the
two identifier write-points: the HostAwaitBuiltin constructor (registration
name) and the response key in PinHostAwaitResponseSets. Identifiers are the
"address" of a host-await and have no parser backstop, so truncation there
is silent mis-routing.

Response values are deliberately not validated: they are payload,
backstopped by from_json_str, and a raw NUL in a value is the accepted
binding-wide "client responsibility" contract shared with AddDataJson /
SetInputJson (a raw NUL truncating to invalid JSON surfaces as a loud parse
error; one truncating to valid JSON is silently accepted, same as elsewhere).

Tests: RejectsEmbeddedNulInIdentifier (name + response key), escaped-\u0000
round-trip on preset and resume value paths, and a raw-NUL resume value that
throws from the parse backstop.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kusha
kusha force-pushed the markbirger/host-await-ffi-csharp-bindings branch from 5e790d9 to f1ac26a Compare July 17, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants