TTCN-3 test-execution runtime (interpreter + ntt exec) and ETSI conformance gate - #776
Open
rafael2knokia wants to merge 64 commits into
Open
TTCN-3 test-execution runtime (interpreter + ntt exec) and ETSI conformance gate#776rafael2knokia wants to merge 64 commits into
rafael2knokia wants to merge 64 commits into
Conversation
rafael2knokia
marked this pull request as ready for review
June 16, 2026 10:46
rafael2knokia
added a commit
that referenced
this pull request
Jun 23, 2026
A shared, async-collaboration starting point (for Matthias + upstream ntt, agents welcome) to align the branch with the rest of ntt: - Inventory: the branch is ~99.6% additive (122,746 ins / 488 del, 437 files) - it extends the parser/LSP and adds interpreter/runtime/exec/ codec/port/conformance, not a rewrite. Current state: 97.22% conformance, PR #776 mergeable. - Hard-constraint assessment (evidence-backed): - LSP for ETSI users: extended, not changed - low risk. - Go-to-definition <100ms: off the heavy path (LookupWithDB, not Analyze); position lookup ~25-30us/op, 0 allocs. - Diagnostics: full 156-check Analyze per change is the perf watch-item on huge files - to benchmark/debounce. - ntt run / internal-tool coexistence: run.go + exec.go both exist; new rejections from 156 checks - to reconcile. - Open questions, goals/checklist, and the async-collaboration + gating conventions. No code change. %INT_NO_SW_CHANGE %AI=CLAUDE Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rmance gate
Grow ntt from a TTCN-3 front-end (parser / LSP / tooling) into a full
TTCN-3 test-execution toolchain in Go: a tree-walking interpreter and an
`ntt exec` runtime that compile, execute and verdict TTCN-3 test suites,
validated against the ETSI TTCN-3 conformance suite (4788/4948, 97.22%).
- interpreter/ tree-walking eval: testcases, templates, alt/interleave,
ports (message + procedure), timers, components, OO classes
- runtime/, runtime/exec/ testcase executor, verdicts, port/queue model, --cfg,
reporting, C-ABI/cgo bridge and a pure-Go (goport) port binding
- runtime/codec/ JSON, XML/XER, RAW encvalue/decvalue
- ttcn3/semantic/ additional static checks surfaced via `ntt check`
- ttcn3/syntax/ parser extensions (classes, new node kinds)
- internal/lsp/ rename, signature help, workspace symbols, type definition,
organize imports, folding, references (Go-to-def off the
heavy path; position lookup is microseconds)
- conformance.go ETSI suite harness gating regressions vs a committed baseline
- docs/ conformance roadmap, Go-only embedding, C/C++ test ports
Additive by design (go.mod/go.sum unchanged from master); the execution
engine is not on the front-end hot path unless explicitly invoked. This
branch is squashed for now; a reviewable per-subsystem commit split is
planned as a follow-up.
An unqualified getreply/catch inside a blocking `call(S,...) { }`
response block must treat only S's reply/exception (ETSI 22.3.1 h).
The loopback model previously matched any queued reply/exception, so a
reply/exception left over from an earlier unhandled call on the same
port wrongly matched the block's guard.
- record the signature identifier on every call/reply/raise envelope
(procSignatureName -> PortMessage.Signature)
- stash the blocking call's signature on the scope for its response
block (procCallSigKey, save/restore so nested call blocks compose)
- skip a queued head whose known signature differs from the block's,
in commGuardMatches (bare `[] p.getreply` guard) and in
evalPortReceiveInfo (the `from`-qualified variant)
Scoped to the implicit call-block qualifier: explicit-signature and
getcall matching stay lenient, so the Sem_2204 check fixtures are
untouched. Fixes Sem_220301_CallOperation_019/020; conformance
4788 -> 4790 (97.22% -> 97.26%), 0 per-file regressions. Guarded by
interpreter/proc_signature_test.go.
Off by default, so the ETSI conformance suite is byte-identical
(4790/4948, verified 0 per-file regressions). Callers opt in via
interpreter.TestcaseOptions{RealScheduler: true}; only an external load
driver that embeds ntt to run TTCN-3-side concurrency needs it.
With RealScheduler on, N `alive` PTCs each running
`while(true){ p.send; alt{ [] p.receive [] t_guard.timeout }; t_pace.start; t_pace.timeout }`
run on real goroutines and generate concurrent traffic through their
mapped Go test ports -- the idiomatic TTCN-3 load pattern, which the
default skip/virtual model turns into a no-op (zero requests issued).
- flag: RealScheduler on TestcaseOptions, carried onto TestcaseExec
(realScheduler/mtcID, mu-guarded), read via FindTestcaseExec.
- scheduler: `alive` starts bypass the skip heuristic and fork onto a
goroutine even with while(true)+timer-guard alts; the alt scheduler
gains a combined park (waitForAltCombined) that wakes on port traffic
OR the soonest timer deadline OR the PTC's stop.
- per-PTC stop: `all component.stop` now cancels worker goroutines
(StopPTC + unmap); evalBlockStmts and waitForTimerTimeout honour a
per-PTC stop so a while(true) worker unwinds promptly.
- port routing: TestcaseExec.PortKey qualifies a PTC's port by
component id ("\x00c<id>/<name>"), so N same-named ports get private
instances/queues/drivers and a goport Inject routes each reply back to
the worker that sent it. PortDriver resolves the port type via the
bare name. goport clears its per-instance cache at each testcase
teardown (RegisterExecTeardownHook) so restarting component ids can't
alias a stale TestPort across runs.
Tests (interpreter/realsched_test.go, incl. -race): single-worker loop,
default-off skip (0 sends), 4 workers each receiving their own replies
(4 distinct instances), and cross-testcase instance isolation.
The 97.26% match rate conflates genuinely different outcomes: real execution verdict matches, static parse/semantic rejections, parse-only acceptances (noexecution / no testcase), and negative tests the interpreter merely aborted on. Nothing recorded which was which, so "semantically correct by execution" was invisible. Add ConformanceResult.Provenance (executed | exec-reject | static | parse | parse-only | runtime-error | timeout), set at each classification site, and surface a ConformanceSummary.Provenance histogram plus RealExecRate = executed-matches / considered. The existing PassRate and the --baseline gate are unchanged, so nothing regresses; the new number is purely additive. On the current suite the 97.26% match rate decomposes as 2489 executed (50.54% real execution) + 1764 static + 135 parse + 207 parse-only + 195 exec-reject. Also drop the stale, unreferenced root conformance-baseline.json (a Jun-1 real-execution-only snapshot at 50.98%, now superseded by the live RealExecRate); CI gates on testdata/conformance-baseline.json.
Introduce runtime.SemanticsProfile (Approximate|Strict) as the single coherent carrier the semantic-correctness work converges on; the shipped RealScheduler bool folds into it (RealScheduler:true selects Strict, and TestcaseExec.RealScheduler() now means "strict profile active"). Real concurrent PTC execution is simply the first domain of Strict; further correct-semantics domains attach to the same profile as they land, and the transitional flags are removed once a domain reaches parity. Add `ntt conformance --profile approximate|strict` and `--differential`: the latter re-runs each executed testcase under the strict profile and reports verdict divergences from the approximate (gate) run - the work-list showing exactly where strict semantics change behaviour. The default gate stays approximate, so nothing regresses (97.26% match, 50.54% real execution unchanged).
…tic) First Phase-1 increment toward faithful alt semantics (ES 201 873-4 clause 20). Under ProfileStrict, `alt` now runs evalAltStmtStrict: it keeps the correct part of the best-effort path -- source-order, first-match-wins guard evaluation (a matching guard consumes only its own selected event), [else], repeat, activated defaults -- but REPLACES the verdict-preferring heuristic with honest blocking: when no guard matches it parks on the alt's event sources (port traffic, soonest timer deadline, component transition, or this PTC's stop) and re-snapshots, never fabricating a verdict for a branch whose guard did not fire. Reached only under ProfileStrict (interleave still uses best-effort); the default approximate path and the conformance gate are byte-identical (97.26%, 0 regressions), and the RealScheduler tests now exercise the strict evaluator (all green under -race). Next sub-task, surfaced by --differential: connection-topology routing under strict. Today PortKey namespaces a send to the SENDER's component queue, so a `connect(mtc:p, peer:p)` loopback send doesn't reach the connected peer's queue and the receiver blocks (6 alt-2002 fixtures diverge pass->timeout under strict). Routing connected sends to the peer's qualified queue via the connect graph is the next increment.
Second Phase-1 increment. Under ProfileStrict a send on a CONNECTED port
is now delivered to the connected peer(s)' queue(s) via the connect
graph (new ConnectedPeers + PortKeyFor), tagged with the actual sending
component so the receiver's `from` matches -- instead of the sender's
own component-qualified queue. Falls through to self-delivery when the
port has no peer (loopback-to-self / self-connect).
This fixes the connected-component alt fixtures the differential harness
flagged: e.g. `connect(mtc:p, peer:p); peer.start(fsend()); alt { []
p.receive ... }` now delivers the peer's send to the MTC's queue and the
strict alt matches it (was pass->timeout under strict). On alt-2002 the
strict-vs-approximate divergences collapse to a single pre-existing
NegSem miss unrelated to alt (the rest were strict correctly waiting
real 5s timers, cut off only by a short differential --timeout).
The approximate path and the conformance gate are byte-identical
(97.26%). New guards in interpreter/strict_alt_test.go (connected-peer
receive; timer guard actually fires). -race clean.
Add TestcaseOptions.Context: on cancellation RunTestcaseWith stops the executor (exec.Stop), unwinding a blocked strict alt / timer wait via a bounded watcher goroutine (closed on return, so it never outlives the run). The conformance harness's execVerdict now passes its per-testcase timeout context, so a strict run that blocks forever (honest alt semantics on a stuck fixture) is cancelled on timeout instead of leaking a spinning goroutine after we record "timeout". This unblocks a clean full-suite strict differential: the comm-chapter strict sweep now finishes in bounded wall time (~24s / ~48MB RSS). The default approximate path and the conformance gate are byte-identical (97.26%, 0 regressions); Context is nil for RunTestcase. Guard: TestStrictAlt_ContextCancelsBlockedAlt (-race clean).
Fourth Phase-1 increment, so the strict differential measures real divergences instead of real-timer artifacts. Add TestcaseOptions.DeterministicClock (strict-only, orthogonal to Profile): timers advance the per-testcase virtual clock to their deadline and fire instantly instead of sleeping real wall-clock time. timerExpired reads the virtual clock under the flag; the alt block step advances to the SOONEST timer deadline (nextAltTimerVirtualDeadline) so multi-timer ordering holds (guard evaluation no longer advances the clock in this mode). The conformance harness enables it for strict runs; the default gate (approximate) keeps the real clock and is byte-identical (97.26%). A real load driver leaves it off so timers pace real I/O. Effect: the alt-2002 strict differential drops 6->1 divergences at a 2s budget (timer artifacts vanish; the last is a pre-existing NegSem miss), and the 88 communication-chapter divergences are confirmed genuine (unchanged at 2s vs 5s) -- the real proc-comm/send work-list. Guards (interpreter/strict_alt_test.go, -race): a 30s timer fires instantly; the soonest of two timers wins regardless of clause order.
Extend strict connection-topology routing (previously message-send only) to procedure-based communication. call/reply/raise now deliver to the connected peer(s)' queue(s) via the connect graph (shared helpers strictConnectedTargets / enqueueEnvelopeRouted), so a caller's call reaches the server and the server's reply/raise reaches the caller, instead of landing on the sender's own component-qualified queue. On the communication chapter the strict differential drops 88->63 divergences (pass->fail 28->10). The default (approximate) gate is byte-identical (97.26%), full -race suite green. Guard: TestStrictProc_ConnectedCallReply. Remaining comm divergences are separate sub-clusters (skip/deferred- responder timing for single-port servers, port-array any-from routing, blocking-call semantics) tracked for follow-up.
Extend the "finite responder runs at start when a call is already queued" exception to single-port `getcall; reply/raise` servers under the strict profile. The approximate exception (startBodyIsFiniteResponder) only covered indexed-port responders, so a single-port server on a non-alive PTC stayed skipped and its caller's getreply/catch never matched under strict. Comm-chapter strict differential 63->49. Default (approximate) gate is byte-identical (97.26%), full -race suite green. Guard: TestStrictProc_ConnectedCallReply now uses a plain non-alive `create`.
Under the strict profile, procedure receives (getcall/getreply/catch,
including inside check) now match the signature parameter record and
value/exception template via procReceiveMatches, instead of the lenient
"any envelope of this kind" short-circuit. So e.g.
check(getreply(S:{p:=(100..200)} value ?)) no longer matches a reply
whose parameter is out of range (2204 check fixtures 057/058/081/082).
Comm-chapter strict differential 49->45, and the actively-wrong
pass->fail 2204 divergences drop 12->4; verified 0 fixtures regress
(none goes from correct to diverging). The default (approximate) gate
keeps the lenient match and is byte-identical (97.26%); -race green.
Guard: TestStrictProc_CheckHonoursTemplate.
connect(self:p[i], v:p[i]) records the endpoint under the array BASE name
("p") because resolvePortEndpoint/portRefName drops the `[i]`, while comm
(send/call/reply) uses the indexed name "p[i]". Under approximate this is
harmless (shared name-keyed queues); under strict the connect-graph
lookup missed and the call/reply self-delivered, so a caller's
`any from p.getreply` never matched.
Fix strictConnectedTargets (strict-only): when the exact indexed
endpoint has no peer, fall back to the base-name endpoint and re-apply
the element suffix to the peer's port (splitPortIndex). Comm-chapter
strict differential 45->35, 0 regressions (set-diff verified). Default
gate byte-identical (97.26%), -race green. Guard:
TestStrictProc_PortArrayConnectedRouting.
…trict
Under the strict semantics profile, a non-alive PTC started with a body
that blocks on inter-component procedure communication now runs on a real
goroutine instead of being skipped by the synchronous loopback model. This
lets the two-PTC blocking-call shape execute: a server component blocks in
`getcall` while a client component issues a blocking `call{...}`, its reply
routed back over the connection.
- startBodyBlocksOnComm: fork a started non-alive PTC only when its body
blocks on a `call{}` (CallStmt) or a `getcall`, and only when no call is
already queued (the call-before-start case stays on the inline
finite-responder path). Bodies with an `[else]` clause (finite) or a
`@decoded` redirect (codec decoding not yet implemented) are left on the
skip path. Standalone `getreply`/`receive` bodies are not forked on
their own, since their counterpart (a nowait caller or a send-only
sender) is not forked either.
- deterministicClockEnabled: fall back to the real clock while concurrent
PTCs are live. The virtual clock is only sound single-threaded; with
concurrent PTCs, inter-component events already flow in real time
(fast), and a virtual-clock advance in one goroutine would race a safety
timer in another.
- ComponentRef.{done,alive,verdict} are now private and guarded by a
mutex (IsDone/SetDone/IsAlive/SetAlive/GetVerdict, locked MergeVerdict):
a forked PTC updates its completion/verdict on its own goroutine while
the parent observes them via comp.done / comp.alive / comp.running.
Default (approximate) execution is unchanged; the conformance gate stays
at 97.26%. Strict differential across core_language: 73 -> 54 divergences
(19 fixed, 0 regressions). Race detector clean.
A strict alt clause `[expr] op {...}` is now eligible in a snapshot round
only when its boolean guard `expr` holds (ETSI ES 201 873-4 §20.2). The
snapshot gates on a concretely-false boolean; an Undefined/unmodelled or
non-boolean guard falls through to the communication match, preserving the
prior behaviour where the guard was ignored entirely.
This lets a snapshot discriminate clauses that differ only by their guard
across `repeat` rounds. Default (approximate) execution is unchanged; the
conformance gate stays at 97.26%. Strict differential across core_language
gains 2 fixtures with no regressions.
Adds quiesceScheduler, a discrete-event "quiescence barrier" for the strict interpreter's deterministic clock. It models the same rule Go's testing/synctest applies in the runtime: virtual time advances only when every live participant (the MTC and all live PTC goroutines) is parked, and then jumps to the soonest registered timer deadline, waking the goroutine(s) whose timer fires; a communication event or a peer finishing wakes parked participants at the current instant without advancing time. This is the correct basis for making concurrent timer-driven execution fast (no real sleeps), sound (a safety timer can never fire before an earlier event), and free of the current 2ms-polling backstop's load-dependent flakiness — replacing the ad-hoc mix of a real clock, a virtual clock, and a tick counter selected per call site. The type is self-contained (owns its virtual clock, takes explicit goroutine ids) and fully unit-tested under -race: instant single-timer fire, soonest-timer-wins, comm-wakes-without-advance, terminal deadlock detection, stop, done-waiter wakeups, and a producer/consumer stress. It is not yet referenced by the interpreter; wiring the interpreter's block points onto it is a separate, gated step, so this commit changes no execution behaviour and the conformance gate is unaffected.
…direct fixes) Wires the discrete-event quiescence scheduler (runtime/scheduler.go) into the strict interpreter behind a new DeterministicScheduler option: - runtime: TestcaseExec grows a *quiesceScheduler; VirtualClock/ AdvanceVirtualClock delegate to it when active; SchedGoLive/SchedGoDone/ SchedSignal/SchedPark bridge the interpreter to the barrier; FinishPTC deregisters a finished PTC and signalMessageReady broadcasts a wake. - interpreter: every strict block point parks through the scheduler when active — the alt wait (blockForAltEvents), the bare `T.timeout` (both selector and method forms), and the timer-array `any/all timer.timeout` wait. Timer expiry reads the virtual clock via useVirtualClock. The PTC fork registers with the scheduler (SchedGoLive) and the 50 ms start barrier is skipped (the scheduler orders startup). Validated directly by TestSched_TwoPTCBlockingCall: a two-PTC blocking call with a boolean guard, a return value, and a catch(timeout) fail branch runs to a correct pass at virtual time 0 — the 30 s server timer and 5 s call timer never really elapse. The option is left OFF in the conformance harness for now: enabling it routes blocking-call response blocks through the strict alt, which exposes two pre-existing procedure-redirect gaps the approximate path masked — the getcall `-> param` redirect is dropped in the RedirectExpr evaluator, and positional `param(a,-,b)` binding assigns the whole payload rather than per-field. Fixing those is the next step to turning the scheduler on. Default execution is unchanged; the gate stays at 97.26% and -race is clean.
…rict) Three related procedure-based communication fixes under the strict profile, all gate-safe (default execution unchanged, conformance gate stays 97.26%): - getcall redirect routing: the RedirectExpr evaluator handled receive/trigger/getreply/catch/check but not getcall, so a `p.getcall(...) -> param(...)` redirect was silently dropped. Route getcall through evalPortReceiveInfo like the other receive ops. - positional param binding: a `-> param(v1, -, v3)` redirect bound every target to the whole parameter record instead of the per-position field. Carry the signature declaration in TypeDesc and bind positional targets onto the record's fields in formal-parameter order (signatureParamNames + applyParamRedirect); the named `x := field` form and the single-parameter scalar case are unchanged. - multi-client broadcast: a blocking-`call` client PTC was skipped when another component already had a queued call, because the forkStrict gate used the global HasPendingCalls flag. A client produces a call rather than consuming one, so it must always fork; only getcall responders gate on HasPendingCalls. This unhangs `reply ... to all component` fixtures with two or more clients. Together these make the blocking-call two-PTC fixtures genuinely correct (previously the approximate verdict-preferring heuristic fabricated their passes): Sem_220301_CallOperation_001/002/003/007, Sem_220302_GetcallOperation_001/002/005, and the multi-client Sem_220303_ReplyOperation_001/002. Strict differential across core_language shows no regressions; -race clean. Guards: TestStrictProc_GetcallPositionalParamRedirect, _MultiClientBroadcast.
…ive scheduler Replaces the broadcast quiescence scheduler with coopScheduler, a single-runner "token" cooperative scheduler: exactly one component participant (MTC or PTC) executes at a time, and the token is handed to the next participant in a deterministic order (lowest component id first). Virtual time still advances only at quiescence, to the globally-soonest timer deadline — but interleaving is now deterministic too. This removes the residual goroutine-interleaving nondeterminism of the plain quiescence model: with real goroutines running concurrently between park points, two components' snapshots could race (a client's getreply vs. a server's reply), making blocking-call verdicts flaky run-to-run. Under the token model a client blocking call is reproducibly pass (verified: Sem_220301_CallOperation_001 goes from flaky to 8/8 pass under the scheduler). Wiring: the scheduler is keyed on component id (deterministic, unlike a goroutine id); a forked PTC registers via SchedGoLive(id) and takes the token via SchedAcquireToken(id) at goroutine start; SchedPark(id, ...), SchedGoDone(id) and SchedSignal complete the model. Self-contained and unit-tested under -race: instant single-timer, deterministic token handoff, soonest-timer-wins, terminal deadlock, stop, and a mutual-exclusion stress proving the single-runner invariant. Still gated OFF in the conformance harness: enabling it needs a few procedure-comm fixtures resolved first (the call `catch(timeout)` timer ordering, the nowait-call + separate-alt shape, and any-port getcall). Default (approximate) execution is unchanged; the gate stays at 97.26% and -race is clean.
Under the strict semantics profile, interleave reused the best-effort alt
evaluator, which takes only ONE alternative. Add evalInterleaveStmtStrict
implementing the ES 201 873-1 clause 20.4 rule that every alternative is
taken exactly once: each round re-snapshots the not-yet-taken branches,
takes the first whose guard matches, and blocks on the remaining branch
event sources between rounds.
It models the subset it can handle correctly and otherwise defers to the
best-effort evaluator, so nothing outside that subset changes:
- branch bodies that may themselves block need cooperative suspension
(not yet modelled), so those interleaves fall back;
- an interleave with active defaults but no @nodefault needs a
fired-not-verdict-changed default distinction runDefaults cannot make,
so it falls back too.
The remaining subset (@nodefault or no active defaults, with non-blocking
bodies) is modelled correctly, respecting @nodefault and taking every
branch once. The approximate profile is untouched, so the conformance gate
is unchanged (4790/4948, 97.26%); the strict differential gains
Sem_2004_InterleaveStatement_013 with no regressions. Adds unit tests.
An alt whose alternative is an altstep call (`alt { [] a() }`) with a timer
guard living inside the altstep (`a` has `[] t.timeout {}`) hung under the
strict deterministic clock: nextAltTimerVirtualDeadline only scanned direct
`x.timeout` clauses, so it found no deadline, never advanced the virtual
clock, and the altstep's timer could never fire.
Make nextAltTimerVirtualDeadline recurse through an altstep-call guard into
that altstep's own clauses (bounded depth for mutually-recursive altsteps),
resolving the timers in the current env — which reaches component-scope
timers, the common case.
Strict-only (the deterministic clock is strict-only), so the approximate
gate is unchanged (4790/4948, 97.26%). Full-suite strict differential:
strict-worse barriers 53->50 — Sem_1101_ValueVars_001,
Sem_1102_TemplateVars_001, Sem_160201_invoking_altsteps_004 now pass — with
zero regressions. -race clean. Adds a unit test.
A PTC whose body only blocks on a finite timer is not forked under the strict profile; it is skipped and its `.done`/`.running`/`.killed`/`.alive` state is modelled from a duration. componentCompleted measured that window with wall-clock time.Since(StartedAt) — but under the deterministic clock the MTC's observation window (`t.timeout`) advances VIRTUAL time, not wall time, so the wall measure read ~0 and the modelled body was never seen as completed. `any from v_ptc.done` then fell through to `[else]` and failed. Record StartedAtVirtual on the component at start and, when the virtual clock is in effect (useVirtualClock), measure completion as VirtualClock-StartedAtVirtual >= ModeledDuration; wall time still drives it otherwise. The four comp* predicates take env to reach the active clock. Strict-only (approximate keeps the wall-clock path), gate unchanged (4790/4948, 97.26%). Full-suite strict differential: worse barriers 50->42 — done_operation 003/006/008/009/010 and killed_operation 005/011/012 now pass — zero regressions, -race clean. Adds a unit test.
When the MTC's testcase body terminates, every still-running PTC is implicitly stopped (ETSI ES 201 873-1 §21.3). WaitPTCs instead gave each PTC the full join budget to finish on its own before stopping it, so a body that never completes (a long timer nobody stopped, a getcall/reply half that is done being needed) held teardown for the whole budget and pushed the run past the harness timeout — reporting `timeout` even though the testcase verdict was already decided in the MTC body. Signal every remaining PTC to unwind first, then join with the same budget. A PTC that was about to finish still reports Done within the budget; one that would never finish now exits promptly. Approximate has no forked PTCs (no-op there); strict/real-scheduler runs join faster. Gate unchanged (4790/4948, 97.26%). Full-suite strict differential: worse barriers 42->38 (Sem_220302_GetcallOperation_003, Sem_220305_raise_operation_002/003/004), zero regressions, -race clean; RealScheduler / async-PTC tests still green.
… fork) Two correctness fixes to the cooperative scheduler that make it handle component lifecycle correctly — the prerequisite for using it as the strict execution engine. Both are exercised only when the scheduler is active (still gated off in conformance), so no measured behaviour changes yet. 1. Blocking `comp.done` / `comp.killed` park. The standalone forms are blocking (ETSI ES 201 873-1 §21.3.7/21.3.8). Under the single-runner scheduler they must release the token so the target PTC is granted a turn and can actually run to completion; the old non-parking Bool check let the MTC race to the end holding the token, starving the PTC (deadlock). Added blockUntilComponentState: a forked PTC is woken by its goDone; a modelled PTC parks on its virtual completion deadline so the clock advances to it. 2. Stoppable acquireToken. A PTC that was started but never scheduled (the MTC finished without ever parking) blocked forever waiting for a turn, so teardown hung. acquireToken now also selects on the PTC's stop channel and returns stopped=true, and the fork goroutine then exits without running its body. Measured with the scheduler ON (config+comm+alt differential): strict-worse barriers 33->15 (+18 net, 1 known proc-comm regression pending), fully deterministic (identical across runs), gate unchanged (4790/4948, 97.26% — the scheduler is strict-only), -race clean. Adds unit tests (TestCoop_AcquireTokenStop, TestStrictSched_CompDoneParks).
Two new static rejects land the +2:
- permutation() on a set-of target (ETSI B.1.3.3)
- reply with a NotUsed out via a named signature template (ETSI 15.3
restriction e), caught by resolving the Sig:tmplRef form.
Both fixtures now reject at analyze time under both execution profiles, so
they also drop out of the strict-vs-approximate differential.
A non-blocking multicast `p.call(S:{}) to (v1, v3)` was dropped by the
TO-clause dispatch (no `call` case) and fell through to a broadcast, so an
un-addressed server also received the call; and under the scheduler a bare
`getcall; reply` server was forked, its non-blocking getcall fell through on
the empty queue, and it replied unconditionally. Either way an un-addressed
component replied and the caller mis-attributed it to the wrong sender
(Sem_220301_CallOperation_015 "Wrong component").
Two strict-only changes:
- Route `p.call(...) to (targets)` to only the addressed components'
connected peer ports (evalProcedureCallTo / callTargetComponentIDs),
resolving each target through the connect graph; broadcast fallback when
the target set is unresolved. The approximate path keeps its legacy
broadcast.
- Take a BARE `getcall; reply` responder (straight-line, no alt/timer/loop)
off the fork path and register it as a deferred responder even under the
scheduler; RunDeferredResponders then replays a responder ONLY when a
call is actually queued on its own component's ports (selective replay),
so an un-addressed server never replies. An alt-based responder still
forks — its guard must park on its own goroutine, so an inline replay
would deadlock. The approximate path keeps its unconditional replay.
Full-suite strict differential: worse barriers drop to 1 (only
NegSem_1503_..._008, a contradictory-suite non-barrier where strict is
already the gate-correct verdict); CallOperation_015 now passes; 0 new worse;
better set unchanged (13); deterministic across two runs; -race clean; gate
4792/4948 (97.30%).
The strict operational-semantics engine (deterministic discrete-event
scheduler + virtual clock) now scores higher on the ETSI suite than the
legacy approximate engine — 4798 vs 4792 matched (97.42% vs 97.30%) — and its
verdicts are reproducible and free of real-clock races. Make it the default.
- `ntt conformance` defaults to --profile=strict; the gate now measures the
strict engine. --profile=approximate still selects the legacy engine.
Baseline refreshed 4792 -> 4798.
- `ntt exec` defaults to the strict engine (deterministic scheduler + clock,
60s safety timeout); the retiring engine is opt-in via --approximate
(replacing the experimental --deterministic opt-in).
- `--differential` now runs BOTH profiles explicitly and reports each, so
the strict-vs-approximate diagnostic keeps working with strict as default.
The approximate engine remains reachable behind the flag during the
transition; retiring it (and the SemanticsProfile toggle) is a follow-up that
first needs altstep bodies routed through the strict evaluator. Full suite:
gate 4798 exit 0, deterministic across two runs, -race clean, all package
tests green.
Toward a single engine: run DIRECT altstep calls on the strict evaluator
instead of the verdict-preferring best-effort heuristic. A direct altstep
call blocks like `alt { [] a() }` (ETSI 20.5.2), so under the cooperative
scheduler evalAltstepBody now dispatches to evalAltStmtStrict. (Activated
defaults stay on the non-blocking best-effort pass — runDefaults detects a
fired default via the branch flag — and the routing is gated on the coop
scheduler actually being active, so the deterministic-clock-only path is
unaffected.)
That routing needs `any timer` / `all timer` to work inside a `runs on`
altstep. They name the running timers of the CURRENT component (ETSI 23.7) —
a dynamic notion — but resolved lexically via the env chain, which a direct
altstep's scope (chained to its module-level definition) can't follow to a
caller/testcase-body timer. Add an exec/component timer registry: `.start`
registers the handle under the current component, and collectScopeTimers
unions the current component's timers with the lexical set (deduped by
pointer). This mirrors how ports already resolve dynamically via the
TestcaseExec, and is sound under the scheduler (unlike a per-goroutine scheme,
which breaks on goroutine-id reuse across forked participants).
Full suite: gate 4798/4948 (97.42%) — identical match-set to before (0
regressions, 0 gains); strict differential worse=1 (the NegSem_1503_008
non-barrier), better=13, deterministic across two runs; -race clean; full
`go test ./...` green. Unblocks retiring the best-effort heuristic (defaults
next, then the delete).
Continue retiring the verdict-preferring heuristic: an activated altstep default now evaluates on the strict evaluator too, not best-effort. evalAltstepBody dispatches every altstep body to evalAltStmtStrict under the cooperative scheduler (dropping the direct-call-only restriction), and the strict alt loop concludes without parking whenever it is running as a default body (defaultCtx active) — a default is a single non-blocking snapshot pass: it fires only on a real guard match (detected via the branch flag), and a `repeat` still re-snapshots so a repeating default drains every queued message (default_mechanism_006). This generalises the previous `defaultCtx && altHasRealPortGuard` early-out to all default bodies. Under the scheduler the best-effort heuristic is now used only by interleave's unsafe-case fallback; regular alts, direct altstep calls, and defaults all run on strict. Full suite: gate 4798/4948 (97.42%) — identical match-set (0 regressions, 0 gains); strict differential worse=1 (NegSem_1503_008 non-barrier), better=13, deterministic across two runs; -race clean; full go test ./... green.
State the current 4798/4948 (97.42%) baseline and mark the procedure payload matching and async multi-PTC clusters as closed by the strict discrete-event engine. Re-order the remaining work around engine convergence, the reject->pass bulk, and real-execution depth. %INT_NO_SW_CHANGE %AI=CURSOR
Engine convergence, step 1 of the remaining-work order: shrink what the strict path still hands to the legacy best-effort evaluator. An interleave without `@nodefault` used to fall back whenever defaults were active, because leaving the interleave requires knowing that a default FIRED, and runDefaults could only report one that changed the verdict. It now reports a default that actually took a branch, so the snapshot evaluator handles the case directly and an interleave with active defaults keeps its take-each-branch-once semantics instead of degrading to the best-effort "take one alternative" model. Honour `@nodefault` on a plain `alt` too. The strict evaluator ignored it and ran the activated defaults unconditionally; only the interleave path consulted it. The remaining fallback is an interleave branch body that may itself block, which needs cooperative suspension we do not model yet. Conformance unchanged at 4798/4948 (97.42%), 0 per-file regressions, real-execution rate flat at 50.72%. The three new tests each fail on the previous implementation. Adds diff_runs.py, which diffs two conformance reports file by file so a slice can be checked for per-file regressions rather than just a headline delta.
evalAltstepBody chose the strict evaluator only when the discrete-event scheduler was engaged, so a strict run WITHOUT the scheduler executed every altstep body on the legacy verdict-preferring heuristic. The correct axis is the profile: strict owes callers strict altstep semantics whether or not the scheduler happens to be active. ntt exec and the conformance gate always enable the scheduler, which is why the corpus never exposed this; interpreter/strict_alt_test.go runs in exactly that configuration. Conformance unchanged, 0 per-file regressions. Removes one of the two strict-path callers of the approximate engine.
An interleave whose branch body could itself block deferred to the legacy best-effort evaluator, on the stated assumption that such a body needs cooperative suspend/resume at the blocking point (ETSI 20.4). Measured against the full corpus, that fallback changes no verdict: with it disabled all 4948 files produce identical results, including the two executed fixtures that exercise the shape (Sem_2004_InterleaveStatement_ 001/002). A nested alt inside a body parks on its own event sources, finds nothing that can ever fire, and concludes without matching; the interleave then re-snapshots and takes the sibling the first body just enabled, and a later round re-offers the branch whose blocking read is now satisfiable. So delete the fallback and interleaveBodyMayBlock rather than building the branch-suspension machinery. Adds a test mirroring Sem_2004_InterleaveStatement_001, where the two branches are mutually dependent: branch 1's body enables branch 2's guard, branch 2's body supplies what branch 1 is blocked on. evalAltStmtBestEffort now has no caller from the strict path. Conformance unchanged, 0 per-file regressions.
ProfileApproximate was the zero value of SemanticsProfile, so every caller that did not set TestcaseOptions.Profile explicitly - including interpreter.RunTestcase, the documented embedding API, and the interpreterdriver - silently ran the legacy engine. Reorder the enum so ProfileStrict is the zero value: a caller that does not choose now gets the correct engine. interpreterdriver keeps DeterministicClock and DeterministicScheduler off deliberately, since it executes real suites where timers must pace real I/O; that intent is now stated rather than implied. Drops two tests that no longer guard anything: - TestExecApproximateOptOut exercised --approximate, which is about to be deleted. - TestRealScheduler_DefaultOffSkipsWorker claimed to pin the approximate opt-in boundary, but its worker body is skipped identically under approximate, strict, and strict-with-scheduler, so it discriminated nothing. Full suite green under -race, conformance unchanged, 0 per-file regressions.
Deletes --approximate (exec), --profile and --differential (conformance), along with runProfile, staticDriver.deterministic, the execVerdict profile parameter and the Diverged reporting in both the per-file result and the summary. --differential was the only built-in way to diagnose a strict regression against a known-good verdict. Its replacement is diffing two conformance reports file by file with docs/conformance/diff_runs.py, which compares two commits rather than two engines and reports per-file provenance moves that the pass-rate gate cannot see. Conformance unchanged, 0 per-file regressions.
Removes evalAltStmtBestEffort and the helpers only it reached:
nextAltTimerDeadline, waitForAltPortTraffic, altHasExternalPortGuard,
altHasRealPortGuard, isCheckGuard, branchVerdictKind with the altVerdict
type, and branchHasSetverdict (already dead). The three dispatch arms
that selected it - plain alt, the blocking call{} response block, and a
directly-invoked altstep body - now go straight to the snapshot
evaluator.
This is the verdict-preferring heuristic that picked whichever clause
contained a setverdict rather than matching a guard. Nothing in the
strict path had called it since the interleave fallback went.
Comments naming the deleted helpers are corrected in place rather than
left pointing at symbols that no longer exist.
Full suite green under -race, conformance unchanged, 0 per-file
regressions.
With one engine left the profile carried no information, so remove the type, its constants, SetProfile/Profile, the SetRealScheduler/ RealScheduler shims and the TestcaseExec.profile field, along with TestcaseOptions.Profile and TestcaseOptions.RealScheduler. Every gate that read it collapses: schedulerEnabled (which was just "profile == strict" behind a misleading name) is gone and its five PTC fork/skip call sites are now unconditional, PortKey and PortKeyFor always qualify a non-MTC component's ports, RunDeferredResponders always checks whether a responder was actually addressed, and strictConnectedTargets no longer short-circuits. deterministicClockEnabled and useVirtualClock survive, minus their profile term: DeterministicClock and DeterministicScheduler remain real options, since a driver running real suites wants timers to pace real I/O. That is now the only axis left. Comments describing the two-engine world are corrected. Several said "the approximate path" where they meant "without the discrete-event scheduler", which is a distinction that still exists. Full suite green under -race, conformance unchanged, 0 per-file regressions.
Rewrites the strict-engine section for a single evaluator, records what
replaced --differential, and marks engine convergence done in the
suggested order.
Two measured corrections to claims this document was making:
The real-execution rate is capped at 55.33%, not open-ended. A file
expecting `reject` can never land in the `executed` bucket, because a
completed run yields a TTCN-3 verdict and `reject` is not one; 2200 of
4925 considered files are in that category. Real headroom is 227 files
(+4.61 points), of which 143 have nothing to execute at all and 40 carry
an ETSI noexecution directive, leaving roughly 84 (+1.71).
The 23 "skipped" files are not inconclusive tests. They are negative
tests the analyzer fails to catch, demoted out of the denominator rather
than counted as misses.
Re-ranks the remaining levers on measured files-per-effort: executing
control {} dominates, codec wiring is really a match-rate lever and moves
to that bucket, and cross-module resolution via ttcn3/types is called out
as not worth doing - TypeOf handles literals and operators only, so it
means writing a symbol table from scratch for a measured return of one
file.
The concurrent-PTC behaviour it describes is now unconditional, and the "default model skips this worker" note pointed at a test deleted with the opt-in boundary it guarded.
Adds RunControlWith: a module's `control` part runs as a statement
sequence, `execute(TC(args))` runs a testcase and yields its verdict, and
the module's verdict is the worst over the testcases the control part
actually ran. execute() also honours its timeout and host operands, both
of which make a testcase yield `error` without its body reaching a
verdict.
The harness routes a module through its control part only when that part
decides something running the first testcase alone cannot reproduce:
several executes selected and ordered by control flow, a verdict threaded
from one into the next as an actual parameter, or an execute carrying a
timeout or host. A plain `control { execute(TheOnlyTestcase()); }` keeps
the direct path, which is equivalent and far better exercised. Measured:
routing every control-part module through it instead costs more than it
gains, mostly on shapes this does not yet model.
Three fixes that fell out, each a real defect the control path exposed:
- A loop no longer ignores its context. `while(true){}` has no statements
at which the per-statement stop check fires, so it spun forever and
leaked the goroutine; the conformance harness only survived it by
abandoning the run and reporting a timeout.
- "Undeclared verdict resolves to pass" (22.4.1) now tests whether
setverdict was called, not whether the verdict is none. An explicit
setverdict(none) has to survive, because a control part can branch on
it.
- execute() outside the control part is a dynamic error rather than a
silent no-op, which is what 16.3 and 26.2 require.
Conformance 4798 -> 4803 (97.42% -> 97.52%), real execution 50.72% ->
50.82%, 0 per-file regressions.
Four related gaps in how values and templates are read back, each found
by a conformance file that our engine ran and got wrong.
Assignment notation keeps unmentioned fields. `v := {field1 := 3,
field3 := 2.0}` over an initialised record leaves field2 alone rather
than dropping it (6.2). A union is excluded: it carries exactly one
alternative, so re-assigning it selects a new one instead of merging.
Referencing a field of a template assigned AnyValue yields that wildcard
again rather than an uninitialised value (15.6.5 restriction b), so
`m.b.u1` on `m.b := ?` is `?`. An OPTIONAL field admits absence and so
yields `*` instead. The rule sits below the Annex E branches, because
`mw_msg.encode` still has to resolve the attribute of the wildcard
template's declared type.
ischosen answers about the alternative the union carries, not about the
value the reference yields, so it now inspects the receiver: `{f2 := ?}`
has f2 chosen even though its value is a wildcard, while a union
template that is itself `?` has chosen nothing at all (16.1.2). The old
"is the value bound" reading could not tell those apart.
Concatenating a fixed-length wildcard onto a binary string contributes
that many unknown units, so `'ABCD'O & ? length(2)` spans four octets
rather than three (15.11). A `?` inside a binary string stands for one
unit of that string's type - a whole octet for an octetstring - so one
per unit is right.
Conformance 4803 -> 4806 (97.52% -> 97.58%), 0 per-file regressions.
ETSI 8.2.3.1: in the context of an enumerated type, an imported enumerated value wins over a same-named definition in the importing module, which then has to be referenced by its qualified name. So `c_enumVal == enumX` compares against the enum member even when the testcase declares `var integer enumX := 1` (Sem_08020301_GeneralFormatOfImport_004). Applied to the comparison operators, where the enum-typed operand establishes the context and only a bare identifier can clash. A name the enumerated type does not declare keeps whatever it evaluated to, so ordinary variables are untouched. Conformance 4806 -> 4807 (97.58% -> 97.60%), 0 per-file regressions.
Sem_2303_timer_stop_004 asks for `none` from a body that never calls setverdict, and 22.4.1 agrees: a component's verdict starts at none and only setverdict moves it. Our engine coerces an undeclared verdict to pass instead, so the file misses. Removing the coercion costs 64 files, not the 2 a source-level scan suggests. Only 4 files in the suite contain no setverdict at all; the other 60 do call it and never reach it, because the branch that would is never taken. Those are real defects the coercion has been hiding, and they have to be fixed before the coercion can go, not after. One of them is next door: a standalone `all component.done` answers a snapshot the statement context throws away rather than blocking as 21.3.7 requires, so the MTC races past it and the testcase ends while its PTCs are still parked. Making it park makes four of our own strict scheduler tests genuine - they were asserting a pass the coercion manufactured - but costs two more files, because the PTC bodies then reach branches that expose a `send ... to <component>` routing defect. No behaviour change here, just the two comments that say so.
Neither operator existed: `v of MySubClass` fell through to the generic binary path and died with "type mismatch: object reference of class descriptor", and `v => MySubClass` was read as the decoded field reference that shares `=>`, so it looked for a cached encoding and answered Undefined. `of` now answers instanceIsA - the object's runtime class or any class it derives from - which is the same subsumption test `select class` already used. A null reference is of no class at all, so it answers false. The cast re-types a reference the object satisfies and leaves the object alone, references being handles; a cast the object does not satisfy, or one applied to null, is a dynamic error so the testcase reports the bad cast instead of failing some later assertion for no visible reason. Conformance is a wash: Sem_5010206_Casting_001 now passes, and Sem_5010205_OfOperator_001 now misses. The latter is a mislabelled fixture - `Sem_` with `@verdict pass reject`, a distinction only it and Sem_5010103_externalClasses_001 share among the 50 reject-annotated oo files, against a body that is plainly legal TTCN-3 and a @purpose ("Ensure that of operator gives the most specific class instance") that reads as positive. It was only satisfied because `of` errored, so the runtime-error relaxation counted it as a caught rejection.
….5.1) runDefaults() discarded the result of the default body it evaluated, so a `stop` in an activated default let the statements after the alt statement run - Sem_200501_the_default_mechanism_008 reached its `setverdict(fail, "Component stop expected")`. It now hands the body's control-flow result back to the alt evaluator, which propagates it. `stop` itself was only unwinding the current behaviour. It now also ends the component, and the testcase when that component is the MTC, while an `alive` component stays reusable as it must. Absorbing `break` at the altstep boundary is the other half: it terminates the altstep and its alt statement (not a loop), so it must not reach unwrap() and become "break outside loops". Propagating errors out of defaults exposed that on 200501_007. Sem_2601_ExecuteStatement_003 is NOT fixed. Its blocking alt should park until execute()'s timeout produces `error`, and leaving the scheduler's deadlocked participants parked does exactly that - at the cost of 51 files whose alts cannot fire for reasons of our own and which rely on that release to reach a verdict. Recorded in the scheduler and deferred.
An `external function` has no TTCN-3 body: a deployment's SUT adapter
supplies one. We bound the name to Undefined instead, so every fixture
that asserts something about the result failed - three of them in the
suite, each documenting its own contract in a doc comment ("@return
always 1").
The engine now resolves such a call against a binding registry, and the
conformance harness plays the adapter for exactly those three fixtures.
Bindings are module-qualified so one answers only for the fixture that
documented it.
An unbound external function still yields Undefined rather than raising.
472 fixtures declare one, nearly all negative tests that never consume
the result; erroring at the call would fail them for a reason unrelated
to what they test.
Conformance: 4808 -> 4811 matched, no regressions.
…x C.5)
encvalue_o returned a single zero octet for everything except an
integer, so the two fixtures that inspect the result byte for byte
failed - and 107 could not even index the blob it got back.
A bitstring is now left-aligned into whole octets with the low bits of
the last octet zeroed, behind a 32-bit little-endian bit count:
encvalue_o('011'B) is '0300000060'O. A record encodes its fields in
order, each in the shape this engine already used for that type, so
{"testText", 5} is '74657374546578740005000000'O - the value 107
documents for it. Other operands keep the round-trip placeholder rather
than get an invented encoding.
decvalue_o now reports 2, "could not be completed, not enough octets",
when the input is shorter than the output slot's declared field width,
matching what the bitstring path already did. That width can be declared
on the variable (`var integer v with { variant "32 bit" }`), which no
TypeDesc could carry, so a declaration's own variant attributes are
recorded alongside its type.
Conformance: 4811 -> 4813 matched, no regressions.
4803 -> 4813 matched (97.52% -> 97.73%). Eleven of the thirteen files that executed and produced a wrong verdict are fixed; the two that are not both stopped at the same wall, and that is the useful result. Removing the undeclared-verdict coercion costs 64 files, 60 of which call setverdict on a branch that never executes. Leaving a deadlocked alt parked - which is what the standard asks for - costs 51 files whose alts cannot fire for reasons of our own. So roughly 115 files are being held up by two pieces of scaffolding, which is a measurement of how much of the alt and event-delivery model is still missing, and it sets the order the two have to be unpicked in. Also retires the 1d "no safe win" entries: three of them were framing rather than feature problems.
refresh_artifacts.py refused to lower the baseline. That put a ratchet in a bookkeeping script, which is the wrong place for one: removing engine behaviour that existed only to make fixtures pass lowers the rate on purpose, and the script's job is to record what was measured. The ratchet that actually prevents an accidental regression is --regress on the conformance command, enforced per commit by CI. A drop has to be justified there.
When every component is parked and no timer can advance the clock, the scheduler released everyone so their blocked alts concluded as though nothing had matched. That existed to keep fixtures passing, and 51 of them were resting on it - several vacuously. Sem_2204_the_check_operation_025 sets pass after its alt either way, so whether check(getcall) works was never actually tested. Quiescence is a provable deadlock in the loopback model: there is no outside, so nothing can ever arrive. It is now reported as an `error` verdict naming the deadlock. With an external port driver installed the inference does not hold, since a real peer may still send, so the old release stands there. `error` outranks an earlier `pass` in the aggregation, deliberately: a verdict reached before the test system broke is not worth reporting. Deliberate drop of 0.89 points, and the baseline moves with it. 51 files lose a pass they had not earned; 7 are gained, six being NegSem fixtures that expect a rejection and now get one, plus Sem_2601_ExecuteStatement_003 which asked for this `error` all along. TestStrictInterleave_BlockingBodyRunsOnSnapshotEvaluator asserted pass and was vacuous for the same reason - the coercion supplied it. Rewritten as TestStrictInterleave_MutuallyDependentBranchesDeadlock, asserting the deadlock and recording what Sem_2004_InterleaveStatement_001 really needs: suspending a blocked interleave branch to run another ready one, which we do not implement.
ETSI 22.4.1 says the verdict starts at none and setverdict is what moves
it. The engine coerced an undeclared verdict to pass, which made a
testcase that does nothing look successful - and fabricated a verdict for
bodies whose setverdict is never reached, which is most of what it was
actually doing.
28 files lose that fabricated pass. 26 of them explicitly assert
ttcn3verdict:pass, so they are real defects now failing honestly. Six Go
tests were vacuous for the same reason and are rewritten to assert what
the engine really does, each naming what must be restored when the gap
closes:
- four procedure-communication tests (TestStrictSched_*) in which the
PTC bodies never run at all, so neither the server's getcall nor its
safety timer nor the client's call happened. Nothing they describe was
ever exercised.
- TestAsyncPTC_InjectWakesAltAndBodyRuns, which hid a user-facing bug in
the path a C/C++ test port uses: an injected message sits in the queue
under exactly the key the daemon PTC reads, unconsumed, while its alt
re-polls for 500ms. Not a lost wake-up and not a template mismatch -
diagnosis recorded on the test.
- the two that encoded the old rule directly.
The harness also over-read the annotation. A bare `@verdict pass accept`
with no ttcn3verdict: tag and no setverdict in the source asserts only
that the run is acceptable, so `none` satisfies it; expecting `pass` was
this harness inventing a requirement. Constrained to positive headers:
applying it to `reject` fixtures too would have handed a match to 21
negative files we genuinely fail, which measurement caught.
Baseline moves with the commit. Predicted 4747, measured 4744: seven files
outside the original 64 also depended on the coercion, two of the
predicted ones no longer do.
ETSI 21.3.3 says `stop` on a component created with `alive` only suspends
it and it stays reusable. Ours does not run the new behaviour, which is
six of the files phase 2 stopped fabricating a verdict for
(Sem_210303_Stop_test_component_005..010).
Two layers, both diagnosed, neither fixed:
1. `start` does not clear the component's `done` flag, so the following
`comp.done` is satisfied by the previous run and the MTC leaves the
testcase before the new body is scheduled.
2. With that cleared, the re-forked PTC goroutine still never gets
scheduled and the run deadlocks instead.
Also tried, and reverted with them: dropping `timeout` from the
skip-the-body predicate so a timer-only PTC body runs on the scheduler,
which the virtual clock now makes possible. On its own that stalls the
parent (start is non-blocking), and with a fork added it still recovered
nothing while costing 7 files in 21_configuration. -7 / +0, so it is out.
Kept as a tripwire test carrying the diagnosis, so the next attempt starts
from what was already learned rather than rediscovering it.
A PTC body containing any procedure operation was not executed at all: a syntactic predicate declared it unrunnable and a model stood in for it. That predicate predates the discrete-event scheduler, which can now run such a body on its own goroutine and park it properly. Dropping the procedure operations from it recovers three getcall fixtures honestly, with no regressions. One shape stays out, and narrowly: a BARE finite responder - straight-line `getcall; reply`, no alt, no loop, no blocking call of its own - started before any call is queued. Its getcall is non-blocking, so running it would fall through on the empty queue and reply to nobody, which a multicast `call to (...)` fixture then mis-attributes to the wrong sender. Measured: without that carve-out this change costs Sem_220301_CallOperation_015 for a net +3 instead of +3 clean. It is still replayed on demand for the call actually addressed to it. This is 3 of the 48 procedure-communication files that phase 2 stopped fabricating verdicts for. Running the bodies was necessary but is not sufficient: the four TestStrictSched_* tripwires still report no verdict, so a client/server pair still fails to complete a call/reply round trip. That is the next layer, and it is matching and routing rather than scheduling.
Adds docs/engine-trust.md, for an engineer deciding whether to rely on the engine. It leads with the correction rather than the rate: two mechanisms existed so that fixtures would pass, 74 files rested on them, some of those files tested nothing at all, and six of our own Go tests had been captured by the same mechanism - which is the part that should worry a reader, since those tests were the reason to believe the engine worked. Also states plainly where the engine is dependable and where it is not: multi-component procedure-based communication does not complete a call/reply round trip, and an injected message does not reach a PTC with an external driver bound, which is the C/C++ test-port path. remaining-work.md is corrected throughout. The masked set was 74 files, not the ~115 it claimed, and 47 of them are one chapter-22 cluster rather than a diffuse wall. Every figure in both pages is checked against the run that produced them: 4747 / 4925 matched, 96.39%, 178 real misses, 102 of which expect a rejection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch grows
nttfrom a TTCN-3 front-end (parser / LSP / tooling)into a full TTCN-3 test-execution toolchain in Go: a tree-walking
interpreter and an
ntt execruntime that can compile, execute andverdict real TTCN-3 test suites, validated against the ETSI TTCN-3
conformance suite.
ETSI conformance: 4798 / 4948 matched (97.42%), 23 skipped, tracked as
a CI regression gate.
What's included
interpreter/) — tree-walking evaluation of modules,testcases, functions/altsteps, templates,
alt/interleave, ports(message and procedure-based communication), timers, components, and
TTCN-3 object orientation (classes, inheritance, nested classes).
runtime/,runtime/exec/) — testcaseexecutor, verdict handling, port/queue model, config (
--cfg,[MODULE_PARAMETERS]), reporting, and a C ABI / cgo bridge so C/C++test ports can be driven from the Go runtime.
runtime/scheduler.go) — a cooperativediscrete-event scheduler with a virtual clock: exactly one component
runs at a time, the token is handed over deterministically in component
order, and virtual time advances only at quiescence. Concurrent PTCs
fork and genuinely interleave,
altblocks on real matches, and timersfire without sleeping in real time, so runs are reproducible across
machines and the whole corpus gates in seconds. This is the only
engine: the earlier heuristic evaluator, the
SemanticsProfiletoggleand the
--approximate/--profileflags have been removed.runtime/codec/) — JSON and XML/XER encode/decode pathsplus RAW
encvalue/decvalueround-tripping.ttcn3/semantic/) — additional static checks(attributes, parametrization, restrictions, type rules, …) surfaced
through
ntt check.conformance.go) — runs the ETSI suite,classifies each file's outcome against its
@verdictannotation, andgates regressions against a committed baseline
(
testdata/conformance-baseline.json).It also reports a real-execution rate — files whose verdict came
from actually executing the testcase rather than from a parse or
semantic rejection — currently 2498 files (50.72%). That figure is
structurally capped at 55.33%, because a file annotated
@verdict pass rejectexpects a rejection and can never be counted asexecuted-and-matching; 2200 of the 4925 considered files are in that
category.
Testing
--regress 0.5against the baseline,with a per-file diff requiring zero regressions for every change.
docs/conformance/diff_runs.pycompares two
--jsonreports file by file, so a slice is checked forper-file regressions and provenance moves rather than just a headline
delta.
(17 pass / 0 fail / 2 expected inconc) across changes.
go test ./...andgo test -race ./....Status & remaining work
The remaining 127 misses are documented with a per-cluster triage in
docs/conformance/remaining-work.md.reject->passaccounts for 107 of the 150 unmatched files, spread across23 clusters, so most of what is left is a question of adding static
rejections — each needing a precise, narrowly scoped analysis pass rather
than a blanket rule. The rest need dedicated deep features
(union-alternative tracking, a real RAW codec, a host binding for
external functions, program-level
control {}execution) or arecontradictory / mislabeled suite fixtures left intentionally as-is. The
miss inventory and per-slice history live alongside it under
docs/conformance/.Engine convergence is complete, so the branch now presents one semantics
rather than two. Measured against the full corpus, every convergence step
held 4798 / 4948 with zero per-file changes — including deleting the
interleave fallback, which turned out to change no verdict and so needed
none of the branch-suspension machinery it appeared to require.