[Feature]: Add a public API for custom extension operations

[zhongkechen]

What would you like?

Split durable operations into two API categories:

1. Primitive operations are directly supported by the durable execution backend and owned by the SDK, such as step, chained invoke, wait, callback, and child context operations.
2. Extension operations are higher-level operations implemented by composing primitive operations, such as waitForCondition, map, parallel, withRetry, waitForCallback, and dag.

This feature request is only about making the second category extensible. Developers should be able to create reusable custom extension operations in a separate Maven module without modifying or rebuilding the SDK. It should not allow developers to add new backend operation types or implement primitive checkpoint protocols.

Extension libraries can expose operations as public static methods. Applications can statically import those methods and call them with ergonomics similar to native SDK operations.

A DAG operation is a concrete example. Under this extension model, it could be exposed from a separate DAG module as:

import static software.amazon.lambda.durable.dag.DagOperations.dag;

DagResult result = dag("etl", d -> {
    var a = d.step("a", String.class, (deps, step) -> "A");
    var b = d.step("b", String.class, (deps, step) -> deps.get(a).orElseThrow() + "B")
            .reads(a);
    var c = d.step("c", String.class, (deps, step) -> deps.get(a).orElseThrow() + "C")
            .reads(a);
    d.step(
                    "join",
                    String.class,
                    (deps, step) -> deps.get(b).orElseThrow() + deps.get(c).orElseThrow())
            .reads(b, c);
});

DagOperations.dag would retrieve the active context from SDK-managed thread-local storage through DurableContext.getCurrentContext(). This avoids adding dag and dagAsync to the core DurableContext interface and does not require extension registration or a typed context adapter.

Existing SDK operations can also be implemented or exposed as static methods backed by the same current-context mechanism. This allows primitive operations, built-in extension operations, and third-party extension operations to use one consistent style:

import static software.amazon.lambda.durable.DurableOperations.map;
import static software.amazon.lambda.durable.DurableOperations.step;

var validated = step("validate", Input.class, stepContext -> validate(input));
var results = map("process", items, ItemResult.class, (item, index, context) -> process(item));

The existing instance methods on DurableContext can remain for compatibility. Static SDK methods can delegate to the active DurableContext, just as external extension methods do.

The SDK should provide:

• Stable public interfaces for composing extension operations from primitive operations.
• A supported extension scope with deterministic operation IDs for the primitives used internally.
• Reliable access to the active DurableContext from operation methods running on SDK-managed context threads.
• Synchronous and asynchronous extension forms consistent with built-in operations.
• A uniform static-import style that can be used by existing SDK operations and third-party extensions.
• Compatibility guarantees so extension implementations depend only on public SDK contracts.

The SDK would continue to own operation IDs, checkpoint/replay behavior, suspension, serialization, and communication with the backend for each primitive.

Possible Implementation

The DAG extension could provide static entry points:

public final class DagOperations {
    private DagOperations() {}

    public static DagResult dag(String name, Consumer<DagContext> register) {
        return dagAsync(name, register, DagConfig.builder().build()).get();
    }

    public static DurableFuture<DagResult> dagAsync(
            String name,
            Consumer<DagContext> register,
            DagConfig config) {
        var context = DurableContext.getCurrentContext();

        return DurableExtensions.runAsync(
                context,
                name,
                DagResult.class,
                extensionContext -> executeDag(extensionContext, register, config));
    }
}

DurableExtensions.runAsync(...) is illustrative. The RFC should decide whether the SDK needs a dedicated extension runner or can stabilize an existing scoping primitive.

DAG demonstrates important requirements for that API. A DAG can run as a child-context container, delegate each task to existing durable operation machinery, and give tasks name-derived IDs so arbitrary graph scheduling remains replay-safe. The extension contract should support those requirements without exposing DurableContextImpl, adding operation-specific methods to DurableContext, or requiring an extension to send raw checkpoint updates.

An extension should receive an ExtensionContext exposing a stable set of primitive durable operations and extension-scope metadata. It should not expose OperationUpdate, backend polling, raw checkpoint state, internal execution managers, or the ability to define new backend operation types/subtypes.

Existing SDK operation methods could use the same structure internally: retrieve the current DurableContext, then delegate to the existing context-based implementation. This provides a common authoring and calling model without removing the current API.

The RFC should address:

• Which operations are primitives and therefore part of the stable extension contract.
• Which existing SDK operations should expose static facades and how those methods are organized and named.
• The contract and failure behavior of DurableContext.getCurrentContext() outside an active durable context or from a step thread.
• Thread-local context propagation across SDK-managed handler, child-context, executor, and virtual threads.
• Deterministic ID allocation and namespacing for primitives inside an extension.
• Stable named operation identities within an extension, as required by DAG tasks whose execution order may vary during replay.
• Sync and async execution and interoperability with DurableFuture combinators.
• Nested extensions and whether recursion or composition requires limits.
• Replay compatibility when an extension implementation changes between deployments.
• Serialization, exception, cancellation, completion, and plugin-hook behavior at the extension boundary.
• Whether existing SDK extension operations should be implemented on the same public contract.

Acceptance criteria:

• A separate Maven module can define and publish custom extension operations as static methods using only supported public APIs.
• Applications can statically import and invoke an extension operation without modifying DurableContext, registering a provider, or qualifying a utility class at every call site.
• Existing SDK primitive and extension operations can expose the same static-import calling style through the current-context mechanism.
• Existing DurableContext instance methods remain compatible.
• An operation method can reliably retrieve the current DurableContext while running on a supported SDK context thread.
• Calls outside a supported durable context fail with a clear, documented exception.
• Custom extensions can compose primitive operations but cannot define primitive/backend operation types or send raw checkpoint updates.
• Extension code does not subclass or reference BaseDurableOperation, DurableContextImpl, ExecutionManager, or other implementation details.
• Primitive operations inside an extension have deterministic, isolated IDs and replay correctly.
• Custom extensions work across initial execution, replay, and suspension/resume by relying on primitive operation semantics.
• Both synchronous and asynchronous extension usage are supported.
• DurableFuture combinators work with asynchronous extension results without internal downcasts.
• Nested extension behavior and plugin lifecycle behavior are documented and tested.
• Public extension interfaces have documented compatibility and versioning guarantees.
• A DAG extension can be implemented in a separate module and imported statically without adding dag/dagAsync to DurableContext or modifying DurableContextImpl.
• DAG task identities remain deterministic and replay-safe through supported public extension APIs rather than internal explicit-ID methods.

Is this a breaking change?

No

Does this require an RFC?

Yes

Additional Context

This deliberately does not make the backend primitive-operation protocol extensible. The SDK remains responsible for backend-supported operation types and their checkpoint/replay implementation.

DAG is the motivating example. A core implementation would otherwise need DAG-specific entry points on DurableContext, implementation code in DurableContextImpl, and internal explicit-ID variants for primitive operations. This feature should stabilize the general capabilities needed by DAG so similarly complex extension operations can live outside the core SDK.

This is also related to #493, which tracks hard-coded operation types, and #492, which discussed optional features in extra packages.