Features for working with synthetitc data in sandbox

[Rens88]

The combined commits were necessary for me to get the code working with my ams synthetic data use case. It involves deletion, uploading and adding a new day of synthetic data.
Have a critical look at what part belongs in the ams-api repo and which parts may be better stored in the synthetic-data repo.

Written by Codex:

Summary

• implement the stateful daily synthetic-data pipeline with deterministic date transitions and missing-date catch-up
• preserve immutable daily commits, hash-linked checkpoints, stable identifiers, pending events, and longitudinal athlete state
• support the same local and scheduled preparation workflow for WSV, KNLTB, or both datasets
• add read-only remote scope classification and immutable daily upload-plan generation
• keep scheduled Databricks execution prepare-only, with no unattended AMS mutation
• add interactive event-only publication with exact sandbox, scope, plan, hash, and reconciliation checks
• support safe effective-dated athlete roster changes without rewriting historical data
• document setup, storage, scheduling, recovery, publication, and operator responsibilities

Safety

Daily generation and scheduled execution remain read-only with respect to AMS. Live publication is a separate interactive sandbox-only workflow that requires an exact immutable plan, a fresh TTY-based typed confirmation, bounded event inserts, single-use authorization, and immediate read-only reconciliation.

Unexpected, incomplete, ambiguous, partial, or unknown states fail closed. No profile upserts, updates, overwrites, deletions, automatic mutation retries, unattended confirmations, or production operations are introduced.

Validation

• 47 daily-focused synthesis tests passed
• the complete connector suite passed 147 tests with one opt-in live test skipped
• synthesis and connector wheels built successfully
• external wheel-path import smoke test passed
• CLI help, form-map JSON, Databricks YAML safeguards, compilation, task completion, and scoped git diff --check passed
• full sibling synthesis suite: 186 passed and two pre-existing fixture-dependent refresh tests errored
• no live AMS mutation, Databricks deployment, bundle validation, or schedule activation was performed

Governance

Applicable aims: B2, B4, B6–B8, C1–C3, C5–C8, E1, E4, E6–E9, F3, F5–F8.

Highest applicable risk: High.

[Rens88]

In the other branch we discussed whether a more detailed confirmation with a summary was desirable. I had already implemented it in this to-be-merged branch. I propose to leave it in, experience it and later decided to either keep or drop it.

Feel free to close this comment and -if the rest looks good- proceed to merge.

[TeamNL1]

This part seems in contradiction with the --auto mode. --auto mode was designed to accept certain risks when approved by a human. If we do not explicitly mention the exception for --auto, this might cause some issues in the actual usage of the API, I think. What is your thought on this @Rens88

[TeamNL1]

I believe we should avoid the mentioning of specific tools such as databricks and the way databricks works. Tomorrow it could be Fabric, Snowflake etc. I think we should keep the API as clean as possible. I feel that maybe we are complicating things too much. Do we need sandbox, databricks job parameter checks, if we already built in the different modes. Maybe we should discuss the design choises to make sure our ideas are fully aligned? @nielsd01 @Rens88

[TeamNL1]

I left a two notes in the reviews of the adjusted files. I also scheduled a meeting about the direction and designchoises of the API to make sure we al have the same idea about how the product should work. I believe it would be good to discuss our development strategy for the next phases.

PS: I was able to extract data with the API from the AMS sandbox and load it directly into the Azure Fabric platform, which shows the basics of the API are working!🥳