test: add reusable Fabric Spark end-to-end validation - #2669
Open
Rana Singh (ranadeepsingh) wants to merge 42 commits into
Open
test: add reusable Fabric Spark end-to-end validation#2669Rana Singh (ranadeepsingh) wants to merge 42 commits into
Rana Singh (ranadeepsingh) wants to merge 42 commits into
Conversation
Rana Singh (ranadeepsingh)
requested a review
from Brendan Walsh (BrendanWalsh)
as a code owner
August 19, 2026 11:32
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
5 tasks
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Hey Rana Singh (@ranadeepsingh) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
August 19, 2026 11:35
View session
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a manifest-driven Fabric E2E runner under tools/fabric_e2e/ and wires it into pipeline.yaml to run a managed Fabric platform-notebook scenario that exercises OpenAIPrompt using Fabric’s implicit LLM authentication. This closes a real-runtime validation gap by capturing machine-readable evidence (JAR provenance, runtime metadata, diagnostics, and JUnit) from an actual Fabric Spark execution.
Changes:
- Introduce a reusable
tools/fabric_e2e/run.pyrunner that loads scenarios fromtools/fabric_e2e/scenarios.json, submits either batch jobs or platform notebooks, captures structured markers, writes JUnit, and retains evidence. - Add checked-in Fabric scenarios for runtime smoke, jar provenance, LightGBM streaming (native diagnostics + repeated fits), and an
OpenAIPrompt“AI Functions origin behaviors” notebook scenario. - Extend
pipeline.yamlFabricE2E job to skip fork PRs, run the OpenAIPrompt notebook gate onmaster/spark3.5, and publish both JUnit results and retained evidence; add unit tests to pin the pipeline and runner behavior.
Show a summary per file
| File | Description |
|---|---|
| tools/fabric_e2e/scenarios/runtime_smoke.py | Minimal Spark action + structured result marker for basic managed runtime proof. |
| tools/fabric_e2e/scenarios/jar_provenance.py | Validates a provided jar is actually used via class-source provenance, plus a Spark action. |
| tools/fabric_e2e/scenarios/lightgbm_streaming.py | Repeated LightGBM streaming fits with native-load diagnostics and provenance assertions. |
| tools/fabric_e2e/scenarios/openai_prompt_ai_functions.py | Platform-notebook scenario exercising OpenAIPrompt using implicit Fabric auth and structured assertions/diagnostics. |
| tools/fabric_e2e/scenarios.json | Manifest describing scenarios, default args, execution mode, and Spark conf defaults. |
| tools/fabric_e2e/run.py | Core runner: scenario loading/validation, command construction, execution + log retention, marker parsing, cleanup, evidence + JUnit output. |
| tools/ci/tests/test_fabric_e2e.py | Unit tests for manifest validation, command construction, notebook generation, marker parsing, and run-id safety. |
| tools/ci/tests/test_pipeline_yaml.py | Verifies the pipeline includes the new OpenAIPrompt Fabric step and publishes results/evidence as intended. |
| pipeline.yaml | Adds the OpenAIPrompt Fabric gate, increases FabricE2E timeouts, skips untrusted fork PRs, and publishes JUnit + evidence artifacts. |
| .github/skills/fabric-e2e/SKILL.md | New repo skill documenting how to run Fabric E2E scenarios and interpret evidence. |
| .github/skills/fabric-e2e/references/scenarios.md | Scenario-by-scenario reference documentation and expected outputs. |
Review details
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 11/11 changed files
- Comments generated: 0
- Review effort level: Lite
Rana Singh (ranadeepsingh)
pushed a commit
that referenced
this pull request
Aug 19, 2026
## Summary Make the independent OpenAIPrompt Fabric task run after either success or failure of the preceding legacy Fabric E2E task, while still excluding canceled runs and preserving the existing branch and test gates. ## Prompting Intent The engineer asked for a pipeline path that actually runs OpenAIPrompt on Fabric. The first trusted PR build showed that a pre-existing certificate authentication failure in the legacy Fabric cleanup task prevented the new, service-connection-based scenario from starting. ## Linked Sources - Pull request: #2669 - Trusted pipeline run: https://msdata.visualstudio.com/A365/_build/results?buildId=231832947 ## Rationale The OpenAIPrompt scenario has independent authentication, execution, evidence, and cleanup. Using succeededOrFailed lets it report its own result when an unrelated earlier Fabric test fails, rather than coupling the new confidence gate to legacy certificate health. Canceled jobs remain excluded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
August 19, 2026 11:56
View session
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
tools/fabric_e2e/run.py:352
parse_runtime_evidenceaccepts duplicated result markers when the JSON payload is identical (becausemarker_payloadsde-duplicates payload strings). The docstring currently claims it parses a single marker, which is misleading and makes the behavior look accidental.
def parse_runtime_evidence(output: str) -> Mapping[str, object]:
"""Parse the scenario's single structured result marker."""
matches = marker_payloads(output, RESULT_MARKER)
- Files reviewed: 11/11 changed files
- Comments generated: 0 new
- Review effort level: Lite
Rana Singh (ranadeepsingh)
pushed a commit
that referenced
this pull request
Aug 19, 2026
## Summary Use PipAuthenticate for the project-scoped SynapseMaven PyPI feed before installing the pinned fabric-spark-cli package, and remove the unsupported ARM service-principal token exchange from the script. ## Prompting Intent The engineer asked for a pipeline that actually executes OpenAIPrompt on Fabric. Trusted pipeline execution reached the CLI installation step but the Azure service principal's ARM access token received a 401 from Azure Artifacts. ## Linked Sources - Pull request: #2669 - Failed trusted run: https://msdata.visualstudio.com/A365/_build/results?buildId=231835811 - Azure Artifacts feed: A365/SynapseMaven ## Rationale PipAuthenticate is the pipeline-native credential provider for Azure Artifacts and uses the build identity already authorized in the A365 project. This avoids embedding credentials, assigning packaging rights to the ARM service principal, or weakening the pinned CLI provenance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
August 19, 2026 12:19
View session
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
pipeline.yaml:393
pip installuses--preeven though an exactfabric-spark-cli==...version is pinned.--prealso relaxes dependency resolution to allow pre-release dependency versions, reducing reproducibility and potentially changing behavior over time. Consider dropping--preunless it is strictly required to install this specific version.
export PIP_DISABLE_PIP_VERSION_CHECK=1
python -m pip install --quiet --pre \
'fabric-spark-cli==0.1.20260807.5'
- Files reviewed: 11/11 changed files
- Comments generated: 0 new
- Review effort level: Lite
Rana Singh (ranadeepsingh)
pushed a commit
that referenced
this pull request
Aug 19, 2026
## Summary Point the OpenAIPrompt pipeline gate at the dedicated SynapseML build workspace and remove the legacy integration-user workspace derivation from the task. ## Prompting Intent The engineer asked for a trusted pipeline run of OpenAIPrompt on Fabric. After feed authentication succeeded, the build service principal could not discover the per-user integration workspace selected by the initial pipeline wiring. ## Linked Sources - Pull request: #2669 - Failed trusted run: https://msdata.visualstudio.com/A365/_build/results?buildId=231838511 - Managed PoC workspace: DONT_DELETE_SynapseML_Build ## Rationale Workspace visibility is identity-specific. The dedicated build workspace matches the Azure service connection and was already proven by the managed PoC, whereas deriving a workspace from the integration account couples the new CLI path to an unrelated certificate-authenticated identity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/azp run |
## Summary Remove the workspace-token and alternate-endpoint experiments after managed Fabric probes proved that the shared certificate identity receives the same backend HTTP 500 from both APIs. ## Prompting Intent Keep investigating PR #2669 until the required Fabric E2E failure was either fixed or proven external, while avoiding speculative code, suppressed tests, or non-blocking success fallbacks. ## Linked Sources - Pull request: #2669 - Fabric-scope probe: https://msdata.visualstudio.com/A365/_build/results?buildId=232335698 - Power BI endpoint probe: https://msdata.visualstudio.com/A365/_build/results?buildId=232338824 - Base-branch reproduction: https://msdata.visualstudio.com/A365/_build/results?buildId=232305251 ## Rationale Both token audiences acquired successfully, yet the same certificate identity received the identical generic backend 500 before any test ran. Keeping either change would add unproven behavior without repairing the gate. Restoring the previously validated tree leaves the external account failure visible and keeps the independent exact-jar Fabric gate intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Move the legacy Fabric Spark Job Definition suite and the Internal Scala compatibility lane from the failing certificate identity to the existing SynapseML Build Azure CLI identity. Resolve the dedicated workspace explicitly, refresh Power BI tokens through Azure CLI, and retain all existing Fabric smoke, notebook, cleanup, and OpenAIPrompt coverage. ## Prompting Intent Investigate and fix PR #2669's failing SynapseML E2E pipeline in an isolated worktree by following the SynapseML PR loop. Preserve blocking E2E coverage, avoid suppressing environmental failures, validate on real Fabric, and continue through current-head CI readiness. ## Linked Sources - Pull request: #2669 - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 - Master baseline build: https://msdata.visualstudio.com/A365/_build/results?buildId=232305251 - PR diagnostic build: https://msdata.visualstudio.com/A365/_build/results?buildId=232324496 - Fabric-scoped token probe: https://msdata.visualstudio.com/A365/_build/results?buildId=232335698 - Power BI endpoint probe: https://msdata.visualstudio.com/A365/_build/results?buildId=232338824 ## Rationale Reusing the proven Azure CLI service connection repairs the broken identity and resource tuple without weakening the gate or exposing tokens. Keeping the existing Scala suite preserves Spark Job Definition creation, upload, execution, monitoring, notebook coverage, and cleanup; replacing it with direct Livy batch submission would have dropped control-plane coverage. The credential fallback remains available for non-CI callers, while CI no longer materializes the retired certificate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Replace the failing legacy Power BI and certificate-based Fabric test path with Azure CLI authentication, public Lakehouse lifecycle APIs, and fabric-spark-cli direct batches. Build and side-load the exact core jar, verify its SHA-256 in every managed notebook, retain logs, cancel failed batches, and hard-delete controlled test artifacts. Preserve structured runner evidence when execution or cleanup raises unexpectedly. ## Prompting Intent Investigate PR #2669's failing SynapseML Fabric E2E pipeline in an isolated worktree and follow the SynapseML PR readiness loop. Keep smoke coverage and all five notebook tests blocking, preserve deterministic cleanup, prove the exact checkout runs on Fabric, and maintain spark4.1 release compatibility. ## Linked Sources - Pull request: #2669 - Original Fabric E2E gap: #2662 - Pipeline failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232398344 - Tracking work item: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_workitems/edit/5538675 - Suppressed review findings: #2669 (review) ## Rationale Direct CLI batches use the supported per-run jar upload and classpath path, whereas Spark Job Definitions reject jar libraries and temporary Fabric Environment publication repeatedly timed out. Public Lakehouse APIs keep resource ownership explicit, and runtime hashing proves Fabric loaded the locally built jar rather than a bundled version. Azure CLI-specific logic lives in a new helper so divergent legacy authentication files remain unchanged in the net PR patch and replay cleanly onto spark4.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Stage a uniquely named, byte-identical SynapseML core jar for every direct Fabric batch so concurrent fabric-spark-cli processes upload to distinct OneLake objects. Preserve runtime SHA-256 provenance and clean every temporary script and jar, including when an earlier cleanup operation fails. ## Prompting Intent Investigate PR #2669's failing SynapseML Fabric E2E pipeline in an isolated worktree and follow the SynapseML PR readiness loop. Fix the exact-head concurrency failure without reducing the three-notebook parallelism, weakening exact-jar verification, or regressing master and spark4.1 compatibility. ## Linked Sources - Pull request: #2669 - Original Fabric E2E gap: #2662 - Failing exact-head build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232406734 - Tracking work item: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_workitems/edit/5538675 ## Rationale The failed batches concurrently appended the same jar basename in OneLake. Unique per-batch basenames remove that remote-object race while retaining the existing bounded concurrency. Staging copies is safer than serializing all notebooks or changing fabric-spark-cli internals; the in-process digest check still proves each copied jar matches the locally built artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Route FabricTestConstants workspace resolution through the shared Azure CLI configuration so pipeline-provided INTEGRATION_WORKSPACE_ID values bypass credential-era account discovery. Preserve the existing credential discovery fallback and add regression coverage for lazy, explicit workspace resolution. ## Prompting Intent Investigate PR #2669's failing SynapseML E2E pipeline in an isolated worktree and follow the SynapseML PR readiness loop. Triage every exact-head failure, fix product and test defects without restoring retired certificate authentication, and preserve master and spark4.1 compatibility. ## Linked Sources - Pull request: #2669 - Original Fabric E2E gap: #2662 - Exact-head compatibility failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232412561 - Tracking work item: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_workitems/edit/5538675 ## Rationale SynapseML-Internal accesses FabricTestConstants directly, so exporting an explicit workspace ID in the pipeline did not reach the newer notebook helper and instead triggered missing INTEGRATION_ACCOUNT errors. Reusing the shared resolver fixes every caller, retains GUID validation and legacy discovery, and avoids reintroducing certificate secrets or private-repository special cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Resolve pipeline-provided Fabric workspace IDs before entering branch-specific workspace discovery. Keep the credential discovery fallback intact and add coverage for explicit and absent workspace overrides. ## Prompting Intent Investigate and fix PR #2669's failing SynapseML E2E pipeline in an isolated worktree by following the SynapseML PR readiness loop. Triage exact-head failures, preserve managed Fabric coverage, and prove both master and spark4.1 compatibility without restoring retired certificate authentication. ## Linked Sources - Pull request: #2669 - Failed release replay: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232418095 - Managed Fabric validation: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232412561 - Tracking work item: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_workitems/edit/5538675 ## Rationale The prior override at the end of FabricTestConstants overlapped spark4.1's branch-specific workspace provisioning implementation, so the release gate could not replay the otherwise compatible change. Resolving an explicit ID before discovery keeps each branch's existing fallback implementation intact and gives the generated patch a shared, non-overlapping insertion point. This avoids a pipeline-only workaround, credential restoration, or taking either branch's full constants file during replay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Forward the pipeline-resolved Fabric workspace ID into the Internal Scala SBT JVM and let Fabric test configuration resolve validated JVM overrides before falling back to environment-based discovery. ## Prompting Intent Investigate and fix PR #2669's failing SynapseML E2E pipeline in an isolated worktree by following the SynapseML PR readiness loop. Continue triaging every exact-head failure until managed Fabric, release replay, and Internal compatibility all exercise the intended code without retired credentials. ## Linked Sources - Pull request: #2669 - Exact-head Internal failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232421537 - Managed Fabric validation: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232421537 - Tracking work item: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_workitems/edit/5538675 ## Rationale The Internal suites consume SynapseML's test-classifier jar in a later SBT invocation, where the shell-only workspace override was not visible and credential-era discovery still evaluated INTEGRATION_ACCOUNT. Passing the already validated GUID as an SBT JVM property creates an explicit process boundary contract. Property resolution retains GUID validation, keeps environment compatibility, and avoids restoring certificate secrets or patching the private repository during CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Join structured marker lines with explicit newline boundaries when aggregating multiple Fabric driver stdout files, and cover logs whose final marker lacks a newline. ## Prompting Intent Investigate and fix the failing SynapseML Fabric E2E pipeline in a dedicated worktree by following the SynapseML PR loop, including exhausting automated review findings and retaining reliable exact-head Fabric evidence. ## Linked Sources - GitHub PR #2669: #2669 - Automated review head: 695e07c - Azure Boards work item: https://msdata.visualstudio.com/DefaultCollection/_workitems/edit/5538675 ## Rationale Normalizing each retained marker line and joining with explicit newlines keeps one structured payload per logical line even when a downloaded stdout file lacks a final line terminator. This is safer than assuming Fabric driver logs always end with a newline and avoids weakening the downstream JSON parser. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Forward the Azure CLI authentication mode and resolved Fabric workspace ID through JAVA_TOOL_OPTIONS so SynapseML-Internal's SBT launcher and child test JVMs receive the same explicit configuration. ## Prompting Intent Investigate and fix the failing SynapseML Fabric E2E pipeline in a dedicated worktree by following the SynapseML PR loop, including deterministic failures in downstream compatibility lanes and cross-branch replay validation. ## Linked Sources - GitHub PR #2669: #2669 - Failed exact-head build: https://msdata.visualstudio.com/A365/_build/results?buildId=232429027 - Azure Boards work item: https://msdata.visualstudio.com/DefaultCollection/_workitems/edit/5538675 ## Rationale SBT_OPTS configures the SBT launcher but does not reliably reach forked test JVMs. The Internal job already uses JAVA_TOOL_OPTIONS for options that must cross every JVM boundary, so appending the dynamic Fabric properties there preserves existing JVM flags and follows the established job-level mechanism. JVM properties take precedence over environment values while retaining the legacy credential-mode fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Make the Scala Internal compatibility lane consume FabricTestConstants from the exact OSS core test-classifier jar instead of compiling Internal's stale same-package duplicate. ## Prompting Intent Investigate and fix PR #2669's failing SynapseML E2E pipeline in an isolated worktree by following the SynapseML PR loop, preserving the Internal EBM and Predict coverage without restoring retired credentials or accepting a false green. ## Linked Sources - Pull request: #2669 - Failing exact-head build: https://dev.azure.com/msdata/A365/_build/results?buildId=232434303 - Review disposition: #2669 (comment) ## Rationale SynapseML-Internal defines the same fully qualified FabricTestConstants object as the OSS test classifier, so its local test class wins before Maven resolver precedence matters. Removing that duplicate only in the Scala compatibility checkout exercises the canonical class and its Azure CLI workspace override. The lane verifies the exact published test jar and class before removal so a missing or stale classifier fails early. This keeps all compatibility suites enabled and avoids fake integration-account values, retired certificate authentication, or package skips. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Clarify that legacy authentication may fall through to branch-specific workspace discovery, while Azure CLI authentication requires an explicit workspace override. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness without bypassing review or CI, including resolving valid current-head automated findings on its Fabric CI prerequisite PR #2669. ## Linked Sources - Pull request: #2669 - Copilot review: #2669 (review) - Tracking work item: AB#5538675 ## Rationale The runtime control flow is already correct and covered by the active full build. Updating only the misleading comment addresses the review finding without changing authentication behavior or introducing another fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Keep Azure CLI as the default Fabric E2E authentication path while adding an explicit key-vault mode for trusted runs that need the legacy integration-user certificate and per-user workspace discovery. Scope vault reads to the required secrets, validate certificate lookup inputs, and retain the legacy publish and Spark Job Definition flow only when selected. ## Prompting Intent The engineer asked to preserve and clean up the former Key Vault-based Fabric test-tenant path, add a switch that can restore it in the future, and update pull request documentation without weakening the current working Azure CLI E2E default. ## Linked Sources - Pull request: #2669 - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 - Passing Azure CLI baseline: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232448412 ## Rationale A queue-time parameter makes rollback explicit and reviewable while compile-time conditions keep legacy secrets out of the default job. Reusing the existing credential-mode Scala implementation preserves the test-tenant behavior, and retaining Azure CLI for OpenAIPrompt avoids mixing a certificate identity with a runner that only supports az login. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Make the checked-in Fabric E2E batch runner explicitly headless, use the long-form output-directory option, and disable Maven-local resolution whenever exact override jars are supplied. Add regression assertions for each command-line guarantee. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness without bypassing Fabric or review policy. While monitoring its prerequisite PR #2669, address all current-head Copilot findings and preserve exact-jar provenance for unattended Fabric validation. ## Linked Sources - Pull request: #2669 - Copilot review: #2669 (review) - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 - Scala batch precedent: https://github.com/microsoft/SynapseML/blob/418398a982951b66b2aabbf1815ae8ad316332a1/core/src/test/scala/com/microsoft/azure/synapse/ml/fabric/FabricPublicOperations.scala ## Rationale Use the live fabric-spark-cli long-form flags to make CI behavior explicit and stable. Apply --no-m2 only when override jars are present, matching the Scala helper and preserving package-free managed-runtime smoke behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Document that the JDK URL-safe Base64 decoder accepts valid JWT final units without explicit padding in both Fabric token paths. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness by keeping prerequisite PR #2669 current-head review-clean. Investigate and resolve Copilot's claim that unpadded JWT payloads force repeated Azure CLI token refreshes. ## Linked Sources - Pull request: #2669 - Copilot review: #2669 (review) - JDK 11 Base64.Decoder contract: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Base64.Decoder.html - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 ## Rationale JDK 11 accepts unpadded two- and three-character final Base64 units by contract and a direct JDK 11 probe decoded representative JWT payloads successfully. Documenting that behavior avoids redundant padding logic while preventing the same false concern in the Azure CLI and credential token providers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Decode legacy Fabric JWT payloads directly from their Base64 URL segment and construct JSON strings with the required UTF-8 charset. Add an unpadded, non-ASCII regression case. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness by keeping prerequisite PR #2669 current-head review-clean. Resolve Copilot's valid portability finding in the sibling Fabric credential token provider without weakening JWT or exact-jar safeguards. ## Linked Sources - Pull request: #2669 - Copilot review: #2669 (review) - JWT specification: https://www.rfc-editor.org/rfc/rfc7519 - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 ## Rationale Use Base64.Decoder.decode(String) to avoid platform-default input encoding and StandardCharsets.UTF_8 for JWT JSON, as required by the format. Keep the helper package-private only for deterministic test coverage of both unpadded decoding and non-ASCII text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Accept uppercase or lowercase SHA-256 text for exact-core Fabric submissions and normalize it to lowercase before embedding the runtime provenance assertion. Add coverage for uppercase input. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness by keeping prerequisite PR #2669 current-head review-clean. Resolve Copilot's digest-casing finding without weakening exact-jar provenance enforcement. ## Linked Sources - Pull request: #2669 - Copilot review: #2669 (review) - FIPS SHA standard: https://csrc.nist.gov/pubs/fips/180-4/upd1/final - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 ## Rationale Hexadecimal SHA-256 text is case-insensitive, but Python's hexdigest output is lowercase. Normalize validated input with Locale.ROOT so uppercase values are accepted while the embedded comparison remains deterministic and provenance checks stay fail-closed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Compute the exact-core Fabric provenance digest in one-megabyte chunks instead of loading the entire jar into driver memory. Add regression assertions for the bounded hashing preamble. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness by keeping prerequisite PR #2669 current-head review-clean. Resolve Copilot's memory-safety finding without changing the provenance value or weakening the exact-jar assertion. ## Linked Sources - Pull request: #2669 - Copilot review: #2669 (review) - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 ## Rationale A fixed one-megabyte read bound preserves the same SHA-256 evidence while making peak memory independent of jar size. The generated Python remains simple, deterministic, and compatible with constrained managed Fabric drivers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#5538675 ## Summary Require test-enabled builds and reject fork pull requests at the Fabric E2E job boundary for both Azure CLI and Key Vault authentication modes. Add pipeline-policy coverage for both guards. ## Prompting Intent Drive SynapseML PR #2664 to merge readiness without exposing trusted Fabric resources or creating false-green CI. Resolve the current-head review finding that default Azure CLI mode could execute external fork code against the shared build workspace. ## Linked Sources - Pull request: #2669 - Review thread: #2669 (comment) - Credential incident: https://msdata.visualstudio.com/A365/_workitems/edit/5538675 ## Rationale Apply the fork and runTests gates at the job boundary so setup, feed authentication, service connections, and Fabric execution are all skipped together. Authentication mode is intentionally irrelevant because both modes access trusted shared resources. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Force-destroy Fabric Spark CLI processes when graceful termination is interrupted, restore the thread interrupt flag, and cover the cleanup path with a deterministic regression test. ## Prompting Intent The engineer asked to rebase PR #2669 onto master, investigate DXT test-tenant compatibility, and keep the pull request merge-ready. Current-head automated review identified that a second interruption during process cleanup could mask the original Fabric E2E failure and leave the child process running. ## Linked Sources - Pull request: #2669 - Review comment: #2669 (comment) ## Rationale Cleanup now treats interruption as a termination constraint rather than a replacement failure: it requests forceful process destruction, restores the interrupt status, and returns so the existing timeout or execution error remains the primary failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Replace both pipeline JMESPath workspace-name interpolations with a shared JSON resolver that performs exact matching, rejects ambiguous responses, and handles workspace names containing quotes safely. ## Prompting Intent The engineer asked to keep PR #2669 merge-ready after rebasing it onto master. Current-head automated review identified that embedding the configured Fabric workspace name inside a JMESPath literal could make valid names containing an apostrophe fail with a query parse error. ## Linked Sources - Pull request and automated review: #2669 ## Rationale Parsing the API response in a small tested helper removes shell/query-language interpolation entirely, preserves exact-match and single-workspace guarantees, and avoids rejecting otherwise valid Fabric workspace names. Sharing the helper also keeps the Fabric E2E and Internal compatibility lanes consistent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Capture the resolved workspace ID before exporting it so Bash preserves the workspace API/resolver pipeline exit status under `set -e -o pipefail`. ## Prompting Intent The engineer asked to keep PR #2669 merge-ready after its master rebase and current-head review fixes. The safe JSON resolver must stop both Fabric lanes immediately when the API fails or workspace matching is missing or ambiguous. ## Linked Sources - Pull request: #2669 ## Rationale An `export NAME="$(command)"` statement reports the status of `export`, which can hide a failed command substitution. A plain assignment retains the substitution pipeline status; exporting only after that succeeds keeps the existing fail-fast contract without adding custom error plumbing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Restore the thread interrupt flag when either the primary Fabric CLI failure or a suppressed cancellation failure was caused by interruption, with regression coverage for both the interrupted and ordinary failure paths. ## Prompting Intent The engineer asked to keep PR #2669 merge-ready after rebasing and resolving all automated review findings. A suppressed current-series review finding identified that interruption during cancellation of an otherwise failed batch was attached to the primary error but did not restore the thread interrupt flag. ## Linked Sources - Pull request and automated review: #2669 ## Rationale Cancellation errors remain suppressed so they cannot mask the original batch failure. Inspecting that existing primary-plus-suppressed error set restores higher-level cancellation semantics without changing which exception callers receive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Teach the disposable SynapseML-Internal compatibility checkout to recognize the exact multiline PEP 561 package-data declaration already emitted by current OSS code generation, while preserving Internal's typing and wheel validation. ## Prompting Intent The engineer asked to rebase PR #2669 onto master and keep it merge-ready. Current-head build 232658505 exposed a post-rebase cross-repository mismatch: the newly merged Internal typing helper recognized only its former one-line setup.py declaration and rejected OSS master’s already-correct representation. ## Linked Sources - Pull request: #2669 - Failing current-head build: https://dev.azure.com/msdata/A365/_build/results?buildId=232658505 - Internal typing feature PR: https://dev.azure.com/msdata/A365/_git/SynapseML-Internal/pullrequest/2258535 ## Rationale The compatibility lane already retargets and commits a disposable Internal checkout. Updating only its known legacy constant avoids weakening the typing gate or reverting OSS package metadata. AST-based validation fails closed if Internal changes the assignment, and focused probes confirm both legacy mutation and already-typed OSS setup.py handling remain idempotent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Wrap interruptions raised during Fabric long-running-operation polling in the existing interrupt-aware exception, restore the thread interrupt flag, and add a deterministic regression test for the polling sleep path. ## Prompting Intent The engineer asked to keep PR #2669 merge-ready after rebasing onto master and to resolve all current-head review findings while preserving reliable Fabric end-to-end execution. ## Linked Sources - Pull request: #2669 - Automated review finding: #2669 (review) ## Rationale Fabric polling used a raw Thread.sleep, which clears the interrupt flag before throwing InterruptedException. Reusing the runner's existing typed interruption path preserves cancellation semantics without changing the polling API, and a pre-interrupted-thread test exercises the failure immediately without network access or timing-sensitive waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Keep streaming the complete fabric-spark-cli transcript to the console and runner.log, but retain only structured result and diagnostic marker lines in memory for evidence parsing. Add a subprocess regression test that proves ordinary remote logs are not accumulated while the full transcript persists. ## Prompting Intent The engineer asked to keep PR #2669 merge-ready after rebasing onto master and to resolve every current-head review finding without weakening managed Fabric end-to-end evidence. ## Linked Sources - Pull request: #2669 - Automated review finding: #2669 (review) ## Rationale Fabric log downloads can be large, and the runner already writes their complete contents to a retained artifact. Keeping only the two controlled marker types in memory removes unbounded duplicate retention while preserving parsing, console visibility, diagnostics, and the full on-disk transcript. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Run SynapseML-Internal's non-live Python compatibility tests and installed wheel typing checks through its common target, while excluding the typing directory only from the first pytest process. This keeps the typing assertions in a fresh process after Spark-dependent tests and adds a pipeline contract. ## Prompting Intent The engineer asked to rebase PR #2669 onto master and make the Fabric E2E and compatibility pipeline reliably merge-ready while preserving all meaningful Internal validation. ## Linked Sources - Pull request: #2669 - Current-head validation: https://dev.azure.com/msdata/A365/_build/results?buildId=232670400 - Internal typing feature: https://dev.azure.com/msdata/A365/_git/SynapseML-Internal/pullrequest/2258535 ## Rationale The combined compatibility invocation ran Spark tests before a typing test that intentionally asserts imports do not create a Py4J gateway. Reusing Internal's own testPythonAIFuncCommon sequencing isolates the typing checks in a new Python process, preserves Internal's evolving default live-test exclusions, and avoids weakening the assertion or mutating global Spark state solely for CI order. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Collaborator
Author
|
Rebased exact-head validation completed successfully for
The rebase itself is patch-equivalent to the previously reviewed head: all 42 range-diff entries are equal and the changed-file list is unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a reusable, manifest-driven path for agents and CI to run the exact
SynapseML checkout on managed Microsoft Fabric Spark, and restore the existing
blocking
Fabric E2Egate with public Fabric APIs and directfabric-spark-clibatches.Azure CLI remains the safe default. A cleaned, explicit
fabricE2EAuthModeswitch also preserves the former Key Vault-backed integration-user path for
trusted manual runs that need to target the legacy per-user Fabric test tenant.
This addresses the runtime-validation gap highlighted by #2662 and the
pipeline failure tracked by ADO #5538675.
Authentication modes
fabricE2EAuthModeazure-cli(default)SynapseML Buildservice principal and the configured build workspacefabric-spark-clibatches with exact local jarskey-vault(opt-in)The default path does not load the legacy account or certificate. Key Vault
mode is compile-time gated and scopes the bootstrap-vault read to the locator
and publication secrets it needs. Both modes are unavailable to untrusted fork
jobs. The certificate lookup validates the account shape, does not print the
derived secret name, stores the PFX as a masked pipeline variable, and maps it
only into the credential-mode test process environment.
FabricOpenAIPromptE2Edeliberately remains on Azure CLI in both configurationsbecause
fabric-spark-cliauthenticates throughaz login; it does not supportthe legacy PFX flow. The switch applies to the Scala Fabric E2E and
SynapseML-Internal compatibility authentication paths.
Coverage
runtime-smokejar-provenancelightgbm-streamingLightGBMClassifierstreaming transfer, validation data, repeated fits/predictions, and exact Scala/JNI/native provenanceopenai-prompt-ai-functionsOpenAIPromptbehavior inspired by AI Functions, with exact core/cognitive provenancecore/packageBinjar in the default modeAll direct-runner profiles emit commit or package provenance, SHA-256 evidence,
runtime diagnostics, JUnit, retained logs, and deterministic scratch-resource
cleanup. Unexpected runner execution, marker-reading, and cleanup exceptions
are converted into failed JSON/JUnit evidence instead of masking the original
failure.
Pipeline gates
Fabric E2E
integration-user password is required.
an untrusted fork, before setup, feed authentication, or service connections.
core/packageBinfrom the checked-out merge commit.batches with maximum concurrency three.
--extra-jars --no-m2, hash the loadedBuildInfo$code-source jar inside every notebook, and fail on any mismatch.failure or interruption, and retain evidence as a pipeline artifact.
fabricE2EAuthMode=key-vault, load the legacy identity only on trustedruns, publish the checkout for Maven resolution, discover its per-user
workspace, and select the retained credential/SJD implementation.
Fabric OpenAIPrompt E2E
endpoint, token, or custom auth header.
structured extraction, usage, error-column, and null-propagation cases in a
Fabric platform notebook.
finally.Both default-mode jobs install pinned
fabric-spark-cli==0.1.20260807.5from the existing authenticated feed.SynapseML-Internal compatibility
FabricTestConstants, then removeSynapseML-Internal's stale same-package duplicate in the Scala compatibility
checkout so it cannot shadow the OSS class.
legacy typing constant is present, so current OSS PEP 561 package metadata
remains authoritative without disabling Internal's wheel/typing gate.
checks, with typing isolated in a fresh process so Spark gateway state cannot
make import-only assertions order-dependent.
workspace ID through
JAVA_TOOL_OPTIONSto SBT and every test JVM.the process environment rather than command-line or JVM options.
spark.aifunc,ebm, andpredictsuites enabled and fail if fewerthan 150 tests execute.
DXT test-tenant assessment
The DXT configuration used by SynapseML-Internal is a viable future backend for
the opt-in
key-vaultmode, but it is not a Key Vault name-only replacementand is not selected by this PR. The Internal change retained its existing
metadata and certificate vaults, selected DXT-specific account and region
aliases, derived the certificate name from that account, and continued to
discover the per-user
SemPy <username>workspace.The account and certificate path worked, but the raw tenant switch left the
Semantic Link job non-blocking. Its follow-up also passes
spark.trident.pbienv=dxt,spark.synapse.ml.powerbi.environment=dxt, and the resolved workspace ID toremote Spark jobs. That follow-up is now merged into Internal
master; build232661708passed all 18 jobs, including
Scala Semantic Link Notebook Tests.The OSS Key Vault contract instead expects generic account, region, and
workspace-prefix aliases. A future DXT rollout must either repoint those aliases
operationally or add an explicit trusted-only tenant selector, and must retain
the extra runtime environment/workspace wiring where Power BI behavior is
exercised. Azure CLI remains the default because it is already proven here,
uses the dedicated build workspace, and avoids loading a long-lived PFX.
Managed Fabric proof
The current-head Azure CLI validation, build
232693731,validated merge SHA
8dac070b46285e91ba28f9a733ddbc5d05cf1eeb,whose second parent is PR head
0d06b40d9abff9b1f5e02dab8e4d7771fed7ccf3:SynapseML-Internal compatibility lanes.
FabricSmokeTests/OnePlusOneand all five blockingFabricNotebookTestspassed, 6/6 total, with notebook concurrency three.synapseml-core_2.12-1.1.3-180-8dac070b-SNAPSHOT.jarwith SHA-256bebd75fb5af5c62fd984aa3cc4f7ea1d8fda680b11d411057269a3f5e1de4b67.OpenAIPromptpassed inDONT_DELETE_SynapseML_Buildthrough Fabric'simplicit endpoint with no explicit endpoint, AAD token, subscription key,
or custom auth header. It used the same core jar plus
synapseml-cognitive_2.12-1.1.3-180-8dac070b-SNAPSHOT.jarwith SHA-256c9ad3676f6203f27b9745e27238e7f3906e5dd48fd88bb3e6411f8490251d358.Lakehouse deletion all returned zero.
installed-wheel typing Python tests against the exact OSS build; all 527
executed tests passed, with three Python tests skipped.
notebook and Lakehouse. Both jobs published retained logs and evidence from
the same build.
Validation
86 passed, 21 skippedin the Fabric runner and pipeline contract suites.FabricAuthenticationSuiteandFabricPublicOperationsSuitetests pass on JDK 11.black==22.3.0.expanded jobs contain unconditional test-enabled and non-fork gates; only
Key Vault mode expands the legacy certificate/SJD tasks.
findings.