perf: run InternalCompat Scala and Python validation in parallel lanes

[Rana Singh (ranadeepsingh)]

What

Splits the SynapseML-Internal Compatibility Check (InternalCompat) job into two parallel matrix legs — scala and python.

Measured: 67.6 min → 39.2 min (−28.4 min, −42%), with identical test counts on both sides.

Rebased onto master after #2655 merged. Two commits, two files, +74 / −9.

Note

Replaces #2657, which was opened from a fork. InternalCompat is guarded by ne(variables['System.PullRequest.IsFork'], 'True') — because SynapseML-Internal is private and fork PRs cannot authenticate to it — so on that PR the job was skipped entirely and the change went unexercised. This branch lives on microsoft/SynapseML so the job actually runs.

Results

Two full runs of this branch, against the serial baseline:

	baseline 231549771	run 1 231589470	run 2 (+ KV gating) 231600597
scala leg	—	38.2 min	39.2 min
python leg	—	38.7 min	37.2 min
Job wall clock	67.6 min	38.7 min	39.2 min
Agent-minutes	67.6	76.9	76.4

Test counts are unchanged — this buys latency, it does not skip work:

	baseline	run 1	run 2
spark.aifunc	129 / 12 suites	129 / 12	129 / 12
ebm	71 / 9	71 / 9	71 / 9
predict	11 / 2	11 / 2	11 / 2
Scala total	211	211	211
Python ExcludeAIFunc	26 passed, 2 skipped	26 passed, 2 skipped	26 passed, 2 skipped

Measured step breakdown (run 2)

=== scala : succeeded  TOTAL 39.2m ===        === python : succeeded  TOTAL 37.2m ===
   Publish OSS to local Maven      8.3m          Publish OSS to local Maven      8.2m
   Compile Internal against OSS    1.6m          Compile Internal against OSS    1.5m
   Free disk space                 0.6m          Free disk space                 0.6m
   Fetch AI service secrets        0.1m          Create Internal conda env       6.3m
   Get certificate from Key Vault  0.1m          Package Internal Python         0.9m
   Run Internal Scala tests       26.6m          Run Internal Python tests      17.8m
   (no conda env)                                (no Key Vault steps)

Scope of the win — this is not 28 min off every PR

InternalCompat was the critical path in the baseline; it no longer is:

	baseline 231549771	run 1 231589470
InternalCompat finishes at	72.1 min — critical path	47.7 min
Databricks GPU E2E finishes at	61.9 min	79.4 min — critical path
Build total	72.4 min	79.6 min

The InternalCompat answer now arrives ~24 min earlier and the job is off the critical path. Total build time in run 1 was higher, because Databricks GPU E2E independently went 58.7 → 71.5 min and every job in that run started 5–10 min later under heavier agent-queue contention. Neither is related to this diff, which touches only InternalCompat. Databricks GPU E2E is the next bottleneck worth attacking.

Why

Its Scala and Python halves are independent — neither reads the other's output — and only ran back to back because they happened to share one checkout.

Segment	Baseline time	Needed by
Shared prefix — checkout, publish OSS to local Maven, retarget, compile	12.5 min	both legs
Create Internal conda env	7.1 min	python only
Scala — spark.aifunc 22.8m + ebm 2.5m + predict 0.4m	25.9 min	scala only
Python — package 1.0m + testPythonExcludeAIFunc 19.2m	20.2 min	python only

How

This mirrors what SynapseML-Internal's own pipeline already does — it shards into 6 Scala jobs and 8 Python jobs via templates/scala_test_job.yml and templates/python_test_job.yml.

strategy:
  matrix:
    scala:
      COMPAT_LANE: scala
    python:
      COMPAT_LANE: python

Shared steps stay ungated; lane-specific steps get eq(variables['COMPAT_LANE'], ...).

The Scala leg skips conda env creation (−6.3 min, measured). This was not a guess: SynapseML-Internal runs ScalaAIFuncTests*, ScalaEBMTests and ScalaPredictTests with useConda unset, reserving that env for its PowerBI and nbtest jobs — exactly the three packages this leg runs. Both runs confirm it: the scala leg has no conda step and still executes all 211 tests. Activation is guarded on the env actually existing, so the step still works unchanged if the env is ever reintroduced.

Key Vault secrets are fetched only in the Scala leg (second commit, in response to review). Run Internal Scala tests is the sole consumer — it maps the four sempy-integration-* values into its env: block. Nothing in the Python lane reads them: ADO exposes Key Vault values as secret variables, which are never auto-exported into a script environment, and the Python steps declare no env: block; sbt's project/Secrets.scala is inert here too (every accessor is behind SYNAPSEML_ENABLE_PUBLISH, unset in this job) and shells out to az directly. Left as-is, the split would have materialised a downloaded certificate and the mmlspark-keys secrets on an agent that never used them — an exposure the serial job did not have, since it had only one agent.

templates/fabric_kv.yml gains a condition parameter defaulting to succeeded() — the implicit step condition — because a - template: reference cannot carry a condition:, and gating only the mmlspark-keys task would have broken the Python leg, which resolves KeyVaultName: '$(fabric-test-kv-name)' from it. The Fabric E2E job includes the template without parameters and is therefore unchanged.

Why two legs and not more

Every leg re-pays the ~12.5 min shared prefix, so sharding has sharply diminishing returns:

Legs	Wall clock	Cost
1 (before)	67.6 min	1 agent
2 (this PR)	39.2 min (measured)	2 agents
3 (split spark.aifunc from ebm+predict)	~38 min	3 agents

The trade is +9 agent-minutes (+13%) — the duplicated prefix — for −28 min of latency. A third leg would buy ~1 min for another full prefix. Internal can shard 6-and-8 ways because its legs resolve a published OSS artifact rather than building one from the change under test, so they have no comparable prefix to amortise.

Validation

Executed, not just reviewed:

• Two green runs of the job on this branch (68/68 jobs, 0 failures), with test counts matching the baseline exactly.
• YAML parses; all 12 inline scripts pass bash -n.
• Lane simulation over the parsed job confirms shared steps run in both legs and each lane-specific step runs in exactly one — lane=scala runs 14 / skips 7, lane=python runs 18 / skips 3.
• Template regression check asserts fabric_kv.yml has exactly two includes — one gated to scala (InternalCompat), one with parameters=None (Fabric E2E, default succeeded()), proving Fabric E2E is unaffected.
• The conda-presence predicate was executed against 5 realistic conda env list shapes — env present, present-and-active (*), absent, conda missing entirely, and a similar-but-different name (synapseml-internal-old) — plus a set -e safety check. All 5 correct.

Risk

• InternalCompat keeps continueOnError: true and is not a required status check, so the new per-leg check names (... Compatibility Check scala / ... python) do not affect branch protection.
• Both legs compute the OSS version from the same pinned build commit, so the retarget is identical in each — and fix: stop InternalCompat requesting an Internal version that was never published #2655 already made that step idempotent.
• One behavioural change beyond the split: the two Python steps go from succeededOrFailed() back to succeeded(). That condition existed only to keep Python running when the Scala tests failed in the same job; the split now guarantees that structurally.

Not done here

The ~12.5 min shared prefix is now the floor, and Databricks GPU E2E is the build's new critical path. Caching the Internal conda env, or publishing the OSS M2 artifacts once and fanning out, would attack the prefix — but both add serialisation and belong in their own change with their own measurements.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Rana Singh (ranadeepsingh)]

/azp run

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR optimizes the SynapseML-Internal Compatibility Check Azure Pipelines job by converting it into a 2-leg matrix (scala and python) so the Scala and Python validation paths can run in parallel, reducing overall PR feedback latency.

Changes:

• Introduces a strategy.matrix with COMPAT_LANE set to scala or python to run two parallel legs.
• Gates lane-specific steps (conda env creation, Scala tests, Python packaging/tests, and per-lane test result publishing) using condition: eq(variables['COMPAT_LANE'], ...).
• Makes Scala test execution resilient to the conda env being absent by guarding conda activation on env presence.

Show a summary per file

File	Description
pipeline.yaml	Splits InternalCompat into parallel Scala/Python matrix lanes and adds lane-conditional execution for conda, test, and publish steps.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Measured results — build 231589470 ✅

Build succeeded, 68/68 jobs, 0 failures. Both matrix legs were created and ran on separate agents.

Wall clock for the InternalCompat job

Leg	Projected	Measured
scala	~39m	38.2m
python	~41m	38.7m
Job wall clock (max of legs)	~41m	38.7m
Serial baseline (231549771)	—	67.6m

−28.9 min (−43%).

Test parity — no tests were lost

This is the part that actually needed proving, since the scala leg now runs without the conda env.

Suite group	Baseline (serial, conda present)	scala leg (no conda)
spark.aifunc	129 tests / 12 suites	129 / 12
ebm	71 / 9	71 / 9
predict	11 / 2	11 / 2
Scala total	211 / 23	211 / 23
Python (ExcludeAIFunc)	26 passed, 2 skipped	26 passed, 2 skipped

Identical counts on both sides. The scala leg's step list contains no Accept Anaconda TOS, no Create Internal conda env, and no conda-feed auth — confirming on a real agent what SynapseML-Internal/templates/scala_test_job.yml implies via useConda: false for the aifunc / ebm / predict suites.

Per-leg step breakdown

=== InternalCompat scala : succeeded  TOTAL 38.2m ===
   Publish OSS to local Maven                        8.5m
   Compile Internal against OSS                      1.6m
   Free disk space                                   0.9m
   Run Internal Scala tests                         25.1m     <- 211 tests
   (no conda steps)

=== InternalCompat python : succeeded  TOTAL 38.7m ===
   Publish OSS to local Maven                        7.9m
   Compile Internal against OSS                      1.4m
   Create Internal conda env                         7.0m
   Package Internal Python against OSS               0.9m
   Run Internal Python tests (ExcludeAIFunc)        18.1m     <- 26 passed / 2 skipped

Honest scope of the win

This shortens the InternalCompat signal, not necessarily total PR turnaround:

	baseline 231549771	this build 231589470
InternalCompat finishes at	72.1m — was the critical path	47.7m
Databricks GPU E2E finishes at	61.9m	79.4m — now the critical path
Build total	72.4m	79.6m

InternalCompat is now off the critical path — its answer arrives ~24 min earlier. The build total rose in this particular run because Databricks GPU E2E independently went 58.7m → 71.5m and every job started ~5-10 min later (heavier agent-queue contention that run). Neither is related to this diff, which touches only the InternalCompat job. Databricks GPU E2E is the next bottleneck worth attacking.

Cost

Two agents re-pay the ~12 min shared prefix (checkout + Publish OSS to local Maven + Compile Internal against OSS), so agent-minutes go 67.6 → 76.9 (+9.3, +14%). That is the deliberate trade: +14% agent-minutes for −43% latency on the slowest signal in the build. It is also why this stops at 2 legs — a 3rd would buy ~3 min for another full prefix.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite