docs: document Spark 4 install coordinates

[Rana Singh (ranadeepsingh)]

Summary

• add a Spark runtime / Scala binary / Python baseline / release tag / Maven artifact matrix
• document copy-paste Spark 3.5, 4.0, and 4.1 installs across the README, canonical/versioned install guides, and website landing page
• explain that the PyPI wheel supplies wrappers but does not load JVM classes
• gate Spark port coordinates with two explicit published artifact versions in a deterministic, non-network regression test
• keep the Isolation Forest quickstart honestly scoped to its Spark 3.5 / Scala 2.12 dependency baseline

Review blockers resolved

1. Primary website surface: website/src/pages/index.js uses centralized Spark 3.5/4.0/4.1 coordinates and complete Python variants; installDocs.test.js covers this surface.
2. Complete Spark 3.5 Python variant: README, canonical install docs, and current versioned docs specify Python 3.11, pyspark>=3.5,<3.6, and the _2.12 JVM coordinate together.
3. Isolation Forest compatibility: the notebook remains explicitly Spark 3.5 / Scala 2.12 because its pinned numpy==1.22.4 dependency is not a Python 3.12/3.13 setup.
4. No manufactured port releases: published-spark-ports.lock contains only the two independently verified Spark 4 artifact versions. Tests compare rendered documentation and website coordinates to that lock; they do not invoke Git or assume full repository history.

Before

Current installation surfaces directed users to Scala 2.12 artifacts such as:

com.microsoft.azure:synapseml_2.12:1.1.3

On Spark 4, which uses Scala 2.13, that can leave the Python wrapper importable but the matching JVM class unavailable, producing LightGBMClassifier does not exist in the JVM.

After

Spark 3.5 / Scala 2.12 / Python 3.11 -> com.microsoft.azure:synapseml_2.12:1.1.3
Spark 4.0 / Scala 2.13 / Python 3.12 -> com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0
Spark 4.1 / Scala 2.13 / Python 3.13 -> com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1

All variants use synapseml==1.1.3; Spark 4 artifacts require https://mmlspark.blob.core.windows.net/maven.

Published artifact evidence

• live tags v1.1.3-spark4.0 and v1.1.3-spark4.1 exist
• Spark 4.0 aggregate POM and LightGBM JAR return HTTP 200
• Spark 4.1 aggregate POM and LightGBM JAR return HTTP 200
• both JARs contain com/microsoft/azure/synapse/ml/lightgbm/LightGBMClassifier.class

Future version updates

1. Run the normal context-anchored base-version update. website/src/installArtifacts.js exposes the base version from its release-managed synapseml==... pin, and the current docs reuse SYNAPSEML_VERSION instead of repeating the number in each command.
2. Publish and verify each Spark port artifact independently.
3. Grep website/test/published-spark-ports.lock and update its two values only after the corresponding artifacts exist. The lock is intentionally excluded from automatic base-version replacement.
4. Run npm test from website. A base-version bump fails until both explicitly locked port versions agree with the documented coordinates.

No commit mapping, tag checkout, git show, or full-history clone is required.

Validation

• npm test in website: 31/31 passed
• documentation projection completed successfully
• JavaScript syntax, lock JSON, and git diff --check passed
• real local Spark sessions resolved the published artifacts, instantiated UDFTransformer, and completed spark.range(3).count() on Spark 3.5.0 / Python 3.11, Spark 4.0.1 / Python 3.12, and Spark 4.1.0 / Python 3.12
• Spark 4.0.0 was explicitly rejected after reproducing a JVM NoSuchMethodError; the install floor is now pyspark>=4.0.1,<4.1, matching the published POM compile baseline
• documentation tests have no child-process or Git-history dependency
• no commit SHA mappings were introduced
• exact-head Linux website production build and dead-link scan passed
• local Windows production build reaches client/server compilation but is blocked by existing generated-MDX parsing failures in three unchanged docs
• repository-wide Black reports nine pre-existing non-doc formatting failures; this PR changes no checked Python source
• exact-head Azure validation was requested after this update

Review follow-up

• All three review threads have evidence replies and are resolved.
• Every historical suppressed Copilot finding was fixed and documented in PR comments.
• Current-head Copilot review covered 15/16 changed files with zero new or suppressed findings.
• Current-head website build, dead-link scan, CodeQL, dependency review, title, WIP, and CLA checks passed.

Closes #2402

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR updates SynapseML installation documentation and the website landing page to clearly document Spark 3.5 vs Spark 4.0/4.1 artifact coordinates (Scala 2.12 vs 2.13), and adds a deterministic website test that locks published Spark-port metadata against tagged sources.

Changes:

• Added a Spark runtime / Scala / Python baseline / release tag / Maven coordinate matrix to README + install guides, clarifying that the PyPI wheel doesn’t load JVM artifacts.
• Centralized website install coordinates into website/src/installArtifacts.js and updated the landing page to render all supported runtime variants.
• Added website/test/installDocs.test.js plus a published-spark-ports.lock file to prevent docs/website from advertising unpublished Spark ports.

Show a summary per file

File	Description
website/versioned_docs/version-1.1.3/Reference/R Setup.md	Adds Spark 3.5 vs Spark 4 coordinate guidance and points to the install matrix.
website/versioned_docs/version-1.1.3/Overview.md	Replaces blanket “requires Scala 2.12” with Spark/Scala-specific artifact guidance.
website/versioned_docs/version-1.1.3/Get Started/Install SynapseML.md	Adds install matrix and expanded cross-Spark install examples (Python, SBT, --packages).
website/versioned_docs/version-1.1.3/Explore Algorithms/Deep Learning/Getting Started.md	Clarifies wheel vs JVM artifact responsibilities and links to the install matrix.
website/versioned_docs/version-1.1.3/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md	Explicitly scopes the quickstart to Spark 3.5 / Scala 2.12 and pins the coordinate.
website/test/published-spark-ports.lock	Introduces a locked record of published Spark 4 port tags/coordinates/baselines.
website/test/installDocs.test.js	Adds non-network regression tests ensuring docs/website match the locked port metadata and tagged sources.
website/src/pages/index.js	Updates landing page copy and snippets; renders a runtime matrix using centralized coordinates.
website/src/installArtifacts.js	Adds a central definition of repository + Spark 3.5/4.0/4.1 coordinates and Python baselines.
README.md	Adds install matrix + clarifies wrapper vs JVM artifact and Spark 4 Scala 2.13 requirements.
docs/Reference/R Setup.md	Adds Spark 3.5 vs Spark 4 coordinate guidance and links to install matrix.
docs/Overview.md	Replaces blanket “requires Scala 2.12” with Spark/Scala-specific artifact guidance.
docs/Get Started/Install SynapseML.md	Adds install matrix and expanded cross-Spark install examples (Python, SBT, --packages).
docs/Explore Algorithms/Deep Learning/ONNX.md	Replaces hardcoded Scala 2.12 coordinates with runtime-aware placeholders and install-matrix guidance.
docs/Explore Algorithms/Deep Learning/Getting Started.md	Clarifies wheel vs JVM artifact responsibilities and links to install matrix.
docs/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.ipynb	Updates prerequisites + pins the Spark 3.5 / Scala 2.12 coordinate in the notebook.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 15/16 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

docs/Explore Algorithms/Deep Learning/Getting Started.md:29

• This page now frames the sample as using a generic “Spark 3 / Scala 2.12” runtime, but the repo’s current build baseline is Spark 3.5 (see build.sbt’s sparkVersion). Given this doc also references a specific Databricks runtime in the note above, consider updating this sentence to explicitly state the supported Spark minor line (e.g., Spark 3.5 / Scala 2.12) or clarify that the sample is for an older Spark 3.x runtime and may not be compatible with the current artifacts.

The Python wheel supplies wrappers but does not install the JVM package. This
sample uses a Spark 3 / Scala 2.12 runtime, so add the following Maven library:

website/src/installArtifacts.js:3

• Deriving version by splitting pythonPackage is unnecessarily brittle (it silently yields undefined if the string format changes). Consider making version the single source of truth (e.g., const version = "1.1.3"; const pythonPackage = synapseml==${version};) so the coordinate construction can’t desynchronize from the package string.

const pythonPackage = "synapseml==1.1.3";
const version = pythonPackage.split("==")[1];
const repository = "https://mmlspark.blob.core.windows.net/maven";

website/test/installDocs.test.js:19

• currentVersion is extracted from docusaurus.config.js via a regex that assumes an exact let version = "..." spelling. This is fragile (e.g., changing it to const or single quotes will break tests even if the version is unchanged). Consider reading the version from versions.json (already parsed) or importing/awaiting the config’s exported function and reading customFields.version, so the test tracks the actual config structure rather than its textual formatting.

const config = fs.readFileSync(
  path.join(repoRoot, 'website', 'docusaurus.config.js'),
  'utf8',
);
const currentVersion = config.match(/let version = "([0-9.]+)"/)?.[1];
const publishedVersions = JSON.parse(
  fs.readFileSync(path.join(repoRoot, 'website', 'versions.json'), 'utf8'),
);

• Files reviewed: 15/16 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed all three suppressed findings from the f1f2bb707c84d677daed761b330c7642b33b3445 Copilot review in bb8219c5273f751f93e300346fa1b1dafd81f7d9:

• canonical and versioned Deep Learning Getting Started docs now state that the current Maven example targets Spark 3.5 / Scala 2.12 and warn that historical Databricks 10.4.x needs a compatible older SynapseML release;
• installArtifacts.js now validates the synapseml== pin before slicing it, preventing silent undefined coordinates while preserving the repository's context-anchored version-bump mechanism;
• installDocs.test.js now takes the current stable documentation version from versions.json rather than regex-parsing Docusaurus JavaScript source.

Validation: website tests 30/30, changed Deep Learning MDX compilation passed, metadata/JSX syntax and git diff --check passed. The bump-version dry run reported no new affected-file failures.

[Copilot]

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

website/src/pages/index.js:306

• The Synapse (Spark 3.5 pools) snippet hard-codes the SynapseML coordinate as com.microsoft.azure:synapseml_2.12:1.1.3 even though this page now centralizes install coordinates via installArtifacts. This can drift on future base-version bumps and defeats the goal of a single source of truth. Consider interpolating spark35.coordinate (or a spark35-derived variable) in this snippet the same way Fabric/Spark Packages/Python do.

                  <p>SynapseML can be installed on Synapse adding the following to the first cell of a notebook:</p>
                  For Spark3.5 pools:
                  <CodeSnippet
                    snippet={`%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
      "spark.jars.repositories": "${repository}",

website/test/installDocs.test.js:131

• new RegExp(installArtifacts.repository) treats characters like . as regex wildcards, so this assertion can pass even if the docs contain a malformed repository URL. Consider escaping the repository string before constructing the regex (e.g., via a small escapeRegExp helper) so the test requires an exact literal match.

This issue also appears on line 222 of the same file.

    );
    assert.match(markdown, /--repositories "\$SYNAPSEML_REPOSITORY"/);
    assert.match(markdown, new RegExp(installArtifacts.repository));
    assert.doesNotMatch(markdown, /mmlspark\.azureedge\.net/);

website/test/installDocs.test.js:225

• Same issue as above: assert.match(onnx, new RegExp(installArtifacts.repository)) does not escape regex metacharacters in the repository URL, which weakens the assertion. Escaping the string (or using a plain substring check) would make this regression test reliably detect repository drift.

    );
  }
  assert.match(onnx, new RegExp(installArtifacts.repository));
  assert.match(rSetup, /examples below use Spark 3\.5 \/ Scala 2\.12/);

• Files reviewed: 15/16 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed every suppressed finding from the bb8219c5273f751f93e300346fa1b1dafd81f7d9 Copilot review in 9b420ec0ab4d607772b3bb419395d54ce2cb2fe7:

• the Synapse Spark 3.5 landing-page snippet now interpolates spark35.coordinate instead of duplicating com.microsoft.azure:synapseml_2.12:1.1.3;
• repository checks now use exact substring assertions instead of unescaped regular expressions in both install-guide and ONNX coverage;
• regression coverage rejects the old hard-coded Spark 3.5 coordinate on the landing page.

Evidence: website tests 30/30, JSX/test syntax and git diff --check passed. Azure was not triggered.

[Copilot]

Review details

Suppressed comments (5)

Previously missed (5) — in code that hasn't changed since the last review.

docs/Get Started/Install SynapseML.md:121

• This section instructs users to install a single runtime variant, but the bash snippet immediately below lists three different pyspark version ranges. This makes it easy to run all commands and unintentionally replace the intended PySpark version. Consider restructuring to present mutually exclusive options (separate code blocks, tabs, or an explicit “choose one” block with only one command active).

To try out SynapseML on a Python (or Conda) installation, you can get Spark
installed via pip. Install the Python wrapper and PySpark version for one
complete runtime variant, then start Spark with that variant's JVM artifact:

```bash

README.md:234

• The text says to install PySpark/SynapseML for “one complete runtime variant”, but the following bash block lists three different pyspark pins. A copy/paste user may run all three and end up with the last-installed PySpark version, which contradicts the intent. Consider splitting into three separate code blocks (or adding an explicit “choose exactly one” heading plus commenting-out the non-selected commands).

Install the Python wrapper and PySpark version for one complete runtime variant,
then start Spark with that variant's JVM artifact:

```bash
SYNAPSEML_VERSION=1.1.3

website/versioned_docs/version-1.1.3/Get Started/Install SynapseML.md:121

• This section says to install one runtime variant, but the bash snippet that follows includes three different pyspark pins. Users may run all three and end up with a mismatched PySpark/JVM artifact pairing. Consider presenting the runtime variants as mutually exclusive choices (separate code blocks, tabs, or an explicit “choose one” instruction).

To try out SynapseML on a Python (or Conda) installation, you can get Spark
installed via pip. Install the Python wrapper and PySpark version for one
complete runtime variant, then start Spark with that variant's JVM artifact:

```bash

website/src/pages/index.js:460

• The Python install snippet lists three different pyspark version specs in a single block. Since these are mutually exclusive runtime variants, users may run them sequentially and silently end up with only the last-installed PySpark version. Consider adding a clear “choose one” note or splitting into per-runtime blocks so copy/paste yields a single consistent setup.

                  Install both the Python wrapper and the PySpark version
                  matching the selected JVM artifact.
                  <CodeSnippet
                    snippet={`# Spark 4.1 / Python ${spark41.pythonBaseline}
python -m pip install "${spark41.pythonPackage}" "pyspark${spark41.pysparkSpec}"

website/src/pages/index.js:491

• This code snippet is Scala (libraryDependencies += ...) but the CodeSnippet language is set to jsx, which will apply incorrect syntax highlighting. Set lang to scala (or omit it if CodeSnippet auto-detects).

                    snippet={`resolvers += "SynapseML" at "${repository}"

// Spark 4.1; use "${spark40.coordinate}" for Spark 4.0.
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "${spark41.coordinate.split(":")[2]}"

// Spark 3.5:
// libraryDependencies +=
//   "com.microsoft.azure" % "synapseml_2.12" % "${spark35.coordinate.split(":")[2]}"`}
                    lang="jsx"
                  ></CodeSnippet>

• Files reviewed: 15/16 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

Runtime install validation found and fixed one concrete compatibility issue: the published 1.1.3-spark4.0 artifact is compiled against Spark 4.0.1. It resolves on Spark 4.0.0 but fails when UDFTransformer initializes with NoSuchMethodError, so the documented range now starts at pyspark>=4.0.1. Real local sessions now resolve the published coordinate, instantiate UDFTransformer, and complete spark.range(3).count() == 3 for Spark 3.5.0/Python 3.11, Spark 4.0.1/Python 3.12, and Spark 4.1.0/Python 3.12. Website tests are 31/31.

[Rana Singh (ranadeepsingh)]

/azp run

[Rana Singh (ranadeepsingh)]

Copilot review

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

docs/Explore Algorithms/Deep Learning/ONNX.md:87

• In the Maven XML example, the placeholders are written as angle-bracket tokens inside element bodies (e.g., synapseml-deep-learning_<scala-binary-version> and <version><synapseml-deep-learning-version></version>). Copy/pasting this will produce invalid XML because <...> is parsed as nested tags. Use a valid placeholder form (e.g., escape as <scala-binary-version> / <synapseml-deep-learning-version>, or show concrete ..._2.12 / ..._2.13 examples with matching versions).

  ```xml
  <dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>synapseml-deep-learning_<scala-binary-version></artifactId>
    <version><synapseml-deep-learning-version></version>
    <exclusions>

• Files reviewed: 15/16 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Fixed the current-head suppressed ONNX finding in 331ff1e90b: the Maven example now uses copy-safe uppercase placeholder text instead of angle-bracket tokens that XML interprets as nested elements. Regression assertions reject the invalid form; website tests remain 31/31.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 15/16 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Azure build 231746953 failed before tests in PythonTests vw because SBT could not resolve unchanged external dependencies (org.scoverage artifacts and com.globalmentor:hadoop-bare-naked-local-fs). This is infrastructure-only and unrelated to the documentation patch; requesting a clean exact-head rerun.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.19%. Comparing base (a6fd536) to head (331ff1e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2660   +/-   ##
=======================================
  Coverage   87.19%   87.19%           
=======================================
  Files         337      337           
  Lines       19554    19554           
  Branches     1894     1894           
=======================================
  Hits        17050    17050           
  Misses       2504     2504           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Review details

• Files reviewed: 17/18 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).