fix: validate ComputeModelStatistics input columns - #2633
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey Rana Singh (@ranadeepsingh) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This PR improves ComputeModelStatistics robustness when scoring metadata is missing by adding explicit input-column validation (label/scored-labels/scores) and expanding tests and docs to cover unannotated DataFrame scenarios, addressing the historical NullPointerException reported in #736.
Changes:
- Add schema-based validation to fail fast with actionable
IllegalArgumentExceptionmessages instead of late Spark expression failures. - Expand
VerifyComputeModelStatisticscoverage for missing/incorrect configured columns and for classification/regression behaviors without scoring metadata. - Document how
labelCol,scoredLabelsCol, andscoresColare inferred/required for unannotated DataFrames.
Show a summary per file
| File | Description |
|---|---|
| core/src/main/scala/com/microsoft/azure/synapse/ml/train/ComputeModelStatistics.scala | Adds validateInputColumns and invokes it before metric computation paths to improve failure modes for missing columns. |
| core/src/test/scala/com/microsoft/azure/synapse/ml/train/VerifyComputeModelStatistics.scala | Adds regression tests for missing columns and metadata-less DataFrame configurations. |
| core/src/main/scala/com/microsoft/azure/synapse/ml/train/ComputeModelStatistics.txt | Updates user-facing documentation clarifying column roles and configuration when metadata inference is unavailable. |
Review details
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
67f5cd4 to
1d83994
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
1d83994 to
dd9cbeb
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2633 +/- ##
==========================================
- Coverage 88.60% 88.40% -0.20%
==========================================
Files 331 331
Lines 18531 18557 +26
Branches 1752 1751 -1
==========================================
- Hits 16419 16405 -14
- Misses 2112 2152 +40
🚀 New features to boost your workflow:
|
## Summary Replace null and late Spark failures with schema-first column resolution, ambiguity and type checks, metadata-safe overrides, scalar/vector handling, regression schema agreement, and persisted pipeline coverage. Preserve the precision-recall AUC behavior merged in PR microsoft#2635 and add its Spark 4.1 replay prerequisite. ## Prompting Intent Make microsoft#2633 merge-ready by auditing ComputeModelStatistics across missing and ambiguous columns, classification versus regression, Spark metadata, scalar and vector scores, null rows, schema/runtime behavior, copying, persistence, code generation, Python wrappers, performance, and Spark 4.1 compatibility. Rebase on current master, resolve all valid review feedback, and add focused negative and end-to-end tests. ## Linked Sources - GitHub issue microsoft#736: microsoft#736 - Pull request microsoft#2633: microsoft#2633 - Merged prerequisite PR microsoft#2635: microsoft#2635 - Copilot review thread: microsoft#2633 (comment) - SynapseML review checklist: .github/skills/code-review/SKILL.md ## Rationale Validate resolved columns before any metadata or Catalyst access, preserve explicit parameter precedence over metadata, and accept only runtime-supported numeric/vector shapes so transformSchema and transform fail consistently. Cast numeric scalar inputs to doubles, quote top-level Spark column names, remove obsolete unbounded DataFrame caching, retain binary AUC/areaUnderPR semantics from PR microsoft#2635, and use defaultCopy so configured stages survive pipeline copy and persistence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dd9cbeb to
60739cc
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
## Summary Replace null and late Spark failures with schema-first column resolution, ambiguity and type checks, metadata-safe overrides, scalar/vector handling, regression schema agreement, and persisted pipeline coverage. Preserve the precision-recall AUC behavior merged in PR microsoft#2635 and replay its scoped test baseline before the dependent Spark 4.1 prerequisite. ## Prompting Intent Make microsoft#2633 merge-ready by auditing ComputeModelStatistics across missing and ambiguous columns, classification versus regression, Spark metadata, scalar and vector scores, null rows, schema/runtime behavior, copying, persistence, code generation, Python wrappers, performance, and Spark 4.1 compatibility. Rebase on current master, resolve all valid review feedback, and add focused negative and end-to-end tests. ## Linked Sources - GitHub issue microsoft#736: microsoft#736 - Pull request microsoft#2633: microsoft#2633 - Merged prerequisite PR microsoft#2635: microsoft#2635 - Failed Azure compatibility build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231383604 - Copilot review thread: microsoft#2633 (comment) - SynapseML review checklist: .github/skills/code-review/SKILL.md ## Rationale Validate resolved columns before metadata or Catalyst access, preserve explicit parameter precedence, and accept only runtime-supported numeric/vector shapes so transformSchema and transform fail consistently. Cast numeric scalar inputs to doubles, quote top-level Spark names, remove obsolete unbounded caching, retain PR microsoft#2635 AUC semantics, and use defaultCopy for persisted stages. Teach release replay to activate add-only scoped baselines required by later prerequisites and validate each baseline against that dependent commit parent, avoiding broad replay of PR microsoft#2507. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
60739cc to
506f79a
Compare
## Summary Replace null and late Spark failures with schema-first column resolution, ambiguity and type checks, metadata-safe overrides, scalar/vector handling, regression schema agreement, actionable metric validation, and persisted pipeline coverage. Preserve the precision-recall AUC behavior merged in PR microsoft#2635 and replay its scoped test baseline before the dependent Spark 4.1 prerequisite. ## Prompting Intent Make microsoft#2633 merge-ready by auditing ComputeModelStatistics across missing and ambiguous columns, classification versus regression, Spark metadata, scalar and vector scores, null rows, schema/runtime behavior, copying, persistence, code generation, Python wrappers, performance, and Spark 4.1 compatibility. Rebase on current master, resolve all valid review feedback, and add focused negative and end-to-end tests. ## Linked Sources - GitHub issue microsoft#736: microsoft#736 - Pull request microsoft#2633: microsoft#2633 - Merged prerequisite PR microsoft#2635: microsoft#2635 - Failed Azure compatibility build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231383604 - Copilot review threads: microsoft#2633 (comment) and microsoft#2633 (comment) - SynapseML review checklist: .github/skills/code-review/SKILL.md ## Rationale Validate resolved columns before metadata or Catalyst access, preserve explicit parameter precedence, and accept only runtime-supported numeric/vector shapes so transformSchema and transform fail consistently. Cast numeric scalar inputs to doubles, quote top-level Spark names, remove obsolete unbounded caching, retain PR microsoft#2635 AUC semantics, use defaultCopy for persisted stages, and report invalid metrics as IllegalArgumentException with valid choices. Teach release replay to activate add-only scoped baselines required by later prerequisites and validate each baseline against that dependent commit parent, avoiding broad replay of PR microsoft#2507. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
506f79a to
cd04062
Compare
## Summary Replace null and late Spark failures with schema-first column resolution, ambiguity and type checks, metadata-safe overrides, scalar/vector handling, regression schema agreement, actionable metric validation, Spark-configured case resolution, and persisted pipeline coverage. Preserve the precision-recall AUC behavior merged in PR microsoft#2635 and replay its scoped test baseline before the dependent Spark 4.1 prerequisite. ## Prompting Intent Make microsoft#2633 merge-ready by auditing ComputeModelStatistics across missing and ambiguous columns, classification versus regression, Spark metadata, scalar and vector scores, null rows, schema/runtime behavior, copying, persistence, code generation, Python wrappers, performance, and Spark 4.1 compatibility. Rebase on current master, resolve all valid review feedback, and add focused negative and end-to-end tests. ## Linked Sources - GitHub issue microsoft#736: microsoft#736 - Pull request microsoft#2633: microsoft#2633 - Merged prerequisite PR microsoft#2635: microsoft#2635 - Failed Azure compatibility build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231383604 - Copilot review threads: microsoft#2633 (comment), microsoft#2633 (comment), and microsoft#2633 (comment) - SynapseML review checklist: .github/skills/code-review/SKILL.md ## Rationale Validate and canonicalize resolved columns before metadata or Catalyst access, using Spark SQLConf resolver semantics so configured case sensitivity is honored. Preserve explicit parameter precedence and accept only runtime-supported numeric/vector shapes so transformSchema and transform fail consistently. Cast numeric scalar inputs to doubles, quote top-level Spark names, remove obsolete unbounded caching, retain PR microsoft#2635 AUC semantics, use defaultCopy for persisted stages, and report invalid metrics as IllegalArgumentException with valid choices. Teach release replay to activate add-only scoped baselines required by later prerequisites and validate each baseline against that dependent commit parent, avoiding broad replay of PR microsoft#2507. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cd04062 to
d82903e
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
GitHub#736 ## Summary Validate ComputeModelStatistics inputs before Spark metadata or Catalyst access, align regression schemas with runtime output, add focused public-path coverage, and replay the target metadata baseline during Spark 4.1 compatibility validation. ## Prompting Intent Make microsoft#2633 engineering-ready on master without broadening sensitive CI behavior. Preserve public JVM and generated Python APIs while validating label, prediction, and score columns; actionable error timing and messages; scalar and vector scores; null and boundary inputs; copy and persistence behavior; and Spark 4.1 release compatibility. ## Linked Sources - GitHub issue microsoft#736: microsoft#736 - Pull request microsoft#2633: microsoft#2633 - Merged prerequisite PR microsoft#2635: microsoft#2635 - Related metadata PR microsoft#2632: microsoft#2632 - Copilot review threads: microsoft#2633 (comment), microsoft#2633 (comment), microsoft#2633 (comment) - SynapseML contributor guidance: AGENTS.md - SynapseML review guidance: .github/skills/code-review/SKILL.md ## Rationale Centralize schema-first validation in a private helper so transform and transformSchema share column resolution, case-sensitivity, ambiguity, and type rules without exceeding the repository source-size limit. Preserve explicit parameter precedence, existing serialized Params and generated wrapper shape, and PR microsoft#2635 metric behavior. Accept only runtime-supported numeric scalars and Spark vectors, quote top-level column names, drop null metric rows, and use defaultCopy for persistence. Keep pipeline.yaml unchanged; add only the missing PR microsoft#2632 prerequisite after PR microsoft#2635 because Spark 4.1 lacks both target baselines and the final patch cannot apply until they are replayed in target-history order. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
d82903e to
45352dc
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
7fd1a0b
into
microsoft:master
AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make microsoft#2629 engineering-ready on current master after PRs microsoft#2638 and microsoft#2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue microsoft#1667: microsoft#1667 - GitHub pull request microsoft#2629: microsoft#2629 - Automated review overview: microsoft#2629 (review) - Suppressed review findings: microsoft#2629 (review) - Merged pull request microsoft#2638: microsoft#2638 - Merged pull request microsoft#2633: microsoft#2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained microsoft#2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make microsoft#2629 engineering-ready on current master after PRs microsoft#2638 and microsoft#2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue microsoft#1667: microsoft#1667 - GitHub pull request microsoft#2629: microsoft#2629 - Automated review overview: microsoft#2629 (review) - Suppressed review findings: microsoft#2629 (review) - Merged pull request microsoft#2638: microsoft#2638 - Merged pull request microsoft#2633: microsoft#2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained microsoft#2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make microsoft#2629 engineering-ready on current master after PRs microsoft#2638 and microsoft#2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue microsoft#1667: microsoft#1667 - GitHub pull request microsoft#2629: microsoft#2629 - Automated review overview: microsoft#2629 (review) - Suppressed review findings: microsoft#2629 (review) - Merged pull request microsoft#2638: microsoft#2638 - Merged pull request microsoft#2633: microsoft#2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained microsoft#2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make microsoft#2629 engineering-ready on current master after PRs microsoft#2638 and microsoft#2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue microsoft#1667: microsoft#1667 - GitHub pull request microsoft#2629: microsoft#2629 - Automated review overview: microsoft#2629 (review) - Suppressed review findings: microsoft#2629 (review) - Merged pull request microsoft#2638: microsoft#2638 - Merged pull request microsoft#2633: microsoft#2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained microsoft#2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make microsoft#2629 engineering-ready on current master after PRs microsoft#2638 and microsoft#2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue microsoft#1667: microsoft#1667 - GitHub pull request microsoft#2629: microsoft#2629 - Automated review overview: microsoft#2629 (review) - Suppressed review findings: microsoft#2629 (review) - Merged pull request microsoft#2638: microsoft#2638 - Merged pull request microsoft#2633: microsoft#2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained microsoft#2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: handle empty featurized text features AB#1667 ## Summary Treat an empty CountSelector selection as a valid nullable zero-width vector with size metadata. Allow Featurize and TrainRegressor to ignore collapsed text inputs when useful features remain, fail clearly when every input collapses, and cover schema, null, copy, persistence, generated Python, and public end-to-end behavior. Preserve the merged vectorAssemblerHandleInvalid/defaultCopy behavior and add a combined regression for collapsed text with keep mode. ## Prompting Intent Make #2629 engineering-ready on current master after PRs #2638 and #2633 merged. Rebase surgically, preserve both the empty-text fix and all merged missing-value handling behavior, resolve suppressed performance/test-cleanliness findings, avoid RDD or driver-side production paths, validate with JDK 11 plus generated Python and Spark 4.1 compatibility, and refresh exact-head CI and review evidence. ## Linked Sources - GitHub issue #1667: #1667 - GitHub pull request #2629: #2629 - Automated review overview: #2629 (review) - Suppressed review findings: #2629 (review) - Merged pull request #2638: #2638 - Merged pull request #2633: #2633 ## Rationale VectorSlicer rejects an empty index list, so CountSelectorModel takes a narrow empty-selection branch while preserving the existing implementation for non-empty selections. A single immutable sparse vector and direct null check avoid per-row Option allocation. Featurize validates fitted nested selectors because selected width is known only after fitting; zero-width text inputs remain harmless beside useful features, while an all-zero feature set fails with a clear column-specific error. The rebase retained #2638's default skip, keep/error modes, VectorAssembler wiring, and defaultCopy semantics rather than choosing either conflict side wholesale. The implementation remains Scala-first so generated wrappers expose the behavior without duplicating production logic in Python. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: declare the Featurize release prerequisite ## Summary Declare PR #2638 as a release compatibility prerequisite for PR #2629. ## Prompting Intent Make PR #2629 merge-ready after rebasing it onto the latest master, including the Spark 4.1 compatibility replay required by SynapseML CI. ## Linked Sources - PR #2629: #2629 - PR #2638: #2638 ## Rationale The PR #2629 tests intentionally exercise the missing-numeric Featurize API merged by PR #2638. Spark 4.1 has not yet received that change, so replaying only the #2629 delta cannot compile. Declaring the merged commit lets the existing compatibility job apply the exact dependency first and keeps the prerequisite removable after the next release-branch sync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: retire synced compatibility prerequisites ## Summary Remove release compatibility prerequisites already incorporated by the Spark 4.1 sync while retaining PR #2638, which PR #2629 still requires. ## Prompting Intent Make PR #2629 merge-ready after its current-head Azure compatibility job failed, without weakening conflict detection or dropping the Featurize dependency used by its production and test changes. ## Linked Sources - PR #2629: #2629 - PR #2638: #2638 - Spark 4.1 sync #2645: #2645 - Failed Azure build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=231604593 ## Rationale Spark 4.1 already contains port-adapted equivalents of the five older manifest entries, so replaying their original master patches conflicts before CI reaches the PR. PR #2638 is not yet present and remains necessary because #2629 uses its missing-numeric Featurize API and test imports. Keeping only that exact commit matches the successful local replay and preserves the pipeline's existing validation semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: SynapseML CI <synapseml-ci@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Related Issues/PRs
Fixes #736
Depends on merged #2635 and #2632 for release-replay baselines
What changes are proposed in this pull request?
transformSchema.copy, Pipeline fitting, save/load, codegen, and generated Python wrappers.pipeline.yamlunchanged and replay merged feat(core): add precision-recall AUC metric #2635 followed by fix: resolve current model metadata deterministically #2632 on Spark 4.1, matching target history.How was this patch tested?
ComputeModelStatisticsValidationSuite8/8,VerifyComputeModelStatistics32/32, andVerifyComputePerInstanceStatistics5/5.core/compile,core/Test/compile,core/scalastyle, andcore/Test/scalastylepassed.sbt codegenpassed; the generated wrapper passed AST andpy_compilechecks.tools/ci/tests/test_pipeline_yaml.py61/61; the amended prerequisite config format test passed on the final head.test:compilepassed and the focused suite passed 8/8.reason=pullRequest, including Spark 4.1 compatibility and CPU/GPU/Fabric E2E.Compatibility
No public JVM signatures or serialized parameter shapes were removed or changed.
ComputeModelStatistics.copy(ParamMap)retains itsTransformerreturn type and now usesdefaultCopy. Generated Python parameter APIs are unchanged.