fix(train): Handle subscription-only models in recipe selection#5948
Closed
haardm wants to merge 1 commit into
Closed
fix(train): Handle subscription-only models in recipe selection#5948haardm wants to merge 1 commit into
haardm wants to merge 1 commit into
Conversation
Models like nova-textgeneration-micro-v2 have only IsSubscriptionModel
recipes. The primary recipe filter required not IsSubscriptionModel,
causing ValueError when no standard recipe exists.
Fix:
- Fallback to subscription recipe as primary when no standard one exists
- Handle access point ARN URIs in primary recipe download path
- Guard against merging subscription override_params into itself
- Only resolve {customer_id} placeholder when present in URI
Tests:
- subscription_only_model_lora: Micro v2 LoRA case
- subscription_only_model_full: Micro v2 full-rank case
- mixed_recipes_still_prefers_standard: Lite v2 regression guard
Fixes: V2248468914
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix recipe selection for models that have only
IsSubscriptionModelrecipes (e.g.,nova-textgeneration-micro-v2). Previously, the primary recipe filter requirednot IsSubscriptionModel, causingValueError: No recipes foundwhen no standard recipe exists.Fixes
s3://arn:aws:s3:...format){customer_id}resolution — only call STS when the URI contains the placeholderTesting
Unit tests (75 passed, 0 failed)
New test cases:
test__get_fine_tuning_options_subscription_only_model_lora— micro-v2 LoRAtest__get_fine_tuning_options_subscription_only_model_full— micro-v2 full-ranktest__get_fine_tuning_options_mixed_recipes_still_prefers_standard— lite-v2 regressionE2E validation (prod IAD, account 551952248621)
...training-job/nova-textgeneration-micro-v2-sft-20260615191820...training-job/nova-textgeneration-micro-v2-sft-20260615191835...training-job/nova-textgeneration-lite-v2-sft-20260615191850...training-job/nova-textgeneration-lite-v2-sft-20260615191906Related