[fix](routineload) Persist the current load definition - #66634
Draft
0AyanamiRei wants to merge 4 commits into
Draft
[fix](routineload) Persist the current load definition#666340AyanamiRei wants to merge 4 commits into
0AyanamiRei wants to merge 4 commits into
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64878 Problem Summary: Routine Load image recovery reparsed the immutable original CREATE statement, so CREATE semantics changed by ALTER were not represented in the image. Persist the current load definition, retain the original statement as the legacy-image fallback, journal altered load clauses, and validate failure-prone Kafka and Kinesis changes before mutating runtime state. ### Release note Routine Load jobs now recover the current effective load definition after ALTER. ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.load.routineload.KafkaRoutineLoadJobTest,org.apache.doris.load.routineload.KinesisRoutineLoadJobTest,org.apache.doris.persist.AlterRoutineLoadOperationLogTest - Behavior changed: Yes. Image recovery uses the current effective Routine Load definition while legacy images continue to use origStmt. - Does this need documentation: No
0AyanamiRei
requested review from
gavinchou,
liaoxin01 and
mymeiyi
as code owners
August 11, 2026 02:21
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64878 Problem Summary: Keep the original CREATE statement in new images so an older FE can ignore the new load definition field and use its existing recovery path. ALTERed load clauses are outside the downgrade compatibility guarantee and may not survive rollback. ### Release note Document that Routine Load ALTER semantics are not guaranteed after rolling back to an older FE. ### Check List (For Author) - Test: Not run (per request; compatibility coverage was added) - Behavior changed: No. This records and tests the intended rollback compatibility boundary. - Does this need documentation: Yes. The rollback limitation must be called out in the feature documentation.
0AyanamiRei
marked this pull request as draft
August 11, 2026 03:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #64878
Problem Summary:
This is Part 1 of the Routine Load target-table ALTER work. It contains only the persistence and recovery prerequisite; it does not add target-table ALTER syntax, fields, or behavior.
RoutineLoadJob currently persists origStmt from CREATE and reparses it during image recovery. Because origStmt never changes, load clauses and job/data-source properties changed by ALTER are not represented by that recovery source.
This PR:
Rollback compatibility: an older FE ignores the unknown LoadDefinition field and follows its existing origStmt recovery path. ALTER semantics applied by the new FE are outside this downgrade guarantee. After rollback, an altered Routine Load job may recover its original load clauses or be cancelled during validation and may need to be recreated.
Release note
Routine Load jobs now recover the current effective load definition after ALTER. If ALTER ROUTINE LOAD is executed after upgrading, rolling back to an earlier FE version may cause the altered job to lose the altered load clauses or be cancelled during image recovery.
Check List (For Author)