Skip to content

[fix](routineload) Persist the current load definition - #66634

Draft
0AyanamiRei wants to merge 4 commits into
apache:masterfrom
0AyanamiRei:feature/routine-load-definition-persistence
Draft

[fix](routineload) Persist the current load definition#66634
0AyanamiRei wants to merge 4 commits into
apache:masterfrom
0AyanamiRei:feature/routine-load-definition-persistence

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: #64878

Problem Summary:

This is Part 1 of the Routine Load target-table ALTER work. It contains only the persistence and recovery prerequisite; it does not add target-table ALTER syntax, fields, or behavior.

RoutineLoadJob currently persists origStmt from CREATE and reparses it during image recovery. Because origStmt never changes, load clauses and job/data-source properties changed by ALTER are not represented by that recovery source.

This PR:

  • persists a LoadDefinition snapshot containing the current RoutineLoadDesc, job properties, and data-source properties;
  • reconstructs and validates CREATE semantics from LoadDefinition for new images, while retaining origStmt both for reading old images and for older FEs reading new images;
  • carries only changed RoutineLoadDesc clauses in ALTER edit logs, then reconstructs the complete definition from runtime state during apply/replay;
  • validates failure-prone Kafka and Kinesis ALTER changes before mutating runtime fields, preserving the invariant that LoadDefinition matches the effective runtime CREATE semantics;
  • adds image, legacy compatibility, edit-log compatibility, and failed-ALTER atomicity unit tests.

Rollback compatibility: an older FE ignores the unknown LoadDefinition field and follows its existing origStmt recovery path. ALTER semantics applied by the new FE are outside this downgrade guarantee. After rollback, an altered Routine Load job may recover its original load clauses or be cancelled during validation and may need to be recreated.

Release note

Routine Load jobs now recover the current effective load definition after ALTER. If ALTER ROUTINE LOAD is executed after upgrading, rolling back to an earlier FE version may cause the altered job to lose the altered load clauses or be cancelled during image recovery.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test
    • No need to test or manual test
  • Behavior changed:
    • No.
    • Yes. New images restore the current effective Routine Load definition; legacy images continue to recover from origStmt.
  • Does this need documentation?
    • No.
    • Yes. Document the rollback limitation for jobs altered after upgrade.

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#64878

Problem Summary: Routine Load image recovery reparsed the immutable original CREATE statement, so CREATE semantics changed by ALTER were not represented in the image. Persist the current load definition, retain the original statement as the legacy-image fallback, journal altered load clauses, and validate failure-prone Kafka and Kinesis changes before mutating runtime state.

### Release note

Routine Load jobs now recover the current effective load definition after ALTER.

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.load.routineload.KafkaRoutineLoadJobTest,org.apache.doris.load.routineload.KinesisRoutineLoadJobTest,org.apache.doris.persist.AlterRoutineLoadOperationLogTest
- Behavior changed: Yes. Image recovery uses the current effective Routine Load definition while legacy images continue to use origStmt.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#64878

Problem Summary: Keep the original CREATE statement in new images so an older FE can ignore the new load definition field and use its existing recovery path. ALTERed load clauses are outside the downgrade compatibility guarantee and may not survive rollback.

### Release note

Document that Routine Load ALTER semantics are not guaranteed after rolling back to an older FE.

### Check List (For Author)

- Test: Not run (per request; compatibility coverage was added)
- Behavior changed: No. This records and tests the intended rollback compatibility boundary.
- Does this need documentation: Yes. The rollback limitation must be called out in the feature documentation.
@0AyanamiRei
0AyanamiRei marked this pull request as draft August 11, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants