Skip to content

[AMORO-4139] Delete resource and optimizer atomically when releasing optimizer - #4310

Open
seoyoniee wants to merge 1 commit into
apache:masterfrom
seoyoniee:AMORO-4139-release-optimizer-atomic
Open

[AMORO-4139] Delete resource and optimizer atomically when releasing optimizer#4310
seoyoniee wants to merge 1 commit into
apache:masterfrom
seoyoniee:AMORO-4139-release-optimizer-atomic

Conversation

@seoyoniee

Copy link
Copy Markdown

Why are the changes needed?

Close #4139

OptimizerController#releaseOptimizer deleted the resource record and the optimizer records through two independent transactions:

  optimizerManager.deleteResource(resourceId);                                                                                                                                                                                           
  optimizerManager.deleteOptimizer(resource.getGroupName(), resourceId);                              

Both records describe the same optimizer, so deleting them in separate transaction can leave the resource gone while its optimizer rows survive - a state the release API can no longer recover from.

If the first call committed and the second one failed, the resource row was deleted while the matching optimizer rows survived. That state cannot be recovered through the API:

  • getResource(resourceId) returns null on the next release attempt, so resource.getProperties().putAll(...) throws a NullPointerException.
  • The container startup stats (yarn-application-id, the Kubernetes namespace and name, ...) are persisted only in the resource properties, so once that row is gone the underlying optimizer process cannot be released either.

Brief change log

  • OptimizerManager: rename deleteOptimizer(String, String) to
    deleteOptimizerAndResource(String, String), since the operation now owns both
    records. OptimizerController#releaseOptimizer was its only caller;
    DefaultOptimizingService#deleteOptimizer is a separate method and is left untouched.
  • DefaultOptimizerManager#deleteOptimizerAndResource: run the resource deletion and
    the optimizer deletion inside a single doAsTransaction(...). The existing deletion
    logic is unchanged, only wrapped. Because NestedSqlSession keeps one SQL session per
    thread, the nested doAs commits become no-ops and the outermost transaction commits
    or rolls back both deletions together.
  • OptimizerController#releaseOptimizer: replace the two calls with the single
    transactional one.

How was this patch tested?

Should I add a TestDefaultOptimizerManager to this PR?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@github-actions github-actions Bot added the module:ams-server Ams server module label Aug 4, 2026

@johntomcat7408-cmyk johntomcat7408-cmyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. I traced the transaction flow through PersistentBase and NestedSqlSession: the outer doAsTransaction keeps one thread-local SQL session, so the nested deleteResource/getAs/unregisterOptimizer calls do not commit independently; this is a sound way to prevent future partial deletion. I do think this PR should add regression coverage before merge: (1) the success path deletes the resource and every optimizer row for the resource ID, and (2) a failure during optimizer deletion rolls back the preceding resource deletion, proving the intended atomicity rather than only the final happy-path state. There is also one scope point to clarify. The existing inconsistent state described in #4139 is not recovered by this patch: OptimizerController still dereferences resource.getProperties() immediately after getResource(resourceId), so a missing resource row fails before deleteOptimizerAndResource is reached. If container startup-only properties make terminating the underlying process impossible, could the PR define and test a graceful non-NPE behavior for that state (and document any cleanup limitation), or explicitly agree with maintainers that recovery will be a separate follow-up? Finally, the group argument is still unused; since the API is already being renamed, a resourceId-only signature would be clearer unless group is intentionally retained for future validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: NPE when releasing optimizer due to non-atomic deleteResource + deleteOptimizer

2 participants