Avoid checkpoint work on skipped steps

[SujeethJinesh]

Description

In the Pathways profiler for train.py, we see that the checkpointing logic appears to take the majority of time in a trace for Pathways, but this is due to the maybe_save_checkpoint function misclassifying it as doing work right now. This makes reading the profile harder and is a major red herring for debugging.

With this fix, it shows a more reasonable amount of work performed by the Pathways CPU node (screenshot).

This should also help avoid some of the unnecessary checkpointing related logic for other paths as well.

FIXES: b/530256100

Tests

Tested on internal cluster (details in b/529621972#comment6)

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!