Skip to content

Require stable cluster health before advancing a rolling restart - #887

Open
tomach wants to merge 2 commits into
masterfrom
ta/extended-health-check
Open

Require stable cluster health before advancing a rolling restart#887
tomach wants to merge 2 commits into
masterfrom
ta/extended-health-check

Conversation

@tomach

@tomach tomach commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary of changes

The rolling-restart health gate (is_cluster_healthy) decided it was safe to kill the next node from a single snapshot of sys.nodes and sys.health. Node membership becomes visible before the master finishes recomputing cluster health, so there is a short window where the count is back and sys.health still reports a stale GREEN.

Now it requires the same healthy reading to hold across N consecutive checks before returning healthy. Configurable via HEALTH_CHECK_STABILITY_CHECKS (default 3) and HEALTH_CHECK_STABILITY_DELAY (default 2s)

Checklist

  • Link to issue this PR refers to: https://github.com/crate/support/issues/916
  • Relevant changes are reflected in CHANGES.rst
  • Added or changed code is covered by tests
  • Documentation has been updated if necessary
  • Changed code does not contain any breaking changes (or this is a major version change)

@tomach
tomach marked this pull request as ready for review August 18, 2026 09:20
@tomach
tomach requested review from juanpardo and plaharanne August 18, 2026 10:35
Comment thread tests/test_cratedb.py Outdated

@plaharanne plaharanne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added one note about a comment but except that, it looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants