fix(vm): start VM after restore for all run policies#2577
Open
danilrwx wants to merge 5 commits into
Open
Conversation
danilrwx
force-pushed
the
fix/vm-restore-stuck-stopped
branch
from
e02477a to
5f6d44b
Compare
After a restore the VM is stopped during the maintenance window (its KVVMI and KVVM are deleted). Bringing it back up relied on a side effect: the recreated KVVM getting RunStrategy=Always at creation time. That path is racy — if the first post-restore KVVMI does not survive (e.g. a broken first boot after the root->non-root virt-launcher migration) the run strategy is already flipped to Manual and nothing starts the VM again, leaving it stuck in Stopped even for AlwaysOnUnlessStoppedManually. Make the restart intent explicit and independent of KVVM recreation timing: capture whether the VM was running before entering the restore maintenance window and store it as an annotation on the VirtualMachine (survives KVVM deletion). The power-state handler consumes this intent via checkNeedStartVM for all run policies and clears it once the VM has been started. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
EnterMaintenance records the start-requested-after-restore intent on the live VM, but the ProcessRestore step overwrites vm.Annotations wholesale with the annotations from the snapshot template, dropping the intent. As a result the power-state handler never saw it and the VM stayed stopped after restore (reproduced deterministically for the Manual run policy). Preserve the intent annotation across the annotation overwrite in ProcessRestore, same as the AnnVMOPRestore marker. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Symmetrically to the start-after-restore intent, capture the stopped power state before restore and honor it. The maintenance window deletes the KVVM, dropping the implicit "stopped manually" run-strategy state; recreating the KVVM for AlwaysOnUnlessStoppedManually would otherwise start the VM (RunStrategy=Always on create), violating the "unless stopped manually" contract. EnterMaintenance now records keep-stopped-after-restore for a stopped VM, ProcessRestore preserves it across the annotation overwrite (like the start intent), and createKVVM forces RunStrategy=Manual for AlwaysOnUnlessStoppedManually so the VM stays stopped. Normal VM creation without the intent keeps RunStrategy=Always and starts as before. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Replace the two boolean annotations (start-requested-after-restore and
keep-stopped-after-restore) with a single restore-power-state annotation whose
value is a MachinePhase string ("Running" or "Stopped"). The two intents are
mutually exclusive, so one key makes that explicit by construction, drops a
constant, and simplifies the ProcessRestore preservation to a single key.
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Instead of teaching the power-state handler to read the VM-level intent and clear it, convert it on the freshly created KVVM: "Running" becomes the regular AnnVMStartRequested annotation (reusing the existing start + retry machinery), "Stopped" overrides RunStrategy to Manual for AlwaysOnUnlessStoppedManually. The power-state handler is left untouched, and all consumption lives in one place. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
danilrwx
force-pushed
the
fix/vm-restore-stuck-stopped
branch
from
5690331 to
5dfcef3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
During a restore the VM is stopped inside the maintenance window: its KVVMI and then the KVVM itself are deleted. Bringing the VM back up afterwards relied on an implicit side effect — the recreated KVVM getting
RunStrategy=Alwaysat creation time. That path is racy and only covers the always-on policies.This PR makes the post-restore start intent explicit and independent of KVVM recreation timing:
virtualization.deckhouse.io/start-requested-after-restoreon theVirtualMachine. It is stored on the VM (not the KVVM, which is deleted during maintenance), so it survives KVVM recreation.EnterMaintenancestep captures whether the VM wasRunning/Pendingbefore restore stops it and sets the annotation.SyncPowerStateHandler.checkNeedStartVMconsumes the intent for all run policies (Manual,AlwaysOn,AlwaysOnUnlessStoppedManually) oncekvvmi == niland the configuration is applied.start()clears the annotation after the VM has been started, so it does not re-trigger on a later manual stop.Why do we need it, and what problem does it solve?
A floating bug was observed on version upgrades (perekat): a VM with
runPolicy: AlwaysOnUnlessStoppedManuallysometimes stayedStoppedafter a restore, even though the snapshot was taken while the VM wasRunning.Root cause: the VM restart after restore depended on the recreated KVVM receiving
RunStrategy=Always. The power-state handler flipsAlways -> Manualas soon as the first KVVMI appears. If that first KVVMI does not survive (e.g. the first boot breaks right after the root -> non-root virt-launcher migration, when disks/PVCs created under root are not accessible to the non-root launcher), the run strategy is alreadyManual, there is no KVVMI, and nothing starts the VM again — it stays stuck inStopped.By making the start intent explicit and persistent on the VM, the VM is reliably brought back to
Runningafter restore for every run policy, regardless of KVVM recreation timing or a failed first boot (the intent waits until the configuration is applied and is retried by the existing power-state machinery).What is the expected result?
runPolicy: AlwaysOnUnlessStoppedManually, wait forRunning.VirtualMachineSnapshotwhile the VM isRunning.VirtualMachineOperationof typeRestore(repeatedly / under controller churn / across an upgrade).Running(previously it could remainStopped).Stoppedbefore restore staysStopped(intent is only captured for running VMs;AlwaysOffis unaffected).Checklist
Changelog entries