Repository navigation
Don't send an evacuated VM back to its last host - #14743
Open
bhouse-nexthop wants to merge 1 commit into
Open
bhouse-nexthop wants to merge 1 commit into
bhouse-nexthop wants to merge 1 commit into
Conversation
When a host is put into maintenance, or HA moves its VMs away, each VM went back to its last host whenever that host was Up and had capacity. That skipped the planner and the VM allocation algorithm. Every migration records its source as the VM's last host, so during a rolling maintenance the VMs piled up on the hosts evacuated earlier. With memory overcommit, those hosts could take far more VM memory than they have. deployInVmLastHost() no longer tries the last host for a migration plan. Only the two host-evacuation paths mark a plan as one: VirtualMachineManagerImpl.orchestrateMigrateAway() and ResourceManagerImpl.migrateAwayVmWithVolumes(). Starting a stopped VM still prefers its last host. Signed-off-by: Brad House <bhouse@nexthop.ai>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.22 #14743 +/- ##
=========================================
Coverage 18.32% 18.32%
- Complexity 16644 16651 +7
=========================================
Files 5939 5939
Lines 537787 537790 +3
Branches 65932 65930 -2
=========================================
+ Hits 98544 98561 +17
+ Misses 428033 428020 -13
+ Partials 11210 11209 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Collaborator
Author
|
Tested in a lab: a nested KVM cluster of three hosts, with Setup:
A start still uses the last host as before, which the new unit tests cover. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When a host is put into maintenance, or HA moves VMs off a host, each VM went back to its last host whenever that host was Up and had capacity, skipping the planner and
vm.allocation.algorithm. Every migration records its source as the VM's last host, so in a rolling maintenance the VMs piled up on the hosts evacuated earlier. With memory overcommit, the capacity check let those hosts take far more VM memory than they have. In our case one host ended up with about 2.3 TiB of VM memory on 1.1 TiB of RAM, and VMs were OOM-killed there.deployInVmLastHost()no longer tries the last host for a migration plan, so the planner and allocators choose the destination.Fixes #14742
Changes
New
DeploymentPlanningManagerImpl.shouldConsiderLastHost(). It returnsfalsefor a plan markedisMigrationPlan(), then applies the existing checks: no last host,ConsiderLastHost=false, an HA tag.Only the two host-evacuation paths mark a plan as a migration plan, and nothing read the flag until now:
VirtualMachineManagerImpl.orchestrateMigrateAway(), used by host maintenance and HA, including its HA planner fallback;ResourceManagerImpl.migrateAwayVmWithVolumes(), used by host maintenance for VMs on local storage.The flag was added for KVM rolling maintenance ([KVM] Rolling maintenance #3610).
A user-initiated migration without a destination host already ignores the last host:
UserVmManagerImpl.chooseVmMigrationDestination()clears it before planning.Not changed here
migrate.vm.across.clusters=false,avoidOtherClustersForDeploymentIfMigrationDisabled()still limits the plan to the last host's cluster. That is separate from choosing the host.findHostAndMigrate(), which moves a VM to scale it, doesn't mark its plan as a migration plan, so it keeps its current behavior. Scaling doesn't evacuate a host.Types of changes
Feature/Enhancement Scale or Bug Severity
Bug Severity
How Has This Been Tested?
DeploymentPlanningManagerImplTest:ConsiderLastHost=true;ConsiderLastHost=falseor an HA tag still turn it off.deployInVmLastHost()(Last host [...] of VM [...] is UP and has enough capacity), not by the allocator.How did you try to break this feature and the system with this change?
migrateAway()retries with the HA planner. That retry is still a migration plan, so it doesn't fall back to the last host.