Skip to content

Don't send an evacuated VM back to its last host - #14743

Open
bhouse-nexthop wants to merge 1 commit into
apache:4.22from
bhouse-nexthop:fix-migration-plan-ignore-last-host
Open

bhouse-nexthop wants to merge 1 commit into
apache:4.22from
bhouse-nexthop:fix-migration-plan-ignore-last-host

Conversation

@bhouse-nexthop

Copy link
Copy Markdown
Collaborator

Description

When a host is put into maintenance, or HA moves VMs off a host, each VM went back to its last host whenever that host was Up and had capacity, skipping the planner and vm.allocation.algorithm. Every migration records its source as the VM's last host, so in a rolling maintenance the VMs piled up on the hosts evacuated earlier. With memory overcommit, the capacity check let those hosts take far more VM memory than they have. In our case one host ended up with about 2.3 TiB of VM memory on 1.1 TiB of RAM, and VMs were OOM-killed there.

deployInVmLastHost() no longer tries the last host for a migration plan, so the planner and allocators choose the destination.

Fixes #14742

Changes

  • New DeploymentPlanningManagerImpl.shouldConsiderLastHost(). It returns false for a plan marked isMigrationPlan(), then applies the existing checks: no last host, ConsiderLastHost=false, an HA tag.

  • Only the two host-evacuation paths mark a plan as a migration plan, and nothing read the flag until now:

    • VirtualMachineManagerImpl.orchestrateMigrateAway(), used by host maintenance and HA, including its HA planner fallback;
    • ResourceManagerImpl.migrateAwayVmWithVolumes(), used by host maintenance for VMs on local storage.

    The flag was added for KVM rolling maintenance ([KVM] Rolling maintenance #3610).

A user-initiated migration without a destination host already ignores the last host: UserVmManagerImpl.chooseVmMigrationDestination() clears it before planning.

Not changed here

  • The last host is still recorded on migration and still preferred when a stopped VM is started.
  • The source host is still excluded.
  • With migrate.vm.across.clusters=false, avoidOtherClustersForDeploymentIfMigrationDisabled() still limits the plan to the last host's cluster. That is separate from choosing the host.
  • findHostAndMigrate(), which moves a VM to scale it, doesn't mark its plan as a migration plan, so it keeps its current behavior. Scaling doesn't evacuate a host.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

  • New unit tests in DeploymentPlanningManagerImplTest:
    • a start with a last host considers it;
    • a migration plan never does, even with ConsiderLastHost=true;
    • no last host, ConsiderLastHost=false or an HA tag still turn it off.
  • The cause was found on a 4.22 cluster of 12 KVM hosts during a rolling maintenance. In the management server's log for that day, 376 of 447 migrations off hosts in maintenance were placed by deployInVmLastHost() (Last host [...] of VM [...] is UP and has enough capacity), not by the allocator.
  • I'll add a test report from a live rolling maintenance.

How did you try to break this feature and the system with this change?

  • When no other host fits, planning still fails as before, and migrateAway() retries with the HA planner. That retry is still a migration plan, so it doesn't fall back to the last host.
  • Starting a stopped VM, and HA restarting a VM after its host fails, don't use a migration plan, so they keep preferring the last host.

When a host is put into maintenance, or HA moves its VMs away, each VM
went back to its last host whenever that host was Up and had capacity.
That skipped the planner and the VM allocation algorithm. Every
migration records its source as the VM's last host, so during a rolling
maintenance the VMs piled up on the hosts evacuated earlier. With
memory overcommit, those hosts could take far more VM memory than they
have.

deployInVmLastHost() no longer tries the last host for a migration
plan. Only the two host-evacuation paths mark a plan as one:
VirtualMachineManagerImpl.orchestrateMigrateAway() and
ResourceManagerImpl.migrateAwayVmWithVolumes(). Starting a stopped VM
still prefers its last host.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (b56bbc2) to head (3ba3731).

Files with missing lines Patch % Lines
...om/cloud/deploy/DeploymentPlanningManagerImpl.java 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               4.22   #14743   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
- Complexity    16644    16651    +7     
=========================================
  Files          5939     5939           
  Lines        537787   537790    +3     
  Branches      65932    65930    -2     
=========================================
+ Hits          98544    98561   +17     
+ Misses       428033   428020   -13     
+ Partials      11210    11209    -1     
Flag Coverage Δ
uitests 4.08% <ø> (ø)
unittests 19.41% <87.50%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

Tested in a lab: a nested KVM cluster of three hosts, with vm.allocation.algorithm=firstfitleastconsumed. The lab runs a main-based (24.0) build with this PR cherry-picked, not the 4.22 branch itself. The code this PR changes is the same on both.

Setup:

  1. Three VMs started on kvm1, then kvm1 put into maintenance with kvm3 disabled, so they migrate to kvm2. Their last host is now kvm1.
  2. kvm3 re-enabled. Four filler VMs on kvm1, so kvm1 is the most consumed host (kvm1 8192 MiB allocated, kvm2 3072, kvm3 512).
  3. kvm2 put into maintenance.
Before this PR With this PR
Where kvm2's VMs went all back to kvm1, the most consumed host all to kvm3, the least consumed
Planner log Last host [... kvm1 ...] ... is UP and has enough capacity for each no last-host lines; Found destination ... Host(3) for each
kvm1 allocated memory afterwards 8192 → 11264 MiB 8192 MiB (unchanged)
kvm3 allocated memory afterwards 512 MiB (unchanged) 512 → 3584 MiB

A start still uses the last host as before, which the new unit tests cover.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant