Skip to content

feat(aarch64): add GDB debugging for Linux KVM - #2016

Open
cshung wants to merge 2 commits into
hyperlight-dev:mainfrom
cshung:cshung/kvm-aarch64-gdb
Open

cshung wants to merge 2 commits into
hyperlight-dev:mainfrom
cshung:cshung/kvm-aarch64-gdb

Conversation

@cshung

@cshung cshung commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

ARM64 Linux KVM GDB

KVM guests support register and memory access, two hardware breakpoints,
BRK software breakpoints, and hardware single-step.
KVM reports debug stops through KVM_EXIT_DEBUG.
Debug state survives vCPU reset, and register access respects the active stack pointer.

Dependencies

Requires shared debugger support in #2050. This PR provides KVM integration,
platform enablement, and native GDB CI. The full diff includes the shared
prerequisite.

KVM-only comparison.

Validation

  • Formatting and ARM64 GDB-enabled cross-Clippy pass with default and KVM-only features.
  • Earlier Raspberry Pi 4 validation passed 16 targeted native unit tests and six real GDB client tests.
  • Shared support has native ARM64 MSHV validation in feat(aarch64): add GDB debugging for Linux MSHV #2052.

Pi validation uses a separate, uncommitted ARMv8.0 allocator fix and guests built
with RUSTFLAGS="-C target-feature=-lse". Pi compatibility changes are outside this PR.

Native KVM ARM64 runtime tests were not repeated against the guest startup
ABI in this base.

@cshung cshung added C-enhancement For PRs adding features, improving functionality, docs, tests, etc. S-needs-review PR is ready for (re-)review labels Nov 3, 2026
Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:34
@cshung
cshung marked this pull request as draft November 3, 2026 00:36
@cshung cshung removed the S-needs-review PR is ready for (re-)review label Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical flaws allow guest modification of executable trampoline code and debugger memory corruption, while breakpoint cleanup and snapshot-ordering issues can leave sandboxes unrecoverable.

5 open issues
What changed in this PR

Adds ARM64 guest debugging on Linux/KVM, alongside the shared debugger infrastructure and Windows/WHP support from #1907.

Changes:

  • Supports ARM64 register and memory access, software and hardware breakpoints, and hardware single-step.
  • Adds instruction-cache synchronization, debug-state preservation across reset, and snapshot safeguards.
  • Adds debugger tests and documents platform capabilities.
File Description
src/​hyperlight_host/​src/​sandbox/​uninitialized_evolve.rs Initializes ARM64 debug vectors during VM setup.
src/​hyperlight_host/​src/​sandbox/​snapshot/​memory/​backing.rs Exposes backing ranges for cache synchronization.
src/​hyperlight_host/​src/​sandbox/​initialized.rs Guards snapshots and restores against active software breakpoints.
src/​hyperlight_host/​src/​mem/​mgr.rs Exposes checked host memory ranges to the debugger.
src/​hyperlight_host/​src/​mem/​exe.rs Carries debug-vector addresses in load metadata.
src/​hyperlight_host/​src/​mem/​elf.rs Finds and relocates the debug-vector symbol.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​whp/​x86_64.rs Adapts address translation to the shared interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​whp/​aarch64.rs Implements WHP debugging, cache synchronization, and tests.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mshv/​x86_64.rs Adapts address translation to the shared interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​mod.rs Adds ARM64 debug exits and initialization errors.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​kvm/​x86_64.rs Adapts address translation to the shared interface.
src/​hyperlight_host/​src/​hypervisor/​virtual_machine/​kvm/​aarch64.rs Implements KVM debugging and active-stack-pointer handling.
src/​hyperlight_host/​src/​hypervisor/​regs/​aarch64/​whp_reg.rs Defines WHP debug register identifiers.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​x86_64.rs Uses shared debugger handling and breakpoint storage.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​mod.rs Integrates ARM64 stops, stepping, and mapping safeguards.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​debug.rs Provides shared requests and software-breakpoint lifecycle handling.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​aarch64.rs Connects ARM64 VM setup and execution to GDB.
src/​hyperlight_host/​src/​hypervisor/​gdb/​target.rs Adds ARM64 register conversion to the shared target.
src/​hyperlight_host/​src/​hypervisor/​gdb/​mod.rs Adds page-table translation and cache-flush support.
src/​hyperlight_host/​src/​hypervisor/​gdb/​event_loop.rs Reports ARM64 initial stops to GDB.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch/​x86_64.rs Houses x86 breakpoint constants and stop classification.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch/​aarch64.rs Defines ARM64 software breakpoints and alignment checks.
src/​hyperlight_host/​src/​hypervisor/​gdb/​arch.rs Selects architecture-specific debugger helpers.
src/​hyperlight_host/​examples/​guest-debugging/​main.rs Adds ARM64 breakpoint, stepping, and code-write integration tests.
src/​hyperlight_host/​build.rs Enables GDB for supported ARM64 platforms.
src/​hyperlight_guest_bin/​src/​arch/​aarch64/​exception/​entry.rs Exports debug vectors and forwards synchronous exceptions.
src/​hyperlight_common/​src/​arch/​aarch64/​exn.rs Defines the shared debug-exception HVC immediate.
docs/​how-to-debug-a-hyperlight-guest.md Documents ARM64 capabilities and guest-symbol requirements.
CHANGELOG.md Records Linux/KVM and Windows/WHP ARM64 debugging support.

Review effort: Balanced

@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from a60be9b to 55da94f Compare November 3, 2026 01:58
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 55da94f to 2418379 Compare November 3, 2026 19:28
@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from a26c2d9 to 11361ca Compare November 4, 2026 16:37
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 11361ca to 4849110 Compare November 5, 2026 17:24
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch 2 times, most recently from 3f280ae to 859ef39 Compare November 12, 2026 00:12
@hyperlight-gh-bot

This comment has been minimized.

@cshung
cshung marked this pull request as ready for review November 12, 2026 02:56
@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 859ef39 to 68a8704 Compare November 12, 2026 18:15
@hyperlight-gh-bot

This comment has been minimized.

@yoshuawuyts yoshuawuyts added O-aarch64 Specific to 64-bit Arm processors O-kvm Linux KVM backend O-linux Linux host support S-needs-review PR is ready for (re-)review labels Nov 16, 2026
@cshung
cshung force-pushed the cshung/kvm-aarch64-gdb branch from 68a8704 to 45810c6 Compare November 16, 2026 18:16
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: 45810c6b19e3
Baseline commit: 5fe5c8f8c44f

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 758.56 ns (➖ 1.00x faster)
vec_bytes 600.40 ns (➖ 1.00x slower)
372.95 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 570.15 ns (➖ 1.02x slower)
65536 141.26 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 13.20 ms (➖ 1.00x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
7.68 ns (➖ 1.00x faster) 7.67 ns (➖ 1.00x faster) 7.69 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 96.65 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.08 µs (➖ 1.03x faster) 7.08 µs (➖ 1.28x faster)
65536 1.91 µs (➖ 1.11x faster) 2.05 µs (➖ 1.07x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 28.20 µs (➖ 1.03x slower)
65536 6.13 µs (➖ 1.03x faster) 6.45 µs (➖ 1.01x slower)
8192 1.08 µs (➖ 1.04x slower) 1.05 µs (➖ 1.01x faster)
kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 763.48 ns (➖ 1.01x slower)
vec_bytes 614.49 ns (➖ 1.05x slower)
715.82 µs (➖ 1.03x faster)

payload_allocation

slot_pool_segmented
262144 566.82 ns (➖ 1.00x slower)
65536 194.64 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 15.13 ms (➖ 1.01x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
7.71 ns (➖ 1.01x slower) 7.71 ns (➖ 1.01x slower) 7.70 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 55.91 µs (➖ 1.03x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 8.55 µs (➖ 1.00x faster) 8.59 µs (➖ 1.01x slower)
65536 2.40 µs (➖ 1.01x slower) 2.40 µs (➖ 1.00x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 32.61 µs (➖ 1.09x faster)
65536 7.95 µs (➖ 1.02x faster) 7.75 µs (➖ 1.02x faster)
8192 809.20 ns (➖ 1.01x slower) 813.01 ns (➖ 1.02x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 898.57 ns (➖ 1.02x faster)
vec_bytes 699.79 ns (➖ 1.00x faster)
320.51 µs (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
262144 680.02 ns (➖ 1.01x slower)
65536 182.24 ns (➖ 1.00x faster)

sandboxes

create_initialized_and_drop
medium 68.60 ms (➖ 1.00x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
9.86 ns (➖ 1.00x faster) 9.87 ns (➖ 1.07x faster) 9.87 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 106.36 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 10.43 µs (➖ 1.01x slower) 9.29 µs (➖ 1.06x faster)
65536 2.47 µs (➖ 1.03x faster) 2.65 µs (➖ 1.04x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 35.03 µs (➖ 1.01x faster)
65536 7.94 µs (➖ 1.02x faster) 7.93 µs (➖ 1.07x faster)
8192 1.29 µs (➖ 1.00x faster) 1.30 µs (➖ 1.01x slower)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 922.87 ns (➖ 1.00x slower)
vec_bytes 659.82 ns (➖ 1.00x faster)
707.18 µs (➖ 1.04x slower)

payload_allocation

slot_pool_segmented
262144 665.27 ns (➖ 1.00x slower)
65536 257.06 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 82.16 ms (➖ 1.09x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
8.72 ns (➖ 1.00x slower) 8.92 ns (➖ 1.03x slower) 9.57 ns (➖ 1.10x slower)

snapshot_files

load_snapshot_unverified
small 54.12 µs (➖ 1.02x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.94 µs (➖ 1.01x slower) 7.90 µs (➖ 1.03x faster)
65536 2.27 µs (➖ 1.00x slower) 2.27 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 39.37 µs (➖ 1.01x slower)
65536 7.47 µs (➖ 1.01x faster) 7.52 µs (➖ 1.00x faster)
8192 898.22 ns (➖ 1.02x faster) 916.69 ns (➖ 1.00x faster)
hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.30 µs (➖ 1.06x slower)
vec_bytes 774.30 ns (➖ 1.01x faster)
2.06 ms (➖ 1.03x slower)

payload_allocation

slot_pool_segmented
262144 781.49 ns (➖ 1.01x faster)
65536 233.63 ns (➖ 1.02x slower)

sandboxes

create_initialized_and_drop
medium 13.16 ms (➖ 1.03x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
10.04 ns (➖ 1.00x faster) 10.00 ns (➖ 1.00x slower) 10.18 ns (➖ 1.02x slower)

snapshot_files

load_snapshot_unverified
small 800.53 µs (➖ 1.05x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 9.19 µs (➖ 1.02x faster) 9.05 µs (➖ 1.05x faster)
65536 2.34 µs (➖ 1.01x faster) 2.34 µs (➖ 1.01x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 43.45 µs (➖ 1.09x faster)
65536 8.90 µs (➖ 1.14x faster) 9.45 µs (➖ 1.08x slower)
8192 1.27 µs (➖ 1.03x faster) 1.30 µs (➖ 1.04x slower)
hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.25 µs (➖ 1.04x slower)
vec_bytes 772.06 ns (➖ 1.01x slower)
3.69 ms (➖ 1.01x slower)

payload_allocation

slot_pool_segmented
262144 795.85 ns (➖ 1.08x slower)
65536 212.57 ns (➖ 1.02x slower)

sandboxes

create_initialized_and_drop
medium 14.58 ms (➖ 1.01x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
9.88 ns (➖ 1.02x slower) 10.06 ns (➖ 1.00x faster) 9.94 ns (➖ 1.03x slower)

snapshot_files

load_snapshot_unverified
small 670.83 µs (➖ 1.05x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.70 µs (➖ 1.00x slower) 7.45 µs (➖ 1.03x faster)
65536 2.31 µs (➖ 1.00x slower) 2.30 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 48.48 µs (➖ 1.04x slower)
65536 8.38 µs (➖ 1.02x slower) 8.35 µs (➖ 1.03x faster)
8192 1.04 µs (➖ 1.14x faster) 1.12 µs (➖ 1.05x slower)

Reported by cargo ci bench-report --candidate run:43927116195 --config-file bench_report.toml.

cshung and others added 2 commits November 19, 2026 19:11
Architecture adapters, software breakpoints, page-sized memory access and recovery handling are shared by backend layers. Guest debug vectors and common tests support ARM64. Backend layers control platform enablement.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>
KVM debug exits provide hardware breakpoints and single-step. Register access respects the active stack pointer. Native ARM64 GDB tests run in debug CI. Shared debugger behavior belongs to the parent commit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-enhancement For PRs adding features, improving functionality, docs, tests, etc. O-aarch64 Specific to 64-bit Arm processors O-kvm Linux KVM backend O-linux Linux host support S-needs-review PR is ready for (re-)review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants