Extending LLDB to Debug Applications That Span CPUs and GPUs

Meta developed and upstreamed a vendor-neutral accelerator plugin architecture inside lldb-server to debug applications spanning CPUs and GPUs, as detailed in Meta Engineering's report. Modern artificial intelligence workloads frequently move back and forth between native host applications and accelerator kernels. Meta created this framework to separate shared debugger plumbing from hardware-specific device control, presenting both execution environments in one debugging session.

This report examines the lldb-server plugin interface, the dual-target process model, and the runtime coordination protocols that synchronize execution across disparate hardware. It is written for systems engineers, compiler maintainers, and distributed software developers profiling heterogeneous computing applications. Readers will find the concrete implementation details behind the upstream LLVM patches, including independent address spaces, mock hardware testing harnesses, and synchronized cross-device breakpoints.

01Architectural Divergence Across Host and Device

Traditional debuggers assume an active process runs on a single instruction set with a consistent view of threads, registers, and memory. Graphics processing units challenge every part of that model. A host process might run on x86-64 while device kernels run on an entirely distinct instruction set, executing groups of lanes in lockstep rather than as independent operating system threads.

Memory architectures diverge just as sharply across processor boundaries. Host and device pointers reference separate address spaces where identical virtual values, such as address 0x1000, point to different physical memory locations. Attempting to force both units into one synthetic process complicates register handling, memory inspection, and stepping, forcing the debugger to evaluate hardware contexts at every step.

Hardware vendors historically addressed this challenge by developing separate tools such as AMD ROCgdb and NVIDIA CUDA-GDB. While those debuggers provide deep platform support, common capabilities must be implemented and maintained independently across separate toolchains. Meta designed an accelerator plugin framework within LLDB that preserves device boundaries rather than attempting to hide physical architectural differences.

02Server Plugin Interface and Protocol Translation

The LLDB architecture relies on a companion binary called lldb-server to launch programs, attach to workloads, and execute low-level debugging actions. The LLDB client issues commands to this server through the GDB remote serial protocol, which provides controls for reading memory, inspecting registers, and setting breakpoints. Because lldb-server already commands native host processes, it serves as the natural coordination point for accelerator runtimes.

The plugin interface allows hardware backends to operate alongside native CPU debugging infrastructure. The backend translates vendor-specific debugging APIs into concepts LLDB natively understands, including processes, threads, registers, memory, modules, and stop reasons. This separation keeps the LLDB client vendor-neutral, allowing shared enhancements in command parsing, symbol resolution, and development environment integrations to benefit all accelerator hardware equally.

Debugging features incur zero overhead when running pure host workloads. The accelerator debugging path activates only when an inspected application invokes a supported accelerator runtime environment. Applications running exclusively on host processors execute through standard native debugging paths without experiencing latency, memory tracking overhead, or runtime compute penalties.

03Independent Targets Inside a Unified Session

Rather than merging disparate computing units into one artificial process container, the architecture represents the host CPU and the accelerator as separate LLDB targets. Each target maintains its own architecture, execution state, register context, and address space while associating with an independent debugger process. The host processor uses standard native debugging paths, while the plugin provisions a secondary connection to instantiate the accelerator target.

To the underlying debugger, the CPU and accelerator targets remain completely independent entities. To the developer working in a terminal or graphical environment, both execution spaces appear within a single coordinated debugging session. This separation avoids conflating incompatible hardware models while maintaining a unified workflow for engineers tracing bugs across boundaries.

A vector dot-product sample implemented with the HIP runtime illustrates this operational flow. In an abridged session, target zero represents the stopped x86-64 host application while target one tracks the amdgcn kernel. Developers can set pending breakpoints on device routines like compute_element, resume host execution to trigger kernel launches, and switch targets to inspect active kernel backtraces.

04Runtime Breakpoint Handshake and Hardware Simulation

A live debugging session follows a coordinated sequence between host and device components. LLDB connects to lldb-server and identifies available accelerator plugins before the plugin asks LLDB to place internal breakpoints at critical runtime events in the host process. When the host hits one of these points, lldb-server alerts the plugin, which initializes vendor APIs and exposes an accelerator debugging endpoint.

LLDB then instantiates the secondary target and connects directly to the new device endpoint. Accelerator threads, registers, memory maps, and compiled binary modules immediately leverage LLDB's native abstractions through standardized coordination commands. This structured protocol manages host breakpoints, reports loaded modules, and synchronizes cross-device execution without exposing proprietary vendor APIs to the client debugger.

To validate this coordination protocol without physical silicon, Meta engineers built a mock accelerator plugin. This testing component exercises plugin discovery, breakpoint coordination, connection creation, and device state handling within the LLVM test suite. The mock tool allows the wider compiler community to test and maintain shared debugger infrastructure independently of proprietary hardware.

05Upstream LLVM Architecture and Vendor Integration

The framework establishes its architectural boundaries on general debugger operations rather than platform-specific hardware abstractions. This design allows the plugin interface to support diverse hardware families that feature contrasting instruction sets, driver interfaces, and execution models. The framework also supports postmortem analysis by providing shared base classes for core dumps while delegating hardware-specific memory and register parsing to vendor plugins.

Meta already deploys this architecture across its production infrastructure to support a heterogeneous fleet of accelerators running diverse hardware and software stacks. Operating varied hardware backends under one common LLDB abstraction proved that the core debugging layer can remain stable while vendor plugins address device peculiarities. Engineers can diagnose faults across contrasting hardware configurations without switching between specialized vendor tools.

Meta has upstreamed the core framework to the LLVM repository, including the lldb-server plugin interface, GDB-remote packets, multiple address space support, and mock tests. The team plans to upstream the remaining components to streamline vendor onboarding. Under this model, integrating new accelerator hardware requires writing only a vendor plugin rather than maintaining an entirely separate debugger.