How Debuggers Work • Sy Brand • GOTO 2025

The primary role of a native code debugger is to translate the low-level, machine-code execution of a process into a human-understandable, source-code level view. This is achieved through a coordinated effort between the debugger, the operating system, the program’s binary file, and the debug information generated by the compiler.

Core Debugging Architecture

The debugging environment consists of several key components that communicate exclusively through the operating system:

• The Debugger: The user-facing tool (e.g., GDB, Visual Studio Debugger) that interprets user commands.
• The Debuggee Process: The running instance of the program, consisting of its executable code and data in memory.
• The Binary File: The on-disk executable which, unlike the running process, contains crucial debug information.
• The Operating System: The intermediary that provides privileged access and control over the debuggee process through system calls.

The debugger’s main task is to use the debug information from the binary to map the state of the running process back to the original source code.

Fundamental Debugger Operations

Memory and Register Manipulation

At the lowest level, a debugger needs to read and write to the memory and registers of the process it’s controlling. This is a privileged operation facilitated by the operating system. On Linux, the primary tool for this is the ptrace system call.

ptrace is a versatile, if complex, API that allows a parent process to observe and control the execution of another process. It supports operations like reading/writing memory (PTRACE_PEEKDATA), accessing registers, and controlling execution flow (e.g., single-stepping).

How Breakpoints Work

Setting a breakpoint on a line of source code involves two main steps:

1. Mapping Source to Machine Code: The debugger consults the debug information, typically in the DWARF format on Linux systems. DWARF contains a line table that maps source file lines and columns to the memory addresses of the corresponding machine instructions.
2. Halting Execution:
   • Software Breakpoints: This is the most common method. The debugger reads the machine instruction at the target address, saves its first byte, and overwrites it with the byte 0xCC. This byte corresponds to the INT 3 instruction on x86/x64 systems—a special, one-byte software interrupt designed for debuggers. When the CPU executes this instruction, it triggers a trap, transferring control to the operating system. The OS then sends a signal (like SIGTRAP) to the process, which the debugger intercepts, effectively pausing the program.
   • Continuing from a Breakpoint: To continue, the debugger must restore the original instruction byte, rewind the instruction pointer back by one, single-step over the original instruction, and then re-insert the 0xCC breakpoint before continuing normal execution. This ensures the breakpoint can be hit again, for example, in a loop.
   • Hardware Breakpoints: These use special-purpose debug registers on the CPU (x86 provides four). They are limited in number but more powerful, as they can be configured to break not just on execution, but also on data reads or writes to a specific memory address.

Watchpoints

Watchpoints, which halt execution when a variable’s memory is read or written, are implemented using hardware breakpoints. The debugger programs one of the hardware debug registers with the memory address of the variable and configures it to trigger a break on the desired access type (read, write, or both). This is invaluable for tracking down memory corruption bugs.

Source-Level Stepping

Stepping through code is built upon lower-level primitives:

• Step In: A simple implementation involves single-stepping through machine instructions and continuing until the debug information indicates that the current source line has changed.
• Step Over: The debugger inspects the next instruction. If it’s a function call, it determines the address of the instruction after the call, sets a temporary breakpoint there, and continues execution. If it’s not a call, it behaves like a “Step In”.
• Step Out: The debugger determines the function’s return address on the stack, sets a temporary breakpoint there, and continues execution until that breakpoint is hit.

Stack Unwinding

To produce a backtrace or inspect variables in a caller’s frame, the debugger must unwind the stack. This process reconstructs the state of previous stack frames.

• Frame Pointers: If the code is compiled with frame pointers, each stack frame contains a pointer to the start of the previous frame, forming a simple linked list that the debugger can walk.
• Call Frame Information (CFI): A more robust method, required when frame pointers are omitted for optimization. CFI is a complex, Turing-complete bytecode language within the DWARF debug information. It provides a set of rules for any given instruction address, describing how to restore register values and find the return address to get back to the state of the caller function.

Reading Variables

To display a variable, the debugger again relies on DWARF information. The Debug Information Entries (DIEs) provide a complete schema for the program’s data:

• They describe the location of a variable (e.g., a static memory address or an offset from a register).
• They define custom types, such as structs, including their size, member names, types, and byte offsets.

The debugger uses this schema to parse the raw bytes from the process’s memory and present a structured, human-readable representation.

Calling a Function

Injecting a function call into a running process is one of the most complex debugger features. It requires adhering to the platform’s Application Binary Interface (ABI), which defines the rules for function calls.

The process involves:

1. Parsing the expression (e.g., my_cat.feed(10)).
2. Using debug info to find the memory address of the my_cat object and the feed function. C++ functions require decoding the mangled name to find the correct symbol.
3. Saving the current state of the CPU registers.
4. Following the ABI’s calling convention to set up the arguments. For a C++ member function on x64, this means placing the object’s this pointer into the RDI register and the integer 10 into the RSI register.
5. Setting the instruction pointer to the address of the feed function.
6. Continuing execution until the function returns.
7. Retrieving the return value from the designated register (e.g., RAX on x64).
8. Restoring the saved CPU registers to return the program to its original state.

• Tags:
• Feature
• Systems Programming
• Debugging
• Compilers
• DWARF
• Ptrace
• Debuggers
• X86-64
• Linux
• Assembly