DRGN(1) | drgn | DRGN(1) |
drgn - drgn 0.0.25
drgn (pronounced "dragon") is a debugger with an emphasis on programmability. drgn exposes the types and variables in a program for easy, expressive scripting in Python. For example, you can debug the Linux kernel:
>>> from drgn.helpers.linux import list_for_each_entry >>> for mod in list_for_each_entry('struct module', ... prog['modules'].address_of_(), ... 'list'): ... if mod.refcnt.counter > 10: ... print(mod.name) ... (char [56])"snd" (char [56])"evdev" (char [56])"i915"
Although other debuggers like GDB have scripting support, drgn aims to make scripting as natural as possible so that debugging feels like coding. This makes it well-suited for introspecting the complex, inter-connected state in large programs.
Additionally, drgn is designed as a library that can be used to build debugging and introspection tools; see the official tools.
drgn was developed at Meta for debugging the Linux kernel (as an alternative to the crash utility), but it can also debug userspace programs written in C. C++ support is in progress.
In addition to the main Python API, an experimental C library, libdrgn, is also available.
See the Installation instructions. Then, start with the User Guide.
Copyright (c) Meta Platforms, Inc. and affiliates.
drgn is licensed under the LGPLv2.1 or later.
drgn is named after this because dragons eat dwarves.
There are several options for installing drgn.
drgn depends on:
It optionally depends on:
The build requires:
Building from the Git repository (rather than a release tarball) additionally requires:
drgn can be installed using the package manager on some Linux distributions.
$ sudo dnf install drgn
Enable EPEL. Then:
$ sudo dnf install drgn
Install the drgn package from the AUR.
$ sudo apt install python3-drgn
$ sudo zypper install python3-drgn
Enable the michel-slm/kernel-utils PPA. Then:
$ sudo apt install python3-drgn
If your Linux distribution doesn't package the latest release of drgn, you can install it with pip.
First, install pip. Then, run:
$ sudo pip3 install drgn
This will install a binary wheel by default. If you get a build error, then pip wasn't able to use the binary wheel. Install the dependencies listed below and try again.
Note that RHEL/CentOS 6, Debian Stretch, Ubuntu Trusty, and Ubuntu Xenial (and older) ship Python versions which are too old. Python 3.6 or newer must be installed.
To get the development version of drgn, you will need to build it from source. First, install dependencies:
$ sudo dnf install autoconf automake elfutils-devel gcc git libkdumpfile-devel libtool make pkgconf python3 python3-devel python3-pip python3-setuptools
$ sudo dnf install autoconf automake elfutils-devel gcc git libtool make pkgconf python3 python3-devel python3-pip python3-setuptools
Optionally, install libkdumpfile-devel from EPEL on RHEL/CentOS >= 8 or install libkdumpfile from source if you want support for the makedumpfile format.
Replace dnf with yum for RHEL/CentOS < 8.
$ sudo apt-get install autoconf automake gcc git liblzma-dev libelf-dev libdw-dev libtool make pkgconf python3 python3-dev python3-pip python3-setuptools zlib1g-dev
Optionally, install libkdumpfile from source if you want support for the makedumpfile format.
$ sudo pacman -S --needed autoconf automake gcc git libelf libtool make pkgconf python python-pip python-setuptools
Optionally, install libkdumpfile from the AUR or from source if you want support for the makedumpfile format.
$ sudo zypper install autoconf automake gcc git libdw-devel libelf-devel libkdumpfile-devel libtool make pkgconf python3 python3-devel python3-pip python3-setuptools
Then, run:
$ git clone https://github.com/osandov/drgn.git $ cd drgn $ python3 setup.py build $ sudo python3 setup.py install
The above options all install drgn globally. You can also install drgn in a virtual environment, either with pip:
$ python3 -m venv drgnenv $ source drgnenv/bin/activate (drgnenv) $ pip3 install drgn (drgnenv) $ drgn --help
Or from source:
$ python3 -m venv drgnenv $ source drgnenv/bin/activate (drgnenv) $ python3 setup.py install (drgnenv) $ drgn --help
If you build drgn from source, you can also run it without installing it:
$ python3 setup.py build_ext -i $ python3 -m drgn --help
drgn debugs the running kernel by default; run sudo drgn. To debug a running program, run sudo drgn -p $PID. To debug a core dump (either a kernel vmcore or a userspace core dump), run drgn -c $PATH. Make sure to install debugging symbols for whatever you are debugging.
Then, you can access variables in the program with prog['name'] and access structure members with .:
$ sudo drgn >>> prog['init_task'].comm (char [16])"swapper/0"
You can use various predefined helpers:
>>> len(list(bpf_prog_for_each())) 11 >>> task = find_task(115) >>> cmdline(task) [b'findmnt', b'-p']
You can get stack traces with stack_trace() and access parameters or local variables with trace['name']:
>>> trace = stack_trace(task) >>> trace[5] #5 at 0xffffffff8a5a32d0 (do_sys_poll+0x400/0x578) in do_poll at ./fs/select.c:961:8 (inlined) >>> poll_list = trace[5]['list'] >>> file = fget(task, poll_list.entries[0].fd) >>> d_path(file.f_path.address_of_()) b'/proc/115/mountinfo'
The most important interfaces in drgn are programs, objects, and helpers.
A program being debugged is represented by an instance of the drgn.Program class. The drgn CLI is initialized with a Program named prog; unless you are using the drgn library directly, this is usually the only Program you will need.
A Program is used to look up type definitions, access variables, and read arbitrary memory:
>>> prog.type('unsigned long') prog.int_type(name='unsigned long', size=8, is_signed=False) >>> prog['jiffies'] Object(prog, 'volatile unsigned long', address=0xffffffffbe405000) >>> prog.read(0xffffffffbe411e10, 16) b'swapper/0\x00\x00\x00\x00\x00\x00\x00'
The drgn.Program.type(), drgn.Program.variable(), drgn.Program.constant(), and drgn.Program.function() methods look up those various things in a program. drgn.Program.read() reads memory from the program's address space. The [] operator looks up a variable, constant, or function:
>>> prog['jiffies'] == prog.variable('jiffies') True
It is usually more convenient to use the [] operator rather than the variable(), constant(), or function() methods unless the program has multiple objects with the same name, in which case the methods provide more control.
Variables, constants, functions, and computed values are all called objects in drgn. Objects are represented by the drgn.Object class. An object may exist in the memory of the program (a reference):
>>> Object(prog, 'int', address=0xffffffffc09031a0)
Or, an object may be a constant or temporary computed value (a value):
>>> Object(prog, 'int', value=4)
What makes drgn scripts expressive is that objects can be used almost exactly like they would be in the program's own source code. For example, structure members can be accessed with the dot (.) operator, arrays can be subscripted with [], arithmetic can be performed, and objects can be compared:
>>> print(prog['init_task'].comm[0]) (char)115 >>> print(repr(prog['init_task'].nsproxy.mnt_ns.mounts + 1)) Object(prog, 'unsigned int', value=34) >>> prog['init_task'].nsproxy.mnt_ns.pending_mounts > 0 False
Python doesn't have all of the operators that C or C++ do, so some substitutions are necessary:
A common use case is converting a drgn.Object to a Python value so it can be used by a standard Python library. There are a few ways to do this:
Objects have several attributes; the most important are drgn.Object.prog_ and drgn.Object.type_. The former is the drgn.Program that the object is from, and the latter is the drgn.Type of the object.
Note that all attributes and methods of the Object class end with an underscore (_) in order to avoid conflicting with structure or union members. The Object attributes and methods always take precedence; use drgn.Object.member_() if there is a conflict.
The main difference between reference objects and value objects is how they are evaluated. References are read from the program's memory every time they are evaluated; values simply return the stored value (drgn.Object.read_() reads a reference object and returns it as a value object):
>>> import time >>> jiffies = prog['jiffies'] >>> jiffies.value_() 4391639989 >>> time.sleep(1) >>> jiffies.value_() 4391640290 >>> jiffies2 = jiffies.read_() >>> jiffies2.value_() 4391640291 >>> time.sleep(1) >>> jiffies2.value_() 4391640291 >>> jiffies.value_() 4391640593
References have a drgn.Object.address_ attribute, which is the object's address as a Python int. This is slightly different from the drgn.Object.address_of_() method, which returns the address as a drgn.Object. Of course, both references and values can have a pointer type; address_ refers to the address of the pointer object itself, and drgn.Object.value_() refers to the value of the pointer (i.e., the address it points to):
>>> address = prog['jiffies'].address_ >>> type(address) <class 'int'> >>> print(hex(address)) 0xffffffffbe405000 >>> jiffiesp = prog['jiffies'].address_of_() >>> jiffiesp Object(prog, 'volatile unsigned long *', value=0xffffffffbe405000) >>> print(hex(jiffiesp.value_())) 0xffffffffbe405000
In addition to reference objects and value objects, objects may also be absent.
>>> Object(prog, "int").value_() Traceback (most recent call last): File "<console>", line 1, in <module> _drgn.ObjectAbsentError: object absent
This represents an object whose value or address is not known. For example, this can happen if the object was optimized out of the program by the compiler.
Any attempt to operate on an absent object results in a drgn.ObjectAbsentError exception, although basic information including its type may still be accessed.
Some programs have common data structures that you may want to examine. For example, consider linked lists in the Linux kernel:
struct list_head { struct list_head *next, *prev; }; #define list_for_each(pos, head) \ for (pos = (head)->next; pos != (head); pos = pos->next)
When working with these lists, you'd probably want to define a function:
def list_for_each(head): pos = head.next while pos != head: yield pos pos = pos.next
Then, you could use it like so for any list you need to look at:
>>> for pos in list_for_each(head): ... do_something_with(pos)
Of course, it would be a waste of time and effort for everyone to have to define these helpers for themselves, so drgn includes a collection of helpers for many use cases. See Helpers.
Validators are a special category of helpers that check the consistency of a data structure. In general, helpers assume that the data structures that they examine are valid. Validators do not make this assumption and do additional (potentially expensive) checks to detect broken invariants, corruption, etc.
Validators raise drgn.helpers.ValidationError if the data structure is not valid or drgn.FaultError if the data structure is invalid in a way that causes a bad memory access. They have names prefixed with validate_.
For example, drgn.helpers.linux.list.validate_list() checks the consistency of a linked list in the Linux kernel (in particular, the consistency of the next and prev pointers):
>>> validate_list(prog["my_list"].address_of_()) drgn.helpers.ValidationError: (struct list_head *)0xffffffffc029e460 next 0xffffffffc029e000 has prev 0xffffffffc029e450
drgn.helpers.linux.list.validate_list_for_each_entry() does the same checks while also returning the entries in the list for further validation:
def validate_my_list(prog): for entry in validate_list_for_each_entry( "struct my_entry", prog["my_list"].address_of_(), "list", ): if entry.value < 0: raise ValidationError("list contains negative entry")
In addition to the core concepts above, drgn provides a few additional abstractions.
The drgn.Thread class represents a thread. drgn.Program.threads(), drgn.Program.thread(), drgn.Program.main_thread(), and drgn.Program.crashed_thread() can be used to find threads:
>>> for thread in prog.threads(): ... print(thread.tid) ... 39143 39144 >>> print(prog.main_thread().tid) 39143 >>> print(prog.crashed_thread().tid) 39144
drgn represents stack traces with the drgn.StackTrace and drgn.StackFrame classes. drgn.stack_trace(), drgn.Program.stack_trace(), and drgn.Thread.stack_trace() return the call stack for a thread. The [] operator looks up an object in the scope of a StackFrame:
>>> trace = stack_trace(115) >>> trace #0 context_switch (./kernel/sched/core.c:4683:2) #1 __schedule (./kernel/sched/core.c:5940:8) #2 schedule (./kernel/sched/core.c:6019:3) #3 schedule_hrtimeout_range_clock (./kernel/time/hrtimer.c:2148:3) #4 poll_schedule_timeout (./fs/select.c:243:8) #5 do_poll (./fs/select.c:961:8) #6 do_sys_poll (./fs/select.c:1011:12) #7 __do_sys_poll (./fs/select.c:1076:8) #8 __se_sys_poll (./fs/select.c:1064:1) #9 __x64_sys_poll (./fs/select.c:1064:1) #10 do_syscall_x64 (./arch/x86/entry/common.c:50:14) #11 do_syscall_64 (./arch/x86/entry/common.c:80:7) #12 entry_SYSCALL_64+0x7c/0x15b (./arch/x86/entry/entry_64.S:113) #13 0x7f3344072af7 >>> trace[5] #5 at 0xffffffff8a5a32d0 (do_sys_poll+0x400/0x578) in do_poll at ./fs/select.c:961:8 (inlined) >>> prog['do_poll'] (int (struct poll_list *list, struct poll_wqueues *wait, struct timespec64 *end_time))<absent> >>> trace[5]['list'] *(struct poll_list *)0xffffacca402e3b50 = { .next = (struct poll_list *)0x0, .len = (int)1, .entries = (struct pollfd []){}, }
The symbol table of a program is a list of identifiers along with their address and size. drgn represents symbols with the drgn.Symbol class, which is returned by drgn.Program.symbol().
drgn automatically obtains type definitions from the program. Types are represented by the drgn.Type class and created by various factory functions like drgn.Program.int_type():
>>> prog.type('int') prog.int_type(name='int', size=4, is_signed=True)
You won't usually need to work with types directly, but see Types if you do.
Certain operations and objects in a program are platform-dependent; drgn allows accessing the platform that a program runs with the drgn.Platform class.
The drgn CLI is basically a wrapper around the drgn library which automatically creates a drgn.Program. The CLI can be run in interactive mode or script mode.
Script mode is useful for reusable scripts. Simply pass the path to the script along with any arguments:
$ cat script.py import sys from drgn.helpers.linux import find_task pid = int(sys.argv[1]) uid = find_task(pid).cred.uid.val.value_() print(f'PID {pid} is being run by UID {uid}') $ sudo drgn script.py 601 PID 601 is being run by UID 1000
It's even possible to run drgn scripts directly with the proper shebang:
$ cat script2.py #!/usr/bin/env drgn mounts = prog['init_task'].nsproxy.mnt_ns.mounts.value_() print(f'You have {mounts} filesystems mounted') $ sudo ./script2.py You have 36 filesystems mounted
Interactive mode uses the Python interpreter's interactive mode and adds a few nice features, including:
The default behavior of the Python REPL is to print the output of repr(). For drgn.Object and drgn.Type, this is a raw representation:
>>> print(repr(prog['jiffies'])) Object(prog, 'volatile unsigned long', address=0xffffffffbe405000) >>> print(repr(prog.type('atomic_t'))) prog.typedef_type(name='atomic_t', type=prog.struct_type(tag=None, size=4, members=(TypeMember(prog.type('int'), name='counter', bit_offset=0),)))
The standard print() function uses the output of str(). For drgn objects and types, this is a representation in programming language syntax:
>>> print(prog['jiffies']) (volatile unsigned long)4395387628 >>> print(prog.type('atomic_t')) typedef struct { int counter; } atomic_t
In interactive mode, the drgn CLI automatically uses str() instead of repr() for objects and types, so you don't need to call print() explicitly:
$ sudo drgn >>> prog['jiffies'] (volatile unsigned long)4395387628 >>> prog.type('atomic_t') typedef struct { int counter; } atomic_t
Refer to the API Reference. Look through the Helpers. Read some Case Studies. Browse through the tools. Check out the community contributions.
The User Guide covers basic usage of drgn, but drgn also supports more advanced use cases which are covered here.
drgn will automatically load debugging information based on the debugged program (e.g., from loaded kernel modules or loaded shared libraries). drgn.Program.load_debug_info() can be used to load additional debugging information:
>>> prog.load_debug_info(['./libfoo.so', '/usr/lib/libbar.so'])
In addition to the CLI, drgn is also available as a library. drgn.program_from_core_dump(), drgn.program_from_kernel(), and drgn.program_from_pid() correspond to the -c, -k, and -p command line options, respectively; they return a drgn.Program that can be used just like the one initialized by the CLI:
>>> import drgn >>> prog = drgn.program_from_kernel()
The core functionality of drgn is implemented in C and is available as a C library, libdrgn. See drgn.h.
Full documentation can be generated by running doxygen in the libdrgn directory of the source code. Note that the API and ABI are not yet stable.
The main components of a drgn.Program are the program memory, types, and symbols. The CLI and equivalent library interfaces automatically determine these. However, it is also possible to create a "blank" Program and plug in the main components. The drgn.cli.run_interactive() function allows you to run the same drgn CLI once you've created a drgn.Program, so it's easy to make a custom program which allows interactive debugging.
drgn.Program.add_memory_segment() defines a range of memory and how to read that memory. The following example uses a Btrfs filesystem image as the program "memory":
import drgn import os import sys from drgn.cli import run_interactive def btrfs_debugger(dev): file = open(dev, 'rb') size = file.seek(0, 2) def read_file(address, count, offset, physical): file.seek(offset) return file.read(count) platform = drgn.Platform(drgn.Architecture.UNKNOWN, drgn.PlatformFlags.IS_LITTLE_ENDIAN) prog = drgn.Program(platform) prog.add_memory_segment(0, size, read_file) prog.load_debug_info([f'/lib/modules/{os.uname().release}/kernel/fs/btrfs/btrfs.ko']) return prog prog = btrfs_debugger(sys.argv[1] if len(sys.argv) >= 2 else '/dev/sda') print(drgn.Object(prog, 'struct btrfs_super_block', address=65536)) run_interactive(prog, banner_func=lambda _: "BTRFS debugger")
drgn.Program.add_type_finder() and drgn.Program.add_object_finder() are the equivalent methods for plugging in types and objects.
Some of drgn's behavior can be modified through environment variables:
When debugging the Linux kernel, there are some special drgn.Objects accessible with drgn.Program.object() and drgn.Program[]. Some of these are available even without debugging information, thanks to metadata called "vmcoreinfo" which is present in kernel core dumps. These special objects include:
This corresponds to the UTS_RELEASE macro in the Linux kernel source code. This is the exact kernel release (i.e., the output of uname -r).
To use this as a Python string, you must convert it:
>>> release = prog["UTS_RELEASE"].string_().decode("ascii")
This is available without debugging information.
These correspond to the macros of the same name in the Linux kernel source code. The page size is the smallest contiguous unit of physical memory which can be allocated or mapped by the kernel.
>>> prog['PAGE_SIZE'] (unsigned long)4096 >>> prog['PAGE_SHIFT'] (int)12 >>> prog['PAGE_MASK'] (unsigned long)18446744073709547520 >>> 1 << prog['PAGE_SHIFT'] == prog['PAGE_SIZE'] True >>> ~(prog['PAGE_SIZE'] - 1) == prog['PAGE_MASK'] True
These are available without debugging information.
This is a counter of timer ticks. It is actually an alias of jiffies_64 on 64-bit architectures, or the least significant 32 bits of jiffies_64 on 32-bit architectures. Since this alias is defined via the linker, drgn handles it specially.
This is not available without debugging information.
This is a pointer to the "virtual memory map", an array of struct page for each physical page of memory. While the purpose and implementation details of this array are beyond the scope of this documentation, it is enough to say that it is represented in the kernel source in an architecture-dependent way, frequently as a macro or constant. The definition provided by drgn ensures that users can access it without resorting to architecture-specific logic.
This is not available without debugging information.
This is the data contained in the vmcoreinfo note, which is present either as an ELF note in /proc/kcore or ELF vmcores, or as a special data section in kdump-formatted vmcores. The vmcoreinfo note contains critical data necessary for interpreting the kernel image, such as KASLR offsets and data structure locations.
In the Linux kernel, this data is normally stored in a variable called vmcoreinfo_data. However, drgn reads this information from ELF note or from the diskdump header. It is possible (in rare cases, usually with vmcores created by hypervisors) for a vmcore to contain vmcoreinfo which differs from the data in vmcoreinfo_data, so it is important to distinguish the contents. For that reason, we use the name VMCOREINFO to distinguish it from the kernel variable vmcoreinfo_data.
This is available without debugging information.
The main functionality of a Program is looking up objects (i.e., variables, constants, or functions). This is usually done with the [] operator.
This is used for interpreting the type name given to type() and when creating an Object without an explicit type.
For the Linux kernel, this defaults to Language.C. For userspace programs, this defaults to the language of main in the program, falling back to Language.C. This heuristic may change in the future.
This can be explicitly set to a different language (e.g., if the heuristic was incorrect).
This is equivalent to prog.object(name) except that this raises KeyError instead of LookupError if no objects with the given name are found.
If there are multiple objects with the same name, one is returned arbitrarily. In this case, the variable(), constant(), function(), or object() methods can be used instead.
>>> prog['jiffies'] Object(prog, 'volatile unsigned long', address=0xffffffff94c05000)
>>> prog.variable('jiffies') Object(prog, 'volatile unsigned long', address=0xffffffff94c05000)
This is equivalent to prog.object(name, FindObjectFlags.VARIABLE, filename).
Note that support for macro constants is not yet implemented for DWARF files, and most compilers don't generate macro debugging information by default anyways.
>>> prog.constant('PIDTYPE_MAX') Object(prog, 'enum pid_type', value=4)
This is equivalent to prog.object(name, FindObjectFlags.CONSTANT, filename).
>>> prog.function('schedule') Object(prog, 'void (void)', address=0xffffffff94392370)
This is equivalent to prog.object(name, FindObjectFlags.FUNCTION, filename).
When debugging the Linux kernel, this can look up certain special objects documented in Linux Kernel Special Objects, sometimes without any debugging information loaded.
Global symbols are preferred over weak symbols, and weak symbols are preferred over other symbols. In other words: if a matching SymbolBinding.GLOBAL or SymbolBinding.UNIQUE symbol is found, it is returned. Otherwise, if a matching SymbolBinding.WEAK symbol is found, it is returned. Otherwise, any matching symbol (e.g., SymbolBinding.LOCAL) is returned. If there are multiple matching symbols with the same binding, one is returned arbitrarily. To retrieve all matching symbols, use symbols().
If a string argument is given, this returns all symbols matching that name. If an integer-like argument given, this returns a list of all symbols containing that address. If no argument is given, all symbols in the program are returned. In all cases, the symbols are returned in an unspecified order.
thread may be a thread ID (as defined by gettid(2)), in which case this will unwind the stack for the thread with that ID. The ID may be a Python int or an integer Object
thread may also be a struct pt_regs or struct pt_regs * object, in which case the initial register values will be fetched from that object.
Finally, if debugging the Linux kernel, thread may be a struct task_struct * object, in which case this will unwind the stack for that task. See drgn.helpers.linux.pid.find_task().
This is implemented for the Linux kernel (both live and core dumps) as well as userspace core dumps; it is not yet implemented for live userspace processes.
>>> prog.type('long') prog.int_type(name='long', size=8, is_signed=True)
This is mainly useful so that helpers can use prog.type() to get a Type regardless of whether they were given a str or a Type. For example:
def my_helper(obj: Object, type: Union[str, Type]) -> bool: # type may be str or Type. type = obj.prog_.type(type) # type is now always Type. return sizeof(obj) > sizeof(type)
This is only defined for userspace programs.
For userspace programs, this is the thread that received the fatal signal (e.g., SIGSEGV or SIGQUIT).
For the kernel, this is the thread that panicked (either directly or as a result of an oops, BUG_ON(), etc.).
>>> prog.read(0xffffffffbe012b40, 16) b'swapper/0'
read_u8(), read_u16(), read_u32(), and read_u64() read an 8-, 16-, 32-, or 64-bit unsigned integer, respectively. read_word() reads a program word-sized unsigned integer.
For signed integers, alternate byte order, or other formats, you can use read() and int.from_bytes() or the struct module.
If it overlaps a previously registered segment, the new segment takes precedence.
Callbacks are called in reverse order of the order they were added until the type is found. So, more recently added callbacks take precedence.
Callbacks are called in reverse order of the order they were added until the object is found. So, more recently added callbacks take precedence.
This loads the memory segments from the core dump and determines the mapped executable and libraries. It does not load any debugging symbols; see load_default_debug_info().
This loads the memory of the running kernel and thus requires root privileges. It does not load any debugging symbols; see load_default_debug_info().
This loads the memory of the process and determines the mapped executable and libraries. It does not load any debugging symbols; see load_default_debug_info().
Note that this is parallelized, so it is usually faster to load multiple files at once rather than one by one.
Also load debugging information which can automatically be determined from the program.
For the Linux kernel, this tries to load vmlinux and any loaded kernel modules from a few standard locations.
For userspace programs, this tries to load the executable and any loaded libraries.
This implies main=True.
Also load debugging information for the main executable.
For the Linux kernel, this tries to load vmlinux.
This is currently ignored for userspace programs.
This is equivalent to load_debug_info(None, True).
This isn't used by drgn itself. It is intended to be used by helpers to cache metadata about the program. For example, if a helper for a program depends on the program version or an optional feature, the helper can detect it and cache it for subsequent invocations:
def my_helper(prog): try: have_foo = prog.cache['have_foo'] except KeyError: have_foo = detect_foo_feature(prog) prog.cache['have_foo'] = have_foo if have_foo: return prog['foo'] else: return prog['bar']
ProgramFlags are flags that can apply to a Program (e.g., about what kind of program it is).
FindObjectFlags are flags for Program.object(). These can be combined to search for multiple kinds of objects at once.
This is equivalent to prog.stack_trace(thread.tid). See Program.stack_trace().
The Program.type(), Program.object(), Program.variable(), Program.constant(), and Program.function() methods all take a filename parameter to distinguish between multiple definitions with the same name. The filename refers to the source code file that contains the definition. It is matched with filename_matches(). If multiple definitions match, one is returned arbitrarily.
The filename is matched from right to left, so 'stdio.h', 'include/stdio.h', 'usr/include/stdio.h', and '/usr/include/stdio.h' would all match a definition in /usr/include/stdio.h. If needle is None or empty, it matches any definition. If haystack is None or empty, it only matches if needle is also None or empty.
The drgn command line interface automatically creates a Program named prog. However, drgn may also be used as a library without the CLI, in which case a Program must be created manually.
Most functions that take a Program can be called without the prog argument. In that case, the default program argument is used, which is determined by the rules below.
NOTE:
# Equivalent in the CLI. find_task(pid) find_task(prog, pid) find_task(prog["init_pid_ns"].address_of_(), pid)
The default program is set automatically in the CLI. Library users can get and set it manually. The default program is a per-thread setting. See Thread Safety.
Error raised when trying to use the default program when it is not set.
For helpers, it is recommended to use the decorators from the drgn.helpers.common.prog module instead.
An Architecture represents an instruction set architecture.
PlatformFlags are flags describing a Platform.
This class cannot be constructed; there are singletons for the supported languages.
All instances of this class have two attributes: prog_, the program that the object is from; and type_, the type of the object. Reference objects also have an address_ and a bit_offset_. Objects may also have a bit_field_size_.
repr() of an object returns a Python representation of the object:
>>> print(repr(prog['jiffies'])) Object(prog, 'volatile unsigned long', address=0xffffffffbf005000)
str() returns a "pretty" representation of the object in programming language syntax:
>>> print(prog['jiffies']) (volatile unsigned long)4326237045
The output format of str() can be modified by using the format_() method instead:
>>> sysname = prog['init_uts_ns'].name.sysname >>> print(sysname) (char [65])"Linux" >>> print(sysname.format_(type_name=False)) "Linux" >>> print(sysname.format_(string=False)) (char [65]){ 76, 105, 110, 117, 120 }
NOTE:
Objects support the following operators:
These operators all have the semantics of the program's programming language. For example, adding two objects from a program written in C results in an object with a type and value according to the rules of C:
>>> Object(prog, 'unsigned long', 2**64 - 1) + Object(prog, 'int', 1) Object(prog, 'unsigned long', value=0)
If only one operand to a binary operator is an object, the other operand will be converted to an object according to the language's rules for literals:
>>> Object(prog, 'char', 0) - 1 Object(prog, 'int', value=-1)
The standard int(), float(), and bool() functions convert an object to that Python type. Conversion to bool uses the programming language's notion of "truthiness". Additionally, certain Python functions will automatically coerce an object to the appropriate Python type (e.g., hex(), round(), and list subscripting).
Object attributes and methods are named with a trailing underscore to avoid conflicting with structure, union, or class members. The attributes and methods always take precedence; use member_() if there is a conflict.
Objects are usually obtained directly from a Program, but they can be constructed manually, as well (for example, if you got a variable address from a log file).
This is used to emulate a literal number in the source code of the program. The type is deduced from value according to the language's rules for literals.
This is False for all values and references (even if the reference has an invalid address).
This corresponds to both the member access (.) and member access through pointer (->) operators in C.
Note that if name is an attribute or method of the Object class, then that takes precedence. Otherwise, this is equivalent to member_().
>>> print(prog['init_task'].pid) (pid_t)0
>>> print(prog['init_task'].comm[1]) (char)119
[0] is also the equivalent of the pointer dereference (*) operator in C:
>>> ptr_to_ptr *(void **)0xffff9b86801e2968 = 0xffff9b86801e2460 >>> print(ptr_to_ptr[0]) (void *)0xffff9b86801e2460
This is only valid for pointers and arrays.
NOTE:
>>> len(prog['init_task'].comm) 16
This is only valid for arrays.
For basic types (integer, floating-point, boolean), this returns an object of the directly corresponding Python type (int, float, bool). For pointers, this returns the address value of the pointer. For enums, this returns an int. For structures and unions, this returns a dict of members. For arrays, this returns a list of values.
This is only valid for pointers and arrays. The element type is ignored; this operates byte-by-byte.
For pointers and flexible arrays, this stops at the first null byte.
For complete arrays, this stops at the first null byte or at the end of the array.
This is valid for structures, unions, classes, and pointers to any of those. If the object is a pointer, it is automatically dereferenced first.
Normally the dot operator (.) can be used to accomplish the same thing, but this method can be used if there is a name conflict with an Object attribute or method.
This corresponds to the address-of (&) operator in C. It is only possible for reference objects, as value objects don't have an address in the program.
As opposed to address_, this returns an Object, not an int.
This is useful if the object can change in the running program (but of course nothing stops the program from modifying the object while it is being read).
As opposed to value_(), this returns an Object, not a standard Python type.
Various format options can be passed (as keyword arguments) to control the output. Options that aren't passed or are passed as None fall back to a default. Specifically, obj.format_() (i.e., with no passed options) is equivalent to str(obj).
>>> workqueues = prog['workqueues'] >>> print(workqueues) (struct list_head){ .next = (struct list_head *)0xffff932ecfc0ae10, .prev = (struct list_head *)0xffff932e3818fc10, } >>> print(workqueues.format_(type_name=False, ... member_type_names=False, ... member_names=False, ... members_same_line=True)) { 0xffff932ecfc0ae10, 0xffff932e3818fc10 }
This is equivalent to Object(prog, type, 0).
Objects with a scalar type (integer, boolean, enumerated, floating-point, or pointer) can be casted to a different scalar type. Other objects can only be casted to the same type. This always results in a value object. See also drgn.reinterpret().
This reinterprets the raw memory of the object, so an object can be reinterpreted as any other type. However, value objects with a scalar type cannot be reinterpreted, as their memory layout in the program is not known. Reinterpreting a reference results in a reference, and reinterpreting a value results in a value. See also drgn.cast().
This corresponds to the container_of() macro in C.
A SymbolBinding describes the linkage behavior and visibility of a symbol.
A SymbolKind describes the kind of entity that a symbol represents.
Stack traces are retrieved with stack_trace(), Program.stack_trace(), or Thread.stack_trace().
See Program.stack_trace() for more details.
len(trace) is the number of stack frames in the trace. trace[0] is the innermost stack frame, trace[1] is its caller, and trace[len(trace) - 1] is the outermost frame. Negative indexing also works: trace[-1] is the outermost frame and trace[-len(trace)] is the innermost frame. It is also iterable:
for frame in trace: if frame.name == 'io_schedule': print('Thread is doing I/O')
str() returns a pretty-printed stack trace:
>>> prog.stack_trace(1) #0 context_switch (kernel/sched/core.c:4339:2) #1 __schedule (kernel/sched/core.c:5147:8) #2 schedule (kernel/sched/core.c:5226:3) #3 do_wait (kernel/exit.c:1534:4) #4 kernel_wait4 (kernel/exit.c:1678:8) #5 __do_sys_wait4 (kernel/exit.c:1706:13) #6 do_syscall_64 (arch/x86/entry/common.c:47:14) #7 entry_SYSCALL_64+0x7c/0x15b (arch/x86/entry/entry_64.S:112) #8 0x4d49dd
The format is subject to change. The drgn CLI is set up so that stack traces are displayed with str() by default.
str() returns a pretty-printed stack frame:
>>> prog.stack_trace(1)[0] #0 at 0xffffffffb64ac287 (__schedule+0x227/0x606) in context_switch at kernel/sched/core.c:4339:2 (inlined)
This includes more information than when printing the full stack trace. The format is subject to change. The drgn CLI is set up so that stack frames are displayed with str() by default.
The [] operator can look up function parameters, local variables, and global variables in the scope of the stack frame:
>>> prog.stack_trace(1)[0]['prev'].pid (pid_t)1 >>> prog.stack_trace(1)[0]['scheduler_running'] (int)1
The name cannot be determined if debugging information is not available for the function, e.g., because it is implemented in assembly. It may be desirable to use the symbol name or program counter as a fallback:
name = frame.name if name is None: try: name = frame.symbol().name except LookupError: name = hex(frame.pc)
An inline frame shares the same stack frame in memory as its caller. Therefore, it has the same registers (including program counter and thus symbol).
If this is True, then the register values in this frame are the values at the time that the frame was interrupted.
This is False if the frame is for a function call, in which case the register values are the values when control returns to this frame. In particular, the program counter is the return address, which is typically the instruction after the call instruction.
If the object exists but has been optimized out, this returns an absent object.
Not all names may have present values, but they can be used with the [] operator to check.
This is equivalent to:
prog.symbol(frame.pc - (0 if frame.interrupted else 1))
repr() of a Type returns a Python representation of the type:
>>> print(repr(prog.type('sector_t'))) prog.typedef_type(name='sector_t', type=prog.int_type(name='unsigned long', size=8, is_signed=False))
str() returns a representation of the type in programming language syntax:
>>> print(prog.type('sector_t')) typedef unsigned long sector_t
The drgn CLI is set up so that types are displayed with str() instead of repr() by default.
This class cannot be constructed directly. Instead, use one of the Type Constructors.
For other types, this attribute is not present.
Note that the original qualifiers are replaced, not added to.
If this type has any unnamed members, this also matches members of those unnamed members, recursively. If the member is found in an unnamed member, TypeMember.bit_offset and TypeMember.offset are adjusted accordingly.
If this type has any unnamed members, this also matches members of those unnamed members, recursively.
One of:
This is the default initializer for the member, or an absent object if the member has no default initializer. (However, the DWARF specification as of version 5 does not actually support default member initializers, so this is usually absent.)
This is a shortcut for TypeMember.object.type.
This is a shortcut for TypeMember.object.bit_field_size_.
Its name and value may be accessed as attributes or unpacked:
>>> prog.type('enum pid_type').enumerators[0].name 'PIDTYPE_PID' >>> name, value = prog.type('enum pid_type').enumerators[0] >>> value 0
One of:
If the parameter does not have a default argument, then this is an absent object.
NOTE:
This is the same as TypeParameter.default_argument.type_.
One of:
If this is a type template parameter, then this is a Type. If this is a non-type template parameter, then this is an Object.
NOTE:
Compilers are inconsistent about which interpretation they use.
GCC added this information in version 4.9. Clang added it in version 11 (and only when emitting DWARF version 5). If the program was compiled by an older version, this is always false.
A TypeKind represents a kind of type.
A PrimitiveType represents a primitive type known to drgn.
Qualifiers are modifiers on types.
This corresponds to offsetof() in C.
Custom drgn types can be created with the following factory functions. These can be used just like types obtained from Program.type().
The script is executed in the same context as the caller: currently defined globals are available to the script, and globals defined by the script are added back to the calling context.
This is most useful for executing scripts from interactive mode. For example, you could have a script named exe.py:
"""Get all tasks executing a given file.""" import sys from drgn.helpers.linux.fs import d_path from drgn.helpers.linux.pid import find_task def task_exe_path(task): if task.mm: return d_path(task.mm.exe_file.f_path).decode() else: return None tasks = [ task for task in for_each_task() if task_exe_path(task) == sys.argv[1] ]
Then, you could execute it and use the defined variables and functions:
>>> execscript('exe.py', '/usr/bin/bash') >>> tasks[0].pid (pid_t)358442 >>> task_exe_path(find_task(357954)) '/usr/bin/vim'
An int or integer-like object.
Parameters annotated with this type expect an integer which may be given as a Python int or an Object with integer type.
Parameters annotated with this type accept a filesystem path as str, bytes, or os.PathLike.
This error is raised when a bad memory access is attempted (i.e., when accessing a memory address which is not valid in a program).
This error is raised when one or more files in a program do not have debug information.
This error is raised when attempting to use an absent object.
This error is raised when attempting to access beyond the bounds of a value object.
Functions for embedding the drgn CLI.
The run_interactive() function does not include this banner at the beginning of an interactive session. Use this function to retrieve one line of text to add to the beginning of the drgn banner, or print it before calling run_interactive().
This function allows your application to embed the same REPL that drgn provides when it is run on the command line in interactive mode.
NOTE:
Applications using readline should save their history and clear any custom settings before calling this function. After calling this function, applications should restore their history and settings before using readline.
drgn logs using the standard logging module to a logger named "drgn".
Only one thread at a time should access the same Program (including Object, Type, StackTrace, etc. from that program). It is safe to use different Programs from concurrent threads.
The drgn.helpers package contains subpackages which provide helpers for working with particular types of programs. Currently, there are common helpers and helpers for the Linux kernel. In the future, there may be helpers for, e.g., glibc and libstdc++.
Error raised by a validator when an inconsistent or invalid state is detected.
The drgn.helpers.common package provides helpers that can be used with any program. The helpers are available from the individual modules in which they are defined and from this top-level package. E.g., the following are both valid:
>>> from drgn.helpers.common.memory import identify_address >>> from drgn.helpers.common import identify_address
Some of these helpers may have additional program-specific behavior but are otherwise generic.
The drgn.helpers.common.format module provides generic helpers for formatting different things as text.
By default, flags are specified by their bit number:
>>> decode_flags(2, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)]) 'ITALIC'
They can also be specified by their value:
>>> decode_flags(2, [("BOLD", 1), ("ITALIC", 2), ("UNDERLINE", 4)], ... bit_numbers=False) 'ITALIC'
Multiple flags are combined with "|":
>>> decode_flags(5, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)]) 'BOLD|UNDERLINE'
If there are multiple names for the same bit, they are all included:
>>> decode_flags(2, [("SMALL", 0), ("BIG", 1), ("LARGE", 1)]) 'BIG|LARGE'
If there are any unknown bits, their raw value is included:
>>> decode_flags(27, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)]) 'BOLD|ITALIC|0x18'
Zero is returned verbatim:
>>> decode_flags(0, [("BOLD", 0), ("ITALIC", 1), ("UNDERLINE", 2)]) '0'
This supports enums where the values are bit numbers:
>>> print(bits_enum) enum style_bits { BOLD = 0, ITALIC = 1, UNDERLINE = 2, } >>> decode_enum_type_flags(5, bits_enum) 'BOLD|UNDERLINE'
Or the values of the flags:
>>> print(flags_enum) enum style_flags { BOLD = 1, ITALIC = 2, UNDERLINE = 4, } >>> decode_enum_type_flags(5, flags_enum, bit_numbers=False) 'BOLD|UNDERLINE'
See decode_flags().
>>> number_in_binary_units(1280) '1.2K'
A precision can be specified:
>>> number_in_binary_units(1280, precision=2) '1.25K'
Exact numbers are printed without a fractional part:
>>> number_in_binary_units(1024 * 1024) '1M'
Numbers less than 1024 are not scaled:
>>> number_in_binary_units(10) '10'
The drgn.helpers.common.memory module provides helpers for working with memory and addresses.
For all programs, this will identify addresses as follows:
Additionally, for the Linux kernel, this will identify:
This may recognize other types of addresses in the future.
The drgn.helpers.common.prog module provides decorators to transparently use the default program argument.
@takes_program_or_default def my_helper(prog: Program, n: IntegerLike) -> Foo: ... my_helper(1) # is equivalent to my_helper(get_default_prog(), 1) obj = Object(...) my_helper(obj) # is equivalent to my_helper(obj.prog_, obj)
@takes_object_or_program_or_default def my_helper(prog: Program, obj: Optional[Object], n: IntegerLike) -> Foo: ... my_helper(prog, 1) # is equivalent to my_helper.__wrapped__(prog, None, 1) obj = Object(...) my_helper(obj, 1) # is equivalent to my_helper.__wrapped__(obj.prog_, obj, 1) my_helper(1) # is equivalent to my_helper.__wrapped__(get_default_prog(), None, 1) one_obj = Object(..., 1) my_helper(one_obj) # is equivalent to my_helper.__wrapped__(one_obj.prog_, None, one_obj)
WARNING:
# NOT ALLOWED @takes_object_or_program_or_default def my_helper(prog: Program, obj: Optional[Object], foo: str = ""): ... # OK @takes_object_or_program_or_default def my_helper(prog: Program, obj: Optional[Object], *, foo: str = ""): ...
NOTE:
The drgn.helpers.common.stack module provides helpers for working with stack traces.
Currently, this will identify any addresses on the stack with identify_address().
>>> print_annotated_stack(stack_trace(1)) STACK POINTER VALUE [stack frame #0 at 0xffffffff8dc93c41 (__schedule+0x429/0x488) in context_switch at ./kernel/sched/core.c:5209:2 (inlined)] [stack frame #1 at 0xffffffff8dc93c41 (__schedule+0x429/0x488) in __schedule at ./kernel/sched/core.c:6521:8] ffffa903c0013d28: ffffffff8d8497bf [function symbol: __flush_tlb_one_user+0x5] ffffa903c0013d30: 000000008d849eb5 ffffa903c0013d38: 0000000000000001 ffffa903c0013d40: 0000000000000004 ffffa903c0013d48: efdea37bb7cb1f00 ffffa903c0013d50: ffff926641178000 [slab object: task_struct+0x0] ffffa903c0013d58: ffff926641178000 [slab object: task_struct+0x0] ffffa903c0013d60: ffffa903c0013e10 ffffa903c0013d68: ffff926641177ff0 [slab object: mm_struct+0x70] ffffa903c0013d70: ffff926641178000 [slab object: task_struct+0x0] ffffa903c0013d78: ffff926641178000 [slab object: task_struct+0x0] ffffa903c0013d80: ffffffff8dc93d29 [function symbol: schedule+0x89] ...
The drgn.helpers.common.type module provides generic helpers for working with types in ways that aren't provided by the core drgn library.
The drgn.helpers.linux package contains several modules for working with data structures and subsystems in the Linux kernel. The helpers are available from the individual modules in which they are defined and from this top-level package. E.g., the following are both valid:
>>> from drgn.helpers.linux.list import list_for_each_entry >>> from drgn.helpers.linux import list_for_each_entry
Iterator macros (for_each_foo) are a common idiom in the Linux kernel. The equivalent drgn helpers are implemented as Python generators. For example, the following code in C:
list_for_each(pos, head) do_something_with(pos);
Translates to the following code in Python:
for pos in list_for_each(head): do_something_with(pos)
The drgn.helpers.linux.bitops module provides helpers for common bit operations in the Linux kernel.
The drgn.helpers.linux.block module provides helpers for working with the Linux block layer, including disks (struct gendisk) and partitions.
Since Linux v5.11, partitions are represented by struct block_device. Before that, they were represented by struct hd_struct.
The drgn.helpers.linux.boot module provides helpers for inspecting the Linux kernel boot configuration.
The drgn.helpers.linux.bpf module provides helpers for working with BPF interface in include/linux/bpf.h, include/linux/bpf-cgroup.h, etc.
This is only supported since Linux v4.18.
This is only supported since Linux v5.8.
This is only supported since Linux v4.13.
This is only supported since Linux v4.13.
The drgn.helpers.linux.cgroup module provides helpers for working with the cgroup interface in include/linux/cgroup.h. Only cgroup v2 is supported.
The drgn.helpers.linux.cpumask module provides helpers for working with CPU masks from include/linux/cpumask.h.
>>> cpumask_to_cpulist(mask) 0-3,8-11
The drgn.helpers.linux.device module provides helpers for working with Linux devices, including the kernel encoding of dev_t.
The drgn.helpers.linux.fs module provides helpers for working with the Linux virtual filesystem (VFS) layer, including mounts, dentries, and inodes.
>>> path_lookup('/usr/include/stdlib.h') ... Exception: could not find '/usr/include/stdlib.h' in dcache >>> open('/usr/include/stdlib.h').close() >>> path_lookup('/usr/include/stdlib.h') (struct path){ .mnt = (struct vfsmount *)0xffff8b70413cdca0, .dentry = (struct dentry *)0xffff8b702ac2c480, }
The drgn.helpers.linux.idr module provides helpers for working with the IDR data structure in include/linux/idr.h. An IDR provides a mapping from an ID to a pointer.
The drgn.helpers.linux.kconfig module provides helpers for reading the Linux kernel build configuration.
>>> get_kconfig()['CONFIG_SMP'] 'y' >>> get_kconfig()['CONFIG_HZ'] '300'
This is only supported if the kernel was compiled with CONFIG_IKCONFIG. Note that most Linux distributions do not enable this option.
The drgn.helpers.linux.kernfs module provides helpers for working with the kernfs pseudo filesystem interface in include/linux/kernfs.h.
The drgn.helpers.linux.list module provides helpers for working with the doubly-linked list implementations (struct list_head and struct hlist_head) in include/linux/list.h.
The list is assumed to be non-empty.
See also list_first_entry_or_null().
See also list_first_entry().
The list is assumed to be non-empty.
The drgn.helpers.linux.list_nulls module provides helpers for working with the special version of lists (struct hlist_nulls_head and struct hlist_nulls_node) in include/linux/list_nulls.h where the end of list is not a NULL pointer, but a "nulls" marker.
The drgn.helpers.linux.llist module provides helpers for working with the lockless, NULL-terminated, singly-linked list implementation in include/linux/llist.h (struct llist_head and struct llist_node).
The list is assumed to be non-empty.
See also llist_first_entry_or_null().
See also llist_first_entry().
The drgn.helpers.linux.mapletree module provides helpers for working with maple trees from include/linux/maple_tree.h.
Maple trees were introduced in Linux 6.1.
>>> entry = mtree_load(task.mm.mm_mt.address_of_(), 0x55d65cfaa000) >>> cast("struct vm_area_struct *", entry) *(struct vm_area_struct *)0xffff97ad82bfc930 = { ... }
>>> for first_index, last_index, entry in mt_for_each(task.mm.mm_mt.address_of_()): ... print(hex(first_index), hex(last_index), entry) ... 0x55d65cfaa000 0x55d65cfaafff (void *)0xffff97ad82bfc930 0x55d65cfab000 0x55d65cfabfff (void *)0xffff97ad82bfc0a8 0x55d65cfac000 0x55d65cfacfff (void *)0xffff97ad82bfc000 0x55d65cfad000 0x55d65cfadfff (void *)0xffff97ad82bfcb28 ...
The drgn.helpers.linux.mm module provides helpers for working with the Linux memory management (MM) subsystem. Only AArch64, ppc64, s390x, and x86-64 are currently supported.
If page is a tail page, this returns the head page of the compound page it belongs to. Otherwise, it returns page.
>>> decode_page_flags(page) 'PG_uptodate|PG_dirty|PG_lru|PG_reclaim|PG_swapbacked|PG_readahead|PG_savepinned|PG_isolated|PG_reported'
NOTE:
>>> for page in for_each_page(): ... try: ... if PageLRU(page): ... print(hex(page)) ... except drgn.FaultError: ... continue 0xfffffb4a000c0000 0xfffffb4a000c0040 ...
This may be fixed in the future.
NOTE:
But not:
For vmalloc or vmap addresses, use vmalloc_to_page(addr). For arbitrary kernel addresses, use follow_page(prog["init_mm"].address_of_(), addr).
NOTE:
NOTE:
>>> task = find_task(113) >>> follow_page(task.mm, 0x7fffbbb6d4d0) *(struct page *)0xffffbe4bc0337b80 = { ... }
>>> task = find_task(113) >>> follow_pfn(task.mm, 0x7fffbbb6d4d0) (unsigned long)52718
>>> task = find_task(113) >>> follow_phys(task.mm, 0x7fffbbb6d4d0) (phys_addr_t)215934160
>>> task = find_task(113) >>> vmalloc_to_page(task.stack) *(struct page *)0xffffbe4bc00a2200 = { ... }
>>> task = find_task(113) >>> vmalloc_to_pfn(task.stack) (unsigned long)10376
>>> task = find_task(1490152) >>> access_process_vm(task, 0x7f8a62b56da0, 12) b'hello, world'
>>> task = find_task(1490152) >>> access_remote_vm(task.mm, 0x7f8a62b56da0, 12) b'hello, world'
>>> cmdline(find_task(1495216)) [b'vim', b'drgn/helpers/linux/mm.py']
$ tr '\0' ' ' < /proc/1495216/cmdline vim drgn/helpers/linux/mm.py
>>> environ(find_task(1497797)) [b'HOME=/root', b'PATH=/usr/local/sbin:/usr/local/bin:/usr/bin', b'LOGNAME=root']
$ tr '\0' '\n' < /proc/1497797/environ HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/usr/bin LOGNAME=root
>>> for vma in for_each_vma(task.mm): ... print(vma) ... *(struct vm_area_struct *)0xffff97ad82bfc930 = { ... } *(struct vm_area_struct *)0xffff97ad82bfc0a8 = { ... } ...
The drgn.helpers.linux.net module provides helpers for working with the Linux kernel networking subsystem.
>>> dev = netdev_get_by_name("wlp0s20f3") >>> netdev_priv(dev) (void *)0xffff9419c9dec9c0 >>> netdev_priv(dev, "struct ieee80211_sub_if_data") *(struct ieee80211_sub_if_data *)0xffff9419c9dec9c0 = { ... }
The drgn.helpers.linux.nodemask module provides helpers for working with NUMA node masks from include/linux/nodemask.h.
The drgn.helpers.linux.percpu module provides helpers for working with per-CPU allocations from include/linux/percpu.h and per-CPU counters from include/linux/percpu_counter.h.
>>> prog["init_net"].loopback_dev.pcpu_refcnt (int *)0x2c980 >>> per_cpu_ptr(prog["init_net"].loopback_dev.pcpu_refcnt, 7) *(int *)0xffff925e3ddec980 = 4
>>> print(repr(prog["runqueues"])) Object(prog, 'struct rq', address=0x278c0) >>> per_cpu(prog["runqueues"], 6).curr.comm (char [16])"python3"
The drgn.helpers.linux.pid module provides helpers for looking up process IDs and processes.
The drgn.helpers.linux.printk module provides helpers for reading the Linux kernel log buffer.
Kernel log record.
This is available if the message was logged from task context and if the kernel saves the printk() caller ID.
As of Linux 5.10, the kernel always saves the caller ID. From Linux 5.1 through 5.9, it is saved only if the kernel was compiled with CONFIG_PRINTK_CALLER. Before that, it is never saved.
This is available only if the message was logged when not in task context (e.g., in an interrupt handler) and if the kernel saves the printk() caller ID.
See caller_tid for when the kernel saves the caller ID.
See the /dev/kmsg documentation for an explanation of the keys and values.
The format of each line is:
[ timestamp] message
If you need to format the log buffer differently, use get_printk_records() and format it yourself.
The drgn.helpers.linux.radixtree module provides helpers for working with radix trees from include/linux/radix-tree.h.
SEE ALSO:
The drgn.helpers.linux.rbtree module provides helpers for working with red-black trees from include/linux/rbtree.h.
Note that this function does not have an analogue in the Linux kernel source code, as tree searches are all open-coded.
This checks that:
The drgn.helpers.linux.sched module provides helpers for working with the Linux CPU scheduler.
>>> cpu_curr(7).comm (char [16])"python3"
>>> idle_task(1).comm (char [16])"swapper/1"
>>> loadavg() (2.34, 0.442, 1.33)
The drgn.helpers.linux.slab module provides helpers for working with the Linux slab allocator.
WARNING:
Unless configured otherwise, the kernel may merge slab caches of similar sizes together. See the SLUB users guide and slab_merge/slab_nomerge in the kernel parameters documentation.
This can cause confusion, as only the name of the first cache will be found, and objects of different types will be mixed in the same slab cache.
For example, suppose that we have two types, struct foo and struct bar, which have the same size but are otherwise unrelated. If the kernel creates a slab cache named foo for struct foo, then another slab cache named bar for struct bar, then slab cache foo will be reused instead of creating another cache for bar. So the following will fail:
find_slab_cache("bar")
And the following will also return struct bar * objects errantly casted to struct foo *:
slab_cache_for_each_allocated_object(find_slab_cache("foo"), "struct foo")
Unfortunately, these issues are difficult to work around generally, so one must be prepared to handle them on a case-by-case basis (e.g., by looking up the slab cache by its variable name and by checking that members of the structure make sense for the expected type).
The SLAB and SLUB subsystems can merge caches with similar settings and object sizes, as described in the documentation of slab_cache_is_merged(). In some cases, the information about which caches were merged is lost, but in other cases, we can reconstruct the info. This function reconstructs the mapping, but requires that the kernel is configured with CONFIG_SLUB and CONFIG_SYSFS.
The returned dict maps from original cache name, to merged cache name. You can use this mapping to discover the correct cache to lookup via find_slab_cache(). The dict contains an entry only for caches which were merged into a cache of a different name.
>>> cache_to_merged = get_slab_cache_aliases() >>> cache_to_merged["dnotify_struct"] 'avc_xperms_data' >>> "avc_xperms_data" in cache_to_merged False >>> find_slab_cache("dnotify_struct") is None True >>> find_slab_cache("avc_xperms_data") is None False
Only the SLUB and SLAB allocators are supported; SLOB does not store enough information to identify objects in a slab cache.
>>> dentry_cache = find_slab_cache("dentry") >>> next(slab_cache_for_each_allocated_object(dentry_cache, "struct dentry")) *(struct dentry *)0xffff905e41404000 = { ... }
>>> ptr = find_task(1).comm.address_of_() >>> info = slab_object_info(ptr) >>> info SlabObjectInfo(slab_cache=Object(prog, 'struct kmem_cache *', address=0xffffdb93c0045e18), slab=Object(prog, 'struct slab *', value=0xffffdb93c0045e00), address=0xffffa2bf81178000, allocated=True)
Note that SlabObjectInfo.address is the start address of the object, which may be less than addr if addr points to a member inside of the object:
>>> ptr.value_() - info.address 1496 >>> offsetof(prog.type("struct task_struct"), "comm") 1496
Note that SLOB does not store enough information to identify slab objects, so if the kernel is configured to use SLOB, this will always return None.
Since Linux v5.17, this is a struct slab *. Before that, it is a struct page *.
Note that SLOB does not store enough information to identify objects in a slab cache, so if the kernel is configured to use SLOB, this will always return NULL.
The drgn.helpers.linux.tc module provides helpers for working with the Linux kernel Traffic Control (TC) subsystem.
The drgn.helpers.linux.tcp module provides helpers for working with the TCP protocol in the Linux kernel.
The drgn.helpers.linux.user module provides helpers for working with users in the Linux kernel.
The drgn.helpers.linux.wait module provides helpers for working with wait queues (wait_queue_head_t and wait_queue_entry_t) from include/linux/wait.h.
NOTE:
WARNING:
The drgn.helpers.linux.xarray module provides helpers for working with the XArray data structure from include/linux/xarray.h.
NOTE:
Specifically, xa_load() is equivalent to radix_tree_lookup(), and xa_for_each() is equivalent to radix_tree_for_each(), except that the radix tree helpers assume advanced=False. (Therefore, xa_load() and xa_for_each() also accept a struct radix_tree_root *, and radix_tree_lookup() and radix_tree_for_each() also accept a struct xarray *.)
>>> entry = xa_load(inode.i_mapping.i_pages.address_of_(), 2) >>> cast("struct page *", entry) *(struct page *)0xffffed6980306f40 = { ... }
>>> for index, entry in xa_for_each(inode.i_mapping.i_pages.address_of_()): ... print(index, entry) ... 0 (void *)0xffffed6980356140 1 (void *)0xffffed6980306f80 2 (void *)0xffffed6980306f40 3 (void *)0xffffed6980355b40
See xa_to_value().
In addition to pointers, XArrays can store integers between 0 and LONG_MAX. If xa_is_value() returns True, use this to get the stored integer.
>>> entry = xa_load(xa, 9) >>> entry (void *)0xc9 >>> xa_is_value(entry) True >>> xa_to_value(entry) (unsigned long)100
A zero entry is an entry that was reserved but is not present. These are only visible to the XArray advanced API, so they are only returned by xa_load() and xa_for_each() when advanced = True.
>>> entry = xa_load(xa, 10, advanced=True) >>> entry (void *)0x406 >>> xa_is_zero(entry) True >>> xa_load(xa, 10) (void *)0
Some features in drgn require architecture-specific support. The current status of this support is:
Architecture | Linux Kernel Modules [1] | Stack Traces [2] | Virtual Address Translation [3] |
x86-64 | ✓ | ✓ | ✓ |
AArch64 | ✓ | ✓ | ✓ |
s390x | ✓ | ✓ | ✓ |
ppc64 | ✓ | ✓ | ✓ |
i386 | ✓ | ||
Arm | ✓ | ||
RISC-V | ✓ |
Key
The listed architectures are recognized in drgn.Architecture. Other architectures are represented by drgn.Architecture.UNKNOWN. Features not mentioned above should work on any architecture, listed or not.
drgn can debug architectures different from the host. For example, you can debug an AArch64 (kernel or userspace) core dump from an x86-64 machine.
drgn officially supports the current mainline, stable, and longterm kernel releases from kernel.org. (There may be some delay before a new mainline version is fully supported.) End-of-life versions are supported until it becomes too difficult to do so. The kernel versions currently fully supported are:
Other versions are not tested. They'll probably mostly work, but support is best-effort.
drgn supports debugging kernels with various configurations:
drgn requires a kernel configured with CONFIG_PROC_KCORE=y for live kernel debugging.
These are writeups of real-world problems solved with drgn.
Author: Omar Sandoval Date: June 9th, 2021
Jakub Kicinski reported a crash in the Kyber I/O scheduler when he was testing Linux 5.12. He captured a core dump and asked me to debug it. This is a quick writeup of that investigation.
First, we can get the task that crashed:
>>> task = per_cpu(prog["runqueues"], prog["crashing_cpu"]).curr
Then, we can get its stack trace:
>>> trace = prog.stack_trace(task) >>> trace #0 queued_spin_lock_slowpath (../kernel/locking/qspinlock.c:471:3) #1 queued_spin_lock (../include/asm-generic/qspinlock.h:85:2) #2 do_raw_spin_lock (../kernel/locking/spinlock_debug.c:113:2) #3 spin_lock (../include/linux/spinlock.h:354:2) #4 kyber_bio_merge (../block/kyber-iosched.c:573:2) #5 blk_mq_sched_bio_merge (../block/blk-mq-sched.h:37:9) #6 blk_mq_submit_bio (../block/blk-mq.c:2182:6) #7 __submit_bio_noacct_mq (../block/blk-core.c:1015:9) #8 submit_bio_noacct (../block/blk-core.c:1048:10) #9 submit_bio (../block/blk-core.c:1125:9) #10 submit_stripe_bio (../fs/btrfs/volumes.c:6553:2) #11 btrfs_map_bio (../fs/btrfs/volumes.c:6642:3) #12 btrfs_submit_data_bio (../fs/btrfs/inode.c:2440:8) #13 submit_one_bio (../fs/btrfs/extent_io.c:175:9) #14 submit_extent_page (../fs/btrfs/extent_io.c:3229:10) #15 __extent_writepage_io (../fs/btrfs/extent_io.c:3793:9) #16 __extent_writepage (../fs/btrfs/extent_io.c:3872:8) #17 extent_write_cache_pages (../fs/btrfs/extent_io.c:4514:10) #18 extent_writepages (../fs/btrfs/extent_io.c:4635:8) #19 do_writepages (../mm/page-writeback.c:2352:10) #20 __writeback_single_inode (../fs/fs-writeback.c:1467:8) #21 writeback_sb_inodes (../fs/fs-writeback.c:1732:3) #22 __writeback_inodes_wb (../fs/fs-writeback.c:1801:12) #23 wb_writeback (../fs/fs-writeback.c:1907:15) #24 wb_check_background_flush (../fs/fs-writeback.c:1975:10) #25 wb_do_writeback (../fs/fs-writeback.c:2063:11) #26 wb_workfn (../fs/fs-writeback.c:2091:20) #27 process_one_work (../kernel/workqueue.c:2275:2) #28 worker_thread (../kernel/workqueue.c:2421:4) #29 kthread (../kernel/kthread.c:292:9) #30 ret_from_fork+0x1f/0x2a (../arch/x86/entry/entry_64.S:294)
It looks like kyber_bio_merge() tried to lock an invalid spinlock. For reference, this is the source code of kyber_bio_merge():
static bool kyber_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio, unsigned int nr_segs) { struct kyber_hctx_data *khd = hctx->sched_data; struct blk_mq_ctx *ctx = blk_mq_get_ctx(hctx->queue); struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw[hctx->type]]; unsigned int sched_domain = kyber_sched_domain(bio->bi_opf); struct list_head *rq_list = &kcq->rq_list[sched_domain]; bool merged; spin_lock(&kcq->lock); merged = blk_bio_list_merge(hctx->queue, rq_list, bio, nr_segs); spin_unlock(&kcq->lock); return merged; }
When printed, the kcq structure containing the spinlock indeed looks like garbage (omitted for brevity).
A crash course on the Linux kernel block layer: for each block device, there is a "software queue" (struct blk_mq_ctx *ctx) for each CPU and a "hardware queue" (struct blk_mq_hw_ctx *hctx) for each I/O queue provided by the device. Each hardware queue has one or more software queues assigned to it. Kyber keeps additional data per hardware queue (struct kyber_hctx_data *khd) and per software queue (struct kyber_ctx_queue *kcq).
Let's try to figure out where the bad kcq came from. It should be an element of the khd->kcqs array (khd is optimized out, but we can recover it from hctx->sched_data):
>>> trace[4]["khd"] (struct kyber_hctx_data *)<absent> >>> hctx = trace[4]["hctx"] >>> khd = cast("struct kyber_hctx_data *", hctx.sched_data) >>> trace[4]["kcq"] - khd.kcqs (ptrdiff_t)1 >>> hctx.nr_ctx (unsigned short)1
So the kcq is for the second software queue, but the hardware queue is only supposed to have one software queue. Let's see which CPU was assigned to the hardware queue:
>>> hctx.ctxs[0].cpu (unsigned int)6
Here's the problem: we're not running on CPU 6, we're running on CPU 19:
>>> prog["crashing_cpu"] (int)19
And CPU 19 is assigned to a different hardware queue that actually does have two software queues:
>>> ctx = per_cpu_ptr(hctx.queue.queue_ctx, 19) >>> other_hctx = ctx.hctxs[hctx.type] >>> other_hctx == hctx False >>> other_hctx.nr_ctx (unsigned short)2
The bug is that the caller gets the hctx for the current CPU, then kyber_bio_merge() gets the ctx for the current CPU, and if the thread is migrated to another CPU in between, they won't match. The fix is to get a consistent view of the hctx and ctx. The commit that fixes this is here.
Most Linux distributions don't install debugging symbols for installed packages by default. This page documents how to install debugging symbols on common distributions. If drgn prints an error like:
$ sudo drgn could not get debugging information for: kernel (could not find vmlinux for 5.14.14-200.fc34.x86_64) ...
Then you need to install debugging symbols.
Fedora makes it very easy to install debugging symbols with the DNF debuginfo-install plugin, which is installed by default. Simply run sudo dnf debuginfo-install $package:
$ sudo dnf debuginfo-install python3
To find out what package owns a binary, use rpm -qf:
$ rpm -qf $(which python3) python3-3.9.7-1.fc34.x86_64
To install symbols for the running kernel:
$ sudo dnf debuginfo-install kernel-$(uname -r)
Also see the Fedora documentation.
Debian requires you to manually add the debugging symbol repositories:
$ sudo tee /etc/apt/sources.list.d/debug.list << EOF deb http://deb.debian.org/debian-debug/ $(lsb_release -cs)-debug main deb http://deb.debian.org/debian-debug/ $(lsb_release -cs)-proposed-updates-debug main EOF $ sudo apt update
Then, debugging symbol packages can be installed with sudo apt install. Some debugging symbol packages are named with a -dbg suffix:
$ sudo apt install python3-dbg
And some are named with a -dbgsym suffix:
$ sudo apt install coreutils-dbgsym
You can use the find-dbgsym-packages command from the debian-goodies package to find the correct name:
$ sudo apt install debian-goodies $ find-dbgsym-packages $(which python3) libc6-dbg libexpat1-dbgsym python3.9-dbg zlib1g-dbgsym $ find-dbgsym-packages $(which cat) coreutils-dbgsym libc6-dbg
To install symbols for the running kernel:
$ sudo apt install linux-image-$(uname -r)-dbg
Also see the Debian documentation.
On Ubuntu, you must install the debugging symbol archive signing key and manually add the debugging symbol repositories:
$ sudo apt update $ sudo apt install ubuntu-dbgsym-keyring $ sudo tee /etc/apt/sources.list.d/debug.list << EOF deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse EOF $ sudo apt update
Like Debian, some debugging symbol packages are named with a -dbg suffix and some are named with a -dbgsym suffix:
$ sudo apt install python3-dbg $ sudo apt install coreutils-dbgsym
You can use the find-dbgsym-packages command from the debian-goodies package to find the correct name:
$ sudo apt install debian-goodies $ find-dbgsym-packages $(which python3) libc6-dbg libexpat1-dbgsym python3.9-dbg zlib1g-dbgsym $ find-dbgsym-packages $(which cat) coreutils-dbgsym libc6-dbg
To install symbols for the running kernel:
$ sudo apt install linux-image-$(uname -r)-dbgsym
Also see the Ubuntu documentation.
Arch Linux unfortunately does not make debugging symbols available. Packages must be manually rebuilt with debugging symbols enabled. See the ArchWiki and the feature request.
These are highlights of each release of drgn focusing on a few exciting items from the full release notes.
These are some of the highlights of drgn 0.0.25. See the GitHub release for the full release notes, including more improvements and bug fixes.
As a usability improvement, prog can now be omitted from most function calls. For example, instead of find_task(prog, 1234), you can now simply write find_task(1234). Additionally, instead of prog.stack_trace(1234), you can now write stack_trace(1234). (The old way will continue to be supported.)
Most CLI users don't need to worry about how this works, but library users may want to understand the Default Program.
It's tricky balancing interactive convenience and sensible APIs for scripting, but we think this is a nice improvement overall!
drgn debugs the live Linux kernel via /proc/kcore, which can only be accessed by the root user (or a user with the CAP_SYS_RAWIO capability, to be precise). However, it's not necessary (or ideal) for the rest of drgn to run as root.
Now when drgn is run against the live kernel as an unprivileged user, it will attempt to open /proc/kcore via sudo(8). The rest of drgn will then run without extra privileges.
In other words, in order to debug the live kernel, all you need to do is install debugging symbols and run:
$ drgn
This feature was contributed by Stephen Brennan.
Maple trees were introduced in Linux 6.1, initially to store virtual memory areas (VMAs). This release adds a couple of helpers for working with them.
mtree_load() looks up an entry in a maple tree:
>>> mtree_load(task.mm.mm_mt.address_of_(), 0x55d65cfaa000) (void *)0xffff97ad82bfc930
mt_for_each() iterates over a maple tree:
>>> for first_index, last_index, entry in mt_for_each(task.mm.mm_mt.address_of_()): ... print(hex(first_index), hex(last_index), entry) ... 0x55d65cfaa000 0x55d65cfaafff (void *)0xffff97ad82bfc930 0x55d65cfab000 0x55d65cfabfff (void *)0xffff97ad82bfc0a8 0x55d65cfac000 0x55d65cfacfff (void *)0xffff97ad82bfc000 0x55d65cfad000 0x55d65cfadfff (void *)0xffff97ad82bfcb28 ...
This release also adds higher-level helpers specifically for VMAs.
vma_find() looks up a VMA by address:
>>> vma_find(task.mm, 0x55d65cfaa000) *(struct vm_area_struct *)0xffff97ad82bfc930 = { ... } >>> vma_find(task.mm, 0x55d65cfa9fff) (struct vm_area_struct *)0
for_each_vma() iterates over every VMA in an address space:
>>> for vma in for_each_vma(task.mm): ... print(vma) ... *(struct vm_area_struct *)0xffff97ad82bfc930 = { ... } *(struct vm_area_struct *)0xffff97ad82bfc0a8 = { ... } ...
These helpers also handle older kernels without maple trees.
Wait queues are a fundamental data structure and synchronization mechanism in the Linux kernel. Imran Khan contributed a few helpers for working with them.
waitqueue_active() returns whether a wait queue has any waiters:
>>> wq *(wait_queue_head_t *)0xffff8da80d618e18 = { .lock = (spinlock_t){ .rlock = (struct raw_spinlock){ .raw_lock = (arch_spinlock_t){ .val = (atomic_t){ .counter = (int)0, }, .locked = (u8)0, .pending = (u8)0, .locked_pending = (u16)0, .tail = (u16)0, }, }, }, .head = (struct list_head){ .next = (struct list_head *)0xffffae44e3007ce8, .prev = (struct list_head *)0xffffae44e3007ce8, }, } >>> waitqueue_active(wq) True
waitqueue_for_each_entry() iterates over each entry in a wait queue:
>>> for entry in waitqueue_for_each_entry(wq): ... print(entry) ... *(wait_queue_entry_t *)0xffffae44e3007cd0 = { .flags = (unsigned int)0, .private = (void *)0xffff8da7863ec000, .func = (wait_queue_func_t)woken_wake_function+0x0 = 0xffffffffa8181010, .entry = (struct list_head){ .next = (struct list_head *)0xffff8da80d618e20, .prev = (struct list_head *)0xffff8da80d618e20, }, }
waitqueue_for_each_task() iterates over each task waiting on a wait queue (although note that this does not work for some special wait queues that don't store tasks):
>>> for task in waitqueue_for_each_task(wq): ... print(task.pid, task.comm) ... (pid_t)294708 (char [16])"zsh"
Sourabh Jain contributed ppc64 radix MMU virtual address translation support. This is the state of architecture support in this release:
Architecture | Linux Kernel Modules | Stack Traces | Virtual Address Translation |
x86-64 | ✓ | ✓ | ✓ |
AArch64 | ✓ | ✓ | ✓ |
s390x | ✓ | ✓ | ✓ |
ppc64 | ✓ | ✓ | ✓ |
i386 | ✓ | ||
Arm | ✓ | ||
RISC-V | ✓ |
These are some of the highlights of drgn 0.0.24. See the GitHub release for the full release notes, including more improvements and bug fixes.
This release added list_count_nodes(), which returns the length of a Linux kernel linked list:
>>> list_count_nodes(prog["workqueues"].address_of_()) 29
This release added a couple of Linux kernel networking helpers requested by Jakub Kicinski.
netdev_priv() returns the private data of a network device:
>>> dev = netdev_get_by_name(prog, "wlp0s20f3") >>> netdev_priv(dev) (void *)0xffff9419c9dec9c0 >>> netdev_priv(dev, "struct ieee80211_sub_if_data") *(struct ieee80211_sub_if_data *)0xffff9419c9dec9c0 = { ... }
skb_shinfo() returns the shared info for a socket buffer.
This release added support for a few C++ features.
Unlike C, C++ allows referring to class, struct, union, and enum types without their respective keywords. For example:
class Foo { ... }; Foo foo; // Equivalent to class Foo foo;
Previously, drgn always required the keyword, so prog.type("class Foo") would succeed but prog.type("Foo") would fail with a LookupError. This requirement was surprising to C++ developers, so it was removed. For C++ programs, prog.type("Foo") will now find a class, struct, union, or enum type named Foo (for C programs, the keyword is still required).
Again unlike C, C++ allows class, struct, and union types to be defined inside of other class, struct, and union types. For example:
class Foo { public: class Bar { ... }; ... }; Foo::Bar bar;
drgn can now find such types with prog.type("Foo::Bar").
C++ supports member functions (a.k.a. methods). For example:
class Foo { int method() { ... } };
drgn can now find member functions with drgn.Program.function(), drgn.Program.object(), or drgn.Program[] (e.g., prog.function("Foo::method") or prog["Foo::method"]).
drgn now supports split DWARF object (.dwo) files. This is enabled by the -gsplit-dwarf option in GCC and Clang or for the Linux kernel with CONFIG_DEBUG_INFO_SPLIT=y.
Split DWARF package (.dwp) file support is still in progress.
Thierry Treyer found a bug that made us search through much more debugging information than necessary when getting a stack trace. Fixing this made stack traces almost twice as fast.
The C++ lookup and split DWARF support mentioned above require processing more information in drgn's debugging information indexing step, which it does on startup and whenever debugging information is manually loaded. This could've been a performance regression, but instead, indexing was reworked from the ground up in a way that's usually faster despite the added features.
These are some of the highlights of drgn 0.0.23. See the GitHub release for the full release notes, including more improvements and bug fixes.
This release added several Linux kernel helpers for translating virtual addresses.
follow_phys() translates a virtual address to a physical address in a given address space. For example, to get the physical address that virtual address 0x7f7fe46a4270 maps to in process 115:
>>> task = find_task(prog, 115) >>> address = 0x7f7fe46a4270 >>> print(hex(follow_phys(task.mm, address))) 0x4090270
follow_page() translates a virtual address to the struct page * that it maps to:
>>> follow_page(task.mm, address) *(struct page *)0xffffd20ac0102400 = { ... }
follow_pfn() translates a virtual address to the page frame number (PFN) of the page that it maps to:
>>> follow_pfn(task.mm, address) (unsigned long)16528
These can be used to translate arbitrary kernel virtual addresses by passing prog["init_mm"].address_of_():
>>> print(hex(follow_phys(prog["init_mm"].address_of_(), 0xffffffffc0483000))) 0x2e4b000
vmalloc_to_page() is a special case of follow_page() for vmalloc and vmap addresses:
>>> vmalloc_to_page(prog, 0xffffffffc0477000) *(struct page *)0xffffc902400b8980 = { ... }
Likewise, vmalloc_to_pfn() is a special case of follow_pfn() for vmalloc and vmap addresses:
>>> vmalloc_to_pfn(prog, 0xffffffffc0477000) (unsigned long)11814
Martin Liška, Boris Burkov, and Johannes Thumshirn added lots of new scripts to the contrib directory:
drgn.cli.run_interactive() runs drgn's interactive mode. It can be used to embed drgn in another application. For example, you could use it for a custom drgn.Program that the standard drgn CLI can't set up:
import drgn import drgn.cli prog = drgn.Program() prog.add_type_finder(...) prog.add_object_finder(...) prog.add_memory_segment(...) drgn.cli.run_interactive(prog)
Sven Schnelle contributed s390x virtual address translation support. This is the state of architecture support in this release:
Architecture | Linux Kernel Modules | Stack Traces | Virtual Address Translation |
x86-64 | ✓ | ✓ | ✓ |
AArch64 | ✓ | ✓ | ✓ |
ppc64 | ✓ | ✓ | |
s390x | ✓ | ✓ | ✓ |
i386 | ✓ | ||
Arm | ✓ | ||
RISC-V | ✓ |
Linux 6.3 and 6.4 had an unusual number of breaking changes for drgn. Here are some errors you might see with older versions of drgn that are fixed in this release.
On startup (fixed by Ido Schimmel):
warning: could not get debugging information for: kernel modules (could not find loaded kernel modules: 'struct module' has no member 'core_size')
From drgn.Program.stack_trace() and drgn.Thread.stack_trace():
Exception: unknown ORC entry type 3
From compound_order() and compound_nr():
AttributeError: 'struct page' has no member 'compound_order'
From for_each_disk() and for_each_partition():
AttributeError: 'struct class' has no member 'p'
Python 3.12, currently in beta, changed an implementation detail that drgn depended on, which caused crashes like:
Py_SIZE: Assertion `ob->ob_type != &PyLong_Type' failed.
Stephen Brennan fixed this.
These are some of the highlights of drgn 0.0.22. See the GitHub release for the full release notes, including more improvements and bug fixes.
drgn.StackFrame.locals() returns the names of all arguments and local variables in the scope of a stack frame. This allows you to get a quick idea of what's going on in a function without needing to read the source code right away.
Let's use the __schedule stack frame from the following trace as an example:
>>> trace = prog.stack_trace(1) >>> trace #0 context_switch (./kernel/sched/core.c:5209:2) #1 __schedule (./kernel/sched/core.c:6521:8) #2 schedule (./kernel/sched/core.c:6597:3) #3 do_wait (./kernel/exit.c:1562:4) #4 kernel_wait4 (./kernel/exit.c:1706:8) #5 __do_sys_wait4 (./kernel/exit.c:1734:13) #6 do_syscall_x64 (./arch/x86/entry/common.c:50:14) #7 do_syscall_64 (./arch/x86/entry/common.c:80:7) #8 entry_SYSCALL_64+0x9b/0x197 (./arch/x86/entry/entry_64.S:120) #9 0x7f6a34a00057 >>> trace[1].locals() ['sched_mode', 'prev', 'next', 'switch_count', 'prev_state', 'rf', 'rq', 'cpu'] >>> for name in trace[1].locals(): ... print(name, trace[1][name].format_(dereference=False)) ... sched_mode (unsigned int)0 prev (struct task_struct *)0xffffa3b601178000 next (struct task_struct *)0xffffa3b6026db800 switch_count (unsigned long *)0xffffa3b601178528 prev_state (unsigned long)<absent> rf (struct rq_flags){ .flags = (unsigned long)1, .cookie = (struct pin_cookie){}, .clock_update_flags = (unsigned int)4, } rq (struct rq *)0xffffa3b67fda9640 cpu (int)<absent>
Compare this to the kernel source code. Note that some of the variables have been optimized out by the compiler.
This feature was contributed by Stephen Brennan.
The Linux kernel slab allocator merges "similar" slab caches as an optimization, which often causes confusion. slab_cache_is_merged() (added back in 0.0.20) returns whether or not a slab cache has been merged, but not what it was merged with. In this release, Stephen Brennan added get_slab_cache_aliases(), which provides a mapping from a slab cache name to the name of the cache it was merged into:
>>> get_slab_cache_aliases(prog) {'io_kiocb': 'maple_node', 'ip_dst_cache': 'uid_cache', 'aio_kiocb': 'uid_cache', 'ip_fib_alias': 'Acpi-Parse', 'pid_namespace': 'pid', 'iommu_iova': 'vmap_area', 'fasync_cache': 'ftrace_event_field', 'dnotify_mark': 'Acpi-State', 'tcp_bind2_bucket': 'vmap_area', 'nsproxy': 'Acpi-Operand', 'shared_policy_node': 'ftrace_event_field', 'eventpoll_epi': 'pid', 'fib6_nodes': 'vmap_area', 'Acpi-Namespace': 'ftrace_event_field', 'posix_timers_cache': 'maple_node', 'inotify_inode_mark': 'Acpi-State', 'kernfs_iattrs_cache': 'trace_event_file', 'fs_cache': 'vmap_area', 'UDP-Lite': 'UDP', 'anon_vma_chain': 'vmap_area', 'ip6_dst_cache': 'maple_node', 'eventpoll_pwq': 'vmap_area', 'inet_peer_cache': 'uid_cache', 'fsnotify_mark_connector': 'numa_policy', 'ip_fib_trie': 'ftrace_event_field', 'filp': 'maple_node', 'dnotify_struct': 'numa_policy', 'UDPLITEv6': 'UDPv6', 'biovec-16': 'maple_node', 'PING': 'signal_cache', 'ep_head': 'blkdev_ioc', 'tcp_bind_bucket': 'pid', 'Acpi-ParseExt': 'Acpi-State', 'cred_jar': 'pid', 'ovl_aio_req': 'pid', 'pool_workqueue': 'maple_node', 'sigqueue': 'Acpi-State', 'file_lock_ctx': 'Acpi-Parse', 'kernfs_node_cache': 'pid'}
This means that if you're looking for io_kiocb allocations, you actually need to look at the maple_node slab cache. Conversely, if you're looking at the maple_node slab cache, you need to be aware that it also contains allocations from all of the following slab caches:
>>> [merged for merged, canonical in get_slab_cache_aliases(prog).items() if canonical == "maple_node"] ['io_kiocb', 'posix_timers_cache', 'ip6_dst_cache', 'filp', 'biovec-16', 'pool_workqueue']
This release extended identify_address() to show additional information about slab allocations:
>>> ptr1 = 0xffffa3b601178438 >>> ptr2 = 0xffffa3b601176cc0 >>> identify_address(prog, ptr1) 'slab object: task_struct+0x438' >>> identify_address(prog, ptr2) 'free slab object: mm_struct+0x0'
This means that ptr1 is an address 0x438 bytes into an allocated object from the task_struct slab cache, and ptr2 is a free object from the mm_struct slab cache.
slab_object_info() provides the same information programmatically:
>>> slab_object_info(prog, ptr1) SlabObjectInfo(slab_cache=Object(prog, 'struct kmem_cache *', value=0xffffa3b601045500), slab=Object(prog, 'struct slab *', value=0xffffe80840045e00), address=0xffffa3b601178000, allocated=True) >>> slab_object_info(prog, ptr2) SlabObjectInfo(slab_cache=Object(prog, 'struct kmem_cache *', value=0xffffa3b601045900), slab=Object(prog, 'struct slab *', value=0xffffe80840045c00), address=0xffffa3b601176cc0, allocated=False)
print_annotated_stack() prints a stack trace and all of its memory, identifying anything that it can:
>>> print_annotated_stack(prog.stack_trace(1)) STACK POINTER VALUE [stack frame #0 at 0xffffffffaf8a68e9 (__schedule+0x429/0x488) in context_switch at ./kernel/sched/core.c:5209:2 (inlined)] [stack frame #1 at 0xffffffffaf8a68e9 (__schedule+0x429/0x488) in __schedule at ./kernel/sched/core.c:6521:8] ffffbb1ac0013d28: ffffffffaf4498f5 [function symbol: __flush_tlb_one_user+0x5] ffffbb1ac0013d30: 00000000af449feb ffffbb1ac0013d38: 0000000000000001 ffffbb1ac0013d40: 0000000000000004 ffffbb1ac0013d48: 25c5ff9539edc200 ffffbb1ac0013d50: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013d58: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013d60: ffffbb1ac0013e10 ffffbb1ac0013d68: ffffa3b601177ff0 [slab object: mm_struct+0x70] ffffbb1ac0013d70: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013d78: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013d80: ffffffffaf8a69d1 [function symbol: schedule+0x89] [stack frame #2 at 0xffffffffaf8a69d1 (schedule+0x89/0xc7) in schedule at ./kernel/sched/core.c:6597:3] ffffbb1ac0013d88: ffffbb1ac0013de8 ffffbb1ac0013d90: 0000000000000000 ffffbb1ac0013d98: ffffffffaf4595ee [function symbol: do_wait+0x231] [stack frame #3 at 0xffffffffaf4595ee (do_wait+0x231/0x2e3) in do_wait at ./kernel/exit.c:1562:4] ffffbb1ac0013da0: ffffa3b601178450 [slab object: task_struct+0x450] ffffbb1ac0013da8: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013db0: 0000000000000004 ffffbb1ac0013db8: 0000000000000000 ffffbb1ac0013dc0: 00007ffe0984a170 ffffbb1ac0013dc8: 0000000000000000 ffffbb1ac0013dd0: fffffffffffffffd ffffbb1ac0013dd8: 0000000000000004 ffffbb1ac0013de0: ffffffffaf45a42f [function symbol: kernel_wait4+0xc2] [stack frame #4 at 0xffffffffaf45a42f (kernel_wait4+0xc2/0x11b) in kernel_wait4 at ./kernel/exit.c:1706:8] ffffbb1ac0013de8: 0000000400000004 ffffbb1ac0013df0: 0000000000000000 ffffbb1ac0013df8: 0000000000000000 ffffbb1ac0013e00: 0000000000000000 ffffbb1ac0013e08: 0000000000000000 ffffbb1ac0013e10: ffffffff00000000 ffffbb1ac0013e18: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013e20: ffffffffaf45890c [function symbol: child_wait_callback+0x0] ffffbb1ac0013e28: ffffa3b601188028 [slab object: signal_cache+0x28] ffffbb1ac0013e30: ffffa3b601188028 [slab object: signal_cache+0x28] ffffbb1ac0013e38: 000055d500000000 ffffbb1ac0013e40: 25c5ff9539edc200 ffffbb1ac0013e48: 0000000000000000 ffffbb1ac0013e50: ffffbb1ac0013f30 ffffbb1ac0013e58: ffffbb1ac0013f58 ffffbb1ac0013e60: 0000000000000000 ffffbb1ac0013e68: 0000000000000000 ffffbb1ac0013e70: 0000000000000000 ffffbb1ac0013e78: ffffffffaf45a4c0 [function symbol: __do_sys_wait4+0x38] [stack frame #5 at 0xffffffffaf45a4c0 (__do_sys_wait4+0x38/0x8c) in __do_sys_wait4 at ./kernel/exit.c:1734:13] ffffbb1ac0013e80: ffffffffaf8aaa21 [function symbol: _raw_spin_unlock_irq+0x10] ffffbb1ac0013e88: ffffffffaf46460c [function symbol: do_sigaction+0xf8] ffffbb1ac0013e90: ffffa3b601180020 [slab object: sighand_cache+0x20] ffffbb1ac0013e98: ffffa3b6028d02d0 [slab object: vm_area_struct+0x0] ffffbb1ac0013ea0: 25c5ff9539edc200 ffffbb1ac0013ea8: 0000000000000002 ffffbb1ac0013eb0: 00007ffe09849fb0 ffffbb1ac0013eb8: ffffbb1ac0013f58 ffffbb1ac0013ec0: 0000000000000000 ffffbb1ac0013ec8: 0000000000000000 ffffbb1ac0013ed0: 0000000000000046 ffffbb1ac0013ed8: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013ee0: ffffa3b601178000 [slab object: task_struct+0x0] ffffbb1ac0013ee8: ffffbb1ac0013f58 ffffbb1ac0013ef0: 0000000000000000 ffffbb1ac0013ef8: ffffffffaf426def [function symbol: fpregs_assert_state_consistent+0x1b] ffffbb1ac0013f00: 0000000000000000 ffffbb1ac0013f08: ffffffffaf4b2f53 [function symbol: exit_to_user_mode_prepare+0xa6] ffffbb1ac0013f10: 0000000000000000 ffffbb1ac0013f18: 25c5ff9539edc200 ffffbb1ac0013f20: ffffbb1ac0013f58 ffffbb1ac0013f28: 0000000000000000 ffffbb1ac0013f30: ffffbb1ac0013f48 ffffbb1ac0013f38: ffffffffaf8a1573 [function symbol: do_syscall_64+0x70] [stack frame #6 at 0xffffffffaf8a1573 (do_syscall_64+0x70/0x8a) in do_syscall_x64 at ./arch/x86/entry/common.c:50:14 (inlined)] [stack frame #7 at 0xffffffffaf8a1573 (do_syscall_64+0x70/0x8a) in do_syscall_64 at ./arch/x86/entry/common.c:80:7] ffffbb1ac0013f40: 0000000000000000 ffffbb1ac0013f48: 0000000000000000 ffffbb1ac0013f50: ffffffffafa0009b [symbol: entry_SYSCALL_64+0x9b] [stack frame #8 at 0xffffffffafa0009b (entry_SYSCALL_64+0x9b/0x197) at ./arch/x86/entry/entry_64.S:120] ffffbb1ac0013f58: 0000000000000000 [stack frame #9 at 0x7f6a34a00057]
Like drgn.StackFrame.locals(), this provides a nice overview of everything happening in a function, which might include useful hints. Keep in mind that it may identify "stale" addresses for anything that a function hasn't reinitialized yet, and as always, be careful of slab cache merging.
This was inspired by the crash bt -FF command. It was contributed by Nhat Pham.
XArrays were introduced in Linux 4.20 as a replacement for radix trees. drgn's radix tree helpers also support XArrays in some cases, but this is awkward, not obvious, and doesn't work for new, XArray-only functionality.
This release added dedicated XArray helpers like xa_load() and xa_for_each().
Sven Schnelle contributed s390x support for Linux kernel modules and stack traces. This is the state of architecture support in this release:
Architecture | Linux Kernel Modules | Stack Traces | Virtual Address Translation |
x86-64 | ✓ | ✓ | ✓ |
AArch64 | ✓ | ✓ | ✓ |
ppc64 | ✓ | ✓ | |
s390x | ✓ | ✓ | |
i386 | ✓ | ||
Arm | ✓ | ||
RISC-V | ✓ |
drgn was originally licensed as GPLv3+. In this release, it was changed to LGPLv2.1+. The motivation for this change was to enable the long term vision for drgn that more projects can use it as a library providing programmatic interfaces for debugger functionality. For example, Object Introspection, a userspace memory profiler recently open sourced by Meta, uses drgn to parse debugging information.
Omar Sandoval
Omar Sandoval
April 1, 2024 | 0.0.25 |