Linux Kernel Debugging
Duration: 3 days

Back to course list Home
Synopsis This course, a companion course to "Debugging Techniques in Linux" focuses on debugging and tracing in the kernel environment. Using Linux's own interfaces and built in tools , it provides techniques to analyze crash dumps, as well as intercept potential errors in kernel module code before they result in crashes.
Target Audience Application developers, and/or anyone interested in obtaining deeper insights into ekeing out the maximum from the Linux OS
Prerequisites
Objectives
  • Understand the Linux Kernel architecture from a fault-finding perspective
  • Quickly navigate kernel memory and extract runtime information
  • Use the /proc and /sys filesystems to get statistics and information on kernel and modules
  • Trace the kernel activity with a variety of tools and mechanisms, like ftrace and perf
  • Analyze and determine cause of crash dumps
Exercises This course is a workshop, allocating at least 33% of the time for hands-on practice.
The hands-on exercises include:
  • Debugging a kernel mode crash dumps ("Panics")
  • Compiling the kernel with debug options
  • Using ftrace for a live module
Modules
1. Refresher: The Linux Architecture
2 hours
Introduction to the Linux architecture. We discuss the design and implementation of the Linux Kernel and its various subsystems, at a modular "black box" level, without going into the source code level. We discuss the Kernel architecture and tunable parameters that allow for performance optimization.
  • Linux Kernel features
    • Version differences - 2.6 through 3.17 subversions
      • User-Mode and Kernel-Mode Architecture
        • The Linux Task Scheduler (process manager)
          • The Linux I/O Schedulers:
            • Elevator/NoOp
              • DeadLine
                • Anticipatory
                  • Complete Fair Queueing (CFQ)
                  2. Knobs & levers: The /proc and /sys filesystems
                  2 hours
                  		In many cases, /proc and /sys provide a plethora of information which may be useful to quickly diagnose a problem in a running kernel. Topics include:
                  • /proc/kallsyms and /proc/kcore
                    • Virtual memory information files (slabinfo, vmallocinfo, and more)
                      • Hardware diagnostics in /sys
                        • sysctl settings used for debugging in /sys/kernel and /sys/vm
                          3. Kernel-Level Debugging
                          1 hours
                          		Kernel-level coding techniques offers a plethora of new debugging techniques. While this is not a full-fledged Linux Kernel course, we introduce the common concepts for a quick-and-dirty module, to allow for the injection of code into the Kernel, with specific focus on debugging.
                          • Why Kernel? The benefits of writing a module:
                            • System call interception
                              • Precise timing measurements
                                • Faking system calls - Virtualizing or stress testing
                                • Interfacing through proc/sys with module parameters and pseudo-files
                                  4. Compiling for debugging
                                  2 hours
                                  		The Linux kernel offers a plethora of debug options, but the really useful ones often require recompilation. We focus on the various settings, including:
                                  • Debug Symbols and a quick recap of the kernel source
                                    • "Kernel hacking" options
                                      5. Using kprobes
                                      2 hours
                                      		Using the kprobes mechanism (as of 2.6.9), a module can intercept access to any kernel code, logging as well as modifying its behavior.
                                      6. Tracing with ftrace and kernel events
                                      2 hours
                                      		Linux's ftrace mechanism is second to none, and allows investigation of kernel code flow using a simple and effective interface. We focus on utilizing ftrace and tweaking some of its lesser known options, including tracing by events, different tracer types, and even custom tracers.
                                      7. Post Mortem Debugging
                                      2 hours
                                      		When bugs can't be intercepted in time, the kernel will often panic. We discuss how to analyze various faults in kernel space from panics and oops messages, including:
                                      • BUG() oops and Panic macros
                                        • Figuring out output in dmesg()
                                          • Configuring crash dumps
                                            • Analyze crash dumps
                                              8. Kernel Network Debugging
                                              2 hours
                                              		Simple, yet surprisingly effective techniques to debug networking (socket) code. We introduce socket level API hooking, and packet level hooking (via NetFilter).

                                              • struct sock - The Kernel implementation of sockets
                                                • struct sk_buff - Socket buffers
                                                  • struct proto - the implementations of TCP and UDP
                                                    • NetFilter hooks