Debugging Techniques in Linux
Duration: 24 hours

Back to course list Home
Synopsis This course introduces debugging tools in Linux. Powerful utilities ubiquitous in Linux distributions, such as strace/ltrace, proc, and - the built-in debugger, gdb. We discuss how to develop and use debugging skills to deal with common bugs, such as memory corruptions, erroneous usage of pointers, multi-threaded code and race-conditions, socket bugs, and more. Special consideration is given to analyzing core dumps and crashes. The techniques shown are applicable across all flavors of Linux - including Android distributions (at the CLI level)
Target Audience Application developers, and/or anyone interested in obtaining deeper insights into ekeing out the maximum from the Linux OS
Prerequisites
  • Knowledge of user mode programming. Familiarity with POSIX (unistd.h) and system calls
Objectives
  • Use common Linux performance monitoring tools
  • Use Linux's built-in debugging tools - strace, ltrace, gdb, and others
  • Understand and effectively use GCC's myriad compiler optimizations
  • Effectively code multi-threaded programs
Exercises This course allocates plenty of time for hands-on practice.
The hands-on exercises include:
  • Debugging user mode crash dumps ("core" files)
  • Writing, profiling and debugging code
Modules
1. Introduction to Linux Architecture
4 hours
Introduction to the Linux architecture. We discuss the design and implementation of the Linux Kernel and its various subsystems, at a modular "black box" level, without going into the source code level. We discuss the Kernel architecture and tunable parameters that allow for performance optimization.
  • Linux Kernel features
    • Version differences - from 2.6.x, through 5.11
      • User-Mode and Kernel-Mode Architecture
        • The Linux Task Scheduler (CFS)
          • The Linux I/O architecture:
            • I/O Schedulers
              • Buffer cache and page cache
              • The /proc filesystem
                • The /sys filesystem
                  2. Compiling, Linking and Debugging
                  3 hours
                  		In-depth discussion of code at the assembly (machine code) level. Students will learn:
                  • Familiarity with assembly code - x86_64 and ARM64
                    • GCC compiler optimizations
                      • Compiler level tricks - Preprocessor
                        • Compiler level tricks - Debugging Info
                          • Compiler level tricks - Inline Assembly
                            • Using GDB - Tips & Tricks
                              3. Processes and Threads
                              2 hours
                              		The inside view of Linux "tasks" - processes and threads - and how the scheduler manages. Using process diagnostic tools effectively.
                              • Processes
                                • Daemons
                                  • Threads
                                    • Priorities and Nice
                                      • The kernel perspective - clone()
                                        • Process scheduler states
                                          • Real-Time priorities
                                            4. Process Tracing & Hooking
                                            2 hours
                                            		Methods for code-injection into running processes via libraries and the ptrace(2) API.
                                            • Library calls vs. system calls - when to use which?
                                              • Libraries
                                                • Compiling a shared library
                                                  • Injecting libraries - load time
                                                    • Injecting libraries - runtime
                                                    • System call tracing with ptrace
                                                      • The ptrace(2) API
                                                        • System call level tracing with strace(1)
                                                          • Library call level tracing with ltrace(1)
                                                            • Advanced tracing with jtrace
                                                            • gprof
                                                              • valgrind

                                                                This module demonstrates the following tools:
                                                                • Strace
                                                                • LTrace
                                                                • JTrace
                                                                • Valgrind
                                                                5. Virtual Memory
                                                                3 hours
                                                                		A deeper look into virtual memory :
                                                                • Process memory layout
                                                                  • System memory management
                                                                    • Page/Buffer Cache maintenance
                                                                      • Kernel tunables
                                                                        • Kernel optimizations (KSM, compressed memory, etc)
                                                                          6. Memory management bugs
                                                                          3 hours
                                                                          		A detailed discussion of memory allocation/management bugs, and how they manifest themselves in stack and heap corruption.
                                                                          • Stack overrun
                                                                            • Stack buffer overflows
                                                                              • Heap buffer overflows
                                                                                • Double Free bugs
                                                                                  • Uninitialized memory
                                                                                    • Use after free
                                                                                      • Memory leaks

                                                                                        This module demonstrates the following tools:
                                                                                        • Val/Cal/MemGrind
                                                                                        • Mcheck
                                                                                        7. Concurrency
                                                                                        3 hours
                                                                                        		A discussion of the challenges involved with writing multi-threaded and multi-process code.
                                                                                        • Concurrency
                                                                                          • Inter-Process Communication (IPC) techniques
                                                                                            • Locking techniques
                                                                                              • Types of locks
                                                                                                • Optimizing locks
                                                                                                  • Deadlocks
                                                                                                  • GDB and multi-threaded programs
                                                                                                    8. Kernel-Level Debugging for User-mode
                                                                                                    2 hours
                                                                                                    		Basic knowledge of Kernel-level coding techniques offers a plethora of new debugging techniques. While this is not a full-fledged Linux Kernel course, we introduce the basic concepts for a quick-and-dirty module, to allow for the injection of code into the Kernel.
                                                                                                    • Why Kernel? The benefits of writing a module:
                                                                                                      • System call interception
                                                                                                        • Precise timing measurements
                                                                                                          • Faking system calls - Virtualizing or stress testing
                                                                                                          • Writing a basic module
                                                                                                            • KProbes
                                                                                                              • UProbes (3.5+)