$ 

Debugging Techniques in Linux -- Duration: 24 hours
----------------------------------------------
Synopsis

This course introduces debugging tools in Linux. Powerful utilities ubiquitous in Linux distributions, such as strace/ltrace, proc, and - the built-in debugger, gdb. We discuss how to develop and use debugging skills to deal with common bugs, such as memory corruptions, erroneous usage of pointers, multi-threaded code and race-conditions, socket bugs, and more. Special consideration is given to analyzing core dumps and crashes. The techniques shown are applicable across all flavors of Linux - including Android distributions (at the CLI level).

Target Audience

Application developers, and/or anyone interested in obtaining deeper insights into ekeing out the maximum from the Linux OS

This course is NOT intended for user mode developers who wish to develop GUI applications or use the Android Java SDKs - but can be a great recommended followup for those who already do.

Prerequisites

Knowledge of user mode programming. Familiarity with POSIX (unistd.h) and system calls.

Objectives Exercises

This course allocates plenty of time for hands-on practice. The hands-on exercises include:

MODULES
1. Introduction to Linux Architecture
4 hours
Introduction to the Linux architecture. We discuss the design and implementation of the Linux Kernel and its various subsystems, at a modular "black box" level, without going into the source code level. We discuss the Kernel architecture and tunable parameters that allow for performance optimization.
  • Linux Kernel features
    • Version differences - from 2.6.x, through 5.11
      • User-Mode and Kernel-Mode Architecture
        • The Linux Task Scheduler (CFS)
          • The Linux I/O architecture:
            • I/O Schedulers
              • Buffer cache and page cache
              • The /proc filesystem
                • The /sys filesystem
                  2. Compiling, Linking and Debugging
                  3 hours
                  		In-depth discussion of code at the assembly (machine code) level. Students will learn:
                  • Familiarity with assembly code - x86_64 and ARM64
                    • GCC compiler optimizations
                      • Compiler level tricks - Preprocessor
                        • Compiler level tricks - Debugging Info
                          • Compiler level tricks - Inline Assembly
                            • Using GDB - Tips & Tricks
                              3. Processes and Threads
                              2 hours
                              		The inside view of Linux "tasks" - processes and threads - and how the scheduler manages. Using process diagnostic tools effectively.
                              • Processes
                                • Daemons
                                  • Threads
                                    • Priorities and Nice
                                      • The kernel perspective - clone()
                                        • Process scheduler states
                                          • Real-Time priorities
                                            4. Process Tracing & Hooking
                                            2 hours
                                            		Methods for code-injection into running processes via libraries and the ptrace(2) API.
                                            • Library calls vs. system calls - when to use which?
                                              • Libraries
                                                • Compiling a shared library
                                                  • Injecting libraries - load time
                                                    • Injecting libraries - runtime
                                                    • System call tracing with ptrace
                                                      • The ptrace(2) API
                                                        • System call level tracing with strace(1)
                                                          • Library call level tracing with ltrace(1)
                                                            • Advanced tracing with jtrace
                                                            • gprof
                                                              • valgrind

                                                                This module demonstrates the following tools:
                                                                • Strace
                                                                • LTrace
                                                                • JTrace
                                                                • Valgrind
                                                                5. Virtual Memory
                                                                3 hours
                                                                		A deeper look into virtual memory :
                                                                • Process memory layout
                                                                  • System memory management
                                                                    • Page/Buffer Cache maintenance
                                                                      • Kernel tunables
                                                                        • Kernel optimizations (KSM, compressed memory, etc)
                                                                          6. Memory management bugs
                                                                          3 hours
                                                                          		A detailed discussion of memory allocation/management bugs, and how they manifest themselves in stack and heap corruption.
                                                                          • Stack overrun
                                                                            • Stack buffer overflows
                                                                              • Heap buffer overflows
                                                                                • Double Free bugs
                                                                                  • Uninitialized memory
                                                                                    • Use after free
                                                                                      • Memory leaks

                                                                                        This module demonstrates the following tools:
                                                                                        • Val/Cal/MemGrind
                                                                                        • Mcheck
                                                                                        7. Concurrency
                                                                                        3 hours
                                                                                        		A discussion of the challenges involved with writing multi-threaded and multi-process code.
                                                                                        • Concurrency
                                                                                          • Inter-Process Communication (IPC) techniques
                                                                                            • Locking techniques
                                                                                              • Types of locks
                                                                                                • Optimizing locks
                                                                                                  • Deadlocks
                                                                                                  • GDB and multi-threaded programs
                                                                                                    8. Kernel-Level Debugging for User-mode
                                                                                                    2 hours
                                                                                                    		Basic knowledge of Kernel-level coding techniques offers a plethora of new debugging techniques. While this is not a full-fledged Linux Kernel course, we introduce the basic concepts for a quick-and-dirty module, to allow for the injection of code into the Kernel.
                                                                                                    • Why Kernel? The benefits of writing a module:
                                                                                                      • System call interception
                                                                                                        • Precise timing measurements
                                                                                                          • Faking system calls - Virtualizing or stress testing
                                                                                                          • Writing a basic module
                                                                                                            • KProbes
                                                                                                              • UProbes (3.5+)


                                                                                                                ------------------------------------------------

                                                                                                                 Completed in 0.01s