diff options
Diffstat (limited to 'docs/www.kernel.org_doc_html_v4.15_dev-tools_kgdb.txt')
| -rw-r--r-- | docs/www.kernel.org_doc_html_v4.15_dev-tools_kgdb.txt | 1242 |
1 files changed, 1242 insertions, 0 deletions
diff --git a/docs/www.kernel.org_doc_html_v4.15_dev-tools_kgdb.txt b/docs/www.kernel.org_doc_html_v4.15_dev-tools_kgdb.txt new file mode 100644 index 0000000..c1cb07a --- /dev/null +++ b/docs/www.kernel.org_doc_html_v4.15_dev-tools_kgdb.txt @@ -0,0 +1,1242 @@ + #[1]Index [2]Search [3]The Linux Kernel documentation [4]Development + tools for the kernel [5]Linux Kernel Selftests [6]Debugging kernel and + modules via gdb + + [7]The Linux Kernel + 4.15.0 + ____________________ + * [8]The Linux kernel user's and administrator's guide + + * [9]The Linux kernel user-space API guide + + * [10]Working with the kernel development community + * [11]Development tools for the kernel + + [12]Coccinelle + + [13]Sparse + + [14]kcov: code coverage for fuzzing + + [15]Using gcov with the Linux kernel + + [16]The Kernel Address Sanitizer (KASAN) + + [17]The Undefined Behavior Sanitizer - UBSAN + + [18]Kernel Memory Leak Detector + + [19]Debugging kernel and modules via gdb + + [20]Using kgdb, kdb and the kernel debugger internals + o [21]Introduction + o [22]Compiling a kernel + # [23]Kernel config options for kgdb + # [24]Kernel config options for kdb + o [25]Kernel Debugger Boot Arguments + # [26]Kernel parameter: kgdboc + # [27]Kernel parameter: kgdbwait + # [28]Kernel parameter: kgdbcon + # [29]Run time parameter: kgdbreboot + # [30]Kernel parameter: nokaslr + o [31]Using kdb + # [32]Quick start for kdb on a serial port + # [33]Quick start for kdb using a keyboard connected + console + o [34]Using kgdb / gdb + # [35]Connecting with gdb to a serial port + o [36]kgdb and kdb interoperability + # [37]Switching between kdb and kgdb + # [38]Running kdb commands from gdb + o [39]kgdb Test Suite + o [40]Kernel Debugger Internals + # [41]Architecture Specifics + # [42]kgdboc internals + o [43]Credits + + [44]Linux Kernel Selftests + * [45]How to write kernel documentation + * [46]Kernel Hacking Guides + + * [47]The Linux driver implementer's API guide + * [48]Core API Documentation + * [49]Linux Media Subsystem Documentation + * [50]Linux Networking Documentation + * [51]The Linux Input Documentation + * [52]Linux GPU Driver Developer's Guide + * [53]Security Documentation + * [54]Linux Sound Subsystem Documentation + * [55]Linux Kernel Crypto API + * [56]Linux Filesystems API + + * [57]SuperH Interfaces Guide + + * [58]Korean translations + + * [59]Chinese translations + + * [60]Japanese translations + + [61]The Linux Kernel + * [62]Docs » + * [63]Development tools for the kernel » + * Using kgdb, kdb and the kernel debugger internals + * [64]View page source + __________________________________________________________________ + +Using kgdb, kdb and the kernel debugger internals[65]¶ + + Author: Jason Wessel + +Introduction[66]¶ + + The kernel has two different debugger front ends (kdb and kgdb) which + interface to the debug core. It is possible to use either of the + debugger front ends and dynamically transition between them if you + configure the kernel properly at compile and runtime. + + Kdb is simplistic shell-style interface which you can use on a system + console with a keyboard or serial console. You can use it to inspect + memory, registers, process lists, dmesg, and even set breakpoints to + stop in a certain location. Kdb is not a source level debugger, + although you can set breakpoints and execute some basic kernel run + control. Kdb is mainly aimed at doing some analysis to aid in + development or diagnosing kernel problems. You can access some symbols + by name in kernel built-ins or in kernel modules if the code was built + with CONFIG_KALLSYMS. + + Kgdb is intended to be used as a source level debugger for the Linux + kernel. It is used along with gdb to debug a Linux kernel. The + expectation is that gdb can be used to "break in" to the kernel to + inspect memory, variables and look through call stack information + similar to the way an application developer would use gdb to debug an + application. It is possible to place breakpoints in kernel code and + perform some limited execution stepping. + + Two machines are required for using kgdb. One of these machines is a + development machine and the other is the target machine. The kernel to + be debugged runs on the target machine. The development machine runs an + instance of gdb against the vmlinux file which contains the symbols + (not a boot image such as bzImage, zImage, uImage...). In gdb the + developer specifies the connection parameters and connects to kgdb. The + type of connection a developer makes with gdb depends on the + availability of kgdb I/O modules compiled as built-ins or loadable + kernel modules in the test machine's kernel. + +Compiling a kernel[67]¶ + + * In order to enable compilation of kdb, you must first enable kgdb. + * The kgdb test compile options are described in the kgdb test suite + chapter. + +Kernel config options for kgdb[68]¶ + + To enable CONFIG_KGDB you should look under Kernel hacking > Kernel + debugging and select KGDB: kernel debugger. + + While it is not a hard requirement that you have symbols in your + vmlinux file, gdb tends not to be very useful without the symbolic + data, so you will want to turn on CONFIG_DEBUG_INFO which is called + Compile the kernel with debug info in the config menu. + + It is advised, but not required, that you turn on the + CONFIG_FRAME_POINTER kernel option which is called Compile the kernel + with frame pointers in the config menu. This option inserts code to + into the compiled executable which saves the frame information in + registers or on the stack at different points which allows a debugger + such as gdb to more accurately construct stack back traces while + debugging the kernel. + + If the architecture that you are using supports the kernel option + CONFIG_STRICT_KERNEL_RWX, you should consider turning it off. This + option will prevent the use of software breakpoints because it marks + certain regions of the kernel's memory space as read-only. If kgdb + supports it for the architecture you are using, you can use hardware + breakpoints if you desire to run with the CONFIG_STRICT_KERNEL_RWX + option turned on, else you need to turn off this option. + + Next you should choose one of more I/O drivers to interconnect + debugging host and debugged target. Early boot debugging requires a + KGDB I/O driver that supports early debugging and the driver must be + built into the kernel directly. Kgdb I/O driver configuration takes + place via kernel or module parameters which you can learn more about in + the in the section that describes the parameter kgdboc. + + Here is an example set of .config symbols to enable or disable for + kgdb: +# CONFIG_STRICT_KERNEL_RWX is not set +CONFIG_FRAME_POINTER=y +CONFIG_KGDB=y +CONFIG_KGDB_SERIAL_CONSOLE=y + +Kernel config options for kdb[69]¶ + + Kdb is quite a bit more complex than the simple gdbstub sitting on top + of the kernel's debug core. Kdb must implement a shell, and also adds + some helper functions in other parts of the kernel, responsible for + printing out interesting data such as what you would see if you ran + lsmod, or ps. In order to build kdb into the kernel you follow the same + steps as you would for kgdb. + + The main config option for kdb is CONFIG_KGDB_KDB which is called + KGDB_KDB: include kdb frontend for kgdb in the config menu. In theory + you would have already also selected an I/O driver such as the + CONFIG_KGDB_SERIAL_CONSOLE interface if you plan on using kdb on a + serial port, when you were configuring kgdb. + + If you want to use a PS/2-style keyboard with kdb, you would select + CONFIG_KDB_KEYBOARD which is called KGDB_KDB: keyboard as input device + in the config menu. The CONFIG_KDB_KEYBOARD option is not used for + anything in the gdb interface to kgdb. The CONFIG_KDB_KEYBOARD option + only works with kdb. + + Here is an example set of .config symbols to enable/disable kdb: +# CONFIG_STRICT_KERNEL_RWX is not set +CONFIG_FRAME_POINTER=y +CONFIG_KGDB=y +CONFIG_KGDB_SERIAL_CONSOLE=y +CONFIG_KGDB_KDB=y +CONFIG_KDB_KEYBOARD=y + +Kernel Debugger Boot Arguments[70]¶ + + This section describes the various runtime kernel parameters that + affect the configuration of the kernel debugger. The following chapter + covers using kdb and kgdb as well as providing some examples of the + configuration parameters. + +Kernel parameter: kgdboc[71]¶ + + The kgdboc driver was originally an abbreviation meant to stand for + "kgdb over console". Today it is the primary mechanism to configure how + to communicate from gdb to kgdb as well as the devices you want to use + to interact with the kdb shell. + + For kgdb/gdb, kgdboc is designed to work with a single serial port. It + is intended to cover the circumstance where you want to use a serial + console as your primary console as well as using it to perform kernel + debugging. It is also possible to use kgdb on a serial port which is + not designated as a system console. Kgdboc may be configured as a + kernel built-in or a kernel loadable module. You can only make use of + kgdbwait and early debugging if you build kgdboc into the kernel as a + built-in. + + Optionally you can elect to activate kms (Kernel Mode Setting) + integration. When you use kms with kgdboc and you have a video driver + that has atomic mode setting hooks, it is possible to enter the + debugger on the graphics console. When the kernel execution is resumed, + the previous graphics mode will be restored. This integration can serve + as a useful tool to aid in diagnosing crashes or doing analysis of + memory with kdb while allowing the full graphics console applications + to run. + +kgdboc arguments[72]¶ + + Usage: +kgdboc=[kms][[,]kbd][[,]serial_device][,baud] + + The order listed above must be observed if you use any of the optional + configurations together. + + Abbreviations: + * kms = Kernel Mode Setting + * kbd = Keyboard + + You can configure kgdboc to use the keyboard, and/or a serial device + depending on if you are using kdb and/or kgdb, in one of the following + scenarios. The order listed above must be observed if you use any of + the optional configurations together. Using kms + only gdb is generally + not a useful combination. + +Using loadable module or built-in[73]¶ + + 1. As a kernel built-in: + Use the kernel boot argument: +kgdboc=<tty-device>,[baud] + + 2. As a kernel loadable module: + Use the command: +modprobe kgdboc kgdboc=<tty-device>,[baud] + + Here are two examples of how you might format the kgdboc string. + The first is for an x86 target using the first serial port. The + second example is for the ARM Versatile AB using the second serial + port. + 1. kgdboc=ttyS0,115200 + 2. kgdboc=ttyAMA1,115200 + +Configure kgdboc at runtime with sysfs[74]¶ + + At run time you can enable or disable kgdboc by echoing a parameters + into the sysfs. Here are two examples: + 1. Enable kgdboc on ttyS0: +echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc + + 2. Disable kgdboc: +echo "" > /sys/module/kgdboc/parameters/kgdboc + + Note + + You do not need to specify the baud if you are configuring the console + on tty which is already configured or open. + +More examples[75]¶ + + You can configure kgdboc to use the keyboard, and/or a serial device + depending on if you are using kdb and/or kgdb, in one of the following + scenarios. + 1. kdb and kgdb over only a serial port: +kgdboc=<serial_device>[,baud] + + Example: +kgdboc=ttyS0,115200 + + 2. kdb and kgdb with keyboard and a serial port: +kgdboc=kbd,<serial_device>[,baud] + + Example: +kgdboc=kbd,ttyS0,115200 + + 3. kdb with a keyboard: +kgdboc=kbd + + 4. kdb with kernel mode setting: +kgdboc=kms,kbd + + 5. kdb with kernel mode setting and kgdb over a serial port: +kgdboc=kms,kbd,ttyS0,115200 + + Note + + Kgdboc does not support interrupting the target via the gdb remote + protocol. You must manually send a SysRq-G unless you have a proxy that + splits console output to a terminal program. A console proxy has a + separate TCP port for the debugger and a separate TCP port for the + "human" console. The proxy can take care of sending the SysRq-G for + you. + + When using kgdboc with no debugger proxy, you can end up connecting the + debugger at one of two entry points. If an exception occurs after you + have loaded kgdboc, a message should print on the console stating it is + waiting for the debugger. In this case you disconnect your terminal + program and then connect the debugger in its place. If you want to + interrupt the target system and forcibly enter a debug session you have + to issue a Sysrq sequence and then type the letter g. Then you + disconnect the terminal session and connect gdb. Your options if you + don't like this are to hack gdb to send the SysRq-G for you as well as + on the initial connect, or to use a debugger proxy that allows an + unmodified gdb to do the debugging. + +Kernel parameter: kgdbwait[76]¶ + + The Kernel command line option kgdbwait makes kgdb wait for a debugger + connection during booting of a kernel. You can only use this option if + you compiled a kgdb I/O driver into the kernel and you specified the + I/O driver configuration as a kernel command line option. The kgdbwait + parameter should always follow the configuration parameter for the kgdb + I/O driver in the kernel command line else the I/O driver will not be + configured prior to asking the kernel to use it to wait. + + The kernel will stop and wait as early as the I/O driver and + architecture allows when you use this option. If you build the kgdb I/O + driver as a loadable kernel module kgdbwait will not do anything. + +Kernel parameter: kgdbcon[77]¶ + + The kgdbcon feature allows you to see [78]printk() messages inside gdb + while gdb is connected to the kernel. Kdb does not make use of the + kgdbcon feature. + + Kgdb supports using the gdb serial protocol to send console messages to + the debugger when the debugger is connected and running. There are two + ways to activate this feature. + 1. Activate with the kernel command line option: +kgdbcon + + 2. Use sysfs before configuring an I/O driver: +echo 1 > /sys/module/kgdb/parameters/kgdb_use_con + + Note + + If you do this after you configure the kgdb I/O driver, the setting + will not take effect until the next point the I/O is reconfigured. + + Important + + You cannot use kgdboc + kgdbcon on a tty that is an active system + console. An example of incorrect usage is: +console=ttyS0,115200 kgdboc=ttyS0 kgdbcon + + It is possible to use this option with kgdboc on a tty that is not a + system console. + +Run time parameter: kgdbreboot[79]¶ + + The kgdbreboot feature allows you to change how the debugger deals with + the reboot notification. You have 3 choices for the behavior. The + default behavior is always set to 0. + 1 echo -1 > /sys/module/debug_core/parameters/kgdbreboot Ignore the + reboot notification entirely. + 2 echo 0 > /sys/module/debug_core/parameters/kgdbreboot Send the detach + message to any attached debugger client. + 3 echo 1 > /sys/module/debug_core/parameters/kgdbreboot Enter the + debugger on reboot notify. + +Kernel parameter: nokaslr[80]¶ + + If the architecture that you are using enable KASLR by default, you + should consider turning it off. KASLR randomizes the virtual address + where the kernel image is mapped and confuse gdb which resolve kernel + symbol address from symbol table of vmlinux. + +Using kdb[81]¶ + +Quick start for kdb on a serial port[82]¶ + + This is a quick example of how to use kdb. + 1. Configure kgdboc at boot using kernel parameters: +console=ttyS0,115200 kgdboc=ttyS0,115200 nokaslr + + OR + Configure kgdboc after the kernel has booted; assuming you are + using a serial port console: +echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc + + 2. Enter the kernel debugger manually or by waiting for an oops or + fault. There are several ways you can enter the kernel debugger + manually; all involve using the SysRq-G, which means you must have + enabled CONFIG_MAGIC_SysRq=y in your kernel config. + + When logged in as root or with a super user session you can + run: +echo g > /proc/sysrq-trigger + + + Example using minicom 2.2 + Press: CTRL-A f g + + When you have telneted to a terminal server that supports + sending a remote break + Press: CTRL-] + Type in: send break + Press: Enter g + 3. From the kdb prompt you can run the help command to see a complete + list of the commands that are available. + Some useful commands in kdb include: + + lsmod Shows where kernel modules are loaded + ps Displays only the active processes + ps A Shows all the processes + summary Shows kernel version info and memory usage + bt Get a backtrace of the current process using dump_stack() + dmesg View the kernel syslog buffer + go Continue the system + 4. When you are done using kdb you need to consider rebooting the + system or using the go command to resuming normal kernel execution. + If you have paused the kernel for a lengthy period of time, + applications that rely on timely networking or anything to do with + real wall clock time could be adversely affected, so you should + take this into consideration when using the kernel debugger. + +Quick start for kdb using a keyboard connected console[83]¶ + + This is a quick example of how to use kdb with a keyboard. + 1. Configure kgdboc at boot using kernel parameters: +kgdboc=kbd + + OR + Configure kgdboc after the kernel has booted: +echo kbd > /sys/module/kgdboc/parameters/kgdboc + + 2. Enter the kernel debugger manually or by waiting for an oops or + fault. There are several ways you can enter the kernel debugger + manually; all involve using the SysRq-G, which means you must have + enabled CONFIG_MAGIC_SysRq=y in your kernel config. + + When logged in as root or with a super user session you can + run: +echo g > /proc/sysrq-trigger + + + Example using a laptop keyboard: + Press and hold down: Alt + Press and hold down: Fn + Press and release the key with the label: SysRq + Release: Fn + Press and release: g + Release: Alt + + Example using a PS/2 101-key keyboard + Press and hold down: Alt + Press and release the key with the label: SysRq + Press and release: g + Release: Alt + 3. Now type in a kdb command such as help, dmesg, bt or go to continue + kernel execution. + +Using kgdb / gdb[84]¶ + + In order to use kgdb you must activate it by passing configuration + information to one of the kgdb I/O drivers. If you do not pass any + configuration information kgdb will not do anything at all. Kgdb will + only actively hook up to the kernel trap hooks if a kgdb I/O driver is + loaded and configured. If you unconfigure a kgdb I/O driver, kgdb will + unregister all the kernel hook points. + + All kgdb I/O drivers can be reconfigured at run time, if CONFIG_SYSFS + and CONFIG_MODULES are enabled, by echo'ing a new config string to + /sys/module/<driver>/parameter/<option>. The driver can be unconfigured + by passing an empty string. You cannot change the configuration while + the debugger is attached. Make sure to detach the debugger with the + detach command prior to trying to unconfigure a kgdb I/O driver. + +Connecting with gdb to a serial port[85]¶ + + 1. Configure kgdboc + Configure kgdboc at boot using kernel parameters: +kgdboc=ttyS0,115200 + + OR + Configure kgdboc after the kernel has booted: +echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc + + 2. Stop kernel execution (break into the debugger) + In order to connect to gdb via kgdboc, the kernel must first be + stopped. There are several ways to stop the kernel which include + using kgdbwait as a boot argument, via a SysRq-G, or running the + kernel until it takes an exception where it waits for the debugger + to attach. + + When logged in as root or with a super user session you can + run: +echo g > /proc/sysrq-trigger + + + Example using minicom 2.2 + Press: CTRL-A f g + + When you have telneted to a terminal server that supports + sending a remote break + Press: CTRL-] + Type in: send break + Press: Enter g + 3. Connect from gdb + Example (using a directly connected port): +% gdb ./vmlinux +(gdb) set remotebaud 115200 +(gdb) target remote /dev/ttyS0 + + Example (kgdb to a terminal server on TCP port 2012): +% gdb ./vmlinux +(gdb) target remote 192.168.2.2:2012 + + Once connected, you can debug a kernel the way you would debug an + application program. + If you are having problems connecting or something is going + seriously wrong while debugging, it will most often be the case + that you want to enable gdb to be verbose about its target + communications. You do this prior to issuing the target remote + command by typing in: +set debug remote 1 + + Remember if you continue in gdb, and need to "break in" again, you need + to issue an other SysRq-G. It is easy to create a simple entry point by + putting a breakpoint at sys_sync and then you can run sync from a shell + or script to break into the debugger. + +kgdb and kdb interoperability[86]¶ + + It is possible to transition between kdb and kgdb dynamically. The + debug core will remember which you used the last time and automatically + start in the same mode. + +Switching between kdb and kgdb[87]¶ + +Switching from kgdb to kdb[88]¶ + + There are two ways to switch from kgdb to kdb: you can use gdb to issue + a maintenance packet, or you can blindly type the command $3#33. + Whenever the kernel debugger stops in kgdb mode it will print the + message KGDB or $3#33 for KDB. It is important to note that you have to + type the sequence correctly in one pass. You cannot type a backspace or + delete because kgdb will interpret that as part of the debug stream. + 1. Change from kgdb to kdb by blindly typing: +$3#33 + + 2. Change from kgdb to kdb with gdb: +maintenance packet 3 + + Note + Now you must kill gdb. Typically you press CTRL-Z and issue the + command: +kill -9 % + +Change from kdb to kgdb[89]¶ + + There are two ways you can change from kdb to kgdb. You can manually + enter kgdb mode by issuing the kgdb command from the kdb shell prompt, + or you can connect gdb while the kdb shell prompt is active. The kdb + shell looks for the typical first commands that gdb would issue with + the gdb remote protocol and if it sees one of those commands it + automatically changes into kgdb mode. + 1. From kdb issue the command: +kgdb + + Now disconnect your terminal program and connect gdb in its place + 2. At the kdb prompt, disconnect the terminal program and connect gdb + in its place. + +Running kdb commands from gdb[90]¶ + + It is possible to run a limited set of kdb commands from gdb, using the + gdb monitor command. You don't want to execute any of the run control + or breakpoint operations, because it can disrupt the state of the + kernel debugger. You should be using gdb for breakpoints and run + control operations if you have gdb connected. The more useful commands + to run are things like lsmod, dmesg, ps or possibly some of the memory + information commands. To see all the kdb commands you can run monitor + help. + + Example: +(gdb) monitor ps +1 idle process (state I) and +27 sleeping system daemon (state M) processes suppressed, +use 'ps A' to see all. +Task Addr Pid Parent [*] cpu State Thread Command + +0xc78291d0 1 0 0 0 S 0xc7829404 init +0xc7954150 942 1 0 0 S 0xc7954384 dropbear +0xc78789c0 944 1 0 0 S 0xc7878bf4 sh +(gdb) + +kgdb Test Suite[91]¶ + + When kgdb is enabled in the kernel config you can also elect to enable + the config parameter KGDB_TESTS. Turning this on will enable a special + kgdb I/O module which is designed to test the kgdb internal functions. + + The kgdb tests are mainly intended for developers to test the kgdb + internals as well as a tool for developing a new kgdb architecture + specific implementation. These tests are not really for end users of + the Linux kernel. The primary source of documentation would be to look + in the drivers/misc/kgdbts.c file. + + The kgdb test suite can also be configured at compile time to run the + core set of tests by setting the kernel config parameter + KGDB_TESTS_ON_BOOT. This particular option is aimed at automated + regression testing and does not require modifying the kernel boot + config arguments. If this is turned on, the kgdb test suite can be + disabled by specifying kgdbts= as a kernel boot argument. + +Kernel Debugger Internals[92]¶ + +Architecture Specifics[93]¶ + + The kernel debugger is organized into a number of components: + 1. The debug core + The debug core is found in kernel/debugger/debug_core.c. It + contains: + + A generic OS exception handler which includes sync'ing the + processors into a stopped state on an multi-CPU system. + + The API to talk to the kgdb I/O drivers + + The API to make calls to the arch-specific kgdb implementation + + The logic to perform safe memory reads and writes to memory + while using the debugger + + A full implementation for software breakpoints unless + overridden by the arch + + The API to invoke either the kdb or kgdb frontend to the debug + core. + + The structures and callback API for atomic kernel mode + setting. + Note + kgdboc is where the kms callbacks are invoked. + 2. kgdb arch-specific implementation + This implementation is generally found in arch/*/kernel/kgdb.c. As + an example, arch/x86/kernel/kgdb.c contains the specifics to + implement HW breakpoint as well as the initialization to + dynamically register and unregister for the trap handlers on this + architecture. The arch-specific portion implements: + + contains an arch-specific trap catcher which invokes + kgdb_handle_exception() to start kgdb about doing its work + + translation to and from gdb specific packet format to pt_regs + + Registration and unregistration of architecture specific trap + hooks + + Any special exception handling and cleanup + + NMI exception handling and cleanup + + (optional) HW breakpoints + 3. gdbstub frontend (aka kgdb) + The gdbstub is located in kernel/debug/gdbstub.c. It contains: + + All the logic to implement the gdb serial protocol + 4. kdb frontend + The kdb debugger shell is broken down into a number of components. + The kdb core is located in kernel/debug/kdb. There are a number of + helper functions in some of the other kernel components to make it + possible for kdb to examine and report information about the kernel + without taking locks that could cause a kernel deadlock. The kdb + core contains implements the following functionality. + + A simple shell + + The kdb core command set + + A registration API to register additional kdb shell commands. + o A good example of a self-contained kdb module is the + ftdump command for dumping the ftrace buffer. See: + kernel/trace/trace_kdb.c + o For an example of how to dynamically register a new kdb + command you can build the kdb_hello.ko kernel module from + samples/kdb/kdb_hello.c. To build this example you can + set CONFIG_SAMPLES=y and CONFIG_SAMPLE_KDB=m in your + kernel config. Later run modprobe kdb_hello and the next + time you enter the kdb shell, you can run the hello + command. + + The implementation for kdb_printf() which emits messages + directly to I/O drivers, bypassing the kernel log. + + SW / HW breakpoint management for the kdb shell + 5. kgdb I/O driver + Each kgdb I/O driver has to provide an implementation for the + following: + + configuration via built-in or module + + dynamic configuration and kgdb hook registration calls + + read and write character interface + + A cleanup handler for unconfiguring from the kgdb core + + (optional) Early debug methodology + Any given kgdb I/O driver has to operate very closely with the + hardware and must do it in such a way that does not enable + interrupts or change other parts of the system context without + completely restoring them. The kgdb core will repeatedly "poll" a + kgdb I/O driver for characters when it needs input. The I/O driver + is expected to return immediately if there is no data available. + Doing so allows for the future possibility to touch watchdog + hardware in such a way as to have a target system not reset when + these are enabled. + + If you are intent on adding kgdb architecture specific support for a + new architecture, the architecture should define HAVE_ARCH_KGDB in the + architecture specific Kconfig file. This will enable kgdb for the + architecture, and at that point you must create an architecture + specific kgdb implementation. + + There are a few flags which must be set on every architecture in their + asm/kgdb.h file. These are: + * + + NUMREGBYTES: + The size in bytes of all of the registers, so that we can + ensure they will all fit into a packet. + + * + + BUFMAX: + The size in bytes of the buffer GDB will read into. This + must be larger than NUMREGBYTES. + + * + + CACHE_FLUSH_IS_SAFE: + Set to 1 if it is always safe to call flush_cache_range or + flush_icache_range. On some architectures, these functions + may not be safe to call on SMP since we keep other CPUs in + a holding pattern. + + There are also the following functions for the common backend, found in + kernel/kgdb.c, that must be supplied by the architecture-specific + backend unless marked as (optional), in which case a default function + maybe used if the architecture does not need to provide a specific + implementation. + + int kgdb_skipexception(int exception, struct pt_regs * regs)[94]¶ + (optional) exit kgdb_handle_exception early + + Parameters + + int exception + Exception vector number + + struct pt_regs * regs + Current struct pt_regs. + + Description + + On some architectures it is required to skip a breakpoint exception + when it occurs after a breakpoint has been removed. This can be + implemented in the architecture specific portion of kgdb. + + void kgdb_breakpoint(void)[95]¶ + compiled in breakpoint + + Parameters + + void + no arguments + + Description + + This will be implemented as a static inline per architecture. This + function is called by the kgdb core to execute an architecture specific + trap to cause kgdb to enter the exception processing. + + int kgdb_arch_init(void)[96]¶ + Perform any architecture specific initalization. + + Parameters + + void + no arguments + + Description + + This function will handle the initalization of any architecture + specific callbacks. + + void kgdb_arch_exit(void)[97]¶ + Perform any architecture specific uninitalization. + + Parameters + + void + no arguments + + Description + + This function will handle the uninitalization of any architecture + specific callbacks, for dynamic registration and unregistration. + + void pt_regs_to_gdb_regs(unsigned long * gdb_regs, struct pt_regs + * regs)[98]¶ + Convert ptrace regs to GDB regs + + Parameters + + unsigned long * gdb_regs + A pointer to hold the registers in the order GDB wants. + + struct pt_regs * regs + The struct pt_regs of the current process. + + Description + + Convert the pt_regs in regs into the format for registers that GDB + expects, stored in gdb_regs. + + void sleeping_thread_to_gdb_regs(unsigned long * gdb_regs, struct + task_struct * p)[99]¶ + Convert ptrace regs to GDB regs + + Parameters + + unsigned long * gdb_regs + A pointer to hold the registers in the order GDB wants. + + struct task_struct * p + The struct task_struct of the desired process. + + Description + + Convert the register values of the sleeping process in p to the format + that GDB expects. This function is called when kgdb does not have + access to the struct pt_regs and therefore it should fill the gdb + registers gdb_regs with what has been saved in struct thread_struct + thread field during switch_to. + + void gdb_regs_to_pt_regs(unsigned long * gdb_regs, struct pt_regs + * regs)[100]¶ + Convert GDB regs to ptrace regs. + + Parameters + + unsigned long * gdb_regs + A pointer to hold the registers we've received from GDB. + + struct pt_regs * regs + A pointer to a struct pt_regs to hold these values in. + + Description + + Convert the GDB regs in gdb_regs into the pt_regs, and store them in + regs. + + int kgdb_arch_handle_exception(int vector, int signo, int err_code, + char * remcom_in_buffer, char * remcom_out_buffer, struct + pt_regs * regs)[101]¶ + Handle architecture specific GDB packets. + + Parameters + + int vector + The error vector of the exception that happened. + + int signo + The signal number of the exception that happened. + + int err_code + The error code of the exception that happened. + + char * remcom_in_buffer + The buffer of the packet we have read. + + char * remcom_out_buffer + The buffer of BUFMAX bytes to write a packet into. + + struct pt_regs * regs + The struct pt_regs of the current process. + + Description + + This function MUST handle the `c' and `s' command packets, as well + packets to set / remove a hardware breakpoint, if used. If there are + additional packets which the hardware needs to handle, they are handled + here. The code should return -1 if it wants to process more packets, + and a 0 or 1 if it wants to exit from the kgdb callback. + + void kgdb_roundup_cpus(unsigned long flags)[102]¶ + Get other CPUs into a holding pattern + + Parameters + + unsigned long flags + Current IRQ state + + Description + + On SMP systems, we need to get the attention of the other CPUs and get + them into a known state. This should do what is needed to get the other + CPUs to call kgdb_wait(). Note that on some arches, the NMI approach is + not used for rounding up all the CPUs. For example, in case of MIPS, + smp_call_function() is used to roundup CPUs. In this case, we have to + make sure that interrupts are enabled before calling + smp_call_function(). The argument to this function is the flags that + will be used when restoring the interrupts. There is local_irq_save() + call before [103]kgdb_roundup_cpus(). + + On non-SMP systems, this is not called. + + void kgdb_arch_set_pc(struct pt_regs * regs, unsigned long pc)[104]¶ + Generic call back to the program counter + + Parameters + + struct pt_regs * regs + Current struct pt_regs. + + unsigned long pc + The new value for the program counter + + Description + + This function handles updating the program counter and requires an + architecture specific implementation. + + void kgdb_arch_late(void)[105]¶ + Perform any architecture specific initalization. + + Parameters + + void + no arguments + + Description + + This function will handle the late initalization of any architecture + specific callbacks. This is an optional function for handling things + like late initialization of hw breakpoints. The default implementation + does nothing. + + struct kgdb_arch[106]¶ + Describe architecture specific values. + + Definition +struct kgdb_arch { + unsigned char gdb_bpt_instr; + unsigned long flags; + int (* set_breakpoint) (unsigned long, char *); + int (* remove_breakpoint) (unsigned long, char *); + int (* set_hw_breakpoint) (unsigned long, int, enum kgdb_bptype); + int (* remove_hw_breakpoint) (unsigned long, int, enum kgdb_bptype); + void (* disable_hw_break) (struct pt_regs *regs); + void (* remove_all_hw_break) (void); + void (* correct_hw_break) (void); + void (* enable_nmi) (bool on); +}; + + Members + + gdb_bpt_instr + The instruction to trigger a breakpoint. + + flags + Flags for the breakpoint, currently just KGDB_HW_BREAKPOINT. + + set_breakpoint + Allow an architecture to specify how to set a software + breakpoint. + + remove_breakpoint + Allow an architecture to specify how to remove a software + breakpoint. + + set_hw_breakpoint + Allow an architecture to specify how to set a hardware + breakpoint. + + remove_hw_breakpoint + Allow an architecture to specify how to remove a hardware + breakpoint. + + disable_hw_break + Allow an architecture to specify how to disable hardware + breakpoints for a single cpu. + + remove_all_hw_break + Allow an architecture to specify how to remove all hardware + breakpoints. + + correct_hw_break + Allow an architecture to specify how to correct the hardware + debug registers. + + enable_nmi + Manage NMI-triggered entry to KGDB + + struct kgdb_io[107]¶ + Describe the interface for an I/O driver to talk with KGDB. + + Definition +struct kgdb_io { + const char * name; + int (* read_char) (void); + void (* write_char) (u8); + void (* flush) (void); + int (* init) (void); + void (* pre_exception) (void); + void (* post_exception) (void); + int is_console; +}; + + Members + + name + Name of the I/O driver. + + read_char + Pointer to a function that will return one char. + + write_char + Pointer to a function that will write one char. + + flush + Pointer to a function that will flush any pending writes. + + init + Pointer to a function that will initialize the device. + + pre_exception + Pointer to a function that will do any prep work for the I/O + driver. + + post_exception + Pointer to a function that will do any cleanup work for the I/O + driver. + + is_console + 1 if the end device is a console 0 if the I/O device is not a + console + +kgdboc internals[108]¶ + +kgdboc and uarts[109]¶ + + The kgdboc driver is actually a very thin driver that relies on the + underlying low level to the hardware driver having "polling hooks" to + which the tty driver is attached. In the initial implementation of + kgdboc the serial_core was changed to expose a low level UART hook for + doing polled mode reading and writing of a single character while in an + atomic context. When kgdb makes an I/O request to the debugger, kgdboc + invokes a callback in the serial core which in turn uses the callback + in the UART driver. + + When using kgdboc with a UART, the UART driver must implement two + callbacks in the struct uart_ops. Example from drivers/8250.c: +#ifdef CONFIG_CONSOLE_POLL + .poll_get_char = serial8250_get_poll_char, + .poll_put_char = serial8250_put_poll_char, +#endif + + Any implementation specifics around creating a polling driver use the + #ifdef CONFIG_CONSOLE_POLL, as shown above. Keep in mind that polling + hooks have to be implemented in such a way that they can be called from + an atomic context and have to restore the state of the UART chip on + return such that the system can return to normal when the debugger + detaches. You need to be very careful with any kind of lock you + consider, because failing here is most likely going to mean pressing + the reset button. + +kgdboc and keyboards[110]¶ + + The kgdboc driver contains logic to configure communications with an + attached keyboard. The keyboard infrastructure is only compiled into + the kernel when CONFIG_KDB_KEYBOARD=y is set in the kernel + configuration. + + The core polled keyboard driver driver for PS/2 type keyboards is in + drivers/char/kdb_keyboard.c. This driver is hooked into the debug core + when kgdboc populates the callback in the array called + kdb_poll_funcs[]. The kdb_get_kbd_char() is the top-level function + which polls hardware for single character input. + +kgdboc and kms[111]¶ + + The kgdboc driver contains logic to request the graphics display to + switch to a text context when you are using kgdboc=kms,kbd, provided + that you have a video driver which has a frame buffer console and + atomic kernel mode setting support. + + Every time the kernel debugger is entered it calls + kgdboc_pre_exp_handler() which in turn calls con_debug_enter() in the + virtual console layer. On resuming kernel execution, the kernel + debugger calls kgdboc_post_exp_handler() which in turn calls + con_debug_leave(). + + Any video driver that wants to be compatible with the kernel debugger + and the atomic kms callbacks must implement the mode_set_base_atomic, + fb_debug_enter and fb_debug_leave operations. For the fb_debug_enter + and fb_debug_leave the option exists to use the generic drm fb helper + functions or implement something custom for the hardware. The following + example shows the initialization of the .mode_set_base_atomic operation + in drivers/gpu/drm/i915/intel_display.c: +static const struct drm_crtc_helper_funcs intel_helper_funcs = { +[...] + .mode_set_base_atomic = intel_pipe_set_base_atomic, +[...] +}; + + Here is an example of how the i915 driver initializes the + fb_debug_enter and fb_debug_leave functions to use the generic drm + helpers in drivers/gpu/drm/i915/intel_fb.c: +static struct fb_ops intelfb_ops = { +[...] + .fb_debug_enter = drm_fb_helper_debug_enter, + .fb_debug_leave = drm_fb_helper_debug_leave, +[...] +}; + +Credits[112]¶ + + The following people have contributed to this document: + 1. Amit Kale <[113]amitkale@linsyssoft.com> + 2. Tom Rini <[114]trini@kernel.crashing.org> + + In March 2008 this document was completely rewritten by: + * Jason Wessel <[115]jason.wessel@windriver.com> + + In Jan 2010 this document was updated to include kdb. + * Jason Wessel <[116]jason.wessel@windriver.com> + + [117]Next [118]Previous + __________________________________________________________________ + + © Copyright The kernel development community. + Built with [119]Sphinx using a [120]theme provided by [121]Read the + Docs. + +References + + 1. https://www.kernel.org/doc/html/v4.15/genindex.html + 2. https://www.kernel.org/doc/html/v4.15/search.html + 3. https://www.kernel.org/doc/html/v4.15/index.html + 4. https://www.kernel.org/doc/html/v4.15/dev-tools/index.html + 5. https://www.kernel.org/doc/html/v4.15/dev-tools/kselftest.html + 6. https://www.kernel.org/doc/html/v4.15/dev-tools/gdb-kernel-debugging.html + 7. https://www.kernel.org/doc/html/v4.15/index.html + 8. https://www.kernel.org/doc/html/v4.15/admin-guide/index.html + 9. https://www.kernel.org/doc/html/v4.15/userspace-api/index.html + 10. https://www.kernel.org/doc/html/v4.15/process/index.html + 11. https://www.kernel.org/doc/html/v4.15/dev-tools/index.html + 12. https://www.kernel.org/doc/html/v4.15/dev-tools/coccinelle.html + 13. https://www.kernel.org/doc/html/v4.15/dev-tools/sparse.html + 14. https://www.kernel.org/doc/html/v4.15/dev-tools/kcov.html + 15. https://www.kernel.org/doc/html/v4.15/dev-tools/gcov.html + 16. https://www.kernel.org/doc/html/v4.15/dev-tools/kasan.html + 17. https://www.kernel.org/doc/html/v4.15/dev-tools/ubsan.html + 18. https://www.kernel.org/doc/html/v4.15/dev-tools/kmemleak.html + 19. https://www.kernel.org/doc/html/v4.15/dev-tools/gdb-kernel-debugging.html + 20. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html + 21. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#introduction + 22. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#compiling-a-kernel + 23. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-config-options-for-kgdb + 24. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-config-options-for-kdb + 25. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-debugger-boot-arguments + 26. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-kgdboc + 27. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-kgdbwait + 28. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-kgdbcon + 29. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#run-time-parameter-kgdbreboot + 30. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-nokaslr + 31. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#using-kdb + 32. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#quick-start-for-kdb-on-a-serial-port + 33. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#quick-start-for-kdb-using-a-keyboard-connected-console + 34. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#using-kgdb-gdb + 35. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#connecting-with-gdb-to-a-serial-port + 36. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdb-and-kdb-interoperability + 37. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#switching-between-kdb-and-kgdb + 38. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#running-kdb-commands-from-gdb + 39. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdb-test-suite + 40. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-debugger-internals + 41. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#architecture-specifics + 42. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdboc-internals + 43. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#credits + 44. https://www.kernel.org/doc/html/v4.15/dev-tools/kselftest.html + 45. https://www.kernel.org/doc/html/v4.15/doc-guide/index.html + 46. https://www.kernel.org/doc/html/v4.15/kernel-hacking/index.html + 47. https://www.kernel.org/doc/html/v4.15/driver-api/index.html + 48. https://www.kernel.org/doc/html/v4.15/core-api/index.html + 49. https://www.kernel.org/doc/html/v4.15/media/index.html + 50. https://www.kernel.org/doc/html/v4.15/networking/index.html + 51. https://www.kernel.org/doc/html/v4.15/input/index.html + 52. https://www.kernel.org/doc/html/v4.15/gpu/index.html + 53. https://www.kernel.org/doc/html/v4.15/security/index.html + 54. https://www.kernel.org/doc/html/v4.15/sound/index.html + 55. https://www.kernel.org/doc/html/v4.15/crypto/index.html + 56. https://www.kernel.org/doc/html/v4.15/filesystems/index.html + 57. https://www.kernel.org/doc/html/v4.15/sh/index.html + 58. https://www.kernel.org/doc/html/v4.15/translations/ko_KR/index.html + 59. https://www.kernel.org/doc/html/v4.15/translations/zh_CN/index.html + 60. https://www.kernel.org/doc/html/v4.15/translations/ja_JP/index.html + 61. https://www.kernel.org/doc/html/v4.15/index.html + 62. https://www.kernel.org/doc/html/v4.15/index.html + 63. https://www.kernel.org/doc/html/v4.15/dev-tools/index.html + 64. https://www.kernel.org/doc/html/v4.15/_sources/dev-tools/kgdb.rst.txt + 65. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#using-kgdb-kdb-and-the-kernel-debugger-internals + 66. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#introduction + 67. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#compiling-a-kernel + 68. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-config-options-for-kgdb + 69. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-config-options-for-kdb + 70. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-debugger-boot-arguments + 71. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-kgdboc + 72. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdboc-arguments + 73. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#using-loadable-module-or-built-in + 74. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#configure-kgdboc-at-runtime-with-sysfs + 75. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#more-examples + 76. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-kgdbwait + 77. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-kgdbcon + 78. https://www.kernel.org/doc/html/v4.15/driver-api/basics.html#c.printk + 79. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#run-time-parameter-kgdbreboot + 80. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-parameter-nokaslr + 81. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#using-kdb + 82. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#quick-start-for-kdb-on-a-serial-port + 83. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#quick-start-for-kdb-using-a-keyboard-connected-console + 84. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#using-kgdb-gdb + 85. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#connecting-with-gdb-to-a-serial-port + 86. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdb-and-kdb-interoperability + 87. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#switching-between-kdb-and-kgdb + 88. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#switching-from-kgdb-to-kdb + 89. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#change-from-kdb-to-kgdb + 90. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#running-kdb-commands-from-gdb + 91. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdb-test-suite + 92. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kernel-debugger-internals + 93. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#architecture-specifics + 94. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_skipexception + 95. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_breakpoint + 96. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_arch_init + 97. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_arch_exit + 98. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.pt_regs_to_gdb_regs + 99. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.sleeping_thread_to_gdb_regs + 100. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.gdb_regs_to_pt_regs + 101. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_arch_handle_exception + 102. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_roundup_cpus + 103. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_roundup_cpus + 104. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_arch_set_pc + 105. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_arch_late + 106. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_arch + 107. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#c.kgdb_io + 108. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdboc-internals + 109. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdboc-and-uarts + 110. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdboc-and-keyboards + 111. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#kgdboc-and-kms + 112. https://www.kernel.org/doc/html/v4.15/dev-tools/kgdb.html#credits + 113. mailto:amitkale%40linsyssoft.com + 114. mailto:trini%40kernel.crashing.org + 115. mailto:jason.wessel%40windriver.com + 116. mailto:jason.wessel%40windriver.com + 117. https://www.kernel.org/doc/html/v4.15/dev-tools/kselftest.html + 118. https://www.kernel.org/doc/html/v4.15/dev-tools/gdb-kernel-debugging.html + 119. http://sphinx-doc.org/ + 120. https://github.com/snide/sphinx_rtd_theme + 121. https://readthedocs.org/ |