{{{#!html

Linux Troubleshooting

See also:

Kernel Debugging

ring buffer

The kernel ring buffer is where various messages are sent by the kernel and its modules (ie debug/info/warn) as long as CONFIG_PRINTK is enabled. You can also enable CONFIG_PRINTK_TIME to enable timestamps on these messages.

Typically an application such as syslog will pull messages from here and write them to a log, however the dmesg command will display them directly from the ring buffer: 

dmesg # display all messages
dmesg -c  # display and clear all messages

printk and log levels

According to Linux Device Drivers, the most common debugging technique is monitoring which is done by calling printf at suitable points. In the kernel you accomplish the same with printk.

The kernel defines various log levels or severity levels which are strings that you can pre-pend to debug statements defined in ​linux/include/linux/kern_levels.h:

To enable various severity levels you can:

  1. set DEFAULT_MESSAGE_LOGLEVEL in kernel at build time
  2. use /proc/sys/kernel/printk: 
    $ cat /proc/sys/kernel/printk
	7       4       1       7
	current	default	minimum	boot-time-default
$ echo 8 > /proc/sys/kernel/printk  # enable highest level
  3. pass the debug kernel commandline option

References:

pr-debug() and dev_dbg()

Instead of using printk() directly it is recommended to use the pr_debug() and dev_dbg() macros for the following reasons:

dynamic debug

Regardless of the system log level each printk() must still being rendered in the kernel (formatted with locking buffers) which is very expensive performance-wise. Dynamic Debug (if enabled via CONFIG_DYNAMIC_DEBUG) will instead conditionally skip debugs before they are rendered. Dynamic Debug allows you to enable/disable various debug messages at runtime based on various levels of scope:

The CONFIG_DYNAMIC_DEBUG option implicitly compiles in all pr_debug() and dev_dbg() calls which can enlarge the kernel text size by about 2%. Note that it does not work with printk and if not enabled in the kernel all pr_debug() dev_dbg() calls simply become static printk's.

If a source file is compiled with the DEBUG flag set (which can be turned on by many CONFIG_*DEBUG* options), any pr_debug() calls in it are enabled by default, but can be disabled at runtime.

Dynamic debugging is controlled by the 'dynamic_debug/control' file in the debugfs filesystem. The following examples assume that debugfs is enabled and mounted to /sys/kernel/debug.

The format of each line of the file is: filename:lineno [module]function flags format:

For details and examples please see the kernel ​Documentation/dynamic-debug-howto.txt

References:

Early Serial Debug

Typically the serial drivers don't get initialized until after some of the more critical components. This can result in any kernel crash/hang not reporting anything from the serial console after Starting kernel ....

For platforms that early serial support you can enable this in the kernel config. For example on a Ventana board (all ventana boards use ttymxc1 (UART2) for serial console):

Linux Magic System Request Key Hacks

There exists a magic key combo you can hit which the kernel will respond to regardless of whatever else it is doing (unless its completely locked up).

To enable this build with CONFIG_MAGIC_SYSRQ when configuring the kernel.

This will enable a /proc/sysrq-trigger file which you can use to send various commands to. Some very useful commands for debugging are:

References:

Last modified 21 months ago Last modified on 01/11/16 16:04:46
}}}