Changes between Initial Version and Version 1 of troubleshooting

10/31/2017 09:32:21 PM (5 years ago)
Tim Harvey

restored html from 2017/10/13 cache


  • troubleshooting

    v1 v1  
     3<h1 id="LinuxTroubleshooting"><b style="color:#000;background:#ffcc99">Linux Troubleshooting</b></h1>
     5See also:
     7<ul><li><a class="wiki" href="/wiki/linux/profiling"><b style="color:#000;background:#ffff66">linux</b>/profiling</a>
     8</li></ul><h2 id="KernelDebugging">Kernel Debugging</h2>
     10<span class="wikianchor" id="dmesg"></span>
     12<h3 id="ringbuffer">ring buffer</h3>
     14The kernel ring buffer is where various messages are sent by the kernel and its modules (ie debug/info/warn) as long as CONFIG_PRINTK is enabled. You can also enable CONFIG_PRINTK_TIME to enable timestamps on these messages.
     17Typically an application such as syslog will pull messages from here and write them to a log, however the <tt>dmesg</tt> command will display them directly from the ring buffer:
     19<pre class="wiki">dmesg # display all messages
     20dmesg -c  # display and clear all messages
     22<span class="wikianchor" id="printk"></span>
     24<h3 id="printkandloglevels">printk and log levels</h3>
     26According to <b style="color:#000;background:#ffff66">Linux</b> Device Drivers, the most common debugging technique is monitoring which is done by calling printf at suitable points. In the kernel you accomplish the same with printk.
     29The kernel defines various log levels or severity levels which are strings that you can pre-pend to debug statements defined in <a class="ext-link" href=""><span class="icon">​</span><b style="color:#000;background:#ffff66">linux</b>/include/<b style="color:#000;background:#ffff66">linux</b>/kern_levels.h</a>:
     31<ul><li>KERN_EMERG (0) - system is unusable
     32</li><li>KERN_ALERT (1) - actino must be taken immediately
     33</li><li>KERN_CRIT (2) - critical conditions
     34</li><li>KERN_ERR (3) - error conditions
     35</li><li>KERN_WARNING (4) - warning conditions
     36</li><li>KERN_NOTICE (5) - normal but significant condition
     37</li><li>KERN_INFO (6) - informational
     38</li><li>KERN_DEBUG (7) - debug-level messages
     39</li><li>KERN_DEFAULT (d) - the default kernel loglevel
     41To enable various severity levels you can:
     43<ol><li>set DEFAULT_MESSAGE_LOGLEVEL in kernel at build time
     44</li><li>use /proc/sys/kernel/printk:
     45<pre class="wiki">$ cat /proc/sys/kernel/printk
     46        7       4       1       7
     47        current default minimum boot-time-default
     48$ echo 8 &gt; /proc/sys/kernel/printk  # enable highest level
     49</pre></li><li>pass the <strong>debug</strong> kernel commandline option
     53<ul><li><a class="ext-link" href=""><span class="icon">​</span>Documentation/printk-formats.txt</a>
     54</li><li><a class="ext-link" href=""><span class="icon">​</span><b style="color:#000;background:#ffff66">linux</b>/include/<b style="color:#000;background:#ffff66">linux</b>/kern_levels.h</a>
     55</li><li><a class="ext-link" href=""><span class="icon">​</span>Debugging by printing</a>
     57<span class="wikianchor" id="pr-debug"></span>
     59<h3 id="pr-debuganddev_dbg">pr-debug() and dev_dbg()</h3>
     61Instead of using printk() directly it is recommended to use the pr_debug() and dev_dbg() macros for the following reasons:
     63<ul><li>they use <a class="wiki" href="/wiki/linux/troubleshooting#dynamic-debugging">dynamic debugging</a> (if enabled in the kernel, otherwise those are just printk's) which avoids performance hits caused by rendering the printk's regardless of loglevel tuning
     64</li><li>they can display device-specific details and/or can have easily formatted prefixes
     66<span class="wikianchor" id="dynamic-debug"></span>
     68<h3 id="dynamicdebug">dynamic debug</h3>
     70Regardless of the system log level each printk() must still being rendered in the kernel (formatted with locking buffers) which is very expensive performance-wise. Dynamic Debug (if enabled via CONFIG_DYNAMIC_DEBUG) will instead conditionally skip debugs before they are rendered. Dynamic Debug allows you to enable/disable various debug messages at runtime based on various levels of scope:
     72<ul><li>per source file
     73</li><li>per function
     74</li><li>per module
     75</li><li>format string
     76</li><li>line number
     78The <tt>CONFIG_DYNAMIC_DEBUG</tt> option implicitly compiles in all pr_debug() and dev_dbg() calls which can enlarge the kernel text size by about 2%. Note that it does not work with printk and if not enabled in the kernel all pr_debug() dev_dbg() calls simply become static printk's.
     81If a source file is compiled with the DEBUG flag set (which can be turned on by many CONFIG_*DEBUG* options), any pr_debug() calls in it are enabled by default, but can be disabled at runtime.
     84Dynamic debugging is controlled by the 'dynamic_debug/control' file in the debugfs filesystem. The following examples assume that debugfs is enabled and mounted to /sys/kernel/debug.
     87The format of each line of the file is: <tt>filename:lineno [module]function flags format</tt>:
     89<ul><li>filename : source file of the debug statement
     90</li><li>lineno : line number of the debug statement
     91</li><li>module : module that contains the debug statement
     92</li><li>function : function that contains the debug statement
     93</li><li>flags : '+p' means the line is turned 'on' for printing
     94</li><li>format : the format used for the debug statement
     96For details and examples please see the kernel <a class="ext-link" href=""><span class="icon">​</span>Documentation/dynamic-debug-howto.txt</a>
     101<ul><li><a class="ext-link" href=""><span class="icon">​</span>Documentation/dynamic-debug-howto.txt</a>
     102</li><li><a class="ext-link" href=""><span class="icon">​</span>Dynamic Debug by Jason Baron of Red Hat (2009)</a>
     104<span class="wikianchor" id="early-serial-debug"></span>
     106<h2 id="EarlySerialDebug">Early Serial Debug</h2>
     108Typically the serial drivers don't get initialized until after some of the more critical components. This can result in any kernel crash/hang not reporting anything from the serial console after <strong>Starting kernel ...</strong>.
     111For platforms that early serial support you can enable this in the kernel config. For example on a Ventana board (all ventana boards use ttymxc1 (UART2) for serial console):
     113<ul><li>make menuconfig
     114<ul><li>Kernel hacking -&gt; Kernel low-level debugging functions -&gt; Kernel low-level debugging port (i.MX6Q/DL Debug UART)
     115</li><li>Kernel hacking -&gt; Kernel low-level debugging functions -&gt; i.MX Debug UART Port Selection : 2
     116</li></ul></li><li>results in the following additions to .config:
     117<pre class="wiki">CONFIG_DEBUG_LL=y
     124</pre></li></ul><ul><li>Finally, some kernels require a special parameter to be passed in through the bootloader. See below for instructions:
     125<pre class="wiki">setenv extra ${extra} earlyprintk=serial,${console},${baudrate}
     128<span class="wikianchor" id="sysrq"></span>
     130<h2 id="LinuxMagicSystemRequestKeyHacks"><b style="color:#000;background:#ffff66">Linux</b> Magic System Request Key Hacks</h2>
     132There exists a magic key combo you can hit which the kernel will respond to regardless of whatever else it is doing (unless its completely locked up).
     135To enable this build with CONFIG_MAGIC_SYSRQ when configuring the kernel.
     138This will enable a /proc/sysrq-trigger file which you can use to send various commands to. Some very useful commands for debugging are:
     140<ul><li>'b' - reboot immediately (without sync)
     141</li><li>'c' - perform a system crash via a NULL pointer dereference (which will show a crashdump if enabled)
     142</li><li>'d' - show all locks that are held
     143</li><li>'e' - send a SIGTERM to all processes except for PID1 (init)
     144</li><li>'f' - call oom_kill to kill a memory hog process
     145</li><li>'h' - display help
     146</li><li>'i' - send a SIGKILL to all processes except for PID1 (init)
     147</li><li>'l' - show a stack backtrace for all active CPU's
     151<ul><li><a class="ext-link" href=""><span class="icon">​</span></a>
     154          <div class="trac-modifiedby">
     155            <span><a href="/wiki/linux/troubleshooting?action=diff&amp;version=6" title="Version 6 by psidhu: fix flag">Last modified</a> <a class="timeline" href="/timeline?from=2016-01-11T16%3A04%3A46-08%3A00&amp;precision=second" title="See timeline at 01/11/16 16:04:46">21 months ago</a></span>
     156            <span class="trac-print">Last modified on 01/11/16 16:04:46</span>
     157          </div>
     160      </div>
     163    </div>
     164    <div id="altlinks">
     165      <h3>Download in other formats:</h3>
     166      <ul>
     167        <li class="last first">
     168          <a rel="nofollow" href="/wiki/linux/troubleshooting?format=txt">Plain Text</a>
     169        </li>
     170      </ul>
     171    </div>
     172    </div>
     173    <div id="footer" lang="en" xml:lang="en"><hr />
     174      <a id="tracpowered" href=""><img src="/chrome/common/trac_logo_mini.png" height="30" width="107" alt="Trac Powered" /></a>
     175      <p class="left">Powered by <a href="/about"><strong>Trac 1.0</strong></a><br />
     176        By <a href="">Edgewall Software</a>.</p>
     177      <p class="right">Visit the Trac open source project at<br /><a href=""></a></p>
     178    </div>