Changes between Initial Version and Version 1 of troubleshooting


Ignore:
Timestamp:
10/31/2017 09:32:21 PM (7 years ago)
Author:
Tim Harvey
Comment:

restored html from 2017/10/13 cache

Legend:

Unmodified
Added
Removed
Modified

  • troubleshooting

    v1 v1  
     1{{{#!html
     2
     3<h1 id="LinuxTroubleshooting"><b style="color:#000;background:#ffcc99">Linux Troubleshooting</b></h1>
     4<p>
     5See also:
     6</p>
     7<ul><li><a class="wiki" href="/wiki/linux/profiling"><b style="color:#000;background:#ffff66">linux</b>/profiling</a>
     8</li></ul><h2 id="KernelDebugging">Kernel Debugging</h2>
     9<p>
     10<span class="wikianchor" id="dmesg"></span>
     11</p>
     12<h3 id="ringbuffer">ring buffer</h3>
     13<p>
     14The kernel ring buffer is where various messages are sent by the kernel and its modules (ie debug/info/warn) as long as CONFIG_PRINTK is enabled. You can also enable CONFIG_PRINTK_TIME to enable timestamps on these messages.
     15</p>
     16<p>
     17Typically an application such as syslog will pull messages from here and write them to a log, however the <tt>dmesg</tt> command will display them directly from the ring buffer:
     18</p>
     19<pre class="wiki">dmesg # display all messages
     20dmesg -c  # display and clear all messages
     21</pre><p>
     22<span class="wikianchor" id="printk"></span>
     23</p>
     24<h3 id="printkandloglevels">printk and log levels</h3>
     25<p>
     26According to <b style="color:#000;background:#ffff66">Linux</b> Device Drivers, the most common debugging technique is monitoring which is done by calling printf at suitable points. In the kernel you accomplish the same with printk.
     27</p>
     28<p>
     29The kernel defines various log levels or severity levels which are strings that you can pre-pend to debug statements defined in <a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/linux/include/linux/kern_levels.h"><span class="icon">​</span><b style="color:#000;background:#ffff66">linux</b>/include/<b style="color:#000;background:#ffff66">linux</b>/kern_levels.h</a>:
     30</p>
     31<ul><li>KERN_EMERG (0) - system is unusable
     32</li><li>KERN_ALERT (1) - actino must be taken immediately
     33</li><li>KERN_CRIT (2) - critical conditions
     34</li><li>KERN_ERR (3) - error conditions
     35</li><li>KERN_WARNING (4) - warning conditions
     36</li><li>KERN_NOTICE (5) - normal but significant condition
     37</li><li>KERN_INFO (6) - informational
     38</li><li>KERN_DEBUG (7) - debug-level messages
     39</li><li>KERN_DEFAULT (d) - the default kernel loglevel
     40</li></ul><p>
     41To enable various severity levels you can:
     42</p>
     43<ol><li>set DEFAULT_MESSAGE_LOGLEVEL in kernel at build time
     44</li><li>use /proc/sys/kernel/printk:
     45<pre class="wiki">$ cat /proc/sys/kernel/printk
     46        7       4       1       7
     47        current default minimum boot-time-default
     48$ echo 8 &gt; /proc/sys/kernel/printk  # enable highest level
     49</pre></li><li>pass the <strong>debug</strong> kernel commandline option
     50</li></ol><p>
     51References:
     52</p>
     53<ul><li><a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/printk-formats.txt"><span class="icon">​</span>Documentation/printk-formats.txt</a>
     54</li><li><a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/linux/include/linux/kern_levels.h"><span class="icon">​</span><b style="color:#000;background:#ffff66">linux</b>/include/<b style="color:#000;background:#ffff66">linux</b>/kern_levels.h</a>
     55</li><li><a class="ext-link" href="http://elinux.org/Debugging_by_printing#Log_Levels"><span class="icon">​</span>Debugging by printing</a>
     56</li></ul><p>
     57<span class="wikianchor" id="pr-debug"></span>
     58</p>
     59<h3 id="pr-debuganddev_dbg">pr-debug() and dev_dbg()</h3>
     60<p>
     61Instead of using printk() directly it is recommended to use the pr_debug() and dev_dbg() macros for the following reasons:
     62</p>
     63<ul><li>they use <a class="wiki" href="/wiki/linux/troubleshooting#dynamic-debugging">dynamic debugging</a> (if enabled in the kernel, otherwise those are just printk's) which avoids performance hits caused by rendering the printk's regardless of loglevel tuning
     64</li><li>they can display device-specific details and/or can have easily formatted prefixes
     65</li></ul><p>
     66<span class="wikianchor" id="dynamic-debug"></span>
     67</p>
     68<h3 id="dynamicdebug">dynamic debug</h3>
     69<p>
     70Regardless of the system log level each printk() must still being rendered in the kernel (formatted with locking buffers) which is very expensive performance-wise. Dynamic Debug (if enabled via CONFIG_DYNAMIC_DEBUG) will instead conditionally skip debugs before they are rendered. Dynamic Debug allows you to enable/disable various debug messages at runtime based on various levels of scope:
     71</p>
     72<ul><li>per source file
     73</li><li>per function
     74</li><li>per module
     75</li><li>format string
     76</li><li>line number
     77</li></ul><p>
     78The <tt>CONFIG_DYNAMIC_DEBUG</tt> option implicitly compiles in all pr_debug() and dev_dbg() calls which can enlarge the kernel text size by about 2%. Note that it does not work with printk and if not enabled in the kernel all pr_debug() dev_dbg() calls simply become static printk's.
     79</p>
     80<p>
     81If a source file is compiled with the DEBUG flag set (which can be turned on by many CONFIG_*DEBUG* options), any pr_debug() calls in it are enabled by default, but can be disabled at runtime.
     82</p>
     83<p>
     84Dynamic debugging is controlled by the 'dynamic_debug/control' file in the debugfs filesystem. The following examples assume that debugfs is enabled and mounted to /sys/kernel/debug.
     85</p>
     86<p>
     87The format of each line of the file is: <tt>filename:lineno [module]function flags format</tt>:
     88</p>
     89<ul><li>filename : source file of the debug statement
     90</li><li>lineno : line number of the debug statement
     91</li><li>module : module that contains the debug statement
     92</li><li>function : function that contains the debug statement
     93</li><li>flags : '+p' means the line is turned 'on' for printing
     94</li><li>format : the format used for the debug statement
     95</li></ul><p>
     96For details and examples please see the kernel <a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/dynamic-debug-howto.txt"><span class="icon">​</span>Documentation/dynamic-debug-howto.txt</a>
     97</p>
     98<p>
     99References:
     100</p>
     101<ul><li><a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/dynamic-debug-howto.txt"><span class="icon">​</span>Documentation/dynamic-debug-howto.txt</a>
     102</li><li><a class="ext-link" href="https://www.kernel.org/doc/ols/2009/ols2009-pages-39-46.pdf"><span class="icon">​</span>Dynamic Debug by Jason Baron of Red Hat (2009)</a>
     103</li></ul><p>
     104<span class="wikianchor" id="early-serial-debug"></span>
     105</p>
     106<h2 id="EarlySerialDebug">Early Serial Debug</h2>
     107<p>
     108Typically the serial drivers don't get initialized until after some of the more critical components. This can result in any kernel crash/hang not reporting anything from the serial console after <strong>Starting kernel ...</strong>.
     109</p>
     110<p>
     111For platforms that early serial support you can enable this in the kernel config. For example on a Ventana board (all ventana boards use ttymxc1 (UART2) for serial console):
     112</p>
     113<ul><li>make menuconfig
     114<ul><li>Kernel hacking -&gt; Kernel low-level debugging functions -&gt; Kernel low-level debugging port (i.MX6Q/DL Debug UART)
     115</li><li>Kernel hacking -&gt; Kernel low-level debugging functions -&gt; i.MX Debug UART Port Selection : 2
     116</li></ul></li><li>results in the following additions to .config:
     117<pre class="wiki">CONFIG_DEBUG_LL=y
     118CONFIG_DEBUG_IMX6Q_UART=y
     119CONFIG_DEBUG_IMX_UART_PORT=2
     120CONFIG_DEBUG_LL_INCLUDE="debug/imx.S"
     121CONFIG_DEBUG_UNCOMPRESS=y
     122CONFIG_EARLY_PRINTK=y
     123CONFIG_EARLY_PRINTK_DIRECT=y
     124</pre></li></ul><ul><li>Finally, some kernels require a special parameter to be passed in through the bootloader. See below for instructions:
     125<pre class="wiki">setenv extra ${extra} earlyprintk=serial,${console},${baudrate}
     126saveenv
     127</pre></li></ul><p>
     128<span class="wikianchor" id="sysrq"></span>
     129</p>
     130<h2 id="LinuxMagicSystemRequestKeyHacks"><b style="color:#000;background:#ffff66">Linux</b> Magic System Request Key Hacks</h2>
     131<p>
     132There exists a magic key combo you can hit which the kernel will respond to regardless of whatever else it is doing (unless its completely locked up).
     133</p>
     134<p>
     135To enable this build with CONFIG_MAGIC_SYSRQ when configuring the kernel.
     136</p>
     137<p>
     138This will enable a /proc/sysrq-trigger file which you can use to send various commands to. Some very useful commands for debugging are:
     139</p>
     140<ul><li>'b' - reboot immediately (without sync)
     141</li><li>'c' - perform a system crash via a NULL pointer dereference (which will show a crashdump if enabled)
     142</li><li>'d' - show all locks that are held
     143</li><li>'e' - send a SIGTERM to all processes except for PID1 (init)
     144</li><li>'f' - call oom_kill to kill a memory hog process
     145</li><li>'h' - display help
     146</li><li>'i' - send a SIGKILL to all processes except for PID1 (init)
     147</li><li>'l' - show a stack backtrace for all active CPU's
     148</li></ul><p>
     149References:
     150</p>
     151<ul><li><a class="ext-link" href="https://www.kernel.org/doc/Documentation/sysrq.txt"><span class="icon">​</span>https://www.kernel.org/doc/Documentation/sysrq.txt</a>
     152</li></ul></div>
     153         
     154          <div class="trac-modifiedby">
     155            <span><a href="/wiki/linux/troubleshooting?action=diff&amp;version=6" title="Version 6 by psidhu: fix flag">Last modified</a> <a class="timeline" href="/timeline?from=2016-01-11T16%3A04%3A46-08%3A00&amp;precision=second" title="See timeline at 01/11/16 16:04:46">21 months ago</a></span>
     156            <span class="trac-print">Last modified on 01/11/16 16:04:46</span>
     157          </div>
     158       
     159       
     160      </div>
     161     
     162
     163    </div>
     164    <div id="altlinks">
     165      <h3>Download in other formats:</h3>
     166      <ul>
     167        <li class="last first">
     168          <a rel="nofollow" href="/wiki/linux/troubleshooting?format=txt">Plain Text</a>
     169        </li>
     170      </ul>
     171    </div>
     172    </div>
     173    <div id="footer" lang="en" xml:lang="en"><hr />
     174      <a id="tracpowered" href="http://trac.edgewall.org/"><img src="/chrome/common/trac_logo_mini.png" height="30" width="107" alt="Trac Powered" /></a>
     175      <p class="left">Powered by <a href="/about"><strong>Trac 1.0</strong></a><br />
     176        By <a href="http://www.edgewall.org/">Edgewall Software</a>.</p>
     177      <p class="right">Visit the Trac open source project at<br /><a href="http://trac.edgewall.org/">http://trac.edgewall.org/</a></p>
     178    </div>
     179}}}