Changes between Version 1 and Version 2 of troubleshooting


Ignore:
Timestamp:
01/09/2018 11:27:13 PM (6 years ago)
Author:
Tim Harvey
Comment:

convert restored html to wiki markup

Legend:

Unmodified
Added
Removed
Modified

  • troubleshooting

    v1 v2  
    1 {{{#!html
     1[[PageOutline]]
    22
    3 <h1 id="LinuxTroubleshooting"><b style="color:#000;background:#ffcc99">Linux Troubleshooting</b></h1>
    4 <p>
     3= Linux Troubleshooting =
     4
    55See also:
    6 </p>
    7 <ul><li><a class="wiki" href="/wiki/linux/profiling"><b style="color:#000;background:#ffff66">linux</b>/profiling</a>
    8 </li></ul><h2 id="KernelDebugging">Kernel Debugging</h2>
    9 <p>
    10 <span class="wikianchor" id="dmesg"></span>
    11 </p>
    12 <h3 id="ringbuffer">ring buffer</h3>
    13 <p>
    14 The kernel ring buffer is where various messages are sent by the kernel and its modules (ie debug/info/warn) as long as CONFIG_PRINTK is enabled. You can also enable CONFIG_PRINTK_TIME to enable timestamps on these messages.
    15 </p>
    16 <p>
    17 Typically an application such as syslog will pull messages from here and write them to a log, however the <tt>dmesg</tt> command will display them directly from the ring buffer:
    18 </p>
    19 <pre class="wiki">dmesg # display all messages
     6 * [wiki:linux/profiling linux/profiling]
     7
     8== Kernel Debugging ==
     9
     10[=#dmesg]
     11=== ring buffer ===
     12The kernel ring buffer is where various messages are sent by the kernel and its modules (ie debug/info/warn) as long as {{{CONFIG_PRINTK}}} is enabled. You can also enable {{{CONFIG_PRINTK_TIME}}} to enable timestamps on these messages.
     13
     14Typically an application such as {{{syslog}}} will pull messages from here and write them to a log, however the {{{dmesg}}} command will display them directly from the ring buffer:
     15{{{#!bash
     16dmesg # display all messages
    2017dmesg -c  # display and clear all messages
    21 </pre><p>
    22 <span class="wikianchor" id="printk"></span>
    23 </p>
    24 <h3 id="printkandloglevels">printk and log levels</h3>
    25 <p>
    26 According to <b style="color:#000;background:#ffff66">Linux</b> Device Drivers, the most common debugging technique is monitoring which is done by calling printf at suitable points. In the kernel you accomplish the same with printk.
    27 </p>
    28 <p>
    29 The kernel defines various log levels or severity levels which are strings that you can pre-pend to debug statements defined in <a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/linux/include/linux/kern_levels.h"><span class="icon">​</span><b style="color:#000;background:#ffff66">linux</b>/include/<b style="color:#000;background:#ffff66">linux</b>/kern_levels.h</a>:
    30 </p>
    31 <ul><li>KERN_EMERG (0) - system is unusable
    32 </li><li>KERN_ALERT (1) - actino must be taken immediately
    33 </li><li>KERN_CRIT (2) - critical conditions
    34 </li><li>KERN_ERR (3) - error conditions
    35 </li><li>KERN_WARNING (4) - warning conditions
    36 </li><li>KERN_NOTICE (5) - normal but significant condition
    37 </li><li>KERN_INFO (6) - informational
    38 </li><li>KERN_DEBUG (7) - debug-level messages
    39 </li><li>KERN_DEFAULT (d) - the default kernel loglevel
    40 </li></ul><p>
     18}}}
     19
     20[=#printk]
     21=== printk and log levels ===
     22According to Linux Device Drivers, the most common debugging technique is monitoring which is done by calling {{{printf}}} at suitable points. In the kernel you accomplish the same with {{{printk}}}.
     23
     24The kernel defines various log levels or severity levels which are strings that you can pre-pend to debug statements defined in ​{{{linux/include/linux/kern_levels.h}}}:
     25* KERN_EMERG (0) - system is unusable
     26* KERN_ALERT (1) - actino must be taken immediately
     27* KERN_CRIT (2) - critical conditions
     28* KERN_ERR (3) - error conditions
     29* KERN_WARNING (4) - warning conditions
     30* KERN_NOTICE (5) - normal but significant condition
     31* KERN_INFO (6) - informational
     32* KERN_DEBUG (7) - debug-level messages
     33* KERN_DEFAULT (d) - the default kernel loglevel
     34
    4135To enable various severity levels you can:
    42 </p>
    43 <ol><li>set DEFAULT_MESSAGE_LOGLEVEL in kernel at build time
    44 </li><li>use /proc/sys/kernel/printk:
    45 <pre class="wiki">$ cat /proc/sys/kernel/printk
     361. set {{{DEFAULT_MESSAGE_LOGLEVEL}}} in kernel at build time
     372. use {{{/proc/sys/kernel/printk}}}:
     38{{{#!bash
     39$ cat /proc/sys/kernel/printk
    4640        7       4       1       7
    4741        current default minimum boot-time-default
    48 $ echo 8 &gt; /proc/sys/kernel/printk  # enable highest level
    49 </pre></li><li>pass the <strong>debug</strong> kernel commandline option
    50 </li></ol><p>
     42$ echo 8 > /proc/sys/kernel/printk  # enable highest level
     43}}}
     443. pass the {{{debug}}} kernel commandline option
     45
    5146References:
    52 </p>
    53 <ul><li><a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/printk-formats.txt"><span class="icon">​</span>Documentation/printk-formats.txt</a>
    54 </li><li><a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/linux/include/linux/kern_levels.h"><span class="icon">​</span><b style="color:#000;background:#ffff66">linux</b>/include/<b style="color:#000;background:#ffff66">linux</b>/kern_levels.h</a>
    55 </li><li><a class="ext-link" href="http://elinux.org/Debugging_by_printing#Log_Levels"><span class="icon">​</span>Debugging by printing</a>
    56 </li></ul><p>
    57 <span class="wikianchor" id="pr-debug"></span>
    58 </p>
    59 <h3 id="pr-debuganddev_dbg">pr-debug() and dev_dbg()</h3>
    60 <p>
    61 Instead of using printk() directly it is recommended to use the pr_debug() and dev_dbg() macros for the following reasons:
    62 </p>
    63 <ul><li>they use <a class="wiki" href="/wiki/linux/troubleshooting#dynamic-debugging">dynamic debugging</a> (if enabled in the kernel, otherwise those are just printk's) which avoids performance hits caused by rendering the printk's regardless of loglevel tuning
    64 </li><li>they can display device-specific details and/or can have easily formatted prefixes
    65 </li></ul><p>
    66 <span class="wikianchor" id="dynamic-debug"></span>
    67 </p>
    68 <h3 id="dynamicdebug">dynamic debug</h3>
    69 <p>
     47* ​[http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/printk-formats.txt Documentation/printk-formats.txt]
     48* [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/linux/include/linux/kern_levels.h linux/include/linux/kern_levels.h]
     49* [http://elinux.org/Debugging_by_printing#Log_Levels Debugging by printing]
     50
     51
     52[=#pr-debug]
     53=== pr-debug() and dev_dbg() ===
     54Instead of using {{{printk()}}} directly it is recommended to use the {{{pr_debug()}}} and {{{dev_dbg()}}} macros for the following reasons:
     55* they use dynamic debugging (if enabled in the kernel, otherwise those are just printk's) which avoids performance hits caused by rendering the printk's regardless of loglevel tuning
     56* they can display device-specific details and/or can have easily formatted prefixes
     57
     58[=#dynamic]
     59=== dynamic debug ===
    7060Regardless of the system log level each printk() must still being rendered in the kernel (formatted with locking buffers) which is very expensive performance-wise. Dynamic Debug (if enabled via CONFIG_DYNAMIC_DEBUG) will instead conditionally skip debugs before they are rendered. Dynamic Debug allows you to enable/disable various debug messages at runtime based on various levels of scope:
    71 </p>
    72 <ul><li>per source file
    73 </li><li>per function
    74 </li><li>per module
    75 </li><li>format string
    76 </li><li>line number
    77 </li></ul><p>
    78 The <tt>CONFIG_DYNAMIC_DEBUG</tt> option implicitly compiles in all pr_debug() and dev_dbg() calls which can enlarge the kernel text size by about 2%. Note that it does not work with printk and if not enabled in the kernel all pr_debug() dev_dbg() calls simply become static printk's.
    79 </p>
    80 <p>
     61* per source file
     62* per function
     63* per module
     64* format string
     65* line number
     66
     67The {{{CONFIG_DYNAMIC_DEBUG}}} option implicitly compiles in all {{{pr_debug()}}} and {{{dev_dbg()}}} calls which can enlarge the kernel text size by about 2%. Note that it does not work with {{{printk()}}} and if not enabled in the kernel all {{{pr_debug()}}} {{{dev_dbg()}}} calls simply become static printk's.
     68
    8169If a source file is compiled with the DEBUG flag set (which can be turned on by many CONFIG_*DEBUG* options), any pr_debug() calls in it are enabled by default, but can be disabled at runtime.
    82 </p>
    83 <p>
     70
    8471Dynamic debugging is controlled by the 'dynamic_debug/control' file in the debugfs filesystem. The following examples assume that debugfs is enabled and mounted to /sys/kernel/debug.
    85 </p>
    86 <p>
    87 The format of each line of the file is: <tt>filename:lineno [module]function flags format</tt>:
    88 </p>
    89 <ul><li>filename : source file of the debug statement
    90 </li><li>lineno : line number of the debug statement
    91 </li><li>module : module that contains the debug statement
    92 </li><li>function : function that contains the debug statement
    93 </li><li>flags : '+p' means the line is turned 'on' for printing
    94 </li><li>format : the format used for the debug statement
    95 </li></ul><p>
    96 For details and examples please see the kernel <a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/dynamic-debug-howto.txt"><span class="icon">​</span>Documentation/dynamic-debug-howto.txt</a>
    97 </p>
    98 <p>
     72
     73The format of each line of the file is: {{{filename:lineno [module]function flags format}}}:
     74* filename : source file of the debug statement
     75* lineno : line number of the debug statement
     76* module : module that contains the debug statement
     77* function : function that contains the debug statement
     78* flags : '+p' means the line is turned 'on' for printing
     79* format : the format used for the debug statement
     80
     81For details and examples please see the kernel ​Documentation/dynamic-debug-howto.txt
     82
    9983References:
    100 </p>
    101 <ul><li><a class="ext-link" href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/dynamic-debug-howto.txt"><span class="icon">​</span>Documentation/dynamic-debug-howto.txt</a>
    102 </li><li><a class="ext-link" href="https://www.kernel.org/doc/ols/2009/ols2009-pages-39-46.pdf"><span class="icon">​</span>Dynamic Debug by Jason Baron of Red Hat (2009)</a>
    103 </li></ul><p>
    104 <span class="wikianchor" id="early-serial-debug"></span>
    105 </p>
    106 <h2 id="EarlySerialDebug">Early Serial Debug</h2>
    107 <p>
    108 Typically the serial drivers don't get initialized until after some of the more critical components. This can result in any kernel crash/hang not reporting anything from the serial console after <strong>Starting kernel ...</strong>.
    109 </p>
    110 <p>
     84* [http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/dynamic-debug-howto.txt Documentation/dynamic-debug-howto.txt]
     85​* [https://www.kernel.org/doc/ols/2009/ols2009-pages-39-46.pdf Dynamic Debug by Jason Baron of Red Hat (2009)]
     86
     87== Early Serial Debug ==
     88Typically the serial drivers don't get initialized until after some of the more critical components. This can result in any kernel crash/hang not reporting anything from the serial console after {{{'Starting kernel ....'}}}
     89
    11190For platforms that early serial support you can enable this in the kernel config. For example on a Ventana board (all ventana boards use ttymxc1 (UART2) for serial console):
    112 </p>
    113 <ul><li>make menuconfig
    114 <ul><li>Kernel hacking -&gt; Kernel low-level debugging functions -&gt; Kernel low-level debugging port (i.MX6Q/DL Debug UART)
    115 </li><li>Kernel hacking -&gt; Kernel low-level debugging functions -&gt; i.MX Debug UART Port Selection : 2
    116 </li></ul></li><li>results in the following additions to .config:
    117 <pre class="wiki">CONFIG_DEBUG_LL=y
     91 * make menuconfig
     92  - Kernel hacking -> Kernel low-level debugging functions -> Kernel low-level debugging port (i.MX6Q/DL Debug UART)
     93  - Kernel hacking -> Kernel low-level debugging functions -> i.MX Debug UART Port Selection : 2
     94 * results in the following additions to .config:
     95{{{
     96CONFIG_DEBUG_LL=y
    11897CONFIG_DEBUG_IMX6Q_UART=y
    11998CONFIG_DEBUG_IMX_UART_PORT=2
    … …  
    122101CONFIG_EARLY_PRINTK=y
    123102CONFIG_EARLY_PRINTK_DIRECT=y
    124 </pre></li></ul><ul><li>Finally, some kernels require a special parameter to be passed in through the bootloader. See below for instructions:
    125 <pre class="wiki">setenv extra ${extra} earlyprintk=serial,${console},${baudrate}
     103}}}
     104 * Finally, some kernels require a special parameter to be passed in through the bootloader.  See below for instructions:
     105{{{#!bash
     106setenv extra ${extra} earlyprintk=serial,${console},${baudrate}
    126107saveenv
    127 </pre></li></ul><p>
    128 <span class="wikianchor" id="sysrq"></span>
    129 </p>
    130 <h2 id="LinuxMagicSystemRequestKeyHacks"><b style="color:#000;background:#ffff66">Linux</b> Magic System Request Key Hacks</h2>
    131 <p>
     108}}}
     109
     110
     111[=#sysrq]
     112== Linux Magic System Request Key Hacks ==
    132113There exists a magic key combo you can hit which the kernel will respond to regardless of whatever else it is doing (unless its completely locked up).
    133 </p>
    134 <p>
     114
    135115To enable this build with CONFIG_MAGIC_SYSRQ when configuring the kernel.
    136 </p>
    137 <p>
     116
    138117This will enable a /proc/sysrq-trigger file which you can use to send various commands to. Some very useful commands for debugging are:
    139 </p>
    140 <ul><li>'b' - reboot immediately (without sync)
    141 </li><li>'c' - perform a system crash via a NULL pointer dereference (which will show a crashdump if enabled)
    142 </li><li>'d' - show all locks that are held
    143 </li><li>'e' - send a SIGTERM to all processes except for PID1 (init)
    144 </li><li>'f' - call oom_kill to kill a memory hog process
    145 </li><li>'h' - display help
    146 </li><li>'i' - send a SIGKILL to all processes except for PID1 (init)
    147 </li><li>'l' - show a stack backtrace for all active CPU's
    148 </li></ul><p>
     118* 'b' - reboot immediately (without sync)
     119* 'c' - perform a system crash via a NULL pointer dereference (which will show a crashdump if enabled)
     120* 'd' - show all locks that are held
     121* 'e' - send a SIGTERM to all processes except for PID1 (init)
     122* 'f' - call oom_kill to kill a memory hog process
     123* 'h' - display help
     124* 'i' - send a SIGKILL to all processes except for PID1 (init)
     125* 'l' - show a stack backtrace for all active CPU's
     126
    149127References:
    150 </p>
    151 <ul><li><a class="ext-link" href="https://www.kernel.org/doc/Documentation/sysrq.txt"><span class="icon">​</span>https://www.kernel.org/doc/Documentation/sysrq.txt</a>
    152 </li></ul></div>
    153          
    154           <div class="trac-modifiedby">
    155             <span><a href="/wiki/linux/troubleshooting?action=diff&amp;version=6" title="Version 6 by psidhu: fix flag">Last modified</a> <a class="timeline" href="/timeline?from=2016-01-11T16%3A04%3A46-08%3A00&amp;precision=second" title="See timeline at 01/11/16 16:04:46">21 months ago</a></span>
    156             <span class="trac-print">Last modified on 01/11/16 16:04:46</span>
    157           </div>
    158        
    159        
    160       </div>
    161      
    162 
    163     </div>
    164     <div id="altlinks">
    165       <h3>Download in other formats:</h3>
    166       <ul>
    167         <li class="last first">
    168           <a rel="nofollow" href="/wiki/linux/troubleshooting?format=txt">Plain Text</a>
    169         </li>
    170       </ul>
    171     </div>
    172     </div>
    173     <div id="footer" lang="en" xml:lang="en"><hr />
    174       <a id="tracpowered" href="http://trac.edgewall.org/"><img src="/chrome/common/trac_logo_mini.png" height="30" width="107" alt="Trac Powered" /></a>
    175       <p class="left">Powered by <a href="/about"><strong>Trac 1.0</strong></a><br />
    176         By <a href="http://www.edgewall.org/">Edgewall Software</a>.</p>
    177       <p class="right">Visit the Trac open source project at<br /><a href="http://trac.edgewall.org/">http://trac.edgewall.org/</a></p>
    178     </div>
    179 }}}
     128* [​https://www.kernel.org/doc/Documentation/sysrq.txt sysrq.txt]