{{{#!html

Linux OS Code Profiling

There are several options for code profiling on the Linux OS. The kernel itself has a profiling API which can be enabled:

OProfile was the profiling tool of choice for linux devls for nearly 10 years. A few years back various kernel developers defined and implemented a new formal kernel API to access performance monitor counters (PMC's), which are hardware elements in most modern CPU's, to address needs of performance tools. Prior to this new API oPOProfileofile used a special OProfile-specific kernel module while other tools relied on patches (perctr, perfmon).

The developers of the new profiling API also developed an example tool that used the new API called 'perf'. The perf tool has thus matured greatly in the past few years. oprfile is strickly a profiling tool.

There are other options that are not described here:

Reference:

Basic Kernel Profiling (CONFIG_PROFILING and readprofile)

There are several facilities to see where the kernel spends its resources. A simple one which can be built-in with (CONFIG_PROFILING) will store the current EIP (instruction pointer) at each clock tick.

To use this ensure the kernel is built with CONFIG_PROFILING and either boot the kernel with command line option profile=2 or enable at runtime with an echo 2 > /sys/kernel/profiling.

This will cause a file /proc/profile to be created. The number provided (2 in the example above) is the number of positions EIP is shifted right when profiling. So a large number gives a coarse profile. The counters are reset by writing to /proc/profile.

The utility readprofile will output statistics for you. It does not sort so you have to invoke sort explicitly. But given a memory map it will translate addresses to kernel symbols.

Example:

  1. boot kernel compiled with CONFIG_PROFILING
  2. enable (either with placing profile=2 on cmdline or dynamically with: 
    echo 2 > /sys/kernel/profiling # enable profiling
  3. (optional) clear counters 
    echo > /proc/profile # reset counters
  4. do some activity you wish to profile
  5. use readprofile to interpret the results: 
    readprofile -m System.map | sort -nr | head -2
510502 total                                      0.1534
508548 default_idle                           10594.7500

References:

OProfile

OProfile provides a profiler and post-processing tools for analyzing profile data, event counter.

The tool used is called operf. Some processors are not supported by the underlying new perf_events kernel API and thus not supported by operf. If you see Your kernel's Performance Events Subsystem does not support your processor type then you need to try and use opcontrol for the legacy mode.

References:

OProfile Standard Mode (imx6)

Starting with v0.9.8, OProfile switched over to using the new perf_events kernel API with a new set of userspace tools (however OProfile still supports the legacy mode - see below).

Standard mode tools:

Using the standard mode, post-processing of collected raw events is not necessary.

OProfile Legacy Mode (cns3xxx)

The legacy mode (for CPU's that do not implement the new perf_events kernel profiling API. The Gateworks Laguna family using the Cavium cns3xxx CPU falls into this category.

The legacy mode tools consists of:

opcontrol parameters:

Example usage:

  1. copy your current kernel's vmlinux to /tmp
  2. (optional) setup our configuration for vmlinux symbol decrypting, specific session location, and separating events by cpu: 
    opcontrol --setup --vmlinux=/tmp/vmlinux --session-dir=/tmp/session1 --separate=cpu
  3. start capturing events: 
    opcontrol --start
  4. stop capturing events (and flush data): 
    opcontrol --shutdown
  5. report events: 
    opreport --vmlinux=/tmp/vmlinux --session-dir=/tmp/session1

Important notes:

References:

Perf

In general profiling with the perf tool is considered easier to install and run.

Example:

  1. (optional) copy your current kernel's vmlinux to /tmp
  2. capture 120 seconds worth of profiling data 
    perf record -p $(pidofprogram) sleep 120
  3. report data (using kernel symbols): 
    perf report -k /tmp/vmlinux

References:

OpenWrt

OpenWrt has support for both oProfile and perf. Because perf depends on glibc (or at least is configured that way) we recommend oprofile when using OpenWrt.

To enable oProfile on OpenWrt do a make menuconfig and:

To enable perf (glibc required):

You likely want to run non-stripped binaries for anything you want to actually investigate. One way of doing this is to build them with CONFIG_DEBUG=y. For example building compat-wireless: 

make target/linux/mac80211/{clean,compile} V=99 CONFIG_DEBUG=y

References:

Last modified 2 years ago Last modified on 04/07/15 16:03:47
}}}