Changes between Initial Version and Version 1 of performance_tuning

Timestamp:

10/22/2017 05:28:45 AM (7 years ago)

Author:

trac

Comment:

--

performance_tuning

@@ +1 @@
+[[PageOutline]]
+
+= Performance Tuning =
+
+Various performance aspects rely heavily on the configuration of your Linux operating system.  Here are some things to keep in mind:
+ * L2 Cache - enabling L2 cache can greatly increase the performance of some things, but greatly hurt others - see [wiki:laguna/l2cache here]
+ * Kernel modules - kernel modules that are not needed can bog down certain paths, such as network routing (iptables/ebtables) (see below)
+ * Userland services - Various services and daemons that are not needed can chew up system resources (CPU cycles, memory footprint) (see below)
+
+=== See also  [wiki:multicoreprocessing Multicore Processing Page] ===
+
+
+== Routing Performance ==
+
+If you are trying to optimize network routing you can try the following:
+ * GigE:
+  * make sure you have a GigE link (where appropriate) in every network segment between your test endpoints (switch segments, target endpoint computer etc)
+  * if using PoE make sure you have a PoE injector capable of GigE
+ * General:
+  * eliminate unnecessary kernel modules which may be present for packet filtering (such as ipt and ebtables related modules). To see an example on removing kernel modules see the [wiki:OpenWrt/kernelconfig#loadingmodules] page. If you are using hardware that requires some modules be sure to leave them in place.
+   * '''Caution:''' Please make informed decisions when removing kernel modules as removing hardware related modules may have unintended effects.
+  * minimize hardware in-between in case it is problematic (direct connection between endpoints)
+  * eliminate unnecessary userspace applications which may be present.  To eliminate virtually all of them (you may need to configure network by hand after doing this) you can use 'for i in $(ls /etc/init.d); do /etc/init.d/$i disable; done; /etc/init.d/enable boot; /etc/init.d/enable done'
+  * using iperf as a network test tool pay attention to window size which can greatly affect throughput (understand what it means)
+  * be aware that generating traffic on an embedded node creates a performance hit on that node vs sending traffic 'through' the node
+  * run 'top' while testing to see where the bottlenecks may be:  Understand that if virtually 100% of utilization occurs in sirq (soft irq), irq (hard irq), and nic (network driver) then you have maxed out the performance due to raw interrupts and low level packet handling 
+
+== iperf ==
+Use iperf to test throughput.[[BR]]
+[[BR]]
+
+
+Please read up on iperf on google or this tutorial [http://openmaniak.com/iperf.php][[BR]]
+[[BR]]
+
+Basic Setup:[[BR]]
+
+There is a client and a server. This is differentiated based on the command.[[BR]][[BR]]
+Server:
+{{{
+iperf -s
+}}}
+Client:
+{{{
+iperf -c 192.168.1.1
+}}}
+
+Note:  For UDP, a bandwidth limit is needed.  Use the flag -b followed by the bandwidth limit desired (1m, 10m, 100m, 200m, 300m, 500m, 1g, etc).
+Because iperf is processor intensive, there is no need to generate more traffic than the processor can handle.  Therefore, incrementally increase the bandwidth limit until the results are slightly below the limit, thus not creating larger amounts of overhead. 
+[[BR]][[BR]]
+For example a bandwidth limit of 10m will easily be hit on a GBe link.  However, once at 200m bandwidth limit, throughput numbers may hit only 187Mbits/sec thus creating very little processor overhead.[[BR]][[BR]]
+'''VERY IMPORTANT: ORDER MATTERS.  The iperf manual (please read it) states that the bandwidth flag must be placed at the end of the command to work!'''[[BR]][[BR]]
+
+From the client type:
+{{{
+iperf -u -c 192.168.4.1 -b 10m
+}}}
+From the server type:
+{{{
+iperf -s -u
+}}}
+[[BR]][[BR]]
+'''Increasing Bandwidth Limit Example:'''
+{{{
+root@OpenWrt:/# iperf -u -c 192.168.4.1 -b 10m
+------------------------------------------------------------
+Client connecting to 192.168.4.1, UDP port 5001
+Sending 1470 byte datagrams
+UDP buffer size:  160 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.4.2 port 56615 connected with 192.168.4.1 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec  11.9 MBytes  10.0 Mbits/sec
+[  3] Sent 8505 datagrams
+[  3] Server Report:
+[  3]  0.0-10.0 sec  11.9 MBytes  10.0 Mbits/sec   0.028 ms    1/ 8506 (0.012%)
+root@OpenWrt:/# iperf -u -c 192.168.4.1 -b 100m
+------------------------------------------------------------
+Client connecting to 192.168.4.1, UDP port 5001
+Sending 1470 byte datagrams
+UDP buffer size:  160 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.4.2 port 33153 connected with 192.168.4.1 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   120 MBytes   100 Mbits/sec
+[  3] Sent 85304 datagrams
+[  3] Server Report:
+[  3]  0.0-10.0 sec   119 MBytes  99.8 Mbits/sec   0.077 ms  514/85305 (0.6%)
+root@OpenWrt:/# iperf -u -c 192.168.4.1 -b 200m
+------------------------------------------------------------
+Client connecting to 192.168.4.1, UDP port 5001
+Sending 1470 byte datagrams
+UDP buffer size:  160 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.4.2 port 56998 connected with 192.168.4.1 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   222 MBytes   186 Mbits/sec
+[  3] Sent 158150 datagrams
+[  3] Server Report:
+[  3]  0.0-10.2 sec  59.3 MBytes  48.6 Mbits/sec  15.794 ms 115820/158151 (73%)
+root@OpenWrt:/# iperf -u -c 192.168.4.1 -b 300m
+------------------------------------------------------------
+Client connecting to 192.168.4.1, UDP port 5001
+Sending 1470 byte datagrams
+UDP buffer size:  160 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.4.2 port 52109 connected with 192.168.4.1 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   223 MBytes   187 Mbits/sec
+[  3] Sent 158814 datagrams
+[  3] Server Report:
+[  3]  0.0-10.0 sec  58.0 MBytes  48.7 Mbits/sec   0.240 ms 117423/158815 (74%)
+}}}
+
+
+== Examples ==
+
+
+Here are some examples showing various tunings of a GW2388 (dual core 600MHz ARM with dual GigE ports) with iperf network bandwidth test between GW2388 and a PC through a Netgear GigE switch:
+ * pre-built firmware (12-10 release), unmodified (186/491mbps tx/rx)
+{{{
+root@OpenWrt:/# iperf -c 192.168.1.146
+------------------------------------------------------------
+Client connecting to 192.168.1.146, TCP port 5001
+TCP window size: 16.0 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.1.83 port 56088 connected with 192.168.1.146 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   223 MBytes   186 Mbits/sec
+root@OpenWrt:/# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 192.168.1.83 port 5001 connected with 192.168.1.146 port 55929
+[ ID] Interval       Transfer     Bandwidth
+[  4]  0.0-10.0 sec   500 MBytes   419 Mbits/sec
+}}}
+ * disabling all kernel modules (306/732mbps tx/rx) (iptables/ebtables is a big performance hit)
+{{{
+root@OpenWrt:/# mv /etc/modules.d /etc/modules.old
+root@OpenWrt:/# reboot
+root@OpenWrt:/# iperf -c 192.168.1.146
+------------------------------------------------------------
+Client connecting to 192.168.1.146, TCP port 5001
+TCP window size: 16.0 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.1.83 port 58570 connected with 192.168.1.146 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   365 MBytes   306 Mbits/sec
+root@gw2388-test:/# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 192.168.1.83 port 5001 connected with 192.168.1.146 port 56554
+[ ID] Interval       Transfer     Bandwidth
+[  4]  0.0-10.0 sec   873 MBytes   732 Mbits/sec
+}}}
+ * disabling all init scripts as well (378/754mbps tx/rx)
+{{{ 
+root@OpenWrt:/# for i in $(ls /etc/init.d); do /etc/init.d/$i disable; done
+root@OpenWrt:/# /etc/init.d/boot enable
+root@OpenWrt:/# reboot
+root@OpenWrt:/# iperf -c 192.168.1.146
+------------------------------------------------------------
+Client connecting to 192.168.1.146, TCP port 5001
+TCP window size: 16.0 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.1.93 port 52295 connected with 192.168.1.146 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   451 MBytes   378 Mbits/sec
+root@OpenWrt:/# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 192.168.1.93 port 5001 connected with 192.168.1.146 port 37005
+[ ID] Interval       Transfer     Bandwidth
+[  4]  0.0-10.0 sec   900 MBytes   754 Mbits/sec
+}}}
+
+Here are some examples using a GW2380 (single core 300MHz ARM with single GigE port) looking at transmit performance:
+ * pre-built firmware (12-10 release), unmodified (153/138mbps tx/rx)
+{{{
+root@OpenWrt:/# iperf -c 192.168.1.146
+------------------------------------------------------------
+Client connecting to 192.168.1.146, TCP port 5001
+TCP window size: 16.0 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.1.87 port 33327 connected with 192.168.1.146 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   183 MBytes   153 Mbits/sec
+root@OpenWrt:/# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 192.168.1.87 port 5001 connected with 192.168.1.146 port 53363
+[ ID] Interval       Transfer     Bandwidth
+[  4]  0.0-10.1 sec   166 MBytes   138 Mbits/sec
+}}}
+ * disabling all kernel modules (207/214mbps tx/rx) (iptables/ebtables is a big performance hit)
+{{{
+root@OpenWrt:/# mv /etc/modules.d /etc/modules.old
+root@OpenWrt:/# reboot
+root@OpenWrt:/# iperf -c 192.168.1.146
+------------------------------------------------------------
+Client connecting to 192.168.1.146, TCP port 5001
+TCP window size: 16.0 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.1.97 port 60819 connected with 192.168.1.146 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   248 MBytes   207 Mbits/sec
+root@OpenWrt:/# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 192.168.1.87 port 5001 connected with 192.168.1.146 port 53387
+[ ID] Interval       Transfer     Bandwidth
+[  4]  0.0-10.0 sec   255 MBytes   214 Mbits/sec
+}}}
+ * disabling all init scripts as well (222/214mbps tx/rx)
+{{{ 
+root@OpenWrt:/# for i in $(ls /etc/init.d); do /etc/init.d/$i disable; done
+root@OpenWrt:/# /etc/init.d/boot enable
+root@OpenWrt:/# /etc/init.d/done enable
+root@OpenWrt:/# reboot
+root@OpenWrt:/# iperf -c 192.168.1.146
+------------------------------------------------------------
+Client connecting to 192.168.1.146, TCP port 5001
+TCP window size: 16.0 KByte (default)
+------------------------------------------------------------
+[  3] local 192.168.1.97 port 34382 connected with 192.168.1.146 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec   265 MBytes   222 Mbits/sec
+root@OpenWrt:/# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 192.168.1.87 port 5001 connected with 192.168.1.146 port 53387
+[ ID] Interval       Transfer     Bandwidth
+[  4]  0.0-10.0 sec   255 MBytes   214 Mbits/sec
+}}}
+
+
+Notes:
+ * the above tests are using 'iperf -s' on an Unbuntu Linux GigE PC Host through a GigE Netgear switch.  This uses a 16KB TCP window size - you can acheive higher tx throughput by using larger TCP windows.
+
+
+== Wireless Tuning ==
+
+Wireless performance is a very finicky process.[[BR]][[BR]]
+
+Wireless is dependent on so many factors.  Wireless N speeds at 300 Mbps are not typically achieved in real life.  An ARM processor on a Gateworks board is not quite the same as a 3.4 GHz X86 Machine.[[BR]][[BR]]
+
+
+Here are a few tips for wireless:
+ 1. Wireless encryption protocols can affect performance because of computation.  Choose wisely.
+ 2. Antenna type and orientation and distance are VERY important.  Be sure to hook up all antennas to the wireless card and testing 2 boards 3 feet apart may not achieve the best results.
+ 3. Wireless N will sometimes run faster at 40MHz vs 20 MHz.
+ 4. Try Channel 153 for Wireless N.
+ 5. Use a uncluttered channel that other devices are not using. 
+ 6. The hardware mode for openwrt for the N radios would typically be 11na.
+ 7. 5.8GHz will run faster than 2.4GHz.
+ 8. Optimize the software as noted below.  The more items running on the board, the less processor that can be utilized for all of the wireless traffic.
+
+We have removed some modules at times to reduce the software load and try to obtain higher throughput.
+
+These are the commands that have been using to tune performance for wireless testing on both the Cambria and Laguna platforms.
+This is not a fully supported and will disable certain features.  Please use at your own risk. 
+
+'''*WARNING: Run these commands on the Gateworks board once it is booted up.  Please only do so over serial as telnet is disabled below'''
+
+{{{
+/etc/init.d/batmand disable
+/etc/init.d/collectd disable
+/etc/init.d/cron disable
+/etc/init.d/dnsmasq disable
+/etc/init.d/dropbear disable
+/etc/init.d/firewall disable
+/etc/init.d/gpsd disable
+/etc/init.d/gscd disable
+/etc/init.d/led disable
+/etc/init.d/luci_bwc disable
+/etc/init.d/luci_dhcp_migrate disable
+/etc/init.d/luci_fixtime disable
+/etc/init.d/luci_statistics disable
+/etc/init.d/miniupnpd disable
+/etc/init.d/openvpn disable
+/etc/init.d/qos disable
+/etc/init.d/rcS disable
+/etc/init.d/relayd disable
+/etc/init.d/serialoverip disable
+/etc/init.d/sysntpd disable
+/etc/init.d/ntpd disable
+/etc/init.d/telnet disable
+/etc/init.d/tinyproxy disable
+/etc/init.d/uhttpd disable
+/etc/init.d/umount disable
+/etc/init.d/usb disable
+/etc/init.d/vnstat disable
+
+mkdir /etc/modules.old
+mv /etc/modules.d/* /etc/modules.old
+mv /etc/modules.old/*crypto* /etc/modules.d
+mv /etc/modules.old/*80211* /etc/modules.d
+mv /etc/modules.old/*nls-base* /etc/modules.d
+mv /etc/modules.old/*usb-core* /etc/modules.d
+mv /etc/modules.old/*ath5k* /etc/modules.d
+mv /etc/modules.old/*ath9k* /etc/modules.d
+mv /etc/modules.old/26-ath  /etc/modules.d
+mv /etc/modules.old/50-madwifi  /etc/modules.d
+}}}
+
+
+Wireless configuration used:
+{{{
+ACCESS POINT WIRELESS UCI Config:
+root@OpenWrt:/# uci show wireless
+wireless.radio0=wifi-device
+wireless.radio0.type=mac80211
+wireless.radio0.phy=phy0
+wireless.radio0.ht_capab=SHORT-GI-40 TX-STBC RX-STBC1 DSSS_CCK-40
+wireless.radio0.disabled=0
+wireless.radio0.country=US
+wireless.radio0.txpower=17
+wireless.radio0.htmode=HT40-
+wireless.radio0.channel=153
+wireless.radio0.hwmode=11na
+wireless.@wifi-iface[0]=wifi-iface
+wireless.@wifi-iface[0].device=radio0
+wireless.@wifi-iface[0].network=lan
+wireless.@wifi-iface[0].mode=ap
+wireless.@wifi-iface[0].ssid=gateworks
+wireless.@wifi-iface[0].wds=1
+wireless.@wifi-iface[0].encryption=psk2
+wireless.@wifi-iface[0].key=abc
+
+CLIENT WDS WIRELESS UCI Config:
+root@OpenWrt:/# uci show wireless
+wireless.radio0=wifi-device
+wireless.radio0.type=mac80211
+wireless.radio0.phy=phy0
+wireless.radio0.ht_capab=SHORT-GI-40 TX-STBC RX-STBC1 DSSS_CCK-40
+wireless.radio0.disabled=0
+wireless.radio0.txpower=17
+wireless.radio0.country=US
+wireless.radio0.htmode=HT40-
+wireless.radio0.channel=153
+wireless.radio0.hwmode=11na
+wireless.@wifi-iface[0]=wifi-iface
+wireless.@wifi-iface[0].device=radio0
+wireless.@wifi-iface[0].network=lan
+wireless.@wifi-iface[0].ssid=gateworks
+wireless.@wifi-iface[0].mode=sta
+wireless.@wifi-iface[0].wds=1
+wireless.@wifi-iface[0].encryption=psk2
+wireless.@wifi-iface[0].key=abc
+}}}
+
+We then ran iperf from the the two pc's, with one being the server and one the client.
+
+{{{
+Server:
+iperf -s
+
+Client:
+iperf -t10 -w 512k -c 192.168.0.23
+
+iperf TCP results:
+
+------------------------------------------------------------
+Client connecting to 192.168.0.23, TCP port 5001
+TCP window size:  256 KByte (WARNING: requested  512 KByte)
+------------------------------------------------------------
+[  3] local 192.168.0.22 port 52644 connected with 192.168.0.23 port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  3]  0.0-10.0 sec  99.9 MBytes  83.7 Mbits/sec
+}}}
+
+=== Wireless Modulation Rate ===
+The modulation rate is dynamic.  The rate adjusts depending on the quality of the RF signal.  The better the signal, the better the rate.  The table below indicates what rate you are currently achieving.  If you have a terrible rate, you will notice your bandwidth much lower and need to adjust the antennas / distance / obstructions to get a better rate.  The highest rates can be very very hard to achieve.
+{{{
+root@OpenWrt:/# cat /sys/kernel/debug/ieee80211/phy0/netdev\:wlan0/stations/a8\:
+54\:b2\:00\:04\:2e/rc_stats
+type      rate     throughput  ewma prob   this prob  this succ/attempt   success    attempts
+HT20/LGI    MCS0        6.1       92.0      100.0          0(  0)        106         111
+HT20/LGI    MCS1       12.8       96.7      100.0          0(  0)        109         116
+HT20/LGI    MCS2       19.5       99.7      100.0          0(  0)        112         119
+HT20/LGI    MCS3       25.8       99.9      100.0          0(  0)        108         110
+HT20/LGI    MCS4       36.2       95.2      100.0          0(  0)        102         112
+HT20/LGI    MCS5       44.2       89.8      100.0          0(  0)        108         111
+HT20/LGI    MCS6       48.4       88.7      100.0          0(  0)       2442        2816
+HT20/LGI   PMCS7       48.5       79.1       56.6         68(120)       3616        4236
+HT20/LGI    MCS8       12.1       92.1      100.0          0(  0)        103         110
+HT20/LGI    MCS9       24.5       95.1      100.0          0(  0)        105         117
+HT20/LGI    MCS10      37.8       99.5      100.0          0(  0)        118         123
+HT20/LGI    MCS11      48.3       98.1      100.0          0(  0)        121         127
+HT20/LGI    MCS12      58.8       81.7       33.3         13( 39)      14294       17331
+HT20/LGI  t MCS13      66.9       74.2       60.8          0(  0)      62514       85221
+HT20/LGI T  MCS14      67.9       67.2       58.2          0(  0)     110248      163970
+HT20/LGI    MCS15      61.4       55.9       28.0         32(114)     141375      243633
+
+T= what we are on
+t= what we are on next
+}}}
+
+[http://mcsindex.com/ Modulation Table Online ]
+
+== Measuring Performance ==
+In most cases, measuring the performance of a particular system function can be done with specialized tests like {{{iperf}}}. When a more thorough measurement is desired of the overall system, a small and portable benchmarking suite like {{{lmbench}}} is more appropriate.
+
+=== lmbench ===
+{{{lmbench}}} is a micro-benchmark suite designed to focus attention on the basic building blocks of many common system applications, such as databases, simulations, software development, and networking. It provides a suite of benchmarks that attempt to measure the most commonly found performance bottlenecks in a wide range of system applications. {{{lmbench}}} is designed to identify, isolate, and reproduce these performance bottlenecks using a set of small microbenchmarks which measure system latency and bandwidth of data movement among the processor and memory, network, file system, and disk.
+
+Installation can be done on Ubuntu systems via {{{sudo apt-get install lmbench}}}. Alternatively, you can compile the source yourself which can be found at the [http://www.bitmover.com/lmbench/ lmbench website].
+
+Run the following commands to compile the source: 
+{{{#!bash
+tar xvf lmbench3.tar.gz
+cd lmbench3
+mkdir SCCS; touch SCCS/s.ChangeSet
+make -C src/
+}}}
+
+Running the benchmark suite is done via {{{make results}}}. The binaries for the individual benchmarks can be found in the {{{bin/}}} directory.
+
+The {{{src/Makefile}}} has additional make targets for your convenience:
+{{{#!bash
+# lmbench       [default] builds the benchmark suite for the current os/arch
+# results       builds, configures run parameters, and runs the benchmark
+# rerun         reruns the benchmark using the same parameters as last time
+# scaling       reruns the benchmark using same parameters as last time,
+#               except it asks what scaling value to use
+# hardware      reruns the hardware benchmarks using the same parameters
+# os            reruns the OS benchmarks using the same parameters
+# clean         cleans out sources and run configuration
+# clobber       clean and removes the bin directories
+# shar          obsolete, use cd .. && make shar
+# depend        builds make dependencies (needs gcc)
+# debug         builds all the benchmarks with '-g' debugging flag
+# assembler     builds the .s files for each benchmark3
+}}}
+
+After the test has been completed, you can compare results from multiple runs with {{{make -C results/ LIST=*}}}
+
+Read the man pages for the individual benchmarks in the {{{doc/}}} directory, or the [http://www.bitmover.com/lmbench/lmbench-usenix.pdf lmbench introduction] to learn more about the test suite.