Changes between Version 1 and Version 2 of ventana/PCIe


Ignore:
Timestamp:
10/21/2017 10:55:05 PM (12 months ago)
Author:
Chris Lang
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ventana/PCIe

    v1 v2  
     1[[PageOutline]]
     2
    13= Ventana PCI/PCIe Support =
    24
     5== PCIe Pinout ==
     6
     7This chart is contained in each product manual for each respective board.
     8
     9Note that Gateworks prefers to adhere to the industry standard for pin usage.
     10
     11[[Image(pciepins.png,400px)]]
     12
     13[=#resources]
     14== Resource Limits ==
    315The i.MX6 CPU has an internal address translation unit (iATU) that connects the i.MX6 PCI host controller to the memory bus.  This iATU window size '''imposes a resource limit which can ultimately limit the number of PCI devices you can have on the bus'''. The iATU window is 16MB which can technically be broken up in a variety of ways but by default is used as:
    416 * 512KB config space
    517 * 64KB io space
    6  * 15MB mem space
     18 * 15MB mem space available for devices
    719
    820PCI Devices can request 1 or more io regions, and 1 or more mem regions however when devices are behind a bridge (which they will be on a GW52xx, GW53xx, and GW54xx) the various resource requests must go through a PCI bridge which imposes a 1MB granularity for mem regions.  On the GW52xx, GW53xx, GW54xx, each PCIe socket is behind a bridge and thus has this 1MB granularity. The upstream port on a PCIe switch takes a mem resource itself, which ends up leaving 14 more 1MB windows available.
    921
    1022The outcome is complex and is likely best explained with a series of examples of what is possible. The following examples use various hardware combinations of:
    11  * baseboards:
     23 * Baseboards:
    1224  * GW54xx - 2 mem windows used by baseboard (1 for PCIe switch, 1 for eth1 GigE)
    1325  * GW53xx - 2 mem windows used by baseboard (1 for PCIe switch, 1 for eth1 GigE)
     
    2133  * SR71e 802.11n  2x2 MIMO radio - 1 mem window required
    2234  * DNMA H-5 802.11abg radio - 1 mem window required
    23   * WLE900 802.11ac 3x3 MIMO radio - 2 mem windows (1 2mb window)
     35  * WLE900 802.11ac 3x3 MIMO radio - 3 mem windows (1@2mb window, 1@1mb window)
    2436   * Note the ath10k driver/firmware may also request coherent pool memory (coherent memory from the kernel's atomic memory pool) and may require you to increase the kernel atomic coherent memory pool via the 'coherent_pool' kernel command line if you encounter allocation errors using multiple radios.
    2537   *
     
    4153Other configurations are possible if someone for example wants to spread out some PCIe devices across a couple of GW16081 mezzanines to allow many cellular radios (which USE USB, not PCI). The basic rules can be summarized as follows:
    4254 * i.MX6 has 14 available memory resources
    43  * most atheros radios seem to require 1 (ie SR71e, Option GTM671WFS), but some (ie WLE300) requires 2
    44  * each PCIe switch requires 1 (ie GW54xx/GW53xx/GW52xx has one on-board, add another if you have a GW16081 mezz)
     55 * Most atheros radios seem to require 1 (ie SR71e, Option GTM671WFS), but some (ie WLE300 ath9k) requires 2
     56 * Each PCIe switch requires 1 (ie GW54xx/GW53xx/GW52xx has one on-board, add another if you have a GW16081 mezz)
    4557 * 2nd onboard eth1 GigE requires 1
    46  * the PCIe-to-PCI bridge (GW16082 mezz) requires 1 but has the unique case that everything behind it fits into 1 resource regardless of radio.
     58 * The PCIe-to-PCI bridge (GW16082 mezz) requires 1 but has the unique case that everything behind it fits into 1 resource regardless of radio.
    4759
    4860Notes:
    4961 * The above examples refer to the PCIe host controller driver used in the OpenWrt (3.10+) kernel.  The 3.0.35 kernel used for our Yocto and Android BSPs reserve a 14MB mem resource window which leaves 1 less region affecting the examples above.
    5062
    51 == Memory Calculation Example ==
    52 
     63=== Memory Calculation Example ===
    5364To determine how many memory resources a wifi or other card uses, consult the following:
    5465
     
    7586}}}
    7687
     88
    7789Now run the dmesg command and grep for that device 07:00.0
    78 Note the two memory regions below, reg10 and reg 30.  Thus this device uses 2 memory resources.
     90Note the two memory regions below, reg10 and reg 30. 
    7991{{{
    8092root@OpenWrt:/# dmesg | grep 07:00.0
     
    89101}}}
    90102
    91 == PCIe PLX Switch - Temperature discussion ==
     103Calculate the memory usage by looking at the lines such as reg 10 and reg30.
    92104
    93 [wiki:faq#Whyarecertainareasontheboardwarmwithregardstotemperature Please see this discussion here]
     105Look at the mem portion and calculate the size between 0x00 and the other number.
    94106
    95 == What throughput does the PCIe lane support? ==
     107For example:
     108 * 0x001fffff = 2097151÷1024 = 2048 KB = 2 MB
     109 * 0x0001ffff = 131072÷1024 = 128 KB = 1MB (this is because there is only a 1MB granularity in the memory allocation, so the miniumum that can be allocated is 1MB even if the actual is only 128KB.)
    96110
    97 The PCIe lane is a single lane going into the switch with a maximum theoretical of 2.5Gbits/sec on the Ventana boards.
    98111
     112
     113[=#switch]
     114== PCIe Switch ==
     115While the IMX6 has only a single PCIe host controller, many Ventana models have a PLX PCIe switch which allows the board to support more than 1 PCIe endpoint device.
     116
     117A PCIe switch operates like a PCI bridge such that it will create additional subordinate busses.
     118
     119PCIe bus layout can vary board-to-board depending on which PLX switch is used (we use a variety of 4 port, 6 port, and 8 port devices).
     120
     121The {{{lspci}}} command will list the details of the devices enumerated on the bus. Note that PCI enumeration occurs at kernel init time as ARM Linux does not support PCI hotplug.
     122
     123The typical power usage of the PLX switch is appx 1.35W up to a max of appx 2.6W (max being 85% traffic). The switch does support PCIExpress Active State Power Management (ASPM) and also will power down unused !SerDes lanes automatically to reduce power when possible.
     124
     125Some customers choose to place a heatsink on the PLX PCIe switch chip depending on their operating environment, system load, and enclosure.
     126
     127[=#throughput]
     128== PCI Throughput ==
     129The IMX6 PCIe host supports up to PCI Gen2 (5.0Gbits/sec) however this requires an external clock generator which is not present on all board revisions and was added as an enhancement.
     130Therefore we limit the link to PCI Gen1 (2.5Gbits/sec) in our Board Support Packages (BSPs) via a software patch as a conservative effort. A future software update (bootloader and kernel) will remove this limit for board revisions that have an external clock generator.
     131
     132See [wiki:ventana/errata#hw14 Ventana errata HW14] for details on what board revisions have Gen2 capability.
     133
     134The bus speed represents a theoretical maximum throughput and does not account for host processing speed or bus contention from multiple masters.
    99135
    100136[=#msi]
     
    105141
    106142Because MSI interrupts can not be steered to different CPU's in the hardirq context there is no performance benefit of MSI and we have MSI disabled in the Gateworks Ventana kernels. Additionally we have encountered devices/drivers from time-to-time that do not work properly with MSI interrupts enabled.
     143
     144Additionally it should be noted that currently enabling MSI in IMX6 kernels will break legacy PCI interrupt support meaning any card/driver that doesn't support MSI will not work. For this reason we do not currently support MSI on Ventana.
     145
     146== PCIe Reset ==
     147
     148Reset signals are routed to the Mini-PCIe slots.
     149
     150Some boards have the same reset signal across all slots, and some boards have individually controlled reset signals for each slot. Please contact Gateworks support via email for a specific board model.
     151
     152PCIe reset signals are typically always controlled by the kernel and software, however, they also can at times be controlled via a GPIO, with more details [wiki:gpio#GPIOmapping here]