wiki:PCI

Version 9 (modified by Tim Harvey, 4 years ago) ( diff )

added see also links, table of gateworks board pci capabilities, and throughput section

  1. PCI
  2. Mini PCI
  3. PCI Express (PCIe)
  4. PCI Express Mini Card (also known as 'Mini PCIe', 'mPCIe' or 'PEM')
  5. PCI Throughput
  6. Linux PCI Debugging
    1. PCI Advanced Error Reporting (AER)
    2. Examples
  7. Mini-PCIe Mechanical Specification
    1. Mini-PCIe Screws

See also:

Gateworks PCI support:

Product Family Capabilities
Venice PCIe Gen2
Newport PCIe Gen2 2
Ventana PCIe Gen1 1
  1. Ventana boards with external clock generators can theoretically support Gen2 however some software modification would be necessary for the PCIe clock configuration.
  2. Newport can support PCIe Gen3 via a Gateworks special which modifies a strapping resistor to move the coprocessor clock (SCLK) from 350MHz to 550Mhz (at the cost of ~500mW of power draw).

PCI

Peripheral Component Interconnect (PCI) is part of the PCI Local Bus standard. The PCI bus supports the functions found on a processor bus and Devices connected to the PCI bus appear to a bus master to be connected directly to its own bus and are assigned addresses in the processor's address space.

Bus mastering refers to the concept that PCI devices can directly access a processors memory bus independent of the processor similar to a Direct Memory Access (DMA) controller.

PCI History:

  • PCI 1.9 1992 Original issue
  • PCI 2.0 1993 Incorporated connector and add-in card specification
  • PCI 2.1 1995 Incorporated clarifications and added 66 MHz
  • PCI 2.2 1998 Added Mini PCI, Incorporated ECNs and improved readability
  • PCI 2.3 2002 Incorporated ECNs, errata, and deleted 5 volt only keyed add-in cards
  • PCI 3.0 2004 Removed support for 5.0 volt keyed system board connector
  • PCI Express 2004

Conventional PCI had 4 shared level-triggered interrupts and uses a paralle bus architecture where the PCI host and all devices share a common set of address, data and control lines.

References:

Mini PCI

Mini PCI was added to PCI version 2.2 and differs from Conventional PCI in the following ways:

  • 32bit 33Mhz
  • 3.3V only; 5V limited to 100mA
  • three form factors:
    • Type I card uses a 100pin stacking connector
    • Type II card uses a 100pin stacking connector and accomodates a larger size
    • Type III card uses a 124pin edge connector

Older generation Gateworks products such as some Laguna product families support the Mini PCI Type III cards.

PCI Express (PCIe)

While the original PCI bus, now referred to as 'Conventional PCI' was a parallel bus with shared address/data in 2004 The PCI Express (PCIe) specification was released which defined a serialized version of PCI which is commonplace today.

PCI Express is based on a point-to-point topology with separate serial links connecting every device to the host, also known as the root complex (RC). Links may contain from one to 32 lanes (1x, 2x, 4x, 12x, 16x, 32x) with each lane being its own differential pair. PCI Express interrupts are embedded within the serial data.

References:

PCI Express Mini Card (also known as 'Mini PCIe', 'mPCIe' or 'PEM')

The PCI Express Mini Card specification is based on PCI Express with the following differences:

  • uses a 52-pin edge connector with 2 rows of pins
  • incorporates both 1x (1 lane) PCI Express, USB 2.0, and SIM connectivity on the connector

Modern generation Gateworks products such as the Laguna GW2391, Ventana, and Newport product families support Mini PCIe cards.

PCI Throughput

The PCI specification has evolved to support various generations capable of increasing bus speeds:

  • PCI Gen3 (8.0GT/sec or 6.4Gbps)
  • PCI Gen2 (5.0GT/sec or 4Gbps)
  • PCI Gen1 (2.5GT/sec or 2Gbps)

These are backwards compatible such that a PCI Gen3 link will only be established if the device and host controller support it and otherwise it will step down to Gen2 then Gen1 as needed.

The bus speed represents a theoretical maximum throughput and does not account for host processing speed or bus contention from multiple masters.

Linux PCI Debugging

PCI configuration registers can be used to debug various PCI bus issues.

The easiest way to access these registers is via the Linux lspci command with the 'very verbose' flag (-vv) which will decode and display the various PCI config space registers. Note that access to some parts of the PCI configuration space is restricted to root permissions on many operating systems - if this is the case you will see certain data flagged as 'access denied'.

The various registers define bits that are either set (indicated with a '+') or unset (indicated with a '-'). These bits typically have attributes of 'RW1C' meaning you can read and write them and need to write a '1' to clear them. Because these are status bits, if you wanted to 'count' the occurrences of them you would need to write some software that detected the bits getting set, incremented counters, and cleared them over time.

The 'Device Status Register' (DevSta) shows at a high level if there have been correctable errors detected (CorrErr), non-fatal errors detected (UncorrErr), fata errors detected (FataErr), unsupported requests detected (UnsuppReq), if the device requires auxillary power (AuxPwr), and if there are transactions pending (non posted requests that have not been completed).

If you want to delve deeper into types of errors see PCI Advanced Error Reporting below.

References:

PCI Advanced Error Reporting (AER)

Most modern PCI devices support 'Advanced Error Reporting' (AER). For these devices a lspci -vv as root will show additional registers similar to the ones described above:

  • UESta - Uncorrectable Error Status
  • UEMsk - Uncorrectable Error Mask
  • UESvrt - Uncorrectable Error Severity
  • CESta - Correctable Error Status
  • CEMsk - Correctable Error Mask
  • AERCap - Advanced Error Reporting Capabilities

For specifics on what the meaning of the bits in these registers are see the PCI Express Base Specification Revision 3.0]

Examples

Here are some examples:

  • Show all Atheros/QCA radios on the PCI bus (vendor 168c): 
    $ lspci -n | grep 168c
0001:20:00.0 0280: 168c:0046
  • Very Verbose listing of a specific device: 
    $ sudo lspci -s 1:20:00 -vv
0001:20:00.0 Network controller: Qualcomm Atheros Device 0046
        Subsystem: Qualcomm Atheros Device cafe
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 191
        Region 0: Memory at 881010000000 (64-bit, non-prefetchable) [size=2M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
                Address: 0000801000030040  Data: 0000
                Masking: fffffffe  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <4us, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [160 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
        Kernel driver in use: ath10k_pci
        Kernel modules: ath10k_pci
  • Looking at general device status (DevSta) for a specific device: 
    $ sudo lspci -s 1:20:00 -vv | grep DevSta
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
    • Device Status Register above shows only AuxPwr bit is set (this device requires auxiliary power)
  • Looking at AER registers: 
    $ sudo lspci -s 1:20:00 -vv | grep -e "UESta\|UEMsk\|UESvrt\|CESta\|CEMsk\|AERCap"
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    • The Uncorrectable Error Status (UESta) reports error status of individual uncorrectable error sources (no bits are set above):
      • Data Link Protocol Error (DLP)
      • Surprise Down Error (SDES)
      • Poisoned TLP (TLP)
      • Flow Control Protocol Error (FCP)
      • Completion Timeout (CmpltTO)
      • Completer Abort (CmpltAbrt)
      • Unexpected Completion (UnxCmplt)
      • Receiver Overflow (RxOF)
      • Malformed TLP (MalfTLP)
      • ECRC Error (ECRC)
      • Unsupported Request Error (UnsupReq)
      • ACS Violation (ACSViol)
    • The Uncorrectable Error Mask (UEMsk) controls reporting of individual errors by the device to the PCIe root complex. A masked error (bit set) is not recorded or reported. Above shows no errors are being masked)
    • The Uncorrectable Severity controls whether an individual error is reported as a Non-fatal (clear) or Fatal error (set).
    • The Correctable Error Status reports error status of individual correctable error sources: (no bits are set above)
      • Receiver Error (RXErr)
      • Bad TLP status (BadTLP)
      • Bad DLLP status (BadDLLP)
      • REPLAY_NUM Rollover status (Rollover)
      • Replay Timer Timeout status (Timeout)
      • Advisory Non-Fatal Error (NonFatalIErr)
    • The Correctable Erro Mask (CEMsk) controls reporting of individual errors by the device to the PCIe root complex. A masked error (bit set) is not reported to the RC. Above shows that Advisory Non-Fatal Errors are being masked - this bit is set by default to enable compatibility with software that does not comprehend Role-Based error reporting.
    • The Advanced Error Capabilities and Control Register (AERCap) enables various capabilities (The above indicates the device capable of generating ECRC errors but they are not enabled):
      • First Error Pointer identifies the bit position of the first error reported in the Uncorrectable Error Status register
      • ECRC Generation Capable (GenCap) indicates if set that the function is capable of generating ECRC
      • ECRC Generation Enable (GenEn) indicates if ECRC generation is enabled (set)
      • ECRC Check Capable (ChkCap) indicates if set that the function is capable of checking ECRC
      • ECRC Check Enable (ChkEn) indicates if ECRC checking is enabled

Note that by default the Linux kernel will not alter ECRC Generation / Check and considers this configured by boot firmware. This can be overridden by enabling CONFIG_PCIE_ECRC in the kernel and passing the kernel cmdline 'ecrc=0' to force disable or 'ecrc=1' to force enable

Mini-PCIe Mechanical Specification

The below mechanical drawing is for the standard Mini-PCIe connector used on standard Ventana/Newport boards. Note that other heights are available as special order (100 piece MOQ). Contact sales for available options. http://trac.gateworks.com/raw-attachment/wiki/minipciexpressmodules/minipcieconn.png

Mini-PCIe Screws

The hold down screws for the Mini-PCIe cards are pre-loaded into the standoffs on the board. If you need additional screws they can be purchased at McMaster Carr.

Note: See TracWiki for help on using the wiki.