Changes between Initial Version and Version 1 of linux/media

10/05/2018 08:26:18 PM (2 years ago)
Tim Harvey

initial page


  • linux/media

    v1 v1  
     3== Media Controller
     4Modern System-on-Chip (SoC) devices support a wide range of functionality in the way of internal hardware blocks which has resulted in a variety of ways to interconnect functions within the SoC that alter the V4L device content.
     6The Media Controller kernel API has been designed to expose detailed information about media devices and capture devices to userspace in a way that allows them to be interconnected in dynamic and complex ways at runtime.
     8Media controller devices expose entities to the media controller framework. Each entity has one or more source pads and one or more sink pads. You use the media controller kernel API (ie via {{{media-ctl}}}) to query entities, link source pads to sink pads, and set formats of pads.
     10'''Note that the IMX6 media controller driver was added to Linux in the 4.13 kernel. This page applies to Linux 4.13 and beyond. If you are using a kernel earlier than that for the Ventana IMX6 based boards (such as the Gateworks 3.14 kernel) you will need to use the Yocto GStreamer pages:
     11 * [wiki:Yocto/gstreamer]
     12 * [wiki:Yocto/gstreamer/compositing]
     13 * [wiki:Yocto/gstreamer/latency]
     14 * [wiki:Yocto/gstreamer/multimedia]
     15 * [wiki:Yocto/gstreamer/streaming]
     16 * [wiki:Yocto/gstreamer/video]
     20=== media-ctl (v4l-utils package)
     21The {{{media-ctl}}} application from the v4l-utils package is a userspace application that uses the Linux Media Controller API to configure pipelines.
     23{{{media-ctl}}} has no manpage but looking at source:
     24 * --device <dev> (/dev/media0 default)
     25 * --entity <name> print the device name associated with given entity
     26 * --set-v4l2 <v4l2> comma sep list of formats to setup
     27 * --get-v4l2 <pad> print active format on a given pad
     28 * --set-dv <pad> configure DV timings on a given pad
     29 * --interactive modify links interactively
     30 * --links <linux> comma sep list of link descriptors to setup
     31 * --known-mbus-fmts list known formats and their numeric values
     32 * --print-topology print device topology
     33 * --reset reset all links to inactive
     35The best way to visualize what hardware components are avaialble to interconnect is to use print the topology and convert it to a graphic image:
     37media-ctl --print-topology
     38media-ctl --print-dot >
     39dot -Tpng > graph.png
     41 - dashed lines show possible connections; use these to determine the possibilities
     42 - solid lines show active connections
     43 - green boxes show media entities
     44 - yellow boxes show video4linux endpoints
     46For example the following is the graph for a GW5404:
     49You can find graphs for the Video capture devices present on Various gateworks boards at
     50 * [ GW5510]
     51 * [ GW5514]
     52 * [ GW540x]
     54Use {{{media-ctl --links}}} to link an entity source pad to an entity sink pad:
     56media-ctl --links "<entitya>:<srcpadn> -> <entityb>:<sinkpadn>[<flags>]
     58 * flags: 0=inactive 1=active
     60The basic configuration steps for a hardware media pipeline are:
     61 * reset all links with {{{media-ctl --reset}}}
     62 * configure links with {{{media-ctl --links}}}
     63 * configure pad formats with {{{media-ctl --set-v4l2}}}
     64 * configure subdev properties with {{{v4l2-ctl}}}
     65 * capture frames on the {{{/dev/video*}}} device
     69== IMX6 Media Capture
     70The [ IMX media controller documentation] in the Linux kernel is likely the best documentation to explain what's in the SoC and what each component can do:
     72The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
     73handles the flow of image frames to and from capture devices and
     74display devices.
     76For image capture, the IPU contains the following internal subunits:
     77- Image DMA Controller (IDMAC)
     78- Camera Serial Interface (CSI)
     79- Image Converter (IC)
     80- Sensor Multi-FIFO Controller (SMFC)
     81- Image Rotator (IRT)
     82- Video De-Interlacing or Combining Block (VDIC)
     84The IDMAC is the DMA controller for transfer of image frames to and from
     85memory. Various dedicated DMA channels exist for both video capture and
     86display paths. During transfer, the IDMAC is also capable of vertical
     87image flip, 8x8 block transfer (see IRT description), pixel component
     88re-ordering (for example UYVY to YUYV) within the same colorspace, and
     89packed <--> planar conversion. The IDMAC can also perform a simple
     90de-interlacing by interweaving even and odd lines during transfer
     91(without motion compensation which requires the VDIC).
     93The CSI is the backend capture unit that interfaces directly with
     94camera sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
     96The IC handles color-space conversion, resizing (downscaling and
     97upscaling), horizontal flip, and 90/270 degree rotation operations.
     99There are three independent "tasks" within the IC that can carry out
     100conversions concurrently: pre-process encoding, pre-process viewfinder,
     101and post-processing. Within each task, conversions are split into three
     102sections: downsizing section, main section (upsizing, flip, colorspace
     103conversion, and graphics plane combining), and rotation section.
     105The IPU time-shares the IC task operations. The time-slice granularity
     106is one burst of eight pixels in the downsizing section, one image line
     107in the main processing section, one image frame in the rotation section.
     109The SMFC is composed of four independent FIFOs that each can transfer
     110captured frames from sensors directly to memory concurrently via four
     111IDMAC channels.
     113The IRT carries out 90 and 270 degree image rotation operations. The
     114rotation operation is carried out on 8x8 pixel blocks at a time. This
     115operation is supported by the IDMAC which handles the 8x8 block transfer
     116along with block reordering, in coordination with vertical flip.
     118The VDIC handles the conversion of interlaced video to progressive, with
     119support for different motion compensation modes (low, medium, and high
     120motion). The deinterlaced output frames from the VDIC can be sent to the
     121IC pre-process viewfinder task for further conversions. The VDIC also
     122contains a Combiner that combines two image planes, with alpha blending
     123and color keying.
     125In addition to the IPU internal subunits, there are also two units
     126outside the IPU that are also involved in video capture on i.MX:
     127- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
     128  interface. This is a Synopsys !DesignWare core.
     129- Two video multiplexers for selecting among multiple sensor inputs
     130  to send to a CSI.
     132These entities will differ depending on the processor: IMX6D/Q have two IPU's and IMX6S/DL has only one IPU therefore the number of entities and pads differ. Additionally certain entities will have fixed configuration based on device-tree endpoints. The adv7180 analog media controller driver also presents an entity as does the tda1997x HDMI receiver media controller driver.
     136 * [ Linux IMX Media Driver Documentation]
     140=== Kernel Drivers
     141The following kernel drivers need to be enabled for IMX6 capture:
     142- CONFIG_MEDIA_CONTROLLER - for media controller drivers and framework
     143- CONFIG_VIDEO_V4L2_SUBDEV_API - for V4L2 subdev framework
     144- CONFIG_VIDEO_IMX_MEDIA - for IMX6 media controller
     145- CONFIG_VIDEO_MUX -
     146- CONFIG_VIDEO_ADV7180 - for Ventana Analog Video Capture
     148The kernel will display messages such as:
     150root@ventana:~# dmesg | grep media
     151[    0.343835] media: Linux media interface: v0.10
     152[    6.189587] imx-media: Registered subdev ipu2_csi1_mux
     153[    6.195114] imx-media: Registered subdev ipu1_csi0_mux
     154[    6.200928] imx-media: Registered subdev ipu1_vdic
     155[    6.206251] imx-media: Registered subdev ipu2_vdic
     156[    6.211493] imx-media: Registered subdev ipu1_ic_prp
     157[    6.216580] imx-media: Registered subdev ipu1_ic_prpenc
     158[    6.230109] imx-media: Registered subdev ipu1_ic_prpvf
     159[    6.242937] imx-media: Registered subdev ipu2_ic_prp
     160[    6.248023] imx-media: Registered subdev ipu2_ic_prpenc
     161[    6.261065] imx-media: Registered subdev ipu2_ic_prpvf
     162[    6.273972] imx-media: Registered subdev ipu1_csi0
     163[    6.292605] imx-media: Registered subdev ipu1_csi1
     164[    6.304263] imx-media: Registered subdev ipu2_csi0
     165[    6.315870] imx-media: Registered subdev ipu2_csi1
     166[    6.327352] imx-media: imx_media_create_link: ipu1_csi0:1 -> ipu1_ic_prp:0
     167[    6.334307] imx-media: imx_media_create_link: ipu1_csi0:1 -> ipu1_vdic:0
     168[    6.341128] imx-media: imx_media_create_link: ipu1_csi0_mux:2 -> ipu1_csi0:0
     169[    6.348200] imx-media: imx_media_create_link: ipu1_csi1:1 -> ipu1_ic_prp:0
     170[    6.355253] imx-media: imx_media_create_link: ipu1_csi1:1 -> ipu1_vdic:0
     171[    6.362026] imx-media: imx_media_create_link: ipu2_csi0:1 -> ipu2_ic_prp:0
     172[    6.368924] imx-media: imx_media_create_link: ipu2_csi0:1 -> ipu2_vdic:0
     173[    6.375717] imx-media: imx_media_create_link: ipu2_csi1:1 -> ipu2_ic_prp:0
     174[    6.382661] imx-media: imx_media_create_link: ipu2_csi1:1 -> ipu2_vdic:0
     175[    6.389419] imx-media: imx_media_create_link: ipu2_csi1_mux:2 -> ipu2_csi1:0
     176[    6.396529] imx-media: imx_media_create_link: ipu1_vdic:2 -> ipu1_ic_prp:0
     177[    6.403487] imx-media: imx_media_create_link: ipu1_ic_prp:1 -> ipu1_ic_prpenc:0
     178[    6.410858] imx-media: imx_media_create_link: ipu1_ic_prp:2 -> ipu1_ic_prpvf:0
     179[    6.418104] imx-media: imx_media_create_link: ipu2_vdic:2 -> ipu2_ic_prp:0
     180[    6.425095] imx-media: imx_media_create_link: ipu2_ic_prp:1 -> ipu2_ic_prpenc:0
     181[    6.432764] imx-media: imx_media_create_link: ipu2_ic_prp:2 -> ipu2_ic_prpvf:0
     183 - {{{/dev/media}}} devices are acted on by media controller framework - even though imx-media driver registers subdevs the {{{/dev/media}}} device doesn't show up until the endpoint (adv7180, tda1997x) driver loads because that's the endpoint in devicetree (both endpoints have to exist before it shows up as available)
     187=== Media Controller Pipelines
     188Various pipelines of media controller entities are created by linking them together in a fashion that starts with a Linux v4l2 capture chip and ends with a Linux v4l2 capture node (ie /dev/video4).
     190The media pipelines you create will depend on:
     191 * board model and CPU
     192 * features and performance you are after
     194The Ventana baseboards have the following video capture capabilities:
     195* GW54xx:
     196 - CVBS (Analog) video capture (ADV7180)
     197 - HDMI (Digital) video capture (TDA1997X)
     198* GW551x:
     199 - HDMI (Digital) video capture (TDA1997X)
     200* GW51xx/GW52xx/GW53xx/GW553x:
     201 - CVBS (Analog) video capture (ADV7180)
     203The IPU and CSI port for each capture device depends not only on the board model but the IMX6 SoC variant as well.
     205The following table shows the IPU/CSI combination for each capture device per board and SoC:
     206||= Board =||= SoC  =||= ADV7180 Analog CVBS =||= TDA1997x Digital HDMI =||
     207|| GW540x  || IMX6Q  || IPU2_CSI1             || IPU1_CSI0               ||
     208|| GW540x  || IMX6DL || IPU1_CSI1             || not supported           ||
     209|| GW530x  || IMX6Q  || IPU2_CSI1             || N/A ||
     210|| GW530x  || IMX6DL || IPU1_CSI1             || N/A ||
     211|| GW520x  || IMX6Q  || IPU2_CSI1             || N/A ||
     212|| GW520x  || IMX6DL || IPU1_CSI1             || N/A ||
     213|| GW510x  || IMX6Q/DL || IPU1_CSI0           || N/A ||
     214|| GW551x  || IMX6Q/DL || N/A                 || IPU1_CSI0 ||
     216The pipelines you construct with {{{media-ctl}}} will differ depending on what you are trying to accomplish as different hardware blocks (translated to entities in the media controller graph) can perform different translations such as colorspace conversion, scaling, rotation, or de-interlacing. In some cases there are multiple ways to achieve a certain translation with different trade-offs.
     218The IMX6 media driver registers several entities that take advantage of the IMX6 hardware (See: [ Documentation/media/v4l-drivers/imx]):
     219 * ipu_csi_mux - IPU CSI Multiplexer: Multiplexes the parallel capture bus and the MIPI capture bus to the Camera Sensor Interface (CSI)
     220 * ipu_csi - IPU Capture Sensor Interface (IDMAC) (latches parallel data on the bus, or de-serializes MIPI data from the bus). In addition to raw frame capture this supports:
     221  - cropping using the crop selection subdev API
     222  - fixed divide-by-two down-scaling independently in width and height using the compose selection subdev API
     223  - frame skipping specified by setting the frame intervals at the sink and source pads
     224  - simple de-interlacing without motion compensation (activated if the sink pad's field type is an interlaced type and the source pad field type is set to none)
     225  - pixel reordering within the same colorspace: ie: UYVY2X8 -> YUYV2X8
     226  - pixel plane conversion: YUV -> YUV420 (planar)
     227  - frame interval monitor: For decoders such as the adv718x which occasionally send corrupt fields during NTSC/PAL signal re-sync (too little or too many video lines) the IPU can trigger a mechanism to re-establish vertical sync by adding a dummy line every frame causing a rolling effect from image to image or even a permanent split image. This mechanism can be used to detect corrupt fields and send a V4L2_EVENT_IMX_FRAME_INTERVAL_ERROR that can be caught to issue a restart to correct the rolling/split image. The ipu_csi subdev includes custom controls to configure this 'Frame Interval Monitor' or FIM. The V4L FIM controls are V4L2_CID_IMX_FIM_NUM_SKIP, V4L2_CID_IMX_FIM_ICAP_CHANNEL and V4L2_CID_IMX_FIM_ICAP_EDGE.
     228 * ipu_vdic - Conversion of interlaced video to progressive with support for different motion compensation modes (low/med/high motion). The mode is specified with the V4L2_CID_DEINTERLACING_MODE menu control. This entity has a direct connection pad from the ipu_csi entity (in which only high motion mode can be used) but has an additional source pad that can be used with other output or mem2mem device nodes (where high, low, or medium motion mode can be used).
     229 * ipu_ic_prp - IC pre-processing entity: acts as a router routing data between source and sink pads. This entity can source to either ipu_ic_prpenc or ipu_ic_prpvf if it is sinking from ipu1_vdic, or it can sink directly from ipu_csi and source both output pads at the same time.
     230 * ipu_ic_prpenc - IC pre-processing encode entity provides:
     231  - color-space conversion
     232  - resizing (downscaling and upscaling)
     233  - horizontal and vertical flip, and 90/270 degree rotation using the standard V4L2 controls
     234  - simple de-interlace without motion compensation
     235  - pixel reordering
     236 * ipu_ic_prpvf - IC pre-process viewfinder entity: This is identical to the ipu_ic_prpenc entity supporting:
     237  - color-space conversion
     238  - horizontal and vertical flip, and 90/270 degree rotation using the standard V4L2 controls
     239  - simple de-interlace without motion compensation (unless the ipu_vdic is included in the pipeline as the the de-interlacing has already been done by the vdic with motion compensation)
     240  - pixel reordering
     241Ignoring the input and output pad numbering, and the IPU and CSI indices the following pseudo pipelines are useful:
     242 * raw capture of input resolution:
     244sensor -> ipu_csi_mux -> ipu_csi -> ipu_csi capture
     246  - use {{{tda19971}}} source for digital HDMI and {{{adv7180}}} source for analog CVBS
     247  - use correct indices on ipu and csi above depending on board/soc/input (see above table)
     248  - the IDMAC can be used for simple de-interlacing of interlaced video from the source by specifying a format with 'field:seq-tb' on the ipu_csi_mux and ipu_csi entities.
     249 * de-interlace via VDIC high motion compensation:
     251sensor -> ipu_csi_mux -> ipu_csi -> ipu_vdic -> ipu_ic_prp -> ipu_ic_prpvf -> ipu_ic_prpvf capture
     253  - use {{{tda19971}}} source for digital HDMI and {{{adv7180}}} source for analog CVBS
     254  - use correct indices on ipu and csi above depending on board/soc/input (see above table)
     255  - specify 'field:interlaced' on the format for ipu_csi_mux, ipu_csi, ipu_vdic and 'field:none' on ipu_ic_prp and ipu2_ic_prpvf (because ipu_ic_prp is the entity that can deinterlace with the IDMAC)
     256 * IC direct conversions:
     257  - uses preprocess encode entity to route frames directly from the CSI to the IC to carry out scaling up to 1024x1024 resolution, CSC, flipping, and image rotation:
     259sensor -> ipu_csi_mux -> ipu_csi -> ipu_ic_prp -> ipu_ic_prpenc -> ipu_ic_prpenc capture
     261 * Motion compensated de-interlace:
     262  - routes frames from the CSI direct to the VDIC for motion-compensated (high mode) de-interlacing, scaling up to 1024x1024, CSC, flip, and rotation:
     264sensor -> ipu_csi_mux -> ipu_csi -> ipu_vdic -> ipu_ic_prp -> ipu_ic_prpvf -> ipu_ic_prpvf capture
     267Note that to aid in the configuration of and for backward compatibility with V4L2 applications that use controls the capture device interfaces inherit controls from the active entities in the current pipeline thus these controls can be accessed via the entity subdev or from the active capture device interface. For example the FIM controls are available from the ipu_csi subdevs or from the active capture device.