[[PageOutline]]

See also:
 * [wiki:ventana/ipu IMX6 Image Processing Unit]
 * [wiki:ventana/gpu IMX6 3D Graphics Processing Unit]
 * [wiki:ventana/codec IMX6 Codecs]

[=#vpu]
= IMX6 VPU - Video Processing Unit (Codec Support)
The Video Processing Unit (VPU) in the IMX6 is a CODA960 licensed IP block provided by Chips&Media supporting the following codecs:

Encode:
 * H.264 BP/CBP
 * H.263 V2
 * MPEG-4 SP
 * MPEG-2
 * MJPEG Baseline

Decode:
 * MPEG-2 MP/HP
 * H.264 BP/MP/HP
 * H.264 MVC BP/MP/HP
 * VC1 SP/MP/AP
 * MPEG-4/Xvid SP/ASP
 * DivX v3/4/5/6
 * H.263 V2
 * MJPEG Baseline
 * VP6/VebM VP8

Performance:
 * IMX6SOLO/DUALLITE (IMX6SDL):
  * HD 1080p30Hz + D1 decode
  * HD 1080p30Hz or 2x 720p encode
 * IMX6DUAL/I.MX6QUAD (IMX6DQ) (2 IPU's):
  * HD 1080p60Hz + D1 or 2x 1080p30Hz Decode
  * HD 1080p30Hz or 2x 720p encode

Software support:
 * Yocto BSP packages:
  * firmware-imx-vpu-imx6q - contains firmware (/lib/firmware/vpu/vpu_fw_imx6q.bin) for IMX6DUAL/IMX6QUAD
  * firmware-imx-vpu-imx6d - contains firmware (/lib/firmware/vpu/vpu_fw_imx6d.bin) for IMX6DUALLITE/IMX6SOLO
  * libfslvpuwrap3 / libvpu4 - libraries
  * gst-fsl-plugin - gstreamer plugin support for IMX6 VPU
  * libfslcodec-* - various gstreamer libraries

Documentation:
 * Yocto BSP's - i.MX_6_VPU_Application_Programming_Interface_Linux_Reference_Manual.pdf (from the Freescale i.MX6 Yocto BSP releases which you must download from Freescale)

References:
 * ​[https://community.freescale.com/servlet/JiveServlet/download/317735-256858/VPU_API_RM_L3.0.35_1.1.0.pdf Freescale IMX6VPUAPI] documents the VPU API - This is from the LINUXDOCS_BUNDLE downloadable from Freescale's IMX6 site - check there for the most recent version


[=#software]
= VPU Software Support

'''This page refers to the Linux coda driver and the GStreamer elements that use it. Note that coda support for the IMX6 was added to Linux in the 3.16 kernel. This page applies to Linux 3.16 and beyond. If you are using a kernel earlier than that for the Ventana IMX6 based boards (such as the Gateworks 3.14 kernel) you will need to refer the Yocto GStreamer pages documenting the proprietary VPU drivers and software from Freescale:
 * [wiki:Yocto/gstreamer]
 * [wiki:Yocto/gstreamer/compositing]
 * [wiki:Yocto/gstreamer/latency]
 * [wiki:Yocto/gstreamer/multimedia]
 * [wiki:Yocto/gstreamer/streaming]
 * [wiki:Yocto/gstreamer/video]
'''

[=#coda]
== Kernel Driver (coda)
'''Note that coda support for the IMX6 was added to Linux in the 3.16 kernel. This page applies to Linux 3.16 and beyond. If you are using a kernel earlier than that for the Ventana IMX6 based boards (such as the Gateworks 3.14 kernel) you will need to refer the Yocto GStreamer pages documenting the proprietary VPU drivers and software from Freescale:
 * [wiki:Yocto/gstreamer]
 * [wiki:Yocto/gstreamer/compositing]
 * [wiki:Yocto/gstreamer/latency]
 * [wiki:Yocto/gstreamer/multimedia]
 * [wiki:Yocto/gstreamer/streaming]
 * [wiki:Yocto/gstreamer/video]
'''

The Linux [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/coda coda] media driver as of Linux 3.16 supports the CODA960 in the IMX6 and currently provides access to H264 encode/decode, MPEG4 encode/decode, and MPEG2 decode.

Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the {{{v4l2-ctl -L}}} and {{{v4l2-ctl -l}}} parameters on the devices exposed by the coda driver:
{{{#!bash
# v4l2-ctl -d6 -L

User Controls

                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0

Codec Controls

                 video_gop_size 0x009909cb (int)    : min=0 max=99 step=1 default=16 value=16
                  video_bitrate 0x009909cf (int)    : min=0 max=32767000 step=1000 default=0 value=0
    number_of_intra_refresh_mbs 0x009909d6 (int)    : min=0 max=8160 step=1 default=0 value=0
           sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1
                                1: Joined With 1st Frame
       maximum_bytes_in_a_slice 0x009909db (int)    : min=1 max=1073741823 step=1 default=500 value=500
       number_of_mbs_in_a_slice 0x009909dc (int)    : min=1 max=1073741823 step=1 default=1 value=1
      slice_partitioning_method 0x009909dd (menu)   : min=0 max=2 default=0 value=0
                                0: Single
                                1: Max Macroblocks
                                2: Max Bytes
                vbv_buffer_size 0x009909de (int)    : min=0 max=262144 step=1 default=0 value=0
  initial_delay_for_vbv_control 0x009909e1 (int)    : min=0 max=32767 step=1 default=0 value=0
          h264_i_frame_qp_value 0x00990a5e (int)    : min=0 max=51 step=1 default=25 value=25
          h264_p_frame_qp_value 0x00990a5f (int)    : min=0 max=51 step=1 default=25 value=25
          h264_maximum_qp_value 0x00990a62 (int)    : min=0 max=51 step=1 default=51 value=51
                     h264_level 0x00990a67 (menu)   : min=0 max=11 default=11 value=11
                                5: 2
                                8: 3
                                9: 3.1
                                10: 3.2
                                11: 4
  h264_loop_filter_alpha_offset 0x00990a68 (int)    : min=0 max=15 step=1 default=0 value=0
   h264_loop_filter_beta_offset 0x00990a69 (int)    : min=0 max=15 step=1 default=0 value=0
          h264_loop_filter_mode 0x00990a6a (menu)   : min=0 max=1 default=0 value=0
                                0: Enabled
                                1: Disabled
                   h264_profile 0x00990a6b (menu)   : min=0 max=0 default=0 value=0
                                0: Baseline
         mpeg4_i_frame_qp_value 0x00990a90 (int)    : min=1 max=31 step=1 default=2 value=2
         mpeg4_p_frame_qp_value 0x00990a91 (int)    : min=1 max=31 step=1 default=2 value=2
                    mpeg4_level 0x00990a95 (menu)   : min=0 max=7 default=7 value=7
                                7: 5
                  mpeg4_profile 0x00990a96 (menu)   : min=0 max=0 default=0 value=0
                                0: Simple
root@bionic-armhf:~# v4l2-ctl -d6 -l

User Controls

                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0

Codec Controls

                 video_gop_size 0x009909cb (int)    : min=0 max=99 step=1 default=16 value=16
                  video_bitrate 0x009909cf (int)    : min=0 max=32767000 step=1000 default=0 value=0
    number_of_intra_refresh_mbs 0x009909d6 (int)    : min=0 max=8160 step=1 default=0 value=0
           sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1
       maximum_bytes_in_a_slice 0x009909db (int)    : min=1 max=1073741823 step=1 default=500 value=500
       number_of_mbs_in_a_slice 0x009909dc (int)    : min=1 max=1073741823 step=1 default=1 value=1
      slice_partitioning_method 0x009909dd (menu)   : min=0 max=2 default=0 value=0
                vbv_buffer_size 0x009909de (int)    : min=0 max=262144 step=1 default=0 value=0
  initial_delay_for_vbv_control 0x009909e1 (int)    : min=0 max=32767 step=1 default=0 value=0
          h264_i_frame_qp_value 0x00990a5e (int)    : min=0 max=51 step=1 default=25 value=25
          h264_p_frame_qp_value 0x00990a5f (int)    : min=0 max=51 step=1 default=25 value=25
          h264_maximum_qp_value 0x00990a62 (int)    : min=0 max=51 step=1 default=51 value=51
                     h264_level 0x00990a67 (menu)   : min=0 max=11 default=11 value=11
  h264_loop_filter_alpha_offset 0x00990a68 (int)    : min=0 max=15 step=1 default=0 value=0
   h264_loop_filter_beta_offset 0x00990a69 (int)    : min=0 max=15 step=1 default=0 value=0
          h264_loop_filter_mode 0x00990a6a (menu)   : min=0 max=1 default=0 value=0
                   h264_profile 0x00990a6b (menu)   : min=0 max=0 default=0 value=0
         mpeg4_i_frame_qp_value 0x00990a90 (int)    : min=1 max=31 step=1 default=2 value=2
         mpeg4_p_frame_qp_value 0x00990a91 (int)    : min=1 max=31 step=1 default=2 value=2
                    mpeg4_level 0x00990a95 (menu)   : min=0 max=7 default=7 value=7
                  mpeg4_profile 0x00990a96 (menu)   : min=0 max=0 default=0 value=0
}}}


[=#firmware]
=== VPU Firmware
The VPU requires firmware that is loaded by the coda driver (see [https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/coda/coda-common.c#L2477 here]). You can obtain and extract the firmware from various Freescale / NXP sources included in the Freescale / NXP Yocto BSP.

Example:
 * v3.1.1 from firmware-imx_7.6
{{{#!bash
wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/firmware-imx-7.6.bin
/bin/sh firmware-imx-*
mkdir /lib/firmware/vpu
mv firmware-imx-7.6/firmware/vpu/vpu_fw_imx6*.bin /lib/firmware/vpu
}}}


[=#gstreamer]
== GStreamer
The popular GStreamer framework provides access to hardware decode via plugins.

The [https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2videodec.c gstv4l2videodec plugin] (since GStreamer 1.1) can utilize the Linux coda driver to provide hardware decode for H264, MPEG2 and MPEG4.

The [https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2videoenc.c gstv4l2videoenc plugin] (since GStreamer 1.14) can utilize the Linux coda driver to provide hardware encode for H264 and MPEG4.

The GStreamer elements that are exposed are typed elements:
 * gstv4l2videodec:
{{{#!bash
root@bionic-armhf:~# gst-inspect-1.0 | grep -P "v4l2.*dec"
video4linux2:  v4l2mpeg4dec: V4L2 MPEG4 Decoder
video4linux2:  v4l2mpeg2dec: V4L2 MPEG2 Decoder
video4linux2:  v4l2h264dec: V4L2 H264 Decoder
}}}
 * gstv4l2videoenc
{{{#!bash
root@bionic-armhf:~# gst-inspect-1.0 | grep -P "v4l2.*enc"
video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
video4linux2:  v4l2mpeg4enc: V4L2 MPEG4 Encoder
}}}


Examples:
 * Encode and stream 1080p videotestsrc using UDP/RTP/H264:
{{{#!bash
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=1920, height=1080, framerate=30/1, format=NV12, colorimetry=bt709' ! v4l2h264enc output-io-mode=4 ! rtph264pay ! udpsink host=$SERVER port=5000
}}}
 * Decode and display an H264 video stream:
{{{#!bash
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,payload=96 ! rtph264depay ! v4l2h264dec ! kmssink
}}}

=== Troubleshooting
- If you do not have the VPU firmware installed you will see these elements. 
- CODA driver needs CMA memory.
 - In bootloader "setenv extra cma=64M"