wiki:ventana/vpu

  1. IMX6 VPU - Video Processing Unit (Codec Support)
  2. VPU Software Support
    1. Kernel Driver (coda)
      1. VPU Firmware
    2. GStreamer
      1. Troubleshooting

See also:

IMX6 VPU - Video Processing Unit (Codec Support)

The Video Processing Unit (VPU) in the IMX6 is a CODA960 licensed IP block provided by Chips&Media supporting the following codecs1:

Encode:

  • H.264 BP/CBP
  • H.263 V2
  • MPEG-4 SP
  • MPEG-2
  • MJPEG Baseline

Decode:

  • MPEG-2 MP/HP
  • H.264 BP/MP/HP
  • H.264 MVC BP/MP/HP
  • VC1 SP/MP/AP
  • MPEG-4/Xvid SP/ASP
  • DivX v3/4/5/6
  • H.263 V2
  • MJPEG Baseline
  • VP6/VebM VP8

Performance:

  • IMX6SOLO/DUALLITE (IMX6SDL):
    • HD 1080p30Hz + D1 decode
    • HD 1080p30Hz or 2x 720p encode
  • IMX6DUAL/I.MX6QUAD (IMX6DQ) (2 IPU's):
    • HD 1080p60Hz + D1 or 2x 1080p30Hz Decode
    • HD 1080p30Hz or 2x 720p encode

Software support:

  • Yocto BSP packages:
    • firmware-imx-vpu-imx6q - contains firmware (/lib/firmware/vpu/vpu_fw_imx6q.bin) for IMX6DUAL/IMX6QUAD
    • firmware-imx-vpu-imx6d - contains firmware (/lib/firmware/vpu/vpu_fw_imx6d.bin) for IMX6DUALLITE/IMX6SOLO
    • libfslvpuwrap3 / libvpu4 - libraries
    • gst-fsl-plugin - gstreamer plugin support for IMX6 VPU
    • libfslcodec-* - various gstreamer libraries

Notes:

  1. support for certain codecs may require software from NXP
  2. The CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry

Documentation:

  • Yocto BSP's - i.MX_6_VPU_Application_Programming_Interface_Linux_Reference_Manual.pdf (from the Freescale i.MX6 Yocto BSP releases which you must download from Freescale)

References:

  • ​​Freescale IMX6VPUAPI documents the VPU API - This is from the LINUXDOCS_BUNDLE downloadable from Freescale's IMX6 site - check there for the most recent version

VPU Software Support

This page refers to the Linux coda driver and the GStreamer elements that use it. Note that coda support for the IMX6 was added to Linux in the 3.16 kernel. This page applies to Linux 3.16 and beyond. If you are using a kernel earlier than that for the Ventana IMX6 based boards (such as the Gateworks 3.14 kernel) you will need to refer the Yocto GStreamer pages documenting the proprietary VPU drivers and software from Freescale:

Kernel Driver (coda)

The Linux ​coda media driver as of Linux 3.16 supports the CODA960 in the IMX6 and currently provides access to the following:

  • decode
    • H264
    • MPEG4
    • MPEG2
  • encode:
    • H264
    • MPEG4

If the CODA driver is enabled in the kernel (CONFIG_VIDEO_CODA=m) and firmware is able to be loaded from /lib/firmware/vpu two /dev/videoN devices are created (one for encode, the other for decode): 

root@imx6q-gw5404:~/# dmesg | grep coda
[   12.893365] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2
[   12.893389] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin
[   15.168874] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin
[   15.201391] coda 2040000.vpu: Firmware code revision: 570363
[   15.201421] coda 2040000.vpu: Initialized CODA960.
[   15.201433] coda 2040000.vpu: Firmware version: 3.1.1
[   15.205867] coda 2040000.vpu: codec registered as /dev/video[8-9]
root@imx6q-gw5404:~/# cat /sys/class/video4linux/video8/name 
coda-encoder
root@imx6q-gw5404:~/# cat /sys/class/video4linux/video9/name 
coda-decoder

Note that the CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry.

Note also that the CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough)

Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the v4l2-ctl -L and v4l2-ctl -l parameters on the devices exposed by the coda driver: 

# v4l2-ctl -d6 -L

User Controls

                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0

Codec Controls

                 video_gop_size 0x009909cb (int)    : min=0 max=99 step=1 default=16 value=16
                  video_bitrate 0x009909cf (int)    : min=0 max=32767000 step=1000 default=0 value=0
    number_of_intra_refresh_mbs 0x009909d6 (int)    : min=0 max=8160 step=1 default=0 value=0
           sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1
                                1: Joined With 1st Frame
       maximum_bytes_in_a_slice 0x009909db (int)    : min=1 max=1073741823 step=1 default=500 value=500
       number_of_mbs_in_a_slice 0x009909dc (int)    : min=1 max=1073741823 step=1 default=1 value=1
      slice_partitioning_method 0x009909dd (menu)   : min=0 max=2 default=0 value=0
                                0: Single
                                1: Max Macroblocks
                                2: Max Bytes
                vbv_buffer_size 0x009909de (int)    : min=0 max=262144 step=1 default=0 value=0
  initial_delay_for_vbv_control 0x009909e1 (int)    : min=0 max=32767 step=1 default=0 value=0
          h264_i_frame_qp_value 0x00990a5e (int)    : min=0 max=51 step=1 default=25 value=25
          h264_p_frame_qp_value 0x00990a5f (int)    : min=0 max=51 step=1 default=25 value=25
          h264_maximum_qp_value 0x00990a62 (int)    : min=0 max=51 step=1 default=51 value=51
                     h264_level 0x00990a67 (menu)   : min=0 max=11 default=11 value=11
                                5: 2
                                8: 3
                                9: 3.1
                                10: 3.2
                                11: 4
  h264_loop_filter_alpha_offset 0x00990a68 (int)    : min=0 max=15 step=1 default=0 value=0
   h264_loop_filter_beta_offset 0x00990a69 (int)    : min=0 max=15 step=1 default=0 value=0
          h264_loop_filter_mode 0x00990a6a (menu)   : min=0 max=1 default=0 value=0
                                0: Enabled
                                1: Disabled
                   h264_profile 0x00990a6b (menu)   : min=0 max=0 default=0 value=0
                                0: Baseline
         mpeg4_i_frame_qp_value 0x00990a90 (int)    : min=1 max=31 step=1 default=2 value=2
         mpeg4_p_frame_qp_value 0x00990a91 (int)    : min=1 max=31 step=1 default=2 value=2
                    mpeg4_level 0x00990a95 (menu)   : min=0 max=7 default=7 value=7
                                7: 5
                  mpeg4_profile 0x00990a96 (menu)   : min=0 max=0 default=0 value=0
                                0: Simple
root@bionic-armhf:~# v4l2-ctl -d6 -l

User Controls

                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0

Codec Controls

                 video_gop_size 0x009909cb (int)    : min=0 max=99 step=1 default=16 value=16
                  video_bitrate 0x009909cf (int)    : min=0 max=32767000 step=1000 default=0 value=0
    number_of_intra_refresh_mbs 0x009909d6 (int)    : min=0 max=8160 step=1 default=0 value=0
           sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1
       maximum_bytes_in_a_slice 0x009909db (int)    : min=1 max=1073741823 step=1 default=500 value=500
       number_of_mbs_in_a_slice 0x009909dc (int)    : min=1 max=1073741823 step=1 default=1 value=1
      slice_partitioning_method 0x009909dd (menu)   : min=0 max=2 default=0 value=0
                vbv_buffer_size 0x009909de (int)    : min=0 max=262144 step=1 default=0 value=0
  initial_delay_for_vbv_control 0x009909e1 (int)    : min=0 max=32767 step=1 default=0 value=0
          h264_i_frame_qp_value 0x00990a5e (int)    : min=0 max=51 step=1 default=25 value=25
          h264_p_frame_qp_value 0x00990a5f (int)    : min=0 max=51 step=1 default=25 value=25
          h264_maximum_qp_value 0x00990a62 (int)    : min=0 max=51 step=1 default=51 value=51
                     h264_level 0x00990a67 (menu)   : min=0 max=11 default=11 value=11
  h264_loop_filter_alpha_offset 0x00990a68 (int)    : min=0 max=15 step=1 default=0 value=0
   h264_loop_filter_beta_offset 0x00990a69 (int)    : min=0 max=15 step=1 default=0 value=0
          h264_loop_filter_mode 0x00990a6a (menu)   : min=0 max=1 default=0 value=0
                   h264_profile 0x00990a6b (menu)   : min=0 max=0 default=0 value=0
         mpeg4_i_frame_qp_value 0x00990a90 (int)    : min=1 max=31 step=1 default=2 value=2
         mpeg4_p_frame_qp_value 0x00990a91 (int)    : min=1 max=31 step=1 default=2 value=2
                    mpeg4_level 0x00990a95 (menu)   : min=0 max=7 default=7 value=7
                  mpeg4_profile 0x00990a96 (menu)   : min=0 max=0 default=0 value=0

VPU Firmware

The VPU requires firmware that is loaded by the coda driver (see ​here). You can obtain and extract the firmware from various Freescale / NXP sources included in the Freescale / NXP Yocto BSP.

Example:

  • v3.1.1 from firmware-imx_7.6 
    wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/firmware-imx-7.6.bin
/bin/sh firmware-imx-*
mkdir /lib/firmware/vpu
mv firmware-imx-7.6/firmware/vpu/vpu_fw_imx6*.bin /lib/firmware/vpu

GStreamer

The popular GStreamer framework provides access to hardware decode via plugins.

The ​gstv4l2videodec plugin (since GStreamer 1.1) can utilize the Linux coda driver to provide hardware decode for H264, MPEG2 and MPEG4.

The ​gstv4l2videoenc plugin (since GStreamer 1.14) can utilize the Linux coda driver to provide hardware encode for H264 and MPEG4.

The GStreamer elements that are exposed are typed elements:

  • gstv4l2videodec: 
    root@bionic-armhf:~# gst-inspect-1.0 | grep -P "v4l2.*dec"
video4linux2:  v4l2mpeg4dec: V4L2 MPEG4 Decoder
video4linux2:  v4l2mpeg2dec: V4L2 MPEG2 Decoder
video4linux2:  v4l2h264dec: V4L2 H264 Decoder
  • gstv4l2videoenc 
    root@bionic-armhf:~# gst-inspect-1.0 | grep -P "v4l2.*enc"
video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
video4linux2:  v4l2mpeg4enc: V4L2 MPEG4 Encoder

Note that the CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry

Examples:

  • Encode and stream 1080p videotestsrc using UDP/RTP/H264: 
    gst-launch-1.0 videotestsrc ! \
  'video/x-raw,width=1920,height=1080,framerate=30/1,format=NV12,colorimetry=bt709' ! \
  v4l2h264enc output-io-mode=4 ! rtph264pay ! udpsink host=$SERVER port=5000
  • Decode and display an H264 video stream: 
    gst-launch-1.0 udpsrc port=5000 ! \
  application/x-rtp,payload=96 ! \
  rtph264depay ! v4l2h264dec ! kmssink

Troubleshooting

  • If you do not have the VPU firmware installed you will see the /dev/videoN devices associated with the coda driver.
  • CODA driver needs CMA memory.
    • In bootloader "setenv extra cma=64M"
Last modified 6 years ago Last modified on 01/22/2019 08:24:56 PM
Note: See TracWiki for help on using the wiki.