[[PageOutline]] See also: * [wiki:ventana/ipu IMX6 Image Processing Unit] * [wiki:ventana/gpu IMX6 3D Graphics Processing Unit] * [wiki:ventana/codec IMX6 Codecs] [=#vpu] = IMX6 VPU - Video Processing Unit (Codec Support) The Video Processing Unit (VPU) in the IMX6 is a CODA960 licensed IP block provided by Chips&Media supporting the following codecs^^^1^^^: Encode: * H.264 BP/CBP * H.263 V2 * MPEG-4 SP * MPEG-2 * MJPEG Baseline Decode: * MPEG-2 MP/HP * H.264 BP/MP/HP * H.264 MVC BP/MP/HP * VC1 SP/MP/AP * MPEG-4/Xvid SP/ASP * DivX v3/4/5/6 * H.263 V2 * MJPEG Baseline * VP6/VebM VP8 Performance: * IMX6SOLO/DUALLITE (IMX6SDL): * HD 1080p30Hz + D1 decode * HD 1080p30Hz or 2x 720p encode * IMX6DUAL/I.MX6QUAD (IMX6DQ) (2 IPU's): * HD 1080p60Hz + D1 or 2x 1080p30Hz Decode * HD 1080p30Hz or 2x 720p encode Software support: * Yocto BSP packages: * firmware-imx-vpu-imx6q - contains firmware (/lib/firmware/vpu/vpu_fw_imx6q.bin) for IMX6DUAL/IMX6QUAD * firmware-imx-vpu-imx6d - contains firmware (/lib/firmware/vpu/vpu_fw_imx6d.bin) for IMX6DUALLITE/IMX6SOLO * libfslvpuwrap3 / libvpu4 - libraries * gst-fsl-plugin - gstreamer plugin support for IMX6 VPU * libfslcodec-* - various gstreamer libraries Notes: 1. **support for certain codecs may require software from NXP** 2. The CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry Documentation: * Yocto BSP's - i.MX_6_VPU_Application_Programming_Interface_Linux_Reference_Manual.pdf (from the Freescale i.MX6 Yocto BSP releases which you must download from Freescale) References: * ​[https://community.freescale.com/servlet/JiveServlet/download/317735-256858/VPU_API_RM_L3.0.35_1.1.0.pdf Freescale IMX6VPUAPI] documents the VPU API - This is from the LINUXDOCS_BUNDLE downloadable from Freescale's IMX6 site - check there for the most recent version [=#software] = VPU Software Support '''This page refers to the Linux coda driver and the GStreamer elements that use it. Note that coda support for the IMX6 was added to Linux in the 3.16 kernel. This page applies to Linux 3.16 and beyond. If you are using a kernel earlier than that for the Ventana IMX6 based boards (such as the Gateworks 3.14 kernel) you will need to refer the Yocto GStreamer pages documenting the proprietary VPU drivers and software from Freescale: * [wiki:Yocto/gstreamer] * [wiki:Yocto/gstreamer/compositing] * [wiki:Yocto/gstreamer/latency] * [wiki:Yocto/gstreamer/multimedia] * [wiki:Yocto/gstreamer/streaming] * [wiki:Yocto/gstreamer/video] ''' [=#coda] == Kernel Driver (coda) The Linux [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/coda coda] media driver as of Linux 3.16 supports the CODA960 in the IMX6 and currently provides access to the following: * decode - H264 - MPEG4 - MPEG2 * encode: - H264 - MPEG4 If the CODA driver is enabled in the kernel (CONFIG_VIDEO_CODA=m) and firmware is able to be loaded from {{{/lib/firmware/vpu}}} two {{{/dev/videoN}}} devices are created (one for encode, the other for decode): {{{#!bash root@imx6q-gw5404:~/# dmesg | grep coda [ 12.893365] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2 [ 12.893389] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin [ 15.168874] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin [ 15.201391] coda 2040000.vpu: Firmware code revision: 570363 [ 15.201421] coda 2040000.vpu: Initialized CODA960. [ 15.201433] coda 2040000.vpu: Firmware version: 3.1.1 [ 15.205867] coda 2040000.vpu: codec registered as /dev/video[8-9] root@imx6q-gw5404:~/# cat /sys/class/video4linux/video8/name coda-encoder root@imx6q-gw5404:~/# cat /sys/class/video4linux/video9/name coda-decoder }}} Note that the CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry. Note also that the CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough) Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the {{{v4l2-ctl -L}}} and {{{v4l2-ctl -l}}} parameters on the devices exposed by the coda driver: {{{#!bash # v4l2-ctl -d6 -L User Controls horizontal_flip 0x00980914 (bool) : default=0 value=0 vertical_flip 0x00980915 (bool) : default=0 value=0 Codec Controls video_gop_size 0x009909cb (int) : min=0 max=99 step=1 default=16 value=16 video_bitrate 0x009909cf (int) : min=0 max=32767000 step=1000 default=0 value=0 number_of_intra_refresh_mbs 0x009909d6 (int) : min=0 max=8160 step=1 default=0 value=0 sequence_header_mode 0x009909d8 (menu) : min=0 max=1 default=1 value=1 1: Joined With 1st Frame maximum_bytes_in_a_slice 0x009909db (int) : min=1 max=1073741823 step=1 default=500 value=500 number_of_mbs_in_a_slice 0x009909dc (int) : min=1 max=1073741823 step=1 default=1 value=1 slice_partitioning_method 0x009909dd (menu) : min=0 max=2 default=0 value=0 0: Single 1: Max Macroblocks 2: Max Bytes vbv_buffer_size 0x009909de (int) : min=0 max=262144 step=1 default=0 value=0 initial_delay_for_vbv_control 0x009909e1 (int) : min=0 max=32767 step=1 default=0 value=0 h264_i_frame_qp_value 0x00990a5e (int) : min=0 max=51 step=1 default=25 value=25 h264_p_frame_qp_value 0x00990a5f (int) : min=0 max=51 step=1 default=25 value=25 h264_maximum_qp_value 0x00990a62 (int) : min=0 max=51 step=1 default=51 value=51 h264_level 0x00990a67 (menu) : min=0 max=11 default=11 value=11 5: 2 8: 3 9: 3.1 10: 3.2 11: 4 h264_loop_filter_alpha_offset 0x00990a68 (int) : min=0 max=15 step=1 default=0 value=0 h264_loop_filter_beta_offset 0x00990a69 (int) : min=0 max=15 step=1 default=0 value=0 h264_loop_filter_mode 0x00990a6a (menu) : min=0 max=1 default=0 value=0 0: Enabled 1: Disabled h264_profile 0x00990a6b (menu) : min=0 max=0 default=0 value=0 0: Baseline mpeg4_i_frame_qp_value 0x00990a90 (int) : min=1 max=31 step=1 default=2 value=2 mpeg4_p_frame_qp_value 0x00990a91 (int) : min=1 max=31 step=1 default=2 value=2 mpeg4_level 0x00990a95 (menu) : min=0 max=7 default=7 value=7 7: 5 mpeg4_profile 0x00990a96 (menu) : min=0 max=0 default=0 value=0 0: Simple root@bionic-armhf:~# v4l2-ctl -d6 -l User Controls horizontal_flip 0x00980914 (bool) : default=0 value=0 vertical_flip 0x00980915 (bool) : default=0 value=0 Codec Controls video_gop_size 0x009909cb (int) : min=0 max=99 step=1 default=16 value=16 video_bitrate 0x009909cf (int) : min=0 max=32767000 step=1000 default=0 value=0 number_of_intra_refresh_mbs 0x009909d6 (int) : min=0 max=8160 step=1 default=0 value=0 sequence_header_mode 0x009909d8 (menu) : min=0 max=1 default=1 value=1 maximum_bytes_in_a_slice 0x009909db (int) : min=1 max=1073741823 step=1 default=500 value=500 number_of_mbs_in_a_slice 0x009909dc (int) : min=1 max=1073741823 step=1 default=1 value=1 slice_partitioning_method 0x009909dd (menu) : min=0 max=2 default=0 value=0 vbv_buffer_size 0x009909de (int) : min=0 max=262144 step=1 default=0 value=0 initial_delay_for_vbv_control 0x009909e1 (int) : min=0 max=32767 step=1 default=0 value=0 h264_i_frame_qp_value 0x00990a5e (int) : min=0 max=51 step=1 default=25 value=25 h264_p_frame_qp_value 0x00990a5f (int) : min=0 max=51 step=1 default=25 value=25 h264_maximum_qp_value 0x00990a62 (int) : min=0 max=51 step=1 default=51 value=51 h264_level 0x00990a67 (menu) : min=0 max=11 default=11 value=11 h264_loop_filter_alpha_offset 0x00990a68 (int) : min=0 max=15 step=1 default=0 value=0 h264_loop_filter_beta_offset 0x00990a69 (int) : min=0 max=15 step=1 default=0 value=0 h264_loop_filter_mode 0x00990a6a (menu) : min=0 max=1 default=0 value=0 h264_profile 0x00990a6b (menu) : min=0 max=0 default=0 value=0 mpeg4_i_frame_qp_value 0x00990a90 (int) : min=1 max=31 step=1 default=2 value=2 mpeg4_p_frame_qp_value 0x00990a91 (int) : min=1 max=31 step=1 default=2 value=2 mpeg4_level 0x00990a95 (menu) : min=0 max=7 default=7 value=7 mpeg4_profile 0x00990a96 (menu) : min=0 max=0 default=0 value=0 }}} [=#firmware] === VPU Firmware The VPU requires firmware that is loaded by the coda driver (see [https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/coda/coda-common.c#L2477 here]). You can obtain and extract the firmware from various Freescale / NXP sources included in the Freescale / NXP Yocto BSP. Example: * v3.1.1 from firmware-imx_7.6 {{{#!bash wget https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/firmware-imx-7.6.bin /bin/sh firmware-imx-* mkdir /lib/firmware/vpu mv firmware-imx-7.6/firmware/vpu/vpu_fw_imx6*.bin /lib/firmware/vpu }}} [=#gstreamer] == GStreamer The popular GStreamer framework provides access to hardware decode via plugins. The [https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2videodec.c gstv4l2videodec plugin] (since GStreamer 1.1) can utilize the Linux coda driver to provide hardware decode for H264, MPEG2 and MPEG4. The [https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2videoenc.c gstv4l2videoenc plugin] (since GStreamer 1.14) can utilize the Linux coda driver to provide hardware encode for H264 and MPEG4. The GStreamer elements that are exposed are typed elements: * gstv4l2videodec: {{{#!bash root@bionic-armhf:~# gst-inspect-1.0 | grep -P "v4l2.*dec" video4linux2: v4l2mpeg4dec: V4L2 MPEG4 Decoder video4linux2: v4l2mpeg2dec: V4L2 MPEG2 Decoder video4linux2: v4l2h264dec: V4L2 H264 Decoder }}} * gstv4l2videoenc {{{#!bash root@bionic-armhf:~# gst-inspect-1.0 | grep -P "v4l2.*enc" video4linux2: v4l2h264enc: V4L2 H.264 Encoder video4linux2: v4l2mpeg4enc: V4L2 MPEG4 Encoder }}} Note that the CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry Examples: * Encode and stream 1080p videotestsrc using UDP/RTP/H264: {{{#!bash gst-launch-1.0 videotestsrc ! \ 'video/x-raw,width=1920,height=1080,framerate=30/1,format=NV12,colorimetry=bt709' ! \ v4l2h264enc output-io-mode=4 ! rtph264pay ! udpsink host=$SERVER port=5000 }}} * Decode and display an H264 video stream: {{{#!bash gst-launch-1.0 udpsrc port=5000 ! \ application/x-rtp,payload=96 ! \ rtph264depay ! v4l2h264dec ! kmssink }}} === Troubleshooting - If you do not have the VPU firmware installed you will see the {{{/dev/videoN}}} devices associated with the coda driver. - CODA driver needs CMA memory. - In bootloader "setenv extra cma=64M"