Changes between Version 4 and Version 5 of linux/media

Timestamp:

02/19/2019 10:20:10 PM (6 years ago)

Author:

Tim Harvey

Comment:

add mem2mem and coda details and examples

linux/media

@@ -323 +323 @@
 # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=5001 caps=application/x-rtp,payload=96 ! rtph264depay ! decodebin ! autovideosink'
 }}}
+
+
+[=#mem2mem]
+== IMX6 ipu_ic_pp mem2mem CSC/scale/crop/rotate/flip conversion
+A Linux V4L2 MEM2MEM imx-media driver exists that allows utilizing the IMX6 IPU Image Converter hardware blocks (IC) to perform hardware colorspace conversion (CSC), scaling, cropping, rotation, and flip operations.
+
+The GStreamer {{{video4linux2}}} plugin provides an element that uses this driver to expose these capabilities to GStreamer applications.
+
+Notes:
+ - for GStreamer-1.14 the name of the element depends on the video device the driver registers with the kernel (ie v4l2video8convert if mem2mem driver registers /dev/video8)
+ - for GStreamer master (in development) the name of the element is always 'v4l2videoconvert'
+ - the {{{kmssink}}} examples below need a {{{can-scale=false}}} property to tell GStreamer not to scale via the KMS driver (as the IMX6 KMS driver does not support scaling)
+ - ensure that the input format differs from the output format otherwise GStreamer will bypass the conversion completely; note that GStreamer doesn't understand flipping or rotation as part of the format. Gstreamer master (in development) adds a 'disable-passthrough' property to the v4l2videoconvert entity that can be set to force the conversion regardless of input and output format
+ - when using imx entities (ie capture, encode/decode, mem2mem, display) you can specify 'output-io-mode=dmabuf-import' to share dmabuf pointers for a zero-copy pipeline however if using non imx entities (ie videotestsrc) you must omit these as you can not ensure the buffers share the alignment/stride necessary to share dmabuf pointers
+
+Examples:
+ * Ensure mem2mem is in your kernel:
+{{{#!bash
+~# dmesg | grep mem2mem
+[   18.356023] imx-media: Registered ipu_ic_pp mem2mem as /dev/video8
+}}}
+ * Ensure GStreamer element exists:
+{{{#!bash
+~# gst-inspect-1.0 | grep -e "v4l2.*convert"
+video4linux2:  v4l2video8convert: V4L2 Video Converter
+}}}
+  - Note that for GStreamer-1.14, the name of the element depends on the video device the driver registers with the kernel (video8 in the above example). This changes in GStreamer-1.16 to always be 'v4l2videoconvert'
+ * scale/rotate/flip using {{{videotestsrc}}} (can not use dmabufs for this as it is a non-imx entity)
+{{{#!bash
+# upscale
+gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
+    v4l2convert ! \
+    video/x-raw,width=640,height=480 ! kmssink can-scale=false
+# downscale
+gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480 ! \
+    v4l2convert ! \
+    video/x-raw,width=320,height=240 ! kmssink can-scale=false
+# rotate
+gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
+    v4l2convert extra-controls=cid,rotate=90 ! \
+    video/x-raw,width=240,height=320 ! kmssink can-scale=false
+# hflip
+gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
+    v4l2convert extra-controls=cid,horizontal_flip=1 ! \
+    video/x-raw,width=640,height=480 ! kmssink can-scale=false
+# vflip
+gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
+    v4l2convert extra-controls=cid,vertical_flip=1 ! \
+    video/x-raw,width=640,height=480 ! kmssink can-scale=false
+}}}
+  - note the above examples force the input format (resolution in this case) to differ from the output format otherwise gstreamer will bypass the v4l2convert entity thinking it not necessary as gstreamer does not understand the flip/rotation properties. GStreamer master (in development) adds the 'disable-passthrough' property which can be enabled to force disabling passthrough
+
+
+[=#coda]
+== IMX6 coda encode
+The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer {{{video4linux2}}} plugin provides encode/decode elements that tap into this.
+
+Examples:
+ * Ensure CODA is in your kernel:
+{{{#!bash
+~# dmesg | grep coda    
+[   16.721698] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2
+[   16.721724] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin
+[   18.381136] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin
+[   18.433648] coda 2040000.vpu: Firmware code revision: 570363
+[   18.433683] coda 2040000.vpu: Initialized CODA960.
+[   18.433706] coda 2040000.vpu: Firmware version: 3.1.1
+[   18.442312] coda 2040000.vpu: codec registered as /dev/video[9-10]
+}}}
+ * Ensure GStreamer encode elements exists:
+{{{#!bash
+~# gst-inspect-1.0 | grep -e "v4l2.*enc"
+video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
+video4linux2:  v4l2mpeg4enc: V4L2 MPEG4 Encoder
+}}}
+ * capture, scale, rotate, flip and encode using imx-media capture device mem2mem device and coda device (can use dmabufs for zero-copy)
+{{{#!bash
+# encode
+gst-launch-1.0 v4l2src device=$DEVICE ! \
+     v4l2convert output-io-mode=dmabuf-import ! \
+     v4l2h264enc output-io-mode=dmabuf-import ! \
+     rtph264pay ! udpsink host=$SERVER port=$PORT
+# scale/encode
+gst-launch-1.0 v4l2src device=$DEVICE ! \
+     v4l2convert output-io-mode=dmabuf-import ! \
+     video/x-raw,width=1440,height=960 ! \
+     v4l2h264enc output-io-mode=dmabuf-import ! \
+     rtph264pay ! udpsink host=$SERVER port=$PORT
+# scale/flip/encode
+gst-launch-1.0 v4l2src device=$DEVICE ! \
+     v4l2convert output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \
+     video/x-raw,width=1440,height=960 ! \
+     v4l2h264enc output-io-mode=dmabuf-import ! \
+     rtph264pay ! udpsink host=$SERVER port=$PORT
+# scale/rotate/encode
+gst-launch-1.0 v4l2src device=$DEVICE ! \
+     v4l2convert output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \
+     video/x-raw,width=1440,height=960 ! \
+     v4l2h264enc output-io-mode=dmabuf-import ! \
+     rtph264pay ! udpsink host=$SERVER port=$PORT
+}}}