Changes between Version 4 and Version 5 of linux/media


Ignore:
Timestamp:
02/19/2019 10:20:10 PM (4 years ago)
Author:
Tim Harvey
Comment:

add mem2mem and coda details and examples

Legend:

Unmodified
Added
Removed
Modified
  • linux/media

    v4 v5  
    323323# client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=5001 caps=application/x-rtp,payload=96 ! rtph264depay ! decodebin ! autovideosink'
    324324}}}
     325
     326
     327[=#mem2mem]
     328== IMX6 ipu_ic_pp mem2mem CSC/scale/crop/rotate/flip conversion
     329A Linux V4L2 MEM2MEM imx-media driver exists that allows utilizing the IMX6 IPU Image Converter hardware blocks (IC) to perform hardware colorspace conversion (CSC), scaling, cropping, rotation, and flip operations.
     330
     331The GStreamer {{{video4linux2}}} plugin provides an element that uses this driver to expose these capabilities to GStreamer applications.
     332
     333Notes:
     334 - for GStreamer-1.14 the name of the element depends on the video device the driver registers with the kernel (ie v4l2video8convert if mem2mem driver registers /dev/video8)
     335 - for GStreamer master (in development) the name of the element is always 'v4l2videoconvert'
     336 - the {{{kmssink}}} examples below need a {{{can-scale=false}}} property to tell GStreamer not to scale via the KMS driver (as the IMX6 KMS driver does not support scaling)
     337 - ensure that the input format differs from the output format otherwise GStreamer will bypass the conversion completely; note that GStreamer doesn't understand flipping or rotation as part of the format. Gstreamer master (in development) adds a 'disable-passthrough' property to the v4l2videoconvert entity that can be set to force the conversion regardless of input and output format
     338 - when using imx entities (ie capture, encode/decode, mem2mem, display) you can specify 'output-io-mode=dmabuf-import' to share dmabuf pointers for a zero-copy pipeline however if using non imx entities (ie videotestsrc) you must omit these as you can not ensure the buffers share the alignment/stride necessary to share dmabuf pointers
     339
     340Examples:
     341 * Ensure mem2mem is in your kernel:
     342{{{#!bash
     343~# dmesg | grep mem2mem
     344[   18.356023] imx-media: Registered ipu_ic_pp mem2mem as /dev/video8
     345}}}
     346 * Ensure GStreamer element exists:
     347{{{#!bash
     348~# gst-inspect-1.0 | grep -e "v4l2.*convert"
     349video4linux2:  v4l2video8convert: V4L2 Video Converter
     350}}}
     351  - Note that for GStreamer-1.14, the name of the element depends on the video device the driver registers with the kernel (video8 in the above example). This changes in GStreamer-1.16 to always be 'v4l2videoconvert'
     352 * scale/rotate/flip using {{{videotestsrc}}} (can not use dmabufs for this as it is a non-imx entity)
     353{{{#!bash
     354# upscale
     355gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
     356    v4l2convert ! \
     357    video/x-raw,width=640,height=480 ! kmssink can-scale=false
     358# downscale
     359gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480 ! \
     360    v4l2convert ! \
     361    video/x-raw,width=320,height=240 ! kmssink can-scale=false
     362# rotate
     363gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
     364    v4l2convert extra-controls=cid,rotate=90 ! \
     365    video/x-raw,width=240,height=320 ! kmssink can-scale=false
     366# hflip
     367gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
     368    v4l2convert extra-controls=cid,horizontal_flip=1 ! \
     369    video/x-raw,width=640,height=480 ! kmssink can-scale=false
     370# vflip
     371gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
     372    v4l2convert extra-controls=cid,vertical_flip=1 ! \
     373    video/x-raw,width=640,height=480 ! kmssink can-scale=false
     374}}}
     375  - note the above examples force the input format (resolution in this case) to differ from the output format otherwise gstreamer will bypass the v4l2convert entity thinking it not necessary as gstreamer does not understand the flip/rotation properties. GStreamer master (in development) adds the 'disable-passthrough' property which can be enabled to force disabling passthrough
     376
     377
     378[=#coda]
     379== IMX6 coda encode
     380The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer {{{video4linux2}}} plugin provides encode/decode elements that tap into this.
     381
     382Examples:
     383 * Ensure CODA is in your kernel:
     384{{{#!bash
     385~# dmesg | grep coda   
     386[   16.721698] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2
     387[   16.721724] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin
     388[   18.381136] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin
     389[   18.433648] coda 2040000.vpu: Firmware code revision: 570363
     390[   18.433683] coda 2040000.vpu: Initialized CODA960.
     391[   18.433706] coda 2040000.vpu: Firmware version: 3.1.1
     392[   18.442312] coda 2040000.vpu: codec registered as /dev/video[9-10]
     393}}}
     394 * Ensure GStreamer encode elements exists:
     395{{{#!bash
     396~# gst-inspect-1.0 | grep -e "v4l2.*enc"
     397video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
     398video4linux2:  v4l2mpeg4enc: V4L2 MPEG4 Encoder
     399}}}
     400 * capture, scale, rotate, flip and encode using imx-media capture device mem2mem device and coda device (can use dmabufs for zero-copy)
     401{{{#!bash
     402# encode
     403gst-launch-1.0 v4l2src device=$DEVICE ! \
     404     v4l2convert output-io-mode=dmabuf-import ! \
     405     v4l2h264enc output-io-mode=dmabuf-import ! \
     406     rtph264pay ! udpsink host=$SERVER port=$PORT
     407# scale/encode
     408gst-launch-1.0 v4l2src device=$DEVICE ! \
     409     v4l2convert output-io-mode=dmabuf-import ! \
     410     video/x-raw,width=1440,height=960 ! \
     411     v4l2h264enc output-io-mode=dmabuf-import ! \
     412     rtph264pay ! udpsink host=$SERVER port=$PORT
     413# scale/flip/encode
     414gst-launch-1.0 v4l2src device=$DEVICE ! \
     415     v4l2convert output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \
     416     video/x-raw,width=1440,height=960 ! \
     417     v4l2h264enc output-io-mode=dmabuf-import ! \
     418     rtph264pay ! udpsink host=$SERVER port=$PORT
     419# scale/rotate/encode
     420gst-launch-1.0 v4l2src device=$DEVICE ! \
     421     v4l2convert output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \
     422     video/x-raw,width=1440,height=960 ! \
     423     v4l2h264enc output-io-mode=dmabuf-import ! \
     424     rtph264pay ! udpsink host=$SERVER port=$PORT
     425}}}