Changes between Initial Version and Version 1 of Yocto/gstreamer/compositing


Ignore:
Timestamp:
10/22/2017 05:28:45 AM (6 years ago)
Author:
trac
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Yocto/gstreamer/compositing

    v1 v1  
     1[[PageOutline]]
     2
     3[=#compositing]
     4= Compositing =
     5Compositing is the term used when combining two or more images/video streams into a single frame. This is useful when you want to apply the same effect over several videos from a single stream. There are a few ways of accomplishing this with GStreamer: Either through a 'compositor' element, or by using individualized sinks to transform video streams for you. However, using a compositor element is preferred as it truly combines video streams into a single frame which ideal for streaming over the network.
     6
     7In the past, a software based compositor was available, but was extremely burdensome on the CPU, which led to very choppy video. However, in the latest {{{gstreamer-imx}}}, a hardware accelerated compositor was added for imx6 based products. For some documentation, please see [https://github.com/Freescale/gstreamer-imx#compositing this page]. Please see below for examples on the two ways you can composite video together.
     8
     9
     10== Pseudo Compositing to Local Video Out ==
     11When there is a system with multiple video inputs that are then directly displayed on an attached video output, it is best to simply create the composition through coordinates with the hardware sinks.
     12
     13Instead of using a compositor software element, you can pseudo composite by running multiple video input pipelines together.
     14
     15Please see the following for an example of taking 2 camera inputs and displaying on one HDMI monitor side by side:
     16This example uses two separate pipelines from source to sink, relying on the {{{imxg2dvideosink}}} element to draw video frames where instructed.
     17{{{#!bash
     18gst-launch-1.0 \
     19    imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink window-width=960 window-height=544 window-x-coord=0 window-y-coord=0 \
     20    imxv4l2videosrc device=/dev/video5 ! imxg2dvideosink window-width=960 window-height=544 window-x-coord=960 window-y-coord=0
     21}}}
     22
     23
     24=== Displaying 8 camera inputs onto an HDMI monitor using AVC8000nano ===
     25This assumes a 1920x1080 HDMI monitor is connected to the HDMI output on the SBC.
     26{{{#!bash
     27gst-launch-1.0 \
     28    imxv4l2videosrc device=/dev/video2 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=0 window-y-coord=0 \
     29    imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=490 window-y-coord=0 \
     30    imxv4l2videosrc device=/dev/video4 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=980 window-y-coord=0 \
     31    imxv4l2videosrc device=/dev/video5 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=1470 window-y-coord=0 \
     32    imxv4l2videosrc device=/dev/video6 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=0 window-y-coord=540 \
     33    imxv4l2videosrc device=/dev/video7 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=490 window-y-coord=540 \
     34    imxv4l2videosrc device=/dev/video8 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=980 window-y-coord=540 \
     35    imxv4l2videosrc device=/dev/video9 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=1470 window-y-coord=540
     36}}}
     37
     38
     39== Compositor Element for Streaming ==
     40Using a compositor element has the benefit of allowing affecting the frame wholly (after frames are combined) instead of working on individual pipelines. For example, video streaming where you want to compress the stream to mpeg4 would benefit. Instead of creating four separate video streams, each doing an encoding, you now stream a single frame with only a single encoding.
     41
     42
     43=== gstreamer-imx imxg2dcompositor Hardware Accelerated Single Frame Compositing for Streaming ===
     44This new gstreamer-imx compositor uses the same notation the software-based compositor used on this page. However, this is now hardware accelerated by either the GPU or IPU.
     45
     46A simple example using the {{{videotestsrc}}} plugin is shown below:
     47{{{#!bash
     48gst-launch-1.0 \
     49  imxg2dcompositor name=comp \
     50    sink_0::alpha=1 sink_0::xpos=0 sink_0::ypos=0 \
     51    sink_1::alpha=0.5 sink_1::xpos=320 sink_1::ypos=240 ! \
     52   queue2 ! video/x-raw, width=800, height=600 ! imxipuvideotransform ! imxg2dvideosink \
     53  videotestsrc ! "video/x-raw" ! comp.sink_0 \
     54  videotestsrc ! "video/x-raw" ! comp.sink_1
     55}}}
     56
     57You can combine live video streaming via the following:
     58{{{#!bash
     59gst-launch-1.0 \
     60 imxg2dcompositor name=c background-color=0x223344 \
     61   sink_0::xpos=0 sink_0::ypos=0 sink_0::width=400 sink_0::height=300 sink_0::fill_color=0x00000000 \
     62   sink_1::xpos=400 sink_1::ypos=0 sink_1::width=400 sink_1::height=300 sink_1::fill_color=0x11111111 \
     63   sink_2::xpos=0 sink_2::ypos=300 sink_2::width=400 sink_2::height=300 sink_2::fill_color=0x22222222 \
     64   sink_3::xpos=400 sink_3::ypos=300 sink_3::width=400 sink_3::height=300 sink_3::fill_color=0x33333333 \
     65   sink_4::xpos=200 sink_4::ypos=150 sink_4::width=400 sink_4::height=300 sink_4::fill_color=0x44444444 sink_4::alpha=0.5 ! \
     66  queue2 ! video/x-raw, width=800, height=600 ! imxg2dvideosink \
     67 imxv4l2videosrc device=/dev/video3 ! queue2 ! c.sink_0 \
     68 imxv4l2videosrc device=/dev/video5 ! queue2 ! c.sink_1 \
     69 videotestsrc pattern=0  ! queue2 ! c.sink_2 \
     70 videotestsrc pattern=18 ! queue2 ! c.sink_3 \
     71 videotestsrc pattern=1  ! queue2 ! c.sink_4
     72}}}
     73
     74An alternate way of writing the above:
     75{{{#!bash
     76gst-launch-1.0 \
     77  imxv4l2videosrc device=/dev/video3 ! queue2 ! c.sink_0 \
     78  imxv4l2videosrc device=/dev/video5 ! queue2 ! c.sink_1 \
     79  videotestsrc pattern=0  ! queue2 ! c.sink_2 \
     80  videotestsrc pattern=18 ! queue2 ! c.sink_3 \
     81  videotestsrc pattern=1  ! queue2 ! c.sink_4 \
     82  imxg2dcompositor name=c background-color=0x223344 \
     83   sink_0::xpos=0 sink_0::ypos=0 sink_0::width=400 sink_0::height=300 sink_0::fill_color=0x00000000 \
     84   sink_1::xpos=400 sink_1::ypos=0 sink_1::width=400 sink_1::height=300 sink_1::fill_color=0x11111111 \
     85   sink_2::xpos=0 sink_2::ypos=300 sink_2::width=400 sink_2::height=300 sink_2::fill_color=0x22222222 \
     86   sink_3::xpos=400 sink_3::ypos=300 sink_3::width=400 sink_3::height=300 sink_3::fill_color=0x33333333 \
     87   sink_4::xpos=200 sink_4::ypos=150 sink_4::width=400 sink_4::height=300 sink_4::fill_color=0x44444444 sink_4::alpha=0.5 ! \
     88  queue2 ! video/x-raw, width=800, height=600 ! imxg2dvideosink
     89}}}
     90
     91This above examples showcases several ideas: resizing video, filling background color if video doesn't fit the full parameters, combines a total of five (5) pipelines, and places one of them in the middle of the screen with half it's transparency. Please note that you can use the {{{imxipucompositor}}}, but because the IPU is very pick with it's input type (width and height), it is generally not recommended for video composition.
     92
     93The below image (annotated) is how the above pipeline would look:
     94
     95[[Image(composite_example.png)]]
     96
     97Notes:
     98 * The colors {{{0x00000000}}} and {{{0x11111111}}} are used as background fill colors for that frame. Note that the {{{sink_n::fill_color}}} property is a 32bit ABGR value
     99 * The total size is 800x600, with each of the five frames being 400x300 (outlined in [[span(style=color: #FF0000, Red)]])
     100 * The top-left corner is coordinate {{{0, 0}}}
     101 * You can rotate a single inner video frame by adding {{{sink_n::rotation=1}}}, the video stream will be fit for the new aspect ratio and background will be colored with the {{{sink_n::fill_color}}} property
     102 * Borders can be added to frames via the {{{sink_n::left-margin}}}, {{{sink_n::right-margin}}}, {{{sink_n::top-margin}}}, and {{{sink_n::bottom-margin}}} properties
     103 * For a complete list of properties to the compositor, please see [https://github.com/Freescale/gstreamer-imx#compositing this section] of the {{{gstreamer-imx}}} README.
     104
     105If you want to stream this via rtsp, you can either use the {{{gst-variable-rtsp-server}}} application, which is provided by default on our Yocto 1.8 BSP or use the {{{udpsink/src}}} elements. The following is an example of using the {{{gst-variable-rtsp-server}}} application.
     106{{{#!bash
     107# Streaming at birate of 10mbit/s
     108gst-variable-rtsp-server -u \
     109 "imxv4l2videosrc device=/dev/video3 ! queue2 ! c.sink_0 \
     110  imxv4l2videosrc device=/dev/video5 ! queue2 ! c.sink_1 \
     111  videotestsrc pattern=0  ! queue2 ! c.sink_2 \
     112  videotestsrc pattern=18 ! queue2 ! c.sink_3 \
     113  videotestsrc pattern=1  ! queue2 ! c.sink_4 \
     114  imxg2dcompositor name=c background-color=0x223344 \
     115   sink_0::xpos=0 sink_0::ypos=0 sink_0::width=400 sink_0::height=300 sink_0::fill_color=0x00000000 \
     116   sink_1::xpos=400 sink_1::ypos=0 sink_1::width=400 sink_1::height=300 sink_1::fill_color=0x11111111 \
     117   sink_2::xpos=0 sink_2::ypos=300 sink_2::width=400 sink_2::height=300 sink_2::fill_color=0x22222222 \
     118   sink_3::xpos=400 sink_3::ypos=300 sink_3::width=400 sink_3::height=300 sink_3::fill_color=0x33333333 \
     119   sink_4::xpos=200 sink_4::ypos=150 sink_4::width=400 sink_4::height=300 sink_4::fill_color=0x44444444 sink_4::alpha=0.5 ! \
     120  queue2 ! video/x-raw, width=800, height=600 ! imxipuvideotransform ! imxvpuenc_h264 bitrate=10000 ! rtph264pay name=pay0 pt=96"
     121}}}
     122
     123An example of the pipeline on the receiving side that then displays to an HDMI monitor:
     124{{{#!bash
     125gst-launch-1.0 rtspsrc location=rtsp://172.24.10.210:9099/stream latency=10 ! decodebin ! autovideosink
     126}}}
     127For other methods of network streaming, please view our [wiki:Yocto/gstreamer/streaming streaming] page.
     128
     129
     130=== Compositor Example with 8 Video Cameras ===
     131[[Image(http://trac.gateworks.com/raw-attachment/wiki/minipciexpressmodules/avc8000nano/final_screen_cameras.jpg)]]
     132
     133 1. First, start the server side code on the SBC with the cameras connected
     134{{{#!bash
     135gst-variable-rtsp-server -u \
     136 "imxv4l2videosrc device=/dev/video2 queue-size=55 ! queue2 ! c.sink_0 \
     137 imxv4l2videosrc device=/dev/video3 queue-size=55 ! queue2 ! c.sink_1 \
     138 imxv4l2videosrc device=/dev/video4 queue-size=55 ! queue2 ! c.sink_2 \
     139 imxv4l2videosrc device=/dev/video5 queue-size=55 ! queue2 ! c.sink_3 \
     140 imxv4l2videosrc device=/dev/video6 queue-size=55 ! queue2 ! c.sink_4 \
     141 imxv4l2videosrc device=/dev/video7 queue-size=55 ! queue2 ! c.sink_5 \
     142 imxv4l2videosrc device=/dev/video8 queue-size=55 ! queue2 ! c.sink_6 \
     143 imxv4l2videosrc device=/dev/video9 queue-size=55 ! queue2 ! c.sink_7 \
     144 imxg2dcompositor name=c background-color=0xffffff \
     145  sink_0::xpos=0 sink_0::ypos=0 sink_0::width=320 sink_0::height=360 sink_0::fill_color=0x00000000 \
     146  sink_1::xpos=320 sink_1::ypos=0 sink_1::width=320 sink_1::height=360 sink_1::fill_color=0x00000000 \
     147  sink_2::xpos=640 sink_2::ypos=0 sink_2::width=320 sink_2::height=360 sink_2::fill_color=0x00000000 \
     148  sink_3::xpos=960 sink_3::ypos=0 sink_3::width=320 sink_3::height=360 sink_3::fill_color=0x00000000 \
     149  sink_4::xpos=0 sink_4::ypos=360 sink_4::width=320 sink_4::height=360 sink_4::fill_color=0x00000000 \
     150  sink_5::xpos=320 sink_5::ypos=360 sink_5::width=320 sink_5::height=360 sink_5::fill_color=0x00000000 \
     151  sink_6::xpos=640 sink_6::ypos=360 sink_6::width=320 sink_6::height=360 sink_6::fill_color=0x00000000 \
     152  sink_7::xpos=960 sink_7::ypos=360 sink_7::width=320 sink_7::height=360 sink_7::fill_color=0x00000000 \
     153 ! queue2 ! video/x-raw, width=1280, height=720 ! imxipuvideotransform \
     154 ! imxvpuenc_h264 bitrate=20000 ! rtph264pay name=pay0 pt=96"
     155}}}
     156
     157 2. Then, start the receiving code on the other SBC, pointing to the IP of the server
     158{{{#!bash
     159gst-launch-1.0 --gst-debug=3 rtspsrc location=rtsp://172.24.10.210:9099/stream latency=100 ! queue2 ! decodebin ! autovideosink
     160}}}
     161
     162
     163=== Compositor Software Accelerated - Not recommended ===
     164This compositor element is a software based compositor and is extremely painful to use. It is very slow at copying buffers around and thus causes quite a bit of latency and causes video stuttering. Please consider using [wiki:Yocto/gstreamer/compositing#gstreamer-imx the gstreamer-imx] compositor instead. However, as an example, here is a possible pipeline to use it:
     165
     166'''(Not recommended, use gstreamer-imx)'''
     167{{{#!bash
     168gst-launch-1.0 \
     169  compositor name=comp \
     170    sink_0::alpha=1 sink_0::xpos=0 sink_0::ypos=0 \
     171    sink_1::alpha=0.5 sink_1::xpos=320 sink_1::ypos=240 ! \
     172      videoconvert ! imxg2dvideosink \
     173  videotestsrc ! "video/x-raw,format=AYUV,width=320,height=240,framerate=(fraction)30/1" ! queue2 ! comp.sink_0 \
     174  videotestsrc ! "video/x-raw,format=AYUV,width=320,height=240,framerate=(fraction)30/1" ! queue2 ! comp.sink_1
     175}}}