wiki:Yocto/gstreamer/compositing

Compositing

Compositing is the term used when combining two or more images/video streams into a single frame. This is useful when you want to apply the same effect over several videos from a single stream. There are a few ways of accomplishing this with GStreamer: Either through a 'compositor' element, or by using individualized sinks to transform video streams for you. However, using a compositor element is preferred as it truly combines video streams into a single frame which ideal for streaming over the network.

In the past, a software based compositor was available, but was extremely burdensome on the CPU, which led to very choppy video. However, in the latest gstreamer-imx, a hardware accelerated compositor was added for imx6 based products. For some documentation, please see this page. Please see below for examples on the two ways you can composite video together.

The following gstreamer-imx elements have compositing capability:

  • imxipuvideosink
  • imxg2dvideosink
  • imxpxpvideosink (IMX6DL only, not recommended)
  • imxg2dcompositor (a stand-alone non-sink element useful if streaming the output)

Note that if you are interested in de-interlacing captured video you must do this prior to compositing as the compositing merges fields into a progressive frame. See here for details on how to de-interlace using the imx6.

Pseudo Compositing to Local Video Out

When there is a system with multiple video inputs that are then directly displayed on an attached video output, it is best to simply create the composition through coordinates with the hardware sinks.

Instead of using a compositor software element, you can pseudo composite by running multiple video input pipelines together.

Please see the following for an example of taking 2 camera inputs and displaying on one HDMI monitor side by side: This example uses two separate pipelines from source to sink, relying on the imxg2dvideosink element to draw video frames where instructed.

gst-launch-1.0 \
    imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink window-width=960 window-height=544 window-x-coord=0 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video5 ! imxg2dvideosink window-width=960 window-height=544 window-x-coord=960 window-y-coord=0

Displaying 8 camera inputs onto an HDMI monitor using AVC8000nano

This assumes a 1920x1080 HDMI monitor is connected to the HDMI output on the SBC.

gst-launch-1.0 \
    imxv4l2videosrc device=/dev/video2 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=0 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=490 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video4 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=980 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video5 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=1470 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video6 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=0 window-y-coord=540 \
    imxv4l2videosrc device=/dev/video7 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=490 window-y-coord=540 \
    imxv4l2videosrc device=/dev/video8 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=980 window-y-coord=540 \
    imxv4l2videosrc device=/dev/video9 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=1470 window-y-coord=540

Compositor Element for Streaming

Using a compositor element has the benefit of allowing affecting the frame wholly (after frames are combined) instead of working on individual pipelines. For example, video streaming where you want to compress the stream to mpeg4 would benefit. Instead of creating four separate video streams, each doing an encoding, you now stream a single frame with only a single encoding.

gstreamer-imx imxg2dcompositor Hardware Accelerated Single Frame Compositing for Streaming

This new gstreamer-imx compositor uses the same notation the software-based compositor used on this page. However, this is now hardware accelerated by either the GPU or IPU.

A simple example using the videotestsrc plugin is shown below:

gst-launch-1.0 \
  imxg2dcompositor name=comp \
    sink_0::alpha=1 sink_0::xpos=0 sink_0::ypos=0 \
    sink_1::alpha=0.5 sink_1::xpos=320 sink_1::ypos=240 ! \
   queue2 ! video/x-raw, width=800, height=600 ! imxipuvideotransform ! imxg2dvideosink \
  videotestsrc ! "video/x-raw" ! comp.sink_0 \
  videotestsrc ! "video/x-raw" ! comp.sink_1

You can combine live video streaming via the following:

gst-launch-1.0 \
 imxg2dcompositor name=c background-color=0x223344 \
   sink_0::xpos=0 sink_0::ypos=0 sink_0::width=400 sink_0::height=300 sink_0::fill_color=0x00000000 \
   sink_1::xpos=400 sink_1::ypos=0 sink_1::width=400 sink_1::height=300 sink_1::fill_color=0x11111111 \
   sink_2::xpos=0 sink_2::ypos=300 sink_2::width=400 sink_2::height=300 sink_2::fill_color=0x22222222 \
   sink_3::xpos=400 sink_3::ypos=300 sink_3::width=400 sink_3::height=300 sink_3::fill_color=0x33333333 \
   sink_4::xpos=200 sink_4::ypos=150 sink_4::width=400 sink_4::height=300 sink_4::fill_color=0x44444444 sink_4::alpha=0.5 ! \
  queue2 ! video/x-raw, width=800, height=600 ! imxg2dvideosink \
 imxv4l2videosrc device=/dev/video3 ! queue2 ! c.sink_0 \
 imxv4l2videosrc device=/dev/video5 ! queue2 ! c.sink_1 \
 videotestsrc pattern=0  ! queue2 ! c.sink_2 \
 videotestsrc pattern=18 ! queue2 ! c.sink_3 \
 videotestsrc pattern=1  ! queue2 ! c.sink_4

An alternate way of writing the above:

gst-launch-1.0 \
  imxv4l2videosrc device=/dev/video3 ! queue2 ! c.sink_0 \
  imxv4l2videosrc device=/dev/video5 ! queue2 ! c.sink_1 \
  videotestsrc pattern=0  ! queue2 ! c.sink_2 \
  videotestsrc pattern=18 ! queue2 ! c.sink_3 \
  videotestsrc pattern=1  ! queue2 ! c.sink_4 \
  imxg2dcompositor name=c background-color=0x223344 \
   sink_0::xpos=0   sink_0::ypos=0   sink_0::width=400 sink_0::height=300 sink_0::fill_color=0x00000000 \
   sink_1::xpos=400 sink_1::ypos=0   sink_1::width=400 sink_1::height=300 sink_1::fill_color=0x11111111 \
   sink_2::xpos=0   sink_2::ypos=300 sink_2::width=400 sink_2::height=300 sink_2::fill_color=0x22222222 \
   sink_3::xpos=400 sink_3::ypos=300 sink_3::width=400 sink_3::height=300 sink_3::fill_color=0x33333333 \
   sink_4::xpos=200 sink_4::ypos=150 sink_4::width=400 sink_4::height=300 sink_4::fill_color=0x44444444 sink_4::alpha=0.5 ! \
  queue2 ! video/x-raw, width=800, height=600 ! imxg2dvideosink

This above examples showcases several ideas: resizing video, filling background color if video doesn't fit the full parameters, combines a total of five (5) pipelines, and places one of them in the middle of the screen with half it's transparency. Please note that you can use the imxipucompositor, but because the IPU is very pick with it's input type (width and height), it is generally not recommended for video composition.

The below image (annotated) is how the above pipeline would look:

picture showcasing compositing

Notes:

  • The colors 0x00000000 and 0x11111111 are used as background fill colors for that frame. Note that the sink_n::fill_color property is a 32bit ABGR value
  • The total size is 800x600, with each of the five frames being 400x300 (outlined in Red)
  • The top-left corner is coordinate 0, 0
  • You can rotate a single inner video frame by adding sink_n::rotation=1, the video stream will be fit for the new aspect ratio and background will be colored with the sink_n::fill_color property
  • Borders can be added to frames via the sink_n::left-margin, sink_n::right-margin, sink_n::top-margin, and sink_n::bottom-margin properties
  • For a complete list of properties to the compositor, please see this section of the gstreamer-imx README.

If you want to stream this via rtsp, you can either use the gst-variable-rtsp-server application, which is provided by default on our Yocto 1.8 BSP or use the udpsink/src elements. The following is an example of using the gst-variable-rtsp-server application.

# Streaming at birate of 10mbit/s
gst-variable-rtsp-server --port=9000 --mount-point=/mystream --user-pipeline="\
 "imxv4l2videosrc device=/dev/video3 ! queue2 ! c.sink_0 \
  imxv4l2videosrc device=/dev/video5 ! queue2 ! c.sink_1 \
  videotestsrc pattern=0  ! queue2 ! c.sink_2 \
  videotestsrc pattern=18 ! queue2 ! c.sink_3 \
  videotestsrc pattern=1  ! queue2 ! c.sink_4 \
  imxg2dcompositor name=c background-color=0x223344 \
   sink_0::xpos=0   sink_0::ypos=0   sink_0::width=400 sink_0::height=300 sink_0::fill_color=0x00000000 \
   sink_1::xpos=400 sink_1::ypos=0   sink_1::width=400 sink_1::height=300 sink_1::fill_color=0x11111111 \
   sink_2::xpos=0   sink_2::ypos=300 sink_2::width=400 sink_2::height=300 sink_2::fill_color=0x22222222 \
   sink_3::xpos=400 sink_3::ypos=300 sink_3::width=400 sink_3::height=300 sink_3::fill_color=0x33333333 \
   sink_4::xpos=200 sink_4::ypos=150 sink_4::width=400 sink_4::height=300 sink_4::fill_color=0x44444444 sink_4::alpha=0.5 ! \
  queue2 ! video/x-raw,width=800,height=600 ! imxipuvideotransform ! imxvpuenc_h264 bitrate=10000 ! rtph264pay name=pay0 pt=96"

An example of the pipeline on the receiving side that then displays to an HDMI monitor:

gst-launch-1.0 rtspsrc location=rtsp://192.168.1.1/mystream latency=10 ! decodebin ! autovideosink

For other methods of network streaming, please view our streaming page.

Compositor Example with 8 Video Cameras

http://trac.gateworks.com/raw-attachment/wiki/minipciexpressmodules/avc8000nano/final_screen_cameras.jpg

Here we will show how to use the compositing features to show 8x video capture streams on a single display. In these examples a GW5304 is used where /dev/video0 is the on-board CVBS analog decoder, and /dev/video1 through /dev/video8 are the 8x D1 video inputs from an avc8000nano miniPCIe card.

We display the composited stream to the IMX6 HDMI at 1080p and we carve up the 1920x1080 into 4x2 490x490 frames for the 8 inputs.

Capture, composite and display on Ventna

When capturing, compositing, and displaying on Ventana you can use the compositing properties of any of the IMX video sink elements.

Here we use imxg2dvideosink:

gst-launch-1.0 -v \
    imxv4l2videosrc device=/dev/video1 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=0    window-y-coord=0 \
    imxv4l2videosrc device=/dev/video2 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=490  window-y-coord=0 \
    imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=980  window-y-coord=0 \
    imxv4l2videosrc device=/dev/video4 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=1470 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video5 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=0    window-y-coord=540 \
    imxv4l2videosrc device=/dev/video6 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=490  window-y-coord=540 \
    imxv4l2videosrc device=/dev/video7 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=980  window-y-coord=540 \
    imxv4l2videosrc device=/dev/video8 ! imxg2dvideosink window-width=490 window-height=490 window-x-coord=1470 window-y-coord=540

Because the avc8000nano captures D1 interlaced analog video, you may wish to de-interlace the inputs. Here we will use the imxipuvideosink as it has the ability to de-interlace:

gst-launch-1.0 -v \
    imxv4l2videosrc device=/dev/video1 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=0    window-y-coord=0 \
    imxv4l2videosrc device=/dev/video2 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=490  window-y-coord=0 \
    imxv4l2videosrc device=/dev/video3 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=980  window-y-coord=0 \
    imxv4l2videosrc device=/dev/video4 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=1470 window-y-coord=0 \
    imxv4l2videosrc device=/dev/video5 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=0    window-y-coord=540 \
    imxv4l2videosrc device=/dev/video6 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=490  window-y-coord=540 \
    imxv4l2videosrc device=/dev/video7 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=980  window-y-coord=540 \
    imxv4l2videosrc device=/dev/video8 ! imxipuvideosink deinterlace=true window-width=490 window-height=490 window-x-coord=1470 window-y-coord=540
  • Note that only the IPU can hardware de-interlace

Capture, composite, compress and stream from Ventana and display remotely via RTSP

When capturing, compositing, compressing and streaming you must use the imxg2dcompositor element to composite the 8x sources into 1 stream before compressing, and packetizing. In this example we will stream via RTSP using gst-variable-rtsp-server.

Ventana server (192.168.1.1):

gst-variable-rtsp-server --port=9000 --mount-point=/mystream --user-pipeline="\
 imxv4l2videosrc device=/dev/video1 queue-size=20 ! queue2 ! c.sink_0 \
 imxv4l2videosrc device=/dev/video2 queue-size=20 ! queue2 ! c.sink_1 \
 imxv4l2videosrc device=/dev/video3 queue-size=20 ! queue2 ! c.sink_2 \
 imxv4l2videosrc device=/dev/video4 queue-size=20 ! queue2 ! c.sink_3 \
 imxv4l2videosrc device=/dev/video5 queue-size=20 ! queue2 ! c.sink_4 \
 imxv4l2videosrc device=/dev/video6 queue-size=20 ! queue2 ! c.sink_5 \
 imxv4l2videosrc device=/dev/video7 queue-size=20 ! queue2 ! c.sink_6 \
 imxv4l2videosrc device=/dev/video8 queue-size=20 ! queue2 ! c.sink_7 \
 imxg2dcompositor name=c background-color=0xffffff \
  sink_0::xpos=0   sink_0::ypos=0   sink_0::width=320 sink_0::height=360 sink_0::fill_color=0x00000000 \
  sink_1::xpos=320 sink_1::ypos=0   sink_1::width=320 sink_1::height=360 sink_1::fill_color=0x00000000 \
  sink_2::xpos=640 sink_2::ypos=0   sink_2::width=320 sink_2::height=360 sink_2::fill_color=0x00000000 \
  sink_3::xpos=960 sink_3::ypos=0   sink_3::width=320 sink_3::height=360 sink_3::fill_color=0x00000000 \
  sink_4::xpos=0   sink_4::ypos=360 sink_4::width=320 sink_4::height=360 sink_4::fill_color=0x00000000 \
  sink_5::xpos=320 sink_5::ypos=360 sink_5::width=320 sink_5::height=360 sink_5::fill_color=0x00000000 \
  sink_6::xpos=640 sink_6::ypos=360 sink_6::width=320 sink_6::height=360 sink_6::fill_color=0x00000000 \
  sink_7::xpos=960 sink_7::ypos=360 sink_7::width=320 sink_7::height=360 sink_7::fill_color=0x00000000 \
 ! queue2 ! video/x-raw,width=1280,height=720 ! imxipuvideotransform \
 ! imxvpuenc_h264 bitrate=20000 ! rtph264pay name=pay0 pt=96"
  • Note we need to increase the buffers for the sources with the queue-size property to the imxv4l2videosrc element (which defaults to 6) otherwise we will start seeing unavailable buffer messages

Networked Client via GStreamer:

gst-launch-1.0 --gst-debug=3 rtspsrc location=rtsp://192.168.1.1:9000/mystream latency=10 ! queue2 ! decodebin ! autovideosink

If you need to de-interlace you will need to do it before the compositing. For example, to de-interlace and composite 4 D1 inputs from an avc8000:

gst-variable-rtsp-server --port=9000 --mount-point=/mystream --user-pipeline="\
 imxv4l2videosrc device=/dev/video1 queue-size=20 ! imxipuvideotransform deinterlace=true ! queue2 ! c.sink_0 \
 imxv4l2videosrc device=/dev/video2 queue-size=20 ! imxipuvideotransform deinterlace=true ! queue2 ! c.sink_1 \
 imxv4l2videosrc device=/dev/video3 queue-size=20 ! imxipuvideotransform deinterlace=true ! queue2 ! c.sink_2 \
 imxv4l2videosrc device=/dev/video4 queue-size=20 ! imxipuvideotransform deinterlace=true ! queue2 ! c.sink_3 \
 imxg2dcompositor name=c background-color=0xffffff \
  sink_0::xpos=0   sink_0::ypos=0   sink_0::width=320 sink_0::height=360 sink_0::fill_color=0x00000000 \
  sink_1::xpos=320 sink_1::ypos=0   sink_1::width=320 sink_1::height=360 sink_1::fill_color=0x00000000 \
  sink_2::xpos=640 sink_2::ypos=0   sink_2::width=320 sink_2::height=360 sink_2::fill_color=0x00000000 \
  sink_3::xpos=960 sink_3::ypos=0   sink_3::width=320 sink_3::height=360 sink_3::fill_color=0x00000000 \
 ! queue2 ! video/x-raw,width=1280,height=720 ! imxipuvideotransform \
 ! imxvpuenc_h264 bitrate=20000 ! rtph264pay name=pay0 pt=96"

Compositor Software Accelerated - Not recommended

This compositor element is a software based compositor and is extremely painful to use. It is very slow at copying buffers around and thus causes quite a bit of latency and causes video stuttering. Please consider using the gstreamer-imx compositor instead. However, as an example, here is a possible pipeline to use it:

(Not recommended, use gstreamer-imx)

gst-launch-1.0 \
  compositor name=comp \
    sink_0::alpha=1 sink_0::xpos=0 sink_0::ypos=0 \
    sink_1::alpha=0.5 sink_1::xpos=320 sink_1::ypos=240 ! \
      videoconvert ! imxg2dvideosink \
  videotestsrc ! "video/x-raw,format=AYUV,width=320,height=240,framerate=(fraction)30/1" ! queue2 ! comp.sink_0 \
  videotestsrc ! "video/x-raw,format=AYUV,width=320,height=240,framerate=(fraction)30/1" ! queue2 ! comp.sink_1
Last modified 13 months ago Last modified on 05/19/16 13:45:05

Attachments (1)

Download all attachments as: .zip