GStreamer on Gateworks SBCs

The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs), specifically the Ventana family that utilizes the Freescale i.MX6 processors.

Gateworks SBCs can be viewed at the following link: ​http://www.gateworks.com

Important Note

The IMX6 (Ventana family) software has been evolving over the years. This page is focused on mainline Linux kernels and not the Freescale GStreamer gstreamer-imx plugin legacy software that used the 3.14 Freescale Vendor kernel. If using the Gateworks 3.14 kernel with the Freescale capture drivers please see Yocto/gstreamer

Also note that GStreamer capture devices are /dev/videoN which require some configuration if using Processor specific video input busses such as the IMX6 capture using Analog (CVBS) or Digital video (HDMI). If using these capture interfaces please see linux/media for details on how to use media-ctl to configure those devices.

Video4Linux2 Devices

The Video4Linux2 API is the device driver API used for kernel capture drivers as well as video encode and video decode drivers.

Media Control API

The MediaControl API is used to configure Video4Linux2 capture devices. See linux/media for more info.

V4L2 Controls (brightness, contrast, saturation, rotation, flip, bitrate etc)

When capturing video from a video4linux2 device you can use the v4l2 API to get and set various controls the device may provide access to such as:

• brightness
• contrast
• hue
• saturation
• horizontal flip
• veritcal flip
• rotation

What controls are available depend on the device. Furthermore when using capture devices that have media-ctl pipelines controls provided by various elements within the pipeline are passed on to the capture device.

Examples:

• list available controls of a capture device:

  media-ctl-setup adv7180 > setup
  source setup
  v4l2-ctl --device $DEVICE --list-ctrls # list available controls
  

• get brightness control for adv7180 analog video capture device:

  media-ctl-setup adv7180 > setup
  source setup
  v4l2-ctl --device $DEVICE --get-ctrl=brightness
  

• set brightness control for adv7180 analog video capture device:

  media-ctl-setup adv7180 > setup
  source setup
  v4l2-ctl --device $DEVICE --set-ctrl=brightness=50
  

• set video bitrate of the v4l2h264enc H264 encoder element:

  media-ctl-setup adv7180 > setup
  source setup
  v4l2-ctl --device $ENCODER --set-ctrl=video_bitrate=10000000
  

Note that we 'source' the setup file above instead of executing it. This is because it sets some env variables that we will use throughout our examples below. If you are using a shell that does not have 'source' the notation is '. ./setup'

Test Media Files

There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing:

• ​Big Buck Bunny - Creative Commons
• ​Sintel - Creative Commons
• ​Tears of Steel - Creative Commons

GStreamer

​GStreamer is an opensource library and framework created to handle multimedia. Pipelines are used to source and sink media however you would like (e.g. decoding the mp3 out of a video file and playing it back through speakers).

It is important to understand that GStreamer is foremost a library meant for developing applications. While it has an extremely useful test application called gst-launch meant for testing pipelines using that application alone can't solve all problems. There are plenty of resources online including mailing lists and IRC channels that can be helpful in understanding GStreamer and writing applications that use it.

To install GStreamer on Ubuntu:

apt-get install gstreamer1.0-x gstreamer1.0-tools libgstreamer1.0-0 \
  gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio \
  v4l-utils alsa-utils

Plugins and Elements

GStreamer works on 'plugins' that provide pipeline 'elements'. A plugin comprises of elements that can do work on a media stream. For example, the mpeg2dec can decode mpeg1 and mpeg2 video streams (software based decoding, not hardware accelerated).

Pipeline Construction

​gst-launch is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type gst-inspect element where element is the element you are looking for. To see a full listing of elements installed, you can type just gst-inspect.

Here is a graphical example of a pipeline.

The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next

​gstreamer.com contains much more information on the construction of pipelines.

Caps Filters

GStreamer has a concept called ​caps filters. A 'cap' (short for capability) is used to describe the type of data that links a src (output) pad from one element to a sink (input) pad of another element.

Adding a -v flag to gst-launch-1.0 will output the capabilities negotiated between elements:

~# gst-launch-1.0 -v videotestsrc ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

In the output, the caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" are the video caps negotiated between the two.

You can force a caps filter between two elements by treating it as a pipeline element:

~# gst-launch-1.0 -v videotestsrc ! 'video/x-raw, format=UYVY, width=1920, height=1080, framerate=10/1' ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

As you can see, the caps filter above changed both the video format, resolution, and framerate of the video stream coming out of the videotestsrc element.

Caps filters are useful when you want to use a specific format where a device may support multiple. For example when capturing audio from an ALSA sound device that supports a variety of samplerates, channels, and bit formats you can use a capsfilter to override any default configuration.

Applications

GStreamer is a library and framework for multimedia processing. It is not a user-space application. It is easy to forget this because of the usefulness of the gst-launch application that can be used to connect simple pipelines together for testing. For your needs you very likely need to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that.

GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see ​here for more details.

Furthermore, Gateworks has created a gst-variable-rtsp-server example application that demonstrates how to server simple GStreamer pipelines via an RTSP server and automatically adjust compression quality based on client connections.

fakesink

The fakesink element is a very useful video sink when debugging or learning GStreamer. It takes whatever frames is given to it and drops them. This might help debugging pipelines if problems ever arise.

Example:

gst-launch-1.0 videotestsrc ! fakesink

gst-inspect

The gst-inspect-1.0 tool is an excellent tool if you're unfamiliar with the functionality, source/sink formats, or properties of any Gstreamer element.

Usage:

root@bionic-armhf:~# gst-inspect-1.0 videotestsrc
Factory Details:
  Rank                     none (0)
  Long-name                Video test source
  Klass                    Source/Video
  Description              Creates a test video stream
  Author                   David A. Schleef <ds@schleef.org>

Plugin Details:
  Name                     videotestsrc
  Description              Creates a test video stream
  Filename                 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstvideotestsrc.so
  Version                  1.14.5
  License                  LGPL
  Source module            gst-plugins-base
  Source release date      2019-05-29
  Binary package           GStreamer Base Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-base1.0

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstVideoTestSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)I420, (string)YV12, (string)YUY2, (string)UYVY, (string)AYUV, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGB, (string)BGR, (string)Y41B, (string)Y42B, (string)YVYU, (string)Y444, (string)v210, (string)v216, (string)NV12, (string)NV21, (string)GRAY8, (string)GRAY16_BE, (string)GRAY16_LE, (string)v308, (string)RGB16, (string)BGR16, (string)RGB15, (string)BGR15, (string)UYVP, (string)A420, (string)RGB8P, (string)YUV9, (string)YVU9, (string)IYU1, (string)ARGB64, (string)AYUV64, (string)r210, (string)I420_10BE, (string)I420_10LE, (string)I422_10BE, (string)I422_10LE, (string)Y444_10BE, (string)Y444_10LE, (string)GBR, (string)GBR_10BE, (string)GBR_10LE, (string)NV16, (string)NV24, (string)NV12_64Z32, (string)A420_10BE, (string)A420_10LE, (string)A422_10BE, (string)A422_10LE, (string)A444_10BE, (string)A444_10LE, (string)NV61, (string)P010_10BE, (string)P010_10LE, (string)IYU2, (string)VYUY, (string)GBRA, (string)GBRA_10BE, (string)GBRA_10LE, (string)GBR_12BE, (string)GBR_12LE, (string)GBRA_12BE, (string)GBRA_12LE, (string)I420_12BE, (string)I420_12LE, (string)I422_12BE, (string)I422_12LE, (string)Y444_12BE, (string)Y444_12LE, (string)GRAY10_LE32, (string)NV12_10LE32, (string)NV16_10LE32 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
         multiview-mode: { (string)mono, (string)left, (string)right }
      video/x-bayer
                 format: { (string)bggr, (string)rggb, (string)grbg, (string)gbrg }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
         multiview-mode: { (string)mono, (string)left, (string)right }

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "videotestsrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  typefind            : Run typefind before negotiating (deprecated, non-functional)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: false
  pattern             : Type of test pattern to generate
                        flags: readable, writable
                        Enum "GstVideoTestSrcPattern" Default: 0, "smpte"
                           (0): smpte            - SMPTE 100% color bars
                           (1): snow             - Random (television snow)
                           (2): black            - 100% Black
                           (3): white            - 100% White
                           (4): red              - Red
                           (5): green            - Green
                           (6): blue             - Blue
                           (7): checkers-1       - Checkers 1px
                           (8): checkers-2       - Checkers 2px
                           (9): checkers-4       - Checkers 4px
                           (10): checkers-8       - Checkers 8px
                           (11): circular         - Circular
                           (12): blink            - Blink
                           (13): smpte75          - SMPTE 75% color bars
                           (14): zone-plate       - Zone plate
                           (15): gamut            - Gamut checkers
                           (16): chroma-zone-plate - Chroma zone plate
                           (17): solid-color      - Solid color
                           (18): ball             - Moving ball
                           (19): smpte100         - SMPTE 100% color bars
                           (20): bar              - Bar
                           (21): pinwheel         - Pinwheel
                           (22): spokes           - Spokes
                           (23): gradient         - Gradient
                           (24): colors           - Colors
  timestamp-offset    : An offset added to timestamps set on buffers (in ns)
                        flags: readable, writable
                        Integer64. Range: 0 - 2147483646999999999 Default: 0 
  is-live             : Whether to act as a live source
                        flags: readable, writable
                        Boolean. Default: false
  k0                  : Zoneplate zero order phase, for generating plain fields or phase offsets
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kx                  : Zoneplate 1st order x phase, for generating constant horizontal frequencies
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  ky                  : Zoneplate 1st order y phase, for generating contant vertical frequencies
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kt                  : Zoneplate 1st order t phase, for generating phase rotation as a function of time
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kxt                 : Zoneplate x*t product phase, normalised to kxy/256 cycles per vertical pixel at width/2 from origin
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kyt                 : Zoneplate y*t product phase
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kxy                 : Zoneplate x*y product phase
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kx2                 : Zoneplate 2nd order x phase, normalised to kx2/256 cycles per horizontal pixel at width/2 from origin
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  ky2                 : Zoneplate 2nd order y phase, normailsed to ky2/256 cycles per vertical pixel at height/2 from origin
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kt2                 : Zoneplate 2nd order t phase, t*t/256 cycles per picture
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  xoffset             : Zoneplate 2nd order products x offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  yoffset             : Zoneplate 2nd order products y offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  foreground-color    : Foreground color to use (big-endian ARGB)
                        flags: readable, writable, controllable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  background-color    : Background color to use (big-endian ARGB)
                        flags: readable, writable, controllable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4278190080 
  horizontal-speed    : Scroll image number of pixels per frame (positive is scroll to the left)
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  animation-mode      : For pattern=ball, which counter defines the position of the ball.
                        flags: readable, writable
                        Enum "GstVideoTestSrcAnimationMode" Default: 0, "frames"
                           (0): frames           - frame count
                           (1): wall-time        - wall clock time
                           (2): running-time     - running time
  motion              : For pattern=ball, what motion the ball does
                        flags: readable, writable
                        Enum "GstVideoTestSrcMotionType" Default: 0, "wavy"
                           (0): wavy             - Ball waves back and forth, up and down
                           (1): sweep            - 1 revolution per second
                           (2): hsweep           - 1/2 revolution per second, then reset to top
  flip                : For pattern=ball, invert colors every second.
                        flags: readable, writable
                        Boolean. Default: false

GStreamer Video

This section will introduce several concepts regarding how video media is handled by GStreamer and provide several example pipelines.

Video Output

Generally, a GStreamer 'sink' element is one that will take a video stream and output it to a display.

Some commonly used video output sinks:

• autovideosink: Wrapper video sink for automatically detecting video output device
• kmssink: KMS video device sink
• fbdevsink: Framebuffer video device sink

Execute a gst-inspect-1.0 | grep sink to see a complete list of sinks available and gst-inspect-1.0 <element> to see specific details about the applicable formats and properties of a specific element.

Important Note: Video display drivers for use with video display sinks such as kmssink and fbdevsink for HDMI out and LVDS require proper kernel commandline parameters to configure them. Sometimes the Bootloader bootscript will do this for you - Please see linux/display for details and check the kernel command-line after booting to make sure the desired device is enabled and configured for the proper resolution and mode.

Examples:

• HDMI enabled and configured for 1080p60Hz LVDS disabled:

  ~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done
  video=HDMI-A-1:1920x1080M@60
  video=LVDS-1:d
  

• HDMI disabled and LVDS configured for 1280x800 60Hz

  ~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done
  video=LVDS-1:1280x800@60M
  video=HDMI-A-1:d
  

autovideosink

This GStreamer sink is not really a 'video' sink in the traditional sense. Similar to playbin and decodebin, this element selects what it thinks is the best available video sink and uses it.

This will typically use kmssink unless format choices require one of the other sinks.

You can add a verbose flag gst-launch-1.0 -v to see details about the elements and caps chosen when using any type of 'auto' element or 'bin' element.

Example:

gst-launch-1.0 videotestsrc ! autovideosink

• Note that adding a '-v' flag for verbose mode will show the element configuration to let you know what sink was actually chosen

kmssink

This sink allows you to directly output to a KMS (kernelmode setting API) driver. This is the most efficient video output method.

Example:

gst-launch-1.0 videotestsrc ! kmssink

The Framebuffer device sink allows you to directly output to a Linux Framebuffer device if you have one (ls /sys/class/graphics/fb*).

Example:

gst-launch-1.0 videotestsrc ! fbdevsink

Video Input (Capture)

A video input source is anything coming from an input on the device, e.g. HDMI input/USB Web Cam. This is referred to as Video Capture. In GStreamer terms a video capture device is a 'source' or 'src' element.

Please refer to the linux/media page for details on the video in devices on the Ventana platform.

Some commonly used GStreamer video input sources:

• videotestsrc
• v4l2src

Execute a gst-inspect-1.0 | grep src to see a complete list of source elements available and gst-inspect-1.0 <element> to see specific details about the applicable formats and properties of a specific element.

If the is-live property is set to true this will cause buffers to be discarded on a pipeline paused state and pipelines will not participate in the PREROLL phase of a pipeline.

Important Note: Linux video capture devices provided by CPU's such as the IMX6 Analog (CVBS) and Digital (HDMI) inputs provided on Ventana boards require a hardware pipeline configuration. See linux/media for details on how to configure the video capture device pipeline.

videotestsrc

This is a very useful element for testing pipelines as it creates video content. It can output a huge number of video formats for raw video video/x-raw and bayer video video/x-bayer. Use gst-inspect-1.0 videotestsrc to see its output formats and properties such as video test pattern, orientation, colrs, etc.

Examples:

• display SMPTE pattern to connected and configured display

  gst-launch-1.0 videotestsrc ! kmssink
  

• set the 'pattern' property to display a moving ball:

  gst-launch-1.0 videotestsrc pattern=18 ! kmssink
  

• Use a 'capsfilter' to force output of videotestsrc to a specific resolution, format, and framerate (ie 1080p 10fps YUV):

  gst-launch-1.0 videotestsrc ! 'video/x-raw,format=UYVY,width=1920,height=1080,framerate=10/1' ! kmssink
  

v4l2src

This element uses the video4linux2 API to capture video from input sources (/dev/video<n>).

Note that v4l2src is always live regardless of the is-live property.

Note that you need to configure the media-ctl pipeline that feeds the /dev/video<n> devices - see linux/media for details

You can use the v4l2-ctl application from the v4l-utils package to to interact with the device to get/set various capabilities and controls.

Examples:

• show all video4linux2 devices:

  root@bionic-armhf:~# v4l2-ctl --list-devices
  CODA960 (platform:coda):
          /dev/video8
          /dev/video9
  
  imx-media-mem2mem (platform:imx-media-mem2mem):
          /dev/video10
  
  imx-media-capture (platform:ipu1_csi0):
          /dev/video4
  
  imx-media-capture (platform:ipu1_csi1):
          /dev/video5
  
  imx-media-capture (platform:ipu1_ic_prpenc):
          /dev/video0
  
  imx-media-capture (platform:ipu1_ic_prpvf):
          /dev/video1
  
  imx-media-capture (platform:ipu2_csi0):
          /dev/video6
  
  imx-media-capture (platform:ipu2_csi1):
          /dev/video7
  
  imx-media-capture (platform:ipu2_ic_prpenc):
          /dev/video2
  
  imx-media-capture (platform:ipu2_ic_prpvf):
          /dev/video3
  
  

• display all details about /dev/video1:

  v4l2-ctl -d /dev/video1 --all
  

• Use Gateworks media-ctl-setup to configure the media-ctl pipeline for Analog Video capture using the adv7180 chip. If you don't already have this on your root filesystem you can install it via:

  wget https://raw.githubusercontent.com/Gateworks/media-ctl-setup/master/media-ctl-setup -O /usr/local/bin/media-ctl-setup
  chmod +x /usr/local/bin/media-ctl-setup
  media-ctl-setup adv7180 > setup
  . ./setup
  

  • you can look at the setup script to see what was done - the details vary between SoC (IMX6DL vs IMX6Q), board, and sensor (adv7180 for Analog video capture and tda1997x for digital video capture)
  • note that the setup script exports the video capture device to the DEVICE env variable which you can use as shown below
• After sourcing setup above you can capture and display to the connected and configured output display (video loopback):

  gst-launch-1.0 v4l2src device=$DEVICE ! kmssink
  

Colorspace Converting and/or Video Scaling (via IMX6 IPU mem2mem driver)

Often times, a colorspace conversion or scaling is required in order to link GStreamer elements together. This is often due to the fact that not all elements can every format available.

A Linux V4L2 MEM2MEM imx-media driver exists that allows utilizing the IMX6 IPU Image Converter hardware blocks (IC) to perform hardware colorspace conversion (CSC), scaling, cropping, rotation, and flip operations.

The GStreamer video4linux2 element provides an element that uses this driver to expose these capabilities to GStreamer applications.

Notes:

• For GStreamer-1.14 the name of the element depends on the video device the driver registers with the kernel (ie v4l2video8convert if mem2mem driver registers /dev/video8) but for GStreamer-1.16 and beyond the name of the element is stable and always 'v4l2videoconvert'. For the examples on this page we use the GST_CONVERT env variable that is set from the output of the media-ctl-setup script to avoid confusion.
• the kmssink examples below need a can-scale=false property to tell GStreamer not to scale via the KMS driver (as the IMX6 KMS driver does not support scaling)
• ensure that the input format differs from the output format otherwise GStreamer will bypass the conversion completely; note that GStreamer doesn't understand flipping or rotation as part of the format. Gstreamer master (in development) adds a 'disable-passthrough' property to the v4l2videoconvert entity that can be set to force the conversion regardless of input and output format
• when using imx entities (ie capture, encode/decode, mem2mem, display) you can specify 'output-io-mode=dmabuf-import' to share dmabuf pointers for a zero-copy pipeline however if using non imx entities (ie videotestsrc) you must omit these as you can not ensure the buffers share the alignment/stride necessary to share dmabuf pointers

Examples:

• Ensure mem2mem is in your kernel:

  ~# dmesg | grep mem2mem
  [   18.356023] imx-media: Registered ipu_ic_pp mem2mem as /dev/video8
  

• Ensure GStreamer element exists:

  ~# gst-inspect-1.0 | grep -e "v4l2.*convert"
  video4linux2:  v4l2convert: V4L2 Video Converter
  

• Obtain the name of the element (as it can vary between GStreaqmer-1.14 and GStreamer 1.16)

  GST_CONVERT=$(gst-inspect-1.0 | grep -e "v4l2.*convert*" | sed -e 's/.*:\s*\(v4l2.*convert\):.*/\1/')
  

• scale/rotate/flip using videotestsrc (can not use dmabufs for this as it is a non-imx entity)

  # upscale
  gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
      $GST_CONVERT ! \
      video/x-raw,width=640,height=480 ! kmssink can-scale=false
  # downscale
  gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480 ! \
      $GST_CONVERT ! \
      video/x-raw,width=320,height=240 ! kmssink can-scale=false
  # rotate
  gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
      $GST_CONVERT extra-controls=cid,rotate=90 ! \
      video/x-raw,width=240,height=320 ! kmssink can-scale=false
  # hflip
  gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
      $GST_CONVERT extra-controls=cid,horizontal_flip=1 ! \
      video/x-raw,width=640,height=480 ! kmssink can-scale=false
  # vflip
  gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
      $GST_CONVERT extra-controls=cid,vertical_flip=1 ! \
      video/x-raw,width=640,height=480 ! kmssink can-scale=false
  

  • note the above examples force the input format (resolution in this case) to differ from the output format otherwise gstreamer will bypass the v4l2convert entity thinking it not necessary as gstreamer does not understand the flip/rotation properties. GStreamer master (in development) adds the 'disable-passthrough' property which can be enabled to force disabling passthrough
• scale/rotate/flip using imx-media capture device and KMS display driver (can use dmabufs for this as they are all imx hardware entitites):

  # scale sensor input to 720p display
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import ! \
       video/x-raw,width=1280,height=720 ! \
       kmssink can-scale=false
  # scale sensor input to 1080p display
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import ! \
       video/x-raw,width=1920,height=1080 ! \
       kmssink can-scale=false
  # scale/flip
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \
       video/x-raw,width=1920,height=1080 ! \
       kmssink can-scale=false
  # scale/rotate
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \
       video/x-raw,width=720,height=1280 ! \
       kmssink can-scale=false
  

• capture, scale, rotate, flip and encode using imx-media capture device mem2mem device and coda device (can use dmabufs for zero-copy)

  # encode
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import ! \
       v4l2h264enc output-io-mode=dmabuf-import ! \
       rtph264pay ! udpsink host=$SERVER port=$PORT
  # scale/encode
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import ! \
       video/x-raw,width=1440,height=960 ! \
       v4l2h264enc output-io-mode=dmabuf-import ! \
       rtph264pay ! udpsink host=$SERVER port=$PORT
  # scale/flip/encode
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \
       video/x-raw,width=1440,height=960 ! \
       v4l2h264enc output-io-mode=dmabuf-import ! \
       rtph264pay ! udpsink host=$SERVER port=$PORT
  # scale/rotate/encode
  gst-launch-1.0 v4l2src device=$DEVICE ! \
       $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \
       video/x-raw,width=1440,height=960 ! \
       v4l2h264enc output-io-mode=dmabuf-import ! \
       rtph264pay ! udpsink host=$SERVER port=$PORT
  

For anything not able to be handled by the video4linux2 element and any mem2mem drivers there are software based elements that can convert pixel colorspace and/or scale however note these are extremely CPU intensive:

• autovideoconvert
• videoconvert
• rgb2bayer

Interlaced Video and Deinterlacing

Interlaced video is a technique for doubling the perceived frame rate of a video display without consuming extra bandwidth. The interlaced signal contains two fields of a video frame captured at two different times. The alternative to interlaced video is called progressive video. While reducing bandwidth this can cause a perceived flicker effect as well as a very apparent artifact seen during motion. For example a car moving horizontally across a scene will show every other vertical line differently: the second field interlaced with the first field will be a full frame period ahead in time from the other. The visual affect can be seen in ​this image from Wikipedia

Television signals are typically interlaced, or at least were until recently. For example, analog television standards such as NTSC used in North America as well as the PAL and SECAM formats used abroad use interlaced video and therefore any analog video decoder such as the ADV7180 found on many Gateworks Ventana boards will capture interlaced video and are subject to interlacing artifacts. Interlaced video is still used in High Definition signals as well and the letter at the end of the format tells you if its interlaced (ie 480i, 720i, 1080i) or progressive (ie 480p, 720p, 1080p).

To deinterlace video captured from the adv7180 analog decoder you need to use media-ctl to configure your pipeline to include the IMX6 Video Deinterlacer block. This is done for you if you use the Gateworks media-ctl-setup script to configure your pipeline. See linux/media for more info.

References:

• ​Wikipedia interlaced video - includes several images demonstrating interlacing artifacts
• ​Wikipedia deinterlacing

Video Encoding (via IMX6 coda driver and video4linux2 element)

Encoding video means taking the raw captured pixels and converting them to a standard compressed format.

The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer video4linux2 plugin provides various encode/decode elements that tap into this.

Notes:

• Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded:

  ~# dmesg | grep coda    
  [   10.798192] coda 2040000.vpu: Firmware code revision: 570363
  [   10.804115] coda 2040000.vpu: Initialized CODA960.
  [   10.808929] coda 2040000.vpu: Firmware version: 3.1.1
  [   10.817951] coda 2040000.vpu: encoder registered as video8
  [   10.825635] coda 2040000.vpu: decoder registered as video9
  [   10.832117] coda 2040000.vpu: encoder registered as video10
  [   10.839520] coda 2040000.vpu: decoder registered as video11
  ...
  ~# for i in $(ls -d /sys/class/video4linux/video*); do echo $i:$(cat $i/name); do
  ne | grep encoder
  /sys/class/video4linux/video10:coda-encoder
  /sys/class/video4linux/video8:coda-jpeg-encoder
  

• Ensure GStreamer encode elements exists:

  ~# gst-inspect-1.0 | grep -e "v4l2.*enc"
  video4linux2:  v4l2jpegenc: V4L2 JPEG Encoder
  video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
  video4linux2:  v4l2mpeg4enc: V4L2 MPEG4 Encoder
  

• The CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry
• The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough)
• Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the v4l2-ctl -L and v4l2-ctl -l parameters on the devices exposed by the coda driver
• Often you will need to use the mem2mem v4l2convert driver for format conversions (colorspace, colorimetry) to reconcile the format of the source device and the encoder element - notice the $GST_CONVERT env variable below.
• V4L2 JPEG Encoder support was added to the CODA driver in Linux 5.6 and has been backported to the Gateworks 5.4 kernel
• V4l2jpegenc JPEG Encoder support requires GStreamer-1.16+

Examples:

• Encode to H264 (hardware based encode) and stream via RTP/UDP:

  # stream H264/RTP/UDP
  GST_CONVERT=$(gst-inspect-1.0 | grep -e "v4l2.*convert*" | sed -e 's/.*:\s*\(v4l2.*convert\):.*/\1/')
  gst-launch-1.0 v4l2src device=$DEVICE ! \
          $GST_CONVERT output-io-mode=dmabuf-import ! \
          v4l2h264enc output-io-mode=dmabuf-import ! \
          rtph264pay ! udpsink host=$SERVER port=$PORT
  # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT caps=application/x-rtp,payload=96 ! rtph264depay ! decodebin ! autovideosink'
  

• Encode to JPEG (software based encode) and stream via RTP/UDP:

  # stream JPEG/RTP/UDP
  gst-launch-1.0 v4l2src device=$DEVICE ! jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
  # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay ! jpegdec ! autovideosink'
  

• Encode to JPEG (hardware based encode) and stream via RTP/UDP:

  # stream JPEG/RTP/UDP
  gst-launch-1.0 v4l2src device=$DEVICE ! v4l2jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
  # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay ! jpegdec ! autovideosink'
  

  • requires GStreamer 1.16+ and kernel supporting CODA JPEG encode

See also:

• ventana/vpu

Video Decoding (via IMX6 coda driver and video4linux2 element)

Decoding video means taking compressed formats such as MPEG and H264 and decoding them into frames comprised of raw pixels (typically so they can be displayed on a video display output device).

The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer video4linux2 plugin provides encode/decode elements that tap into this.

Notes:

• Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded:

  ~# dmesg | grep coda
  [   10.798192] coda 2040000.vpu: Firmware code revision: 570363
  [   10.804115] coda 2040000.vpu: Initialized CODA960.
  [   10.808929] coda 2040000.vpu: Firmware version: 3.1.1
  [   10.817951] coda 2040000.vpu: encoder registered as video8
  [   10.825635] coda 2040000.vpu: decoder registered as video9
  [   10.832117] coda 2040000.vpu: encoder registered as video10
  [   10.839520] coda 2040000.vpu: decoder registered as video11
  ~# for i in $(ls -d /sys/class/video4linux/video*); do echo $i:$(cat $i/name); do
  ne | grep decoder
  /sys/class/video4linux/video11:coda-decoder
  /sys/class/video4linux/video9:coda-jpeg-decoder
  

• Ensure GStreamer decode elements exists:

  ~# gst-inspect-1.0 | grep -e "v4l2.*dec"
  video4linux2:  v4l2jpegdec: V4L2 JPEG Decoder
  video4linux2:  v4l2mpeg4dec: V4L2 MPEG4 Decoder
  video4linux2:  v4l2mpeg2dec: V4L2 MPEG2 Decoder
  video4linux2:  v4l2h264dec: V4L2 H264 Decoder
  

• The CODA960 decoder outputs NV12/I420/YV12 YUV pixel formats
• The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough)
• V4L2 JPEG Decoder support has not yet been added to the CODA driver as of Linux 5.6 - patches have been posted and are in the Gateworks 5.4 kernel

Note that the following examples assume you are using raw video encoded files, not container formats used for mimxed multimedia types (audio + video) such as ogg, avi, or mov (Quicktime). For information on de-muxing container formats see #multimedia

Examples:

• Decoding H264 video might look like this:

  gst-launch-1.0 filesrc location=file.h264 ! h264parse ! v4l2h264dec  ! kmssink
  

• Decoding MPEG4 video might look like this:

  gst-launch-1.0 filesrc location=file.mp4v ! mpeg4videoparse !  v4l2mpeg4dec ! kmssink
  

• Streaming JPEG video might look like this:

  # transmit JPEG/RTP/UDP on on board:
  gst-launch-1.0 v4l2src device=$DEVICE ! v4l2jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
  # recieve JPEG/RTP/UDP on another board:
  gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay !  v4l2jpegdec ! video/x-raw,format=I420 ! autovideosink
  

  • Note the fact that we need to use a capsfilter to force v4l2jpegdec to output I420

For more examples including working with 'multimedia' files that contain both audio and video see #multimedia

GStreamer Audio

The GStreamer alsasrc and alsasink provide audio capture and playback to Advanced Linux Sound Architecture (ALSA) devices. ALSA is the modern device-driver API for the Linux kernel.

The Gstreamer alsasrc and alsasink can be passed a device property which can represent either the 'device alias' (reported from aplay -L for playback, or arecord -L for record) (ie sysdefault:CARD=sgtl5000audio) or the 'hw:<card>,<device>' notation (ie hw:0,0 for first device, first subdevice)

If not specified via the device property, the device used for alsasrc and alsasink will depend on /etc/asound.conf and/or ~/.asoundrc.

See here for details on specifying audio input and output devices on the Ventana product family.

Playback and Capture also depends on your ALSA mixer (amixer) settings. See mixer settings for more info.

Audio Output via Linux ALSA driver and GStreamer alsasink

Generally, a 'sink' element is one that will take an audio stream and send it to an audio device. Please refer to this audio page for more details on audio devices on the ventana platform.

This alsasink element can accept any of the following formats via audio/x-raw: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE

Examples:

• Generate and play a 1kHz tone to the analog audio output device on Ventana (sgtl5000audio):

  gst-launch-1.0 audiotestsrc ! \
    alsasink device="sysdefault:CARD=sgtl5000audio"
  

Audio Input via Linux ALSA driver and GStreamer alsasrc

An input source is anything coming from a capture device on the SBC, e.g. HDMI audio in/analog audio in. Please refer to this audio page for more details on audio devices on the ventana platform.

The alsasrc element can output the following source types in audio/x-raw: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE

Examples:

• Capture 32kHz stereo 16bit audio from the Analog Audio input on Ventana (sgtl5000audio) and mux into into an AVI file:

  gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \
    audioconvert ! avimux ! filesink location=./audio.avi
  

audiotestsrc

The audiotestsrc is a very useful element for testing. It can output 16-64bit audio/x-raw: S16LE, S32LE, F32LE, F64LE

This can output audio signals from sine to violet-noise.

Examples:

• Generate and play a 1kHz tone to the analog audio output device on Ventana (sgtl5000audio):

  gst-launch-1.0 audiotestsrc ! \
    alsasink device="sysdefault:CARD=sgtl5000audio"
  

Audio Encoding

Encoding is the term used to capture audio and encode it to a new format type.

Commonly used GStreamer Audio encoders:

• vorbisenc - encode audio/x-raw F32LE to audio/x-vorbis
• mulawenc - encode audio/x-raw S16LE to audio/x-mulaw
• wavenc - encode audio/x-raw S32LE, S24LE, S16LE, U8, F32LE, F64LE, audio/x-alaw, and audio/x-mulaw to audio/x-wav
• alawenc - encode audio/x-raw S16LE to audio/x-alaw
• flacenc - encode audio/x-raw S24LE, S24_32LE, S16LE, S8 to audio/x-flac
• lamemp3enc - encode audio/x-raw S16LE to audio/mpeg

There are many more. You can search for your specified one by running a similar search: gst-inspect-1.0 | grep enc.

Note that each encoder has its own limits on the samples it accepts (S16LE, S24LE etc). You can find the formats accepted via gst-inspect-1.0 <element> and you can use the #audioconvertaudioconvert element to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements.

Examples:

• Capture audio from the Ventana Analog audio input (sgtl5000audio), encode it using the MPEG Layer 3 audio codec and store it to a file:

  gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! audioconvert ! lamemp3enc ! filesink location=file.mp3
  

Audio Decoding

Decoding is the term used to decode an encoded audio stream to a raw audio stream.

Commonly used GStreamer Audio decoders:

• mpg123audiodec - decode audio/mpeg to audio/x-raw
• vorbisdec - decode audio/x-vorbis to audio/x-raw
• ivorbisdec - decode audio/x-vorbis to audio/x-raw
• a52dec - decode audio/x-ac3, audio/ac3, and audio/x-private1-ac3 to audio/x-raw
• mulawdec - decode audio/x-mulaw to audio/x-raw
• alawdec - decode audio/x-alaw to audio/x-raw
• flacdec - decode audio/x-flac to audio/x-raw

Note that each decoder has its own limits on the samples it produces at its src pad (S16LE, S24LE etc). You can find the formats accepted via gst-inspect-1.0 <element> and you can use the audioconvert element to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements.

Examples:

• Decode MPEG Layer 3 audio and playback to the default audio device with the following:

  gst-launch-1.0 filesrc location=file.mp3 ! mpegaudioparse ! mpg123audiodec ! alsasink
  

Audio Formats and Conversion

Audio samples specified by the audio/x-raw vary between bit width and endieness as well as the sampling rate (how many times a second audio is measured) and the number of channels that are sampled.

To specify the details of the audio format, you use a GStreamer capsfilter and to convert it from one format to another you use the GStreamer audioconvert element.

Examples:

• specify the sample-rate (32kHz), channels (2:stereo), and bit depth (16bit) and capture to an AVI file:

  gst-launch-1.0 audiotestsrc ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \
    audioconvert ! avimux ! filesink location=./audio.avi
  

Audio loopback (Useful for testing audio input and output)

A simple aduio loopback test can take audio input from an input device and output it to an output device.

loopback audio from the first audio card:

gst-launch-1.0 alsasrc device="hw:0,0" ! alsasink device="hw:0,0"

To send audio from the Ventana Analog input (sgtl5000audio) to the Ventana HDMI output (DWHDMI):

• first, enable Analog input Line in and set capture volume:

  card=$(cat /proc/asound/cards | grep sgtl5000 | head -1 | awk '{print $1}')
  amixer -c $card sset "Capture Mux" LINE_IN
  amixer -c $card sset "Capture" 8 # 0-15
  

• using device/function numbers (ie from aplay -l):

  gst-launch-1.0 alsasrc device="hw:2,0" ! \
    audioconvert ! \
    alsasink device="hw:0,0"
  

• or, using device aliases (ie from aplay -L):

  gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! \
    audioconvert ! \
    alsasink device="sysdefault:CARD=imxhdmisoc"
  

• Note the use of audioconvert above to handle any format conversions needed between input and output devices

To send audio from the tda1997x HDMI receiver (digital input) to the imxhdmisoc (digital output):

gst-launch-1.0 alsasrc device="sysdefault:CARD=tda1997xaudio" ! \
  "audio/x-raw,rate=44100" ! alsasink device="sysdefault:CARD=imxhdmisoc"

• Note here we need to specify the audio sample-rate as it can vary per input stream (and gstreamer does not validate the rate). This rate must match the source stream samplrate which can be found via sysfs /sys/bus/i2c/drivers/tda1997x/2-0048/audmode. If your output device requires a different sample-rate than the source input device, you need to perform a sample-rate conversion.

Multimedia (Video and Audio combined)

Handling media content that has both audio and video stream types requires a slightly more complex pipeline than the standard audio and video pipeline examples.

Generally, a mixed media pipeline which consumes multimedia will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and queue elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data).

Conversely if producing multimedia content your pipeline will consist of a muxer to join audio and video streams and queue elements.

Named Elements, queues, and Multiple pipelines with gst-launch

When mixing audio and video elements with gst-launch one must make use of multiple pipelines using named elements.

The name property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any).

Multiple pipelines can be provided to gst-launch and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'.

This is best explained with some examples:

• Encoding a stream with audio and video content into an AVI file:

  gst-launch-1.0 \
    videotestsrc \
      ! $VIDEO_CAPABILITIES \
      ! mux. \
    audiotestsrc \
      ! $AUDIO_CAPABILITIES \
      ! mux. \
    avimux name=mux \
      ! filesink location=test.avi
  

  • The videotestsrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux
  • The audiotestsrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux
  • The avimux pipeline specifies name=mux therefore it takes as a source all pipelines that ended with mux.
• Decoding a Matroska container file containing H264 video and AC3 audio

  gst-launch-1.0 \
    filesrc location=file.mkv \
      ! matroskademux name=demux \
    demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \
    demux. ! queue ! h264parse ! v4l2h264dec ! kmssink
  

  • The filesrc pipeline ends with name=demux which means the output of this pipeline will be sent to all pipelines with a demux. source (who's types have been successfully negotiated)
  • The audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3)
  • The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
  • Queue elements are used to keep one pipeline or element from blocking another. For example, if the v4l2h264dec element needs more data from the demux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering
• Decoding a MOV file containing H264 video and AAC audio:

  gst-launch-1.0 \
    filesrc location=file.mov \
      ! qtdemux name=demux \
    demux. ! queue ! aacparse ! avdec_aac ! alsasink \
    demux. ! queue ! h264parse ! v4l2h264dec ! kmssink
  

Muxing Mixed Content

Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'.

You can use gst-inspect to see a list of most of these using grep:

gst-inspect-1.0 | grep -i muxer | grep -vi de

Some common examples:

• mpegtsmux: MPEG Transport Stream Muxer
• mpegpsmux: MPEG Program Stream Muxer
• matroskamux: Matroska muxer
• avimux: Avi muxer
• qtmux: QuickTime Muxer
• oggmux: Ogg muxer

To mux mixed content together include one of these elements following the audio and video pipelines.

Examples:

• Encoding a stream with audio and video content into an AVI file:

  gst-launch-1.0 \
    videotestsrc \
      ! $VIDEO_CAPABILITIES \
      ! mux. \
    audiotestsrc \
      ! $AUDIO_CAPABILITIES \
      ! mux. \
    avimux name=mux \
      ! filesink location=test.avi
  

  • the videotestsrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux and who's format has been successfully negotiated.
  • the audiotestsrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux and who's format has been successfully negotiated.
  • the avimux pipeline specifies name=mux therefore it takes as a source all pipelines that ended with mux. and it understands how to multiplex the two types of data together into its output which is written to the file test.avi

De-muxing mixed content

Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'.

You can use gst-inspect to see a list of most of these using grep:

gst-inspect-1.0 | grep -i 'de\?muxer'

Some common examples:

• tsparse: MPEG transport stream parser
• tsdemux: MPEG transport stream demuxer
• matroskademux: Matroska demuxer
• avidemux: Avi demuxer
• qtdemux: QuickTime demuxer
• oggdemux: Ogg demuxer

To demux mixed content include one of these elements following the audio and video pipelines.

Unlike muxing you also need to use a parser element to parse the bitstream and break it into discrete buffers (frames) that the downstream decoder expects. In other words this 'frames' the data going to decoders.

Some common parsers:

• ogmaudioparse: OGM audio stream parser
• ogmvideoparse: OGM video stream parser
• aacparse: AAC audio stream parser
• amrparse: AMR audio stream parser
• ac3parse: AC3 audio stream parser
• flacparse: FLAC audio parser
• mpegaudioparse: MPEG1 Audio Parser
• h263parse: H.263 parser
• h264parse: H.264 parser
• mpegvideoparse: MPEG video elementary stream parser
• mpeg4videoparse: MPEG 4 video elementary stream parser
• pngparse: PNG parser
• vc1parse: VC1 parser

Examples:

• Demuxing a Matroska container file containing H264 video and AC3 audio into its raw components:

  gst-launch-1.0 \
    filesrc location=file.mkv \
      ! matroskademux name=demux \
    demux. ! queue ! ac3parse ! filesink location=file.ac3 \
    demux. ! queue ! h264parse ! filesink location=file.h264
  

• Demuxing a MOV file containing H264 video and AAC audio into its raw components:

  gst-launch-1.0 \
    filesrc location=file.mov \
      ! qtdemux name=demux \
    demux. ! queue ! aacparse ! filesink location=file.aac \
    demux. ! queue ! h264parse ! filesink locatino=file.h264
  

bin elements

A bin element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept.

Sometimes the bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream. The gst-launch application provides a couple of useful debugging tools that can help with this:

• using GST_DEBUG_DUMP_DOT_DIR and Graphviz
• adding the '-v' parameter to gst-launch will provide verbose feedback on the pipeline configuration that can tell you what is going on

See #troubleshooting for more details on these methods.

GStreamer playbin

GStreamer playbin element attempts to create a pipeline that will play both the audio and video portions of a file. For example:

gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi

The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by providing a video-sink and audio-sink property for example to decode audio to the Analog audio output on Ventana:

gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=sysdefault:CARD=sgtl5000audio"

Please type gst-inspect-1.0 playbin to see more options.

GStreamer decodebin

The GStreamer decodebin element is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under the first example with the following:

gst-launch-1.0 \
  filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \
  matroskademux name=d \
  d. ! queue ! ivorbisdec ! queue ! alsasink \
  d. ! queue ! decodebin  ! queue ! kmssink

Note that decodebin doesn't always choose the correct decoder, so be wary of this. It is similar to playbin in that it aids in creating a dynamic pipeline.

GStreamer gst-play

The stand-alone application gst-play is is a program that utilizes the playbin element and thus can be used for playback of many file types. The above example gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi can be replaced with:

gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi

Streaming (send multimedia to or receive from Network)

GStreamer has elements that allow for network streaming to occur.

There are several ways to accomplish networked streaming over Internet Protocol (IP):

• Raw UDP/IP
• Raw TCP/IP
• Real-time Transport Protocol (RTP)
• Real Time Streaming Protocol (RTSP) (recommended)
• Adaptive Bitrate Streaming

Raw UDP

Using UDP/IP is the simplest mechanism for streaming and utilizes the least amount of bandwidth. Because UDP does not provide any error detection, packet ordering, or error correction the bitrate is deterministic and simply the bitrate of the media you are streaming.

The limitations of raw UDP is:

• requires codec that can handle missing/corrupt data (most do these days)
• does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media)
• does not fragment packets - will try to send a raw udp packet for whatever size buffer the udpsink is passed (which can lead to pipeline errors). To fragment packets use RTP

The only benefit of using raw UDP is that it is the simplest pipeline you can create for streaming and requires the least amount of dependencies (albeit you might run into one or all of the above problems).

Note that it is recommended that you use RTP or RTSP unless you know exactly what you are doing to overcome the limitations listed above

The udpsrc element can be used to render/save a stream originated from a udpsink pipeline.

Examples:

• encode and send H264 video from Ventana:
  1. Start decoder first:

     ifconfig eth0 192.168.1.1
     gst-launch-1.0 udpsrc port=9001 ! h264parse ! v4l2h264dec ! kmssink sync=false
     

  2. Start encoder second:

     ifconfig eth0 192.168.1.2
     gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc ! udpsink host=192.168.1.1 port=9001
     

Notes:

• On the client (stream receiver and renderer) you must use the sync=false property to render frames as they are received otherwise the stream will stall because their is no headers containing timestamps
• the decoder (udpsrc) needs to be started first because udpsink will fail if nothing is listening to the socket

TCP

Using TCP/IP brings error detection, packet re-ordering, and error correction to the network stream. This however causes the bitrate to be non-deterministic because as the error rate increases so does the bitrate and latency.

The limitations of using TCP:

• non-deterministic bitrate
• added latency
• does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media)

Note that it is recommended that you use RTP or RTSP unless you know exactly what you are doing to overcome the limitations listed above

TCP/IP introduces the concept of a socket connection therefore there must exist a server and a client in which case the server must be started first to listen for a connection. You can use a server sink or a server source. The tcpserversrc source can be used to create a TCP server that waits for a connection from a tcpclientsink to render/save. Alternatively the tcpserversink sink can be used to create a TCP server that waits for a connection from a tcpclientsrc that will send data.

Examples:

• encode and send H264 video from Ventana with decoder as server:
  1. Start decoder (server) first:

     ifconfig eth0 192.168.1.1
     gst-launch-1.0 tcpserversrc host=0.0.0.0 port=9001 ! decodebin ! autovideosink sync=false
     

  2. Start encoder (client) second:

     ifconfig eth0 192.168.1.2
     ENCODER=/dev/$(for i in $(ls -d /sys/class/video4linux/video*); do [ "coda-encoder" = "$(cat $i/name)" ] && basename $i; done)
     gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! tcpclientsink host=192.168.1.1 port=9001
     

Notes:

• TCP is connection oriented therefore the TCP 'server' must be started first. You can choose your elements such that the stream originator is the server or the stream renderer is the server however doing so can be problematic for certain codecs because the client decoding the stream may pick up the stream somewhere in the middle and not know how to parse it.
• the host=0.0.0.0 property means listen to all network interfaces

RTP (raw/session-less)

The ​Real-time Transport Protocol (RTP) is a network protocol for delivering audio and video over IP networks. RTP is used extensively in communication and entertainment systems that involve streaming media, such as telephony, video teleconference applications, television services and web-based push-to-talk features.

The RTP packet type encapsulates multimedia data with a payload type and time-stamp and therefore can be used to compensate for jitter, out of sequence packets, and time synchronization between streams of different types (ie audio/video lip-sync).

RTP is typically used in conjunction with other protocols such as RTP Control Protocol (RTCP) and Real Time Streaming Protocol (RTSP) to manage stream sessions however can be used on its own in a raw session-less fashion using udpsink and udpsrc elements.

The limitations of using raw/session-less RTP:

• session management needs to be handled manually (capsfilter is needed to specify stream format)

Note that it is recommended that you use RTSP unless you know exactly what you are doing to overcome the limitations listed above

Example: Capture, encode and stream H264 via RTP with GStreamer playback:

• Encode and send H264 video from Ventana:
  1. Start decoder first:

     ifconfig eth0 192.168.1.1
     gst-launch-1.0 udpsrc port=9001 \
      caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" \
      ! decodebin ! autovideosink
     

  2. Start encoder second:

     gst-launch-1.0 videotestsrc is-live=true \
      ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! rtph264pay ! udpsink host=192.168.1.1 port=9001
     

Notes:

• when using RTP a capsfilter must be used to specify the payload as application/x-rtp as above. You can determine the capsfilter required by starting the encoder with a verbose flag -v and looking for caps = "application/x-rtp"

Example: Capture, encode and stream H264 via RTP with VLC playback:

• Encode and send H264 video from Ventana to a PC with VLC:

1. Start decoder first:
   1. Create SDP file like below (IP address in example is that of the Ventana board)

      v=0
      m=video 5000 RTP/AVP 96
      c=IN IP4 172.24.20.207
      a=rtpmap:96 H264/90000
      

   2. Open SDP file in VLC
2. Start encoder (Ventana) second: (IP address in below example is IP of the PC)

   gst-launch-1.0 videotestsrc ! v4l2h264enc ! rtph264pay config-interval=3 ! udpsink host=172.24.20.26 port=5000
   

RTSP (Real Time Streaming Protocol) (recommended)

The ​Real Time Streaming Protocol (RTSP) is a network control protocol designed for use in entertainment and communications systems to control streaming media servers. The protocol is used for establishing and controlling media sessions between end points. Clients of media servers issue VCR-style commands, such as play and pause, to facilitate real-time control of playback of media files from the server. This protocol uses the Real-time Transport Protocol (RTP) in conjunction with Real-time Control Protocol (RTCP) for media stream delivery.

The limitations of using RTSP are:

• gst-inspect has no way of using a simple pipeline to create an RTSP server - you must create or use an existing gstreamer based application (keep reading below).

Creating an RTSP server is not possible via a simple pipeline to gst-launch however GStreamer libraries do exist to make writing an RTSP server trivial. The source for gst-rtsp-server contains an example application ​test-launch.c which provides a simple example that can take a GStreamer 'bin' element consisting of everything but the sink element and serves it via RTSP.

Gateworks has extended this with gst-variable-rtsp-server which demonstrates how to auto-adjusting properties like the encoding bitrate depending on the number of clients connected in addition to serving as a fairly simple example of how to write a gstreamer application.

gst-variable-rtsp-server

​gst-variable-rtsp-server is a solution created by Gateworks that allows changing the quality of a live stream on the fly based on the number of clients connected to the server. Similar to other adaptive bitrate technologies, it can change quality based on information is knows about the stream. In our case, the number of connected clients was deemed as appropriate.

This application can change the video_bitrate control of the v4l2h264enc element based on the --steps input (default to 5). For example, if using the default steps value of 5, if the min bitrate was 500 and max bitrate was 2000, it would take 5 clients to get from the best to the worst quality.

Example:

• Start a RTSP server using videotestsrc on default port of 9099 serving H264:

  gst-variable-rtsp-server "videotestsrc ! v4l2h264enc ! rtph264pay name=pay0 pt
  =96"
  

To connect as a client, you can use any program that can connect to an RTSP stream. Below is an example connecting with GStreamer:

gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! decodebin ! autovideosink

With the above, if there is only a single client, the stream will be at it's best quality. But as you connect more clients, the quality of the stream will drop in order to account for network congestion.

RTMP Streaming

Real-Time Messaging Protocol (RTMP) is a protocol for streaming audio, video and data over the internet. It was originally developed by Macromedia and proprietary but has since been released for public use. It is a TCP-based protocol which maintains persistent connections.

Using RTMP requires you have:

• an RTMP server (ie Youtube or Nginx)
• a stream transmitter (ie Gateworks board)
• a stream receiver (ie Web browser or Gateworks board)

YouTube Live Streaming

It is possible to stream video to !​YouTube Live Streaming from the Gateworks board using GStreamer's rtmpsink element.

To create a YouTube live feed via RTMP:

• goto YouTube Live Control Room (​https://www.youtube.com/livestreaming/)
• goto ​http://youtube.com and login to your account
• click the Camera icon on the top left to add a stream and select 'Go Live'
• select Stream at the top
• create a title and specify your YouTube attributes (ie Public, etc etc) then hit 'Create Stream'
• take note of your 'Stream key' needed for the encoder pipeline
• start your gstreamer pipeline to capture/encode/stream via rtpmsink to the youtube 'Streamer URL' (see below)
• once YouTube says your stream is ready (indicating its receiving data) hit the 'GO LIVE' button at the top left
• Now you can hit the 'Share' icon at the top left to get a URL to share to view the stream

GStreamer rtmpsink transmitter

Once you have an RTMP server you can use GStreamer to stream live content to it with the rtmpsink element.

Examples:

• color-bar pattern from videotestsrc and audio from audiotestsrc

  gst-launch-1.0 \
    videotestsrc ! "video/x-raw,width=640,height=360,framerate=15/1" ! \
    v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=620000" ! \
    h264parse ! \
    mux. \
  audiotestsrc is-live=true ! "audio/x-raw,format=S16LE,endianness=1234, signed=true,width=16,depth=16,rate=44100,channels=2" ! \
    voaacenc bitrate=128000 ! \
    aacparse ! "audio/mpeg,mpegversion=4,stream-format=raw" ! \
    mux. \
  flvmux streamable=true name=mux ! queue ! \
   rtmpsink location="rtmp://a.rtmp.youtube.com/live2/ee12-bsgt-e22v-8qr1"
  

• captured video/audio:

  gst-launch-1.0 \
    v4l2src device=$DEVICE ! \
    v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=620000" ! \
    h264parse ! \
    mux. \
  alsasrc device="sysdefault:CARD=sgtl5000audio" ! \
    voaacenc bitrate=128000 ! \
    aacparse ! "audio/mpeg,mpegversion=4,stream-format=raw" ! \
    mux. \
  flvmux streamable=true name=mux ! queue ! \
   rtmpsink location="rtmp://a.rtmp.youtube.com/live2/ee12-bsgt-e22v-8qr1"
  

Note that the location property of the rtmpsink is configured on your server. For YouTube for example they provide the URL and you need to append the 'Stream Key' to that path

Troubleshooting

If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device.

You can see all the properties of a particular element with the 'gst-inspect <elementname>' command. You can see a list of all elements with 'gst-inspect' without arguments.

You can enable verbose output for GStreamer by adding the '-v' flag which will show you the negotiated pipeline details and the state machine details.

You can enable debug output for GStreamer by using the ​--gst-debug parameter to gst-launch or by setting the ​GST_DEBUG environment variable.

For example, if you wanted to see DEBUG level messages for the videotestsrc element you can set GST_DEBUG=videotestsrc:4

GST_DEBUG_DUMP_DOT_DIR

You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause gst-launch to output a .dot file for each phase of the pipeline then use a tool such as Graphviz to visualize the .dot file.

Example:

• use playbin to playback a file:

  root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
  

  • hit Cntl-C after decoding starts to exit early
• see the dot files created:

  root@ventana:~# ls /tmp/dot
  0.00.00.108710334-gst-launch.NULL_READY.dot
  0.00.00.490807334-gst-launch.READY_PAUSED.dot
  0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot
  0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
  0.00.03.254000001-gst-launch.PAUSED_READY.dot
  

• transfer to a PC and use something like xdot to view:

  xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
  

  • zoom in along the graph and you can see that:
    • GstFileSrc is the source,
    • GstAviDemux is used to demux to audio/x-ac3,
    • GstAc3Parse is used to parse the audio into audio frames,
    • GstMpeg4VParse is used to parse the video into video frames,
    • GstV4l2VideoDec is used to decode the video from video/mpeg to video/x-raw,
    • GstA52Dec is used to decode the audio from audio/x-ac3 to audio/x-raw,
    • etc
  • Note that some hunting with gst-inspect must be done to determine what elements coorespond to the above class names

Reference:

• ​http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+11%3A+Debugging+tools

gst-launch -v

The verbose debugging from gst-launch -v can show you the negotiation that takes place as a pipeline moves through its stages.

Example:

gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi

examining the verbose output can show you the following:

• container: AVI: avidemux
• video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! v4l2mpeg4dec
• audio: AC3 48khz 5channels: ac3parse ! a52dec

Therefore you can use these pipelines to decode and play:

• video only (output to kmssink)

  gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink
  

• audio only (output to hdmi audio sink)

  gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
  

• both audio and video

  gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \
    d. ! queue ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink \
    d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
  

streaming

If you're having issues with network streaming:

• Verify that both sides can ping one another
• If the message There may be a timestamping problem, or this computer is too slow appears and the video display appears choppy, try the following:
  • Lower the bitrate from the server
  • Place a sync=false on the sink side of the server and client.
• If video appears choppy, try using UDP over TCP.
• Verify that the network is not congested.
• Verify your gstreamer pipeline is correct. The best way to find the element that causes a negotiation failure is to end your pipeline in a fakesink and one-by-one eliminate elements leading up to it until it negotiates successfully.