[[PageOutline]]

= GStreamer on Gateworks SBCs
The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These '''GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs)''', specifically the Ventana family that utilizes the Freescale i.MX6 processors.

Gateworks SBCs can be viewed at the following link: [http://www.gateworks.com]

[[Image(http://trac.gateworks.com/raw-attachment/wiki/OpenWrt/wireless/relayd/gw5100realsmall.png,200px)]]

'''Important Note'''

The IMX6 (Ventana family) software has been evolving over the years. This page is focused on mainline Linux kernels and not the Freescale GStreamer {{{gstreamer-imx}}} plugin legacy software that used the 3.14 Freescale Vendor kernel. If using the Gateworks 3.14 kernel with the Freescale capture drivers please see [wiki:Yocto/gstreamer]

Also note that GStreamer capture devices are {{{/dev/videoN}}} which require some configuration if using Processor specific video input busses such as the IMX6 capture using Analog (CVBS) or Digital video (HDMI). If using these capture interfaces please see [wiki:linux/media] for details on how to use {{{media-ctl}}} to configure those devices.

= OS Support

== Ubuntu
GStreamer packages are available on Ubuntu and the instructions here have been tested with the Ventana Ubuntu Focal image with the Gateworks 5.4 kernel.

== Buildroot
GStreamer packages are available in buildroot and if you are using the Gateworks 5.4 kernel you will be able to use the information on this page.

A specific wiki page shows what configuration you will need: http://trac.gateworks.com/wiki/buildroot/gstreamer


== OpenWrt 
OpenWrt is not intended for Video support. While there are some gstreamer packages available they are limited and the kernel does not have video input and output enabled with the exception of video console support.

Gateworks does not support OpenWrt for video capture and display using Gstreamer.


= Video4Linux2 Devices
The Video4Linux2 API is the device driver API used for kernel capture drivers as well as video encode and video decode drivers.


[=#media-ctl]
== Media Control API
The !MediaControl API is used to configure Video4Linux2 capture devices. See [wiki:linux/media] for more info.


[=#v4l2-ctrls]
== V4L2 Controls (brightness, contrast, saturation, rotation, flip, bitrate etc)
When capturing video from a video4linux2 device you can use the v4l2 API to get and set various controls the device may provide access to such as:
 * brightness
 * contrast
 * hue
 * saturation
 * horizontal flip
 * veritcal flip
 * rotation

What controls are available depend on the device. Furthermore when using capture devices that have media-ctl pipelines controls provided by various elements within the pipeline are passed on to the capture device.

Examples:
 * list available controls of a capture device:
{{{#!bash
media-ctl-setup adv7180 > setup
source setup
v4l2-ctl --device $DEVICE --list-ctrls # list available controls
}}}
 * get brightness control for adv7180 analog video capture device:
{{{#!bash
media-ctl-setup adv7180 > setup
source setup
v4l2-ctl --device $DEVICE --get-ctrl=brightness
}}}
 * set brightness control for adv7180 analog video capture device:
{{{#!bash
media-ctl-setup adv7180 > setup
source setup
v4l2-ctl --device $DEVICE --set-ctrl=brightness=50
}}}
 * set video bitrate of the v4l2h264enc H264 encoder element:
{{{#!bash
media-ctl-setup adv7180 > setup
source setup
v4l2-ctl --device $ENCODER --set-ctrl=video_bitrate=10000000
}}}

**Note that we 'source' the setup file above instead of executing it. This is because it sets some env variables that we will use throughout our examples below. If you are using a shell that does not have 'source' the notation is '. ./setup'**


[=#test-media]
= Test Media Files
There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing:
 * [https://peach.blender.org/download/ Big Buck Bunny] - Creative Commons
 * [https://durian.blender.org/download/ Sintel] - Creative Commons
 * [https://mango.blender.org/download/ Tears of Steel] - Creative Commons


= GStreamer
[http://GStreamer.freedesktop.org GStreamer] is an opensource library and framework created to handle multimedia. Pipelines are used to source and sink media using a multitude of different methods.

It is important to understand that GStreamer is foremost a library meant for developing applications. While it has an extremely useful test application called {{{gst-launch}}} intended for testing purposes, that application alone can't solve all problems. There are plenty of resources online including mailing lists and IRC channels that can be helpful in understanding GStreamer and writing applications that use it.

To install GStreamer on Ubuntu:
{{{#!bash
apt-get install gstreamer1.0-x gstreamer1.0-tools libgstreamer1.0-0 \
  gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio \
  v4l-utils alsa-utils
}}}


[=#plugins]
== Plugins and Elements
GStreamer works on 'plugins' that provide pipeline 'elements'. A plugin comprises of elements that can do work on a media stream. For example, the {{{mpeg2dec}}} can decode mpeg1 and mpeg2 video streams (software based decoding, not hardware accelerated).


[=#pipelines]
== Pipeline Construction
[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-applications.html gst-launch] is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type {{{gst-inspect element}}} where {{{element}}} is the element you are looking for. To see a full listing of elements installed, you can type just {{{gst-inspect}}}.

Here is a graphical example of a pipeline.

[[Image(https://gstreamer.freedesktop.org/documentation/tutorials/basic/images/figure-1.png, 512px)]]

The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next

[http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+10%3A+gstreamer+tools gstreamer.com] contains much more information on the construction of pipelines.


[=#capsfilters]
== Caps Filters
GStreamer has a concept called [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-caps-api.html caps filters]. A 'cap' (short for capability) is used to describe the type of data that links a src (output) pad from one element to a sink (input) pad of another element.

Adding a {{{-v}}} flag to {{{gst-launch-1.0}}} will output the capabilities negotiated between elements:
{{{#!bash
~# gst-launch-1.0 -v videotestsrc ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
}}}

In the output, the {{{caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"}}} are the video caps negotiated between the two.

You can force a caps filter between two elements by treating it as a pipeline element:
{{{#!bash
~# gst-launch-1.0 -v videotestsrc ! 'video/x-raw, format=UYVY, width=1920, height=1080, framerate=10/1' ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
}}}

As you can see, the caps filter above changed both the video format, resolution, and framerate of the video stream coming out of the {{{videotestsrc}}} element.

Caps filters are useful when you want to use a specific format where a device may support multiple. For example when capturing audio from an ALSA sound device that supports a variety of samplerates, channels, and bit formats you can use a capsfilter to override any default configuration.



[=#applications]
== Applications
GStreamer is a library and framework for multimedia processing. It is not a user-space application. It is easy to forget this because of the usefulness of the {{{gst-launch}}} application that can be used to connect simple pipelines together for testing. For your needs you very likely need to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that.

GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html here] for more details.

Furthermore, Gateworks has created a [#gst-variable-rtsp-server gst-variable-rtsp-server] example application that demonstrates how to server simple GStreamer pipelines via an RTSP server and automatically adjust compression quality based on client connections.


[=#fakesink]
== fakesink
The {{{fakesink}}} element is a very useful video sink when debugging or learning GStreamer. It takes whatever frames is given to it and drops them. This might help debugging pipelines if problems ever arise.

Example:
{{{#!bash
gst-launch-1.0 videotestsrc ! fakesink
}}}


== gst-inspect
The {{{gst-inspect-1.0}}} tool is an excellent tool if you're unfamiliar with the functionality, source/sink formats, or properties of any Gstreamer element. 

Usage:
{{{#!bash
root@bionic-armhf:~# gst-inspect-1.0 videotestsrc
Factory Details:
  Rank                     none (0)
  Long-name                Video test source
  Klass                    Source/Video
  Description              Creates a test video stream
  Author                   David A. Schleef <ds@schleef.org>

Plugin Details:
  Name                     videotestsrc
  Description              Creates a test video stream
  Filename                 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstvideotestsrc.so
  Version                  1.14.5
  License                  LGPL
  Source module            gst-plugins-base
  Source release date      2019-05-29
  Binary package           GStreamer Base Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-base1.0

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstVideoTestSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)I420, (string)YV12, (string)YUY2, (string)UYVY, (string)AYUV, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGB, (string)BGR, (string)Y41B, (string)Y42B, (string)YVYU, (string)Y444, (string)v210, (string)v216, (string)NV12, (string)NV21, (string)GRAY8, (string)GRAY16_BE, (string)GRAY16_LE, (string)v308, (string)RGB16, (string)BGR16, (string)RGB15, (string)BGR15, (string)UYVP, (string)A420, (string)RGB8P, (string)YUV9, (string)YVU9, (string)IYU1, (string)ARGB64, (string)AYUV64, (string)r210, (string)I420_10BE, (string)I420_10LE, (string)I422_10BE, (string)I422_10LE, (string)Y444_10BE, (string)Y444_10LE, (string)GBR, (string)GBR_10BE, (string)GBR_10LE, (string)NV16, (string)NV24, (string)NV12_64Z32, (string)A420_10BE, (string)A420_10LE, (string)A422_10BE, (string)A422_10LE, (string)A444_10BE, (string)A444_10LE, (string)NV61, (string)P010_10BE, (string)P010_10LE, (string)IYU2, (string)VYUY, (string)GBRA, (string)GBRA_10BE, (string)GBRA_10LE, (string)GBR_12BE, (string)GBR_12LE, (string)GBRA_12BE, (string)GBRA_12LE, (string)I420_12BE, (string)I420_12LE, (string)I422_12BE, (string)I422_12LE, (string)Y444_12BE, (string)Y444_12LE, (string)GRAY10_LE32, (string)NV12_10LE32, (string)NV16_10LE32 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
         multiview-mode: { (string)mono, (string)left, (string)right }
      video/x-bayer
                 format: { (string)bggr, (string)rggb, (string)grbg, (string)gbrg }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
         multiview-mode: { (string)mono, (string)left, (string)right }

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "videotestsrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  typefind            : Run typefind before negotiating (deprecated, non-functional)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: false
  pattern             : Type of test pattern to generate
                        flags: readable, writable
                        Enum "GstVideoTestSrcPattern" Default: 0, "smpte"
                           (0): smpte            - SMPTE 100% color bars
                           (1): snow             - Random (television snow)
                           (2): black            - 100% Black
                           (3): white            - 100% White
                           (4): red              - Red
                           (5): green            - Green
                           (6): blue             - Blue
                           (7): checkers-1       - Checkers 1px
                           (8): checkers-2       - Checkers 2px
                           (9): checkers-4       - Checkers 4px
                           (10): checkers-8       - Checkers 8px
                           (11): circular         - Circular
                           (12): blink            - Blink
                           (13): smpte75          - SMPTE 75% color bars
                           (14): zone-plate       - Zone plate
                           (15): gamut            - Gamut checkers
                           (16): chroma-zone-plate - Chroma zone plate
                           (17): solid-color      - Solid color
                           (18): ball             - Moving ball
                           (19): smpte100         - SMPTE 100% color bars
                           (20): bar              - Bar
                           (21): pinwheel         - Pinwheel
                           (22): spokes           - Spokes
                           (23): gradient         - Gradient
                           (24): colors           - Colors
  timestamp-offset    : An offset added to timestamps set on buffers (in ns)
                        flags: readable, writable
                        Integer64. Range: 0 - 2147483646999999999 Default: 0 
  is-live             : Whether to act as a live source
                        flags: readable, writable
                        Boolean. Default: false
  k0                  : Zoneplate zero order phase, for generating plain fields or phase offsets
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kx                  : Zoneplate 1st order x phase, for generating constant horizontal frequencies
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  ky                  : Zoneplate 1st order y phase, for generating contant vertical frequencies
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kt                  : Zoneplate 1st order t phase, for generating phase rotation as a function of time
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kxt                 : Zoneplate x*t product phase, normalised to kxy/256 cycles per vertical pixel at width/2 from origin
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kyt                 : Zoneplate y*t product phase
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kxy                 : Zoneplate x*y product phase
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kx2                 : Zoneplate 2nd order x phase, normalised to kx2/256 cycles per horizontal pixel at width/2 from origin
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  ky2                 : Zoneplate 2nd order y phase, normailsed to ky2/256 cycles per vertical pixel at height/2 from origin
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  kt2                 : Zoneplate 2nd order t phase, t*t/256 cycles per picture
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  xoffset             : Zoneplate 2nd order products x offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  yoffset             : Zoneplate 2nd order products y offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  foreground-color    : Foreground color to use (big-endian ARGB)
                        flags: readable, writable, controllable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  background-color    : Background color to use (big-endian ARGB)
                        flags: readable, writable, controllable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4278190080 
  horizontal-speed    : Scroll image number of pixels per frame (positive is scroll to the left)
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  animation-mode      : For pattern=ball, which counter defines the position of the ball.
                        flags: readable, writable
                        Enum "GstVideoTestSrcAnimationMode" Default: 0, "frames"
                           (0): frames           - frame count
                           (1): wall-time        - wall clock time
                           (2): running-time     - running time
  motion              : For pattern=ball, what motion the ball does
                        flags: readable, writable
                        Enum "GstVideoTestSrcMotionType" Default: 0, "wavy"
                           (0): wavy             - Ball waves back and forth, up and down
                           (1): sweep            - 1 revolution per second
                           (2): hsweep           - 1/2 revolution per second, then reset to top
  flip                : For pattern=ball, invert colors every second.
                        flags: readable, writable
                        Boolean. Default: false
}}}



[=#building]
== Building GStreamer from source
GStreamer has recently converted to a single repository which made things much easier to build.

Example building on Ubuntu 20.04 Focal:
 * build
{{{#!bash
# install a few dependencies
sed -i "s/# deb-src/deb-src/" /etc/apt/sources.list
apt update
apt build-dep gstreamer1.0-plugins-good
apt build-dep gstreamer1.0-plugins-bad
apt build-dep gstreamer1.0-plugins-ugly
apt install git python3-pip flex bison ninja-build
# meson version on Ubuntu focal is too old...download a local one
apt purge meson
apt install python3-pip
pip3 install --user meson
PATH=~/.local/bin:$PATH
# clone repo
git clone https://gitlab.freedesktop.org/gstreamer/gstreamer.git
cd gstreamer
# create Makefiles with meson to build with objects in 'build' subdir
meson build
# use ninja to build in 'build' subdir
ninja -C build
# use ninja to create a development environment to use locally without installing
ninja -C build devenv
}}}
 - to reconfigure use 'meson build --reconfigure'


[=#video]
= GStreamer Video
This section will introduce several concepts regarding how video media is handled by GStreamer and provide several example pipelines.


[=#video-output]
== Video Output
Generally, a GStreamer 'sink' element is one that will take a video stream and output it to a display.

Some commonly used video output sinks:
 * autovideosink: Wrapper video sink for automatically detecting video output device 
 * kmssink: KMS video device sink
 * fbdevsink: Framebuffer video device sink

Execute a {{{gst-inspect-1.0 | grep sink}}} to see a complete list of  sinks available and {{{gst-inspect-1.0 <element>}}} to see specific details about the applicable formats and properties of a specific element.

**Important Note**: Video display drivers for use with video display sinks such as kmssink and fbdevsink for HDMI out and LVDS require proper kernel commandline parameters to configure them. Sometimes the Bootloader bootscript will do this for you - Please see [wiki:linux/display] for details and check the kernel command-line after booting to make sure the desired device is enabled and configured for the proper resolution and mode.

Examples:
 * HDMI enabled and configured for 1080p60Hz LVDS disabled:
{{{#!bash
~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done
video=HDMI-A-1:1920x1080M@60
video=LVDS-1:d
}}}
 * HDMI disabled and LVDS configured for 1280x800 60Hz
{{{#!bash
~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done
video=LVDS-1:1280x800@60M
video=HDMI-A-1:d
}}}


[=#autovideosink]
=== autovideosink
This GStreamer sink is not really a 'video' sink in the traditional sense. Similar to {{{playbin}}} and {{{decodebin}}}, this element selects what it thinks is the best available video sink and uses it.

This will typically use {{{kmssink}}} unless format choices require one of the other sinks.

You can add a verbose flag {{{gst-launch-1.0 -v}}} to see details about the elements and caps chosen when using any type of 'auto' element or 'bin' element.

Example:
{{{#!bash
gst-launch-1.0 videotestsrc ! autovideosink
}}}
 * Note that adding a '-v' flag for verbose mode will show the element configuration to let you know what sink was actually chosen


[=#kmssink]
=== kmssink
This sink allows you to directly output to a KMS (kernelmode setting API) driver. This is the most efficient video output method.

Example:
{{{#!bash
gst-launch-1.0 videotestsrc ! kmssink
}}}


[=#fbdevsink]
=== fbdevsink
The Framebuffer device sink allows you to directly output to a Linux Framebuffer device if you have one (ls /sys/class/graphics/fb*).

Example:
{{{#!bash
gst-launch-1.0 videotestsrc ! fbdevsink
}}}


[=#video-input]
== Video Input (Capture)
A video input source is anything coming from an input on the device, e.g. HDMI input/USB Web Cam. This is referred to as Video Capture. In GStreamer terms a video capture device is a 'source' or 'src' element.

Please refer to the [wiki:linux/media linux/media] page for details on the video in devices on the Ventana platform.

Some commonly used GStreamer video input sources:
 * videotestsrc
 * v4l2src

Execute a {{{gst-inspect-1.0 | grep src}}} to see a complete list of source elements available and {{{gst-inspect-1.0 <element>}}} to see specific details about the applicable formats and properties of a specific element.

If the {{{is-live}}} property is set to true this will cause buffers to be discarded on a pipeline paused state and pipelines will not participate in the PREROLL phase of a pipeline.

** Important Note: ** Linux video capture devices provided by CPU's such as the IMX6 Analog (CVBS) and Digital (HDMI) inputs provided on Ventana boards require a hardware pipeline configuration. See [wiki:linux/media] for details on how to configure the video capture device pipeline.


[=#videotestsrc]
=== videotestsrc
This is a very useful element for testing pipelines as it creates video content. It can output a huge number of video formats for raw video {{{video/x-raw}}} and bayer video {{{video/x-bayer}}}. Use {{{gst-inspect-1.0 videotestsrc}}} to see its output formats and properties such as video test pattern, orientation, colrs, etc.

Examples:
 * display SMPTE pattern to connected and configured display
{{{#!bash
gst-launch-1.0 videotestsrc ! kmssink
}}}
 * set the 'pattern' property to display a moving ball:
{{{#!bash
gst-launch-1.0 videotestsrc pattern=18 ! kmssink
}}}
 * Use a '[#capsfilter capsfilter]' to force output of videotestsrc to a specific resolution, format, and framerate (ie 1080p 10fps YUV):
{{{#!bash
gst-launch-1.0 videotestsrc ! 'video/x-raw,format=UYVY,width=1920,height=1080,framerate=10/1' ! kmssink
}}}

[=#v4l2src]
=== v4l2src
This element uses the video4linux2 API to capture video from input sources (/dev/video<n>).

Note that {{{v4l2src}}} is always live regardless of the is-live property.

Note that you need to configure the media-ctl pipeline that feeds the /dev/video<n> devices - see [wiki:linux/media] for details

You can use the {{{v4l2-ctl}}} application from the {{{v4l-utils}}} package to to interact with the device to get/set various capabilities and controls.

Examples:
 * show all video4linux2 devices:
{{{#!bash
CODA960 (platform:coda):
        /dev/video9
        /dev/video10
        /dev/video11
        /dev/video12

imx-media (platform:imx-media):
        /dev/media0

imx-media-csc-scaler (platform:imx-media-csc-scaler):
        /dev/video8

imx-capture (platform:ipu1_csi0):
        /dev/video0

imx-capture (platform:ipu1_csi1):
        /dev/video3

imx-capture (platform:ipu1_ic_prpenc):
        /dev/video1

imx-capture (platform:ipu1_ic_prpvf):
        /dev/video2

imx-capture (platform:ipu2_csi0):
        /dev/video4

imx-capture (platform:ipu2_csi1):
        /dev/video7

imx-capture (platform:ipu2_ic_prpenc):
        /dev/video5

imx-capture (platform:ipu2_ic_prpvf):
        /dev/video6
}}}
  - note the output may vary depending on your kernel version, board, and processor
 * display all details about /dev/video1:
{{{#!bash
v4l2-ctl -d /dev/video1 --all
}}}
 * Use Gateworks {{{media-ctl-setup}}} to configure the media-ctl pipeline for Analog Video capture using the adv7180 chip. If you don't already have this on your root filesystem you can install it via:
{{{#!bash
wget https://raw.githubusercontent.com/Gateworks/media-ctl-setup/master/media-ctl-setup -O /usr/local/bin/media-ctl-setup
chmod +x /usr/local/bin/media-ctl-setup
media-ctl-setup adv7180 > setup
. ./setup
}}}
  - you can look at the setup script to see what was done - the details vary between SoC (IMX6DL vs IMX6Q), board, and sensor (adv7180 for Analog video capture and tda1997x for digital video capture)
  - note that the setup script exports the video capture device to the DEVICE env variable which you can use as shown below
 * After sourcing setup above you can capture and display to the connected and configured output display (video loopback):
{{{#!bash
gst-launch-1.0 v4l2src device=$DEVICE ! kmssink
}}}


[=#colorspace]
[=#scaling]
== Colorspace Converting and/or Video Scaling (via IMX6 imx-media-csc-scaler driver)
Often times, a colorspace conversion or scaling is required in order to link GStreamer elements together. This is often due to the fact that not all elements can every format available.

A Linux V4L2 MEM2MEM imx-media driver exists that allows utilizing the IMX6 IPU Image Converter hardware blocks (IC) to perform hardware colorspace conversion (CSC), scaling, cropping, rotation, and flip operations.

The GStreamer {{{video4linux2}}} element provides an element that uses this driver to expose these capabilities to GStreamer applications.

Notes:
 - For GStreamer-1.14 the name of the element depends on the video device the driver registers with the kernel (ie v4l2video8convert if the driver registers /dev/video8) but for GStreamer-1.16 and beyond the name of the element is stable and always 'v4l2convert'. For the examples on this page we use the GST_CONVERT env variable that is set from the output of the media-ctl-setup script to avoid confusion.
 - the {{{kmssink}}} examples below need a {{{can-scale=false}}} property to tell GStreamer not to scale via the KMS driver (as the IMX6 KMS driver does not support scaling)
 - ensure that the input format differs from the output format otherwise GStreamer will bypass the conversion completely; note that GStreamer doesn't understand flipping or rotation as part of the format. Gstreamer master (in development) adds a 'disable-passthrough' property to the v4l2convert entity that can be set to force the conversion regardless of input and output format
 - when using imx entities (ie capture, encode/decode, csc/scaler, display) you can specify 'output-io-mode=dmabuf-import' to share dmabuf pointers for a zero-copy pipeline however if using non imx entities (ie videotestsrc) you must omit these as you can not ensure the buffers share the alignment/stride necessary to share dmabuf pointers

Examples:
 * Ensure GStreamer element exists:
{{{#!bash
~# gst-inspect-1.0 | grep -e "v4l2.*convert"
video4linux2:  v4l2convert: V4L2 Video Converter
}}}
 * Obtain the name of the element (as it can vary between GStreaqmer-1.14 and GStreamer 1.16)
{{{#!bash
GST_CONVERT=$(gst-inspect-1.0 | grep -e "v4l2.*convert*" | sed -e 's/.*:\s*\(v4l2.*convert\):.*/\1/')
}}}
  - for modern Gstreamer 1.16 this will be 'v4l2convert' but for earlier Gstreamer 1.14 it will vary depending on the video device number which is why we assign it to a variable here for the examples
 * scale/rotate/flip using {{{videotestsrc}}} (can not use dmabufs for this as it is a non-imx entity)
{{{#!bash
# upscale
gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
    $GST_CONVERT ! \
    video/x-raw,width=640,height=480 ! kmssink can-scale=false
# downscale
gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480 ! \
    $GST_CONVERT ! \
    video/x-raw,width=320,height=240 ! kmssink can-scale=false
# rotate
gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
    $GST_CONVERT extra-controls=cid,rotate=90 ! \
    video/x-raw,width=240,height=320 ! kmssink can-scale=false
# hflip
gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
    $GST_CONVERT extra-controls=cid,horizontal_flip=1 ! \
    video/x-raw,width=640,height=480 ! kmssink can-scale=false
# vflip
gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \
    $GST_CONVERT extra-controls=cid,vertical_flip=1 ! \
    video/x-raw,width=640,height=480 ! kmssink can-scale=false
}}}
  - note the above examples force the input format (resolution in this case) to differ from the output format otherwise gstreamer will bypass the v4l2convert entity thinking it not necessary as gstreamer does not understand the flip/rotation properties. GStreamer master (in development) adds the 'disable-passthrough' property which can be enabled to force disabling passthrough
 * scale/rotate/flip using imx-media capture device and KMS display driver (can use dmabufs for this as they are all imx hardware entitites):
{{{#!bash
# scale sensor input to 720p display
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import ! \
     video/x-raw,width=1280,height=720 ! \
     kmssink can-scale=false
# scale sensor input to 1080p display
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import ! \
     video/x-raw,width=1920,height=1080 ! \
     kmssink can-scale=false
# scale/flip
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \
     video/x-raw,width=1920,height=1080 ! \
     kmssink can-scale=false
# scale/rotate
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \
     video/x-raw,width=720,height=1280 ! \
     kmssink can-scale=false
}}}
 * capture, scale, rotate, flip and encode using imx-media capture device, csc/scaler device and coda device (can use dmabufs for zero-copy)
{{{#!bash
# encode
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import ! \
     video/x-raw,colorimetry=bt709 ! \
     v4l2h264enc output-io-mode=dmabuf-import ! \
     rtph264pay ! udpsink host=$SERVER port=$PORT
# scale/encode
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import ! \
     video/x-raw,width=1440,height=960,colorimetry=bt709 ! \
     v4l2h264enc output-io-mode=dmabuf-import ! \
     rtph264pay ! udpsink host=$SERVER port=$PORT
# scale/flip/encode
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \
     video/x-raw,width=1440,height=960,colorimetry=bt709 ! \
     v4l2h264enc output-io-mode=dmabuf-import ! \
     rtph264pay ! udpsink host=$SERVER port=$PORT
# scale/rotate/encode
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \
     video/x-raw,width=1440,height=960,colorimetry=bt709 ! \
     v4l2h264enc output-io-mode=dmabuf-import ! \
     rtph264pay ! udpsink host=$SERVER port=$PORT
}}}
 * note the above examples also use v4l2convert to convert to bt709 coloriemtry required by the encoder elements

For anything not able to be handled by the v4l2convert element and csc/scaler driver there are software based elements that can convert pixel colorspace and/or scale however note these are extremely CPU intensive:
 * autovideoconvert
 * videoconvert
 * rgb2bayer 


[=#interlaced-video]
== Interlaced Video and Deinterlacing
Interlaced video is a technique for doubling the perceived frame rate of a video display without consuming extra bandwidth. The interlaced signal contains two fields of a video frame captured at two different times. The alternative to interlaced video is called progressive video.
 
While reducing bandwidth this can cause a perceived flicker effect as well as a very apparent artifact seen during motion. For example a car moving horizontally across a scene will show every other vertical line differently: the second field interlaced with the first field will be a full frame period ahead in time from the other. The visual affect can be seen in [https://en.wikipedia.org/wiki/File:Interlaced_video_frame_(car_wheel).jpg ​this image from Wikipedia]

Television signals are typically interlaced, or at least were until recently. For example, analog television standards such as NTSC used in North America as well as the PAL and SECAM formats used abroad use interlaced video and therefore any analog video decoder such as the ADV7180 found on many Gateworks Ventana boards will capture interlaced video and are subject to interlacing artifacts. Interlaced video is still used in High Definition signals as well and the letter at the end of the format tells you if its interlaced (ie 480i, 720i, 1080i) or progressive (ie 480p, 720p, 1080p).

To deinterlace video captured from the adv7180 analog decoder you need to use media-ctl to configure your pipeline to include the IMX6 Video Deinterlacer block. This is done for you if you use the Gateworks {{{media-ctl-setup}}} script to configure your pipeline. See [wiki:linux/media#media-ctl-setup linux/media] for more info.

 
References:
* [https://en.wikipedia.org/wiki/Interlaced_video Wikipedia interlaced video - includes several images demonstrating interlacing artifacts]
* [https://en.wikipedia.org/wiki/Deinterlacing​Wikipedia deinterlacing]


[=#encoding] 
== Video Encoding (via IMX6 coda driver and video4linux2 element)
Encoding video means taking the raw captured pixels and converting them to a standard compressed format.

The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer {{{video4linux2}}} plugin provides various encode/decode elements that tap into this.

Notes:
 * Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded:
{{{#!bash
~# dmesg | grep coda    
[   10.798192] coda 2040000.vpu: Firmware code revision: 570363
[   10.804115] coda 2040000.vpu: Initialized CODA960.
[   10.808929] coda 2040000.vpu: Firmware version: 3.1.1
[   10.817951] coda 2040000.vpu: encoder registered as video8
[   10.825635] coda 2040000.vpu: decoder registered as video9
[   10.832117] coda 2040000.vpu: encoder registered as video10
[   10.839520] coda 2040000.vpu: decoder registered as video11
...
~# for i in $(ls -d /sys/class/video4linux/video*); do echo $i:$(cat $i/name); do
ne | grep encoder
/sys/class/video4linux/video10:coda-encoder
/sys/class/video4linux/video8:coda-jpeg-encoder
}}}
 * Ensure GStreamer encode elements exists:
{{{#!bash
~# gst-inspect-1.0 | grep -e "v4l2.*enc"
video4linux2:  v4l2jpegenc: V4L2 JPEG Encoder
video4linux2:  v4l2h264enc: V4L2 H.264 Encoder
video4linux2:  v4l2mpeg4enc: V4L2 MPEG4 Encoder
}}}
 * The CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry (bt709)
 * The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough)
 * Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the v4l2-ctl -L and v4l2-ctl -l parameters on the devices exposed by the coda driver
 * Often you will need to use the csc/scaler driver (v4l2convert gstreamer element) for format conversions (colorspace, colorimetry) to reconcile the format of the source device and the encoder element - notice the $GST_CONVERT env variable below.
 * V4L2 JPEG Encoder support was added to the CODA driver in Linux 5.6 and has been backported to the Gateworks 5.4 kernel
 * V4l2jpegenc JPEG Encoder support requires GStreamer-1.16+

Examples:
 * Encode to H264 (hardware based encode) and stream via RTP/UDP:
{{{#!bash
# stream H264/RTP/UDP
gst-launch-1.0 v4l2src device=$DEVICE ! \
        $GST_CONVERT output-io-mode=dmabuf-import ! video/x-raw,colorimetry=bt709 ! \
        v4l2h264enc output-io-mode=dmabuf-import ! \
        rtph264pay ! udpsink host=$SERVER port=$PORT
# client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT caps=application/x-rtp,payload=96 ! rtph264depay ! decodebin ! autovideosink'
}}}
 * Encode to JPEG (software based encode) and stream via RTP/UDP:
{{{#!bash
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=$DEVICE ! jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
# client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay ! jpegdec ! autovideosink'
}}}
 * Encode to JPEG (hardware based encode) and stream via RTP/UDP:
{{{#!bash
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=$DEVICE ! \
        $GST_CONVERT output-io-mode=dmabuf-import ! video/x-raw,colorimetry=bt709 ! \
        v4l2jpegenc output-io-mode=dmabuf-import ! \
        rtpjpegpay ! udpsink host=$SERVER port=$PORT
# client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay ! jpegdec ! autovideosink'
}}}
  - requires GStreamer 1.16+ and kernel supporting CODA JPEG encode

See also:
 * [wiki:ventana/vpu ventana/vpu]



[=#decoding] 
== Video Decoding (via IMX6 coda driver and video4linux2 element)
Decoding video means taking compressed formats such as MPEG and H264 and decoding them into frames comprised of raw pixels (typically so they can be displayed on a video display output device).

The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer {{{video4linux2}}} plugin provides encode/decode elements that tap into this.

Notes:
 * Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded:
{{{#!bash
~# dmesg | grep coda
[   10.798192] coda 2040000.vpu: Firmware code revision: 570363
[   10.804115] coda 2040000.vpu: Initialized CODA960.
[   10.808929] coda 2040000.vpu: Firmware version: 3.1.1
[   10.817951] coda 2040000.vpu: encoder registered as video8
[   10.825635] coda 2040000.vpu: decoder registered as video9
[   10.832117] coda 2040000.vpu: encoder registered as video10
[   10.839520] coda 2040000.vpu: decoder registered as video11
~# for i in $(ls -d /sys/class/video4linux/video*); do echo $i:$(cat $i/name); do
ne | grep decoder
/sys/class/video4linux/video11:coda-decoder
/sys/class/video4linux/video9:coda-jpeg-decoder
}}}
 * Ensure GStreamer decode elements exists:
{{{#!bash
~# gst-inspect-1.0 | grep -e "v4l2.*dec"
video4linux2:  v4l2jpegdec: V4L2 JPEG Decoder
video4linux2:  v4l2mpeg4dec: V4L2 MPEG4 Decoder
video4linux2:  v4l2mpeg2dec: V4L2 MPEG2 Decoder
video4linux2:  v4l2h264dec: V4L2 H264 Decoder
}}}
 * The CODA960 decoder outputs NV12/I420/YV12 YUV pixel formats
 * The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough but you may need to bump this to 128MB for 1080p)
 * V4L2 JPEG Decoder support has not yet been added to the CODA driver as of Linux 5.6 - patches have been posted and are in the Gateworks 5.4 kernel

Note that the following examples assume you are using raw video encoded files, not container formats used for mimxed multimedia types (audio + video) such as ogg, avi, or mov (Quicktime). For information on de-muxing container formats see [#multimedia]

Examples:
 * Decoding H264 video might look like this:
{{{#!bash
gst-launch-1.0 filesrc location=file.h264 ! h264parse ! v4l2h264dec  ! kmssink
}}}
 * Decoding MPEG4 video might look like this:
{{{#!bash
gst-launch-1.0 filesrc location=file.mp4v ! mpeg4videoparse !  v4l2mpeg4dec ! kmssink
}}}
 * Streaming JPEG video might look like this:
{{{#!bash
# transmit JPEG/RTP/UDP on on board:
gst-launch-1.0 v4l2src device=$DEVICE ! v4l2jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
# recieve JPEG/RTP/UDP on another board:
gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay !  v4l2jpegdec ! video/x-raw,format=I420 ! autovideosink
}}}
  - Note the fact that we need to use a capsfilter to force v4l2jpegdec to output I420

For more examples including working with 'multimedia' files that contain both audio and video see [#multimedia]



[=#gstreamer-alsa]
[=#audio]
= GStreamer Audio
The GStreamer {{{alsasrc}}} and {{{alsasink}}} provide audio capture and playback to Advanced Linux Sound Architecture (ALSA) devices. ALSA is the modern device-driver API for the Linux kernel.

The Gstreamer {{{alsasrc}}} and {{{alsasink}}} can be passed a {{{device}}} property which can represent either the 'device alias' (reported from {{{aplay -L}}} for playback, or {{{arecord -L}}} for record) (ie {{{sysdefault:CARD=sgtl5000audio}}}) or the 'hw:<card>,<device>' notation (ie {{{hw:0,0}}} for first device, first subdevice)

If not specified via the {{{device}}} property, the device used for alsasrc and alsasink will depend on {{{/etc/asound.conf}}} and/or {{{~/.asoundrc}}}.

See [wiki:linux/audio#alsa linux/audio] for details on specifying audio input and output devices on the Ventana product family.

Playback and Capture also depends on your ALSA mixer ({{{amixer}}}) settings. See [wiki:linux/audio#amixer linux/audio#amixer] for more info.


[=#audio-output]
== Audio Output via Linux ALSA driver and GStreamer alsasink
Generally, a 'sink' element is one that will take an audio stream and send it to an audio device. Please refer to [wiki:linux/audio linux/audio] for more details on audio devices on the ventana platform.

This {{{alsasink}}} element can accept any of the following formats via {{{audio/x-raw}}}: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE

Examples:
 * Generate and play a 1kHz tone to the analog audio output device on Ventana (sgtl5000audio):
{{{#!bash
gst-launch-1.0 audiotestsrc ! \
  alsasink device="sysdefault:CARD=sgtl5000audio"
}}}


[=#audio-input]
== Audio Input via Linux ALSA driver and GStreamer alsasrc
An input source is anything coming from a capture device on the SBC, e.g. HDMI audio in/analog audio in. Please refer to [wiki:linux/audio linux/audio] for more details on audio devices on the ventana platform.

The {{{alsasrc}}} element can output the following source types in {{{audio/x-raw}}}: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE

Examples:
 * Capture 32kHz stereo 16bit audio from the Analog Audio input on Ventana (sgtl5000audio) and mux into into an AVI file:
{{{#!bash
gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \
  audioconvert ! avimux ! filesink location=./audio.avi
}}}


[=#audio-test]
== audiotestsrc
The {{{audiotestsrc}}} is a very useful element for testing. It can output 16-64bit {{{audio/x-raw}}}: S16LE, S32LE, F32LE, F64LE

This can output audio signals from {{{sine}}} to {{{violet-noise}}}.

Examples:
 * Generate and play a 1kHz tone to the analog audio output device on Ventana (sgtl5000audio):
{{{#!bash
gst-launch-1.0 audiotestsrc ! \
  alsasink device="sysdefault:CARD=sgtl5000audio"
}}}


[=#audio-encoding]
== Audio Encoding
Encoding is the term used to capture audio and encode it to a new format type.

Commonly used GStreamer Audio encoders:
 * vorbisenc - encode {{{audio/x-raw}}} F32LE to {{{audio/x-vorbis}}}
 * mulawenc - encode {{{audio/x-raw}}} S16LE to {{{audio/x-mulaw}}}
 * wavenc - encode {{{audio/x-raw}}} S32LE, S24LE, S16LE, U8, F32LE, F64LE, {{{audio/x-alaw}}}, and {{{audio/x-mulaw}}} to {{{audio/x-wav}}}
 * alawenc - encode {{{audio/x-raw}}} S16LE to {{{audio/x-alaw}}}
 * flacenc - encode {{{audio/x-raw}}} S24LE, S24_32LE, S16LE, S8 to {{{audio/x-flac}}}
 * lamemp3enc -  encode {{{audio/x-raw}}} S16LE to {{{audio/mpeg}}}

There are many more. You can search for your specified one by running a similar search: {{{gst-inspect-1.0 | grep enc}}}.

Note that each encoder has its own limits on the samples it accepts (S16LE, S24LE etc). You can find the formats accepted via {{{gst-inspect-1.0 <element>}}} and you can use the [#audioconvertaudioconvert] element  to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements.

Examples:
 * Capture audio from the Ventana Analog audio input (sgtl5000audio), encode it using the MPEG Layer 3 audio codec and store it to a file:
{{{#!bash
gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! audioconvert ! lamemp3enc ! filesink location=file.mp3
}}}


[=#audio-decoding]
== Audio Decoding
Decoding is the term used to decode an encoded audio stream to a raw audio stream.

Commonly used GStreamer Audio decoders:
 * mpg123audiodec - decode {{{audio/mpeg}}} to {{{audio/x-raw}}}
 * vorbisdec - decode {{{audio/x-vorbis}}} to {{{audio/x-raw}}}
 * ivorbisdec - decode {{{audio/x-vorbis}}} to {{{audio/x-raw}}}
 * a52dec - decode {{{audio/x-ac3}}}, {{{audio/ac3}}}, and {{{audio/x-private1-ac3}}} to {{{audio/x-raw}}}
 * mulawdec - decode {{{audio/x-mulaw}}} to {{{audio/x-raw}}}
 * alawdec - decode {{{audio/x-alaw}}} to {{{audio/x-raw}}}
 * flacdec - decode {{{audio/x-flac}}} to {{{audio/x-raw}}}

Note that each decoder has its own limits on the samples it produces at its src pad (S16LE, S24LE etc). You can find the formats accepted via {{{gst-inspect-1.0 <element>}}} and you can use the [#audioconvert audioconvert] element  to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements.

Examples:
 * Decode MPEG Layer 3 audio and playback to the default audio device  with the following:
{{{#!bash
gst-launch-1.0 filesrc location=file.mp3 ! mpegaudioparse ! mpg123audiodec ! alsasink
}}}


[=#audio-format]
[=#audioconvert]
== Audio Formats and Conversion
Audio samples specified by the {{{audio/x-raw}}} vary between bit width and endieness as well as the sampling rate (how many times a second audio is measured) and the number of channels that are sampled.

To specify the details of the audio format, you use a [#capsfilter GStreamer capsfilter] and to convert it from one format to another you use the GStreamer {{{audioconvert}}} element.

Examples:
 * specify the sample-rate (32kHz), channels (2:stereo), and bit depth (16bit) and capture to an AVI file:
{{{#!bash
gst-launch-1.0 audiotestsrc ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \
  audioconvert ! avimux ! filesink location=./audio.avi
}}}


[=#audio-loopback]
== Audio loopback (Useful for testing audio input and output)
A simple aduio loopback test can take audio input from an input device and output it to an output device.

loopback audio from the first audio card:
{{{#!bash
gst-launch-1.0 alsasrc device="hw:0,0" ! alsasink device="hw:0,0"
}}}

To send audio from the Ventana Analog input (sgtl5000audio) to the Ventana HDMI output (DWHDMI):
 * first, enable Analog input Line in and set capture volume:
{{{#!bash
card=$(cat /proc/asound/cards | grep sgtl5000 | head -1 | awk '{print $1}')
amixer -c $card sset "Capture Mux" LINE_IN
amixer -c $card sset "Capture" 8 # 0-15
}}}
 * using device/function numbers (ie from {{{aplay -l}}}):
{{{#!bash
gst-launch-1.0 alsasrc device="hw:2,0" ! \
  audioconvert ! \
  alsasink device="hw:0,0"
}}}
 * or, using device aliases (ie from {{{aplay -L}}}):
{{{#!bash
gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! \
  audioconvert ! \
  alsasink device="sysdefault:CARD=imxhdmisoc"
}}}
 * Note the use of {{{audioconvert}}} above to handle any format conversions needed between input and output devices

To send audio from the tda1997x HDMI receiver (digital input) to the imxhdmisoc (digital output):
{{{#!bash
gst-launch-1.0 alsasrc device="sysdefault:CARD=tda1997xaudio" ! \
  "audio/x-raw,rate=44100" ! alsasink device="sysdefault:CARD=imxhdmisoc"
}}}
  * Note here we need to specify the audio sample-rate as it can vary per input stream (and gstreamer does not validate the rate). This rate must match the source stream samplrate which can be found via sysfs {{{/sys/bus/i2c/drivers/tda1997x/2-0048/audmode}}}. If your output device requires a different sample-rate than the source input device, you need to perform a sample-rate conversion.


[=#multimedia]
= Multimedia (Video and Audio combined)
Handling media content that has both audio and video stream types requires a slightly more complex pipeline than the standard [#audio audio] and [#video video] pipeline examples.

Generally, a mixed media pipeline which consumes multimedia will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and {{{queue}}} elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data).

Conversely if producing multimedia content your pipeline will consist of a muxer to join audio and video streams and {{{queue}}} elements.


[=#named-elements]
== Named Elements, queues, and Multiple pipelines with gst-launch
When mixing audio and video elements with {{{gst-launch}}} one must make use of multiple pipelines using {{{named elements}}}.

The {{{name}}} property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any).

Multiple pipelines can be provided to {{{gst-launch}}} and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'.

This is best explained with some examples:
 * Encoding a stream with audio and video content into an AVI file:
{{{#!bash
gst-launch-1.0 \
  videotestsrc \
    ! $VIDEO_CAPABILITIES \
    ! mux. \
  audiotestsrc \
    ! $AUDIO_CAPABILITIES \
    ! mux. \
  avimux name=mux \
    ! filesink location=test.avi
}}}
  - The {{{videotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}}
  - The {{{audiotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}}
  - The {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}}
 * Decoding a Matroska container file containing H264 video and AC3 audio
{{{#!bash
gst-launch-1.0 \
  filesrc location=file.mkv \
    ! matroskademux name=demux \
  demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \
  demux. ! queue ! h264parse ! v4l2h264dec ! kmssink
}}}
  - The {{{filesrc}}} pipeline ends with {{{name=demux}}} which means the output of this pipeline will be sent to all pipelines with a {{{demux.}}} source (who's types have been successfully negotiated)
  - The audio pipeline consisting of the {{{ac3parse}}} element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3)
  - The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
  - Queue elements are used to keep one pipeline or element from blocking another. For example, if the {{{v4l2h264dec}}} element needs more data from the demux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering
 * Decoding a MOV file containing H264 video and AAC audio:
{{{#!bash
gst-launch-1.0 \
  filesrc location=file.mov \
    ! qtdemux name=demux \
  demux. ! queue ! aacparse ! avdec_aac ! alsasink \
  demux. ! queue ! h264parse ! v4l2h264dec ! kmssink
}}}


[=#mux]
== Muxing Mixed Content
Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'.

You can use {{{gst-inspect}}} to see a list of most of these using grep:
{{{#!bash
gst-inspect-1.0 | grep -i muxer | grep -vi de
}}}

Some common examples:
- mpegtsmux: MPEG Transport Stream Muxer
- mpegpsmux: MPEG Program Stream Muxer
- matroskamux: Matroska muxer
- avimux: Avi muxer
- qtmux: !QuickTime Muxer
- oggmux: Ogg muxer

To mux mixed content together include one of these elements following the audio and video pipelines.

Examples:
 * Encoding a stream with audio and video content into an AVI file:
{{{#!bash
gst-launch-1.0 \
  videotestsrc \
    ! $VIDEO_CAPABILITIES \
    ! mux. \
  audiotestsrc \
    ! $AUDIO_CAPABILITIES \
    ! mux. \
  avimux name=mux \
    ! filesink location=test.avi
}}}
  - the {{{videotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated.
  - the {{{audiotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated.
  - the {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} and it understands how to multiplex the two types of data together into its output which is written to the file test.avi



[=#demux]
== De-muxing mixed content
Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'.

You can use {{{gst-inspect}}} to see a list of most of these using grep:
{{{#!bash
gst-inspect-1.0 | grep -i 'de\?muxer'
}}}

Some common examples:
 - tsparse: MPEG transport stream parser
 - tsdemux: MPEG transport stream demuxer
 - matroskademux: Matroska demuxer
 - avidemux: Avi demuxer
 - qtdemux: !QuickTime demuxer
 - oggdemux: Ogg demuxer

To demux mixed content include one of these elements following the audio and video pipelines.

Unlike muxing you also need to use a {{{parser}}} element to parse the bitstream and break it into discrete buffers (frames) that the downstream decoder expects. In other words this 'frames' the data going to decoders.

Some common parsers:
 - ogmaudioparse: OGM audio stream parser
 - ogmvideoparse: OGM video stream parser
 - aacparse: AAC audio stream parser
 - amrparse: AMR audio stream parser
 - ac3parse: AC3 audio stream parser
 - flacparse: FLAC audio parser
 - mpegaudioparse: MPEG1 Audio Parser
 - h263parse: H.263 parser
 - h264parse: H.264 parser
 - mpegvideoparse: MPEG video elementary stream parser
 - mpeg4videoparse: MPEG 4 video elementary stream parser
 - pngparse: PNG parser
 - vc1parse: VC1 parser

Examples:
 * Demuxing a Matroska container file containing H264 video and AC3 audio into its raw components:
{{{#!bash
gst-launch-1.0 \
  filesrc location=file.mkv \
    ! matroskademux name=demux \
  demux. ! queue ! ac3parse ! filesink location=file.ac3 \
  demux. ! queue ! h264parse ! filesink location=file.h264
}}}
 * Demuxing a MOV file containing H264 video and AAC audio into its raw components:
{{{#!bash
gst-launch-1.0 \
  filesrc location=file.mov \
    ! qtdemux name=demux \
  demux. ! queue ! aacparse ! filesink location=file.aac \
  demux. ! queue ! h264parse ! filesink locatino=file.h264
}}}



[=#bin]
== bin elements
A '''bin''' element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept.

Sometimes the bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream. The {{{gst-launch}}} application provides a couple of useful debugging tools that can help with this:
 * using {{{GST_DEBUG_DUMP_DOT_DIR}}} and Graphviz
 * adding the '-v' parameter to gst-launch will provide verbose feedback on the pipeline configuration that can tell you what is going on

See [#troubleshooting] for more details on these methods.


[=#playbin]
=== GStreamer playbin
GStreamer {{{playbin}}} element attempts to create a pipeline that will play both the audio and video portions of a file. For example:
{{{#!bash
gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
}}}

The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by providing a {{{video-sink}}} and {{{audio-sink}}} property for example to decode audio to the Analog audio output on Ventana:
{{{#!bash
gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=sysdefault:CARD=sgtl5000audio"
}}}

Please type {{{gst-inspect-1.0 playbin}}} to see more options.


[=#decodebin]
=== GStreamer decodebin
The GStreamer {{{decodebin}}} element is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under [#ex1 the first example] with the following:
{{{#!bash
gst-launch-1.0 \
  filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \
  matroskademux name=d \
  d. ! queue ! ivorbisdec ! queue ! alsasink \
  d. ! queue ! decodebin  ! queue ! kmssink
}}}

Note that {{{decodebin}}} doesn't always choose the correct decoder, so be wary of this. It is similar to {{{playbin}}} in that it aids in creating a dynamic pipeline.


[=#gst-play]
=== GStreamer gst-play
The stand-alone application {{{gst-play}}} is is a program that utilizes the {{{playbin}}} element and thus can be used for playback of many file types. The above example {{{gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi}}} can be replaced with:
{{{#!bash
gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
}}}


[=#streaming]
= Streaming (send multimedia to or receive from Network)
GStreamer has elements that allow for network streaming to occur.

There are several ways to accomplish networked streaming over Internet Protocol (IP):
 * [#udp Raw UDP/IP]
 * [#tcp Raw TCP/IP]
 * [#rtp Real-time Transport Protocol (RTP)]
 * [#rtsp Real Time Streaming Protocol (RTSP)] ('''recommended''')
 * [#abs Adaptive Bitrate Streaming]


[=#udp]
== Raw UDP
Using UDP/IP is the simplest mechanism for streaming and utilizes the least amount of bandwidth. Because UDP does not provide any error detection, packet ordering, or error correction the bitrate is deterministic and simply the bitrate of the media you are streaming.

The limitations of raw UDP is:
 * requires codec that can handle missing/corrupt data (most do these days)
 * does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media)
 * does not fragment packets - will try to send a raw udp packet for whatever size buffer the udpsink is passed (which can lead to pipeline errors). To fragment packets use RTP

The only benefit of using raw UDP is that it is the simplest pipeline you can create for streaming and requires the least amount of dependencies (albeit you might run into one or all of the above problems).

'''Note that it is recommended that you use [#rtp RTP] or [#rtsp RTSP] unless you know exactly what you are doing to overcome the limitations listed above'''

The {{{udpsrc}}} element can be used to render/save a stream originated from a {{{udpsink}}} pipeline.

Examples:
 * encode and send H264 video from Ventana:
  1. Start decoder first:
 {{{
#!bash
ifconfig eth0 192.168.1.1
gst-launch-1.0 udpsrc port=9001 ! h264parse ! v4l2h264dec ! kmssink sync=false
}}}
  2. Start encoder second:
{{{
#!bash
ifconfig eth0 192.168.1.2
gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc ! udpsink host=192.168.1.1 port=9001
}}}

Notes:
 * On the client (stream receiver and renderer) you must use the {{{sync=false}}} property to render frames as they are received otherwise the stream will stall because their is no headers containing timestamps
 * the decoder (udpsrc) needs to be started first because udpsink will fail if nothing is listening to the socket


[=#tcp]
== TCP
Using TCP/IP brings error detection, packet re-ordering, and error correction to the network stream. This however causes the bitrate to be non-deterministic because as the error rate increases so does the bitrate and latency.

The limitations of using TCP:
 * non-deterministic bitrate
 * added latency
 * does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media)

'''Note that it is recommended that you use [#rtp RTP] or [#rtsp RTSP] unless you know exactly what you are doing to overcome the limitations listed above'''

TCP/IP introduces the concept of a socket connection therefore there must exist a server and a client in which case the server must be started first to listen for a connection. You can use a server sink or a server source. The {{{tcpserversrc}}} source can be used to create a TCP server that waits for a connection from a {{{tcpclientsink}}} to render/save. Alternatively the {{{tcpserversink}}} sink can be used to create a TCP server that waits for a connection from a {{{tcpclientsrc}}} that will send data.

Examples:
 * encode and send H264 video from Ventana with '''decoder as server''':
  1. Start decoder (server) first:
{{{
#!bash
ifconfig eth0 192.168.1.1
gst-launch-1.0 tcpserversrc host=0.0.0.0 port=9001 ! decodebin ! autovideosink sync=false
}}}
  2. Start encoder (client) second:
{{{
#!bash
ifconfig eth0 192.168.1.2
ENCODER=/dev/$(for i in $(ls -d /sys/class/video4linux/video*); do [ "coda-encoder" = "$(cat $i/name)" ] && basename $i; done)
gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! tcpclientsink host=192.168.1.1 port=9001
}}}

Notes:
 * TCP is connection oriented therefore the TCP 'server' must be started first. You can choose your elements such that the stream originator is the server or the stream renderer is the server however doing so can be problematic for certain codecs because the client decoding the stream may pick up the stream somewhere in the middle and not know how to parse it.
 * the {{{host=0.0.0.0}}} property means listen to all network interfaces


[=#rtp]
== RTP (raw/session-less)
The [https://en.wikipedia.org/wiki/Real-time_Transport_Protocol Real-time Transport Protocol (RTP)] is a network protocol for delivering audio and video over IP networks. RTP is used extensively in communication and entertainment systems that involve streaming media, such as telephony, video teleconference applications, television services and web-based push-to-talk features.

The RTP packet type encapsulates multimedia data with a payload type and time-stamp and therefore can be used to compensate for jitter, out of sequence packets, and time synchronization between streams of different types (ie audio/video lip-sync).

RTP is typically used in conjunction with other protocols such as RTP Control Protocol (RTCP) and [#rtsp Real Time Streaming Protocol (RTSP)] to manage stream sessions however can be used on its own in a raw session-less fashion using {{{udpsink}}} and {{{udpsrc}}} elements.

The limitations of using raw/session-less RTP:
 * session management needs to be handled manually (capsfilter is needed to specify stream format)

'''Note that it is recommended that you use [#rtsp RTSP] unless you know exactly what you are doing to overcome the limitations listed above'''

=== Example: Capture, encode and stream H264 via RTP with GStreamer playback:
 * Encode and send H264 video from Ventana:
  1. Start decoder first:
{{{
#!bash
ifconfig eth0 192.168.1.1
gst-launch-1.0 udpsrc port=9001 \
 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" \
 ! decodebin ! autovideosink
}}}
  2. Start encoder second:
{{{
#!bash
gst-launch-1.0 videotestsrc is-live=true \
 ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! rtph264pay ! udpsink host=192.168.1.1 port=9001
}}}

Notes:
 * when using RTP a capsfilter must be used to specify the payload as application/x-rtp as above. You can determine the capsfilter required by starting the encoder with a verbose flag {{{-v}}} and looking for  {{{caps = "application/x-rtp"}}}


=== Example: Capture, encode and stream H264 via RTP with VLC playback:
* Encode and send H264 video from Ventana to a PC with VLC:
1. Start decoder first:
 a. Create SDP file like below (IP address in example is that of the Ventana board)
{{{
v=0
m=video 5000 RTP/AVP 96
c=IN IP4 172.24.20.207
a=rtpmap:96 H264/90000
}}}
 b. Open SDP file in VLC
2. Start encoder (Ventana) second: (IP address in below example is IP of the PC)
{{{
gst-launch-1.0 videotestsrc ! v4l2h264enc ! rtph264pay config-interval=3 ! udpsink host=172.24.20.26 port=5000
}}}


[=#rtsp]
== RTSP (Real Time Streaming Protocol) '''(recommended)''' 
The [https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol Real Time Streaming Protocol (RTSP)] is a network control protocol designed for use in entertainment and communications systems to control streaming media servers. The protocol is used for establishing and controlling media sessions between end points. Clients of media servers issue VCR-style commands, such as play and pause, to facilitate real-time control of playback of media files from the server. This protocol uses the Real-time Transport Protocol (RTP) in conjunction with Real-time Control Protocol (RTCP) for media stream delivery.

The limitations of using RTSP are:
 * {{{gst-inspect}}} has no way of using a simple pipeline to create an RTSP server - you must create or use an existing gstreamer based application (keep reading below).

Creating an RTSP server is not possible via a simple pipeline to gst-launch however GStreamer libraries do exist to make writing an RTSP server trivial. The source for gst-rtsp-server contains an example application [http://cgit.freedesktop.org/gstreamer/gst-rtsp-server/tree/examples/test-launch.c test-launch.c] which provides a simple example that can take a GStreamer 'bin' element consisting of everything but the sink element and serves it via RTSP.

Gateworks has extended this with [#gst-variable-rtsp-server gst-variable-rtsp-server] which demonstrates how to auto-adjusting properties like the encoding bitrate depending on the number of clients connected in addition to serving as a fairly simple example of how to write a gstreamer application.


[=#gst-variable-rtsp-server]
=== gst-variable-rtsp-server
[https://github.com/Gateworks/gst-gateworks-apps/blob/master/src/gst-variable-rtsp-server.c gst-variable-rtsp-server] is a solution created by Gateworks that allows changing the quality of a live stream on the fly based on the number of clients connected to the server. Similar to other adaptive bitrate technologies, it can change quality based on information is knows about the stream. In our case, the number of connected clients was deemed as appropriate.

This application can change the {{{video_bitrate}}} control of the {{{v4l2h264enc}}} element based on the {{{--steps}}} input (default to 5). For example, if using the default {{{steps}}} value of 5, if the min bitrate was 500 and max bitrate was 2000, it would take 5 clients to get from the best to the worst quality.

Example:
 * Start a RTSP server using videotestsrc on default port of 9099 serving H264:
{{{
#!bash
gst-variable-rtsp-server "videotestsrc ! v4l2h264enc ! rtph264pay name=pay0 pt=96"
}}}

To connect as a client, you can use any program that can connect to an RTSP stream. Below is an example connecting with GStreamer:
{{{
#!bash
gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! decodebin ! autovideosink
}}}

With the above, if there is only a single client, the stream will be at it's best quality. But as you connect more clients, the quality of the stream will drop in order to account for network congestion.


[=#rtmp]
== RTMP Streaming
Real-Time Messaging Protocol (RTMP) is a protocol for streaming audio, video and data over the internet. It was originally developed by Macromedia and proprietary but has since been released for public use. It is a TCP-based protocol which maintains persistent connections.

Using RTMP requires you have:
 * an RTMP server (ie Youtube or Nginx)
 * a stream transmitter (ie Gateworks board)
 * a stream receiver (ie Web browser or Gateworks board)

[=#youtube]
=== !YouTube Live Streaming
It is possible to stream video to !​YouTube Live Streaming from the Gateworks board using GStreamer's rtmpsink element.

To create a !YouTube live feed via RTMP:
 - goto !YouTube Live Control Room (https://www.youtube.com/livestreaming/)
 - goto http://youtube.com and login to your account
 - click the Camera icon on the top left to add a stream and select 'Go Live'
 - select Stream at the top
 - create a title and specify your !YouTube attributes (ie Public, etc etc) then hit 'Create Stream'
 - take note of your 'Stream key' needed for the encoder pipeline
 - start your gstreamer pipeline to capture/encode/stream via {{{rtpmsink}}} to the youtube 'Streamer URL' (see [#rtmpsink below])
 - once YouTube says your stream is ready (indicating its receiving data) hit the 'GO LIVE' button at the top left
 - Now you can hit the 'Share' icon at the top left to get a URL to share to view the stream

 
[=#rtmpsink]
=== GStreamer rtmpsink transmitter
Once you have an RTMP server you can use GStreamer to stream live content to it with the {{{rtmpsink}}} element.

Examples:
 * color-bar pattern from videotestsrc and audio from audiotestsrc
{{{#!bash
gst-launch-1.0 \
  videotestsrc ! "video/x-raw,width=640,height=360,framerate=15/1" ! \
  v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=620000" ! \
  h264parse ! \
  mux. \
audiotestsrc is-live=true ! "audio/x-raw,format=S16LE,endianness=1234, signed=true,width=16,depth=16,rate=44100,channels=2" ! \
  voaacenc bitrate=128000 ! \
  aacparse ! "audio/mpeg,mpegversion=4,stream-format=raw" ! \
  mux. \
flvmux streamable=true name=mux ! queue ! \
 rtmpsink location="rtmp://a.rtmp.youtube.com/live2/ee12-bsgt-e22v-8qr1"
}}}
 * captured video/audio:
{{{#!bash
gst-launch-1.0 \
  v4l2src device=$DEVICE ! \
  $GST_CONVERT output-io-mode=dmabuf-import ! \
  v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=620000" ! \
  h264parse ! \
  mux. \
alsasrc device="sysdefault:CARD=sgtl5000audio" ! \
  voaacenc bitrate=128000 ! \
  aacparse ! "audio/mpeg,mpegversion=4,stream-format=raw" ! \
  mux. \
flvmux streamable=true name=mux ! queue ! \
 rtmpsink location="rtmp://a.rtmp.youtube.com/live2/ee12-bsgt-e22v-8qr1"
}}}

Note that the {{{location}}} property of the {{{rtmpsink}}} is configured on your server. For !YouTube for example they provide the URL and you need to append the 'Stream Key' to that path


[=#troubleshooting]
= Troubleshooting
If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device.


[=#GST_DEBUG]
== gst-inspect and GST_DEBUG
You can see all the properties of a particular element with the 'gst-inspect <elementname>' command. You can see a list of all elements with 'gst-inspect' without arguments.

You can enable verbose output for GStreamer by adding the '-v' flag which will show you the negotiated pipeline details and the state machine details.

You can enable debug output for GStreamer by using the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-debug.html '--gst-debug'] parameter to gst-launch or by setting the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html GST_DEBUG]  environment variable.

For example, if you wanted to see DEBUG level messages for the {{{videotestsrc}}} element you can set GST_DEBUG=videotestsrc:4

== Colorspace and Colorimetry compatibility
If you see errors such as 'Device wants bt709 colorimetry' you need to use the v4l2convert ($GST_CONVERT) element to convert your colorspace between elements. This is often needed between v4l2src capture devices and various CODA elements such as v4l2h264enc and vr4l2jpegenc.

For example:
{{{#!bash
gst-launch-1.0 v4l2src device=$DEVICE ! \
     $GST_CONVERT output-io-mode=dmabuf-import ! \
     video/x-raw,colorimetry=bt709 ! \
     v4l2h264enc output-io-mode=dmabuf-import ! \
     rtph264pay ! udpsink host=$SERVER port=$PORT
}}}

== insufficient CMA memory when using CODA
When using the CODA element ($GST_CONVERT) to perform video compression or decompression of high resolution images you may not have enough CMA memory allocated. By default the kernel allocates 64MiB but you can set this via the kernel command line by adding a 'cma=128M' for example to double it.

Insufficient CMA memory for coda will show a 'Failed to allocate required memory' with kernel messages like:
{{{#!bash
cma: cma_alloc: alloc failed, req-size: 512 pages, ret: -12
coda 2040000.vpu: dma_alloc_coherent of size 2097152 failed
}}}


[=#filter-graph]
== GST_DEBUG_DUMP_DOT_DIR
You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause {{{gst-launch}}} to output a {{{.dot}}} file for each phase of the pipeline then use a tool such as {{{Graphviz}}} to visualize the {{{.dot}}} file.

Example:
 * use playbin to playback a file:
{{{#!bash
root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
}}}
  - hit Cntl-C after decoding starts to exit early
 * see the dot files created:
{{{#!bash
root@ventana:~# ls /tmp/dot
0.00.00.108710334-gst-launch.NULL_READY.dot
0.00.00.490807334-gst-launch.READY_PAUSED.dot
0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot
0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
0.00.03.254000001-gst-launch.PAUSED_READY.dot
}}}
 * transfer to a PC and use something like {{{xdot}}} to view:
{{{#!bash
xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
}}}
  - zoom in along the graph and you can see that:
   - {{{GstFileSrc}}} is the source,
   - {{{GstAviDemux}}} is used to demux to audio/x-ac3,
   - {{{GstAc3Parse}}} is used to parse the audio into audio frames,
   - {{{GstMpeg4VParse}}} is used to parse the video into video frames,
   - {{{GstV4l2VideoDec}}} is used to decode the video from video/mpeg to video/x-raw,
   - {{{ GstA52Dec}}} is used to decode the audio from audio/x-ac3 to audio/x-raw,
   - etc
  - Note that some hunting with {{{gst-inspect}}} must be done to determine what elements coorespond to the above class names

Reference:
 - http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+11%3A+Debugging+tools

== gst-launch -v
The verbose debugging from {{{gst-launch -v}}} can show you the negotiation that takes place as a pipeline moves through its stages.

Example:
{{{#!bash
gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
}}}

examining the verbose output can show you the following:
 * container: AVI: avidemux
 * video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! v4l2mpeg4dec
 * audio: AC3 48khz 5channels: ac3parse ! a52dec

Therefore you can use these pipelines to decode and play:
 * video only (output to kmssink)
{{{#!bash
gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink
}}}
 * audio only (output to hdmi audio sink)
{{{#!bash
gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
}}}
 * both audio and video
{{{#!bash
gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \
  d. ! queue ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink \
  d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
}}}


== streaming
If you're having issues with network streaming:
* Verify that both sides can ping one another
* If the message {{{There may be a timestamping problem, or this computer is too slow}}} appears and the video display appears choppy, try the following:
 * Lower the bitrate from the server
 * Place a {{{sync=false}}} on the sink side of the server and client.
* If video appears choppy, try using UDP over TCP.
* Verify that the network is not congested.
* Verify your gstreamer pipeline is correct. The best way to find the element that causes a negotiation failure is to end your pipeline in a fakesink and one-by-one eliminate elements leading up to it until it negotiates successfully.