Version 8 (modified by 5 years ago) ( diff ) | ,
---|
- GStreamer on Gateworks SBCs
- Video4Linux2 Devices
- Test Media Files
- GStreamer
- GStreamer Video
- GStreamer Audio
- Multimedia (Video and Audio combined)
- Streaming (send multimedia to or receive from Network)
- Troubleshooting
GStreamer on Gateworks SBCs
The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs), specifically the Ventana family that utilizes the Freescale i.MX6 processors.
Gateworks SBCs can be viewed at the following link: http://www.gateworks.com
Important Note
The IMX6 (Ventana family) software has been evolving over the years. This page is focused on mainline Linux kernels and not the Freescale GStreamer gstreamer-imx
plugin legacy software that used the 3.14 Freescale Vendor kernel. If using the Gateworks 3.14 kernel with the Freescale capture drivers please see Yocto/gstreamer
Also note that GStreamer capture devices are /dev/videoN
which require some configuration if using Processor specific video input busses such as the IMX6 capture using Analog (CVBS) or Digital video (HDMI). If using these capture interfaces please see linux/media for details on how to use media-ctl
to configure those devices.
Video4Linux2 Devices
The Video4Linux2 API is the device driver API used for kernel capture drivers as well as video encode and video decode drivers.
Media Control API
The MediaControl API is used to configure Video4Linux2 capture devices. See linux/media for more info.
V4L2 Controls (brightness, contrast, saturation, rotation, flip, bitrate etc)
When capturing video from a video4linux2 device you can use the v4l2 API to get and set various controls the device may provide access to such as:
- brightness
- contrast
- hue
- saturation
- horizontal flip
- veritcal flip
- rotation
What controls are available depend on the device. Furthermore when using capture devices that have media-ctl pipelines controls provided by various elements within the pipeline are passed on to the capture device.
Examples:
- list available controls of a capture device:
media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $DEVICE --list-ctrls # list available controls
- get brightness control for adv7180 analog video capture device:
media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $DEVICE --get-ctrl=brightness
- set brightness control for adv7180 analog video capture device:
media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $DEVICE --set-ctrl=brightness=50
- set video bitrate of the v4l2h264enc H264 encoder element:
media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $ENCODER --set-ctrl=video_bitrate=10000000
Note that we 'source' the setup file above instead of executing it. This is because it sets some env variables that we will use throughout our examples below. If you are using a shell that does not have 'source' the notation is '. ./setup'
Test Media Files
There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing:
- Big Buck Bunny - Creative Commons
- Sintel - Creative Commons
- Tears of Steel - Creative Commons
GStreamer
GStreamer is an opensource library and framework created to handle multimedia. Pipelines are used to source and sink media however you would like (e.g. decoding the mp3 out of a video file and playing it back through speakers).
It is important to understand that GStreamer is foremost a library meant for developing applications. While it has an extremely useful test application called gst-launch
meant for testing pipelines using that application alone can't solve all problems. There are plenty of resources online including mailing lists and IRC channels that can be helpful in understanding GStreamer and writing applications that use it.
To install GStreamer on Ubuntu:
apt-get install gstreamer1.0-x gstreamer1.0-tools libgstreamer1.0-0 \ gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \ gstreamer1.0-libav gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio \ v4l-utils alsa-utils
Plugins and Elements
GStreamer works on 'plugins' that provide pipeline 'elements'. A plugin comprises of elements that can do work on a media stream. For example, the mpeg2dec
can decode mpeg1 and mpeg2 video streams (software based decoding, not hardware accelerated).
Pipeline Construction
gst-launch is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type gst-inspect element
where element
is the element you are looking for. To see a full listing of elements installed, you can type just gst-inspect
.
Here is a graphical example of a pipeline.
The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next
gstreamer.com contains much more information on the construction of pipelines.
Caps Filters
GStreamer has a concept called caps filters. A 'cap' (short for capability) is used to describe the type of data that links a src (output) pad from one element to a sink (input) pad of another element.
Adding a -v
flag to gst-launch-1.0
will output the capabilities negotiated between elements:
~# gst-launch-1.0 -v videotestsrc ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock
In the output, the caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
are the video caps negotiated between the two.
You can force a caps filter between two elements by treating it as a pipeline element:
~# gst-launch-1.0 -v videotestsrc ! 'video/x-raw, format=UYVY, width=1920, height=1080, framerate=10/1' ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock
As you can see, the caps filter above changed both the video format, resolution, and framerate of the video stream coming out of the videotestsrc
element.
Caps filters are useful when you want to use a specific format where a device may support multiple. For example when capturing audio from an ALSA sound device that supports a variety of samplerates, channels, and bit formats you can use a capsfilter to override any default configuration.
Applications
GStreamer is a library and framework for multimedia processing. It is not a user-space application. It is easy to forget this because of the usefulness of the gst-launch
application that can be used to connect simple pipelines together for testing. For your needs you very likely need to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that.
GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see here for more details.
Furthermore, Gateworks has created a gst-variable-rtsp-server example application that demonstrates how to server simple GStreamer pipelines via an RTSP server and automatically adjust compression quality based on client connections.
fakesink
The fakesink
element is a very useful video sink when debugging or learning GStreamer. It takes whatever frames is given to it and drops them. This might help debugging pipelines if problems ever arise.
Example:
gst-launch-1.0 videotestsrc ! fakesink
gst-inspect
The gst-inspect-1.0
tool is an excellent tool if you're unfamiliar with the functionality, source/sink formats, or properties of any Gstreamer element.
Usage:
root@bionic-armhf:~# gst-inspect-1.0 videotestsrc Factory Details: Rank none (0) Long-name Video test source Klass Source/Video Description Creates a test video stream Author David A. Schleef <ds@schleef.org> Plugin Details: Name videotestsrc Description Creates a test video stream Filename /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstvideotestsrc.so Version 1.14.5 License LGPL Source module gst-plugins-base Source release date 2019-05-29 Binary package GStreamer Base Plugins (Ubuntu) Origin URL https://launchpad.net/distros/ubuntu/+source/gst-plugins-base1.0 GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstBaseSrc +----GstPushSrc +----GstVideoTestSrc Pad Templates: SRC template: 'src' Availability: Always Capabilities: video/x-raw format: { (string)I420, (string)YV12, (string)YUY2, (string)UYVY, (string)AYUV, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGB, (string)BGR, (string)Y41B, (string)Y42B, (string)YVYU, (string)Y444, (string)v210, (string)v216, (string)NV12, (string)NV21, (string)GRAY8, (string)GRAY16_BE, (string)GRAY16_LE, (string)v308, (string)RGB16, (string)BGR16, (string)RGB15, (string)BGR15, (string)UYVP, (string)A420, (string)RGB8P, (string)YUV9, (string)YVU9, (string)IYU1, (string)ARGB64, (string)AYUV64, (string)r210, (string)I420_10BE, (string)I420_10LE, (string)I422_10BE, (string)I422_10LE, (string)Y444_10BE, (string)Y444_10LE, (string)GBR, (string)GBR_10BE, (string)GBR_10LE, (string)NV16, (string)NV24, (string)NV12_64Z32, (string)A420_10BE, (string)A420_10LE, (string)A422_10BE, (string)A422_10LE, (string)A444_10BE, (string)A444_10LE, (string)NV61, (string)P010_10BE, (string)P010_10LE, (string)IYU2, (string)VYUY, (string)GBRA, (string)GBRA_10BE, (string)GBRA_10LE, (string)GBR_12BE, (string)GBR_12LE, (string)GBRA_12BE, (string)GBRA_12LE, (string)I420_12BE, (string)I420_12LE, (string)I422_12BE, (string)I422_12LE, (string)Y444_12BE, (string)Y444_12LE, (string)GRAY10_LE32, (string)NV12_10LE32, (string)NV16_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] multiview-mode: { (string)mono, (string)left, (string)right } video/x-bayer format: { (string)bggr, (string)rggb, (string)grbg, (string)gbrg } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] multiview-mode: { (string)mono, (string)left, (string)right } Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SRC: 'src' Pad Template: 'src' Element Properties: name : The name of the object flags: readable, writable String. Default: "videotestsrc0" parent : The parent of the object flags: readable, writable Object of type "GstObject" blocksize : Size in bytes to read per buffer (-1 = default) flags: readable, writable Unsigned Integer. Range: 0 - 4294967295 Default: 4096 num-buffers : Number of buffers to output before sending EOS (-1 = unlimited) flags: readable, writable Integer. Range: -1 - 2147483647 Default: -1 typefind : Run typefind before negotiating (deprecated, non-functional) flags: readable, writable, deprecated Boolean. Default: false do-timestamp : Apply current stream time to buffers flags: readable, writable Boolean. Default: false pattern : Type of test pattern to generate flags: readable, writable Enum "GstVideoTestSrcPattern" Default: 0, "smpte" (0): smpte - SMPTE 100% color bars (1): snow - Random (television snow) (2): black - 100% Black (3): white - 100% White (4): red - Red (5): green - Green (6): blue - Blue (7): checkers-1 - Checkers 1px (8): checkers-2 - Checkers 2px (9): checkers-4 - Checkers 4px (10): checkers-8 - Checkers 8px (11): circular - Circular (12): blink - Blink (13): smpte75 - SMPTE 75% color bars (14): zone-plate - Zone plate (15): gamut - Gamut checkers (16): chroma-zone-plate - Chroma zone plate (17): solid-color - Solid color (18): ball - Moving ball (19): smpte100 - SMPTE 100% color bars (20): bar - Bar (21): pinwheel - Pinwheel (22): spokes - Spokes (23): gradient - Gradient (24): colors - Colors timestamp-offset : An offset added to timestamps set on buffers (in ns) flags: readable, writable Integer64. Range: 0 - 2147483646999999999 Default: 0 is-live : Whether to act as a live source flags: readable, writable Boolean. Default: false k0 : Zoneplate zero order phase, for generating plain fields or phase offsets flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kx : Zoneplate 1st order x phase, for generating constant horizontal frequencies flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 ky : Zoneplate 1st order y phase, for generating contant vertical frequencies flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kt : Zoneplate 1st order t phase, for generating phase rotation as a function of time flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kxt : Zoneplate x*t product phase, normalised to kxy/256 cycles per vertical pixel at width/2 from origin flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kyt : Zoneplate y*t product phase flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kxy : Zoneplate x*y product phase flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kx2 : Zoneplate 2nd order x phase, normalised to kx2/256 cycles per horizontal pixel at width/2 from origin flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 ky2 : Zoneplate 2nd order y phase, normailsed to ky2/256 cycles per vertical pixel at height/2 from origin flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kt2 : Zoneplate 2nd order t phase, t*t/256 cycles per picture flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 xoffset : Zoneplate 2nd order products x offset flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 yoffset : Zoneplate 2nd order products y offset flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 foreground-color : Foreground color to use (big-endian ARGB) flags: readable, writable, controllable Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 background-color : Background color to use (big-endian ARGB) flags: readable, writable, controllable Unsigned Integer. Range: 0 - 4294967295 Default: 4278190080 horizontal-speed : Scroll image number of pixels per frame (positive is scroll to the left) flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 animation-mode : For pattern=ball, which counter defines the position of the ball. flags: readable, writable Enum "GstVideoTestSrcAnimationMode" Default: 0, "frames" (0): frames - frame count (1): wall-time - wall clock time (2): running-time - running time motion : For pattern=ball, what motion the ball does flags: readable, writable Enum "GstVideoTestSrcMotionType" Default: 0, "wavy" (0): wavy - Ball waves back and forth, up and down (1): sweep - 1 revolution per second (2): hsweep - 1/2 revolution per second, then reset to top flip : For pattern=ball, invert colors every second. flags: readable, writable Boolean. Default: false
GStreamer Video
This section will introduce several concepts regarding how video media is handled by GStreamer and provide several example pipelines.
Video Output
Generally, a GStreamer 'sink' element is one that will take a video stream and output it to a display.
Some commonly used video output sinks:
- autovideosink: Wrapper video sink for automatically detecting video output device
- kmssink: KMS video device sink
- fbdevsink: Framebuffer video device sink
Execute a gst-inspect-1.0 | grep sink
to see a complete list of sinks available and gst-inspect-1.0 <element>
to see specific details about the applicable formats and properties of a specific element.
Important Note: Video display drivers for use with video display sinks such as kmssink and fbdevsink for HDMI out and LVDS require proper kernel commandline parameters to configure them. Sometimes the Bootloader bootscript will do this for you - Please see linux/display for details and check the kernel command-line after booting to make sure the desired device is enabled and configured for the proper resolution and mode.
Examples:
- HDMI enabled and configured for 1080p60Hz LVDS disabled:
~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done video=HDMI-A-1:1920x1080M@60 video=LVDS-1:d
- HDMI disabled and LVDS configured for 1280x800 60Hz
~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done video=LVDS-1:1280x800@60M video=HDMI-A-1:d
autovideosink
This GStreamer sink is not really a 'video' sink in the traditional sense. Similar to playbin
and decodebin
, this element selects what it thinks is the best available video sink and uses it.
This will typically use kmssink
unless format choices require one of the other sinks.
You can add a verbose flag gst-launch-1.0 -v
to see details about the elements and caps chosen when using any type of 'auto' element or 'bin' element.
Example:
gst-launch-1.0 videotestsrc ! autovideosink
- Note that adding a '-v' flag for verbose mode will show the element configuration to let you know what sink was actually chosen
kmssink
This sink allows you to directly output to a KMS (kernelmode setting API) driver. This is the most efficient video output method.
Example:
gst-launch-1.0 videotestsrc ! kmssink
The Framebuffer device sink allows you to directly output to a Linux Framebuffer device if you have one (ls /sys/class/graphics/fb*).
Example:
gst-launch-1.0 videotestsrc ! fbdevsink
Video Input (Capture)
A video input source is anything coming from an input on the device, e.g. HDMI input/USB Web Cam. This is referred to as Video Capture. In GStreamer terms a video capture device is a 'source' or 'src' element.
Please refer to the linux/media page for details on the video in devices on the Ventana platform.
Some commonly used GStreamer video input sources:
- videotestsrc
- v4l2src
Execute a gst-inspect-1.0 | grep src
to see a complete list of source elements available and gst-inspect-1.0 <element>
to see specific details about the applicable formats and properties of a specific element.
If the is-live
property is set to true this will cause buffers to be discarded on a pipeline paused state and pipelines will not participate in the PREROLL phase of a pipeline.
Important Note: Linux video capture devices provided by CPU's such as the IMX6 Analog (CVBS) and Digital (HDMI) inputs provided on Ventana boards require a hardware pipeline configuration. See linux/media for details on how to configure the video capture device pipeline.
videotestsrc
This is a very useful element for testing pipelines as it creates video content. It can output a huge number of video formats for raw video video/x-raw
and bayer video video/x-bayer
. Use gst-inspect-1.0 videotestsrc
to see its output formats and properties such as video test pattern, orientation, colrs, etc.
Examples:
- display SMPTE pattern to connected and configured display
gst-launch-1.0 videotestsrc ! kmssink
- set the 'pattern' property to display a moving ball:
gst-launch-1.0 videotestsrc pattern=18 ! kmssink
- Use a 'capsfilter' to force output of videotestsrc to a specific resolution, format, and framerate (ie 1080p 10fps YUV):
gst-launch-1.0 videotestsrc ! 'video/x-raw,format=UYVY,width=1920,height=1080,framerate=10/1' ! kmssink
v4l2src
This element uses the video4linux2 API to capture video from input sources (/dev/video<n>).
Note that v4l2src
is always live regardless of the is-live property.
Note that you need to configure the media-ctl pipeline that feeds the /dev/video<n> devices - see linux/media for details
You can use the v4l2-ctl
application from the v4l-utils
package to to interact with the device to get/set various capabilities and controls.
Examples:
- show all video4linux2 devices:
root@bionic-armhf:~# v4l2-ctl --list-devices CODA960 (platform:coda): /dev/video8 /dev/video9 imx-media-mem2mem (platform:imx-media-mem2mem): /dev/video10 imx-media-capture (platform:ipu1_csi0): /dev/video4 imx-media-capture (platform:ipu1_csi1): /dev/video5 imx-media-capture (platform:ipu1_ic_prpenc): /dev/video0 imx-media-capture (platform:ipu1_ic_prpvf): /dev/video1 imx-media-capture (platform:ipu2_csi0): /dev/video6 imx-media-capture (platform:ipu2_csi1): /dev/video7 imx-media-capture (platform:ipu2_ic_prpenc): /dev/video2 imx-media-capture (platform:ipu2_ic_prpvf): /dev/video3
- display all details about /dev/video1:
v4l2-ctl -d /dev/video1 --all
- Use Gateworks
media-ctl-setup
to configure the media-ctl pipeline for Analog Video capture using the adv7180 chip. If you don't already have this on your root filesystem you can install it via:wget https://raw.githubusercontent.com/Gateworks/media-ctl-setup/master/media-ctl-setup -O /usr/local/bin/media-ctl-setup chmod +x /usr/local/bin/media-ctl-setup media-ctl-setup adv7180 > setup . ./setup
- you can look at the setup script to see what was done - the details vary between SoC (IMX6DL vs IMX6Q), board, and sensor (adv7180 for Analog video capture and tda1997x for digital video capture)
- note that the setup script exports the video capture device to the DEVICE env variable which you can use as shown below
- After sourcing setup above you can capture and display to the connected and configured output display (video loopback):
gst-launch-1.0 v4l2src device=$DEVICE ! kmssink
Colorspace Converting and/or Video Scaling (via IMX6 IPU mem2mem driver)
Often times, a colorspace conversion or scaling is required in order to link GStreamer elements together. This is often due to the fact that not all elements can every format available.
A Linux V4L2 MEM2MEM imx-media driver exists that allows utilizing the IMX6 IPU Image Converter hardware blocks (IC) to perform hardware colorspace conversion (CSC), scaling, cropping, rotation, and flip operations.
The GStreamer video4linux2
element provides an element that uses this driver to expose these capabilities to GStreamer applications.
Notes:
- For GStreamer-1.14 the name of the element depends on the video device the driver registers with the kernel (ie v4l2video8convert if mem2mem driver registers /dev/video8) but for GStreamer-1.16 and beyond the name of the element is stable and always 'v4l2videoconvert'. For the examples on this page we use the GST_CONVERT env variable that is set from the output of the media-ctl-setup script to avoid confusion.
- the
kmssink
examples below need acan-scale=false
property to tell GStreamer not to scale via the KMS driver (as the IMX6 KMS driver does not support scaling) - ensure that the input format differs from the output format otherwise GStreamer will bypass the conversion completely; note that GStreamer doesn't understand flipping or rotation as part of the format. Gstreamer master (in development) adds a 'disable-passthrough' property to the v4l2videoconvert entity that can be set to force the conversion regardless of input and output format
- when using imx entities (ie capture, encode/decode, mem2mem, display) you can specify 'output-io-mode=dmabuf-import' to share dmabuf pointers for a zero-copy pipeline however if using non imx entities (ie videotestsrc) you must omit these as you can not ensure the buffers share the alignment/stride necessary to share dmabuf pointers
Examples:
- Ensure mem2mem is in your kernel:
~# dmesg | grep mem2mem [ 18.356023] imx-media: Registered ipu_ic_pp mem2mem as /dev/video8
- Ensure GStreamer element exists:
~# gst-inspect-1.0 | grep -e "v4l2.*convert" video4linux2: v4l2convert: V4L2 Video Converter
- Obtain the name of the element (as it can vary between GStreaqmer-1.14 and GStreamer 1.16)
GST_CONVERT=$(gst-inspect-1.0 | grep -e "v4l2.*convert*" | sed -e 's/.*:\s*\(v4l2.*convert\):.*/\1/')
- scale/rotate/flip using
videotestsrc
(can not use dmabufs for this as it is a non-imx entity)# upscale gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT ! \ video/x-raw,width=640,height=480 ! kmssink can-scale=false # downscale gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480 ! \ $GST_CONVERT ! \ video/x-raw,width=320,height=240 ! kmssink can-scale=false # rotate gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT extra-controls=cid,rotate=90 ! \ video/x-raw,width=240,height=320 ! kmssink can-scale=false # hflip gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT extra-controls=cid,horizontal_flip=1 ! \ video/x-raw,width=640,height=480 ! kmssink can-scale=false # vflip gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT extra-controls=cid,vertical_flip=1 ! \ video/x-raw,width=640,height=480 ! kmssink can-scale=false
- note the above examples force the input format (resolution in this case) to differ from the output format otherwise gstreamer will bypass the v4l2convert entity thinking it not necessary as gstreamer does not understand the flip/rotation properties. GStreamer master (in development) adds the 'disable-passthrough' property which can be enabled to force disabling passthrough
- scale/rotate/flip using imx-media capture device and KMS display driver (can use dmabufs for this as they are all imx hardware entitites):
# scale sensor input to 720p display gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ video/x-raw,width=1280,height=720 ! \ kmssink can-scale=false # scale sensor input to 1080p display gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ video/x-raw,width=1920,height=1080 ! \ kmssink can-scale=false # scale/flip gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \ video/x-raw,width=1920,height=1080 ! \ kmssink can-scale=false # scale/rotate gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \ video/x-raw,width=720,height=1280 ! \ kmssink can-scale=false
- capture, scale, rotate, flip and encode using imx-media capture device mem2mem device and coda device (can use dmabufs for zero-copy)
# encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # scale/encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ video/x-raw,width=1440,height=960 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # scale/flip/encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \ video/x-raw,width=1440,height=960 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # scale/rotate/encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \ video/x-raw,width=1440,height=960 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT
For anything not able to be handled by the video4linux2 element and any mem2mem drivers there are software based elements that can convert pixel colorspace and/or scale however note these are extremely CPU intensive:
- autovideoconvert
- videoconvert
- rgb2bayer
Interlaced Video and Deinterlacing
Interlaced video is a technique for doubling the perceived frame rate of a video display without consuming extra bandwidth. The interlaced signal contains two fields of a video frame captured at two different times. The alternative to interlaced video is called progressive video. While reducing bandwidth this can cause a perceived flicker effect as well as a very apparent artifact seen during motion. For example a car moving horizontally across a scene will show every other vertical line differently: the second field interlaced with the first field will be a full frame period ahead in time from the other. The visual affect can be seen in this image from Wikipedia
Television signals are typically interlaced, or at least were until recently. For example, analog television standards such as NTSC used in North America as well as the PAL and SECAM formats used abroad use interlaced video and therefore any analog video decoder such as the ADV7180 found on many Gateworks Ventana boards will capture interlaced video and are subject to interlacing artifacts. Interlaced video is still used in High Definition signals as well and the letter at the end of the format tells you if its interlaced (ie 480i, 720i, 1080i) or progressive (ie 480p, 720p, 1080p).
To deinterlace video captured from the adv7180 analog decoder you need to use media-ctl to configure your pipeline to include the IMX6 Video Deinterlacer block. This is done for you if you use the Gateworks media-ctl-setup
script to configure your pipeline. See linux/media for more info.
References:
- Wikipedia interlaced video - includes several images demonstrating interlacing artifacts
- Wikipedia deinterlacing
Video Encoding (via IMX6 coda driver and video4linux2 element)
Encoding video means taking the raw captured pixels and converting them to a standard compressed format.
The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer video4linux2
element provides encode/decode elements that tap into this.
Notes:
- Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded:
~# dmesg | grep coda [ 16.721698] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2 [ 16.721724] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin [ 18.381136] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin [ 18.433648] coda 2040000.vpu: Firmware code revision: 570363 [ 18.433683] coda 2040000.vpu: Initialized CODA960. [ 18.433706] coda 2040000.vpu: Firmware version: 3.1.1 [ 18.442312] coda 2040000.vpu: codec registered as /dev/video[9-10] ~# cat /sys/class/video4linux/video9/name coda-encoder ~# cat /sys/class/video4linux/video10/name coda-decoder
- Ensure GStreamer encode elements exists:
~# gst-inspect-1.0 | grep -e "v4l2.*enc" video4linux2: v4l2h264enc: V4L2 H.264 Encoder video4linux2: v4l2mpeg4enc: V4L2 MPEG4 Encoder
- The CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry
- The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough)
- Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the v4l2-ctl -L and v4l2-ctl -l parameters on the devices exposed by the coda driver
- Often you will need to use the mem2mem v4l2convert driver for format conversions (colorspace, colorimetry) to reconcile the format of the source device and the encoder element - notice the $GST_CONVERT env variable below.
Examples:
- Encode to H264 (hardware based encode) and stream via RTP/UDP:
# stream H264/RTP/UDP GST_CONVERT=$(gst-inspect-1.0 | grep -e "v4l2.*convert*" | sed -e 's/.*:\s*\(v4l2.*convert\):.*/\1/') gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT caps=application/x-rtp,payload=96 ! rtph264depay ! decodebin ! autovideosink'
- Encode to JPEG (software based encode) and stream via RTP/UDP:
# stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=$DEVICE ! jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay ! jpegdec ! autovideosink'
See also:
Video Decoding (via IMX6 coda driver and video4linux2 element)
Decoding video means taking compressed formats such as MPEG and H264 and decoding them into frames comprised of raw pixels (typically so they can be displayed on a video display output device).
The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer video4linux2
element provides encode/decode elements that tap into this.
Notes:
- Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded:
~# dmesg | grep coda [ 16.721698] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2 [ 16.721724] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin [ 18.381136] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin [ 18.433648] coda 2040000.vpu: Firmware code revision: 570363 [ 18.433683] coda 2040000.vpu: Initialized CODA960. [ 18.433706] coda 2040000.vpu: Firmware version: 3.1.1 [ 18.442312] coda 2040000.vpu: codec registered as /dev/video[9-10] ~# cat /sys/class/video4linux/video9/name coda-encoder ~# cat /sys/class/video4linux/video10/name coda-decoder
- Ensure GStreamer decode elements exists:
~# gst-inspect-1.0 | grep -e "v4l2.*dec" video4linux2: v4l2mpeg4dec: V4L2 MPEG4 Decoder video4linux2: v4l2mpeg2dec: V4L2 MPEG2 Decoder video4linux2: v4l2h264dec: V4L2 H264 Decoder
- The CODA960 decoder outputs NV12/I420/YV12 YUV pixel formats
- The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough)
Note that the following examples assume you are using raw video encoded files, not container formats used for mimxed multimedia types (audio + video) such as ogg, avi, or mov (Quicktime). For information on de-muxing container formats see #multimedia
Examples:
- Decoding H264 video might look like this:
gst-launch-1.0 filesrc location=file.h264 ! h264parse ! v4l2h264dec ! kmssink
- Decoding MPEG4 video might look like this:
gst-launch-1.0 filesrc location=file.mp4v ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink
For more examples including working with 'multimedia' files that contain both audio and video see #multimedia
GStreamer Audio
The GStreamer alsasrc
and alsasink
provide audio capture and playback to Advanced Linux Sound Architecture (ALSA) devices. ALSA is the modern device-driver API for the Linux kernel.
The Gstreamer alsasrc
and alsasink
can be passed a device
property which can represent either the 'device alias' (reported from aplay -L
for playback, or arecord -L
for record) (ie sysdefault:CARD=sgtl5000audio
) or the 'hw:<card>,<device>' notation (ie hw:0,0
for first device, first subdevice)
If not specified via the device
property, the device used for alsasrc and alsasink will depend on /etc/asound.conf
and/or ~/.asoundrc
.
See here for details on specifying audio input and output devices on the Ventana product family.
Playback and Capture also depends on your ALSA mixer (amixer
) settings. See mixer settings for more info.
Audio Output via Linux ALSA driver and GStreamer alsasink
Generally, a 'sink' element is one that will take an audio stream and send it to an audio device. Please refer to this audio page for more details on audio devices on the ventana platform.
This alsasink
element can accept any of the following formats via audio/x-raw
: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE
Examples:
- Generate and play a 1kHz tone to the analog audio output device on Ventana (sgtl5000audio):
gst-launch-1.0 audiotestsrc ! \ alsasink device="sysdefault:CARD=sgtl5000audio"
Audio Input via Linux ALSA driver and GStreamer alsasrc
An input source is anything coming from a capture device on the SBC, e.g. HDMI audio in/analog audio in. Please refer to this audio page for more details on audio devices on the ventana platform.
The alsasrc
element can output the following source types in audio/x-raw
: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE
Examples:
- Capture 32kHz stereo 16bit audio from the Analog Audio input on Ventana (sgtl5000audio) and mux into into an AVI file:
gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \ audioconvert ! avimux ! filesink location=./audio.avi
audiotestsrc
The audiotestsrc
is a very useful element for testing. It can output 16-64bit audio/x-raw
: S16LE, S32LE, F32LE, F64LE
This can output audio signals from sine
to violet-noise
.
Examples:
- Generate and play a 1kHz tone to the analog audio output device on Ventana (sgtl5000audio):
gst-launch-1.0 audiotestsrc ! \ alsasink device="sysdefault:CARD=sgtl5000audio"
Audio Encoding
Encoding is the term used to capture audio and encode it to a new format type.
Commonly used GStreamer Audio encoders:
- vorbisenc - encode
audio/x-raw
F32LE toaudio/x-vorbis
- mulawenc - encode
audio/x-raw
S16LE toaudio/x-mulaw
- wavenc - encode
audio/x-raw
S32LE, S24LE, S16LE, U8, F32LE, F64LE,audio/x-alaw
, andaudio/x-mulaw
toaudio/x-wav
- alawenc - encode
audio/x-raw
S16LE toaudio/x-alaw
- flacenc - encode
audio/x-raw
S24LE, S24_32LE, S16LE, S8 toaudio/x-flac
- lamemp3enc - encode
audio/x-raw
S16LE toaudio/mpeg
There are many more. You can search for your specified one by running a similar search: gst-inspect-1.0 | grep enc
.
Note that each encoder has its own limits on the samples it accepts (S16LE, S24LE etc). You can find the formats accepted via gst-inspect-1.0 <element>
and you can use the #audioconvertaudioconvert element to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements.
Examples:
- Capture audio from the Ventana Analog audio input (sgtl5000audio), encode it using the MPEG Layer 3 audio codec and store it to a file:
gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! audioconvert ! lamemp3enc ! filesink location=file.mp3
Audio Decoding
Decoding is the term used to decode an encoded audio stream to a raw audio stream.
Commonly used GStreamer Audio decoders:
- mpg123audiodec - decode
audio/mpeg
toaudio/x-raw
- vorbisdec - decode
audio/x-vorbis
toaudio/x-raw
- ivorbisdec - decode
audio/x-vorbis
toaudio/x-raw
- a52dec - decode
audio/x-ac3
,audio/ac3
, andaudio/x-private1-ac3
toaudio/x-raw
- mulawdec - decode
audio/x-mulaw
toaudio/x-raw
- alawdec - decode
audio/x-alaw
toaudio/x-raw
- flacdec - decode
audio/x-flac
toaudio/x-raw
Note that each decoder has its own limits on the samples it produces at its src pad (S16LE, S24LE etc). You can find the formats accepted via gst-inspect-1.0 <element>
and you can use the audioconvert element to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements.
Examples:
- Decode MPEG Layer 3 audio and playback to the default audio device with the following:
gst-launch-1.0 filesrc location=file.mp3 ! mpegaudioparse ! mpg123audiodec ! alsasink
Audio Formats and Conversion
Audio samples specified by the audio/x-raw
vary between bit width and endieness as well as the sampling rate (how many times a second audio is measured) and the number of channels that are sampled.
To specify the details of the audio format, you use a GStreamer capsfilter and to convert it from one format to another you use the GStreamer audioconvert
element.
Examples:
- specify the sample-rate (32kHz), channels (2:stereo), and bit depth (16bit) and capture to an AVI file:
gst-launch-1.0 audiotestsrc ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \ audioconvert ! avimux ! filesink location=./audio.avi
Audio loopback (Useful for testing audio input and output)
A simple aduio loopback test can take audio input from an input device and output it to an output device.
loopback audio from the first audio card:
gst-launch-1.0 alsasrc device="hw:0,0" ! alsasink device="hw:0,0"
To send audio from the Ventana Analog input (sgtl5000audio) to the Ventana HDMI output (DWHDMI):
- first, enable Analog input Line in and set capture volume:
card=$(cat /proc/asound/cards | grep sgtl5000 | head -1 | awk '{print $1}') amixer -c $card sset "Capture Mux" LINE_IN amixer -c $card sset "Capture" 8 # 0-15
- using device/function numbers (ie from
aplay -l
):gst-launch-1.0 alsasrc device="hw:2,0" ! \ audioconvert ! \ alsasink device="hw:0,0"
- or, using device aliases (ie from
aplay -L
):gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! \ audioconvert ! \ alsasink device="sysdefault:CARD=imxhdmisoc"
- Note the use of
audioconvert
above to handle any format conversions needed between input and output devices
To send audio from the tda1997x HDMI receiver (digital input) to the imxhdmisoc (digital output):
gst-launch-1.0 alsasrc device="sysdefault:CARD=tda1997xaudio" ! \ "audio/x-raw,rate=44100" ! alsasink device="sysdefault:CARD=imxhdmisoc"
- Note here we need to specify the audio sample-rate as it can vary per input stream (and gstreamer does not validate the rate). This rate must match the source stream samplrate which can be found via sysfs
/sys/bus/i2c/drivers/tda1997x/2-0048/audmode
. If your output device requires a different sample-rate than the source input device, you need to perform a sample-rate conversion.
Multimedia (Video and Audio combined)
Handling media content that has both audio and video stream types requires a slightly more complex pipeline than the standard audio and video pipeline examples.
Generally, a mixed media pipeline which consumes multimedia will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and queue
elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data).
Conversely if producing multimedia content your pipeline will consist of a muxer to join audio and video streams and queue
elements.
Named Elements, queues, and Multiple pipelines with gst-launch
When mixing audio and video elements with gst-launch
one must make use of multiple pipelines using named elements
.
The name
property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any).
Multiple pipelines can be provided to gst-launch
and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'.
This is best explained with some examples:
- Encoding a stream with audio and video content into an AVI file:
gst-launch-1.0 \ videotestsrc \ ! $VIDEO_CAPABILITIES \ ! mux. \ audiotestsrc \ ! $AUDIO_CAPABILITIES \ ! mux. \ avimux name=mux \ ! filesink location=test.avi
- The
videotestsrc
pipeline ends withmux.
which means its output is sent to the pipeline who's name ismux
- The
audiotestsrc
pipeline ends withmux.
which means its output is sent to the pipeline who's name ismux
- The
avimux
pipeline specifiesname=mux
therefore it takes as a source all pipelines that ended withmux.
- The
- Decoding a Matroska container file containing H264 video and AC3 audio
gst-launch-1.0 \ filesrc location=file.mkv \ ! matroskademux name=demux \ demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \ demux. ! queue ! h264parse ! v4l2h264dec ! kmssink
- The
filesrc
pipeline ends withname=demux
which means the output of this pipeline will be sent to all pipelines with ademux.
source (who's types have been successfully negotiated) - The audio pipeline consisting of the
ac3parse
element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3) - The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
- Queue elements are used to keep one pipeline or element from blocking another. For example, if the
v4l2h264dec
element needs more data from the demux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering
- The
- Decoding a MOV file containing H264 video and AAC audio:
gst-launch-1.0 \ filesrc location=file.mov \ ! qtdemux name=demux \ demux. ! queue ! aacparse ! avdec_aac ! alsasink \ demux. ! queue ! h264parse ! v4l2h264dec ! kmssink
Muxing Mixed Content
Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'.
You can use gst-inspect
to see a list of most of these using grep:
gst-inspect-1.0 | grep -i muxer | grep -vi de
Some common examples:
- mpegtsmux: MPEG Transport Stream Muxer
- mpegpsmux: MPEG Program Stream Muxer
- matroskamux: Matroska muxer
- avimux: Avi muxer
- qtmux: QuickTime Muxer
- oggmux: Ogg muxer
To mux mixed content together include one of these elements following the audio and video pipelines.
Examples:
- Encoding a stream with audio and video content into an AVI file:
gst-launch-1.0 \ videotestsrc \ ! $VIDEO_CAPABILITIES \ ! mux. \ audiotestsrc \ ! $AUDIO_CAPABILITIES \ ! mux. \ avimux name=mux \ ! filesink location=test.avi
- the
videotestsrc
pipeline ends withmux.
which means its output is sent to the pipeline who's name ismux
and who's format has been successfully negotiated. - the
audiotestsrc
pipeline ends withmux.
which means its output is sent to the pipeline who's name ismux
and who's format has been successfully negotiated. - the
avimux
pipeline specifiesname=mux
therefore it takes as a source all pipelines that ended withmux.
and it understands how to multiplex the two types of data together into its output which is written to the file test.avi
- the
De-muxing mixed content
Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'.
You can use gst-inspect
to see a list of most of these using grep:
gst-inspect-1.0 | grep -i 'de\?muxer'
Some common examples:
- tsparse: MPEG transport stream parser
- tsdemux: MPEG transport stream demuxer
- matroskademux: Matroska demuxer
- avidemux: Avi demuxer
- qtdemux: QuickTime demuxer
- oggdemux: Ogg demuxer
To demux mixed content include one of these elements following the audio and video pipelines.
Unlike muxing you also need to use a parser
element to parse the bitstream and break it into discrete buffers (frames) that the downstream decoder expects. In other words this 'frames' the data going to decoders.
Some common parsers:
- ogmaudioparse: OGM audio stream parser
- ogmvideoparse: OGM video stream parser
- aacparse: AAC audio stream parser
- amrparse: AMR audio stream parser
- ac3parse: AC3 audio stream parser
- flacparse: FLAC audio parser
- mpegaudioparse: MPEG1 Audio Parser
- h263parse: H.263 parser
- h264parse: H.264 parser
- mpegvideoparse: MPEG video elementary stream parser
- mpeg4videoparse: MPEG 4 video elementary stream parser
- pngparse: PNG parser
- vc1parse: VC1 parser
Examples:
- Demuxing a Matroska container file containing H264 video and AC3 audio into its raw components:
gst-launch-1.0 \ filesrc location=file.mkv \ ! matroskademux name=demux \ demux. ! queue ! ac3parse ! filesink location=file.ac3 \ demux. ! queue ! h264parse ! filesink location=file.h264
- Demuxing a MOV file containing H264 video and AAC audio into its raw components:
gst-launch-1.0 \ filesrc location=file.mov \ ! qtdemux name=demux \ demux. ! queue ! aacparse ! filesink location=file.aac \ demux. ! queue ! h264parse ! filesink locatino=file.h264
bin elements
A bin element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept.
Sometimes the bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream. The gst-launch
application provides a couple of useful debugging tools that can help with this:
- using
GST_DEBUG_DUMP_DOT_DIR
and Graphviz - adding the '-v' parameter to gst-launch will provide verbose feedback on the pipeline configuration that can tell you what is going on
See #troubleshooting for more details on these methods.
GStreamer playbin
GStreamer playbin
element attempts to create a pipeline that will play both the audio and video portions of a file. For example:
gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by providing a video-sink
and audio-sink
property for example to decode audio to the Analog audio output on Ventana:
gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=sysdefault:CARD=sgtl5000audio"
Please type gst-inspect-1.0 playbin
to see more options.
GStreamer decodebin
The GStreamer decodebin
element is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under the first example with the following:
gst-launch-1.0 \ filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \ matroskademux name=d \ d. ! queue ! ivorbisdec ! queue ! alsasink \ d. ! queue ! decodebin ! queue ! kmssink
Note that decodebin
doesn't always choose the correct decoder, so be wary of this. It is similar to playbin
in that it aids in creating a dynamic pipeline.
GStreamer gst-play
The stand-alone application gst-play
is is a program that utilizes the playbin
element and thus can be used for playback of many file types. The above example gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
can be replaced with:
gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
Streaming (send multimedia to or receive from Network)
GStreamer has elements that allow for network streaming to occur.
There are several ways to accomplish networked streaming over Internet Protocol (IP):
- Raw UDP/IP
- Raw TCP/IP
- Real-time Transport Protocol (RTP)
- Real Time Streaming Protocol (RTSP) (recommended)
- Adaptive Bitrate Streaming
Raw UDP
Using UDP/IP is the simplest mechanism for streaming and utilizes the least amount of bandwidth. Because UDP does not provide any error detection, packet ordering, or error correction the bitrate is deterministic and simply the bitrate of the media you are streaming.
The limitations of raw UDP is:
- requires codec that can handle missing/corrupt data (most do these days)
- does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media)
- does not fragment packets - will try to send a raw udp packet for whatever size buffer the udpsink is passed (which can lead to pipeline errors). To fragment packets use RTP
The only benefit of using raw UDP is that it is the simplest pipeline you can create for streaming and requires the least amount of dependencies (albeit you might run into one or all of the above problems).
Note that it is recommended that you use RTP or RTSP unless you know exactly what you are doing to overcome the limitations listed above
The udpsrc
element can be used to render/save a stream originated from a udpsink
pipeline.
Examples:
- encode and send H264 video from Ventana:
- Start decoder first:
ifconfig eth0 192.168.1.1 gst-launch-1.0 udpsrc port=9001 ! h264parse ! v4l2h264dec ! kmssink sync=false
- Start encoder second:
ifconfig eth0 192.168.1.2 gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc ! udpsink host=192.168.1.1 port=9001
- Start decoder first:
Notes:
- On the client (stream receiver and renderer) you must use the
sync=false
property to render frames as they are received otherwise the stream will stall because their is no headers containing timestamps - the decoder (udpsrc) needs to be started first because udpsink will fail if nothing is listening to the socket
TCP
Using TCP/IP brings error detection, packet re-ordering, and error correction to the network stream. This however causes the bitrate to be non-deterministic because as the error rate increases so does the bitrate and latency.
The limitations of using TCP:
- non-deterministic bitrate
- added latency
- does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media)
Note that it is recommended that you use RTP or RTSP unless you know exactly what you are doing to overcome the limitations listed above
TCP/IP introduces the concept of a socket connection therefore there must exist a server and a client in which case the server must be started first to listen for a connection. You can use a server sink or a server source. The tcpserversrc
source can be used to create a TCP server that waits for a connection from a tcpclientsink
to render/save. Alternatively the tcpserversink
sink can be used to create a TCP server that waits for a connection from a tcpclientsrc
that will send data.
Examples:
- encode and send H264 video from Ventana with decoder as server:
- Start decoder (server) first:
ifconfig eth0 192.168.1.1 gst-launch-1.0 tcpserversrc host=0.0.0.0 port=9001 ! decodebin ! autovideosink sync=false
- Start encoder (client) second:
ifconfig eth0 192.168.1.2 ENCODER=/dev/$(for i in $(ls -d /sys/class/video4linux/video*); do [ "coda-encoder" = "$(cat $i/name)" ] && basename $i; done) gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! tcpclientsink host=192.168.1.1 port=9001
- Start decoder (server) first:
Notes:
- TCP is connection oriented therefore the TCP 'server' must be started first. You can choose your elements such that the stream originator is the server or the stream renderer is the server however doing so can be problematic for certain codecs because the client decoding the stream may pick up the stream somewhere in the middle and not know how to parse it.
- the
host=0.0.0.0
property means listen to all network interfaces
RTP (raw/session-less)
The Real-time Transport Protocol (RTP) is a network protocol for delivering audio and video over IP networks. RTP is used extensively in communication and entertainment systems that involve streaming media, such as telephony, video teleconference applications, television services and web-based push-to-talk features.
The RTP packet type encapsulates multimedia data with a payload type and time-stamp and therefore can be used to compensate for jitter, out of sequence packets, and time synchronization between streams of different types (ie audio/video lip-sync).
RTP is typically used in conjunction with other protocols such as RTP Control Protocol (RTCP) and Real Time Streaming Protocol (RTSP) to manage stream sessions however can be used on its own in a raw session-less fashion using udpsink
and udpsrc
elements.
The limitations of using raw/session-less RTP:
- session management needs to be handled manually (capsfilter is needed to specify stream format)
Note that it is recommended that you use RTSP unless you know exactly what you are doing to overcome the limitations listed above
Example: Capture, encode and stream H264 via RTP with GStreamer playback:
- Encode and send H264 video from Ventana:
- Start decoder first:
ifconfig eth0 192.168.1.1 gst-launch-1.0 udpsrc port=9001 \ caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" \ ! decodebin ! autovideosink
- Start encoder second:
gst-launch-1.0 videotestsrc is-live=true \ ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! rtph264pay ! udpsink host=192.168.1.1 port=9001
- Start decoder first:
Notes:
- when using RTP a capsfilter must be used to specify the payload as application/x-rtp as above. You can determine the capsfilter required by starting the encoder with a verbose flag
-v
and looking forcaps = "application/x-rtp"
Example: Capture, encode and stream H264 via RTP with VLC playback:
- Encode and send H264 video from Ventana to a PC with VLC:
- Start decoder first:
- Create SDP file like below (IP address in example is that of the Ventana board)
v=0 m=video 5000 RTP/AVP 96 c=IN IP4 172.24.20.207 a=rtpmap:96 H264/90000
- Open SDP file in VLC
- Create SDP file like below (IP address in example is that of the Ventana board)
- Start encoder (Ventana) second: (IP address in below example is IP of the PC)
gst-launch-1.0 videotestsrc ! v4l2h264enc ! rtph264pay config-interval=3 ! udpsink host=172.24.20.26 port=5000
RTSP (Real Time Streaming Protocol) (recommended)
The Real Time Streaming Protocol (RTSP) is a network control protocol designed for use in entertainment and communications systems to control streaming media servers. The protocol is used for establishing and controlling media sessions between end points. Clients of media servers issue VCR-style commands, such as play and pause, to facilitate real-time control of playback of media files from the server. This protocol uses the Real-time Transport Protocol (RTP) in conjunction with Real-time Control Protocol (RTCP) for media stream delivery.
The limitations of using RTSP are:
gst-inspect
has no way of using a simple pipeline to create an RTSP server - you must create or use an existing gstreamer based application (keep reading below).
Creating an RTSP server is not possible via a simple pipeline to gst-launch however GStreamer libraries do exist to make writing an RTSP server trivial. The source for gst-rtsp-server contains an example application test-launch.c which provides a simple example that can take a GStreamer 'bin' element consisting of everything but the sink element and serves it via RTSP.
Gateworks has extended this with gst-variable-rtsp-server which demonstrates how to auto-adjusting properties like the encoding bitrate depending on the number of clients connected in addition to serving as a fairly simple example of how to write a gstreamer application.
Troubleshooting
If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device.
You can see all the properties of a particular element with the 'gst-inspect <elementname>' command. You can see a list of all elements with 'gst-inspect' without arguments.
You can enable verbose output for GStreamer by adding the '-v' flag which will show you the negotiated pipeline details and the state machine details.
You can enable debug output for GStreamer by using the --gst-debug parameter to gst-launch or by setting the GST_DEBUG environment variable.
For example, if you wanted to see DEBUG level messages for the videotestsrc
element you can set GST_DEBUG=videotestsrc:4
GST_DEBUG_DUMP_DOT_DIR
You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause gst-launch
to output a .dot
file for each phase of the pipeline then use a tool such as Graphviz
to visualize the .dot
file.
Example:
- use playbin to playback a file:
root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
- hit Cntl-C after decoding starts to exit early
- see the dot files created:
root@ventana:~# ls /tmp/dot 0.00.00.108710334-gst-launch.NULL_READY.dot 0.00.00.490807334-gst-launch.READY_PAUSED.dot 0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot 0.00.03.254000001-gst-launch.PAUSED_READY.dot
- transfer to a PC and use something like
xdot
to view:xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
- zoom in along the graph and you can see that:
GstFileSrc
is the source,GstAviDemux
is used to demux to audio/x-ac3,GstAc3Parse
is used to parse the audio into audio frames,GstMpeg4VParse
is used to parse the video into video frames,GstV4l2VideoDec
is used to decode the video from video/mpeg to video/x-raw,GstA52Dec
is used to decode the audio from audio/x-ac3 to audio/x-raw,- etc
- Note that some hunting with
gst-inspect
must be done to determine what elements coorespond to the above class names
- zoom in along the graph and you can see that:
Reference:
gst-launch -v
The verbose debugging from gst-launch -v
can show you the negotiation that takes place as a pipeline moves through its stages.
Example:
gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
examining the verbose output can show you the following:
- container: AVI: avidemux
- video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! v4l2mpeg4dec
- audio: AC3 48khz 5channels: ac3parse ! a52dec
Therefore you can use these pipelines to decode and play:
- video only (output to kmssink)
gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink
- audio only (output to hdmi audio sink)
gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
- both audio and video
gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \ d. ! queue ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink \ d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
streaming
If you're having issues with network streaming:
- Verify that both sides can ping one another
- If the message
There may be a timestamping problem, or this computer is too slow
appears and the video display appears choppy, try the following:- Lower the bitrate from the server
- Place a
sync=false
on the sink side of the server and client.
- If video appears choppy, try using UDP over TCP.
- Verify that the network is not congested.
- Verify your gstreamer pipeline is correct. The best way to find the element that causes a negotiation failure is to end your pipeline in a fakesink and one-by-one eliminate elements leading up to it until it negotiates successfully.