[[PageOutline]] = GStreamer on Gateworks SBCs The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These '''GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs)''', specifically the Ventana family that utilizes the Freescale i.MX6 processors. Gateworks SBCs can be viewed at the following link: [http://www.gateworks.com] [[Image(http://trac.gateworks.com/raw-attachment/wiki/OpenWrt/wireless/relayd/gw5100realsmall.png,200px)]] '''Important Note''' The IMX6 (Ventana family) software has been evolving over the years. This page is focused on mainline Linux kernels and not the Freescale GStreamer {{{gstreamer-imx}}} plugin legacy software that used the 3.14 Freescale Vendor kernel. If using the Gateworks 3.14 kernel with the Freescale capture drivers please see [wiki:Yocto/gstreamer] Also note that GStreamer capture devices are {{{/dev/videoN}}} which require some configuration if using Processor specific video input busses such as the IMX6 capture using Analog (CVBS) or Digital video (HDMI). If using these capture interfaces please see [wiki:linux/media] for details on how to use {{{media-ctl}}} to configure those devices. = Video4Linux2 Devices The Video4Linux2 API is the device driver API used for kernel capture drivers as well as video encode and video decode drivers. [=#media-ctl] == Media Control API The !MediaControl API is used to configure Video4Linux2 capture devices. See [wiki:linux/media] for more info. [=#v4l2-ctrls] == V4L2 Controls (brightness, contrast, saturation, rotation, flip, bitrate etc) When capturing video from a video4linux2 device you can use the v4l2 API to get and set various controls the device may provide access to such as: * brightness * contrast * hue * saturation * horizontal flip * veritcal flip * rotation What controls are available depend on the device. Furthermore when using capture devices that have media-ctl pipelines controls provided by various elements within the pipeline are passed on to the capture device. Examples: * list available controls of a capture device: {{{#!bash media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $DEVICE --list-ctrls # list available controls }}} * get brightness control for adv7180 analog video capture device: {{{#!bash media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $DEVICE --get-ctrl=brightness }}} * set brightness control for adv7180 analog video capture device: {{{#!bash media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $DEVICE --set-ctrl=brightness=50 }}} * set video bitrate of the v4l2h264enc H264 encoder element: {{{#!bash media-ctl-setup adv7180 > setup source setup v4l2-ctl --device $ENCODER --set-ctrl=video_bitrate=10000000 }}} [=#test-media] = Test Media Files There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing: * [https://peach.blender.org/download/ Big Buck Bunny] - Creative Commons * [https://durian.blender.org/download/ Sintel] - Creative Commons * [https://mango.blender.org/download/ Tears of Steel] - Creative Commons = GStreamer [http://GStreamer.freedesktop.org GStreamer] is an opensource library and framework created to handle multimedia. Pipelines are used to source and sink media however you would like (e.g. decoding the mp3 out of a video file and playing it back through speakers). It is important to understand that GStreamer is foremost a library meant for developing applications. While it has an extremely useful test application called {{{gst-launch}}} meant for testing pipelines using that application alone can't solve all problems. There are plenty of resources online including mailing lists and IRC channels that can be helpful in understanding GStreamer and writing applications that use it. To install GStreamer on Ubuntu: {{{#!bash apt-get install gstreamer1.0-x gstreamer1.0-tools libgstreamer1.0-0 \ gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \ gstreamer1.0-libav gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio \ v4l-utils alsa-utils }}} [=#plugins] == Plugins and Elements GStreamer works on 'plugins' that provide pipeline 'elements'. A plugin comprises of elements that can do work on a media stream. For example, the {{{mpeg2dec}}} can decode mpeg1 and mpeg2 video streams (software based decoding, not hardware accelerated). [=#pipelines] == Pipeline Construction [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-applications.html gst-launch] is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type {{{gst-inspect element}}} where {{{element}}} is the element you are looking for. To see a full listing of elements installed, you can type just {{{gst-inspect}}}. Here is a graphical example of a pipeline. [[Image(https://gstreamer.freedesktop.org/documentation/tutorials/basic/images/figure-1.png, 512px)]] The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next [http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+10%3A+gstreamer+tools gstreamer.com] contains much more information on the construction of pipelines. [=#capsfilters] == Caps Filters GStreamer has a concept called [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-caps-api.html caps filters]. A 'cap' (short for capability) is used to describe the type of data that links a src (output) pad from one element to a sink (input) pad of another element. Adding a {{{-v}}} flag to {{{gst-launch-1.0}}} will output the capabilities negotiated between elements: {{{#!bash ~# gst-launch-1.0 -v videotestsrc ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock }}} In the output, the {{{caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"}}} are the video caps negotiated between the two. You can force a caps filter between two elements by treating it as a pipeline element: {{{#!bash ~# gst-launch-1.0 -v videotestsrc ! 'video/x-raw, format=UYVY, width=1920, height=1080, framerate=10/1' ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)10/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive" Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock }}} As you can see, the caps filter above changed both the video format, resolution, and framerate of the video stream coming out of the {{{videotestsrc}}} element. Caps filters are useful when you want to use a specific format where a device may support multiple. For example when capturing audio from an ALSA sound device that supports a variety of samplerates, channels, and bit formats you can use a capsfilter to override any default configuration. [=#applications] == Applications GStreamer is a library and framework for multimedia processing. It is not a user-space application. It is easy to forget this because of the usefulness of the {{{gst-launch}}} application that can be used to connect simple pipelines together for testing. For your needs you very likely need to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that. GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html here] for more details. Furthermore, Gateworks has created a [#gst-variable-rtsp-server gst-variable-rtsp-server] example application that demonstrates how to server simple GStreamer pipelines via an RTSP server and automatically adjust compression quality based on client connections. [=#fakesink] == fakesink The {{{fakesink}}} element is a very useful video sink when debugging or learning GStreamer. It takes whatever frames is given to it and drops them. This might help debugging pipelines if problems ever arise. Example: {{{#!bash gst-launch-1.0 videotestsrc ! fakesink }}} == gst-inspect The {{{gst-inspect-1.0}}} tool is an excellent tool if you're unfamiliar with the functionality, source/sink formats, or properties of any Gstreamer element. Usage: {{{#!bash root@bionic-armhf:~# gst-inspect-1.0 videotestsrc Factory Details: Rank none (0) Long-name Video test source Klass Source/Video Description Creates a test video stream Author David A. Schleef Plugin Details: Name videotestsrc Description Creates a test video stream Filename /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstvideotestsrc.so Version 1.14.5 License LGPL Source module gst-plugins-base Source release date 2019-05-29 Binary package GStreamer Base Plugins (Ubuntu) Origin URL https://launchpad.net/distros/ubuntu/+source/gst-plugins-base1.0 GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstBaseSrc +----GstPushSrc +----GstVideoTestSrc Pad Templates: SRC template: 'src' Availability: Always Capabilities: video/x-raw format: { (string)I420, (string)YV12, (string)YUY2, (string)UYVY, (string)AYUV, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGB, (string)BGR, (string)Y41B, (string)Y42B, (string)YVYU, (string)Y444, (string)v210, (string)v216, (string)NV12, (string)NV21, (string)GRAY8, (string)GRAY16_BE, (string)GRAY16_LE, (string)v308, (string)RGB16, (string)BGR16, (string)RGB15, (string)BGR15, (string)UYVP, (string)A420, (string)RGB8P, (string)YUV9, (string)YVU9, (string)IYU1, (string)ARGB64, (string)AYUV64, (string)r210, (string)I420_10BE, (string)I420_10LE, (string)I422_10BE, (string)I422_10LE, (string)Y444_10BE, (string)Y444_10LE, (string)GBR, (string)GBR_10BE, (string)GBR_10LE, (string)NV16, (string)NV24, (string)NV12_64Z32, (string)A420_10BE, (string)A420_10LE, (string)A422_10BE, (string)A422_10LE, (string)A444_10BE, (string)A444_10LE, (string)NV61, (string)P010_10BE, (string)P010_10LE, (string)IYU2, (string)VYUY, (string)GBRA, (string)GBRA_10BE, (string)GBRA_10LE, (string)GBR_12BE, (string)GBR_12LE, (string)GBRA_12BE, (string)GBRA_12LE, (string)I420_12BE, (string)I420_12LE, (string)I422_12BE, (string)I422_12LE, (string)Y444_12BE, (string)Y444_12LE, (string)GRAY10_LE32, (string)NV12_10LE32, (string)NV16_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] multiview-mode: { (string)mono, (string)left, (string)right } video/x-bayer format: { (string)bggr, (string)rggb, (string)grbg, (string)gbrg } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] multiview-mode: { (string)mono, (string)left, (string)right } Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SRC: 'src' Pad Template: 'src' Element Properties: name : The name of the object flags: readable, writable String. Default: "videotestsrc0" parent : The parent of the object flags: readable, writable Object of type "GstObject" blocksize : Size in bytes to read per buffer (-1 = default) flags: readable, writable Unsigned Integer. Range: 0 - 4294967295 Default: 4096 num-buffers : Number of buffers to output before sending EOS (-1 = unlimited) flags: readable, writable Integer. Range: -1 - 2147483647 Default: -1 typefind : Run typefind before negotiating (deprecated, non-functional) flags: readable, writable, deprecated Boolean. Default: false do-timestamp : Apply current stream time to buffers flags: readable, writable Boolean. Default: false pattern : Type of test pattern to generate flags: readable, writable Enum "GstVideoTestSrcPattern" Default: 0, "smpte" (0): smpte - SMPTE 100% color bars (1): snow - Random (television snow) (2): black - 100% Black (3): white - 100% White (4): red - Red (5): green - Green (6): blue - Blue (7): checkers-1 - Checkers 1px (8): checkers-2 - Checkers 2px (9): checkers-4 - Checkers 4px (10): checkers-8 - Checkers 8px (11): circular - Circular (12): blink - Blink (13): smpte75 - SMPTE 75% color bars (14): zone-plate - Zone plate (15): gamut - Gamut checkers (16): chroma-zone-plate - Chroma zone plate (17): solid-color - Solid color (18): ball - Moving ball (19): smpte100 - SMPTE 100% color bars (20): bar - Bar (21): pinwheel - Pinwheel (22): spokes - Spokes (23): gradient - Gradient (24): colors - Colors timestamp-offset : An offset added to timestamps set on buffers (in ns) flags: readable, writable Integer64. Range: 0 - 2147483646999999999 Default: 0 is-live : Whether to act as a live source flags: readable, writable Boolean. Default: false k0 : Zoneplate zero order phase, for generating plain fields or phase offsets flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kx : Zoneplate 1st order x phase, for generating constant horizontal frequencies flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 ky : Zoneplate 1st order y phase, for generating contant vertical frequencies flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kt : Zoneplate 1st order t phase, for generating phase rotation as a function of time flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kxt : Zoneplate x*t product phase, normalised to kxy/256 cycles per vertical pixel at width/2 from origin flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kyt : Zoneplate y*t product phase flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kxy : Zoneplate x*y product phase flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kx2 : Zoneplate 2nd order x phase, normalised to kx2/256 cycles per horizontal pixel at width/2 from origin flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 ky2 : Zoneplate 2nd order y phase, normailsed to ky2/256 cycles per vertical pixel at height/2 from origin flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 kt2 : Zoneplate 2nd order t phase, t*t/256 cycles per picture flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 xoffset : Zoneplate 2nd order products x offset flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 yoffset : Zoneplate 2nd order products y offset flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 foreground-color : Foreground color to use (big-endian ARGB) flags: readable, writable, controllable Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 background-color : Background color to use (big-endian ARGB) flags: readable, writable, controllable Unsigned Integer. Range: 0 - 4294967295 Default: 4278190080 horizontal-speed : Scroll image number of pixels per frame (positive is scroll to the left) flags: readable, writable Integer. Range: -2147483648 - 2147483647 Default: 0 animation-mode : For pattern=ball, which counter defines the position of the ball. flags: readable, writable Enum "GstVideoTestSrcAnimationMode" Default: 0, "frames" (0): frames - frame count (1): wall-time - wall clock time (2): running-time - running time motion : For pattern=ball, what motion the ball does flags: readable, writable Enum "GstVideoTestSrcMotionType" Default: 0, "wavy" (0): wavy - Ball waves back and forth, up and down (1): sweep - 1 revolution per second (2): hsweep - 1/2 revolution per second, then reset to top flip : For pattern=ball, invert colors every second. flags: readable, writable Boolean. Default: false }}} [=#video] = GStreamer Video This section will introduce several concepts regarding how video media is handled by GStreamer and provide several example pipelines. [=#video-output] == Video Output Generally, a GStreamer 'sink' element is one that will take a video stream and output it to a display. Some commonly used video output sinks: * autovideosink: Wrapper video sink for automatically detecting video output device * kmssink: KMS video device sink * fbdevsink: Framebuffer video device sink Execute a {{{gst-inspect-1.0 | grep sink}}} to see a complete list of sinks available and {{{gst-inspect-1.0 }}} to see specific details about the applicable formats and properties of a specific element. **Important Note**: Video display drivers for use with video display sinks such as kmssink and fbdevsink for HDMI out and LVDS require proper kernel commandline parameters to configure them. Sometimes the Bootloader bootscript will do this for you - Please see [wiki:linux/display] for details and check the kernel command-line after booting to make sure the desired device is enabled and configured for the proper resolution and mode. Examples: * HDMI enabled and configured for 1080p60Hz LVDS disabled: {{{#!bash ~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done video=HDMI-A-1:1920x1080M@60 video=LVDS-1:d }}} * HDMI disabled and LVDS configured for 1280x800 60Hz {{{#!bash ~# for arg in $(cat /proc/cmdline); do echo $arg | grep video; done video=LVDS-1:1280x800@60M video=HDMI-A-1:d }}} [=#autovideosink] === autovideosink This GStreamer sink is not really a 'video' sink in the traditional sense. Similar to {{{playbin}}} and {{{decodebin}}}, this element selects what it thinks is the best available video sink and uses it. This will typically use {{{kmssink}}} unless format choices require one of the other sinks. You can add a verbose flag {{{gst-launch-1.0 -v}}} to see details about the elements and caps chosen when using any type of 'auto' element or 'bin' element. Example: {{{#!bash gst-launch-1.0 videotestsrc ! autovideosink }}} * Note that adding a '-v' flag for verbose mode will show the element configuration to let you know what sink was actually chosen [=#kmssink] === kmssink This sink allows you to directly output to a KMS (kernelmode setting API) driver. This is the most efficient video output method. Example: {{{#!bash gst-launch-1.0 videotestsrc ! kmssink }}} [=#fbdevsink] The Framebuffer device sink allows you to directly output to a Linux Framebuffer device if you have one (ls /sys/class/graphics/fb*). Example: {{{#!bash gst-launch-1.0 videotestsrc ! fbdevsink }}} [=#video-input] == Video Input (Capture) A video input source is anything coming from an input on the device, e.g. HDMI input/USB Web Cam. This is referred to as Video Capture. In GStreamer terms a video capture device is a 'source' or 'src' element. Please refer to the [wiki:linux/media linux/media] page for details on the video in devices on the Ventana platform. Some commonly used GStreamer video input sources: * videotestsrc * v4l2src Execute a {{{gst-inspect-1.0 | grep src}}} to see a complete list of source elements available and {{{gst-inspect-1.0 }}} to see specific details about the applicable formats and properties of a specific element. If the {{{is-live}}} property is set to true this will cause buffers to be discarded on a pipeline paused state and pipelines will not participate in the PREROLL phase of a pipeline. ** Important Note: ** Linux video capture devices provided by CPU's such as the IMX6 Analog (CVBS) and Digital (HDMI) inputs provided on Ventana boards require a hardware pipeline configuration. See [wiki:linux/media] for details on how to configure the video capture device pipeline. [=#videotestsrc] === videotestsrc This is a very useful element for testing pipelines as it creates video content. It can output a huge number of video formats for raw video {{{video/x-raw}}} and bayer video {{{video/x-bayer}}}. Use {{{gst-inspect-1.0 videotestsrc}}} to see its output formats and properties such as video test pattern, orientation, colrs, etc. Examples: * display SMPTE pattern to connected and configured display {{{#!bash gst-launch-1.0 videotestsrc ! kmssink }}} * set the 'pattern' property to display a moving ball: {{{#!bash gst-launch-1.0 videotestsrc pattern=18 ! kmssink }}} * Use a '[#capsfilter capsfilter]' to force output of videotestsrc to a specific resolution, format, and framerate (ie 1080p 10fps YUV): {{{#!bash gst-launch-1.0 videotestsrc ! 'video/x-raw,format=UYVY,width=1920,height=1080,framerate=10/1' ! kmssink }}} [=#v4l2src] === v4l2src This element uses the video4linux2 API to capture video from input sources (/dev/video). Note that {{{v4l2src}}} is always live regardless of the is-live property. Note that you need to configure the media-ctl pipeline that feeds the /dev/video devices - see [wiki:linux/media] for details You can use the {{{v4l2-ctl}}} application from the {{{v4l-utils}}} package to to interact with the device to get/set various capabilities and controls. Examples: * show all video4linux2 devices: {{{#!bash root@bionic-armhf:~# v4l2-ctl --list-devices CODA960 (platform:coda): /dev/video8 /dev/video9 imx-media-mem2mem (platform:imx-media-mem2mem): /dev/video10 imx-media-capture (platform:ipu1_csi0): /dev/video4 imx-media-capture (platform:ipu1_csi1): /dev/video5 imx-media-capture (platform:ipu1_ic_prpenc): /dev/video0 imx-media-capture (platform:ipu1_ic_prpvf): /dev/video1 imx-media-capture (platform:ipu2_csi0): /dev/video6 imx-media-capture (platform:ipu2_csi1): /dev/video7 imx-media-capture (platform:ipu2_ic_prpenc): /dev/video2 imx-media-capture (platform:ipu2_ic_prpvf): /dev/video3 }}} * display all details about /dev/video1: {{{#!bash v4l2-ctl -d /dev/video1 --all }}} * Use Gateworks {{{media-ctl-setup}}} to configure the media-ctl pipeline for Analog Video capture using the adv7180 chip. If you don't already have this on your root filesystem you can install it via: {{{#!bash wget https://raw.githubusercontent.com/Gateworks/media-ctl-setup/master/media-ctl-setup -O /usr/local/bin/media-ctl-setup chmod +x /usr/local/bin/media-ctl-setup media-ctl-setup adv7180 > setup . ./setup }}} - you can look at the setup script to see what was done - the details vary between SoC (IMX6DL vs IMX6Q), board, and sensor (adv7180 for Analog video capture and tda1997x for digital video capture) - note that the setup script exports the video capture device to the DEVICE env variable which you can use as shown below * After sourcing setup above you can capture and display to the connected and configured output display (video loopback): {{{#!bash gst-launch-1.0 v4l2src device=$DEVICE ! kmssink }}} [=#colorspace] [=#scaling] [=#mem2mem] == Colorspace Converting and/or Video Scaling (via IMX6 IPU mem2mem driver) Often times, a colorspace conversion or scaling is required in order to link GStreamer elements together. This is often due to the fact that not all elements can every format available. A Linux V4L2 MEM2MEM imx-media driver exists that allows utilizing the IMX6 IPU Image Converter hardware blocks (IC) to perform hardware colorspace conversion (CSC), scaling, cropping, rotation, and flip operations. The GStreamer {{{video4linux2}}} element provides an element that uses this driver to expose these capabilities to GStreamer applications. Notes: - for GStreamer-1.14 the name of the element depends on the video device the driver registers with the kernel (ie v4l2video8convert if mem2mem driver registers /dev/video8) - for GStreamer master (in development) the name of the element is always 'v4l2videoconvert' - the {{{kmssink}}} examples below need a {{{can-scale=false}}} property to tell GStreamer not to scale via the KMS driver (as the IMX6 KMS driver does not support scaling) - ensure that the input format differs from the output format otherwise GStreamer will bypass the conversion completely; note that GStreamer doesn't understand flipping or rotation as part of the format. Gstreamer master (in development) adds a 'disable-passthrough' property to the v4l2videoconvert entity that can be set to force the conversion regardless of input and output format - when using imx entities (ie capture, encode/decode, mem2mem, display) you can specify 'output-io-mode=dmabuf-import' to share dmabuf pointers for a zero-copy pipeline however if using non imx entities (ie videotestsrc) you must omit these as you can not ensure the buffers share the alignment/stride necessary to share dmabuf pointers Examples: * Ensure mem2mem is in your kernel: {{{#!bash ~# dmesg | grep mem2mem [ 18.356023] imx-media: Registered ipu_ic_pp mem2mem as /dev/video8 }}} * Ensure GStreamer element exists: {{{#!bash ~# gst-inspect-1.0 | grep -e "v4l2.*convert" video4linux2: v4l2convert: V4L2 Video Converter }}} - Note that for GStreamer-1.14, the name of the element depends on the video device the driver registers with the kernel (video8 in the above example). This changes in GStreamer-1.16 to always be 'v4l2videoconvert' * Obtain the name of the element (as it can vary between GStreaqmer-1.14 and GStreamer 1.16) {{{#!bash GST_CONVERT=$(gst-inspect-1.0 | grep -e "v4l2.*convert*" | sed -e 's/.*:\s*\(v4l2.*convert\):.*/\1/') }}} * scale/rotate/flip using {{{videotestsrc}}} (can not use dmabufs for this as it is a non-imx entity) {{{#!bash # upscale gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT ! \ video/x-raw,width=640,height=480 ! kmssink can-scale=false # downscale gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480 ! \ $GST_CONVERT ! \ video/x-raw,width=320,height=240 ! kmssink can-scale=false # rotate gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT extra-controls=cid,rotate=90 ! \ video/x-raw,width=240,height=320 ! kmssink can-scale=false # hflip gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT extra-controls=cid,horizontal_flip=1 ! \ video/x-raw,width=640,height=480 ! kmssink can-scale=false # vflip gst-launch-1.0 videotestsrc ! video/x-raw,width=320,height=240 ! \ $GST_CONVERT extra-controls=cid,vertical_flip=1 ! \ video/x-raw,width=640,height=480 ! kmssink can-scale=false }}} - note the above examples force the input format (resolution in this case) to differ from the output format otherwise gstreamer will bypass the v4l2convert entity thinking it not necessary as gstreamer does not understand the flip/rotation properties. GStreamer master (in development) adds the 'disable-passthrough' property which can be enabled to force disabling passthrough * scale/rotate/flip using imx-media capture device and KMS display driver (can use dmabufs for this as they are all imx hardware entitites): {{{#!bash # scale sensor input to 720p display gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ video/x-raw,width=1280,height=720 ! \ kmssink can-scale=false # scale sensor input to 1080p display gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ video/x-raw,width=1920,height=1080 ! \ kmssink can-scale=false # scale/flip gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \ video/x-raw,width=1920,height=1080 ! \ kmssink can-scale=false # scale/rotate gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \ video/x-raw,width=720,height=1280 ! \ kmssink can-scale=false }}} * capture, scale, rotate, flip and encode using imx-media capture device mem2mem device and coda device (can use dmabufs for zero-copy) {{{#!bash # encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # scale/encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import ! \ video/x-raw,width=1440,height=960 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # scale/flip/encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,horizontal_flip=1 ! \ video/x-raw,width=1440,height=960 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # scale/rotate/encode gst-launch-1.0 v4l2src device=$DEVICE ! \ $GST_CONVERT output-io-mode=dmabuf-import extra-controls=cid,rotate=90 ! \ video/x-raw,width=1440,height=960 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT }}} For anything not able to be handled by the video4linux2 element and any mem2mem drivers there are software based elements that can convert pixel colorspace and/or scale however note these are extremely CPU intensive: * autovideoconvert * videoconvert * rgb2bayer [=#interlaced-video] == Interlaced Video and Deinterlacing Interlaced video is a technique for doubling the perceived frame rate of a video display without consuming extra bandwidth. The interlaced signal contains two fields of a video frame captured at two different times. The alternative to interlaced video is called progressive video. While reducing bandwidth this can cause a perceived flicker effect as well as a very apparent artifact seen during motion. For example a car moving horizontally across a scene will show every other vertical line differently: the second field interlaced with the first field will be a full frame period ahead in time from the other. The visual affect can be seen in [https://en.wikipedia.org/wiki/File:Interlaced_video_frame_(car_wheel).jpg ​this image from Wikipedia] Television signals are typically interlaced, or at least were until recently. For example, analog television standards such as NTSC used in North America as well as the PAL and SECAM formats used abroad use interlaced video and therefore any analog video decoder such as the ADV7180 found on many Gateworks Ventana boards will capture interlaced video and are subject to interlacing artifacts. Interlaced video is still used in High Definition signals as well and the letter at the end of the format tells you if its interlaced (ie 480i, 720i, 1080i) or progressive (ie 480p, 720p, 1080p). To deinterlace video captured from the adv7180 analog decoder you need to use media-ctl to configure your pipeline to include the IMX6 Video Deinterlacer block. This is done for you if you use the Gateworks {{{media-ctl-setup}}} script to configure your pipeline. See [wiki:linux/media#media-ctl-setup linux/media] for more info. References: * [https://en.wikipedia.org/wiki/Interlaced_video Wikipedia interlaced video - includes several images demonstrating interlacing artifacts] * [https://en.wikipedia.org/wiki/Deinterlacing​Wikipedia deinterlacing] [=#encoding] == Video Encoding (via IMX6 coda driver and video4linux2 element) Encoding video means taking the raw captured pixels and converting them to a standard compressed format. The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer {{{video4linux2}}} element provides encode/decode elements that tap into this. Notes: * Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded: {{{#!bash ~# dmesg | grep coda [ 16.721698] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2 [ 16.721724] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin [ 18.381136] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin [ 18.433648] coda 2040000.vpu: Firmware code revision: 570363 [ 18.433683] coda 2040000.vpu: Initialized CODA960. [ 18.433706] coda 2040000.vpu: Firmware version: 3.1.1 [ 18.442312] coda 2040000.vpu: codec registered as /dev/video[9-10] ~# cat /sys/class/video4linux/video9/name coda-encoder ~# cat /sys/class/video4linux/video10/name coda-decoder }}} * Ensure GStreamer encode elements exists: {{{#!bash ~# gst-inspect-1.0 | grep -e "v4l2.*enc" video4linux2: v4l2h264enc: V4L2 H.264 Encoder video4linux2: v4l2mpeg4enc: V4L2 MPEG4 Encoder }}} * The CODA960 encoder requires NV12/I420/YV12 YUV pixel formats with rec709 colorimetry * The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough) * Encoder and Decoder options are exposed through the v4l2 control IDs (CID) and can be listed with the v4l2-ctl -L and v4l2-ctl -l parameters on the devices exposed by the coda driver: Examples: * Encode to H264 (hardware based encode) and stream via RTP/UDP: {{{#!bash # stream H264/RTP/UDP gst-launch-1.0 v4l2src device=$DEVICE ! \ v4l2video10convert output-io-mode=dmabuf-import ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=$SERVER port=$PORT # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT caps=application/x-rtp,payload=96 ! rtph264depay ! decodebin ! autovideosink' }}} * Encode to JPEG (software based encode) and stream via RTP/UDP: {{{#!bash # stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=$DEVICE ! jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT # client on $SERVER:$PORT could be viewing via 'gst-launch-1.0 udpsrc port=$PORT ! application/x-rtp,payload=96 ! rtpjpegdepay ! jpegdec ! autovideosink' }}} See also: * [wiki:ventana/vpu ventana/vpu] [=#decoding] == Video Decoding (via IMX6 coda driver and video4linux2 element) Decoding video means taking compressed formats such as MPEG and H264 and decoding them into frames comprised of raw pixels (typically so they can be displayed on a video display output device). The Linux CODA driver provides access to the IMX6 hardware encode/decode codecs and the GStreamer {{{video4linux2}}} element provides encode/decode elements that tap into this. Notes: * Ensure the CODA driver is in your kernel (CONFIG_VIDEO_CODA added in Linux 3.16) and that firmware was loaded: {{{#!bash ~# dmesg | grep coda [ 16.721698] coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2 [ 16.721724] coda 2040000.vpu: Falling back to syfs fallback for: vpu_fw_imx6q.bin [ 18.381136] coda 2040000.vpu: Using fallback firmware vpu/vpu_fw_imx6q.bin [ 18.433648] coda 2040000.vpu: Firmware code revision: 570363 [ 18.433683] coda 2040000.vpu: Initialized CODA960. [ 18.433706] coda 2040000.vpu: Firmware version: 3.1.1 [ 18.442312] coda 2040000.vpu: codec registered as /dev/video[9-10] ~# cat /sys/class/video4linux/video9/name coda-encoder ~# cat /sys/class/video4linux/video10/name coda-decoder }}} * Ensure GStreamer decode elements exists: {{{#!bash ~# gst-inspect-1.0 | grep -e "v4l2.*dec" video4linux2: v4l2mpeg4dec: V4L2 MPEG4 Decoder video4linux2: v4l2mpeg2dec: V4L2 MPEG2 Decoder video4linux2: v4l2h264dec: V4L2 H264 Decoder }}} * The CODA960 decoder outputs NV12/I420/YV12 YUV pixel formats * The CODA driver requires CMA memory for buffers when it is used. Make sure to provide the kernel with enough CMA memory with the kernel command-line (ie 'cma=64M' for 64MB which should be enough) Note that the following examples assume you are using raw video encoded files, not container formats used for mimxed multimedia types (audio + video) such as ogg, avi, or mov (Quicktime). For information on de-muxing container formats see [#multimedia] Examples: * Decoding H264 video might look like this: {{{#!bash gst-launch-1.0 filesrc location=file.h264 ! h264parse ! v4l2h264dec ! kmssink }}} * Decoding MPEG4 video might look like this: {{{#!bash gst-launch-1.0 filesrc location=file.mp4v ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink }}} For more examples including working with 'multimedia' files that contain both audio and video see [#multimedia] [=#gstreamer-alsa] [=#audio] = GStreamer Audio The GStreamer {{{alsasrc}}} and {{{alsasink}}} provide audio capture and playback to Advanced Linux Sound Architecture (ALSA) devices. ALSA is the modern device-driver API for the Linux kernel. The Gstreamer {{{alsasrc}}} and {{{alsasink}}} can be passed a {{{device}}} property which can represent either the device name (reported from {{{aplay -L}}} for playback, or {{{arecord -L}}} for record) (ie {{{sysdefault:CARD=sgtl5000audio}}}) or the hw:x,y notation (ie {{{hw:0,0}}} for first device, first subdevice) If not specified via the {{{device}}} property, the device used for alsasrc and alsasink will depend on {{{/etc/asound.conf}}} and/or {{{~/.asoundrc}}}. See [wiki:ventana/audio#devices here] for details on specifying audio input and output devices on the Ventana product family. Playback and Capture also depends on your ALSA mixer ({{{amixer}}}) settings. See [wiki:ventana/audio#Mixermuxgainvolume mixer settings] for more info. [=#audio-output] == Audio Output via Linux ALSA driver and GStreamer alsasink Generally, a 'sink' element is one that will take an audio stream and send it to an audio device. Please refer to [wiki:ventana/audio this audio] page for more details on audio devices on the ventana platform. This {{{alsasink}}} element can accept any of the following formats via {{{audio/x-raw}}}: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE Examples: * Generate and play a 1kHz tone to the 'hw:0,0' device: {{{#!bash gst-launch-1.0 audiotestsrc ! alsasink device="hw:0,0" }}} [=#audio-input] == Audio Input via Linux ALSA driver and GStreamer alsasrc An input source is anything coming from a capture device on the SBC, e.g. HDMI audio in/analog audio in. Please refer to [wiki:ventana/audio this audio] page for more details on audio devices on the ventana platform. The {{{alsasrc}}} element can output the following source types in {{{audio/x-raw}}}: S8, U8, S16LE, S16BE, U16LE, U16BE, S24_32LE, S24_32BE, U24_32LE, U24_32BE, S32LE, S32BE, U32LE, U32BE, S24LE, S24BE, U24LE, U24BE, S20LE, S20BE, U20LE, U20BE, S18LE, S18BE, U18LE, U18BE, F32LE, F32BE, F64LE, F64BE Examples: * Capture 32kHz stereo 16bit audio from the 'hw:0,0' device and mux into into an AVI file: {{{#!bash gst-launch-1.0 alsasrc device="hw:0,0" ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \ audioconvert ! avimux ! filesink location=./audio.avi }}} [=#audio-test] == audiotestsrc The {{{audiotestsrc}}} is a very useful element for testing. It can output 16-64bit {{{audio/x-raw}}}: S16LE, S32LE, F32LE, F64LE This can output audio signals from {{{sine}}} to {{{violet-noise}}}. Examples: * Generate and play a 1kHz tone to the 'hw:0,0' device: {{{#!bash gst-launch-1.0 audiotestsrc ! alsasink device="hw:0,0" }}} [=#audio-encoding] == Audio Encoding Encoding is the term used to capture audio and encode it to a new format type. Commonly used GStreamer Audio encoders: * vorbisenc - encode {{{audio/x-raw}}} F32LE to {{{audio/x-vorbis}}} * mulawenc - encode {{{audio/x-raw}}} S16LE to {{{audio/x-mulaw}}} * wavenc - encode {{{audio/x-raw}}} S32LE, S24LE, S16LE, U8, F32LE, F64LE, {{{audio/x-alaw}}}, and {{{audio/x-mulaw}}} to {{{audio/x-wav}}} * alawenc - encode {{{audio/x-raw}}} S16LE to {{{audio/x-alaw}}} * flacenc - encode {{{audio/x-raw}}} S24LE, S24_32LE, S16LE, S8 to {{{audio/x-flac}}} * lamemp3enc - encode {{{audio/x-raw}}} S16LE to {{{audio/mpeg}}} There are many more. You can search for your specified one by running a similar search: {{{gst-inspect-1.0 | grep enc}}}. Note that each encoder has its own limits on the samples it accepts (S16LE, S24LE etc). You can find the formats accepted via {{{gst-inspect-1.0 }}} and you can use the [#audioconvertaudioconvert] element to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements. Examples: * Capture audio from the 'hw:0,0' device, encode it using the MPEG Layer 3 audio codec and store it to a file: {{{#!bash gst-launch-1.0 alsasrc device="hw:0,0" ! audioconvert ! lamemp3enc ! filesink location=file.mp3 }}} [=#audio-decoding] == Audio Decoding Decoding is the term used to decode an encoded audio stream to a raw audio stream. Commonly used GStreamer Audio decoders: * mpg123audiodec - decode {{{audio/mpeg}}} to {{{audio/x-raw}}} * vorbisdec - decode {{{audio/x-vorbis}}} to {{{audio/x-raw}}} * ivorbisdec - decode {{{audio/x-vorbis}}} to {{{audio/x-raw}}} * a52dec - decode {{{audio/x-ac3}}}, {{{audio/ac3}}}, and {{{audio/x-private1-ac3}}} to {{{audio/x-raw}}} * mulawdec - decode {{{audio/x-mulaw}}} to {{{audio/x-raw}}} * alawdec - decode {{{audio/x-alaw}}} to {{{audio/x-raw}}} * flacdec - decode {{{audio/x-flac}}} to {{{audio/x-raw}}} Note that each decoder has its own limits on the samples it produces at its src pad (S16LE, S24LE etc). You can find the formats accepted via {{{gst-inspect-1.0 }}} and you can use the [#audioconvert audioconvert] element to convert audio samplerate / bitwidth / format (via software algorithms) as needed between two elements. Examples: * Decode MPEG Layer 3 audio and playback to the default audio device with the following: {{{#!bash gst-launch-1.0 filesrc location=file.mp3 ! mpegaudioparse ! mpg123audodec ! alsasink }}} [=#audio-format] [=#audioconvert] == Audio Formats and Conversion Audio samples specified by the {{{audio/x-raw}}} vary between bit width and endieness as well as the sampling rate (how many times a second audio is measured) and the number of channels that are sampled. To specify the details of the audio format, you use a [#capsfilter GStreamer capsfilter] and to convert it from one format to another you use the GStreamer {{{audioconvert}}} element. Examples: * specify the sample-rate (32kHz), channels (2:stereo), and bit depth (16bit) and capture to an AVI file: {{{#!bash gst-launch-1.0 alsasrc device="hw:0,0" ! "audio/x-raw,rate=32000,channels=2,depth=16" ! \ audioconvert ! avimux ! filesink location=./audio.avi }}} [=#audio-loopback] == Audio loopback (Useful for testing audio input and output) A simple aduio loopback test can take audio input from an input device and output it to an output device. loopback audio from the first audio card: {{{#!bash gst-launch-1.0 alsasrc device="hw:0,0" ! alsasink device="hw:0,0" }}} To send audio from the sgtl5000 (analog input) to the imxhdmisoc (HDMI output): * using device/function numbers (ie from {{{aplay -l}}}): {{{#!bash gst-launch-1.0 alsasrc device="hw:0,0" ! alsasink device="hw:2,0" }}} * or, using names (ie from {{{aplay -L}}}): {{{#!bash gst-launch-1.0 alsasrc device="sysdefault:CARD=sgtl5000audio" ! \ alsasink device="sysdefault:CARD=imxhdmisoc" }}} To send audio from the tda1997x HDMI receiver (digital input) to the imxhdmisoc (digital output): {{{#!bash gst-launch-1.0 alsasrc device="sysdefault:CARD=tda1997xaudio" ! \ "audio/x-raw,rate=44100" ! alsasink device="sysdefault:CARD=imxhdmisoc" }}} * Note here we need to specify the audio sample-rate as it can vary per input stream (and gstreamer does not validate the rate). This rate must match the source stream samplrate which can be found via sysfs {{{/sys/bus/i2c/drivers/tda1997x/2-0048/audmode}}}. If your output device requires a different sample-rate than the source input device, you need to perform a sample-rate conversion. [=#multimedia] = Multimedia (Video and Audio combined) Handling media content that has both audio and video stream types requires a slightly more complex pipeline than the standard [#audio audio] and [#video video] pipeline examples. Generally, a mixed media pipeline which consumes multimedia will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and {{{queue}}} elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data). Conversely if producing multimedia content your pipeline will consist of a muxer to join audio and video streams and {{{queue}}} elements. [=#named-elements] == Named Elements, queues, and Multiple pipelines with gst-launch When mixing audio and video elements with {{{gst-launch}}} one must make use of multiple pipelines using {{{named elements}}}. The {{{name}}} property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any). Multiple pipelines can be provided to {{{gst-launch}}} and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'. This is best explained with some examples: * Encoding a stream with audio and video content into an AVI file: {{{#!bash gst-launch-1.0 \ videotestsrc \ ! $VIDEO_CAPABILITIES \ ! mux. \ audiotestsrc \ ! $AUDIO_CAPABILITIES \ ! mux. \ avimux name=mux \ ! filesink location=test.avi }}} - The {{{videotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} - The {{{audiotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} - The {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} * Decoding a Matroska container file containing H264 video and AC3 audio {{{#!bash gst-launch-1.0 \ filesrc location=file.mkv \ ! matroskademux name=demux \ demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \ demux. ! queue ! h264parse ! v4l2h264dec ! kmssink }}} - The {{{filesrc}}} pipeline ends with {{{name=demux}}} which means the output of this pipeline will be sent to all pipelines with a {{{demux.}}} source (who's types have been successfully negotiated) - The audio pipeline consisting of the {{{ac3parse}}} element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3) - The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx) - Queue elements are used to keep one pipeline or element from blocking another. For example, if the {{{v4l2h264dec}}} element needs more data from the demux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering * Decoding a MOV file containing H264 video and AAC audio: {{{#!bash gst-launch-1.0 \ filesrc location=file.mov \ ! qtdemux name=demux \ demux. ! queue ! aacparse ! avdec_aac ! alsasink \ demux. ! queue ! h264parse ! v4l2h264dec ! kmssink }}} [=#mux] == Muxing Mixed Content Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'. You can use {{{gst-inspect}}} to see a list of most of these using grep: {{{#!bash gst-inspect-1.0 | grep -i muxer | grep -vi de }}} Some common examples: - mpegtsmux: MPEG Transport Stream Muxer - mpegpsmux: MPEG Program Stream Muxer - matroskamux: Matroska muxer - avimux: Avi muxer - qtmux: !QuickTime Muxer - oggmux: Ogg muxer To mux mixed content together include one of these elements following the audio and video pipelines. Examples: * Encoding a stream with audio and video content into an AVI file: {{{#!bash gst-launch-1.0 \ videotestsrc \ ! $VIDEO_CAPABILITIES \ ! mux. \ audiotestsrc \ ! $AUDIO_CAPABILITIES \ ! mux. \ avimux name=mux \ ! filesink location=test.avi }}} - the {{{videotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated. - the {{{audiotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated. - the {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} and it understands how to multiplex the two types of data together into its output which is written to the file test.avi [=#demux] == De-muxing mixed content Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'. You can use {{{gst-inspect}}} to see a list of most of these using grep: {{{#!bash gst-inspect-1.0 | grep -i 'de\?muxer' }}} Some common examples: - tsparse: MPEG transport stream parser - tsdemux: MPEG transport stream demuxer - matroskademux: Matroska demuxer - avidemux: Avi demuxer - qtdemux: !QuickTime demuxer - oggdemux: Ogg demuxer To demux mixed content include one of these elements following the audio and video pipelines. Unlike muxing you also need to use a {{{parser}}} element to parse the bitstream and break it into discrete buffers (frames) that the downstream decoder expects. In other words this 'frames' the data going to decoders. Some common parsers: - ogmaudioparse: OGM audio stream parser - ogmvideoparse: OGM video stream parser - aacparse: AAC audio stream parser - amrparse: AMR audio stream parser - ac3parse: AC3 audio stream parser - flacparse: FLAC audio parser - mpegaudioparse: MPEG1 Audio Parser - h263parse: H.263 parser - h264parse: H.264 parser - mpegvideoparse: MPEG video elementary stream parser - mpeg4videoparse: MPEG 4 video elementary stream parser - pngparse: PNG parser - vc1parse: VC1 parser Examples: * Demuxing a Matroska container file containing H264 video and AC3 audio into its raw components: {{{#!bash gst-launch-1.0 \ filesrc location=file.mkv \ ! matroskademux name=demux \ demux. ! queue ! ac3parse ! filesink location=file.ac3 \ demux. ! queue ! h264parse ! filesink location=file.h264 }}} * Demuxing a MOV file containing H264 video and AAC audio into its raw components: {{{#!bash gst-launch-1.0 \ filesrc location=file.mov \ ! qtdemux name=demux \ demux. ! queue ! aacparse ! filesink location=file.aac \ demux. ! queue ! h264parse ! filesink locatino=file.h264 }}} [=#bin] == bin elements A '''bin''' element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept. Sometimes the bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream. The {{{gst-launch}}} application provides a couple of useful debugging tools that can help with this: * using {{{GST_DEBUG_DUMP_DOT_DIR}}} and Graphviz * adding the '-v' parameter to gst-launch will provide verbose feedback on the pipeline configuration that can tell you what is going on See [#troubleshooting] for more details on these methods. [=#playbin] === GStreamer playbin GStreamer {{{playbin}}} element attempts to create a pipeline that will play both the audio and video portions of a file. For example: {{{#!bash gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi }}} The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by: {{{#!bash gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=hw:1,0" }}} Please type {{{gst-inspect-1.0 playbin}}} to see more options. [=#decodebin] === GStreamer decodebin The GStreamer {{{decodebin}}} element is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under [#ex1 the first example] with the following: {{{#!bash gst-launch-1.0 \ filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \ matroskademux name=d \ d. ! queue ! ivorbisdec ! queue ! alsasink device=hw:1,0 \ d. ! queue ! decodebin ! queue ! kmssink }}} Note that {{{decodebin}}} doesn't always choose the correct decoder, so be wary of this. It is similar to {{{playbin}}} in that it aids in creating a dynamic pipeline. [=#gst-play] === GStreamer gst-play The stand-alone application {{{gst-play}}} is is a program that utilizes the {{{playbin}}} element and thus can be used for playback of many file types. The above example {{{gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi}}} can be replaced with: {{{#!bash gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi }}} [=#streaming] = Streaming (send multimedia to or receive from Network) GStreamer has elements that allow for network streaming to occur. There are several ways to accomplish networked streaming over Internet Protocol (IP): * [#udp Raw UDP/IP] * [#tcp Raw TCP/IP] * [#rtp Real-time Transport Protocol (RTP)] * [#rtsp Real Time Streaming Protocol (RTSP)] ('''recommended''') * [#abs Adaptive Bitrate Streaming] [=#udp] == Raw UDP Using UDP/IP is the simplest mechanism for streaming and utilizes the least amount of bandwidth. Because UDP does not provide any error detection, packet ordering, or error correction the bitrate is deterministic and simply the bitrate of the media you are streaming. The limitations of raw UDP is: * requires codec that can handle missing/corrupt data (most do these days) * does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media) * does not fragment packets - will try to send a raw udp packet for whatever size buffer the udpsink is passed (which can lead to pipeline errors). To fragment packets use RTP The only benefit of using raw UDP is that it is the simplest pipeline you can create for streaming and requires the least amount of dependencies (albeit you might run into one or all of the above problems). '''Note that it is recommended that you use [#rtp RTP] or [#rtsp RTSP] unless you know exactly what you are doing to overcome the limitations listed above''' The {{{udpsrc}}} element can be used to render/save a stream originated from a {{{udpsink}}} pipeline. Examples: * encode and send H264 video from Ventana: 1. Start decoder first: {{{ #!bash ifconfig eth0 192.168.1.1 gst-launch-1.0 udpsrc port=9001 ! h264parse ! v4l2h264dec ! kmssink sync=false }}} 2. Start encoder second: {{{ #!bash ifconfig eth0 192.168.1.2 gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc ! udpsink host=192.168.1.1 port=9001 }}} Notes: * On the client (stream receiver and renderer) you must use the {{{sync=false}}} property to render frames as they are received otherwise the stream will stall because their is no headers containing timestamps * the decoder (udpsrc) needs to be started first because udpsink will fail if nothing is listening to the socket [=#tcp] == TCP Using TCP/IP brings error detection, packet re-ordering, and error correction to the network stream. This however causes the bitrate to be non-deterministic because as the error rate increases so does the bitrate and latency. The limitations of using TCP: * non-deterministic bitrate * added latency * does not use headers containing payload type or timestamp info on stream (making it suitable for only a single type of media, or a pre-muxed type of media) '''Note that it is recommended that you use [#rtp RTP] or [#rtsp RTSP] unless you know exactly what you are doing to overcome the limitations listed above''' TCP/IP introduces the concept of a socket connection therefore there must exist a server and a client in which case the server must be started first to listen for a connection. You can use a server sink or a server source. The {{{tcpserversrc}}} source can be used to create a TCP server that waits for a connection from a {{{tcpclientsink}}} to render/save. Alternatively the {{{tcpserversink}}} sink can be used to create a TCP server that waits for a connection from a {{{tcpclientsrc}}} that will send data. Examples: * encode and send H264 video from Ventana with '''decoder as server''': 1. Start decoder (server) first: {{{ #!bash ifconfig eth0 192.168.1.1 gst-launch-1.0 tcpserversrc host=0.0.0.0 port=9001 ! decodebin ! autovideosink sync=false }}} 2. Start encoder (client) second: {{{ #!bash ifconfig eth0 192.168.1.2 ENCODER=/dev/$(for i in $(ls -d /sys/class/video4linux/video*); do [ "coda-encoder" = "$(cat $i/name)" ] && basename $i; done) gst-launch-1.0 videotestsrc is-live=true ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! tcpclientsink host=192.168.1.1 port=9001 }}} Notes: * TCP is connection oriented therefore the TCP 'server' must be started first. You can choose your elements such that the stream originator is the server or the stream renderer is the server however doing so can be problematic for certain codecs because the client decoding the stream may pick up the stream somewhere in the middle and not know how to parse it. * the {{{host=0.0.0.0}}} property means listen to all network interfaces [=#rtp] == RTP (raw/session-less) The [https://en.wikipedia.org/wiki/Real-time_Transport_Protocol Real-time Transport Protocol (RTP)] is a network protocol for delivering audio and video over IP networks. RTP is used extensively in communication and entertainment systems that involve streaming media, such as telephony, video teleconference applications, television services and web-based push-to-talk features. The RTP packet type encapsulates multimedia data with a payload type and time-stamp and therefore can be used to compensate for jitter, out of sequence packets, and time synchronization between streams of different types (ie audio/video lip-sync). RTP is typically used in conjunction with other protocols such as RTP Control Protocol (RTCP) and [#rtsp Real Time Streaming Protocol (RTSP)] to manage stream sessions however can be used on its own in a raw session-less fashion using {{{udpsink}}} and {{{udpsrc}}} elements. The limitations of using raw/session-less RTP: * session management needs to be handled manually (capsfilter is needed to specify stream format) '''Note that it is recommended that you use [#rtsp RTSP] unless you know exactly what you are doing to overcome the limitations listed above''' === Example: Capture, encode and stream H264 via RTP with GStreamer playback: * Encode and send H264 video from Ventana: 1. Start decoder first: {{{ #!bash ifconfig eth0 192.168.1.1 gst-launch-1.0 udpsrc port=9001 \ caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" \ ! decodebin ! autovideosink }}} 2. Start encoder second: {{{ #!bash gst-launch-1.0 videotestsrc is-live=true \ ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=1000;" ! rtph264pay ! udpsink host=192.168.1.1 port=9001 }}} Notes: * when using RTP a capsfilter must be used to specify the payload as application/x-rtp as above. You can determine the capsfilter required by starting the encoder with a verbose flag {{{-v}}} and looking for {{{caps = "application/x-rtp"}}} === Example: Capture, encode and stream H264 via RTP with VLC playback: * Encode and send H264 video from Ventana to a PC with VLC: 1. Start decoder first: a. Create SDP file like below (IP address in example is that of the Ventana board) {{{ v=0 m=video 5000 RTP/AVP 96 c=IN IP4 172.24.20.207 a=rtpmap:96 H264/90000 }}} b. Open SDP file in VLC 2. Start encoder (Ventana) second: (IP address in below example is IP of the PC) {{{ gst-launch-1.0 videotestsrc ! v4l2h264enc ! rtph264pay config-interval=3 ! udpsink host=172.24.20.26 port=5000 }}} [=#rtsp] == RTSP (Real Time Streaming Protocol) '''(recommended)''' The [https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol Real Time Streaming Protocol (RTSP)] is a network control protocol designed for use in entertainment and communications systems to control streaming media servers. The protocol is used for establishing and controlling media sessions between end points. Clients of media servers issue VCR-style commands, such as play and pause, to facilitate real-time control of playback of media files from the server. This protocol uses the Real-time Transport Protocol (RTP) in conjunction with Real-time Control Protocol (RTCP) for media stream delivery. The limitations of using RTSP are: * {{{gst-inspect}}} has no way of using a simple pipeline to create an RTSP server - you must create or use an existing gstreamer based application (keep reading below). Creating an RTSP server is not possible via a simple pipeline to gst-launch however GStreamer libraries do exist to make writing an RTSP server trivial. The source for gst-rtsp-server contains an example application [http://cgit.freedesktop.org/gstreamer/gst-rtsp-server/tree/examples/test-launch.c test-launch.c] which provides a simple example that can take a GStreamer 'bin' element consisting of everything but the sink element and serves it via RTSP. Gateworks has extended this with [#gst-variable-rtsp-server gst-variable-rtsp-server] which demonstrates how to auto-adjusting properties like the encoding bitrate depending on the number of clients connected in addition to serving as a fairly simple example of how to write a gstreamer application. [=#troubleshooting] = Troubleshooting If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device. You can see all the properties of a particular element with the 'gst-inspect ' command. You can see a list of all elements with 'gst-inspect' without arguments. You can enable verbose output for GStreamer by adding the '-v' flag which will show you the negotiated pipeline details and the state machine details. You can enable debug output for GStreamer by using the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-debug.html '--gst-debug'] parameter to gst-launch or by setting the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html GST_DEBUG] environment variable. For example, if you wanted to see DEBUG level messages for the {{{videotestsrc}}} element you can set GST_DEBUG=videotestsrc:4 [=#filter-graph] == GST_DEBUG_DUMP_DOT_DIR You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause {{{gst-launch}}} to output a {{{.dot}}} file for each phase of the pipeline then use a tool such as {{{Graphviz}}} to visualize the {{{.dot}}} file. Example: * use playbin to playback a file: {{{#!bash root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi }}} - hit Cntl-C after decoding starts to exit early * see the dot files created: {{{#!bash root@ventana:~# ls /tmp/dot 0.00.00.108710334-gst-launch.NULL_READY.dot 0.00.00.490807334-gst-launch.READY_PAUSED.dot 0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot 0.00.03.254000001-gst-launch.PAUSED_READY.dot }}} * transfer to a PC and use something like {{{xdot}}} to view: {{{#!bash xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot }}} - zoom in along the graph and you can see that: - {{{GstFileSrc}}} is the source, - {{{GstAviDemux}}} is used to demux to audio/x-ac3, - {{{GstAc3Parse}}} is used to parse the audio into audio frames, - {{{GstMpeg4VParse}}} is used to parse the video into video frames, - {{{GstV4l2VideoDec}}} is used to decode the video from video/mpeg to video/x-raw, - {{{ GstA52Dec}}} is used to decode the audio from audio/x-ac3 to audio/x-raw, - etc - Note that some hunting with {{{gst-inspect}}} must be done to determine what elements coorespond to the above class names Reference: - http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+11%3A+Debugging+tools == gst-launch -v The verbose debugging from {{{gst-launch -v}}} can show you the negotiation that takes place as a pipeline moves through its stages. Example: {{{#!bash gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi }}} examining the verbose output can show you the following: * container: AVI: avidemux * video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! v4l2mpeg4dec * audio: AC3 48khz 5channels: ac3parse ! a52dec Therefore you can use these pipelines to decode and play: * video only (output to kmssink) {{{#!bash gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink }}} * audio only (output to hdmi audio sink) {{{#!bash gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc" }}} * both audio and video {{{#!bash gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \ d. ! queue ! mpeg4videoparse ! v4l2mpeg4dec ! kmssink \ d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc" }}} == streaming If you're having issues with network streaming: * Verify that both sides can ping one another * If the message {{{There may be a timestamping problem, or this computer is too slow}}} appears and the video display appears choppy, try the following: * Lower the bitrate from the server * Place a {{{sync=false}}} on the sink side of the server and client. * If video appears choppy, try using UDP over TCP. * Verify that the network is not congested. * Verify your gstreamer pipeline is correct. The best way to find the element that causes a negotiation failure is to end your pipeline in a fakesink and one-by-one eliminate elements leading up to it until it negotiates successfully.