[[PageOutline]] = GStreamer on Gateworks SBCs = The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These '''GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs)''', specifically the Ventana family that utilizes the Freescale i.MX6 processors. Gateworks SBCs can be viewed at the following link: [http://www.gateworks.com] [[Image(http://trac.gateworks.com/raw-attachment/wiki/OpenWrt/wireless/relayd/gw5100realsmall.png,200px)]] = GStreamer = [http://GStreamer.freedesktop.org GStreamer] is an opensource library created to handle multimedia. Pipelines are used to source and sink media however you would like (e.g. decoding the mp3 out of a video file and playing it back through speakers) [=#plugins] == Plugins == GStreamer works on 'plugins'. A plugin comprises of elements that can do work on a media stream. For example, the {{{mpeg2dec}}} can decode mpeg1 and mpeg2 video streams. By default, we install several GStreamer plugins that can tap into the i.MX6 processor's hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these plugins is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves your CPU free to deal with other programs etc. There are two main versions of GStreamer: 0.10 and 1.0. Our Yocto 'master' branch and Yocto v1.8 'fido' branch include GStreamer 1.0 while Yocto versions prior (Yocto v1.7 (dizzy), Yocto v1.6 (daisy), Yocto v1.3 (danny)) include only GStreamer 0.10. Different sets of plugins exist for each main version of GStreamer. The next few subsections will attempt to showcase plugins that utilize the i.MX6. [=#gstreamer-imx] === GStreamer-imx plugins (for GStreamer 1.0+) === The GStreamer-imx plugins project is a community project created by Carlos Rafael Giani (). These are always being updated which might make them good to use if a certain feature is only implemented in one of the below. Pros: * Open Source Development model: The project is on github and is very active * The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community * The source is very well documented and easy to follow * Things are done in a very standard GStreamer way Cons: * These are relatively new (early 2014) so there might be undiscovered bugs The following is a list of plugins provided by the latest version of gstreamer-imx (0.11.0) ||= Type =||= Plugin(s) =||= Element(s) =||= Comments =|| || Audio Decoder || imxaudio || imxuniaudiodec || Uses i.MX uniaudio codecs for decoding || || Audio Encoder || imxaudio || imxmp3audioenc || Uses i.MX for MP3 audio encoding || || Device Sources || imxv4l2video || imxv4l2videosrc || Get camera source via v4l2 || || Video Decoder || imxvpu || imxvpudec || VPU Based decoder || || Video Encoder || imxvpu || imxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263 || VPU Based encoders || || Video Render (sink) || imxg2d; imxpxp; imxeglvivsink; imxipu || imxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosink || g2d^1^, ipu^1^, pxp^2^, and egl (overlay) video sinks || || Video Converter || imxg2d; imxpxp; imxipu || imxg2dvideotransform; imxpxpvideotransform; imxipuvideotransform || g2d, pxp, egl and ipu video filter/converter/scalars^3^ || || Video Compositing || imxg2d; imxipu || imxg2dcompositor, imxipucompositor || gpu/ipu accelerated compositing || ,,1. The g2d sink is very flexible in the types of input video it can take, but doesn't have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it's input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.,, [[BR]] ,,2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.,, [[BR]] ,,3. Please see note 1 above.,, [[BR]] Please uncollapse the following if you want to view gstreamer plugins specific to the imx6 previous to Yocto v1.8 (Fido). [[CollapsibleStart(Older GStreamer Plugins (Supported previous to Yocto 1.8))]] [=#gst-fsl-plugin] === Freescale GStreamer Plugins (fsl-gst-plugin) === The gst-fsl-plugin package plugins are written by Freescale. While they are open-source (release source tarballs are available) the development cycle is closed and there is no community involvement. Pros: * These have been available for some time now and many examples can be found on the Internet Cons: * There is no community involvement in the development of this code * The code is a bit difficult to follow * They do not follow the gstreamer model very well and do things in a very unorthodox way that don't mix well with the rest of GStreamer: * their decoders rely on their 'aiur' demuxer to do stream parsing * they do not use the PTS and DTS timestamps that come for upstream and instead re-generate their own * they are not very compatible with other gstreamer elements (such as ones that are not DMA buffer aware) * they do not communicate with the GStreamer development community [=#gstreamer-0.10_gst-fsl-plugin-3.0.11] ==== Freescale GStreamer 0.10 (plugin version 3.0.11) plugins ==== These plugins are available in the Gateworks Yocto 1.3, 1.6, and 1.7 BSP's. For a very good reference for this version of the gst-fsl-plugins see the i.MX_Series_Yocto_Project_Multimedia_Users's_Guide.pdf and i.MX_6_Series_Yocto_Project_Multimedia_Release_Notes.pdf found in [http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX6Q&fpsp=1&tab=Documentation_Tab L3.10.17_1.0.0_LINUX_DOCS] download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading. ||= Type =||= Plugin(s) =||= Element(s) =||= Comments =|| || Audio Decoder || beep.imx;amrdec.imx || beepdec; beepdec.vorbis; beepdec.mpr; beepdec.aac; mfw_amrdecoder || -- || || Audio Encoder || mp3enc.imx || mfw_mp3encoder || Encode audio to mp3 || || Audio Converter || audiopeq.imx || mfw_audio_pp || Equalizer || || Video Decoder || vpu.imx; h264.imx || vpudec; mfw_h264decoder || VPU Based decoder and h264 video decoder || || Video Encoder || vpu.imx || vpuenc || VPU Based encoder || || Demuxer || aiur.imx || aiurdemux || Universal Demuxer || || Video Render (sink) || v4lsink.imx; isink.imx || mfw_v4lsink; mfw_isink || Video4Linux2 and IPU based video sinks || || Device Source(src) || tvsrc.imx; v4lsrc.imx || tvsrc; mfw_v4lsrc || tvsrc: v4l2 tv src; mfw_v4lsrc: v4l2 camera src || If there are any elements that you require more information on, please run a 'gst-inspect-0.10 '. [=#gstreamer-0.10_gst-fsl-plugin-4.0.2] ==== Freescale GStreamer 0.10 (plugin version 4.0.2) plugins ==== These plugins are available in the Gateworks Yocto master branch and will be in the Gateworks Yocto 1.8 BSP. For a very good reference for this version of the gst-fsl-plugins see the i.MX_Linux_Release_notes.pdf foudn in [http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX6Q&fpsp=1&tab=Documentation_Tab L3.10.53_1.1.0_LINUX_DOCS] download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading. ||= Type =||= Plugin(s) =||= Element(s) =||= Comments =|| || Audio Decoder || beep.imx;amrdec.imx || beepdec; beepdec.vorbis; beepdec.mpr; beepdec.aac; mfw_amrdecoder || -- || || Audio Encoder || mp3enc.imx || mfw_mp3encoder || Encode audio to mp3 || || Audio Converter || audiopeq.imx || mfw_audio_pp || Equalizer || || Video Decoder || vpu.imx || vpudec || VPU Based decoder || || Video Encoder || vpu.imx || vpuenc || VPU Based encoder || || Demuxer || aiur.imx || aiurdemux || Universal Demuxer || || Video Render (sink) || v4lsink.imx; isink.imx || imxv4l2sink; mfw_isink || Video4Linux2 and IPU based video sinks || || Device Source(src) || tvsrc.imx; v4lsrc.imx || tvsrc; imxv4l2src || tvsrc: v4l2 tv src; imxv4l2src: v4l2 camera src || If there are any elements that you require more information on, please run a 'gst-inspect-0.10 '. [=#gstreamer-1.0_gst-fsl-plugin-4.0.2] ==== Freescale GStreamer 1.0 (plugin version 4.0.2) plugins ==== These plugins are available in the Gateworks Yocto master branch and will be in the Gateworks Yocto 1.8 BSP. For a very good reference for this version of the gst-fsl-plugins see the i.MX_Linux_Release_notes.pdf foudn in [http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX6Q&fpsp=1&tab=Documentation_Tab L3.10.53_1.1.0_LINUX_DOCS] download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading. ||= Type =||= Plugin(s) =||= Element(s) =||= Comments =|| || Audio Decoder || beep.imx;amrdec.imx || beepdec || MP3, AAC, WMA, AC3, etc. "Universal" || || Audio Encoder || imxmp3enc.imx || imxmp3enc || Encode audio to mp3 || || Video Decoder || vpu.imx || vpudec || VPU Based decoder || || Video Encoder || vpu.imx || vpuenc || VPU Based encoder || || Demuxer || aiur.imx || aiurdemux || AVI, MKV, MP4, MPEG4, ASF, OGG, FLV, WebM Demux || || Video Render (sink) || imxv4l2.imx; overlaysink.imx || imxv4l2sink; overlaysink || imxv4l2sink: v4l2 video sink; overlaysink: g2d based video sink || || Device Source(src) || imxv4l2.imx || imxv4l2src || imxv4l2src: v4l2 camera/TVin src || || Video Converter || imxvideoconvert.imx || imxvideoconvert_ipu; imxvideoconvert_g2d || IPU/G2D based video converters (i.e. colorspace etc) || || AVB Streaming || avb.imx || avbpcmsink; avbpcmsrc, avbmpegtssink; avbmpegtssrc || Please use 'gst-inspect-1.0' on element you're interested in || [[CollapsibleEnd]] If there are any elements that you require more information on, please run a 'gst-inspect-1.0 '. == Codecs == For Codec support (Encoder / Decoder) details please see: * [wiki:ventana/codec Ventana Codecs] [=#pipelines] == Pipeline Construction == [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-applications.html gst-launch] is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type {{{gst-inspect element}}} where {{{element}}} is the element you are looking for. To see a full listing of elements installed, you can type just {{{gst-inspect}}}. Here is a graphical example of a pipeline. [[BR]] [[Image(https://gstreamer.freedesktop.org/documentation/tutorials/basic/images/figure-1.png, 512px)]] The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next [=#test-media] === Test Media Files === There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing: * [https://peach.blender.org/download/ Big Buck Bunny] - Creative Commons * [https://durian.blender.org/download/ Sintel] - Creative Commons * [https://mango.blender.org/download/ Tears of Steel] - Creative Commons [=#video] === Video === Please see [wiki:Yocto/gstreamer/video this] page for some [wiki:Yocto/gstreamer/video video] pipeline examples. [=#audio] === Audio === Please see [wiki:Yocto/gstreamer/audio this] page for some [wiki:Yocto/gstreamer/audio audio] pipeline examples. [=#multimedia] === Multimedia === Please see [wiki:Yocto/gstreamer/multimedia this] page for some [wiki:Yocto/gstreamer/multimedia multimedia] pipeline examples. [=#streaming] === Streaming === Please see [wiki:Yocto/gstreamer/streaming this] page for some [wiki:Yocto/gstreamer/streaming streaming] pipeline examples. [=#latency] === Latency === Please see [wiki:Yocto/gstreamer/latency this] page for some [wiki:Yocto/gstreamer/latency latency] information. [=#compositing] === Video Composition === Please see [wiki:Yocto/gstreamer/compositing this] page for some [wiki:Yocto/gstreamer/compositing compositing] information. [=#applications] == Applications == GStreamer uses pipelines as a way to test elements in various ways. In a production environment, you generally want to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that. === Example Applications === GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html here] for more details. Further, Gateworks has created their own suite (currently only consisting of one application) of applications that help to show developers how to create GStreamer apps. [=#gst-gateworks-apps] ==== gst-variable-rtsp-server ==== [https://github.com/Gateworks/gst-gateworks-apps/blob/master/src/gst-variable-rtsp-server.c This application] is included by default on our Yocto 1.8+ BSPs (multimedia/gui images). It is an example GStreamer application that, you can pass in a custom pipeline to create an RTSP server out of, but extends the usage quite a bit in order to showcase GStreamer call-backs, among other things. It is a fully open-source program to provide you, the developer, a working example of a solution to some commonly asked questions. [https://github.com/Gateworks/gst-gateworks-apps/blob/master/src/gst-variable-rtsp-server.c gst-variable-rtsp-server] is a solution created by Gateworks that allows changing the quality of a live stream on the fly based on the number of clients connected to the server. Similar to other [wiki:Yocto/gstreamer/streaming#AdaptiveBitrateStreaming adaptive bitrate] technologies, it can change quality based on information is knows about the stream. In our case, the number of connected clients was deemed as appropriate. Note that this ability will only work if you are not supplying the full pipeline (e.g. not using the {{{-u}}} flag). This application can change either the {{{quant-param}}} or the {{{bitrate}}} parameters of the {{{imxvpuenc_h264}}} encoder. The {{{quant-param}}} will only be used if the pipeline is set to Variable Bitrate mode (VBR). This can be accomplished by passing in the {{{-b 0}}} flag to the program. Otherwise, {{{gst-variable-rtsp-server}}} will change the {{{bitrate}}} of the stream. The algorithm used to change both {{{bitrate}}} and {{{quant-param}}} are based on the {{{--steps}}} input (default to 5). For example, if using the default {{{steps}}} value of 5, if the min bitrate was 500 and max bitrate was 2000, it would take 5 clients to get from the best to the worst quality. To calculate a {{{bitrate}}} or {{{quant-param}}} for a number of clients or steps etc, please use the following formulas ('''integer math'''): 1. First get step factor: step = (max^1^ - min^2^) / (steps - 1), e.g. (51 - 0) / 1 = 51 1. Now, depending on the parameter, pick one: * quant-param * new_quant^3^ = ((clients_connected - 1) * step) + min^2^, e.g. ((2 - 1) * 51) + 0 = 51 * bitrate * new_bitrate^4^ = max^1^ - ((clients_connected - 1) * step), e.g. 10000 - ((2 - 1) * 9999) = 1 ,,1. max means the high number of that particular parameter,,[[BR]] ,,2. min means the low number of that particular parameter,,[[BR]] ,,3. Note that this is capped to the 'max' quant-param value,,[[BR]] ,,4. Note that this is capped to the 'min' bitrate value,,[[BR]] The formulas above slightly differ due to the fact that the lower the {{{bitrate}}}, the lower the quality, but with a higher {{{quant-param}}}, the lower the quality (inverted). Please see below for some example use cases: * Use {{{imxv4l2videosrc}}} as input stream on {{{/dev/video0}}} {{{ #!bash gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 }}} * Use {{{imxv4l2videosrc}}} as input stream on {{{/dev/video0}}}, but go from best to worst quality when 2 clients connect {{{ #!bash gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 --steps 2 }}} * Use {{{imxv4l2videosrc}}} as input stream on {{{/dev/video0}}}, cap max bitrate to 5000kbps and min bitrate to 2500kbps {{{ #!bash gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 --max-bitrate=5000 --min-bitrate=2500 }}} To connect as a client, you can use any program that can connect to an RTSP stream. Below is an example connecting with GStreamer: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! decodebin ! autovideosink }}} There are many more examples [wiki:Yocto/gstreamer/streaming#rtsp on our GStreamer streaming] page. With the above, if there is only a single client, the stream will be at it's best quality. But as you connect more clients, the quality of the stream will drop in order to account for network congestion. For more information, please see [https://github.com/Gateworks/gst-gateworks-apps#gst-variable-rtsp-server this README] section. [=#references] == References == * [http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+10%3A+gstreamer+tools gstreamer.com] contains much more information on the construction of pipelines. * [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html GStreamer Hello World Application explanation.] * [https://github.com/Gateworks/gst-gateworks-apps Gateworks Example Application Suite] [=#misc] = Miscellaneous = The below sections host some miscellaneous and loose information about GStreamer that didn't otherwise fit a specific category. [=#troubleshooting] == Troubleshooting == If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device. You can see all the properties of a particular element with the 'gst-inspect ' command. You can see a list of all elements with 'gst-inspect' without arguments. You can enable debug output for Gstreamer by using the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-debug.html '--gst-debug'] parameter to gst-launch or by setting the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html GST_DEBUG] environment variable. For example, if you wanted to see DEBUG level messages for the {{{videotestsrc}}} plugin you can set GST_DEBUG=videotestsrc:4 References: * http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-debug.html * http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html === Memory Limitation On SBCs with 512MB of memory, such as the GW5510, GW5100, GW5530, etc it is possible that a error appears related to memory. {{{ [WARN] VPU iram is less than needed, some parts don't use iram [ 6194.061104] mxc_vpu 2040000.vpu: Physical memory allocation error! [ 6194.067361] mxc_vpu 2040000.vpu: Physical memory allocation error! [ERR] mem allocation failed! }}} This requires increasing the CMA memory in the bootloader: {{{ setenv mem 'cma=96M' }}} The example of 96M may or may not be enough, and should be experimented with. More information about CMA can be found here: [wiki:ventana/memory Ventana Memory] === Video source loop An efficient way of testing multiple video sources is to use a loop. Beneath is a simple bash script that will do this: {{{#!bash for i in $(seq 1 8);do ##seq 1 8 represents the range to loop of enumerated /dev/video* devices gst-launch-1.0 imxv4l2videosrc device=/dev/video$i ! imxg2dvideosink & pid=$! echo displaying device $i sleep 5 kill $pid done }}} === gst-inspect-1.0 Gst-inspect-1.0 is included in our Yocto BSP. This is an excellent tool if you're unfamiar with the functionality of any Gstreamer module. Usage: {{{#!bash root@ventana:~# gst-inspect-1.0 imxv4l2videosrc ##imxv4l2videosrc is just an example this can be any gstreamer plugin. Factory Details: Rank primary (256) Long-name V4L2 CSI Video Source Klass Source/Video Description Capture video streams using V4L2 CSI interface Author Philip Craig Plugin Details: Name imxv4l2videosrc Description GStreamer i.MX V4L2 CSI video source Filename /usr/lib/gstreamer-1.0/libgstimxv4l2videosrc.so Version 0.11.1 License LGPL Source module gstreamer-imx Binary package Unknown package release Origin URL Unknown package origin GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstBaseSrc +----GstPushSrc +----GstImxV4l2VideoSrc Implemented Interfaces: GstURIHandler Pad Templates: SRC template: 'src' Availability: Always Capabilities: video/x-raw Element Flags: no flags set Element Implementation: Has change_state() function: gst_base_src_change_state Element has no clocking capabilities. URI handling capabilities: Element can act as source. Supported URI protocols: imxv4l2 Pads: SRC: 'src' Implementation: Has getrangefunc(): gst_base_src_getrange Has custom eventfunc(): gst_base_src_event Has custom queryfunc(): gst_base_src_query Has custom iterintlinkfunc(): gst_pad_iterate_internal_links_default Pad Template: 'src' Element Properties: name : The name of the object flags: readable, writable String. Default: "imxv4l2videosrc0" parent : The parent of the object flags: readable, writable Object of type "GstObject" blocksize : Size in bytes to read per buffer (-1 = default) flags: readable, writable Unsigned Integer. Range: 0 - 4294967295 Default: 4096 num-buffers : Number of buffers to output before sending EOS (-1 = unlimited) flags: readable, writable Integer. Range: -1 - 2147483647 Default: -1 typefind : Run typefind before negotiating flags: readable, writable Boolean. Default: false do-timestamp : Apply current stream time to buffers flags: readable, writable Boolean. Default: false capture-mode : Capture mode of camera, varies with each v4l2 driver, for example ov5460: ov5640_mode_VGA_640_480 = 0, ov5640_mode_QVGA_320_240 = 1, ov5640_mode_NTSC_720_480 = 2, ov5640_mode_PAL_720_576 = 3, ov5640_mode_720P_1280_720 = 4, ov5640_mode_1080P_1920_1080 = 5 flags: readable, writable Integer. Range: 0 - 2147483647 Default: 0 fps-n : Numerator of the framerate at whichthe input stream is to be captured flags: readable, writable Integer. Range: 0 - 2147483647 Default: 30 input : Video input selected with VIDIOC_S_INPUT flags: readable, writable Integer. Range: 0 - 2147483647 Default: 1 device : Device location flags: readable, writable String. Default: "/dev/video0" queue-size : Number of V4L2 buffers to request flags: readable, writable Integer. Range: 0 - 2147483647 Default: 6 crop-meta-x : X value for crop metadata flags: readable, writable Integer. Range: 0 - 2147483647 Default: 0 crop-meta-y : Y value for crop metadata flags: readable, writable Integer. Range: 0 - 2147483647 Default: 0 crop-meta-width : WIDTH value for crop metadata flags: readable, writable Integer. Range: 0 - 2147483647 Default: 0 crop-meta-height : HEIGHT value for crop metadata flags: readable, writable Integer. Range: 0 - 2147483647 Default: 0 }}} [=#info] == More Information == For more pipeline examples, please visit the [https://community.freescale.com/docs/DOC-93387 Freescale community site] for the i.MX6 processor.