Changes between Initial Version and Version 1 of Yocto/gstreamer


Ignore:
Timestamp:
10/22/2017 05:28:45 AM (7 years ago)
Author:
trac
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Yocto/gstreamer

    v1 v1  
     1[[PageOutline]]
     2
     3= GStreamer on Gateworks SBCs  =
     4The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These '''GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs)''', specifically the Ventana family that utilizes the Freescale i.MX6 processors.
     5
     6Gateworks SBCs can be viewed at the following link: [http://www.gateworks.com]
     7
     8[[Image(http://trac.gateworks.com/raw-attachment/wiki/OpenWrt/wireless/relayd/gw5100realsmall.png,200px)]]
     9
     10
     11= GStreamer =
     12[http://GStreamer.freedesktop.org GStreamer] is an opensource library created to handle multimedia. Pipelines are used to source and sink media however you would like (e.g. decoding the mp3 out of a video file and playing it back through speakers)
     13
     14
     15[=#plugins]
     16== Plugins ==
     17GStreamer works on 'plugins'. A plugin comprises of elements that can do work on a media stream. For example, the {{{mpeg2dec}}} can decode mpeg1 and mpeg2 video streams.
     18
     19By default, we install several GStreamer plugins that can tap into the i.MX6 processor's hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these plugins is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves your CPU free to deal with other programs etc.
     20
     21There are two main versions of GStreamer: 0.10 and 1.0. Our Yocto 'master' branch and Yocto v1.8 'fido' branch include GStreamer 1.0 while Yocto versions prior (Yocto v1.7 (dizzy), Yocto v1.6 (daisy), Yocto v1.3 (danny)) include only GStreamer 0.10.
     22
     23Different sets of plugins exist for each main version of GStreamer. The next few subsections will attempt to showcase plugins that utilize the i.MX6.
     24
     25[=#gstreamer-imx]
     26=== GStreamer-imx plugins (for GStreamer 1.0+) ===
     27The GStreamer-imx plugins project is a community project created by Carlos Rafael Giani (<dv@pseudoterminal.org>). These are always being updated which might make them good to use if a certain feature is only implemented in one of the below.
     28
     29Pros:
     30 * Open Source Development model: The project is on github and is very active
     31 * The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community
     32 * The source is very well documented and easy to follow
     33 * Things are done in a very standard GStreamer way
     34
     35Cons:
     36 * These are relatively new (early 2014) so there might be undiscovered bugs
     37
     38The following is a list of plugins provided by the latest version of gstreamer-imx (0.11.0)
     39||= Type              =||= Plugin(s)                            =||= Element(s)                                                       =||= Comments                                              =||
     40|| Audio Decoder       || imxaudio                               || imxuniaudiodec                                                     || Uses i.MX uniaudio codecs for decoding                  ||
     41|| Audio Encoder       || imxaudio                               || imxmp3audioenc                                                     || Uses i.MX for MP3 audio encoding                        ||
     42|| Device Sources      || imxv4l2video                           || imxv4l2videosrc                                                    || Get camera source via v4l2                              ||
     43|| Video Decoder       || imxvpu                                 || imxvpudec                                                          || VPU Based decoder                                       ||
     44|| Video Encoder       || imxvpu                                 || imxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263   || VPU Based encoders                                      ||
     45|| Video Render (sink) || imxg2d; imxpxp; imxeglvivsink; imxipu  || imxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosink   || g2d^1^, ipu^1^, pxp^2^, and egl (overlay) video sinks   ||
     46|| Video Converter     || imxg2d; imxpxp; imxipu                 || imxg2dvideotransform; imxpxpvideotransform; imxipuvideotransform   || g2d, pxp, egl and ipu video filter/converter/scalars^3^ ||
     47|| Video Compositing   || imxg2d; imxipu                         || imxg2dcompositor, imxipucompositor                                 || gpu/ipu accelerated compositing                         ||
     48,,1. The g2d sink is very flexible in the types of input video it can take, but doesn't have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it's input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.,,
     49[[BR]]
     50,,2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.,,
     51[[BR]]
     52,,3. Please see note 1 above.,,
     53[[BR]]
     54
     55Please uncollapse the following if you want to view gstreamer plugins specific to the imx6 previous to Yocto v1.8 (Fido).
     56[[CollapsibleStart(Older GStreamer Plugins (Supported previous to Yocto 1.8))]]
     57[=#gst-fsl-plugin]
     58=== Freescale GStreamer Plugins (fsl-gst-plugin) ===
     59The gst-fsl-plugin package plugins are written by Freescale. While they are open-source (release source tarballs are available) the development cycle is closed and there is no community involvement.
     60
     61Pros:
     62 * These have been available for some time now and many examples can be found on the Internet
     63
     64Cons:
     65 * There is no community involvement in the development of this code
     66 * The code is a bit difficult to follow
     67 * They do not follow the gstreamer model very well and do things in a very unorthodox way that don't mix well with the rest of GStreamer:
     68  * their decoders rely on their 'aiur' demuxer to do stream parsing
     69  * they do not use the PTS and DTS timestamps that come for upstream and instead re-generate their own
     70  * they are not very compatible with other gstreamer elements (such as ones that are not DMA buffer aware)
     71  * they do not communicate with the GStreamer development community
     72
     73
     74[=#gstreamer-0.10_gst-fsl-plugin-3.0.11]
     75==== Freescale GStreamer 0.10 (plugin version 3.0.11) plugins ====
     76These plugins are available in the Gateworks Yocto 1.3, 1.6, and 1.7 BSP's.
     77
     78For a very good reference for this version of the gst-fsl-plugins see the i.MX_Series_Yocto_Project_Multimedia_Users's_Guide.pdf and i.MX_6_Series_Yocto_Project_Multimedia_Release_Notes.pdf found in [http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX6Q&fpsp=1&tab=Documentation_Tab L3.10.17_1.0.0_LINUX_DOCS] download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading.
     79
     80||= Type              =||= Plugin(s)             =||= Element(s)                                                       =||= Comments                                      =||
     81|| Audio Decoder       || beep.imx;amrdec.imx     || beepdec; beepdec.vorbis; beepdec.mpr; beepdec.aac; mfw_amrdecoder  || --                                              ||
     82|| Audio Encoder       || mp3enc.imx              || mfw_mp3encoder                                                     || Encode audio to mp3                             ||
     83|| Audio Converter     || audiopeq.imx            || mfw_audio_pp                                                       || Equalizer                                       ||
     84|| Video Decoder       || vpu.imx; h264.imx       || vpudec; mfw_h264decoder                                            || VPU Based decoder and h264 video decoder        ||
     85|| Video Encoder       || vpu.imx                 || vpuenc                                                             || VPU Based encoder                               ||
     86|| Demuxer             || aiur.imx                || aiurdemux                                                          || Universal Demuxer                               ||
     87|| Video Render (sink) || v4lsink.imx; isink.imx  || mfw_v4lsink; mfw_isink                                             || Video4Linux2 and IPU based video sinks          ||
     88|| Device Source(src)  || tvsrc.imx; v4lsrc.imx   || tvsrc; mfw_v4lsrc                                                  || tvsrc: v4l2 tv src; mfw_v4lsrc: v4l2 camera src ||
     89
     90If there are any elements that you require more information on, please run a 'gst-inspect-0.10 <element>'.
     91
     92
     93[=#gstreamer-0.10_gst-fsl-plugin-4.0.2]
     94==== Freescale GStreamer 0.10 (plugin version 4.0.2) plugins ====
     95These plugins are available in the Gateworks Yocto master branch and will be in the Gateworks Yocto 1.8 BSP.
     96
     97For a very good reference for this version of the gst-fsl-plugins see the i.MX_Linux_Release_notes.pdf foudn in [http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX6Q&fpsp=1&tab=Documentation_Tab L3.10.53_1.1.0_LINUX_DOCS] download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading.
     98
     99||= Type              =||= Plugin(s)             =||= Element(s)                                                       =||= Comments                                      =||
     100|| Audio Decoder       || beep.imx;amrdec.imx     || beepdec; beepdec.vorbis; beepdec.mpr; beepdec.aac; mfw_amrdecoder  || --                                              ||
     101|| Audio Encoder       || mp3enc.imx              || mfw_mp3encoder                                                     || Encode audio to mp3                             ||
     102|| Audio Converter     || audiopeq.imx            || mfw_audio_pp                                                       || Equalizer                                       ||
     103|| Video Decoder       || vpu.imx                 || vpudec                                                             || VPU Based decoder                               ||
     104|| Video Encoder       || vpu.imx                 || vpuenc                                                             || VPU Based encoder                               ||
     105|| Demuxer             || aiur.imx                || aiurdemux                                                          || Universal Demuxer                               ||
     106|| Video Render (sink) || v4lsink.imx; isink.imx  || imxv4l2sink; mfw_isink                                             || Video4Linux2 and IPU based video sinks          ||
     107|| Device Source(src)  || tvsrc.imx; v4lsrc.imx   || tvsrc; imxv4l2src                                                  || tvsrc: v4l2 tv src; imxv4l2src: v4l2 camera src ||
     108
     109If there are any elements that you require more information on, please run a 'gst-inspect-0.10 <element>'.
     110
     111
     112[=#gstreamer-1.0_gst-fsl-plugin-4.0.2]
     113==== Freescale GStreamer 1.0 (plugin version 4.0.2) plugins ====
     114These plugins are available in the Gateworks Yocto master branch and will be in the Gateworks Yocto 1.8 BSP.
     115
     116For a very good reference for this version of the gst-fsl-plugins see the i.MX_Linux_Release_notes.pdf foudn in [http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX6Q&fpsp=1&tab=Documentation_Tab L3.10.53_1.1.0_LINUX_DOCS] download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading.
     117
     118||= Type              =||= Plugin(s)                   =||= Element(s)                                                       =||= Comments                                                      =||
     119|| Audio Decoder       || beep.imx;amrdec.imx           || beepdec                                                            || MP3, AAC, WMA, AC3, etc. "Universal"                            ||
     120|| Audio Encoder       || imxmp3enc.imx                 || imxmp3enc                                                          || Encode audio to mp3                                             ||
     121|| Video Decoder       || vpu.imx                       || vpudec                                                             || VPU Based decoder                                               ||
     122|| Video Encoder       || vpu.imx                       || vpuenc                                                             || VPU Based encoder                                               ||
     123|| Demuxer             || aiur.imx                      || aiurdemux                                                          || AVI, MKV, MP4, MPEG4, ASF, OGG, FLV, WebM Demux                 ||
     124|| Video Render (sink) || imxv4l2.imx; overlaysink.imx  || imxv4l2sink; overlaysink                                           || imxv4l2sink: v4l2 video sink; overlaysink: g2d based video sink ||
     125|| Device Source(src)  || imxv4l2.imx                   || imxv4l2src                                                         || imxv4l2src: v4l2 camera/TVin src                                ||
     126|| Video Converter     || imxvideoconvert.imx           || imxvideoconvert_ipu; imxvideoconvert_g2d                           || IPU/G2D based video converters (i.e. colorspace etc)            ||
     127|| AVB Streaming       || avb.imx                       || avbpcmsink; avbpcmsrc, avbmpegtssink; avbmpegtssrc                 || Please use 'gst-inspect-1.0' on element you're interested in    ||
     128[[CollapsibleEnd]]
     129
     130If there are any elements that you require more information on, please run a 'gst-inspect-1.0 <element>'.
     131
     132== Codecs ==
     133For Codec support (Encoder / Decoder) details please see:
     134 * [wiki:ventana/codec Ventana Codecs]
     135
     136
     137[=#pipelines]
     138== Pipeline Construction ==
     139[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-applications.html gst-launch] is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type {{{gst-inspect element}}} where {{{element}}} is the element you are looking for. To see a full listing of elements installed, you can type just {{{gst-inspect}}}.
     140
     141Here is a graphical example of a pipeline. [[BR]]
     142[[Image(http://docs.gstreamer.com/download/attachments/327784/simple-player.png?version=1&modificationDate=1333442995365, 512px)]]
     143
     144The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next
     145
     146[=#test-media]
     147=== Test Media Files ===
     148There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing:
     149 * [https://peach.blender.org/download/ Big Buck Bunny] - Creative Commons
     150 * [https://durian.blender.org/download/ Sintel] - Creative Commons
     151 * [https://mango.blender.org/download/ Tears of Steel] - Creative Commons
     152
     153[=#video]
     154=== Video ===
     155Please see [wiki:Yocto/gstreamer/video this] page for some [wiki:Yocto/gstreamer/video video] pipeline examples.
     156
     157[=#audio]
     158=== Audio  ===
     159Please see [wiki:Yocto/gstreamer/audio this] page for some [wiki:Yocto/gstreamer/audio audio] pipeline examples.
     160
     161[=#multimedia]
     162=== Multimedia ===
     163Please see [wiki:Yocto/gstreamer/multimedia this] page for some [wiki:Yocto/gstreamer/multimedia multimedia] pipeline examples.
     164
     165[=#streaming]
     166=== Streaming ===
     167Please see [wiki:Yocto/gstreamer/streaming this] page for some [wiki:Yocto/gstreamer/streaming streaming] pipeline examples.
     168
     169[=#latency]
     170=== Latency ===
     171Please see [wiki:Yocto/gstreamer/latency this] page for some [wiki:Yocto/gstreamer/latency latency] information.
     172
     173[=#compositing]
     174=== Video Composition ===
     175Please see [wiki:Yocto/gstreamer/compositing this] page for some [wiki:Yocto/gstreamer/compositing compositing] information.
     176
     177[=#applications]
     178== Applications ==
     179GStreamer uses pipelines as a way to test elements in various ways. In a production environment, you generally want to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that.
     180
     181=== Example Applications ===
     182GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html here] for more details.
     183
     184Further, Gateworks has created their own suite (currently only consisting of one application) of applications that help to show developers how to create GStreamer apps.
     185
     186[=#gst-gateworks-apps]
     187==== gst-variable-rtsp-server ====
     188[https://github.com/Gateworks/gst-gateworks-apps/blob/master/src/gst-variable-rtsp-server.c This application] is included by default on our Yocto 1.8+ BSPs (multimedia/gui images). It is an example GStreamer application that, like the GStreamer [http://code.metager.de/source/xref/freedesktop/gstreamer/gst/rtsp-server/examples/test-launch.c test RTSP application], you can pass in a custom pipeline to create an RTSP server out of, but extends the usage quite a bit in order to showcase GStreamer call-backs, among other things. It is a fully open-source program to provide you, the developer, a working example of a solution to some commonly asked questions.
     189
     190[https://github.com/Gateworks/gst-gateworks-apps/blob/master/src/gst-variable-rtsp-server.c gst-variable-rtsp-server] is a solution created by Gateworks that allows changing the quality of a live stream on the fly based on the number of clients connected to the server. Similar to other [wiki:Yocto/gstreamer/streaming#AdaptiveBitrateStreaming adaptive bitrate] technologies, it can change quality based on information is knows about the stream. In our case, the number of connected clients was deemed as appropriate. Note that this ability will only work if you are not supplying the full pipeline (e.g. not using the {{{-u}}} flag).
     191
     192This application can change either the {{{quant-param}}} or the {{{bitrate}}} parameters of the {{{imxvpuenc_h264}}} encoder. The {{{quant-param}}} will only be used if the pipeline is set to Variable Bitrate mode (VBR). This can be accomplished by passing in the {{{-b 0}}} flag to the program. Otherwise, {{{gst-variable-rtsp-server}}} will change the {{{bitrate}}} of the stream.
     193
     194The algorithm used to change both {{{bitrate}}} and {{{quant-param}}} are based on the {{{--steps}}} input (default to 5). For example, if using the default {{{steps}}} value of 5, if the min bitrate was 500 and max bitrate was 2000, it would take 5 clients to get from the best to the worst quality.
     195
     196To calculate a {{{bitrate}}} or {{{quant-param}}} for a number of clients or steps etc, please use the following formulas ('''integer math'''):
     197 1. First get step factor: step = (max^1^ - min^2^) / (steps - 1), e.g. (51 - 0) / 1 = 51
     198 1. Now, depending on the parameter, pick one:
     199  * quant-param
     200   * new_quant^3^ = ((clients_connected - 1) * step) + min^2^, e.g. ((2 - 1) * 51) + 0 = 51
     201  * bitrate
     202   * new_bitrate^4^ = max^1^ - ((clients_connected - 1) * step), e.g. 10000 - ((2 - 1) * 9999) = 1
     203
     204,,1. max means the high number of that particular parameter,,[[BR]]
     205,,2. min means the low number of that particular parameter,,[[BR]]
     206,,3. Note that this is capped to the 'max' quant-param value,,[[BR]]
     207,,4. Note that this is capped to the 'min' bitrate value,,[[BR]]
     208
     209The formulas above slightly differ due to the fact that the lower the {{{bitrate}}}, the lower the quality, but with a higher {{{quant-param}}}, the lower the quality (inverted).
     210
     211Please see below for some example use cases:
     212* Use {{{imxv4l2videosrc}}} as input stream on {{{/dev/video0}}}
     213{{{
     214#!bash
     215gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0
     216}}}
     217
     218* Use {{{imxv4l2videosrc}}} as input stream on {{{/dev/video0}}}, but go from best to worst quality when 2 clients connect
     219{{{
     220#!bash
     221gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 --steps 2
     222}}}
     223
     224* Use {{{imxv4l2videosrc}}} as input stream on {{{/dev/video0}}}, cap max bitrate to 5000kbps and min bitrate to 2500kbps
     225{{{
     226#!bash
     227gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 --max-bitrate=5000 --min-bitrate=2500
     228}}}
     229
     230To connect as a client, you can use any program that can connect to an RTSP stream. Below is an example connecting with GStreamer:
     231{{{
     232#!bash
     233gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! decodebin ! autovideosink
     234}}}
     235
     236There are many more examples [wiki:Yocto/gstreamer/streaming#rtsp on our GStreamer streaming] page.
     237
     238With the above, if there is only a single client, the stream will be at it's best quality. But as you connect more clients, the quality of the stream will drop in order to account for network congestion. For more information, please see [https://github.com/Gateworks/gst-gateworks-apps#gst-variable-rtsp-server this README] section.
     239
     240[=#references]
     241== References ==
     242* [http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+10%3A+gstreamer+tools gstreamer.com] contains much more information on the construction of pipelines.
     243* [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html GStreamer Hello World Application explanation.]
     244* [https://github.com/Gateworks/gst-gateworks-apps Gateworks Example Application Suite]
     245
     246[=#misc]
     247= Miscellaneous =
     248The below sections host some miscellaneous and loose information about GStreamer that didn't otherwise fit a specific category.
     249
     250[=#troubleshooting]
     251== Troubleshooting ==
     252If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device.
     253
     254You can see all the properties of a particular element with the 'gst-inspect <elementname>' command. You can see a list of all elements with 'gst-inspect' without arguments.
     255
     256You can enable debug output for Gstreamer by using the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-debug.html '--gst-debug'] parameter to gst-launch or by setting the [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html GST_DEBUG]  environment variable.
     257
     258For example, if you wanted to see DEBUG level messages for the mfw_v4lsrc plugin you can set GST_DEBUG=mfw_v4lsrc:4
     259
     260References:
     261 * http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-checklist-debug.html
     262 * http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html
     263
     264[=#info]
     265== More Information ==
     266For more pipeline examples, please visit the [https://community.freescale.com/docs/DOC-93387 Freescale community site] for the i.MX6 processor.