wiki:Yocto/gstreamer

Version 8 (modified by Ryan Erbstoesser, 5 years ago) ( diff )

add statement for new video method

  1. GStreamer on Gateworks SBCs
  2. GStreamer
    1. Plugins
      1. GStreamer-imx plugins (for GStreamer 1.0+)
      2. Freescale GStreamer Plugins (fsl-gst-plugin)
        1. Freescale GStreamer 0.10 (plugin version 3.0.11) plugins
        2. Freescale GStreamer 0.10 (plugin version 4.0.2) plugins
        3. Freescale GStreamer 1.0 (plugin version 4.0.2) plugins
    2. Codecs
    3. Pipeline Construction
      1. Test Media Files
      2. Video
      3. Audio
      4. Multimedia
      5. Streaming
      6. Latency
      7. Video Composition
    4. Applications
      1. Example Applications
        1. gst-variable-rtsp-server
    5. References
  3. Miscellaneous
    1. Troubleshooting
      1. Memory Limitation
      2. Video source loop
      3. gst-inspect-1.0
    2. More Information

GStreamer on Gateworks SBCs

The intended goal of this article is to provide the user with enough information so they can create their own GStreamer pipelines. These GStreamer pipelines have been tested and created for use on the Gateworks Single Board Computers (SBCs), specifically the Ventana family that utilizes the Freescale i.MX6 processors.

Gateworks SBCs can be viewed at the following link: ​http://www.gateworks.com

http://trac.gateworks.com/raw-attachment/wiki/OpenWrt/wireless/relayd/gw5100realsmall.png

Important Note

The Ventana family software has been evolving over the years. This page is focused on the legacy software that used the 3.14 Freescale Vendor kernel.

A move to a mainline kernel with open-source drivers has been made and is available on Ubuntu. A different wiki page exists to document using the new drivers and software, at this link: The Linux Media-CTL Wiki

GStreamer

​GStreamer is an opensource library created to handle multimedia. Pipelines are used to source and sink media however you would like (e.g. decoding the mp3 out of a video file and playing it back through speakers)

Plugins

GStreamer works on 'plugins'. A plugin comprises of elements that can do work on a media stream. For example, the mpeg2dec can decode mpeg1 and mpeg2 video streams.

By default, we install several GStreamer plugins that can tap into the i.MX6 processor's hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these plugins is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves your CPU free to deal with other programs etc.

There are two main versions of GStreamer: 0.10 and 1.0. Our Yocto 'master' branch and Yocto v1.8 'fido' branch include GStreamer 1.0 while Yocto versions prior (Yocto v1.7 (dizzy), Yocto v1.6 (daisy), Yocto v1.3 (danny)) include only GStreamer 0.10.

Different sets of plugins exist for each main version of GStreamer. The next few subsections will attempt to showcase plugins that utilize the i.MX6.

GStreamer-imx plugins (for GStreamer 1.0+)

The GStreamer-imx plugins project is a community project created by Carlos Rafael Giani (<dv@…>). These are always being updated which might make them good to use if a certain feature is only implemented in one of the below.

Pros:

  • Open Source Development model: The project is on github and is very active
  • The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community
  • The source is very well documented and easy to follow
  • Things are done in a very standard GStreamer way

Cons:

  • These are relatively new (early 2014) so there might be undiscovered bugs

The following is a list of plugins provided by the latest version of gstreamer-imx (0.11.0)

Type Plugin(s) Element(s) Comments
Audio Decoder imxaudio imxuniaudiodec Uses i.MX uniaudio codecs for decoding
Audio Encoder imxaudio imxmp3audioenc Uses i.MX for MP3 audio encoding
Device Sources imxv4l2video imxv4l2videosrc Get camera source via v4l2
Video Decoder imxvpu imxvpudec VPU Based decoder
Video Encoder imxvpu imxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263 VPU Based encoders
Video Render (sink) imxg2d; imxpxp; imxeglvivsink; imxipu imxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosink g2d1, ipu1, pxp2, and egl (overlay) video sinks
Video Converter imxg2d; imxpxp; imxipu imxg2dvideotransform; imxpxpvideotransform; imxipuvideotransform g2d, pxp, egl and ipu video filter/converter/scalars3
Video Compositing imxg2d; imxipu imxg2dcompositor, imxipucompositor gpu/ipu accelerated compositing

1. The g2d sink is very flexible in the types of input video it can take, but doesn't have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it's input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.
2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.
3. Please see note 1 above.

Please uncollapse the following if you want to view gstreamer plugins specific to the imx6 previous to Yocto v1.8 (Fido). CollapsibleStart(Older GStreamer Plugins (Supported previous to Yocto 1.8))

Freescale GStreamer Plugins (fsl-gst-plugin)

The gst-fsl-plugin package plugins are written by Freescale. While they are open-source (release source tarballs are available) the development cycle is closed and there is no community involvement.

Pros:

  • These have been available for some time now and many examples can be found on the Internet

Cons:

  • There is no community involvement in the development of this code
  • The code is a bit difficult to follow
  • They do not follow the gstreamer model very well and do things in a very unorthodox way that don't mix well with the rest of GStreamer:
    • their decoders rely on their 'aiur' demuxer to do stream parsing
    • they do not use the PTS and DTS timestamps that come for upstream and instead re-generate their own
    • they are not very compatible with other gstreamer elements (such as ones that are not DMA buffer aware)
    • they do not communicate with the GStreamer development community

Freescale GStreamer 0.10 (plugin version 3.0.11) plugins

These plugins are available in the Gateworks Yocto 1.3, 1.6, and 1.7 BSP's.

For a very good reference for this version of the gst-fsl-plugins see the i.MX_Series_Yocto_Project_Multimedia_Users's_Guide.pdf and i.MX_6_Series_Yocto_Project_Multimedia_Release_Notes.pdf found in ​L3.10.17_1.0.0_LINUX_DOCS download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading.

Type Plugin(s) Element(s) Comments
Audio Decoder beep.imx;amrdec.imx beepdec; beepdec.vorbis; beepdec.mpr; beepdec.aac; mfw_amrdecoder --
Audio Encoder mp3enc.imx mfw_mp3encoder Encode audio to mp3
Audio Converter audiopeq.imx mfw_audio_pp Equalizer
Video Decoder vpu.imx; h264.imx vpudec; mfw_h264decoder VPU Based decoder and h264 video decoder
Video Encoder vpu.imx vpuenc VPU Based encoder
Demuxer aiur.imx aiurdemux Universal Demuxer
Video Render (sink) v4lsink.imx; isink.imx mfw_v4lsink; mfw_isink Video4Linux2 and IPU based video sinks
Device Source(src) tvsrc.imx; v4lsrc.imx tvsrc; mfw_v4lsrc tvsrc: v4l2 tv src; mfw_v4lsrc: v4l2 camera src

If there are any elements that you require more information on, please run a 'gst-inspect-0.10 <element>'.

Freescale GStreamer 0.10 (plugin version 4.0.2) plugins

These plugins are available in the Gateworks Yocto master branch and will be in the Gateworks Yocto 1.8 BSP.

For a very good reference for this version of the gst-fsl-plugins see the i.MX_Linux_Release_notes.pdf foudn in ​L3.10.53_1.1.0_LINUX_DOCS download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading.

Type Plugin(s) Element(s) Comments
Audio Decoder beep.imx;amrdec.imx beepdec; beepdec.vorbis; beepdec.mpr; beepdec.aac; mfw_amrdecoder --
Audio Encoder mp3enc.imx mfw_mp3encoder Encode audio to mp3
Audio Converter audiopeq.imx mfw_audio_pp Equalizer
Video Decoder vpu.imx vpudec VPU Based decoder
Video Encoder vpu.imx vpuenc VPU Based encoder
Demuxer aiur.imx aiurdemux Universal Demuxer
Video Render (sink) v4lsink.imx; isink.imx imxv4l2sink; mfw_isink Video4Linux2 and IPU based video sinks
Device Source(src) tvsrc.imx; v4lsrc.imx tvsrc; imxv4l2src tvsrc: v4l2 tv src; imxv4l2src: v4l2 camera src

If there are any elements that you require more information on, please run a 'gst-inspect-0.10 <element>'.

Freescale GStreamer 1.0 (plugin version 4.0.2) plugins

These plugins are available in the Gateworks Yocto master branch and will be in the Gateworks Yocto 1.8 BSP.

For a very good reference for this version of the gst-fsl-plugins see the i.MX_Linux_Release_notes.pdf foudn in ​L3.10.53_1.1.0_LINUX_DOCS download provided by Freescale. In the link, please search for this document. You will have to sign in and sign a EULA in before downloading.

Type Plugin(s) Element(s) Comments
Audio Decoder beep.imx;amrdec.imx beepdec MP3, AAC, WMA, AC3, etc. "Universal"
Audio Encoder imxmp3enc.imx imxmp3enc Encode audio to mp3
Video Decoder vpu.imx vpudec VPU Based decoder
Video Encoder vpu.imx vpuenc VPU Based encoder
Demuxer aiur.imx aiurdemux AVI, MKV, MP4, MPEG4, ASF, OGG, FLV, WebM Demux
Video Render (sink) imxv4l2.imx; overlaysink.imx imxv4l2sink; overlaysink imxv4l2sink: v4l2 video sink; overlaysink: g2d based video sink
Device Source(src) imxv4l2.imx imxv4l2src imxv4l2src: v4l2 camera/TVin src
Video Converter imxvideoconvert.imx imxvideoconvert_ipu; imxvideoconvert_g2d IPU/G2D based video converters (i.e. colorspace etc)
AVB Streaming avb.imx avbpcmsink; avbpcmsrc, avbmpegtssink; avbmpegtssrc Please use 'gst-inspect-1.0' on element you're interested in

CollapsibleEnd

If there are any elements that you require more information on, please run a 'gst-inspect-1.0 <element>'.

Codecs

For Codec support (Encoder / Decoder) details please see:

Pipeline Construction

​gst-launch is a simple GStreamer application that can be used to test pipelines. A pipeline connects media handling components such as sources, sinks, decoders, encoders, muxes, demuxes, etc. Each of these elements are made available through GStreamer plugins. That is, if a certain plugin isn't installed on your device, you won't be able to use that element. To see that element's information, type gst-inspect element where element is the element you are looking for. To see a full listing of elements installed, you can type just gst-inspect.

Here is a graphical example of a pipeline.
https://gstreamer.freedesktop.org/documentation/tutorials/basic/images/figure-1.png

The most basic and important things to know is the pipeline '!'. This is a pipe, similar to Linux's pipe '|'. It takes the output of one element and inputs it into the next

Test Media Files

There is a variety of royalty free media available on the Internet that can be used freely in demos and for codec and file format testing:

Video

Please see this page for some video pipeline examples.

Audio

Please see this page for some audio pipeline examples.

Multimedia

Please see this page for some multimedia pipeline examples.

Streaming

Please see this page for some streaming pipeline examples.

Latency

Please see this page for some latency information.

Video Composition

Please see this page for some compositing information.

Applications

GStreamer uses pipelines as a way to test elements in various ways. In a production environment, you generally want to write a GStreamer Application. GStreamer applications use glib, so one should be somewhat familiar with that.

Example Applications

GStreamer has an excellent hello world example that clearly explain what each step is meant to do. Please see ​here for more details.

Further, Gateworks has created their own suite (currently only consisting of one application) of applications that help to show developers how to create GStreamer apps.

gst-variable-rtsp-server

​This application is included by default on our Yocto 1.8+ BSPs (multimedia/gui images). It is an example GStreamer application that, you can pass in a custom pipeline to create an RTSP server out of, but extends the usage quite a bit in order to showcase GStreamer call-backs, among other things. It is a fully open-source program to provide you, the developer, a working example of a solution to some commonly asked questions.

​gst-variable-rtsp-server is a solution created by Gateworks that allows changing the quality of a live stream on the fly based on the number of clients connected to the server. Similar to other adaptive bitrate technologies, it can change quality based on information is knows about the stream. In our case, the number of connected clients was deemed as appropriate. Note that this ability will only work if you are not supplying the full pipeline (e.g. not using the -u flag).

This application can change either the quant-param or the bitrate parameters of the imxvpuenc_h264 encoder. The quant-param will only be used if the pipeline is set to Variable Bitrate mode (VBR). This can be accomplished by passing in the -b 0 flag to the program. Otherwise, gst-variable-rtsp-server will change the bitrate of the stream.

The algorithm used to change both bitrate and quant-param are based on the --steps input (default to 5). For example, if using the default steps value of 5, if the min bitrate was 500 and max bitrate was 2000, it would take 5 clients to get from the best to the worst quality.

To calculate a bitrate or quant-param for a number of clients or steps etc, please use the following formulas (integer math):

  1. First get step factor: step = (max1 - min2) / (steps - 1), e.g. (51 - 0) / 1 = 51
  2. Now, depending on the parameter, pick one:
    • quant-param
      • new_quant3 = ((clients_connected - 1) * step) + min2, e.g. ((2 - 1) * 51) + 0 = 51
    • bitrate
      • new_bitrate4 = max1 - ((clients_connected - 1) * step), e.g. 10000 - ((2 - 1) * 9999) = 1

1. max means the high number of that particular parameter
2. min means the low number of that particular parameter
3. Note that this is capped to the 'max' quant-param value
4. Note that this is capped to the 'min' bitrate value

The formulas above slightly differ due to the fact that the lower the bitrate, the lower the quality, but with a higher quant-param, the lower the quality (inverted).

Please see below for some example use cases:

  • Use imxv4l2videosrc as input stream on /dev/video0 
    gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0
  • Use imxv4l2videosrc as input stream on /dev/video0, but go from best to worst quality when 2 clients connect 
    gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 --steps 2
  • Use imxv4l2videosrc as input stream on /dev/video0, cap max bitrate to 5000kbps and min bitrate to 2500kbps 
    gst-variable-rtsp-server -s imxv4l2videosrc -i /dev/video0 --max-bitrate=5000 --min-bitrate=2500

To connect as a client, you can use any program that can connect to an RTSP stream. Below is an example connecting with GStreamer: 

gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! decodebin ! autovideosink

There are many more examples on our GStreamer streaming page.

With the above, if there is only a single client, the stream will be at it's best quality. But as you connect more clients, the quality of the stream will drop in order to account for network congestion. For more information, please see ​this README section.

References

Miscellaneous

The below sections host some miscellaneous and loose information about GStreamer that didn't otherwise fit a specific category.

Troubleshooting

If something doesn't work right, make sure all of the defaults being used in your pipeline element properties are correct. Many of the examples above or found online may omit the 'device' property from the source or sink elements which will make it default to the first appropriate device.

You can see all the properties of a particular element with the 'gst-inspect <elementname>' command. You can see a list of all elements with 'gst-inspect' without arguments.

You can enable debug output for Gstreamer by using the ​--gst-debug parameter to gst-launch or by setting the ​GST_DEBUG environment variable.

For example, if you wanted to see DEBUG level messages for the videotestsrc plugin you can set GST_DEBUG=videotestsrc:4

References:

Memory Limitation

On SBCs with 512MB of memory, such as the GW5510, GW5100, GW5530, etc it is possible that a error appears related to memory. 

[WARN]  VPU iram is less than needed, some parts don't use iram
[ 6194.061104] mxc_vpu 2040000.vpu: Physical memory allocation error!
[ 6194.067361] mxc_vpu 2040000.vpu: Physical memory allocation error!
[ERR]   mem allocation failed!

This requires increasing the CMA memory in the bootloader: 

setenv mem 'cma=96M'

The example of 96M may or may not be enough, and should be experimented with. More information about CMA can be found here: Ventana Memory

Video source loop

An efficient way of testing multiple video sources is to use a loop. Beneath is a simple bash script that will do this: 

for i in $(seq 1 8);do ##seq 1 8 represents the range to loop of enumerated /dev/video* devices
gst-launch-1.0 imxv4l2videosrc device=/dev/video$i ! imxg2dvideosink &
pid=$!
echo displaying device $i 
sleep 5
kill $pid
done

gst-inspect-1.0

Gst-inspect-1.0 is included in our Yocto BSP. This is an excellent tool if you're unfamiar with the functionality of any Gstreamer module.

Usage: 

root@ventana:~# gst-inspect-1.0 imxv4l2videosrc ##imxv4l2videosrc is just an example this can be any gstreamer plugin. 
Factory Details:
  Rank                     primary (256)
  Long-name                V4L2 CSI Video Source
  Klass                    Source/Video
  Description              Capture video streams using V4L2 CSI interface
  Author                   Philip Craig <phil@blackmoth.com.au>

Plugin Details:
  Name                     imxv4l2videosrc
  Description              GStreamer i.MX V4L2 CSI video source
  Filename                 /usr/lib/gstreamer-1.0/libgstimxv4l2videosrc.so
  Version                  0.11.1
  License                  LGPL
  Source module            gstreamer-imx
  Binary package           Unknown package release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstImxV4l2VideoSrc

Implemented Interfaces:
  GstURIHandler

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_base_src_change_state

Element has no clocking capabilities.

URI handling capabilities:
  Element can act as source.
  Supported URI protocols:
    imxv4l2

Pads:
  SRC: 'src'
    Implementation:
      Has getrangefunc(): gst_base_src_getrange
      Has custom eventfunc(): gst_base_src_event
      Has custom queryfunc(): gst_base_src_query
      Has custom iterintlinkfunc(): gst_pad_iterate_internal_links_default
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "imxv4l2videosrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  typefind            : Run typefind before negotiating
                        flags: readable, writable
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: false
  capture-mode        : Capture mode of camera, varies with each v4l2 driver,
                                for example ov5460:
                                ov5640_mode_VGA_640_480 = 0,
                                ov5640_mode_QVGA_320_240 = 1,
                                ov5640_mode_NTSC_720_480 = 2,
                                ov5640_mode_PAL_720_576 = 3,
                                ov5640_mode_720P_1280_720 = 4,
                                ov5640_mode_1080P_1920_1080 = 5
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  fps-n               : Numerator of the framerate at whichthe input stream is to be captured
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 30 
  input               : Video input selected with VIDIOC_S_INPUT
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 1 
  device              : Device location
                        flags: readable, writable
                        String. Default: "/dev/video0"
  queue-size          : Number of V4L2 buffers to request
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 6 
  crop-meta-x         : X value for crop metadata
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  crop-meta-y         : Y value for crop metadata
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  crop-meta-width     : WIDTH value for crop metadata
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  crop-meta-height    : HEIGHT value for crop metadata
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0

More Information

For more pipeline examples, please visit the ​Freescale community site for the i.MX6 processor.

Note: See TracWiki for help on using the wiki.