wiki:Yocto/gstreamer/multimedia

  1. Video and Audio
    1. Named Elements, queues, and Multiple pipelines with gst-launch
    2. Muxing Mixed Content
      1. Example: Capture Video and Audio Through HDMI Port
      2. Example: Capture MPEG4 video, MP3 Audio muxed together in a AVI File
    3. De-muxing mixed content
      1. Example: Playback Matroska file with WEBM video and Vorbis Audio
      2. Example: Playback Quicktime file with H.264/AVC video and AAC Audio
      3. Example: Playback MPEG-TS file
    4. Bin elements
      1. GStreamer playbin
      2. GStreamer decodebin
      3. GStreamer gst-play-1.0
    5. How to determine what pipeline is needed to decode and play
      1. GST_DEBUG_DUMP_DOT_DIR
      2. gst-launch -v

Video and Audio

Playback of a file that has both audio and video requires a slightly more complex pipeline than the standard audio and video pipelines.

Generally, a mixed media pipeline will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and queue elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data).

The examples on this page will refer to GStreamer-1.0. To see GStreamer-0.10 (deprecated) examples, please see this older revision page.

Named Elements, queues, and Multiple pipelines with gst-launch

When mixing audio and video elements with gst-launch one must make use of multiple pipelines using named elements. When developing GStreamer based applications you create 'Bin' elements that put multiple elements together in a bin.

The name property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any). Multiple pipelines can be provided to gst-launch and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'.

This is best explained with some examples:

  • Encoding a stream with audio and video content into an AVI file:
    Line 
    1gst-launch-1.0 \
    2 v4l2src \
    3 ! $VIDEO_CAPABILITIES \
    4 ! mux. \
    5 alsasrc \
    6 ! $AUDIO_CAPABILITIES \
    7 ! mux. \
    8 avimux name=mux \
    9 ! filesink location=test.avi
    • The v4l2src pipeline ends with mux. which means its output is sent to the pipeline who's name is mux
    • The alsasrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux
    • The avimux pipeline specifies name=mux therefore it takes as a source all pipelines that ended with mux.
  • Decoding a stream with audio and video content from an AVI file:
    Line 
    1gst-launch-1.0 \
    2 filesource location=test.avi \
    3 ! avidemux name=demux \
    4 demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \
    5 demux. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink
    • The filesource pipeline ends with name=demux which means the output of this pipeline will be sent to all pipelines with a demux. source (who's types have been successfully negotiated)
    • The audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3)
    • The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
    • Queue elements are used to keep one pipeline or element from blocking another. For example, if the mpeg4videoparse element needs more data from the avidemux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering

Muxing Mixed Content

Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'.

You can use gst-inspect to see a list of most of these using grep: 

gst-inspect-1.0 | grep -i muxer | grep -vi de

Some common examples:

  • mpegtsmux: MPEG Transport Stream Muxer
  • mpegpsmux: MPEG Program Stream Muxer
  • matroskamux: Matroska muxer
  • avimux: Avi muxer
  • qtmux: QuickTime Muxer
  • oggmux: Ogg muxer

To mux mixed content together include one of these elements following the audio and video pipelines.

Examples:

  • Encoding a stream with audio and video content into an AVI file:
    Line 
    1gst-launch-1.0 \
    2 videotestsrc \
    3 ! $VIDEO_CAPABILITIES \
    4 ! mux. \
    5 audiotestsrc \
    6 ! $AUDIO_CAPABILITIES \
    7 ! mux. \
    8 avimux name=mux \
    9 ! filesink location=test.avi
    • the videotestsrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux and who's format has been successfully negotiated.
    • the audiotestsrc pipeline ends with mux. which means its output is sent to the pipeline who's name is mux and who's format has been successfully negotiated.
    • the avimux pipeline specifies name=mux therefore it takes as a source all pipelines that ended with mux. and it understands how to multiplex the two types of data together into its output which is written to the file test.avi

Example: Capture Video and Audio Through HDMI Port

  • Note the -e switch is needed to properly terminate for playing back in a video player like VLC
    Line 
    1gst-launch-1.0 -e imxv4l2videosrc device=/dev/video0 queue-size=6 do-timestamp=true \
    2! queue flush-on-eos=true silent=true leaky=0 max-size-buffers=0 max-size-time=0 max-size-bytes=0 \
    3! imxipuvideotransform \
    4! imxvpuenc_h264 bitrate=10000 \
    5! h264parse disable_passthrough=true \
    6! queue flush-on-eos=true silent=true leaky=0 max-size-buffers=0 max-size-time=0 max-size-bytes=0 \
    7! mux. alsasrc device="sysdefault:CARD=tda1997xaudio" \
    8! audioconvert \
    9! imxmp3audioenc bitrate=128 \
    10! mux. mp4mux name=mux \
    11! filesink location=audiovideo.mp4 sync=true

Example: Capture MPEG4 video, MP3 Audio muxed together in a AVI File

To capture, encode, and output audio and video using MPEG4 video compression, MP3 audio compression, and an AVI file format you could use:

Line 
1gst-launch-1.0 \
2 imxv4l2videosrc device=/dev/video0 \
3 ! imxvpuenc_mpeg4 bitrate=10000 \
4 ! mux. \
5 alsasrc device="sysdefault:CARD=sgtl5000audio" \
6 ! audioconvert ! imxmp3audioenc bitrate=128 \
7 ! mux. \
8 avimux name=mux \
9 ! filesink location=test.avi
  • The imxv4l2videosrc pipeline ends with mux. which means its output (video/mpeg) is sent to the pipeline who's name is mux
  • The alsasrc pipeline ends with mux. which means its output (audio/mpeg) is sent to the pipeline who's name is mux
  • The avimux pipeline specifies name=mux therefore it takes as a source all pipelines that ended with mux. and it understands how to multiplex the two types of data together into its output which is written to the file test.avi.

De-muxing mixed content

Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'.

You can use gst-inspect to see a list of most of these using grep: 

gst-inspect-1.0 | grep -i 'de\?muxer'

Some common examples:

  • tsparse: MPEG transport stream parser
  • tsdemux: MPEG transport stream demuxer
  • matroskademux: Matroska demuxer
  • avidemux: Avi demuxer
  • qtdemux: QuickTime demuxer
  • oggdemux: Ogg demuxer

To de-mux mixed content include one of these elements following the audio and video pipelines. Note that unlike muxing typically you also need to use a parser element to parse the bitstream and break it into discrete buffers that the downstream decoder can understand.

Some common parsers:

  • ogmaudioparse: OGM audio stream parser
  • ogmvideoparse: OGM video stream parser
  • aacparse: AAC audio stream parser
  • amrparse: AMR audio stream parser
  • ac3parse: AC3 audio stream parser
  • flacparse: FLAC audio parser
  • mpegaudioparse: MPEG1 Audio Parser
  • h263parse: H.263 parser
  • h264parse: H.264 parser
  • mpegvideoparse: MPEG video elementary stream parser
  • mpeg4videoparse: MPEG 4 video elementary stream parser
  • pngparse: PNG parser
  • vc1parse: VC1 parser

Examples:

  • decoding a stream with audio and video content from an AVI file:
    Line 
    1gst-launch-1.0 \
    2 filesource location=test.avi \
    3 ! avidemux name=demux \
    4 demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \
    5 demux. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink
    • the filesource pipeline ends with name=demux which means the output of this pipeline will be sent to all pipelines with a demux. source (who's types have been successfully negotiated)
    • the audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3)
    • the video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
    • queue elements are used to keep one pipeline or element from blocking another. For example, if the mpeg4videoparse element needs more data from the avidemux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering

Example: Playback Matroska file with WEBM video and Vorbis Audio

An example file consisting of a Matroska file container that includes WEBM encoded video and Vorbis encoded audio can be found at the ​Tears of Steel download site. Tears of Steel is a relatively popular video (Creative Commons License) that has several encodings.

Video file downloadable ​here

For this file we can use the filessrc source element, the matroskademux}} demuxer, the {{{ivorbisdec Vorbos audio decoder, and the imxg2dvideosink sink element: 

gst-launch-1.0 \
  filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \
    matroskademux name=d \
  d. ! queue ! ivorbisdec  ! queue ! alsasink device=hw:1,0 \
  d. ! queue ! imxvpudec   ! queue ! imxg2dvideosink framebuffer=/dev/fb0

Example: Playback Quicktime file with H.264/AVC video and AAC Audio

An example file consisting of a Quicktime file container that includes H.264/AVC encoded video and AAC encoded audio can be found at the ​Sintel download site. Sintel is a relatively popular video (Creative Commons License) that has several encodings.

Video file downloadable ​here

For this file we can use the filesrc source element, the h264parse parser element along with the imxvpudec decoder element, and the avdec_aac audio decoder element. 

 gst-launch-1.0  filesrc location=/home/root/sintel_trailer-1080p.mp4  ! \
   qtdemux name=d \
 d. ! queue ! h264parse ! imxvpudec ! imxg2dvideosink \
 d. ! queue ! avdec_aac ! audioconvert ! alsasink

Example: Playback MPEG-TS file

Video only: 

gst-launch-1.0 \
  filesrc location=/tmp2/T2C00201_1080p60_Crop.ts ! \
  tsdemux ! mpegvideoparse ! imxvpudec ! imxipuvideosink sync=false async=false

Video + Audio 

gst-launch-1.0 \
  filesrc location=/tmp2/T2C00201_1080p60_Crop.ts \
    ! tsdemux name=demux \
  demux. ! queue ! mpegaudioparse ! queue ! mad ! audioconvert ! queue ! alsasink \
  demux. ! queue ! mpegvideoparse ! queue ! imxvpudec ! queue ! imxg2dvideosink sync=false async=false

Bin elements

A Bin element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept.

GStreamer playbin

GStreamer playbin element attempts to create a pipeline that will play both the audio and video portions of a file. For example: 

gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi

The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by: 

gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=hw:1,0"

Please type gst-inspect-1.0 playbin to see more options.

GStreamer decodebin

The GStreamer plugin decodebin is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under the first example with the following: 

gst-launch-1.0 \
  filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \
  matroskademux name=d \
  d. ! queue ! ivorbisdec ! queue ! alsasink device=hw:1,0 \
  d. ! queue ! decodebin  ! queue ! imxg2dvideosink framebuffer=/dev/fb0

Note that decodebin doesn't always choose the correct decoder, so be wary of this. It is similar to playbin in that it aids in creating a dynamic pipeline.

GStreamer gst-play-1.0

The stand-alone application gst-play is is a program that utilizes the playbin element and thus can be used for playback of many file types. The above example gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi can be replaced with: 

gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi

How to determine what pipeline is needed to decode and play

Sometimes the above Bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream.

The gst-launch application provides a couple of useful debugging tools that can help with this:

  • using GST_DEBUG_DUMP_DOT_DIR and Graphviz
  • using gst-launch -v

GST_DEBUG_DUMP_DOT_DIR

You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause gst-launch to output a .dot file for each phase of the pipeline then use a tool such as Graphviz to visualize the .dot file.

Example:

  • use playbin to playback a file: 
    root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
    • hit Cntl-C after decoding starts to exit early
  • see the dot files created: 
    root@ventana:~# ls /tmp/dot
0.00.00.108710334-gst-launch.NULL_READY.dot
0.00.00.490807334-gst-launch.READY_PAUSED.dot
0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot
0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
0.00.03.254000001-gst-launch.PAUSED_READY.dot
  • transfer to a PC and use something like xdot to view: 
    xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
    • zoom in along the graph and you can see that:
      • GstFileSrc is the source,
      • GstAviDemux is used to demux to audio/x-ac3,
      • GstAc3Parse is used to parse the audio into audio frames,
      • GstMpeg4VParse is used to parse the video into video frames,
      • GstImxVpuDec is used to decode the video from video/mpeg to video/x-raw,
      • GstA52Dec is used to decode the audio from audio/x-ac3 to audio/x-raw,
      • etc
    • Note that some hunting with gst-inspect must be done to determine what elements coorespond to the above class names

Reference:

gst-launch -v

The verbose debugging from gst-launch -v can show you the negotiation that takes place as a pipeline moves through its stages.

Example: 

gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi

examining the verbose output can show you the following:

  • container: AVI: avidemux
  • video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! imxvpudec
  • audio: AC3 48khz 5channels: ac3parse ! a52dec

Therefore you can use these pipelines to decode and play:

  • video only (output to imxipuvideosink fb0) 
    gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! imxvpudec ! imxipuvideosink
  • audio only (output to hdmi audio sink) 
    gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
  • both audio and video 
    gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \
  d. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink \
  d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
Last modified 7 years ago Last modified on 10/22/2017 06:18:20 AM
Note: See TracWiki for help on using the wiki.