[[PageOutline]] [=#audio-and-video] = Video and Audio = Playback of a file that has both audio and video requires a slightly more complex pipeline than the standard [wiki:Yocto/gstreamer/audio audio] and [wiki:Yocto/gstreamer/video video] pipelines. Generally, a mixed media pipeline will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and {{{queue}}} elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data). The examples on this page will refer to GStreamer-1.0. To see GStreamer-0.10 (deprecated) examples, please see this [http://trac.gateworks.com/wiki/Yocto/gstreamer/multimedia?version=1 older revision page]. [=#named-elements] == Named Elements, queues, and Multiple pipelines with gst-launch == When mixing audio and video elements with {{{gst-launch}}} one must make use of multiple pipelines using {{{named elements}}}. When developing GStreamer based applications you create 'Bin' elements that put multiple elements together in a bin. The {{{name}}} property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any). Multiple pipelines can be provided to {{{gst-launch}}} and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'. This is best explained with some examples: * Encoding a stream with audio and video content into an AVI file: {{{#!bash lineno=1 gst-launch-1.0 \ v4l2src \ ! $VIDEO_CAPABILITIES \ ! mux. \ alsasrc \ ! $AUDIO_CAPABILITIES \ ! mux. \ avimux name=mux \ ! filesink location=test.avi }}} - The {{{v4l2src}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} - The {{{alsasrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} - The {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} * Decoding a stream with audio and video content from an AVI file: {{{#!bash lineno=1 gst-launch-1.0 \ filesource location=test.avi \ ! avidemux name=demux \ demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \ demux. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink }}} - The {{{filesource}}} pipeline ends with {{{name=demux}}} which means the output of this pipeline will be sent to all pipelines with a {{{demux.}}} source (who's types have been successfully negotiated) - The audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3) - The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx) - Queue elements are used to keep one pipeline or element from blocking another. For example, if the mpeg4videoparse element needs more data from the avidemux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering [=#mux] == Muxing Mixed Content == Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'. You can use {{{gst-inspect}}} to see a list of most of these using grep: {{{ gst-inspect-1.0 | grep -i muxer | grep -vi de }}} Some common examples: - mpegtsmux: MPEG Transport Stream Muxer - mpegpsmux: MPEG Program Stream Muxer - matroskamux: Matroska muxer - avimux: Avi muxer - qtmux: !QuickTime Muxer - oggmux: Ogg muxer To mux mixed content together include one of these elements following the audio and video pipelines. Examples: * Encoding a stream with audio and video content into an AVI file: {{{#!bash lineno=1 gst-launch-1.0 \ videotestsrc \ ! $VIDEO_CAPABILITIES \ ! mux. \ audiotestsrc \ ! $AUDIO_CAPABILITIES \ ! mux. \ avimux name=mux \ ! filesink location=test.avi }}} - the {{{videotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated. - the {{{audiotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated. - the {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} and it understands how to multiplex the two types of data together into its output which is written to the file test.avi === Example: Capture Video and Audio Through HDMI Port === * Note the -e switch is needed to properly terminate for playing back in a video player like VLC {{{#!bash lineno=1 gst-launch-1.0 -e imxv4l2videosrc device=/dev/video0 queue-size=6 do-timestamp=true \ ! queue flush-on-eos=true silent=true leaky=0 max-size-buffers=0 max-size-time=0 max-size-bytes=0 \ ! imxipuvideotransform \ ! imxvpuenc_h264 bitrate=10000 \ ! h264parse disable_passthrough=true \ ! queue flush-on-eos=true silent=true leaky=0 max-size-buffers=0 max-size-time=0 max-size-bytes=0 \ ! mux. alsasrc device="sysdefault:CARD=tda1997xaudio" \ ! audioconvert \ ! imxmp3audioenc bitrate=128 \ ! mux. mp4mux name=mux \ ! filesink location=audiovideo.mp4 sync=true }}} === Example: Capture MPEG4 video, MP3 Audio muxed together in a AVI File === To capture, encode, and output audio and video using MPEG4 video compression, MP3 audio compression, and an AVI file format you could use: {{{#!bash lineno=1 gst-launch-1.0 \ imxv4l2videosrc device=/dev/video0 \ ! imxvpuenc_mpeg4 bitrate=10000 \ ! mux. \ alsasrc device="sysdefault:CARD=sgtl5000audio" \ ! audioconvert ! imxmp3audioenc bitrate=128 \ ! mux. \ avimux name=mux \ ! filesink location=test.avi }}} - The {{{imxv4l2videosrc}}} pipeline ends with {{{mux.}}} which means its output (video/mpeg) is sent to the pipeline who's name is {{{mux}}} - The {{{alsasrc}}} pipeline ends with {{{mux.}}} which means its output (audio/mpeg) is sent to the pipeline who's name is {{{mux}}} - The {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} and it understands how to multiplex the two types of data together into its output which is written to the file test.avi. [=#demux] == De-muxing mixed content == Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'. You can use {{{gst-inspect}}} to see a list of most of these using grep: {{{ gst-inspect-1.0 | grep -i 'de\?muxer' }}} Some common examples: - tsparse: MPEG transport stream parser - tsdemux: MPEG transport stream demuxer - matroskademux: Matroska demuxer - avidemux: Avi demuxer - qtdemux: !QuickTime demuxer - oggdemux: Ogg demuxer To de-mux mixed content include one of these elements following the audio and video pipelines. Note that unlike muxing typically you also need to use a {{{parser}}} element to parse the bitstream and break it into discrete buffers that the downstream decoder can understand. Some common parsers: - ogmaudioparse: OGM audio stream parser - ogmvideoparse: OGM video stream parser - aacparse: AAC audio stream parser - amrparse: AMR audio stream parser - ac3parse: AC3 audio stream parser - flacparse: FLAC audio parser - mpegaudioparse: MPEG1 Audio Parser - h263parse: H.263 parser - h264parse: H.264 parser - mpegvideoparse: MPEG video elementary stream parser - mpeg4videoparse: MPEG 4 video elementary stream parser - pngparse: PNG parser - vc1parse: VC1 parser Examples: * decoding a stream with audio and video content from an AVI file: {{{#!bash lineno=1 gst-launch-1.0 \ filesource location=test.avi \ ! avidemux name=demux \ demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \ demux. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink }}} - the {{{filesource}}} pipeline ends with {{{name=demux}}} which means the output of this pipeline will be sent to all pipelines with a {{{demux.}}} source (who's types have been successfully negotiated) - the audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3) - the video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx) - queue elements are used to keep one pipeline or element from blocking another. For example, if the mpeg4videoparse element needs more data from the avidemux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering [=#ex1] === Example: Playback Matroska file with WEBM video and Vorbis Audio === An example file consisting of a Matroska file container that includes WEBM encoded video and Vorbis encoded audio can be found at the [https://mango.blender.org/download/ Tears of Steel] download site. Tears of Steel is a relatively popular video (Creative Commons License) that has several encodings. Video file downloadable [http://media.xiph.org/mango/tears_of_steel_1080p.webm here] For this file we can use the {{{filessrc}}} source element, the {{{matroskademux}} demuxer, the {{{ivorbisdec}}} Vorbos audio decoder, and the {{{imxg2dvideosink}}} sink element: {{{#!bash gst-launch-1.0 \ filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \ matroskademux name=d \ d. ! queue ! ivorbisdec ! queue ! alsasink device=hw:1,0 \ d. ! queue ! imxvpudec ! queue ! imxg2dvideosink framebuffer=/dev/fb0 }}} [=#ex2] === Example: Playback Quicktime file with H.264/AVC video and AAC Audio === An example file consisting of a Quicktime file container that includes H.264/AVC encoded video and AAC encoded audio can be found at the [https://durian.blender.org/download/ Sintel] download site. Sintel is a relatively popular video (Creative Commons License) that has several encodings. Video file downloadable [https://download.blender.org/durian/trailer/sintel_trailer-1080p.mp4 here] For this file we can use the {{{filesrc}}} source element, the {{{h264parse}}} parser element along with the {{{imxvpudec}}} decoder element, and the {{{avdec_aac}}} audio decoder element. {{{#!bash gst-launch-1.0 filesrc location=/home/root/sintel_trailer-1080p.mp4 ! \ qtdemux name=d \ d. ! queue ! h264parse ! imxvpudec ! imxg2dvideosink \ d. ! queue ! avdec_aac ! audioconvert ! alsasink }}} === Example: Playback MPEG-TS file === Video only: {{{ gst-launch-1.0 \ filesrc location=/tmp2/T2C00201_1080p60_Crop.ts ! \ tsdemux ! mpegvideoparse ! imxvpudec ! imxipuvideosink sync=false async=false }}} Video + Audio {{{ gst-launch-1.0 \ filesrc location=/tmp2/T2C00201_1080p60_Crop.ts \ ! tsdemux name=demux \ demux. ! queue ! mpegaudioparse ! queue ! mad ! audioconvert ! queue ! alsasink \ demux. ! queue ! mpegvideoparse ! queue ! imxvpudec ! queue ! imxg2dvideosink sync=false async=false }}} [=#bin] == Bin elements == A '''Bin''' element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept. [=#playbin] === GStreamer {{{playbin}}} === GStreamer {{{playbin}}} element attempts to create a pipeline that will play both the audio and video portions of a file. For example: {{{ #!bash gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi }}} The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by: {{{ #!bash gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=hw:1,0" }}} Please type {{{gst-inspect-1.0 playbin}}} to see more options. [=#decodebin] === GStreamer {{{decodebin}}} === The GStreamer plugin {{{decodebin}}} is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under [#ex1 the first example] with the following: {{{ #!bash gst-launch-1.0 \ filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \ matroskademux name=d \ d. ! queue ! ivorbisdec ! queue ! alsasink device=hw:1,0 \ d. ! queue ! decodebin ! queue ! imxg2dvideosink framebuffer=/dev/fb0 }}} Note that {{{decodebin}}} doesn't always choose the correct decoder, so be wary of this. It is similar to {{{playbin}}} in that it aids in creating a dynamic pipeline. [=#gst-play] === GStreamer {{{gst-play-1.0}}} === The stand-alone application {{{gst-play}}} is is a program that utilizes the {{{playbin}}} element and thus can be used for playback of many file types. The above example {{{gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi}}} can be replaced with: {{{ #!bash gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi }}} [=#determining-pipelines] == How to determine what pipeline is needed to decode and play == Sometimes the above Bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream. The {{{gst-launch}}} application provides a couple of useful debugging tools that can help with this: * using {{{GST_DEBUG_DUMP_DOT_DIR}}} and Graphviz * using {{{gst-launch -v}}} [=#filter-graph] === GST_DEBUG_DUMP_DOT_DIR === You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause {{{gst-launch}}} to output a {{{.dot}}} file for each phase of the pipeline then use a tool such as {{{Graphviz}}} to visualize the {{{.dot}}} file. Example: * use playbin to playback a file: {{{ root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi }}} - hit Cntl-C after decoding starts to exit early * see the dot files created: {{{ root@ventana:~# ls /tmp/dot 0.00.00.108710334-gst-launch.NULL_READY.dot 0.00.00.490807334-gst-launch.READY_PAUSED.dot 0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot 0.00.03.254000001-gst-launch.PAUSED_READY.dot }}} * transfer to a PC and use something like {{{xdot}}} to view: {{{ xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot }}} - zoom in along the graph and you can see that: - {{{GstFileSrc}}} is the source, - {{{GstAviDemux}}} is used to demux to audio/x-ac3, - {{{GstAc3Parse}}} is used to parse the audio into audio frames, - {{{GstMpeg4VParse}}} is used to parse the video into video frames, - {{{GstImxVpuDec}}} is used to decode the video from video/mpeg to video/x-raw, - {{{ GstA52Dec}}} is used to decode the audio from audio/x-ac3 to audio/x-raw, - etc - Note that some hunting with {{{gst-inspect}}} must be done to determine what elements coorespond to the above class names Reference: - http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+11%3A+Debugging+tools === gst-launch -v === The verbose debugging from {{{gst-launch -v}}} can show you the negotiation that takes place as a pipeline moves through its stages. Example: {{{ gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi }}} examining the verbose output can show you the following: * container: AVI: avidemux * video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! imxvpudec * audio: AC3 48khz 5channels: ac3parse ! a52dec Therefore you can use these pipelines to decode and play: * video only (output to imxipuvideosink fb0) {{{ gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! imxvpudec ! imxipuvideosink }}} * audio only (output to hdmi audio sink) {{{ gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc" }}} * both audio and video {{{ gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \ d. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink \ d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc" }}}