Changes between Version 1 and Version 2 of Yocto/gstreamer/multimedia

Timestamp:

10/22/2017 06:18:20 AM (8 years ago)

Author:

Chris Lang

Comment:

--

Yocto/gstreamer/multimedia

@@ -3 +3 @@
 [=#audio-and-video]
 = Video and Audio =
-Playback of combined video and audio is a bit more complicated as now you need to utilize parsers, named elements to split the audio and video paths, and queues to avoid single-threaded blocking.
-
-The following pipeline examples utilize plugins from the the Freescale gst-fsl-plugin package on GStreamer 0.10.
-
-gst-fsl-plugin elements used:
- * Parsers / Demuxers:
-  * '''aiurdemux''' - parse and identify source stream formats
-
-Example using Big Buck Bunney H.264 (MOV H264 video, AAC suuround sound):
-{{{
-gst-launch filesrc location=/media/sda1/big_buck_bunny_1080p_h264.mov typefind=true ! aiurdemux name=d ! \
-  queue2 ! beepdec ! alsasink d. ! \
-  queue2 ! vpudec ! mfw_v4lsink device=/dev/video16
-}}}
-
-
-== GStreamer '''playbin2''' and '''decodebin2''' elements ==
-GStreamer has some '''bin''' elements that try to 'autoplug' other elements together by inspecting the pipeline.
-
-The '''decodebin2''' element will try to figure out the proper decoder pipeline for the sink specified:
-{{{
-gst-launch filesrc location=/media/sda1/big_buck_bunny_720p_surround.avi ! \
-  decodebin2 ! mfw_v4lsink device=/dev/video16
-}}}
-
-There is also '''playbin2''' which just needs to be given a source and a sink
-{{{
-gst-launch playbin2 uri=file:///home/root/hawaii.mp4 \
-  video-sink="mfw_v4lsink device=/dev/video16"
-}}}
-
-Note that these 'bin' elements may not always do what you want or expect (they may not use hardware decode for example).
-
-
-== Freescale gplay ==
-Or if simplicity is required, the Freescale GStreamer based '''gplay''' app will try to determine the file type and play it back as best as it can.
-{{{
-gplay /media/sda1/big_buck_bunny.mov
-}}}
+Playback of a file that has both audio and video requires a slightly more complex pipeline than the standard [wiki:Yocto/gstreamer/audio audio] and [wiki:Yocto/gstreamer/video video] pipelines.
+
+Generally, a mixed media pipeline will consist of a demuxer (to split audio and video), individualized pipelines per video stream and audio stream, and {{{queue}}} elements to provide asynchronous playback of each stream type (which basically relates to using multiple threads of execution so that one element doesn't block the pipeline waiting for more data).
+
+The examples on this page will refer to GStreamer-1.0. To see GStreamer-0.10 (deprecated) examples, please see this [http://trac.gateworks.com/wiki/Yocto/gstreamer/multimedia?version=1 older revision page].
+
+
+[=#named-elements]
+== Named Elements, queues, and Multiple pipelines with gst-launch ==
+When mixing audio and video elements with {{{gst-launch}}} one must make use of multiple pipelines using {{{named elements}}}. When developing GStreamer based applications you create 'Bin' elements that put multiple elements together in a bin.
+
+The {{{name}}} property can be specified on any element in a pipeline and by default if not specified it will be set to the previous name (if any). Multiple pipelines can be provided to {{{gst-launch}}} and connected together by their names by either sourcing a pipeline with a name followed by a '.' or sinking a pipeline to a name followed by a '.'.
+
+This is best explained with some examples:
+ * Encoding a stream with audio and video content into an AVI file:
+{{{#!bash lineno=1
+gst-launch-1.0 \
+  v4l2src \
+    ! $VIDEO_CAPABILITIES \
+    ! mux. \
+  alsasrc \
+    ! $AUDIO_CAPABILITIES \
+    ! mux. \
+  avimux name=mux \
+    ! filesink location=test.avi
+}}}
+  - The {{{v4l2src}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}}
+  - The {{{alsasrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}}
+  - The {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}}
+ * Decoding a stream with audio and video content from an AVI file:
+{{{#!bash lineno=1
+gst-launch-1.0 \
+  filesource location=test.avi \
+    ! avidemux name=demux \
+  demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \
+  demux. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink
+}}}
+  - The {{{filesource}}} pipeline ends with {{{name=demux}}} which means the output of this pipeline will be sent to all pipelines with a {{{demux.}}} source (who's types have been successfully negotiated)
+  - The audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3)
+  - The video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
+  - Queue elements are used to keep one pipeline or element from blocking another. For example, if the mpeg4videoparse element needs more data from the avidemux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering
+
+
+[=#mux]
+== Muxing Mixed Content ==
+Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the combining or muxiplexing on the stream creation side are called 'Muxers'.
+
+You can use {{{gst-inspect}}} to see a list of most of these using grep:
+{{{
+gst-inspect-1.0 | grep -i muxer | grep -vi de
+}}}
+
+Some common examples:
+- mpegtsmux: MPEG Transport Stream Muxer
+- mpegpsmux: MPEG Program Stream Muxer
+- matroskamux: Matroska muxer
+- avimux: Avi muxer
+- qtmux: !QuickTime Muxer
+- oggmux: Ogg muxer
+
+To mux mixed content together include one of these elements following the audio and video pipelines.
+
+Examples:
+* Encoding a stream with audio and video content into an AVI file:
+{{{#!bash lineno=1
+gst-launch-1.0 \
+  videotestsrc \
+    ! $VIDEO_CAPABILITIES \
+    ! mux. \
+  audiotestsrc \
+    ! $AUDIO_CAPABILITIES \
+    ! mux. \
+  avimux name=mux \
+    ! filesink location=test.avi
+}}}
+  - the {{{videotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated.
+  - the {{{audiotestsrc}}} pipeline ends with {{{mux.}}} which means its output is sent to the pipeline who's name is {{{mux}}} and who's format has been successfully negotiated.
+  - the {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} and it understands how to multiplex the two types of data together into its output which is written to the file test.avi
+
+
+=== Example: Capture Video and Audio Through HDMI Port ===
+
+ * Note the -e switch is needed to properly terminate for playing back in a video player like VLC
+{{{#!bash lineno=1
+gst-launch-1.0 -e imxv4l2videosrc device=/dev/video0 queue-size=6 do-timestamp=true \
+! queue flush-on-eos=true silent=true leaky=0 max-size-buffers=0 max-size-time=0 max-size-bytes=0 \
+! imxipuvideotransform \
+! imxvpuenc_h264 bitrate=10000 \
+! h264parse disable_passthrough=true \
+! queue flush-on-eos=true silent=true leaky=0 max-size-buffers=0 max-size-time=0 max-size-bytes=0 \
+! mux. alsasrc device="sysdefault:CARD=tda1997xaudio" \
+! audioconvert \
+! imxmp3audioenc bitrate=128 \
+! mux.  mp4mux name=mux \
+! filesink location=audiovideo.mp4 sync=true
+}}}
+
+=== Example: Capture MPEG4 video, MP3 Audio muxed together in a AVI File ===
+To capture, encode, and output audio and video using MPEG4 video compression, MP3 audio compression, and an AVI file format you could use:
+{{{#!bash lineno=1
+gst-launch-1.0 \
+  imxv4l2videosrc device=/dev/video0 \
+    ! imxvpuenc_mpeg4 bitrate=10000 \
+    ! mux. \
+  alsasrc device="sysdefault:CARD=sgtl5000audio" \
+    ! audioconvert ! imxmp3audioenc bitrate=128 \
+    ! mux. \
+  avimux name=mux \
+    ! filesink location=test.avi
+}}}
+  - The {{{imxv4l2videosrc}}} pipeline ends with {{{mux.}}} which means its output (video/mpeg) is sent to the pipeline who's name is {{{mux}}}
+  - The {{{alsasrc}}} pipeline ends with {{{mux.}}} which means its output (audio/mpeg) is sent to the pipeline who's name is {{{mux}}}
+  - The {{{avimux}}} pipeline specifies {{{name=mux}}} therefore it takes as a source all pipelines that ended with {{{mux.}}} and it understands how to multiplex the two types of data together into its output which is written to the file test.avi.
+
+
+
+[=#demux]
+== De-muxing mixed content ==
+Often a multi-media stream will consist of mixed audio and video streams that are multiplexed (aka 'muxed') together into a single bitstream. The GStreamer elements that perform the de-multiplexing on the stream consumption side are called 'De-Muxers'.
+
+You can use {{{gst-inspect}}} to see a list of most of these using grep:
+{{{
+gst-inspect-1.0 | grep -i 'de\?muxer'
+}}}
+
+Some common examples:
+- tsparse: MPEG transport stream parser
+- tsdemux: MPEG transport stream demuxer
+- matroskademux: Matroska demuxer
+- avidemux: Avi demuxer
+- qtdemux: !QuickTime demuxer
+- oggdemux: Ogg demuxer
+
+To de-mux mixed content include one of these elements following the audio and video pipelines. Note that unlike muxing typically you also need to use a {{{parser}}} element to parse the bitstream and break it into discrete buffers that the downstream decoder can understand.
+
+Some common parsers:
+- ogmaudioparse: OGM audio stream parser
+- ogmvideoparse: OGM video stream parser
+- aacparse: AAC audio stream parser
+- amrparse: AMR audio stream parser
+- ac3parse: AC3 audio stream parser
+- flacparse: FLAC audio parser
+- mpegaudioparse: MPEG1 Audio Parser
+- h263parse: H.263 parser
+- h264parse: H.264 parser
+- mpegvideoparse: MPEG video elementary stream parser
+- mpeg4videoparse: MPEG 4 video elementary stream parser
+- pngparse: PNG parser
+- vc1parse: VC1 parser
+
+Examples:
+ * decoding a stream with audio and video content from an AVI file:
+{{{#!bash lineno=1
+gst-launch-1.0 \
+  filesource location=test.avi \
+    ! avidemux name=demux \
+  demux. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink \
+  demux. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink
+}}}
+  - the {{{filesource}}} pipeline ends with {{{name=demux}}} which means the output of this pipeline will be sent to all pipelines with a {{{demux.}}} source (who's types have been successfully negotiated)
+  - the audio pipeline consisting of the ac3parse element will source buffers that are supported by its sink capabilities (ie audio/x-ac3, audio/x-eac3, audio/ac3)
+  - the video pipeline consisting of the mpeg4videoparse element will source buffers that are supported by its sink capabilities (ie video/mpeg, video-x-divx)
+  - queue elements are used to keep one pipeline or element from blocking another. For example, if the mpeg4videoparse element needs more data from the avidemux element before it can decode a frame and send it down its pipeline it would normally stall the pipeline unless a queue element was in place to allow buffering
+
+
+[=#ex1]
+=== Example: Playback Matroska file with WEBM video and Vorbis Audio ===
+An example file consisting of a Matroska file container that includes WEBM encoded video and Vorbis encoded audio can be found at the [https://mango.blender.org/download/ Tears of Steel] download site. Tears of Steel is a relatively popular video (Creative Commons License) that has several encodings.
+
+Video file downloadable [http://media.xiph.org/mango/tears_of_steel_1080p.webm here]
+
+For this file we can use the {{{filessrc}}} source element, the {{{matroskademux}} demuxer, the {{{ivorbisdec}}} Vorbos audio decoder, and the {{{imxg2dvideosink}}} sink element:
+{{{#!bash
+gst-launch-1.0 \
+  filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \
+    matroskademux name=d \
+  d. ! queue ! ivorbisdec  ! queue ! alsasink device=hw:1,0 \
+  d. ! queue ! imxvpudec   ! queue ! imxg2dvideosink framebuffer=/dev/fb0
+}}}
+
+
+[=#ex2]
+=== Example: Playback Quicktime file with H.264/AVC video and AAC Audio ===
+An example file consisting of a Quicktime file container that includes H.264/AVC encoded video and AAC encoded audio can be found at the [https://durian.blender.org/download/ Sintel] download site. Sintel is a relatively popular video (Creative Commons License) that has several encodings.
+
+Video file downloadable [https://download.blender.org/durian/trailer/sintel_trailer-1080p.mp4 here]
+
+For this file we can use the {{{filesrc}}} source element, the {{{h264parse}}} parser element along with the {{{imxvpudec}}} decoder element, and the {{{avdec_aac}}} audio decoder element.
+{{{#!bash
+ gst-launch-1.0  filesrc location=/home/root/sintel_trailer-1080p.mp4  ! \
+   qtdemux name=d \
+ d. ! queue ! h264parse ! imxvpudec ! imxg2dvideosink \
+ d. ! queue ! avdec_aac ! audioconvert ! alsasink
+}}}
+
+
+=== Example: Playback MPEG-TS file ===
+
+Video only:
+{{{
+gst-launch-1.0 \
+  filesrc location=/tmp2/T2C00201_1080p60_Crop.ts ! \
+  tsdemux ! mpegvideoparse ! imxvpudec ! imxipuvideosink sync=false async=false
+}}}
+
+Video + Audio
+{{{
+gst-launch-1.0 \
+  filesrc location=/tmp2/T2C00201_1080p60_Crop.ts \
+    ! tsdemux name=demux \
+  demux. ! queue ! mpegaudioparse ! queue ! mad ! audioconvert ! queue ! alsasink \
+  demux. ! queue ! mpegvideoparse ! queue ! imxvpudec ! queue ! imxg2dvideosink sync=false async=false
+
+}}}
+
+
+[=#bin]
+== Bin elements ==
+A '''Bin''' element refers to a group of elements strung together and referenced as one. However, there are stand-alone elements that provide some automatic negotiation of sub-elements which use this concept.
+
+[=#playbin]
+=== GStreamer {{{playbin}}} ===
+GStreamer {{{playbin}}} element attempts to create a pipeline that will play both the audio and video portions of a file. For example:
+{{{
+#!bash
+gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
+}}}
+
+The above pipeline will attempt to output to the first video device and first audio devices found. However, you can further specify this by:
+{{{
+#!bash
+gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi audio-sink="alsasink device=hw:1,0"
+}}}
+
+Please type {{{gst-inspect-1.0 playbin}}} to see more options.
+
+[=#decodebin]
+=== GStreamer {{{decodebin}}} ===
+The GStreamer plugin {{{decodebin}}} is very useful if you're unsure of which decoder to use on a stream. For example, we can replace the example under [#ex1 the first example] with the following:
+{{{
+#!bash
+gst-launch-1.0 \
+  filesrc location=/media/open-media/tears_of_steel_1080p.webm do-timestamp=true typefind=true ! \
+  matroskademux name=d \
+  d. ! queue ! ivorbisdec ! queue ! alsasink device=hw:1,0 \
+  d. ! queue ! decodebin  ! queue ! imxg2dvideosink framebuffer=/dev/fb0
+}}}
+
+Note that {{{decodebin}}} doesn't always choose the correct decoder, so be wary of this. It is similar to {{{playbin}}} in that it aids in creating a dynamic pipeline.
+
+
+[=#gst-play]
+=== GStreamer {{{gst-play-1.0}}} ===
+The stand-alone application {{{gst-play}}} is is a program that utilizes the {{{playbin}}} element and thus can be used for playback of many file types. The above example {{{gst-launch-1.0 playbin uri=file:///media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi}}} can be replaced with:
+{{{
+#!bash
+gst-play-1.0 /media/open-media/big_buck_bunny_1080p_mp4v_ac3_5.1.avi
+}}}
+
+
+[=#determining-pipelines]
+== How to determine what pipeline is needed to decode and play ==
+Sometimes the above Bin elements are not flexible enough and you need to determine exactly what pipeline you can use to decode and play a stream.
+
+The {{{gst-launch}}} application provides a couple of useful debugging tools that can help with this:
+ * using {{{GST_DEBUG_DUMP_DOT_DIR}}} and Graphviz
+ * using {{{gst-launch -v}}}
+
+[=#filter-graph]
+=== GST_DEBUG_DUMP_DOT_DIR ===
+You can set the GST_DEBUG_DUMP_DOT_DIR env variable to a directory which will cause {{{gst-launch}}} to output a {{{.dot}}} file for each phase of the pipeline then use a tool such as {{{Graphviz}}} to visualize the {{{.dot}}} file.
+
+Example:
+ * use playbin to playback a file:
+{{{
+root@ventana:~# GST_DEBUG_DUMP_DOT_DIR=/tmp/dot gst-launch-1.0 playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
+}}}
+  - hit Cntl-C after decoding starts to exit early
+ * see the dot files created:
+{{{
+root@ventana:~# ls /tmp/dot
+0.00.00.108710334-gst-launch.NULL_READY.dot
+0.00.00.490807334-gst-launch.READY_PAUSED.dot
+0.00.00.506736000-gst-launch.PAUSED_PLAYING.dot
+0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
+0.00.03.254000001-gst-launch.PAUSED_READY.dot
+}}}
+ * transfer to a PC and use something like {{{xdot}}} to view:
+{{{
+xdot 0.00.03.135202001-gst-launch.PLAYING_PAUSED.dot
+}}}
+  - zoom in along the graph and you can see that:
+   - {{{GstFileSrc}}} is the source,
+   - {{{GstAviDemux}}} is used to demux to audio/x-ac3,
+   - {{{GstAc3Parse}}} is used to parse the audio into audio frames,
+   - {{{GstMpeg4VParse}}} is used to parse the video into video frames,
+   - {{{GstImxVpuDec}}} is used to decode the video from video/mpeg to video/x-raw,
+   - {{{ GstA52Dec}}} is used to decode the audio from audio/x-ac3 to audio/x-raw,
+   - etc
+  - Note that some hunting with {{{gst-inspect}}} must be done to determine what elements coorespond to the above class names
+
+Reference:
+ - http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+11%3A+Debugging+tools
+
+=== gst-launch -v ===
+The verbose debugging from {{{gst-launch -v}}} can show you the negotiation that takes place as a pipeline moves through its stages.
+
+Example:
+{{{
+gst-launch-1.0 -v playbin uri=file:///mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi
+}}}
+
+examining the verbose output can show you the following:
+ * container: AVI: avidemux
+ * video: MPEG-4 4481kbps min, 6668kbps max: mpeg4videoparse ! imxvpudec
+ * audio: AC3 48khz 5channels: ac3parse ! a52dec
+
+Therefore you can use these pipelines to decode and play:
+ * video only (output to imxipuvideosink fb0)
+{{{
+gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! mpeg4videoparse ! imxvpudec ! imxipuvideosink
+}}}
+ * audio only (output to hdmi audio sink)
+{{{
+gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
+}}}
+ * both audio and video
+{{{
+gst-launch-1.0 -v filesrc location=/mnt/big_buck_bunny_1080p_ac3-5.1_mp4.avi ! avidemux name=d \
+  d. ! queue ! mpeg4videoparse ! imxvpudec ! imxipuvideosink \
+  d. ! queue ! ac3parse ! a52dec ! audioconvert ! alsasink device="sysdefault:CARD=imxhdmisoc"
+}}}