[[PageOutline]] = Latency = Latency is the term used to describe how long it takes for video from the source side to get to the sink side. For example, if a video system was described as having a latency of 1 second, that would mean that it takes 1 second for the video to get from the capture (or reading a file) side to the actual display. This section will attempt to describe some real world latencies that we have seen using gstreamer and our product. Because latency is highly dependent on the different 'knobs' that you can turn, each section will be a very specific test setup. This is important, as you cannot reliably compare latencies when you start to introduce/remove certain feature, video capture ports/types, sink types etc. The below values differ in capture/display device+driver in the following ways that affect latency * Frames Per Second (FPS) * Interlace vs. Progressive * Capture device Output format (RAW vs JPEG) * Capture device's inherent latency * Display device's inherent latency Please note, the below values are '''not''' comparisons, but rather completely isolated and individual examples that happen to be categorized based on input types. This is because latency is affected by several things, some of which aren't always applicable to the scenario demonstrated. To see prior numbers, please see [http://trac.gateworks.com/wiki/Yocto/gstreamer/latency?version=3 this older page revision]. [=#measure] == Measuring Latency == We've come up with a synchronous method of testing total system latency. We wrote [https://github.com/Gateworks/gateworks-sample-apps/blob/master/gw-latency-tester/gw-latency-tester.c a program] that uses the on-board gpio's to control an ambient light sensor ([http://www.digikey.com/product-detail/en/TEPT5700/751-1059-ND/1681193 Vishay Semiconductor Opto Division TEPT5700]) and a bright blue LED. A camera would be pointed to the LED while a GStreamer pipeline would pipe it to some sink. Finally, the [https://github.com/Gateworks/gateworks-sample-apps/blob/master/gw-latency-tester/gw-latency-tester.c program] controlling the LED and light sensor turns the LED on, measures how long it took the ambient light sensor to detect it, and repeat several times. If you would like to repeat our findings, please build the [https://github.com/Gateworks/gateworks-sample-apps/blob/master/gw-latency-tester/gw-latency-tester.c gw-latency-tester] program and use in conjunction with the following hardware: * [http://www.digikey.com/product-detail/en/TEPT5700/751-1059-ND/1681193 Vishay Semiconductor Opto Division TEPT5700] * Bright Blue or White LED Pinout to the DIO connector is as follows: {{{ DIO0 - Power to the ambient light sensor (Collector Side), tie other end to DIO3 DIO1 - Power to the LED, tie other end to GND DIO2 - [OPTIONAL] Status LED, tie other end to GND DIO3 - Receiver end of ambient light sensor (Emitter Side), tie other end to DIO0 GND - }}} Usage is as follows: {{{ root@ventana:~# ./gw-latency-tester gw-latency-tester [] [] [] DIO: 0 1 2 3 ---------- GW54xx: 9 19 41 42 GW53xx: 16 19 17 20 GW52xx: 16 19 17 20 GW51xx: 16 19 17 18 GW551x: Unsupported GW552x: 16 19 17 20 }}} The below results will show how the program was run, so please refer to them for examples. In general (unless otherwise specified), we use the following displays configured in the following ways: * HDMI: Hanns-G HL273 * Resolution: 1920x1080p-60 * Manufacturer specified latency: 2ms * LVDS: !HannStar HSD100PXN1 * Resolution: 1024x768p-60 * Manufacturer specified rise time latency: 3-6ms * Manufacturer specified fall time latency: 9-18ms * CVBS: Generic 5" inch TFT LCD Monitor (AC-530DB) * Resolution: D:720x480i-60 === Accounting for Latency === When we say a system has a latency of {{{66ms}}}, this number only accounts from end to end devices. However, if we know certain device latencies (such as a displays), we can subtract that off and get fine-tuned numbers. Using the measurement method described [wiki:Yocto/gstreamer/latency#measure above], calculating the circuitry involved RC time constants, LED Diode rise/fall times, and the response time of the ambient light sensor, we get a "circuitry latency" of: (sensor fall time response) + (led rise time response) == {{{(2us + 13us) + (200ns + 13us) = 28.200us}}} Because of DIO levels, we'll round that number down to 28us. Plus add in latency from an input (let's assume a camera has exactly 1framerate of latency at 30fps) at 33ms, and finally add in the display latency (we'll assume 4ms). Adding these together produces a total of: {{{28us + 33ms + 4ms = 37.28ms}}} of latency. We can now say that latency inherent to our product is {{{66ms - 37.28ms = 28.72ms}}}. == Latency Summary Table == Here is a summary table of the below results. Note that customers have reported down to 4ms latency on 1080p30 HDMI input to 1080p30 HDMI output eliminating the source (camera) latency. Note the GoPro has been reported to have 60-100ms of latency itself, however, this is unverified by Gateworks. Loopback: ||= Type =||= Capture =||= Display =||= End-To-End Latency =|| || Analog || CVBS(NTSC:'''480i@30fps''') [[BR]] imxv4l2videosrc(IPU) || CVBS monitor [[BR]] imxg2dvideosink(GPU) || 56ms || || Analog || CVBS(NTSC:'''480i@30fps''') [[BR]] imxv4l2videosrc(IPU) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 66ms || || Analog || CVBS(NTSC:'''480i@30fps''') [[BR]] imxv4l2videosrc(IPU) || LVDS monitor [[BR]] imxg2dvideosink(GPU) || 76ms || || || || Analog || PCI-AVC8000(NTSC:'''480i@30fps''') [[BR]] v4l2src || CVBS monitor [[BR]] imxg2dvideosink(GPU) || 55ms || || Analog || PCI-AVC8000(NTSC:'''480i@30fps''') [[BR]] v4l2src || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 67ms || || Analog || PCI-AVC8000(NTSC:'''480i@30fps''') [[BR]] v4l2src || LVDS monitor [[BR]] imxg2dvideosink(GPU) || 78ms || || || || Digital || HDMI(GoPro:'''720p@60fps''') [[BR]] imxv4l2videosrc(IPU) || CVBS monitor [[BR]] imxg2dvideosink(GPU) || 92ms || || Digital || HDMI(GoPro:'''720p@60fps''') [[BR]] imxv4l2videosrc(IPU) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 96ms || || Digital || HDMI(GoPro:'''720p@60fps''') [[BR]] imxv4l2videosrc(IPU) || LVDS monitor [[BR]] imxg2dvideosink(GPU) || 117ms || || || || Digital || HDMI(GoPro:'''480p@60''') [[BR]] imxv4l2videosrc(IPU) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 101ms || || Digital || HDMI(GoPro:'''720p@30''') [[BR]] imxv4l2videosrc(IPU) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 113ms || || Digital || HDMI(GoPro:'''960p@60''') [[BR]] imxv4l2videosrc(IPU) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 157ms || || Digital || HDMI(GoPro:'''1080p@60''') [[BR]] imxv4l2videosrc(IPU) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 175ms || RTSP Streaming: ||= Capture =||= Transform =||= Encode(VPU) =||= Stream =||= Decode(VPU) =||= Display =||= End-To-End Latency =|| || CVBS(NTSC:'''480i@30fps''') [[BR]]yuv422smp [[BR]] imxv4l2videosrc(IPU) || imxipuvideotransform(IPU) || imxvpuenc_h264 [[BR]] 3mbps CBR || RTSP [[BR]] latency=10ms || imxvpudec [[BR]] (h264) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 98ms || || PCI-AVC8000(NTSC:'''480i@30fps''') [[BR]]yuv422smp [[BR]] imxv4l2videosrc(IPU) || imxipuvideotransform(IPU) || imxvpuenc_h264 [[BR]] 3mbps CBR || RTSP [[BR]] latency=10ms || imxvpudec [[BR]] (h264) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 98ms || || HDMI(GoPro:'''720p@60fps''') [[BR]]yuv422smp [[BR]] imxv4l2videosrc(IPU) || imxipuvideotransform(IPU) || imxvpuenc_h264 [[BR]] 5.5mbps CBR || RTSP [[BR]] latency=10ms || imxvpudec [[BR]] (h264) || HDMI monitor [[BR]] imxg2dvideosink(GPU) || 160ms || Notes: 1. {{{imxipuvideosink}}} and {{{imxg2dvideosink}}} have the same latencies. 2. LVDS display requires a transform from YUV to RGB colorspace. 2. HDMI capture using yuv422smp requires a transform before encoding while yuv422bt656 does not. 3. IPU transforms with a width or height greater than 1024 (720p/720i/1080p/1080i) require striping equating to two transforms == CVBS (Analog) In == Using a 208C Wired Micro Mini Camera with NTSC 720x480@30FPS output captured via the Ventana Analog in ([http://www.analog.com/media/en/technical-documentation/data-sheets/ADV7180.pdf adv7180 SDTV Video Decoder]) with 8bit ITU-R BT656 output to the IMX6 CSI, the following latencies were found using a GW5400: === CVBS in to HDMI out === * CVBS Input to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video1 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.082425s Min Latency: 0.052815s Max Jitter : 0.029610s Exponential Moving Average (alpha=0.1): 0.065967s }}} === CVBS in to LVDS out === * CVBS Input to LVDS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video1 ! imxg2dvideosink framebuffer=/dev/fb4 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.082155s Min Latency: 0.055859s Max Jitter : 0.026296s Exponential Moving Average (alpha=0.1): 0.076021s }}} === CVBS in to CVBS out === * CVBS Input to CVBS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video1 ! imxg2dvideosink framebuffer=/dev/fb2 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.066111s Min Latency: 0.045768s Max Jitter : 0.020343s Exponential Moving Average (alpha=0.1): 0.056305s }}} === CVBS in to RTSP out === RTSP is an audio/video streaming standard that is commonly supported by various player clients. Notes: * [https://www.ietf.org/rfc/rfc2326.txt RTSP RFC] * Note that latency=0 on the receiver will get it to display frames as soon as it receives them. However, depending on network congestion/frame sizes, this could cause video stuttering due to varying latencies. * These tests were done through a ZyXEL GS1100-24 Switch with 4 ports having active traffic. * RTSP clients (players) will have a configuration that specifically adds latency by stream buffering to allow for audio/video synchronization which can be on the order of 10 to 1000ms. For gstreamer rtspsrc this is the 'latency' parameter - we set it to 10ms to minimize latency as a value of 0 can confuse gstreamer. * See the [wiki:Yocto/gstreamer/streaming#rtsp gstreamer/streaming] page for more examples on RTSP streaming The below sections show various gstreamer sinks running on a GW5304 running Yocto 1.8 and an Ubuntu based PC. ==== Yocto 1.8 GW5304 ==== Receiver latency set to 10, bitrate set to 3mbit/s: * CVBS input to RTSP Server (gst-variable-rtsp-server) * receiver: {{{ gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! \ rtph264depay ! h264parse ! imxvpudec ! imxg2dvideosink }}} * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video1 -s imxv4l2videosrc -r 0 -b 3000 }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.100264334s Min Latency: 0.057308000s Max Jitter : 0.042956334s Exponential Moving Average (alpha=0.1): 0.092784375s }}} Here is an example of using {{{udpsink}}} instead of the {{{gst-variable-rtsp-server}}} application. Receiver latency set to 10, bitrate set to 3mbit/s: * CVBS input to RTSP Server (gst-variable-rtsp-server) * sender: {{{ gst-launch-1.0 imxv4l2videosrc device=/dev/video3 ! \ imxipuvideotransform ! imxvpuenc_h264 bitrate=3000 ! \ rtph264pay ! udpsink host=172.24.10.63 port=9001 }}} * receiver: {{{ #!bash gst-launch-1.0 udpsrc port=9001 \ caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! \ rtph264depay ! h264parse ! imxvpudec ! imxg2dvideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.116625333s Min Latency: 0.083219333s Max Jitter : 0.033406000s Exponential Moving Average (alpha=0.1): 0.098617987s }}} Receiver latency set to 100, bitrate set to 3mbit/s: * CVBS input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ gst-variable-rtsp-server -i /dev/video1 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=100 ! \ rtph264depay ! h264parse ! imxvpudec ! imxg2dvideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.166653334s Min Latency: 0.131223333s Max Jitter : 0.035430001s Exponential Moving Average (alpha=0.1): 0.139983921s }}} ==== Ubuntu 14.04 i5 Desktop PC ==== Receiver latency set to 10, bitrate set to 3mbit/s: * CVBS input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video1 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! \ decodebin ! autovideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.118248000s Min Latency: 0.056305334s Max Jitter : 0.061942666s Exponential Moving Average (alpha=0.1): 0.081248197s }}} Receiver latency set to 100, bitrate set to 3mbit/s: * CVBS input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video1 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=100 ! \ decodebin ! autovideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.167262000s Min Latency: 0.138012667s Max Jitter : 0.029249333s Exponential Moving Average (alpha=0.1): 0.164639100s }}} == HDMI (Digital) In == This section is for boards using hdmi capture to various outputs. The in-depth results are based off of the !GoPro Hero3 camera. Please note that the camera itself is responsible for a lot of the latency. Here is an example of how various HDMI output devices can vary (using the same output format and capture format): * HDMI input (yuv422smp - '''Dino-Lite Digital Microscope @ 1280x720p@60''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.116650000s Min Latency: 0.068762333s Max Jitter : 0.047887667s Exponential Moving Average (alpha=0.1): 0.109525081s }}} * HDMI input (yuv422smp - '''Incredisonic Dash Cam @ 1280x720p@60''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.130328000s Min Latency: 0.085225000s Max Jitter : 0.045103000s Exponential Moving Average (alpha=0.1): 0.117414606s }}} * HDMI input (yuv422smp - '''!GoPro Hero3 @ 1280x720p@60''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.115457666s Min Latency: 0.095496333s Max Jitter : 0.019961333s Exponential Moving Average (alpha=0.1): 0.096331845s }}} As you can see, camera choice plays a huge factor in latency numbers. In this case, the !GoPro had 20ms less latency than the Incredisonic Dash Cam, and 10ms less latency than the Dino-Lite Digital Microscope. Also note that HDMI sources have variying latencies based on their output formats. For example 1080p@30 on the !GoPro has increased latencies due it its own performance. Here are some !GoPro Hero3 latency numbers at different output modes: * HDMI input (yuv422smp - !GoPro Hero3 @ '''1080p@60''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.198507s Min Latency: 0.167948s Max Jitter : 0.030559s Exponential Moving Average (alpha=0.1): 0.175236s }}} * HDMI input (yuv422smp - !GoPro Hero3 @ '''960p@60''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.176905000s Min Latency: 0.137760667s Max Jitter : 0.039144333s Exponential Moving Average (alpha=0.1): 0.156957710s }}} * HDMI input (yuv422smp - !GoPro Hero3 @ '''720p@30''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.138299333s Min Latency: 0.096406667s Max Jitter : 0.041892666s Exponential Moving Average (alpha=0.1): 0.112956985s }}} * HDMI input (yuv422smp - !GoPro Hero3 @ '''480p@60''') to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.116750333s Min Latency: 0.093424667s Max Jitter : 0.023325666s Exponential Moving Average (alpha=0.1): 0.101177281s }}} Also note that any resolution larger than 1080x720 (720p/720i) requires striping for any IPU transform that needs to be performed on the IMX (ie colorspace transform for encoding when capturing in yuv422smp mode and output transform when displaying on a device that differs in format). ==== HDMI in to HDMI out ==== With the !GoPro configured to '''output 1280x720p@60''', pixel format YUV420, the following latencies were found using a GW5400: ===== While capturing in yuv422smp: ===== * HDMI input ('''yuv422smp''' - !GoPro Hero3) to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.115457666s Min Latency: 0.095496333s Max Jitter : 0.019961333s Exponential Moving Average (alpha=0.1): 0.096331845s }}} ===== While capturing in bt656: ===== * HDMI input ('''yuv422bt656''' - !GoPro Hero3) to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb0 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.113161667s Min Latency: 0.085511333s Max Jitter : 0.027650334s Exponential Moving Average (alpha=0.1): 0.097953780s }}} ==== HDMI in to LVDS out ==== ===== While capturing in yuv422smp: ===== * HDMI input ('''yuv422smp''' - !GoPro Hero3) to LVDS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb4 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.127060667s Min Latency: 0.106143666s Max Jitter : 0.020917001s Exponential Moving Average (alpha=0.1): 0.117725942s }}} ===== While capturing in bt656: ===== * HDMI input ('''yuv422bt656''' - !GoPro Hero3) to LVDS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb4 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.173456000s Min Latency: 0.120619667s Max Jitter : 0.052836333s Exponential Moving Average (alpha=0.1): 0.131980293s }}} ==== HDMI in to CVBS out ==== ===== While capturing in yuv422smp: ===== * HDMI input ('''yuv422smp''' - !GoPro Hero3) to CVBS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb2 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.103408000s Min Latency: 0.091112000s Max Jitter : 0.012296000s Exponential Moving Average (alpha=0.1): 0.092611801s }}} ===== While capturing in bt656: ===== * HDMI input ('''yuv422bt656''' - !GoPro Hero3) to CVBS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video0 ! imxg2dvideosink framebuffer=/dev/fb2 }}} {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.114230334s Min Latency: 0.099739667s Max Jitter : 0.014490667s Exponential Moving Average (alpha=0.1): 0.100241924s }}} ==== HDMI in to RTSP out ==== RTSP is an audio/video streaming standard that is commonly supported by various player clients. Notes: * [https://www.ietf.org/rfc/rfc2326.txt RTSP RFC] * Note that latency=0 on the receiver will get it to display frames as soon as it receives them. However, depending on network congestion/frame sizes, this could cause video stuttering due to varying latencies. * These tests were done through a ZyXEL GS1100-24 Switch with 4 ports having active traffic. * RTSP clients (players) will have a configuration that specifically adds latency by stream buffering to allow for audio/video synchronization which can be on the order of 10 to 1000ms. For gstreamer rtspsrc this is the 'latency' parameter - we set it to 10ms to minimize latency as a value of 0 can confuse gstreamer. * See the [wiki:Yocto/gstreamer/streaming#rtsp gstreamer/streaming] page for more examples on RTSP streaming The below sections show various gstreamer sinks running on a GW5304 running Yocto 1.8 and an Ubuntu based PC. ===== Yocto 1.8 GW5304 ===== Receiver latency set to 10, bitrate set to 5.5mbit/s: * HDMI input (yuv422smp - !GoPro Hero3) to HDMI Output * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video0 -s imxv4l2videosrc -r 0 -b 5500 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! \ rtph264depay ! h264parse ! imxvpudec ! imxg2dvideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.183257333s Min Latency: 0.149575667s Max Jitter : 0.033681666s Exponential Moving Average (alpha=0.1): 0.160407397s }}} ===== Ubuntu 14.04 i5 Desktop PC ===== Receiver latency set to 10, bitrate set to 5.5mbit/s: * HDMI input (yuv422smp - GoPro Hero3) to HDMI Output * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video0 -s imxv4l2videosrc -r 0 -b 5500 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! \ decodebin ! autovideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 === Summary === Ran 60 times Max Latency: 0.127060667s Min Latency: 0.106143666s Max Jitter : 0.020917001s Exponential Moving Average (alpha=0.1): 0.117725942s }}} == avc8000nano (miniPCIe capture card) in == This section is for boards using the avc8000nano miniPCIe capture card to various outputs. Using a 208C Wired Micro Mini Camera with NTSC 720x480@30FPS output captured via the Ventana Analog in ([http://www.analog.com/media/en/technical-documentation/data-sheets/ADV7180.pdf adv7180 SDTV Video Decoder]) with 8bit ITU-R BT656 output to the IMX6 CSI, the following latencies were found using a GW5400: === avc8000nano in to HDMI out === * avc8000nano Input to HDMI Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink framebuffer=/dev/fb0 }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.067630s Min Latency: 0.058137s Max Jitter : 0.009493s Exponential Moving Average (alpha=0.1): 0.066845s }}} === avc8000nano in to LVDS out === * avc8000nano Input to LVDS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink framebuffer=/dev/fb4 }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.090251s Min Latency: 0.059275s Max Jitter : 0.030976s Exponential Moving Average (alpha=0.1): 0.078264s }}} === avc8000nano in to CVBS out === * avc8000nano Input to CVBS Output {{{ #!bash gst-launch-1.0 imxv4l2videosrc device=/dev/video3 ! imxg2dvideosink framebuffer=/dev/fb2 }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 0 === Summary === Ran 60 times Max Latency: 0.062595s Min Latency: 0.049624s Max Jitter : 0.012971s Exponential Moving Average (alpha=0.1): 0.055531s }}} === avc8000nano in to RTSP out === RTSP is an audio/video streaming standard that is commonly supported by various player clients. Notes: * [https://www.ietf.org/rfc/rfc2326.txt RTSP RFC] * Note that latency=0 on the receiver will get it to display frames as soon as it receives them. However, depending on network congestion/frame sizes, this could cause video stuttering due to varying latencies. * These tests were done through a ZyXEL GS1100-24 Switch with 4 ports having active traffic. * RTSP clients (players) will have a configuration that specifically adds latency by stream buffering to allow for audio/video synchronization which can be on the order of 10 to 1000ms. For gstreamer rtspsrc this is the 'latency' parameter - we set it to 10ms to minimize latency as a value of 0 can confuse gstreamer. * See the [wiki:Yocto/gstreamer/streaming#rtsp gstreamer/streaming] page for more examples on RTSP streaming The below sections show various gstreamer sinks running on a GW5304 running Yocto 1.8 and an Ubuntu based PC. ==== Yocto 1.8 GW5304 ==== Receiver latency set to 10, bitrate set to 3mbit/s: * avc8000nano input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video3 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! \ rtph264depay ! h264parse ! imxvpudec ! imxg2dvideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.116920334s Min Latency: 0.059414667s Max Jitter : 0.057505667s Exponential Moving Average (alpha=0.1): 0.098957740s }}} Receiver latency set to 100, bitrate set to 3mbit/s: * avc8000nano input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video3 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=100 ! \ rtph264depay ! h264parse ! imxvpudec ! imxg2dvideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.166873667s Min Latency: 0.039013333s Max Jitter : 0.127860334s Exponential Moving Average (alpha=0.1): 0.156437334s }}} ==== Ubuntu 14.04 i5 Desktop PC ==== Receiver latency set to 10, bitrate set to 3mbit/s: * avc8000nano input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video3 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=10 ! \ decodebin ! autovideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.129218s Min Latency: 0.067037s Max Jitter : 0.062181s Exponential Moving Average (alpha=0.1): 0.091255s }}} Receiver latency set to 100, bitrate set to 3mbit/s: * avc8000nano input to RTSP Server (gst-variable-rtsp-server) * sender: {{{(imxv4l2videosrc ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay pt=96)}}} {{{ #!bash gst-variable-rtsp-server -i /dev/video3 -s imxv4l2videosrc -r 0 -b 3000 }}} * receiver: {{{ #!bash gst-launch-1.0 rtspsrc location=rtsp://172.24.10.180:9099/stream latency=100 ! \ decodebin ! autovideosink }}} * latency: {{{ #!bash gw-latency-tester 9 19 41 42 60 500000 === Summary === Ran 60 times Max Latency: 0.201775s Min Latency: 0.165823s Max Jitter : 0.035952s Exponential Moving Average (alpha=0.1): 0.177493s }}}