| Version 6 (modified by , 21 months ago) ( diff ) |
|---|
Venice Multimedia Connectivity
The GW720x, GW730x, and GW740x boards have a 50-pin MIPI Media Connector to support displays (LCD) and cameras. While this page is related to the software side of the MIPI connector, all hardware (pinouts, diagrams, etc) are on the venice/mipi page.
Note that while the GW740x has the same 50-pin MIPI Media Connector, the imx8mp SoC on the GW740x requires different Linux drivers for MIPI which are not supported yet
Kernel Support
Support for IMX8MM MIPI CSI capture, MIPI DSI display, and VPU VP8/H264 decode are added to the downstream 5.15-venice Gateworks kernel via patches.
While IMX8MM MIPI CSI support will be in the mainline v5.17 kernel, the MIPI DSI support in mainline is still being worked on. While there is support for the IMX8MM VPU decode making its way upstream there is no support currently for the IMX8MM VPU encoder.
For more info on IMX8MM kernel support see venice#linux
Venice Video Display
The IMX8M Mini SoC supports the following video display hardware capabilities:
- LCDIF Display controller supporting up to 1920x1200p60 via 4-lane MIPI DSI (operating up to a max bit rate of 1.5 Gbps)
- 3D GPU Core supporting:
- OpenGL ES 1.1, 2.0
- OpenVG 1.1
- TrustZone support using a local MMU to manage secure regions
- 2D GPU Core supporting:
- multi-source composition
- one-pass filter
Compatible display devices
Display Devices:
- DFROBOT DFR0678 7" 800x480 TFT Raspberry Pi DSI Touchscreen
- DFROBOT DFR0550 5" 800x480 TFT Raspberry Pi DSI Touchscreen
both of these displays consist of almost identical hardware:
- STM32F103C8T6-LQFP48 Microncontroller for backlight providing a backlight enable GPIO (on/off) - there is a physical thumbwheel to adjust brightness
- FT5316 I2C touchscreen controller without IRQ (requires a driver patch to allow 'polling' the controller); however the FT5x06 I2C runs 'through' the MCU which polls it and presents some different I2C API on the 15pin PFC I2C
- 800x480 WVGA TFT LCD panel
- ICN6211 MIPI-DSI bridge (MIPI DSI to MIPI DBI up to WXGA resolution); appears to be TC358762 compatible after the MCU configures it via I2C on power-up
The Linux drivers for these displays are:
- regulator:
- CONFIG_REGULATOR_RASPBERRYPI_TOUCHSCREEN_ATTINY=y drivers/regulator/rpi-panel-attiny-regulator.c rpi_panel_attiny_regulator.ko
- touchscreen:
- CONFIG_TOUCHSCREEN_EDT_FT5X06 edt-ft5x06.ko
- MIPI DSI / DRM:
- CONFIG_DRM drm.ko
- CONFIG_DRM_PANEL_SIMPLE drivers/gpu/drm/panel/panel-simple.c panel_simple.ko
- CONFIG_DRM_TOSHIBA_TC358762 drivers/gpu/drm/bridge/tc358762.c tc358762.ko
- CONFIG_DRM_MXSFB=y drivers/gpu/drm/mxsfb/ mxsfb.ko
- CONFIG_LOGO=y (optional; if you want the tux)
- CONFIG_DRM_SAMSUNG_DSIM=y drivers/gpu/drm/bridge/samsung-dsim.c samsung-dsim.ko (changes required that are not yet in mainline)
- CONFIG_DRM_SAMSUNG_DSIM_IMX=y drivers/gpu/drm/bridge/samsung-dsim-imx.c samsung-dsim-imx.ko (not yet in mainline)
- CONFIG_PHY_EXYNOS_MIPI_VIDEO=y drivers/phy/samsung/phy-exynos-mipi-video.c phy_exynos_mipi_video.ko (not yet in mainline)
Device-Tree changes:
- to add a device to your system you must properly configure your device-tree to describe how the device is connected to the host processor and other peripherals. Often this is most easily done via a device-tree overlay. The Gateworks venice kernels have dt overlays that support the DFROBOT 5in and 7in displays on the GW720x and GW730x boards. The dt overlay can be applied to the dt prior to booting the kernel by U-Boot. Examples:
- GW730x display:
setenv fdt_overlays imx8mm-venice-gw73xx-0x-rpidsi.dtbo saveenv
- GW720x display:
setenv fdt_overlays imx8mm-venice-gw72xx-0x-rpidsi.dtbo saveenv
- GW730x display:
The Linux runtime devices:
- /sys/class/backlight/7inch-touchscreen-panel-bl/ - back-light controller
- /sys/class/regulator/regulator.9/ - tc358762-power
- /sys/class/input/input1/ /dev/input/event1 - generic ft5x06 (79)
- /sys/class/drm/card0
- /sys/class/drm/card0-DPI-1
- /sys/class/graphics/fb0 - mxsfb_drv.c
Notes:
- no kernel cmdline needed for video out
- samsung-dsim-imx does not currently load by itself. Work around this by adding it to /etc/modules
# update /etc/modules to load them on boot echo "samsung-dsim-imx" >> /etc/modules
Hardware Accelerated Qt5
see qt
Hardware Accelerated Desktop UI
While it is not typical to use an embedded system with a desktop UI we often are asked how to do it.
With the Ubuntu Focal (20.04) all the software is in place to give you 2D and 3D hardware acceleration on the IMX8MM with a modern kernel such as the Linux 5.15-venice kernel.
Here are the steps to add a desktop UI:
- Add a USB mouse and keyboard to your hardware configuration
- Add a compatible display to your hardware configuration
- add samsung-dsim-imx to /etc/modules (if you have not done so already)
echo samsung-dsim-imx >> /etc/modules - add a non-root user (required to login)
adduser gateworks
- add packages
apt update apt install gnome-session gnome-terminal # installs about 1.5GB of packages reboot # sync filesystem and reboot
- you'll notice dragging opaque windows around is fairly smooth as it is using hardware 2D acceleration
- Network manager will not allow access to interfaces in the GUI with "ifupdown" installed. If you plan to use network functions within the GUI you will need to remove/purge this package first. All network settings must be configured using the terminal.
- 3D opengl hardware acceleration:
apt install mesa-utils glmark2 glxgears glmark2
- Note you need to run glxgears from a terminal on the display
- glxgears shows ~160 fps on IMX8MM
- glmark2 score is 39 on IMX8MM
- GStreamer Video display
apt-get install gstreamer1.0-x gstreamer1.0-tools \ gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \ gstreamer1.0-libav gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 v4l-utils # using glvideosink will render 'within' GUI gst-launch-1.0 videotestsrc ! autovideosink
- Note: This pipeline is intended to be run in a terminal within the GUI as a non-root user
Venice Video Capture
The IMX8M Mini SoC supports a 4-lane MIPI CSI2 (MIPI Camera Serial Interface) which operates up to a maximum bit rate of 1.5 Gbps with features including:
- Configurable interface logic to support most commonly available CMOS sensors.
- Support for ITU-R BT.656 video interface as well as traditional sensor interface.
- 8-bit / 16-bit / 24-bit data port for YCbCr, YUV, or RGB data input.
- 8-bit / 10-bit / 16-bit data port for Bayer data input.
- Full control of 8-bit/pixel, 10-bit/pixel or 16-bit / pixel data format to 64-bit receive FIFO packing.
- 256 x 64 FIFO to store received image pixel data.
- Receive FIFO overrun protection mechanism.
- Embedded DMA controllers to transfer data from receive FIFO or statistic FIFO through AHB bus.
- Support 2D (data is chosen as x/y coordinate) DMA transfer from the receive FIFO to the frame buffers in the external memory.
- Support double buffering two frames in the external memory.
- Single interrupt source to interrupt controller from maskable interrupt sources: Start of Frame, End of Frame, Change of Field, FIFO full, FIFO overrun, DMA transfer done, BT.656 error and AHB bus response error.
- Configurable master clock frequency output to sensor.
- Supports simple deinterlacing of interlaced input.
CMOS image sensors are separated into two classes, dumb and smart. Dumb sensors are those that support only traditional sensor timing (Vertical SYNC and Horizontal SYNC) and output only RGB, YUV, and Bayer and statistics data, while smart sensors support ITU-R BT.656 video decoder formats and perform additional processing of the image (for example, image compression, image pre-filtering, and various data output formats). Statistics only work for Bayer data is 8-bit per pixel format.
Compatible capture devices
Capture Devices:
- RaspberryPi Camera Module v2: Sony IMX219 8MP image sensor with RAW8/RAW10 output
- Schematics
- Pin 11 is an active high enable pin which enables on-board 1.8V, 2.8V, 1.2V regulators
- Has its own on-board 24MHz osc
- Requires Gateworks GW16136 Raspberry Pi Adapter board
The Linux drives required for this camera are:
- CONFIG_VIDEO_IMX_MEDIA drivers/staging/media/imx/imx-media-common.c imx_media_common.ko
- CONFIG_VIDEO_IMX7_CSI drivers/staging/media/imx/{imx7-media-csi,imx7-mipi-csis,imx8mq-mipi-csi2}.c imx7_media_csi.ko imx7-mipi-csis.ko
- CONFIG_VIDEO_IMX219 drivers/media/i2c/imx219.c imx219.ko
Device-Tree changes:
- to add a device to your system you must properly configure your device-tree to describe how the device is connected to the host processor and other peripherals. Often this is most easily done via a device-tree overlay. The Gateworks venice kernels have dt overlays that support the IMX219 RaspberryPi v2 camera module on the GW720x and GW730x boards. The dt overlay can be applied to the dt prior to booting the kernel by U-Boot. Examples:
- GW730x imx219 camera:
setenv fdt_overlays imx8mm-venice-gw73xx-0x-imx219.dtbo saveenv
- GW720x imx219 camera:
setenv fdt_overlays imx8mm-venice-gw72xx-0x-imx219.dtbo saveenv
- GW730x imx219 camera:
Examples:
- capture 640x480 RAW8 at 10fps from IMX219 and display to framebuffer using gstreamer
# install gstreamer apt update apt-get install gstreamer1.0-x gstreamer1.0-tools \ gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \ gstreamer1.0-libav gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 v4l-utils # using glvideosink will render 'within' GUI gst-launch-1.0 videotestsrc ! autovideosink # find media device whos model is 'imx-media' MDEV=$(for i in `ls -d /sys/bus/media/devices/media*`; do if [ "$(cat $i/model)" = "imx-media" ]; then cat $i/dev | cut -d: -f2; fi; done) # find video device whos name is 'csi capture' VDEV=$(for i in `ls -d /sys/class/video4linux/video*`; do if [ "$(cat $i/name)" = "csi capture" ]; then cut -d: -f2 $i/dev; fi; done) # configure media controller links media-ctl --device /dev/media$MDEV --reset media-ctl --device /dev/media$MDEV --links "'imx219 2-0010':0->'imx7-mipi-csis.0':0[1]" media-ctl --device /dev/media$MDEV -v -V "'imx219 2-0010':0 [fmt:SRGGB8/640x480 field:none]" media-ctl --device /dev/media$MDEV -v -V "'csi':0 [fmt:SRGGB8/640x480 field:none]" # stream 640x480@30fps gst-launch-1.0 v4l2src device=/dev/video$VDEV ! video/x-bayer,format=rggb,width=640,height=480,framerate=10/1 ! bayer2rgb ! fbdevsink
- bayer2rgb can't keep up at 640x480@30fps and uses about 25% of the CPU when converting 640x480@10fps (as shown above)
- bayer2rgbneon uses appx 10% of CPU when converting 640x480@30fps
Notes:
- The DSI display driver only accepts RGB pixel format so a pixel conversion must occur if you want to display video from anything that does not output RGB (such as a RAW8 camera or YUV from the VPU decoder). There is no hardware accelerated mem2mem driver at this time (and its not clear if there will be on the IMX8MM). There is a bayer2rgb lib with NEON support and a gstreamer plugin that can be used for RAW8 to RGB:
- https://gitlab-ext.sigma-chemnitz.de/ensc/bayer2rgb
- https://gitlab-ext.sigma-chemnitz.de/ensc/gst-bayer2rgb-neon
# build bayer2rgb lib git clone https://gitlab-ext.sigma-chemnitz.de/ensc/bayer2rgb cd bayer2rgb #apt install build-essential git autoconf gengetopt apt install build-essential git gengetopt autoconf automake autoconf-archive libtool help2man pkg-config autoreconf -i -f ./configure make make install cd .. # build gstreamer plugin git clone https://gitlab-ext.sigma-chemnitz.de/ensc/gst-bayer2rgb-neon cd gst-bayer2rgb-neon apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev ./autogen.sh ./configure make make install gst-inspect-1.0 bayer2rgbneon
Video Decoding
The IMX8MM contains the following hardware
- VeriSilicon Hantro G1 video decoder:
- 1080p60 MPEG2 decoder
- 1080p60 AVC/H.264 Baseline, Main, High decoder
- 1080p60 VP8 decoder
- VeriSilicon Hantro G2 video decoder:
HEVC/VP9 (1080p60)
- 1080p60 VP9 Profile 0, 2 (10 bit) decoder
- 1080p60 HEVC/H.265 decoder
The Linux driver at drivers/staging/media/hantro:
- currently supports:
- G1: MPEG2/VP8/H.264 decode
- G2: HEVC decode
- G2: VP9 decode (Linux 5.17)
- registers both media control devices (/dev/media*) and video4linux devices (/dev/video*):
root@focal-venice:~/# dmesg | grep -i hantro [ 1.505790] hantro-vpu 38300000.video-codec: registered nxp,imx8mm-vpu-dec as /dev/video0 [ 1.515788] hantro-vpu 38310000.video-codec: registered nxp,imx8mm-vpu-g2-dec as /dev/video1 root@focal-venice:~/# cat /sys/class/video4linux/video*/name nxp,imx8mm-vpu-dec nxp,imx8mm-vpu-g2-dec root@focal-venice:~/# cat /sys/bus/media/devices/media*/model hantro-vpu hantro-vpu
The Hantro G1/G2 decoders are stateless decoders and require the Request API found in modern Linux kernels and a new v4l2codecs plugin to handle 'stateless decoders'. While the v4l2codecs plugin was added in GStreamer 1.18 it has been receiving many updates required for it to work with the Hantro G1/G2 VPU decoder.
It is likely that Gstreamer 1.19 is needed to be able to access the v4l2codec plugin. The following is from a GStreamer 1.19 build using a 5.16 kernel with VPU support and VP9 support backported:
[gst-main] root@focal-venice:~/gstreamer/build# gst-inspect-1.0 v4l2codecs Plugin Details: Name v4l2codecs Description V4L2 CODEC Accelerators plugin Filename /root/gstreamer/build/subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so Version 1.19.3.1 License LGPL Source module gst-plugins-bad Binary package GStreamer Bad Plug-ins git Origin URL Unknown package origin v4l2slh264dec: V4L2 Stateless H.264 Video Decoder v4l2slmpeg2dec: V4L2 Stateless Mpeg2 Video Decoder v4l2slvp8alphadecodebin: VP8 Alpha Decoder v4l2slvp8dec: V4L2 Stateless VP8 Video Decoder v4l2slvp9dec: V4L2 Stateless VP9 Video Decoder 5 features: +-- 5 elements
Note that the Hantro G1/G2 VPU in the IMX8MM outputs raw video in NV12/YUY2/NV12_32L32 pixel formats. Note that IMX8MM MIPI DSI display supports only RGB formats therefore video conversion needs to take place in software in order to 'decode and display.
Examples:
- install Gstreamer:
apt update apt-get install gstreamer1.0-x gstreamer1.0-tools \ gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \ gstreamer1.0-libav gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 v4l-utils # using glvideosink will render 'within' GUI gst-launch-1.0 videotestsrc ! autovideosink
- VP8 hardware decode and display:
gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP8, payload=(int)96, ssrc=(uint)3363940374, timestamp-offset=(uint)3739685909, seqnum-offset=(uint)28161, a-framerate=(string)30' ! rtpvp8depay ! v4l2slvp8dec ! videoconvert ! fbdevsink
- need videoconvert because vpu src is NV12/YUY2/NV12_32L32 and display only supports BGRx (at this resolution and bitrate CPU is using about 85% idle)
- The caps are taken from the udpsink0.GsdPad:sink: caps from the source
- using kmssink I get 'drmModeSetPlane failed: Invalid argument (22)'
- VP9 hardware decode and display:
gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP9, payload=(int)96, ssrc=(uint)1956992802, timestamp-offset=(uint)828951387, seqnum-offset=(uint)28536, a-framerate=(string)30' ! rtpvp9depay ! vp9parse ! v4l2slvp9dec ! videoconvert ! kmssink
- need videoconvert because vpu src is NV12/YUY2/NV12_32L32 and display only supports BGRx (at this resolution and bitrate CPU is using about 93% idle)
- The caps are taken from the udpsink0.GsdPad:sink: caps from the source
- H264
gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)64001f, sprop-parameter-sets=(string)"Z2QAH6zZQMg9sBagwCC0oAAAAwAgAAAHkeMGMsA\=\,aOvssiw\=", payload=(int)96, ssrc=(uint)2753453329, timestamp-offset=(uint)3593065282, seqnum-offset=(uint)12297, a-framerate=(string)30' ! rtph264depay ! v4l2slh264dec ! videoconvert ! kmssink
- need videoconvert because vpu src is NV12/YUY2/NV12_32L32 and display only supports BGRx (at this resolution and bitrate CPU is using about 90% idle)
- The caps are taken from the udpsink0.GsdPad:sink: caps from the source
- using fbdevsink video looks like it has the wrong colorspace
For the examples above streams were generated on a x86 system using GStreamer software encode:
# vp8 software encode gst-launch-1.0 -v videotestsrc ! vp8enc ! rtpvp8pay ! udpsink host=192.168.1.22 port=9001 # vp9 software encode gst-launch-1.0 -v videotestsrc ! 'video/x-raw,width=800,height=480,format=(string)YV12' ! vp9enc ! rtpvp9pay ! udpsink host=192.168.1.22 port=9001 gst-launch-1.0 -v videotestsrc ! vp9enc ! rtpvp8pay ! udpsink host=192.168.1.22 port=9001 # h264 software encode gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 ! x264enc ! rtph264pay ! udpsink host=192.168.1.22 port=9001
- for VP9 vp9parse is required as the new stateless decoder handle frames, while VP9 have this notion of superframe. vp9parse will seperate the frames from the super frames.
Notes:
- The DSI display driver only accepts RGB pixel format so a pixel conversion must occur if you wish to display video from from the VPU which can only output NV12/YUY2/NV12_32L32. There is no hardware accelerated mem2mem driver at this time (and its not clear if there will be on the IMX8MM). There is a libneon and bayer2rgb neon project with a gstreamer plugin that can be used:
- While GStreamer 1.18 introduced the
v4l2codecsplugin required to use the VPU it has been getting a lot of additional features and bugfixes that are required. Therefore you need to use either a development build or wait for a packaged GStreamer 1.20 release. To build a development build see gstreamer
Video Encoding
The IMX8MM contains the following hardware
- VeriSilicon Hantro H1 video encoder: VP8/H.264 (1080p60)
The Linux driver at drivers/staging/media/hantro currently do not have support for the IMX8MM Hantro H1 VPU
Notes:
- Hantro H1 VPU supports VP8/H.264/MVC video encode but kernel support is still missing
- You can derive support from RK3288 support using Google ChromeOS method (a v4l2 plugin that simulate in userspace a stateful encoder):
Attachments (1)
- rpicam.jpg (12.4 KB ) - added by 2 years ago.
Download all attachments as: .zip
Note:
See TracWiki
for help on using the wiki.
