Changes between Version 10 and Version 11 of venice/multimedia


Ignore:
Timestamp:
12/23/2023 01:15:38 AM (5 months ago)
Author:
Tim Harvey
Comment:

updated video capture for newer kernels and the imx8mp gw74xx

Legend:

Unmodified
Added
Removed
Modified

  • venice/multimedia

    v10 v11  
    175175 - [[Image(rpicam.jpg,300px)]]
    176176
    177 The Linux drives required for this camera are:
    178  - CONFIG_VIDEO_IMX_MEDIA drivers/staging/media/imx/imx-media-common.c imx_media_common.ko
    179  - CONFIG_VIDEO_IMX7_CSI drivers/staging/media/imx/{imx7-media-csi,imx7-mipi-csis,imx8mq-mipi-csi2}.c imx7_media_csi.ko imx7-mipi-csis.ko
    180  - CONFIG_VIDEO_IMX219 drivers/media/i2c/imx219.c imx219.ko
    181 
    182 Device-Tree changes:
    183  - to add a device to your system you must properly configure your device-tree to describe how the device is connected to the host processor and other peripherals. Often this is most easily done via a device-tree overlay. The Gateworks venice kernels have dt overlays that support the IMX219 !RaspberryPi v2 camera module on the GW720x and GW730x boards. The dt overlay can be applied to the dt prior to booting the kernel by U-Boot. Examples:
    184   * GW730x imx219 camera:
     177=== Kernel Drivers
     178You must have a driver for both the IMX8M MIPI CSI side as well as the camera sensor:
     179 * IMX8MM:
     180  - 6.6:
     181   - CONFIG_VIDEO_IMX7_CSI drivers/media/platform/nxp/imx7-media-csi.c
     182   - CONFIG_VIDEO_IMX_MIPI_CSIS drivers/media/platform/nxp/imx-mipi-csis.c
     183  - 6.1:
     184   - CONFIG_VIDEO_IMX_CSI drivers/staging/media/imx/imx7-media-csi.c
     185   - CONFIG_VIDEO_IMX_MIPI_CSIS drivers/media/platform/nxp/imx-mipi-csis.c
     186  - 5.15:
     187   - CONFIG_VIDEO_IMX_CSI drivers/staging/media/imx/imx7-media-csi.c
     188   - CONFIG_VIDEO_IMX7_CSI drivers/staging/media/imx/imx7-mipi-csis.c
     189 - IMX8MP:
     190  - 6.6:
     191   - CONFIG_VIDEO_IMX8_ISI drivers/media/platform/nxp/imx8-isi/imx8-isi-core.c
     192   - CONFIG_VIDEO_IMX_MIPI_CSIS drivers/media/platform/nxp/imx-mipi-csis.c
     193 - IMX219 Image Sensor:
     194   - CONFIG_VIDEO_IMX219 drivers/media/i2c/imx219.c
     195
     196=== Device Tree
     197To add a device to your system you must properly configure your device-tree to describe how the device is connected to the host processor and other peripherals. Often this is most easily done via a device-tree overlay.
     198
     199The Gateworks venice kernels have dt overlays that support the IMX219 !RaspberryPi v2 camera module on the GW72xx, GW73xx, and GW74xx boards which can be used as examples for other devices.
     200
     201The dt overlay can be applied to the dt prior to booting the kernel by the Gateworks U-Boot using the 'fdt_overlay' env variable:
     202  * GW73xx imx219 camera:
    185203{{{#!bash
    186204setenv fdt_overlays imx8mm-venice-gw73xx-0x-imx219.dtbo
    187205saveenv
    188206}}}
    189   * GW720x imx219 camera:
     207  * GW72xx imx219 camera:
    190208{{{#!bash
    191209setenv fdt_overlays imx8mm-venice-gw72xx-0x-imx219.dtbo
    192210saveenv
    193211}}}
    194 
    195 
    196 Examples:
    197  * configure Linux video pipeline for 640x480 RAW8 from IMX219 '''(you must do this in Linux prior to capturing from the video device)
    198 {{{#!bash
    199 # find media device whos model is 'imx-media'
     212 * GW74xx imx219 camera:
     213{{{#!bash
     214setenv fdt_overlays imx8mm-venice-gw74xx-imx219.dtbo
     215saveenv
     216}}}
     217
     218=== Media Controller Pipeline
     219To use the IMX8M capture devices you must have a media controller device which will appear in /sys/bus/media/devices/ as long as you have the correct drivers available and enabled and everything is configured properly in the device-tree to link the IMX8M MIPI CSI capture device to the image sensor.
     220
     221This configuration is done in userspace via the v4l2-utils {{{media-ctl}}} application and the Linux media controller API. The pipeline configuration depends on the specific SoC, the image sensor, and how it is connected to the SoC.
     222
     223Ubuntu Examples to configure the pipeline for 640x480 8-bit ray bayer (RAW8) from the IMX219 image sensor:
     224 - imx8mp-venice-gw74xx-imx219:
     225{{{#!bash
     226# find imx8mp media device (FSL Capture Media Device)
     227MDEV=$(for i in `ls -d /sys/bus/media/devices/media*`; do if [ "$(cat $i/model)" = "FSL Capture Media Device" ]; then cat $i/dev | cut -d: -f2; fi; done)
     228# find imx8mm video capture device (GW74xx uses CSI0 on the MIPI connector which is mxc_isi.0.capture)
     229VDEV=$(for i in `ls -d /sys/class/video4linux/video*`; do if [ "$(cat $i/name)" = "mxc_isi.0.capture" ]; then cut -d: -f2 $i/dev; fi; done)
     230# install media-ctl tool
     231apt update && apt install -y v4l-utils
     232# configure media controller links
     233media-ctl --device $MDEV --reset
     234media-ctl --device $MDEV --print-topology
     235media-ctl --device $MDEV --links "'imx219 3-0010':0->'csis-32e40000.csi':0[1]"
     236# configure media entities for 8-bit raw bayer 640x480
     237media-ctl --device $MDEV --verbose --set-v4l2 "'imx219 3-0010':0 [fmt:SRGGB8/640x480 field:none]"
     238media-ctl --device $MDEV --verbose --set-v4l2 "'crossbar':0 [fmt:SRGGB8/640x480 field:none]"
     239media-ctl --device $MDEV --verbose --set-v4l2 "'mxc_isi.0':0 [fmt:SRGGB8/640x480 field:none]"
     240}}}
     241 - imx8mm-venice-gw73xx-0x-imx219 and ixm8mm-venice-gw73xx-0x-imx219:
     242  * 6.1+ kernel:
     243{{{#!bash
     244# find imx8mm media device (imx-media)
    200245MDEV=$(for i in `ls -d /sys/bus/media/devices/media*`; do if [ "$(cat $i/model)" = "imx-media" ]; then cat $i/dev | cut -d: -f2; fi; done)
    201 # find video device whos name is 'csi capture'
     246# find imx8mm video capture device (csi capture)
    202247VDEV=$(for i in `ls -d /sys/class/video4linux/video*`; do if [ "$(cat $i/name)" = "csi capture" ]; then cut -d: -f2 $i/dev; fi; done)
    203248# configure media controller links
    204 media-ctl --device /dev/media$MDEV --reset
    205 media-ctl --device /dev/media$MDEV --links "'imx219 2-0010':0->'imx7-mipi-csis.0':0[1]"
    206 media-ctl --device /dev/media$MDEV -v -V "'imx219 2-0010':0 [fmt:SRGGB8/640x480 field:none]"
    207 media-ctl --device /dev/media$MDEV -v -V "'csi':0 [fmt:SRGGB8/640x480 field:none]"
    208 }}}
    209  * configure video format and capture a raw frame with v4l2-ctl '''(must configure Linux video pipeline first)'''
    210 {{{#!bash
    211 v4l2-ctl --device /dev/video$VDEV --set-fmt-video=pixelformat=RGGB
    212 v4l2-ctl --device /dev/video$VDEV --stream-mmap --stream-to=frame.raw --stream-count=1
     249media-ctl --device $MDEV --reset
     250media-ctl --device $MDEV --print-topology
     251media-ctl --device $MDEV --links "'imx219 2-0010':0 -> 'csis-32e30000.mipi-csi':0 [1]"
     252# configure for 640x480 raw8
     253media-ctl --device $MDEV --verbose --set-v4l2 "'imx219 2-0010':0 [fmt:SRGGB8/640x480 field:none]"
     254media-ctl --device $MDEV --verbose --set-v4l2 "'csis-32e30000.mipi-csi':0 [fmt:SRGGB8/640x480 field:none]"
     255media-ctl --device $MDEV --verbose --set-v4l2 "'csi':0 [fmt:SRGGB8/640x480 field:none]"
     256}}}
     257  * 5.15 kernel:
     258{{{#!bash
     259# find imx8mm media device (imx-media)
     260MDEV=$(for i in `ls -d /sys/bus/media/devices/media*`; do if [ "$(cat $i/model)" = "imx-media" ]; then cat $i/dev | cut -d: -f2; fi; done)
     261# find imx8mm video capture device (csi capture)
     262VDEV=$(for i in `ls -d /sys/class/video4linux/video*`; do if [ "$(cat $i/name)" = "csi capture" ]; then cut -d: -f2 $i/dev; fi; done)
     263# configure media controller links
     264media-ctl --device $MDEV --reset
     265media-ctl --device $MDEV --print-topology
     266media-ctl --device $MDEV --links "'imx219 2-0010':0->'imx7-mipi-csis.0':0[1]"
     267media-ctl --device $MDEV --verbose --set-v4l2 "'imx219 2-0010':0 [fmt:SRGGB8/640x480 field:none]"
     268media-ctl --device $MDEV --verbose --set-v4l2 "'csi':0 [fmt:SRGGB8/640x480 field:none]"
     269}}}
     270
     271=== Capture with v4l
     272Once you have configured the media pipeline (above) you can configure the appropriate video device for a supported pixel format and resolution and use the Video For Linux (v4l) API to capture:
     273 * configure for 640x480 RGB
     274{{{#!bash
     275v4l2-ctl --device $VDEV --set-fmt-video=width=640,height=480,pixelformat=RGGB --verbose
     276v4l2-ctl --device $VDEV --stream-mmap --stream-to=frame.raw --stream-count=1
    213277# install imagemagic for convert tool
    214 apt-get install imagemagic
     278apt-get install imagemagick
    215279convert -size 640x480 -depth 8 gray:frame.raw frame.png
    216280}}}
    217  * capture 640x480 RAW8 at 10fps from IMX219 and display to framebuffer using gstreamer '''(must configure Linux video pipeline first)'''
     281
     282=== Capture with Gstreamer
     283Once you have proved your video device is configured properly and can capture a frame with the v4l API you can try streaming with Gstreamer:
     284 * capture 640x480 RAW8 at 10fps from IMX219 and display to framebuffer using gstreamer
    218285{{{#!bash
    219286# install gstreamer
    … …  
    229296  - bayer2rgbneon uses appx 10% of CPU when converting 640x480@30fps
    230297
    231 
    232298Notes:
    233  - The DSI display driver only accepts RGB pixel format so a pixel conversion must occur if you want to display video from anything that does not output RGB (such as a RAW8 camera or YUV from the VPU decoder). There is no hardware accelerated mem2mem driver at this time (and its not clear if there will be on the IMX8MM). There is a bayer2rgb lib with NEON support and a gstreamer plugin that can be used for RAW8 to RGB:
     299 - The IMX8MM DSI display driver only accepts RGB pixel format so a pixel conversion must occur if you want to display video from anything that does not output RGB (such as a RAW8 camera or YUV from the VPU decoder). There is no hardware accelerated mem2mem driver at this time (and its not clear if there will be on the IMX8MM). There is a bayer2rgb lib with NEON support and a gstreamer plugin that can be used for RAW8 to RGB:
    234300  - https://gitlab-ext.sigma-chemnitz.de/ensc/bayer2rgb
    235301  - https://gitlab-ext.sigma-chemnitz.de/ensc/gst-bayer2rgb-neon