[[PageOutline]] = Venice Boot Firmware The 'Boot Firmware' for Venice is defined as the combination of the firmware stages through up to and including the bootloader. This can be broken down into the following stages: * Boot ROM (internal on i.MX8 SoC): fetch first level boot firmware from boot device (ie MMC or SPI FLASH) into L2 cache * SPL (Secondary Program Loader) - (U-Boot) * ATF (ARM Trusted Firmware) * Bootloader (U-Boot) Gateworks provides [http://dev.gateworks.com/venice/boot_firmware/ pre-built Boot Firmware] ready to flash onto boot devices as well as source for building and/or modifying the boot firmware yourself. [=#bootrom] == i.MX8 BOOT ROM The BOOT ROM is firmware baked into the SoC and is in charge of loading code from the 'boot device' into an L2 cache scratchpad, verifying signatures (if using trusted boot) and executing it. The BOOT ROM fetches code from an offset of 33KiB on the boot device which leaves the first 33KiB available for other firmware needs. [=#firmware-image] == Boot Firmware Image The firmware image contains all of the components of the 'Boot Firmware': - ARM Trusted Firmware (ATF) - SPL - U-Boot - U-Boot environment This firmware is typically put on the eMMC boot0 hardware partition to keep it separate from the eMMC user partition used for the OS. Venice Boot Firmware Image Map: * imx8mm: ||= start-end =||= len =||= item =||= notes =|| || 0x00000000 - 0x00008400 || 33KiB || unused || || 0x00008400 - 0x00060000 || 351KiB || SPL (spl.bin) || Loads and u-boot.itb and transfers control to the ATF || || 0x00060000 - 0x003F0000 || 3648KiB || U-Boot (u-boot.itb) || FIT image containing u-boot, ATF, and fdt || || 0x003F0000 - 0x00400000 || 64KB || U-Boot env || redundant 32KB env || * imx8mn/imx8mp: ||= start-end =||= len =||= item =||= notes =|| || 0x00000000 - 0x00058000 || 352KiB || SPL (spl.bin) || Loads and u-boot.itb and transfers control to the ATF || || 0x00058000 - 0x003F0000 || 3680KiB || U-Boot(u-boot.itb) || FIT image containing u-boot, ATF, and fdt || || 0x003F0000 - 0x00400000 || 64KB || U-Boot env || redundant 32KB env || Note that while the size of the SPL and offset/size of U-Boot can vary based on bootloader configuration the start offset of the SPL is dictated by the SOC-specific BOOTROM and varies based on the boot device and hardware partition: * imx8mm: - eMMC user partition: 33KiB - eMMC boot partition: 33KiB * imx8mn/imx8mp: - eMMC user partition: 32KiB - eMMC boot partition: 0KiB == Legacy Boot Firmware Image The original Gateworks IMX8MM Boot Firmware (now considered legacy) was installed to the eMMC user hardware partition. Legacy Venice IMX8MM Boot Firmware Image Map: ||= start-end =||= len =||= item =||= notes =|| || 0x00000000 - 0x00008400 || 33KiB || MBR Partition Table || only first 512B used || || 0x00008400 - 0x00060000 || 351KiB || SPL (spl.bin) || Loads and u-boot.itb and transfers control to the ATF || || 0x00060000 - 0x00FF0000 || 15MiB || U-Boot (u-boot.itb) || FIT image containing u-boot, ATF, and fdt || || 0x00FF0000 - 0x01000000 || 64KB || U-Boot env || redundant 32KB env || || 0x01000000 - || || Disk Partitions || used by the OS || == Booting from NVMe You can't access NVMe devices from u-boot because IMX8M PCI is not supported in U-Boot and NVMe uses the PCI bus. What is possible, is that you can load a kernel that has PCI/NVMe support from eMMC/USB/microSD and thus run your OS off NVMe. NVMe support has existed in our '''kernels''' for a long time. It works exactly like putting your root filesystem on any block storage device such that you tell the kernel what device is root via 'root= rootwait'. The IMX8 BOOT ROM does not support booting from PCI/NVMe so it will always boot its boot firmware from something it does support (NAND/SDIO/MMC/SD/SPI/USB/QSPI) and in this case that is eMMC. So the boot firmware will always exist on eMMC. If U-Boot doesn't support your the device or filesystem , then the need to put the kernel/bootscript/fdt on something it does support (ie a 'bootfs' or raw mmc using fixed offfsets/sizes) and just feed it the 'root=' parameter that is eventually wanted. This is not at all a new or foriegn concept. Note that in order to mount a rootfs in Linux not only do you need the valid 'root=' arguments (and supporting args like 'rootwait' for slower devs) you also would need to have 'static' support for the device and filesystem you wish to use otherwise you need to use a ramdisk that loads modules to support them mount the rootfs and pivot to it. NVMe and UMS was set to be static in the Venice kernel at this commit: [https://github.com/Gateworks/linux-venice/commit/4158a0890a8498fc6de18a5b4e3dbd22adfd93b0] If you have an NVMe in one of our boards you will see: {{{ root@jammy-venice:~# lspci 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01) 01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch 02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch 02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch 02:03.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch 02:04.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch 03:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. Device 5765 (rev 01) c0:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller root@jammy-venice:~# ls /sys/class/nvme nvme0 root@jammy-venice:~# ls /dev/nvme0 /dev/nvme0 root@jammy-venice:~# ls /dev/nvme0n1 /dev/nvme0n1 root@jammy-venice:~# dmesg | grep nvme [ 7.052622] nvme nvme0: pci function 0000:03:00.0 [ 7.052665] nvme 0000:03:00.0: enabling device (0000 -> 0002) [ 7.275312] nvme nvme0: allocated 64 MiB host memory buffer. [ 7.340566] nvme nvme0: 4/0/0 default/read/poll queues [ 7.353019] nvme nvme0: Ignoring bogus Namespace Identifiers [ 7.366645] nvme0n1: p1 }}} Notes - Kernel must have CONFIG_BLK_DEV_NVME enabled (and obviously PCI) (and static unless relying on a ramdisk to load modules) - The /dev/nvme0 is the controller (0 being the first one) - The /dev/nvme0n1 is the device and you would partition this and use this just like a /dev/mmcblk2 or a /dev/sda - If partitions are found they will be available as p1, p2 etc - The kernel can use this for root just like any block storage device via 'root=/dev/nvme0n1p1 rootwait' Recall that our default bootloaders use a U-Boot convention called 'generic distro boot' where the bootcmd runs distro_bootcmd which works by searching a list of bootable devices for bootable partitions that contain a boot script matching a list. If found it runs that bootscript and the bootscript uses the device and interface that it was located on to know what root device to tell the kernel to use. Because U-Boot and Linux differ in naming conventions between block storage device (U-Boot doesn't know that 'mmc 2:1' may refer to /dev/mmcblk2p1 on Linux) it uses the UUID of that particular filesystem and adds a 'root=PARTUUID=' to boot. We use uuid because that standard is the same across uboot and Linux but device naming and numbering is not a standard. So if you put our standard rootfs on any block storage device that U-Boot supports it just works without modification. One can do the following steps under Linux on a host or target. For example these steps work on our board: Prepare the nvme device by copying our compressed disk image to it {{{ DEV=/dev/nvme0n1 # /dev/sda for first usb mass storage device cd /tmp wget https://dev.gateworks.com/venice/images/jammy-venice.img.gz [ -b $DEV ] || { echo ERROR NVMe device $DEV does not exist; exit 1; } zcat jammy-venice.img.gz | dd of=$DEV rm jammy-venice.img.gz }}} Creating a 'bootfs' disk image: {{{ # set some vars that can easily be adjusted P1_MB=60 # 60M should be enough for kernel/fdt/bootscript P1_OFFSET_MB=1 # we typically start disk partitions at 1M # fetch our kernel tarball and extract the contents of ./boot to a boot directory wget https://dev.gateworks.com/venice/kernel/linux-venice-6.6.8.tar.xz mkdir boot tar -C boot/ --strip-components=2 -xvf linux-venice-6.6.8.tar.xz ./boot # create a 'bootfs' filesystem truncate -s ${P1_MB}M bootfs.ext mkfs.ext4 bootfs.ext # use e2cp/e2mkdir from e2tools package to create/copy files to ext without having to mount it apt update && apt install -y e2tools e2mkdir bootfs.ext:boot for i in $(ls boot); do e2cp boot/$i bootfs.ext:boot/; done # create a bootscript using out default one with one modification wget https://raw.githubusercontent.com/Gateworks/bsp-venice/master/boot.scr # change the root=PARTUUID=${uuid} argument to what we want (I use sed here to do it but they can just edit and replace however they wish) sed -i 's;root=PARTUUID=${uuid};root=/dev/nvme0n1p1;' boot.scr # create bootscript using mkimage from u-boot-tools package apt update && apt install -y u-boot-tools mkimage -A arm64 -T script -C none -d boot.scr boot.scr.uimage # copy the bootscript to filesystem e2cp boot.scr.uimage bootfs.ext:boot/boot.scr # create a disk image (partition table, partitions) truncate -s $(($P1_MB+$P1_OFFSET_MB))M bootfs.img # add a partition table apt update && apt install -y fdisk printf "$(($P1_OFFSET_MB*2*1024)),$(($P1_MB*2*1024)),L,*" | sfdisk -uS bootfs.img # copy filesystem into image dd if=bootfs.ext of=bootfs.img bs=1M seek=$P1_OFFSET_MB # compress image gzip bootfs.img # bootfs.img.gz is what you are going to flash to the eMMC via 'update_all' or manually via 'gzwrite' #^^^ Note there are lots of possible variations on the above... filesystem type doesn't have to be ext4... partition sizes formats can change etc... you just need to do something uboot supports }}} If you are doing the above on our board booted to emmc then you can't write to the emmc while its mounted (unpredictable results) so you would have to have done this from a ramdisk, or you could move that bootfs.img.gz out of /tmp to save it to the existing file system then reboot to uboot and load/flash it from U-Boot mmc: {{{ mv bootfs.img.gz / sync && reboot u-boot=> load mmc 2:1 $loadaddr /bootfs.img.gz && gzwrite mmc $dev $loadaddr $filesize }}} If you copy bootfs.img.gz to a tftp server for example you can install just like any other compressed disk image: {{{ u-boot=> setenv image venice/bootfs.img.gz; run update_all }}} Now you can reset/power-cycle If you inspect you can see your changes: {{{ u-boot=> part list mmc 2 Partition Map for MMC device 2 -- Partition Type: DOS Part Start Sector Num Sectors UUID Type 1 2048 122880 d1a37fad-01 83 Boot #^^^ emmc partition 1 is a bootable Linux partition of 122880 blocks (122880*512=60M) u-boot=> ls mmc 2:1 4096 . 4096 .. 16384 lost+found 4096 boot #^^^ only the boot dir exists u-boot=> ls mmc 2:1 boot 4096 . 4096 .. 16384 lost+found 4096 boot u-boot=> ls mmc 2:1 boot 4096 . 4096 .. 31275520 Image 37426 imx8mm-venice-gw71xx-0x.dtb 51885 imx8mm-venice-gw72xx-0x-gw16157.dtb 2157 imx8mm-venice-gw72xx-0x-gw16157.dtbo 51547 imx8mm-venice-gw72xx-0x-imx219.dtb 2293 imx8mm-venice-gw72xx-0x-imx219.dtbo 51648 imx8mm-venice-gw72xx-0x-rpidsi.dtb 1989 imx8mm-venice-gw72xx-0x-rpidsi.dtbo 51069 imx8mm-venice-gw72xx-0x-rs232-rts.dtb 1241 imx8mm-venice-gw72xx-0x-rs232-rts.dtbo 51156 imx8mm-venice-gw72xx-0x-rs422.dtb 1292 imx8mm-venice-gw72xx-0x-rs422.dtbo 51145 imx8mm-venice-gw72xx-0x-rs485.dtb 1281 imx8mm-venice-gw72xx-0x-rs485.dtbo 50843 imx8mm-venice-gw72xx-0x.dtb 52878 imx8mm-venice-gw73xx-0x-gw16157.dtb 2157 imx8mm-venice-gw73xx-0x-gw16157.dtbo 52540 imx8mm-venice-gw73xx-0x-imx219.dtb 2293 imx8mm-venice-gw73xx-0x-imx219.dtbo 52641 imx8mm-venice-gw73xx-0x-rpidsi.dtb 1989 imx8mm-venice-gw73xx-0x-rpidsi.dtbo 52042 imx8mm-venice-gw73xx-0x-rs232-rts.dtb 1241 imx8mm-venice-gw73xx-0x-rs232-rts.dtbo 52139 imx8mm-venice-gw73xx-0x-rs422.dtb 1292 imx8mm-venice-gw73xx-0x-rs422.dtbo 52128 imx8mm-venice-gw73xx-0x-rs485.dtb 1281 imx8mm-venice-gw73xx-0x-rs485.dtbo 51836 imx8mm-venice-gw73xx-0x.dtb 38309 imx8mm-venice-gw75xx-0x.dtb 44288 imx8mm-venice-gw7901.dtb 42563 imx8mm-venice-gw7902.dtb 39682 imx8mm-venice-gw7903.dtb 40046 imx8mm-venice-gw7904.dtb 38039 imx8mn-venice-gw7902.dtb 46598 imx8mp-venice-gw71xx-2x.dtb 49113 imx8mp-venice-gw72xx-2x.dtb 49684 imx8mp-venice-gw73xx-2x.dtb 68290 imx8mp-venice-gw74xx-imx219.dtb 2094 imx8mp-venice-gw74xx-imx219.dtbo 68430 imx8mp-venice-gw74xx-rpidsi.dtb 2025 imx8mp-venice-gw74xx-rpidsi.dtbo 67594 imx8mp-venice-gw74xx.dtb 47398 imx8mp-venice-gw75xx-2x.dtb 11300992 kernel.itb 2242 boot.scr #^^^ and it contains our kernel/dtbs/script }}} Note that NVMe support in U-Boot for imx8m mini is likely coming soon; NVMe has been supported in U-Boot for a while now, but PCIe for IMX8M has not been. IMX8MP PCI support landed upstream just a few weeks ago (and we've tested/enabled it (PCI/NVME) upstream for imx8mp-venice). imxmm support is hopefully coming soon. [=#u-boot] == U-Boot Bootloader Read more on the dedicated bootloader page here: [wiki:venice/bootloader Venice U-Boot Bootloader]