wiki:venice/boot

Version 14 (modified by Ryan Erbstoesser, 8 months ago) ( diff )

add more info for booting from nvme on venice

  1. Venice Boot Firmware
    1. i.MX8 BOOT ROM
    2. Boot Firmware Image
    3. Legacy Boot Firmware Image
    4. Booting from NVMe
    5. U-Boot Bootloader

Venice Boot Firmware

The 'Boot Firmware' for Venice is defined as the combination of the firmware stages through up to and including the bootloader. This can be broken down into the following stages:

  • Boot ROM (internal on i.MX8 SoC): fetch first level boot firmware from boot device (ie MMC or SPI FLASH) into L2 cache
  • SPL (Secondary Program Loader) - (U-Boot)
  • ATF (ARM Trusted Firmware)
  • Bootloader (U-Boot)

Gateworks provides ​pre-built Boot Firmware ready to flash onto boot devices as well as source for building and/or modifying the boot firmware yourself.

i.MX8 BOOT ROM

The BOOT ROM is firmware baked into the SoC and is in charge of loading code from the 'boot device' into an L2 cache scratchpad, verifying signatures (if using trusted boot) and executing it.

The BOOT ROM fetches code from an offset of 33KiB on the boot device which leaves the first 33KiB available for other firmware needs.

Boot Firmware Image

The firmware image contains all of the components of the 'Boot Firmware':

  • ARM Trusted Firmware (ATF)
  • SPL
  • U-Boot
  • U-Boot environment

This firmware is typically put on the eMMC boot0 hardware partition to keep it separate from the eMMC user partition used for the OS.

Venice Boot Firmware Image Map:

  • imx8mm:
start-end len item notes
0x00000000 - 0x00008400 33KiB unused
0x00008400 - 0x00060000 351KiB SPL (spl.bin) Loads and u-boot.itb and transfers control to the ATF
0x00060000 - 0x003F0000 3648KiB U-Boot (u-boot.itb) FIT image containing u-boot, ATF, and fdt
0x003F0000 - 0x00400000 64KB U-Boot env redundant 32KB env
  • imx8mn/imx8mp:
start-end len item notes
0x00000000 - 0x00058000 352KiB SPL (spl.bin) Loads and u-boot.itb and transfers control to the ATF
0x00058000 - 0x003F0000 3680KiB U-Boot(u-boot.itb) FIT image containing u-boot, ATF, and fdt
0x003F0000 - 0x00400000 64KB U-Boot env redundant 32KB env

Note that while the size of the SPL and offset/size of U-Boot can vary based on bootloader configuration the start offset of the SPL is dictated by the SOC-specific BOOTROM and varies based on the boot device and hardware partition:

  • imx8mm:
    • eMMC user partition: 33KiB
    • eMMC boot partition: 33KiB
  • imx8mn/imx8mp:
    • eMMC user partition: 32KiB
    • eMMC boot partition: 0KiB

Legacy Boot Firmware Image

The original Gateworks IMX8MM Boot Firmware (now considered legacy) was installed to the eMMC user hardware partition.

Legacy Venice IMX8MM Boot Firmware Image Map:

start-end len item notes
0x00000000 - 0x00008400 33KiB MBR Partition Table only first 512B used
0x00008400 - 0x00060000 351KiB SPL (spl.bin) Loads and u-boot.itb and transfers control to the ATF
0x00060000 - 0x00FF0000 15MiB U-Boot (u-boot.itb) FIT image containing u-boot, ATF, and fdt
0x00FF0000 - 0x01000000 64KB U-Boot env redundant 32KB env
0x01000000 - Disk Partitions used by the OS

Booting from NVMe

You can't access NVMe devices from u-boot because IMX8M PCI is not supported in U-Boot and NVMe uses the PCI bus.

What is possible, is that you can load a kernel that has PCI/NVMe support from eMMC/USB/microSD and thus run your OS off NVMe.

NVMe support has existed in our kernels for all of our product families for a long time now (I don't think we enable it on Ventana). It works exactly like putting your root filesystem on any block storage device such that you tell the kernel what device is root via 'root=<device> rootwait'.

The IMX8 BOOT ROM does not support booting from PCI/NVMe so it will always boot its boot firmware from something it does support (NAND/SDIO/MMC/SD/SPI/USB/QSPI) and in this case that is eMMC. So the boot firmware will always exist on eMMC.

If U-Boot doesn't support your the device or filesystem , then the need to put the kernel/bootscript/fdt on something it does support (ie a 'bootfs' or raw mmc using fixed offfsets/sizes) and just feed it the 'root=' parameter that is eventually wanted. This is not at all a new or foriegn concept.

Note that in order to mount a rootfs in Linux not only do you need the valid 'root=' arguments (and supporting args like 'rootwait' for slower devs) you also would need to have 'static' support for the device and filesystem you wish to use otherwise you need to use a ramdisk that loads modules to support them mount the rootfs and pivot to it .

If you have an NVMe in one of our boards you will see: 

root@jammy-venice:~# lspci
00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
02:03.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
02:04.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
03:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. Device 5765 (rev 01)
c0:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller
root@jammy-venice:~# ls /sys/class/nvme
nvme0
root@jammy-venice:~# ls /dev/nvme0
/dev/nvme0
root@jammy-venice:~# ls /dev/nvme0n1
/dev/nvme0n1
root@jammy-venice:~# dmesg | grep nvme
[    7.052622] nvme nvme0: pci function 0000:03:00.0
[    7.052665] nvme 0000:03:00.0: enabling device (0000 -> 0002)
[    7.275312] nvme nvme0: allocated 64 MiB host memory buffer.
[    7.340566] nvme nvme0: 4/0/0 default/read/poll queues
[    7.353019] nvme nvme0: Ignoring bogus Namespace Identifiers
[    7.366645]  nvme0n1: p1

Notes

  • Kernel must have CONFIG_BLK_DEV_NVME enabled (and obviously PCI) (and static unless relying on a ramdisk to load modules)
  • The /dev/nvme0 is the controller (0 being the first one)
  • The /dev/nvme0n1 is the device and you would partition this and use this just like a /dev/mmcblk2 or a /dev/sda
  • If partitions are found they will be available as p1, p2 etc
  • The kernel can use this for root just like any block storage device via 'root=/dev/nvme0n1p1 rootwait'

Recall that our default bootloaders use a U-Boot convention called 'generic distro boot' where the bootcmd runs distro_bootcmd which works by searching a list of bootable devices for bootable partitions that contain a boot script matching a list. If found it runs that bootscript and the bootscript uses the device and interface that it was located on to know what root device to tell the kernel to use. Because U-Boot and Linux differ in naming conventions between block storage device (U-Boot doesn't know that 'mmc 2:1' may refer to /dev/mmcblk2p1 on Linux) it uses the UUID of that particular filesystem and adds a 'root=PARTUUID=<uuid>' to boot. We use uuid because that standard is the same across uboot and Linux but device naming and numbering is not a standard. So if you put our standard rootfs on any block storage device that U-Boot supports it just works without modification.

One can do the following steps under Linux on a host or target. For example these steps work on our board:

Prepare the nvme device by copying our compressed disk image to it 

DEV=/dev/nvme0n1 # /dev/sda for first usb mass storage device
cd /tmp
wget https://dev.gateworks.com/venice/images/jammy-venice.img.gz
[ -b $DEV ] || { echo ERROR NVMe device $DEV does not exist; exit 1; }
zcat jammy-venice.img.gz | dd of=$DEV
rm jammy-venice.img.gz

Creating a 'bootfs' disk image: 

# set some vars that can easily be adjusted
P1_MB=60 # 60M should be enough for kernel/fdt/bootscript
P1_OFFSET_MB=1 # we typically start disk partitions at 1M
# fetch our kernel tarball and extract the contents of ./boot to a boot directory
wget https://dev.gateworks.com/venice/kernel/linux-venice-6.6.8.tar.xz
mkdir boot
tar -C boot/ --strip-components=2 -xvf linux-venice-6.6.8.tar.xz ./boot
# create a 'bootfs' filesystem
truncate -s ${P1_MB}M bootfs.ext
mkfs.ext4 bootfs.ext
# use e2cp/e2mkdir from e2tools package to create/copy files to ext without having to mount it
apt update && apt install -y e2tools
e2mkdir bootfs.ext:boot
for i in $(ls boot); do e2cp boot/$i bootfs.ext:boot/; done
# create a bootscript using out default one with one modification
wget https://raw.githubusercontent.com/Gateworks/bsp-venice/master/boot.scr
# change the root=PARTUUID=${uuid} argument to what we want (I use sed here to do it but they can just edit and replace however they wish)
sed -i 's;root=PARTUUID=${uuid};root=/dev/nvme0n1p1;' boot.scr
# create bootscript using mkimage from u-boot-tools package
apt update && apt install -y u-boot-tools
mkimage -A arm64 -T script -C none -d boot.scr boot.scr.uimage
# copy the bootscript to filesystem
e2cp boot.scr.uimage bootfs.ext:boot/boot.scr
# create a disk image (partition table, partitions)
truncate -s $(($P1_MB+$P1_OFFSET_MB))M bootfs.img
# add a partition table
apt update && apt install -y fdisk
printf "$(($P1_OFFSET_MB*2*1024)),$(($P1_MB*2*1024)),L,*" | sfdisk -uS bootfs.img
# copy filesystem into image
dd if=bootfs.ext of=bootfs.img bs=1M seek=$P1_OFFSET_MB
# compress image
gzip bootfs.img
# bootfs.img.gz is what you are going to flash to the eMMC via 'update_all' or manually via 'gzwrite'
#^^^ Note there are lots of possible variations on the above... filesystem type doesn't have to be ext4... partition sizes formats can change etc... you just need to do something uboot supports

If you are doing the above on our board booted to emmc then you can't write to the emmc while its mounted (unpredictable results) so you would have to have done this from a ramdisk, or you could move that bootfs.img.gz out of /tmp to save it to the existing file system then reboot to uboot and load/flash it from U-Boot mmc: 

mv bootfs.img.gz /
sync && reboot
u-boot=> load mmc 2:1 $loadaddr /bootfs.img.gz && gzwrite mmc $dev $loadaddr $filesize

If you copy bootfs.img.gz to a tftp server for example you can install just like any other compressed disk image: 

u-boot=> setenv image venice/bootfs.img.gz; run update_all

Now you can reset/power-cycle

If you inspect you can see your changes: 

u-boot=> part list mmc 2

Partition Map for MMC device 2  --   Partition Type: DOS

Part    Start Sector    Num Sectors     UUID            Type
  1     2048            122880          d1a37fad-01     83 Boot
#^^^ emmc partition 1 is a bootable Linux partition of 122880 blocks (122880*512=60M)

u-boot=> ls mmc 2:1
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
<DIR>       4096 boot
#^^^ only the boot dir exists

u-boot=> ls mmc 2:1 boot
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
<DIR>       4096 boot
u-boot=> ls mmc 2:1 boot
<DIR>       4096 .
<DIR>       4096 ..
        31275520 Image
           37426 imx8mm-venice-gw71xx-0x.dtb
           51885 imx8mm-venice-gw72xx-0x-gw16157.dtb
            2157 imx8mm-venice-gw72xx-0x-gw16157.dtbo
           51547 imx8mm-venice-gw72xx-0x-imx219.dtb
            2293 imx8mm-venice-gw72xx-0x-imx219.dtbo
           51648 imx8mm-venice-gw72xx-0x-rpidsi.dtb
            1989 imx8mm-venice-gw72xx-0x-rpidsi.dtbo
           51069 imx8mm-venice-gw72xx-0x-rs232-rts.dtb
            1241 imx8mm-venice-gw72xx-0x-rs232-rts.dtbo
           51156 imx8mm-venice-gw72xx-0x-rs422.dtb
            1292 imx8mm-venice-gw72xx-0x-rs422.dtbo
           51145 imx8mm-venice-gw72xx-0x-rs485.dtb
            1281 imx8mm-venice-gw72xx-0x-rs485.dtbo
           50843 imx8mm-venice-gw72xx-0x.dtb
           52878 imx8mm-venice-gw73xx-0x-gw16157.dtb
            2157 imx8mm-venice-gw73xx-0x-gw16157.dtbo
           52540 imx8mm-venice-gw73xx-0x-imx219.dtb
            2293 imx8mm-venice-gw73xx-0x-imx219.dtbo
           52641 imx8mm-venice-gw73xx-0x-rpidsi.dtb
            1989 imx8mm-venice-gw73xx-0x-rpidsi.dtbo
           52042 imx8mm-venice-gw73xx-0x-rs232-rts.dtb
            1241 imx8mm-venice-gw73xx-0x-rs232-rts.dtbo
           52139 imx8mm-venice-gw73xx-0x-rs422.dtb
            1292 imx8mm-venice-gw73xx-0x-rs422.dtbo
           52128 imx8mm-venice-gw73xx-0x-rs485.dtb
            1281 imx8mm-venice-gw73xx-0x-rs485.dtbo
           51836 imx8mm-venice-gw73xx-0x.dtb
           38309 imx8mm-venice-gw75xx-0x.dtb
           44288 imx8mm-venice-gw7901.dtb
           42563 imx8mm-venice-gw7902.dtb
           39682 imx8mm-venice-gw7903.dtb
           40046 imx8mm-venice-gw7904.dtb
           38039 imx8mn-venice-gw7902.dtb
           46598 imx8mp-venice-gw71xx-2x.dtb
           49113 imx8mp-venice-gw72xx-2x.dtb
           49684 imx8mp-venice-gw73xx-2x.dtb
           68290 imx8mp-venice-gw74xx-imx219.dtb
            2094 imx8mp-venice-gw74xx-imx219.dtbo
           68430 imx8mp-venice-gw74xx-rpidsi.dtb
            2025 imx8mp-venice-gw74xx-rpidsi.dtbo
           67594 imx8mp-venice-gw74xx.dtb
           47398 imx8mp-venice-gw75xx-2x.dtb
        11300992 kernel.itb
            2242 boot.scr
#^^^ and it contains our kernel/dtbs/script

Note that NVMe support in U-Boot for imx8m mini is likely coming soon; NVMe has been supported in U-Boot for a while now, but PCIe for IMX8M has not been. IMX8MP PCI support landed upstream just a few weeks ago (and we've tested/enabled it (PCI/NVME) upstream for imx8mp-venice). imxmm support is hopefully coming soon.

U-Boot Bootloader

Read more on the dedicated bootloader page here: Venice U-Boot Bootloader

Note: See TracWiki for help on using the wiki.