Changes between Version 13 and Version 14 of venice/boot

Timestamp:

04/04/2024 09:34:36 PM (5 weeks ago)

Author:

Ryan Erbstoesser

Comment:

add more info for booting from nvme on venice

venice/boot

@@ -58 +58 @@
 || 0x01000000 -            ||        || Disk Partitions || used by the OS ||
 
-= Booting from NVMe
+== Booting from NVMe
 
 You can't access NVMe devices from u-boot because IMX8M PCI is not supported in U-Boot and NVMe uses the PCI bus. 
 
 What is possible, is that you can load a kernel that has PCI/NVMe support from eMMC/USB/microSD and thus run your OS off NVMe.
+
+NVMe support has existed in our '''kernels''' for all of our product families for a long time now (I don't think we enable it on Ventana). It works exactly like putting your root filesystem on any block storage device such that you tell the kernel what device is root via 'root=<device> rootwait'.
+
+The IMX8 BOOT ROM does not support booting from PCI/NVMe so it will always boot its boot firmware from something it does support (NAND/SDIO/MMC/SD/SPI/USB/QSPI) and in this case that is eMMC. So the boot firmware will always exist on eMMC.
+
+If U-Boot doesn't support your the device or filesystem , then the need to put the kernel/bootscript/fdt on something it does support (ie a 'bootfs' or raw mmc using fixed offfsets/sizes) and just feed it the 'root=' parameter that is eventually wanted. This is not at all a new or foriegn concept.
+
+Note that in order to mount a rootfs in Linux not only do you need the valid 'root=' arguments (and supporting args like 'rootwait' for slower devs) you also would need to have 'static' support for the device and filesystem you wish to use otherwise you need to use a ramdisk that loads modules to support them mount the rootfs and pivot to it . 
+
+If you have an NVMe in one of our boards you will see:
+{{{
+root@jammy-venice:~# lspci
+00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
+01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
+02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
+02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
+02:03.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
+02:04.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
+03:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. Device 5765 (rev 01)
+c0:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller
+root@jammy-venice:~# ls /sys/class/nvme
+nvme0
+root@jammy-venice:~# ls /dev/nvme0
+/dev/nvme0
+root@jammy-venice:~# ls /dev/nvme0n1
+/dev/nvme0n1
+root@jammy-venice:~# dmesg | grep nvme
+[    7.052622] nvme nvme0: pci function 0000:03:00.0
+[    7.052665] nvme 0000:03:00.0: enabling device (0000 -> 0002)
+[    7.275312] nvme nvme0: allocated 64 MiB host memory buffer.
+[    7.340566] nvme nvme0: 4/0/0 default/read/poll queues
+[    7.353019] nvme nvme0: Ignoring bogus Namespace Identifiers
+[    7.366645]  nvme0n1: p1
+}}}
+
+Notes
+- Kernel must have CONFIG_BLK_DEV_NVME enabled (and obviously PCI) (and static unless relying on a ramdisk to load modules)
+- The /dev/nvme0 is the controller (0 being the first one)
+- The /dev/nvme0n1 is the device and you would partition this and use this just like a /dev/mmcblk2 or a /dev/sda
+- If partitions are found they will be available as p1, p2 etc
+- The kernel can use this for root just like any block storage device via 'root=/dev/nvme0n1p1 rootwait'
+
+Recall that our default bootloaders use a U-Boot convention called 'generic distro boot' where the bootcmd runs distro_bootcmd which works by searching a list of bootable devices for bootable partitions that contain a boot script matching a list. If found it runs that bootscript and the bootscript uses the device and interface that it was located on to know what root device to tell the kernel to use. Because U-Boot and Linux differ in naming conventions between block storage device (U-Boot doesn't know that 'mmc 2:1' may refer to /dev/mmcblk2p1 on Linux) it uses the UUID of that particular filesystem and adds a 'root=PARTUUID=<uuid>' to boot. We use uuid because that standard is the same across uboot and Linux but device naming and numbering is not a standard. So if you put our standard rootfs on any block storage device that U-Boot supports it just works without modification.
+
+One can do the following steps under Linux on a host or target. For example these steps work on our board:
+
+Prepare the nvme device by copying our compressed disk image to it
+{{{
+DEV=/dev/nvme0n1 # /dev/sda for first usb mass storage device
+cd /tmp
+wget https://dev.gateworks.com/venice/images/jammy-venice.img.gz
+[ -b $DEV ] || { echo ERROR NVMe device $DEV does not exist; exit 1; }
+zcat jammy-venice.img.gz | dd of=$DEV
+rm jammy-venice.img.gz
+}}}
+
+Creating a 'bootfs' disk image:
+{{{
+# set some vars that can easily be adjusted
+P1_MB=60 # 60M should be enough for kernel/fdt/bootscript
+P1_OFFSET_MB=1 # we typically start disk partitions at 1M
+# fetch our kernel tarball and extract the contents of ./boot to a boot directory
+wget https://dev.gateworks.com/venice/kernel/linux-venice-6.6.8.tar.xz
+mkdir boot
+tar -C boot/ --strip-components=2 -xvf linux-venice-6.6.8.tar.xz ./boot
+# create a 'bootfs' filesystem
+truncate -s ${P1_MB}M bootfs.ext
+mkfs.ext4 bootfs.ext
+# use e2cp/e2mkdir from e2tools package to create/copy files to ext without having to mount it
+apt update && apt install -y e2tools
+e2mkdir bootfs.ext:boot
+for i in $(ls boot); do e2cp boot/$i bootfs.ext:boot/; done
+# create a bootscript using out default one with one modification
+wget https://raw.githubusercontent.com/Gateworks/bsp-venice/master/boot.scr
+# change the root=PARTUUID=${uuid} argument to what we want (I use sed here to do it but they can just edit and replace however they wish)
+sed -i 's;root=PARTUUID=${uuid};root=/dev/nvme0n1p1;' boot.scr
+# create bootscript using mkimage from u-boot-tools package
+apt update && apt install -y u-boot-tools
+mkimage -A arm64 -T script -C none -d boot.scr boot.scr.uimage
+# copy the bootscript to filesystem
+e2cp boot.scr.uimage bootfs.ext:boot/boot.scr
+# create a disk image (partition table, partitions)
+truncate -s $(($P1_MB+$P1_OFFSET_MB))M bootfs.img
+# add a partition table
+apt update && apt install -y fdisk
+printf "$(($P1_OFFSET_MB*2*1024)),$(($P1_MB*2*1024)),L,*" | sfdisk -uS bootfs.img
+# copy filesystem into image
+dd if=bootfs.ext of=bootfs.img bs=1M seek=$P1_OFFSET_MB
+# compress image
+gzip bootfs.img
+# bootfs.img.gz is what you are going to flash to the eMMC via 'update_all' or manually via 'gzwrite'
+#^^^ Note there are lots of possible variations on the above... filesystem type doesn't have to be ext4... partition sizes formats can change etc... you just need to do something uboot supports
+}}}
+
+If you are doing the above on our board booted to emmc then you can't write to the emmc while its mounted (unpredictable results) so you would have to have done this from a ramdisk, or you could move that bootfs.img.gz out of /tmp to save it to the existing file system then reboot to uboot and load/flash it from U-Boot mmc:
+{{{
+mv bootfs.img.gz /
+sync && reboot
+u-boot=> load mmc 2:1 $loadaddr /bootfs.img.gz && gzwrite mmc $dev $loadaddr $filesize
+}}}
+
+If you copy bootfs.img.gz to a tftp server for example you can install just like any other compressed disk image:
+{{{
+u-boot=> setenv image venice/bootfs.img.gz; run update_all
+}}}
+
+Now you can reset/power-cycle
+
+If you inspect you can see your changes:
+{{{
+u-boot=> part list mmc 2
+
+Partition Map for MMC device 2  --   Partition Type: DOS
+
+Part    Start Sector    Num Sectors     UUID            Type
+  1     2048            122880          d1a37fad-01     83 Boot
+#^^^ emmc partition 1 is a bootable Linux partition of 122880 blocks (122880*512=60M)
+
+u-boot=> ls mmc 2:1
+<DIR>       4096 .
+<DIR>       4096 ..
+<DIR>      16384 lost+found
+<DIR>       4096 boot
+#^^^ only the boot dir exists
+
+u-boot=> ls mmc 2:1 boot
+<DIR>       4096 .
+<DIR>       4096 ..
+<DIR>      16384 lost+found
+<DIR>       4096 boot
+u-boot=> ls mmc 2:1 boot
+<DIR>       4096 .
+<DIR>       4096 ..
+        31275520 Image
+           37426 imx8mm-venice-gw71xx-0x.dtb
+           51885 imx8mm-venice-gw72xx-0x-gw16157.dtb
+            2157 imx8mm-venice-gw72xx-0x-gw16157.dtbo
+           51547 imx8mm-venice-gw72xx-0x-imx219.dtb
+            2293 imx8mm-venice-gw72xx-0x-imx219.dtbo
+           51648 imx8mm-venice-gw72xx-0x-rpidsi.dtb
+            1989 imx8mm-venice-gw72xx-0x-rpidsi.dtbo
+           51069 imx8mm-venice-gw72xx-0x-rs232-rts.dtb
+            1241 imx8mm-venice-gw72xx-0x-rs232-rts.dtbo
+           51156 imx8mm-venice-gw72xx-0x-rs422.dtb
+            1292 imx8mm-venice-gw72xx-0x-rs422.dtbo
+           51145 imx8mm-venice-gw72xx-0x-rs485.dtb
+            1281 imx8mm-venice-gw72xx-0x-rs485.dtbo
+           50843 imx8mm-venice-gw72xx-0x.dtb
+           52878 imx8mm-venice-gw73xx-0x-gw16157.dtb
+            2157 imx8mm-venice-gw73xx-0x-gw16157.dtbo
+           52540 imx8mm-venice-gw73xx-0x-imx219.dtb
+            2293 imx8mm-venice-gw73xx-0x-imx219.dtbo
+           52641 imx8mm-venice-gw73xx-0x-rpidsi.dtb
+            1989 imx8mm-venice-gw73xx-0x-rpidsi.dtbo
+           52042 imx8mm-venice-gw73xx-0x-rs232-rts.dtb
+            1241 imx8mm-venice-gw73xx-0x-rs232-rts.dtbo
+           52139 imx8mm-venice-gw73xx-0x-rs422.dtb
+            1292 imx8mm-venice-gw73xx-0x-rs422.dtbo
+           52128 imx8mm-venice-gw73xx-0x-rs485.dtb
+            1281 imx8mm-venice-gw73xx-0x-rs485.dtbo
+           51836 imx8mm-venice-gw73xx-0x.dtb
+           38309 imx8mm-venice-gw75xx-0x.dtb
+           44288 imx8mm-venice-gw7901.dtb
+           42563 imx8mm-venice-gw7902.dtb
+           39682 imx8mm-venice-gw7903.dtb
+           40046 imx8mm-venice-gw7904.dtb
+           38039 imx8mn-venice-gw7902.dtb
+           46598 imx8mp-venice-gw71xx-2x.dtb
+           49113 imx8mp-venice-gw72xx-2x.dtb
+           49684 imx8mp-venice-gw73xx-2x.dtb
+           68290 imx8mp-venice-gw74xx-imx219.dtb
+            2094 imx8mp-venice-gw74xx-imx219.dtbo
+           68430 imx8mp-venice-gw74xx-rpidsi.dtb
+            2025 imx8mp-venice-gw74xx-rpidsi.dtbo
+           67594 imx8mp-venice-gw74xx.dtb
+           47398 imx8mp-venice-gw75xx-2x.dtb
+        11300992 kernel.itb
+            2242 boot.scr
+#^^^ and it contains our kernel/dtbs/script
+}}}
+
+
+
+Note that NVMe support in U-Boot for imx8m mini is likely coming soon; NVMe has been supported in U-Boot for a while now, but PCIe for IMX8M has not been. IMX8MP PCI support landed upstream just a few weeks ago (and we've tested/enabled it (PCI/NVME) upstream for imx8mp-venice). imxmm support is hopefully coming soon. 
 
 [=#u-boot]