[[PageOutline]] = Provisioning boards = We refer to the act of installing a firmware image across multiple boards as **Provisioning**. Provisioning always starts with creating an image from a build system and this varies greatly depending on your target system. Ultimately you end up building binary images that need to be programmed to the boot device at various locations. For example, boot firmware, firmware blobs, filesystem data etc. You could stitch all these various images together into a 'disk image' or 'device image' and flash it as a single object however doing so on a particularly large flash device, say an 8GiB eMMC can take a long time depending on how much data you need to write. == Overall Venice Recommendations This wiki page has a great amount of information with many different options depending on needs and desires. Provisioning refers to flashing 100s or 1000s of boards with the same software when placing a large order with Gateworks. Gateworks typically does not want a 900MB .bin JTAG file because this will take forever to flash. Gateworks prefers a compressed image that shortens the time to flash the eMMC. Typical compressed disk images with Ubuntu or Buildroot will be 25-300MB. This disk image can then be flashed to the Gateworks board with the update_all script in the bootloader as defined here: [wiki:venice/firmware#disk-images] A compressed disk image is most easily built with the Venice Ubuntu BSP, here: [wiki:venice/bsp] To understand how the Venice BSP puts together all the bootloader, kernel, rootfs, etc all together into a compressed disk image, study the BSP Makefile, target ubuntu-image: [https://github.com/Gateworks/bsp-venice/blob/master/Makefile] Once completed, send Gateworks support team your .img.gz compressed disk image to be applied to a large order. == More Venice Options There are other recommended options from Gateworks: * Option A: * Pull off / rip an image of a single known configured system ('gold image') and then flash to 100s of more boards * Can require consideration of unique information such as serial-number from /proc/device-tree/serial-number, a mac address, or a unique ID from a crypto device when duplicated across 100s of boards * Option B: * Create an image programatically by building Ubuntu with changes embedded into the build and then flash to 100s of more boards === Option A Details * Pull off an image of a single known configured system ('gold image') and then flash to 100s of more boards * Can require consideration of unique information such as serial-number from /proc/device-tree/serial-number, a mac address, or a unique ID from a crypto device when duplicated across 100s of boards 1. Boot target to a Linux rescue image 2. Use the rescue image to create a tarball of your partitions, for example eMMC partition 1 (mmcblk2p1 on venice) {{{ # copy partition table (or just re-create it manually during the install) dd if=/dev/mmcblk2 of=/tmp/partition-table bs=1M count=1 # create a tarball of the partition mount /dev/mmcblk2p1 /mnt tar --numeric-owner -cJf /tmp/mmcblk2p1.tar.xz -C /mnt . }}} You will either need to store this on a mounted removable storage device partition or store to ramdisk (as above) and transfer them via network. If the compressed data exceeds the size of the ramdisk you will need to go straight to removable storage. The reason you do not want to simple compress the '''entire eMMC device''' (/dev/mmcblk2 on Venice) or partition (/dev/mmcblk2p1 for example) is because this will end up a) compressing unused data which will very likely not compress well because it is un-allocated and likely random data b) also ends up writing all unused data which takes a lot of time during the installation process To install your 'gold image' on a target: 1. Boot to a Linux rescue image (below) 2. Run a script that creates your partition table and partition contents from the partition tarballs, for example eMMC partition 1 (mmcblk2p1 on venice) {{{ # install the saved partition table dd if=/tmp/partition-table of=/dev/mmcblk2 # or create partition table, ie start at 1MiB (2048 512byte blocks) and end at the extent of the device printf "2048,,L,,\n" | sudo sfdisk -uS /dev/mmcblk2 # create an ext4 fs on partition 1 mkfs.ext4 -q -F -L rootfs /dev/mmcblk2p1 # mount it mount /dev/mmcblk2p1 /mnt # extract the tarball to it tar -C /mnt -xf /tmp/mmcblk2p1.tar.xz --keep-directory-symlink # unmount it umount /dev/mmcblk2p1 }}} Repeat the above for all partitions and follow-up with any additional per-device modifications you would like to make (ie doing something that depends on the devices serial-number from /proc/device-tree/serial-number, a mac address, or a unique ID from a crypto device). === Option B Details Create an image programatically by building Ubuntu with changes embedded into the build and then flash to 100s of more boards. This is similar to building the Venice BSP but with modifications to Ubuntu (such as packages pre-installed) If you want a more 'programmatic' way of creating your image that doesn't involve ripping the contents of a 'gold image' that may have been created through a complex variety of manual steps (Some engineers always prefer programmatic approaches): 1. Create Ubuntu image with packages pre-installed and system pre-configured, see instructions here: [wiki:ubuntu#CreatingUbunturootfsusingqemuandchroot Building Ubuntu Image] 1. Boot target to a Linux rescue image (below) 1. Run a script that creates your partition table and partition contents from scratch (using the steps from the previous option) === Linux Rescue Image A Linux Rescue Image is a self-contained Linux OS that does not rely on an on-board rootfs. This can be a kernel and device-tree with a separate rootfs that is on removable storage, or it can be a kernel with the rootfs built-in (which is how we like to do it at Gateworks just to have one less file/filesystem to worry about). It is very easy to create a Linux Rescue image using buildroot to build the root filesystem as well as the kernel. You can include in buildroot all the tools you find necessary for the operations you need to perform. We post pre-built images on http://dev.gateworks.com/buildroot/venice/minimal and document how to build them at http://trac.gateworks.com/wiki/buildroot#VeniceIMX8M Provided you are using a kernel Image with a built-in rootfs (such as the prebuilt Images we post on http://dev.gateworks.com/buildroot/venice/minimal/) you can boot this via the following in U-Boot: {{{ # via TFTP u-boot=> setenv bootargs; tftpboot $kernel_addr_r Image && tftpboot $fdt_addr_r imx8mm-venice-gw72xx-0x.dtb && booti $kernel_addr_r - $fdt_addr_r # via USB Mass Storage device (usb device 0 partition 1) u-boot=> setenv bootargs; usb start && load usb 0:1 $kernel_addr_r Image && load usb 0:1 $fdt_addr_r imx8mm-venice-gw72xx-0x.dtb && booti $kernel_addr_r - $fdt_addr_r # via microSD (mmc device 1 partition 1) u-boot=> setenv bootargs; load mmc 1:1 $kernel_addr_r Image && load mmc 1:1 $fdt_addr_r imx8mm-venice-gw72xx-0x.dtb && booti $kernel_addr_r - $fdt_addr_r }}} == FLASH programming methods and performance The speed at which you can physically write to a FLASH device depends on the method you are using to transfer and write the data as well as the performance of the SoC's access to the device. You have a few options available to write to the FLASH boot device: - Gateworks JTAG programmer (extremely low transfer data rate) - U-Boot (much higher data rate depending on driver capability in U-Boot) - Linux (best data rate) Both the transfer and write data rate are important. JTAG offers an extremely low transfer data rate but is an extremely useful method for getting <32MiB boot firmware and/or provisioning firmware on a board in an extremely reliable fashion. The board does not need to have functioning or compatible firmware to begin with. Note that the performance varies greatly across product family as well. The Venice IMX8MM product family for example has much higher MMC write performance than the older Newport CN803X product family. These considerations as well as your image contents and size need to be taken into account when you decide upon your provisioning method. U-Boot offers ok transfer rates and write performance (depending on driver support in U-Boot and noting that U-Boot is not an OS) but can't be used if the firmware on the board if corrupt, blank, or if the tools and features you need for provisioning do not exist in U-Boot. Note that because Gateworks ships all boards with U-Boot and a Linux based OS you can be guaranteed to boot to a U-Boot prompt with a reliable set of tools and features. Linux offers the best transfer rate and write performance with the notion that it is an interrupt capable full operating system but also requires that you are able to boot to Linux or a Linux that has the tools and features needed for your provisioning needs. Note that Because Gateworks ships all boards with U-Boot you at least know you can start at that point to load a version of Linux supporting whatever provisioning tools you need even if the default Linux OS may not have what you need. Summary of Pros and Cons of FLASH programming methods: - JTAG: - Pro: No dependence on booting firmware already on the boot device (FLASH can be corrupt/blank) - Con: Slow transfer rate - U-Boot: - Pro: minimal dependence on boot firmware - Cons: - must have U-Boot pre-programmed on board - performance maybe sub-par compared to Linux - possible unfamiliarity of U-Boot commands and features - possible lack of features and flexibility - Linux: - Cons: - more dependence and work in setting up a provisioning system - Pros: - best in class transfer and write performance - vast variety and familiarity of tools == Using compressed disk images While Gateworks often distributes single binary compressed firmware images for options such as OpenWrt and Ubuntu compatible images, we take care to keep these images small so that the time to flash them is minimal in order to be able to easily flash them with U-Boot. This very well may not be obtainable in your situation especially if you end up wanting multiple partitions with large spaces between them. The compressed disk image method involves: - creating an uncompressed image that includes things like your boot firmware, partition table, and partition images - stitching all these together putting each binary image at its correct offset using a tool like {{{dd}}} - compressing this with gzip - optionally growing your partition and filesystem to fit the extents of the device at runtime on the first boot Pros of compressed disk image method: - single file compressed image distribution - easy to install via U-Boot (obtaining the image via tftp or removable storage) Cons of compressed disk image method: - the compressed image needs to fit into available RAM (or a complicated method of splitting it be used) - can be slow if your uncompressed image is large (ie over 100's MiB's) - may require resizing the partition(s) and filesystem(s) per your needs Note that in order to keep the time required to install the Gateworks Ubuntu rootfs images for example, we create a filesystem just large enough to fit the rootfs meaning (appx 1.6GiB) meaning that is all we have to write. Upon first boot a script runs that resizes the rootfs partition and filesystem to fit the extents of the device. This not only minimizes the amount of data that must be written but allows the disk image to fit into large devices and provide the user with expansion space without any further actions needed. Some performance examples: - Gateworks Venice GW7301-01 IMX8MM installing Ubuntu - Note that IMX8MM supports HS400 eMMC data rates - U-Boot: - 14 seconds to transfer 460MiB compressed image via tftp to RAM at GbE - 45 seconds to uncompress and write the 1.6GiB of data to eMMC at HS400 speeds - Gateworks Newport GW6404 CN8031 installing focal-newport.img.gz - Note that CN803x supports a max of 50MHz MMC data rates - U-Boot: - 58 seconds to transfer 460MiB compressed image via tftp to RAM at GbE - 2 minutes to uncompress and write the 1.6GiB of data to eMMC at 50Mhz speeds To present an example of where this method might not scale well consider a scenario where you multiple partitions defined across an 8GiB device. Using this method you would need to write almost the entire 8GiB which would take appx 10 minutes using this method. If your board has a 64GiB eMMC device you can see how this can get out of hand fairly quickly. == Hybrid approaches There are hybrid approaches you may consider as well. You could choose to create a JTAG'able image that contained custom firmware that automatically boots into a provisioning system that uses Linux for example. This JTAG'able image could be small enough to flash quickly over JTAG and you could even elect to have Gateworks pre-program your image on your boards if you have a Gateworks special or custom build. In this situation for example your image could pull a script from the network to complete your provisioning so that the image programmed on your boards can stay consistent but provide you flexibility in modifying the instructions for provisioning later on. This is the most flexible approach. Another example of a hybrid approach is to use U-Boot to provision images based on a script of U-Boot commands such that you can program various portions of your disk image at a time. For example if you had a scenario where you have multiple filesystems spread across a large eMMC device you could fetch and program the boot-firmware, partition table, and each filesystem independently keeping the data actually written to your FLASH device as small as possible. While this approach could be done in U-Boot alone, there are likely still multiple advantages to booting a Linux ramdisk image to provide greater flexibility and/or familiarity with tools. == Provisioning from a Linux ramdisk environment By far the most flexible solution is being able to provision your boards live with a Linux environment booting to a ramdisk providing all the tools you need for your custom environment. The most successful provisioning environments Gateworks has seen in the past are those that allow you to control your provisioning environment yourself in-dependent of what firmware may be pre-installed on your boards. While having a Gateworks special or Gateworks custom board allows you to dictate what firmware you want programmed on your boards by keeping that firmware as simple as possible you have an advantage of either not needing custom firmware or not having to change that firmware as new provisioning needs arise. For example, While this is the most flexible it is also the most complicate to setup. A very easy way to build a minimal ramdisk Linux based OS is to use buildroot. All you need is basic kernel support for the target board and a minimal set of tools: - minimal root filesystem (only tools you need for your provisioning needs) - minimal kernel config (only drivers/features you need for your provisioning needs) - ramdisk (build artifact will be a 'Image' which is a kernel + ramdisk) - script that performs the provisioning - init config that runs the script on boot (if you wanted it automated) The tools you need in your minimal rootfs: - partition editor (sgdisk and/or parted) - filesystem creation tools (ie e2fsprogs if using ext2/3/4) - u-boot-tools (fw_printenv/fw_setenv/mkenvimage) if setting or altering uboot env - networking support to fetch your filesystem tarballs (which could also come from removable storage) - optionally your provisioning script (or you can transfer this via network) The pre-built buildroot kernel+ramdisk minimal images that Gateworks provides at http://dev.gateworks.com/buildroot/ can be used for such provisioning. The instructions on how to build such an image can be found on the [wiki:#buildroot] page. When performing this provisioning, just like in creating a disk image, you need to understand what partitions may be needed by boot firmware (for example Newport expects a FATFS partition that exists within its boot firmware) and what Linux device you want to provision (ie eMMC). When we create disk partitions below we also take care to create reserved partitions to cover the boot firmware. It is also important to understand what, if any, bootscript you want to install and where. Examples: * provision a Venice board eMMC with a single ext4 root filesystem from a rootfs and kernel tarball: 1. Load and boot kernel+ramdisk buildroot image from TFTP server {{{#!bash tftpboot $kernel_addr_r Image && booti $kernel_addr_r - $fdtcontroladdr }}} 2. Partition /dev/mmcblk0 (emmc) {{{#!bash DEV=/dev/mmcblk0 GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908 GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4 # use sgdisk to create partitions sgdisk -og $DEV sgdisk -n 1:66:32767 -c 1:"boot" -t 1:$GUID_RESERVED $DEV sgdisk -n 2:32768:0 -c 2:"root" -t 2:$GUID_LINUXFS $DEV # set the legacy 'boot' attribute on rootfs partition sgdisk -A 2:set:2 $DEV # print partition table sgdisk -p $DEV }}} - Note the partitioning scheme chosen above creates protective partitions around the boot firmware which includes the SPL, ATF, and U-Boot (plus it's environment). This is both to protect any other partition tools from thinking there is free space there as well as provide you a partition to be able to easily update those portions of the boot firmware at a later date - Note we set the rootfs partition bit 2 attribute to mark the partition as BIOS bootable in case you are using the U-Boot generic distro config to look for bootscripts on bootable partitions - Note the 'end sector' for the rootfs partition above is 0 meaning it will size to the end of the device 3. Create and populate the rootfs partition {{{#!bash # fetch tar archives from network udhcpc -i eth0 # bring up networking cd /tmp wget http://tharvey/tftpboot/focal-venice.tar.xz wget http://tharvey/tftpboot/linux-venice.tar.xz wget http://dev.gateworks.com/ubuntu/focal/focal-venice.tar.xz wget http://dev.gateworks.com/venice/kernel/linux-venice.tar.xz # create and populate rootfs (P2) from tarballs PART=${DEV}p2 mkfs.ext4 -q -F -L rootfs $PART mount $PART /mnt tar -C /mnt -xf focal-venice.tar.xz --keep-directory-symlink tar -C /mnt -xf linux-venice.tar.xz --keep-directory-symlink umount /mnt }}} 4. Create a bootscript if desired (note this requires understanding of how U-Boot generic distro bootconfig works; you could alternately simply put whatever you need directly into U-boot env below) {{{#!bash cat <<\EOF > /tmp/bootscript echo "Venice Boot Script" # determine root device using uuid part uuid ${devtype} ${devnum}:${distro_bootpart} uuid # bootargs setenv bootargs console=$console root=PARTUUID=${uuid} rootwait $bootargs # load and boot kernel load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${prefix}Image && booti ${kernel_addr_r} - ${fdtcontroladdr} EOF mount $PART /mnt mkimage -A arm64 -T script -C none -d /tmp/bootscript /mnt/boot/boot.scr umount /mnt }}} 5. Customize U-Boot env if desired (note this requires understanding where the U-Boot env is located on FLASH) {{{#!bash # customize U-Boot env cat << EOF > /tmp/fw_env.config # Device Device offset $DEV 0xff0000 0x8000 $DEV 0xff8000 0x8000 EOF # get current config fw_printenv --config /tmp/fw_env.config > /tmp/uboot.env # append any desired config changes to /tmp/uboot.env echo "bootdelay=1" >> /tmp/uboot.env # write new config (perform twice so redundant env gets created as well) fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env }}} * Provision a Newport board eMMC with a single ext4 root filesystem from a rootfs and kernel tarball: 1. Load and boot kernel+ramdisk buildroot image from TFTP server {{{#!bash tftpboot $kernel_addr_r Image && booti $kernel_addr_r - $fdtcontroladdr }}} 2. Partition /dev/mmcblk0 (emmc) {{{#!bash DEV=/dev/mmcblk0 GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908 GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4 # first use parted to create a GPT as sgdisk does not like the default MBR based partition table parted --script $DEV mklabel gpt # use sgdisk to create partitions sgdisk -og $DEV sgdisk -n 1:2048:26623 -c 1:"fatfs" -t 1:$GUID_BASIC_DATA $DEV sgdisk -n 2:28672:30719 -c 2:"atf" -t 2:$GUID_RESERVED $DEV sgdisk -n 3:30720:32767 -c 3:"uboot" -t 3:$GUID_RESERVED $DEV sgdisk -n 4:32768:0 -c 4:"root" -t 4:$GUID_LINUXFS $DEV # set the legacy 'boot' attribute on rootfs partition sgdisk -A 4:set:2 $DEV # print partition table sgdisk -p $DEV }}} - Note the partitioning scheme chosen above creates protective partitions around various portions of the boot firmware, namely the BDK bootstub, the fatfs filesystem, the ATF, and U-Boot (plus it's environment). This is both to protect any other partition tools from thinking there is free space there as well as provide you a partition to be able to easily update those portions of the boot firmware at a later date - Note we set the rootfs partition's bit 2 attribute to mark the partition as BIOS bootable in case you are using the U-Boot generic distro config to look for bootscripts on bootable partitions - Note the 'end sector' for the rootfs partition above is 0 meaning it will size to the end of the device 3. Create and populate the rootfs partition {{{#!bash # fetch tar archives from network udhcpc -i eth0 # bring up networking cd /tmp wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz # create and populate rootfs (P4) from tarballs PART=${DEV}p4 mkfs.ext4 -q -F -L rootfs $PART mount $PART /mnt tar -C /mnt -xf focal-newport.tar.xz --keep-directory-symlink tar -C /mnt -xf linux-newport.tar.xz --keep-directory-symlink umount /mnt }}} 4. Create a bootscript if desired (note this requires understanding of how U-Boot generic distro bootconfig works; you could alternately simply put whatever you need directly into U-boot env below) {{{#!bash cat <<\EOF > /tmp/bootscript echo "Newport Boot Script" setenv bootargs ${bootargs} root=/dev/mmcblk${devnum}p${distro_bootpart} rootwait # disable USB autosuspend (CN81xx errata) setenv bootargs ${bootargs} usbcore.autosuspend=-1 # disable KPTI (expected chip errata) setenv bootargs ${bootargs} kpti=0 # add console setenv bootargs ${bootargs} console=${console} # load and boot kernel echo "Loading kernel Image from ${devtype} ${devnum}:${distro_bootpart} ${prefix}" load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${prefix}Image booti ${kernel_addr_r} - ${fdtcontroladdr} EOF mount $PART /mnt mkimage -A arm64 -T script -C none -d /tmp/bootscript /mnt/boot/newport.scr umount /mnt }}} 5. Customize U-Boot env if desired (note this requires understanding where the U-Boot env is located on FLASH) {{{#!bash # customize U-Boot env cat << EOF > /tmp/fw_env.config # Device Device offset $DEV 0xff0000 0x8000 $DEV 0xff8000 0x8000 EOF # get current config fw_printenv --config /tmp/fw_env.config > /tmp/uboot.env # append any desired config changes to /tmp/uboot.env echo "bootdelay=1" >> /tmp/uboot.env # write new config (perform twice so redundant env gets created as well) fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env }}} == Ventana NAND flash based boards == Products that use NAND flash present an issue in that they can contain bad blocks. As a result the raw flash devices can differ in size making it difficult to implement a JTAG flash upload/download scenario. There are several ways of provisioning NAND bootable boards: * using JTAG (this is what Gateworks uses on our production line) * using U-Boot (more complex, much faster than JTAG, but does not allow provisioning the SPL) * using Linux (even more complex, much faster than JTAG, but allows provisioning the SPL) * a combination of the above Regardless of the method used for provisioning there are several artifacts that you need in order to provision NAND: * SPL (secondary program loader) * u-boot.img (bootloader) * env (bootloader env) * ubi (unsorted block image containing ubifs filesystem) === Pulling Software off of an Existing Board === ==== SPL and Bootloader ==== The SPL and u-boot.img are built artifacts (which can be downloaded from ​http://dev.gateworks.com/ventana/images). ==== Bootloader Environment ==== The env can be blank, which will use built-in defaults, or can be customized and extracted from the flash. To create and extract a bootloader env: 1. Create the env on a board: {{{#!bash # blank per-board vars (which are set from eeprom by default, yet overridable via env) setenv fdt_file setenv ethaddr setenv eth1addr # perform any other desired changes # save saveenv }}} 2. Extract the env from the board and save to removable storage: - from Linux {{{#!bash dd if=/dev/mtd1 of=env bs=1M }}} - from U-Boot: {{{#!bash # read the env (environment) partition into temporary memory, note the size reported below as 0x100000 Ventana > nand read ${loadaddr} env NAND read: device 0 offset 0x1000000, size 0x100000 1048576 bytes read: OK Ventana > # store it to file on micro-SD with an ext4 fs (size re-used from above) mmc dev 0 && ext4write mmc 0:1 ${loadaddr} /env 0x100000 # or store it to file on USB mass storage with an ext4 fs usb start && usb dev 0 && ext4write usb 0:1 ${loadaddr} /env 0x100000 }}} - Note that you may find it easier to build yourself a custom bootloader with defaults that match your needs rather than deal with extracting and imaging an env flash partition - Note that your ext4 filesystem must not have checksums enabled (metadata_csum, a feature added to newer e2fsprogs) as U-Boot does not support this in ext4write. ==== Root Filesystem ==== The ubi root filesystem is originally built by the build system of the specific BSP your using, however if you end up imaging this onto a board, and customizing it, you will not be able to bit-per-bit copy it to another board due to flash geometry changes and bad block mapping on the physical raw NAND parts. Instead, you will have to boot a ramdisk based Linux rescue image, mount the ubifs and create a tarball of the files which you can then use to re-create a ubi on a Linux development host that can be applied to a new target board via U-Boot or Linux. Procedure: 1. Boot a Linux rescue image that has networking and UBI/EXT4/FAT filesystem support. The [wiki:buildroot#VentanaIMX6 buildroot based image] is suitable for this. You will need to copy the uImage and dtb files from http://dev.gateworks.com/buildroot/ventana/minimal/ to a TFTP server on your network or a removable microSD or USB storage device and boot with one of the following: a. boot rescue image from network tftpserver {{{#!bash setenv bootargs console=ttymxc1,115200 setenv fsload tftpboot setenv bootdir ventana # set this to the prefix of your tftp dir run loadfdt && $fsload $loadaddr $bootdir/uImage && bootm $loadaddr - $fdt_addr }}} b. boot rescue image from microSD (dev 0 partition 1) with ext/fat filesystem {{{#!bash setenv bootargs console=ttymxc1,115200 setenv fsload load mmc 0:1 setenv bootdir . # set this to the prefix of your tftp dir run loadfdt && $fsload $loadaddr $bootdir/uImage && bootm $loadaddr - $fdt_addr }}} c. boot rescue image from USB (dev 0 partition 1) with ext/fat filesystem {{{#!bash setenv bootargs console=ttymxc1,115200 setenv fsload load usb 0:1 usb start setenv bootdir . # set this to the prefix of your tftp dir run loadfdt && $fsload $loadaddr $bootdir/uImage && bootm $loadaddr - $fdt_addr }}} 2. Mount a filesystem on removable storage (you could use a tmpfs filesystem to do this on a ramdisk if you have enough space): {{{#!bash mkdir /mnt/removable mount /dev/sda1 /mnt/removable }}} * /dev/sd* would be for USB or mSATA devices and /dev/mmc* would be for microSD devices 3. Mount the ubifs: {{{#!bash ubiattach /dev/ubi_ctrl -m 2 # attach /dev/mtd2 as /dev/ubi0 mkdir -p /mnt/ubifs # create a mountpoint for it mount -t ubifs ubi0:rootfs /mnt/ubifs # mount 'rootfs' volume }}} * Note that at this point you could create a tarball of the rootfs if you want to perfor the following steps later or on a Linux development host (tar --numeric-owner -cvzf /mnt/removable/rootfs.tar.gz -C /mnt/ubifs .) 4. Create a ubi image for the particular flash geometry of the target device (see wiki:linux/ubi#creatingubi): a. 'normal' geometry FLASH devices (2048 byte page size) {{{#!bash # create a ubifs image mkfs.ubifs -F -m 2048 -e 124KiB -c 16248 -x zlib -o /mnt/removable/root.ubifs -d /mnt/ubifs # create a ubi image cat < ubinize.cfg [ubifs] mode=ubi image=/mnt/removable/root.ubifs vol_id=0 vol_type=dynamic vol_name=rootfs vol_flags=autoresize EOF # create ubi suitable for 2048byte page size and 128KiB block size devices ubinize -m 2048 -p 128KiB -o /mnt/removable/root.ubi ubinize.cfg }}} * Note that the mkfs.ubifs process can take quite a bit of time with no output depending on the size of your filesystem; takes about 8 minutes for a 1GiB filesystem b. 'large' geometry FLASH devices (4096 byte page size) {{{#!bash # create a ubif image mkfs.ubifs -F -m 4096 -e 248KiB -c 8124 -x zlib -o /mnt/removable/root.ubifs -d /mnt/ubifs # create a ubi image cat < ubinize.cfg [ubifs] mode=ubi image=/mnt/removable/root.ubifs vol_id=0 vol_type=dynamic vol_name=rootfs vol_flags=autoresize EOF # create ubi suitable for 4096byte page size and 256KiB block size devices ubinize -m 4096 -p 256KiB -o /mnt/removable/root.ubi ubinize.cfg }}} * Note that the mkfs.ubifs process can take quite a bit of time with no output depending on the size of your filesystem; takes about 8 minutes for a 1GiB filesystem 5. Unmount the filesystems: {{{#!bash umount /mnt/removable umount /mnt/ubifs ubidetach /dev/ubi_ctrl -m 2 }}} === Flashing Boards with Pulled Software === Once you have all the artifacts you can re-assemble them into a JTAG image suitable for the Gateworks JTAG adapter and software. The following usage of mkimage_jtag will create a jtagable image matching the partitioning described by 'mtdparts=nand:16m(uboot),1m(env),-(rootfs)' {{{#!bash mkimage_jtag -e SPL@0 u-boot.img@14M env@16M ubi@17M > image.bin }}} Or, for a faster two-step method of imaging using U-Boot with serial and ethernet (to a tftp server with the ubi): 1. create a JTAG image of the SPL + bootloader + env: {{{#!bash mkimage_jtag SPL u-boot.img env > image.bin }}} 2. once the above is flashed with the Gateworks JTAG adapter and software you can flash the ubi (much more quickly than via JTAG) within U-Boot 3. break out into the bootloader and load the ubi image and flash it: a. from network via tftp: {{{#!bash setenv ipaddr 192.168.1.1 # local ip setenv serverip 192.168.1.146 # server ip tftp ${loadaddr} image.ubi # tftp ubi image nand erase.part rootfs # erase the nand partition named rootfs from the mdtparts variable nand write ${loadaddr} rootfs ${filesize} # write the downloaded ubi to rootfs }}} b. from removable storage via msata (device 0 partition 1): {{{#!bash sata init load sata 0:1 $loadaddr root.ubi nand erase.part rootfs nand write $loadaddr rootfs $filesize }}} c. from removable storage via USB (device 0 partition 1): {{{#!bash usb start load usb 0:1 $loadaddr root.ubi nand erase.part rootfs nand write $loadaddr rootfs $filesize }}} d. from removable storage via microSD (device 0 partition 1): {{{#!bash load mmc 0:1 $loadaddr root.ubi nand erase.part rootfs nand write $loadaddr rootfs $filesize }}} * Note writing the nand may take some time; several minutes for a 600MB ubi Notes: * you can always elect to build your own bootloader with a custom config rather than pulling the env data off a board [=#ubootenv] == U-Boot environment provisioning The U-Boot bootloader uses environment variables and scripts to control the booting of a board. Often it is desirable to override the default environment the Gateworks BSP's create and create your own. The device, location, size, and redundancy details of how the U-Boot environment is stored vary from board to board based on U-Boot's configuration: * Venice: 32KiB (0x8000) redundant env stored at 16320KiB (0xff0000) offset for MMC * Newport: 32KiB (0x8000) redundant env stored at 16320KiB (0xff0000) offset for MMC * Ventana: - MMC: 128MiB redundant env stored at 709KiB offset - NAND: 128MiB redundant env stored at 16MiB offset There are some tools from the U-Boot project available in the u-boot-tools package that can be used to alter or create an env that can then be copied directly to flash: * fw_setenv / fw_getenv - requires a config file and can get or alter the env. If there is no valid env where the config file points to a default env hard coded into the bootloader will be returned. * mkenvimage - create an env image from scratch with a specified size and redundancy. For examples of how the default env is created for the various Gateworks products: - Venice BSP uses fw_setenv to alter the hard-coded default env. For details see: * https://github.com/Gateworks/bsp-venice/blob/master/Makefile * https://github.com/Gateworks/bsp-venice/blob/master/venice.env * https://github.com/Gateworks/bsp-venice/blob/master/fw_env.config - Newport BSP uses fw_setenv to alter the hard-coded default env. For details see: * https://github.com/Gateworks/bsp-newport/blob/sdk-10.1.1.0-newport/Makefile * https://github.com/Gateworks/bsp-newport/blob/sdk-10.1.1.0-newport/newport.env * https://github.com/Gateworks/bsp-newport/blob/sdk-10.1.1.0-newport/fw_env.config - Ventana firmware images have a blank env area therefore use hard-coded default env from U-Boot To create a specific env from scratch (ie without inheriting any defaults) use the {{{mkenvimage}}} tool from the u-boot-tools package 1. Create an env.txt file with 'name=value' pairs. Be sure to include everything that is necessary as no defaults will be available other than variables that are set at runtime on the fly such as serial#, model, and macaddr: {{{#!bash cat << EOF >> env.txt baudrate=115200 bootcmd=run distro_boot ... EOF }}} - **Note you need to make sure this is complete and proper for your board - you should do a 'print' of a Gateworks pre-built image as a starting point then remove any variables that are board specific and created by U-Boot at runtime 2. Use {{{mkenvimage}}} taking care to specify the env size the bootloader is configured to use and specify '-r' for redundancy if the bootloader is configured for that: * Venice / Newport {{{#!bash mkenvimage -r -s $((32 * 1024)) -o env.bin env.txt }}} * Ventana: {{{#!bash mkenvimage -r -s $((128 * 1024)) -o env.bin env.txt }}} 3. Place this image in FLASH: * Venice / Newport: - Example: compressed disk image for Venice / Newport: {{{#!bash gunzip disk.img.gz dd if=env.bin of=disk.img bs=1K seek=16320 oflag=notrunc gzip disk.img }}} - Example: update from Linux: {{{#!bash dd if=env.bin of=/dev/mmcblk0 bs=1K seek=16320 oflag=notrunc }}} - Example: update from U-Boot (16320KiB is equivalent to 0x7f80 512byte blocks in hex) {{{#!bash tftp $loadaddr env.bin && mmc write $loadaddr 0x7f80 $filesize }}} * Ventana: - Example: JTAG binary image for a Ventana NAND device: {{{#!bash mkimage_jtag -e SPL@0 u-boot.img@14M env.bin@16M ubi@17M > image.bin }}} - Example: update from Linux: {{{#!bash dd if=env.bin of=/dev/mmcblk0 bs=1K seek=16384 oflag=notrunc }}} - Example: update from U-Boot (16384MiB is equivalent to 0x8000 512byte blocks in hex) {{{#!bash tftp $loadaddr env.bin && mmc write $loadaddr 0x8000 $filesize }}} [=#microsd] == micro-SD provisioning == === Directly Cloning SD Cards === See [wiki:linux/blockdev#Creatingadiskimage linux/blockdev] === U-Boot MicroSD Provisioning === The main difference between provisioning removable storage devices such as micro-SD compared to non-removable storage devices (such as NAND flash) is that the removable devices can be potentially booted on a board with a different model, CPU, or memory configuration. This causes us to treat the U-Boot environment differently when we extract it from a configured board. If you do not want a blank env (which uses built-in defaults) you must provision one board, boot it to the bootloader, customize your env, then extract that env to use when provisioning additional boards. The Ventana bootloader stores its microSD env on raw block sectors from offset 709K and is 256K in size. The env can be blank, which will use built-in defaults, or can be customized and extracted. To create and extract a bootloader env: (only if this is being used, otherwise skip this step) 1. Create the env on a board: {{{#!bash # blank per-board vars (which are set from eeprom by default, yet overridable via env) setenv fdt_file setenv ethaddr setenv eth1addr # perform any other desired changes # save saveenv }}} 2. extract the env from a board that boots to micro-SD: * from Linux: {{{#!bash # copy 256KB from offset 709KB to 'env' file dd if=/dev/sdc of=env bs=1K skip=709 count=256 oflag=sync }}} * from U-Boot: {{{#!bash mmc read ${loadaddr} 0x58a 0x200 # read 512x 512byte blocks (256K) from block 0x1418 # store it to file on micro-SD with an ext4 fs ext4write mmc 0:1 ${loadaddr} /mmc.env 0x40000 # store it to file on USB mass storage with an ext4 fs usb start && usb dev 0 && ext4write usb 0:1 ${loadaddr} /mmc.env 0x40000 }}} - Note that your ext4 filesystem must not have checksums enabled (metadata_csum, a feature added to newer e2fsprogs) as U-Boot does not support this in ext4write. To place an extracted env onto a micro-SD: * from Linux: {{{#!bash # copy 256KB from file env to offset 709KB: dd if=env of=/dev/sdc bs=1K seek=709 count=256 oflag=sync }}} = Provisioning root file system from live Newport board It's always best to create your rootfs from scratch using the Newport [wiki:/newport/bsp BSP] when this option is available. If you're prevented from doing so it is possible to provision the root file system from some Newport boards without too much difficulty. Newport SBC's are capable of booting from both eMMC and MMC. The boot device can be selected by the GSC performing a 5x press on the power button. In the following section we will create a bootable microSD card, load it as the primary boot device, image partition 2 of the eMMC using "dd", then create a compressed disk image from this data. == Requirements * A Newport board with MMC card slot and push button (GSC must be configured to accept push button input—this is default) * A MMC with capacity enough to accommodate full size of eMMC (8GB) along with BSP for performing recovery. Recommended would be a minimum 16GB or 32GB card. * A desktop computer with Linux natively installed. This workstation must have a drive capable of accepting an MMC and have packages for DD and Mount. == On your workstation Download a Gateworks pre-built image. All of our images include the tools necessary to perform this operation, though Ubuntu will likely be the easiest to work with. {{{ wget -N http://dev.gateworks.com/newport/images/focal-newport.img.gz }}} Insert a micro SD card into your workstation and identify the device name it's been assigned. This can be done using a variety of methods, for example the "dmesg" output. In this example the MMC is "/dev/sdb" Image the MMC. {{{ zcat focal-newport.img.gz | sudo dd of=/dev/sdb bs=4M }}} Remove the SD card from your workstation. == On your Newport SBC Insert the SD card into the SD card reader, then apply power to the SBC (or otherwise turn it on). 5x press the power button, you will see the status LED turn off and back on. The BDK will display microSD as MMC0. Example output: {{{ Gateworks Newport SPL (12.7.0-96865d0 Tue Jul 7 21:19:32 UTC 2020) GSC : v55 0xe7e2 RST:BOOT_WDT2 Thermal Protection Enabled Temp : Board:34C/86C CPU:42C/100C Model : GW6400-B1 MFGDate : 09-23-2019 Serial : 802864 RTC : 8 SoC : CN8020-800BG676-SCP-P12-G 1024KB 800/550MHz 0xa2 Pass 1.2 MMC0 : microSD MMC1 : eMMC }}} Alterntitively you can use a ramdisk (performance will be faster). [http://wiki:/provisioning#ProvisioningfromaLinuxramdiskenvironment Link to Ramdisk section] Proceed with booting to Linux user-space. Once there if you would like to view the rootfs you're about to image it's located at /dev/mmcblk1p2. Doing so is optional. {{{ mount /dev/mmcblk1p2 /mnt ls /mnt #this will display the eMMC's rootfs umount /mnt }}} Image the partition to a file. {{{ dd if=/dev/mmcblk1p2 of=myrootfs.img }}} "sync" the filesystem and power board off {{{ sync #remove power }}} Remove MMC from the SBC. == Create an image Navigate to the "/tmp" directory on your workstation. In actuality this can be any directory that you have read and write permissions, "/tmp" should only be used if you don't care what happens to these files later. {{{ cd /tmp }}} Insert the MMC into the SD card reader on your workstation, as before note the the name the device is assigned. Because we created our image from within the rootfs we will need to mount the second partition, for example "/dev/sdb2": {{{ sudo mount /dev/sdb2 /mnt/ ls /mnt/ }}} Copy the image you created of mmcblk1p2 into "/tmp" {{{ cp /mnt/myrootfs.img . }}} Executing the command "file" on this file will return that it is ext4 filesystem data and the volume name is "rootfs". Example: {{{ user@workstation:/tmp$ file myrootfs.img myrootfs.img: Linux rev 1.0 ext4 filesystem data, UUID=951f19a1-cc54-4a07-9b7f-41a54ea8acb4, volume name "rootfs" (needs journal recovery) (extents) (64bit) (large files) (huge files) }}} Add boot firmware and create a gzipped image. * Download boot firmware {{{ wget http://dev.gateworks.com/newport/boot_firmware/firmware-newport.img }}} * name it however you please {{{ cp firmware-newport.img myimg-newport.img }}} * Create a disk image containing the root filesystem and boot firmware. The boot firmware is 16M so we will "dd" the rootfs using this offset. {{{ dd if=myrootfs.img of=myimg-newport.img bs=16M seek=1 }}} * gzip the image so it can be installed using standard methods on other Newport boards. {{{ gzip -k -f myimg-newport.img }}} Be mindful that the image size can't exceed the total DRAM of the board you plan to install it on if you're using the bootloader command "tftpboot". The "update_all", and "update_rootfs" scripts both use this command {{{ ls -lh myimg-newport.img.gz #size must be less than total DRAM of board }}} == Flash this image to a Newport SBC: In the bootloader: {{{ GW6400-B1> setenv ipaddr 192.168.1.52 GW6400-B1> setenv serverip 192.168.1.56 GW6400-B1> setenv image myimg-newport.img.gz GW6400-B1> setenv dev 0 GW6400-B1> run update_all }}} = Provisioning Newport, direct write to eMMC on target Usint the bootloader's "gzwrite" method for flashig embedded flash has a lot of downsides; the gzipped image must be smaller than the capacity of the ram and it's difficult to create a minimal filesystem to speed up the writes. Depending on your application a more flexible and potentially time efficient aproach would be to use 'linux kernel+ramdisk + provisioning' script approach. Using this method may be neceessary if there are scripts or applications that must be run during the provisioning process. * [wiki:/buildroot#initrdinitialramdisk Buildroot Ramdisk Wiki] 1. Boot to Target - either from the alternate boot device (ie boot from microSD if working on eMMC or visa versa) or boot a kernel+ramdisk so that you can work off either device. We provide a pre-built ramdisk image with all the necessary tools. A secondary boot media can be use although it will not be as fast as the ramdisk. 1. Partition the device with a partition size just large enough to fit a filesystem that can contain your files. Determine the size in bytes needed for your filesystem, there are several ways to do this including simple trial and error. If you have a directory with your files you can use 'du -b --summarize' to get bytes. For a tar.gz a good estimate is the uncompressed size "gzip -l ". {{{#!bash # for an existing directory using du SZ_B=$(du -b --summarize /usr/src/ubuntu-rootfs/focal-newport | awk '{ print $1; }' # or just guess and use trial and error SZ_B=$((1536*1024*1024)) # 1536MiB fits the Gateworks ubuntu image }}} 1. Create partition table [wiki:/provisioning#ProvisioningfromaLinuxramdiskenvironment Example using GPT and sgdisk]. For Newport we need to preserve the FATFS filesystem used by the boot firmware. {{{#!bash DEV=/dev/mmcblk0 GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908 GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4 # first use parted to create a GPT as sgdisk does not like the default MBR based partition table parted --script $DEV mklabel gpt # use sgdisk to create partitions sgdisk -og $DEV sgdisk -n 1:2048:26623 -c 1:"fatfs" -t 1:$GUID_BASIC_DATA $DEV sgdisk -n 2:28672:30719 -c 2:"atf" -t 2:$GUID_RESERVED $DEV sgdisk -n 3:30720:32767 -c 3:"uboot" -t 3:$GUID_RESERVED $DEV END=$(( 32768 + $((SZ_B/512)) )) sgdisk -n 4:32768:$END -c 4:"root" -t 4:$GUID_LINUXFS $DEV # set the legacy 'boot' attribute on rootfs partition so the uboot env will boot from it sgdisk -A 4:set:4 $DEV # print partition table sgdisk -p $DEV }}} 1. Create your filesystem using the index of the partition for root as shown above. Note that rootfs is p4, this is because of the boot firmware (atf/uboot). Format the partition that will contain the rootfs to ext4. {{{#!bash mkfs.ext4 -q -F -L rootfs ${DEV}p4 }}} 1. Mount your filesystem, populate it, then umount. {{{#!bash mount ${DEV}p4 /mnt cd /tmp udhcpc -e eth0 wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz tar -C /mnt -xf focal-newport.tar.xz --keep-directory-symlink 2>/dev/null || echo Error!!! tar -C /mnt -xf linux-newport.tar.xz --keep-directory-symlink 2>/dev/null || echo Error!!! umount /mnt #check for errors returned from tar such as not enough space }}} ** Note: When creating tarballs use the --numeric-owner to preserve UID/GID numbers vs names and there is no need to be selective about files/directories on a non-live filesystem ** 6. Now you can dd and decompress your image up to the last block in the partition. The last block in the partition is $END which you can see by printing the partition table: {{{#!bash rm /tmp/* dd if=$DEV bs=512 count=$END | gzip > /tmp/disk.img.gz }}} 7. Install a compressed disk image: {{{#!bash cd /tmp wget disk.img.gz zcat disk.img.gz | dd of=$DEV bs=4M }}} == Provisioning Newport removable block storage on Desktop workstation Provisioning on the target is less than ideal in situations where a traditional workstation can be used. Both methods have their advantages and disadvantges. When using a desktop an issue may arrise when using partitioning tools, as most of them do not like working on 'files' and won't let you do things like partition past the size of the file. In those cases there are, and we provide, tools that create partition tables for you without adhering to any rules. 1. Create a 'file' for your filesystem image. 'truncate' can be used as it creates a 'sparse' file which reports the size it was created with but won't take up space for zero'd blocks. {{{#!bash truncate -s 1536M fs.img #create a sparse file mkfs.ext4 -q -F -L rootfs fs.img #format it. }}} 1. Mount your filesystem image and populate it: {{{#!bash sudo mount fs.img /mnt/disk wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz sudo tar -C /mnt/disk/ -xf focal-newport.tar.xz --keep-directory-symlink >/dev/null || echo "Error!!!" sudo tar -C /mnt/disk/ -xf linux-newport.tar.xz --keep-directory-symlink >/dev/null || echo "Error!!!" sudo umount /mnt/disk }}} 1. Create a 'file' for your disk image. This needs to be large enough for your boot firmware (which includes the partition table) as well as the filesystems you will copy to it. {{{#!bash truncate -s $((16+1536))M disk.img }}} 1. Copy the boot firmware to it at offset 0 {{{#!bash dd if=firmware-newport.img of=disk.img conv=notrunc #make sure to use conv=notrunc so it doesn't truncate your image file }}} 1. Copy your filesystem image to the correct sectors dictated by the partition table. If needed adjust/re-create the partition table taking care to ensure the boot firmware FATFS does not move. {{{#!bash dd if=fs.img of=disk.img bs=512 seek=32768 conv=notrunc }}} 1. Compress. {{{#!bash gzip disk.img }}} In conclusion, a partition is simply a chunk of space dictated by a header in the partition table which is in the first few blocks of the disk. It doesn't need to be filled, there can be a large partition with a small filesystem inside of it, that filesystem can be enlarged on first-boot with 'resize2fs'. Additionally the partition size can be incresead as well.