Changes between Version 14 and Version 15 of provisioning


Ignore:
Timestamp:
02/13/2021 02:03:33 AM (18 months ago)
Author:
Tim Harvey
Comment:

added sections regarding flash programming methods and provisioning from linux

Legend:

Unmodified
Added
Removed
Modified
  • provisioning

    v14 v15  
    22
    33= Provisioning boards =
    4 We refer to the act of duplicating a firmware image across multiple boards as **Provisioning**.
    5 
    6 The easiest way to provision boards or removable storage devices is to build the particular BSP you are interested in, and use its tools to create a JTAG image suitable for programming with the Gateworks JTAG dongle (for NAND flash boards) or to create removable storage devices (for NAND-less boards). See [wiki:linux/ubi linux/ubi].
    7  * For Ventana click [wiki:/ventana/bootloader#BuildingfromSource here] for instructions on building custom a U-boot environment. 
    8 
    9 If however you wish to customize a board's configuration in some way that you have not configured into the build system you will want to boot a board, make your customizations, then pull those customizations off and use them to provision further boards. This is what is presented in detail on this page.
    10 
    11 
    12 == SPI / NOR FLASH based boards (Laguna) ==
    13 Products that use NOR and/or SPI flash have the ability to be uploaded to a host PC via the Gateworks GW16042 JTAG dongle and jtag_usb application.
    14 
    15 For these products simply configuring a board the way you want it at runtime then uploading the flash to a file provides you with a firmware image that can then be programmed onto other boards.
    16 
    17 Please read more about JTAG upload here: [wiki:jtag_instructions JTAG Instructions]
    18 
     4We refer to the act of installing a firmware image across multiple boards as **Provisioning**.
     5
     6Provisioning always starts with creating an image from a build system and this varies greatly depending on your target system. Ultimately you end up building binary images that need to be programmed to the boot device at various locations. For example, boot firmware, firmware blobs, filesystem data etc.
     7
     8You could stitch all these various images together into a 'disk image' or 'device image' and flash it as a single object however doing so on a particularly large flash device, say an 8GiB eMMC can take a long time depending on how much data you need to write.
     9
     10== FLASH programming methods and performance
     11The speed at which you can physically write to a FLASH device depends on the method you are using to transfer and write the data as well as the performance of the SoC's access to the device.
     12
     13You have a few options available to write to the FLASH boot device:
     14 - Gateworks JTAG programmer (extremely low transfer data rate)
     15 - U-Boot (much higher data rate depending on driver capability in U-Boot)
     16 - Linux (best data rate)
     17
     18Both the transfer and write data rate are important. JTAG offers an extremely low transfer data rate but is an extremely useful method for getting <32MiB boot firmware and/or provisioning firmware on a board in an extremely reliable fashion. The board does not need to have functioning or compatible firmware to begin with.
     19
     20Note that the performance varies greatly across product family as well. The Venice IMX8MM product family for example has much higher MMC write performance than the older Newport CN803X product family. These considerations as well as your image contents and size need to be taken into account when you decide upon your provisioning method.
     21
     22U-Boot offers ok transfer rates and write performance (depending on driver support in U-Boot and noting that U-Boot is not an OS) but can't be used if the firmware on the board if corrupt, blank, or if the tools and features you need for provisioning do not exist in U-Boot. Note that because Gateworks ships all boards with U-Boot and a Linux based OS you can be guaranteed to boot to a U-Boot prompt with a reliable set of tools and features.
     23
     24Linux offers the best transfer rate and write performance with the notion that it is an interrupt capable full operating system but also requires that you are able to boot to Linux or a Linux that has the tools and features needed for your provisioning needs. Note that Because Gateworks ships all boards with U-Boot you at least know you can start at that point to load a version of Linux supporting whatever provisioning tools you need even if the default Linux OS may not have what you need.
     25
     26Summary of Pros and Cons of FLASH programming methods:
     27- JTAG:
     28 - Pro: no dependence on booting firmware already on the boot device (FLASH can be corrupt/blank)
     29 - Con: slow transfer rate
     30- U-Boot:
     31 - Pro: minimal dependence on boot firmware
     32 - Cons:
     33  - must have U-Boot pre-programmed on board
     34  - performance maybe sub-par compared to Linux
     35  - possible unfamiliarity of U-Boot commands and features
     36  - possible lack of features and flexibility
     37- Linux:
     38 - Cons:
     39  - more dependence and work in setting up a provisioning system
     40 - Pros:
     41  - best in class transfer and write performance
     42  - vast variety and familiarity of tools
     43
     44
     45== Using compressed disk images
     46While Gateworks often distributes single binary compressed firmware images for options such as OpenWrt and Ubuntu compatible images, we take care to keep these images small so that the time to flash them is minimal in order to be able to easily flash them with U-Boot. This very well may not be obtainable in your situation especially if you end up wanting multiple partitions with large spaces between them.
     47
     48The compressed disk image method involves:
     49- creating an uncompressed image that includes things like your boot firmware, partition table, and partition images
     50- stitching all these together putting each binary image at its correct offset using a tool like {{{dd}}}
     51- compressing this with gzip
     52- optionally growing your partition and filesystem to fit the extents of the device at runtime on the first boot
     53
     54Pros of compressed disk image method:
     55- single file compressed image distribution
     56- easy to install via U-Boot (obtaining the image via tftp or removable storage)
     57
     58Cons of compressed disk image method:
     59- the compressed image needs to fit into available RAM (or a complicated method of splitting it be used)
     60- can be slow if your uncompressed image is large (ie over 100's MiB's)
     61- may require resizing the partition(s) and filesystem(s) per your needs
     62
     63Note that in order to keep the time required to install the Gateworks Ubuntu rootfs images for example, we create a filesystem just large enough to fit the rootfs meaning (appx 1.6GiB) meaning that is all we have to write. Upon first boot a script runs that resizes the rootfs partition and filesystem to fit the extents of the device. This not only minimizes the amount of data that must be written but allows the disk image to fit into large devices and provide the user with expansion space without any further actions needed.
     64
     65Some performance examples:
     66- Gateworks Venice GW7301-01 IMX8MM installing focal-venice.img.gz
     67 - Note that IMX8MM supports HS400 eMMC data rates
     68 - U-Boot:
     69  - 14 seconds to transfer 460MiB compressed image via tftp to RAM at GbE
     70  - 45 seconds to uncompress and write the 1.6GiB of data to eMMC at HS400 speeds
     71- Gateworks Newport GW6404 CN8031 installing focal-newport.img.gz
     72 - Note that CN803x supports a max of 50MHz MMC data rates
     73 - U-Boot:
     74  - 58 seconds to transfer 460MiB compressed image via tftp to RAM at GbE
     75  - 2 minutes to uncompress and write the 1.6GiB of data to eMMC at 50Mhz speeds
     76
     77To present an example of where this method might not scale well consider a scenario where you multiple partitions defined across an 8GiB device. Using this method you would need to write almost the entire 8GiB which would take appx 10 minutes using this method. If your board has a 64GiB eMMC device you can see how this can get out of hand fairly quickly.
     78
     79
     80== Hybrid approaches
     81There are hybrid approaches you may consider as well. You could choose to create a JTAG'able image that contained custom firmware that automatically boots into a provisioning system that uses Linux for example. This JTAG'able image could be small enough to flash quickly over JTAG and you could even elect to have Gateworks pre-program your image on your boards if you have a Gateworks special or custom build. In this situation for example your image could pull a script from the network to complete your provisioning so that the image programmed on your boards can stay consistent but provide you flexibility in modifying the instructions for provisioning later on. This is the most flexible approach.
     82
     83Another example of a hyrid approach is to use U-Boot to provision images based on a script of U-Boot commands such that you can program various portions of your disk image at a time. For example if you had a scenario where you have multiple filesystems spread across a large eMMC device you could fetch and program the boot-firmware, partition table, and each filesystem independently keeping the data actually written to your FLASH device as small as possible. While this approach could be done in U-Boot alone, there are likely still multiple advantages to booting a Linux ramdisk image to provide greater flexibility and/or familiarity with tools.
     84
     85
     86== Provisioning from a Linux ramdisk environment
     87By far the most flexible solution is being able to provision your boards live with a Linux environment booting to a ramdisk providing all the tools you need for your custom environment.
     88
     89The most successful provisioning environments Gateworks has seen in the past are those that allow you to control your provisioning environment yourself in-dependent of what firmware may be pre-installed on your boards. While having a Gateworks special or Gateworks custom board allows you to dictate what firmware you want programmed on your boards by keeping that firmware as simple as possible you have an advantage of either not needing custom firmware or not having to change that firmware as new provisioning needs arise. For example,
     90
     91While this is the most flexible it is also the most complicate to setup.
     92
     93A very easy way to build a minimal ramdisk Linux based OS is to use buildroot. All you need is basic kernel support for the target board and a minimal set of tools:
     94- minimal root filesystem (only tools you need for your provisioning needs)
     95- minimal kernel config (only drivers/features you need for your
     96provisioning needs)
     97- ramdisk (build artifact will be a 'Image' which is a kernel + ramdisk)
     98- script that performs the provisioning
     99- init config that runs the script on boot (if you wanted it automated)
     100
     101The tools you need in your minimal rootfs:
     102- partition editor (sgdisk and/or parted)
     103- filesystem creation tools (ie e2fsprogs if using ext2/3/4)
     104- u-boot-tools (fw_printenv/fw_setenv) if setting or altering uboot env
     105- networking support to fetch your filesystem tarballs (which could also come from removable storage)
     106- optionally your provisioning script (or you can transfer this via network)
     107
     108The pre-built buildroot kernel+ramdisk minimal images that Gateworks provides at http://dev.gateworks.com/buildroot/ can be used for such provisioning. The instructions on how to build such an image can be found on the [wiki:#buildroot] page.
     109
     110When performing this provisioning, just like in creating a disk image, you need to understand what partitions may be needed by boot firmware (for example Newport expects a FATFS partition that exists within its boot firmware) and what Linux device you want to provision (ie eMMC). When we create disk partitions below we also take care to create reserved partitions to cover the boot firmware. It is also important to understand what, if any, bootscript you want to install and where.
     111
     112Examples:
     113 * provision a Venice board eMMC with a single ext4 root filesystem from a rootfs and kernel tarball:
     114  1. Load and boot kernel+ramdisk buildroot image from TFTP server
     115{{{#!bash
     116tftpboot $kernel_addr_r Image && booti $kernel_addr_r - $fdtcontroladdr
     117}}}
     118  2. Partition /dev/mmcblk0 (emmc)
     119{{{#!bash
     120DEV=/dev/mmcblk0
     121GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
     122GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908
     123GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4
     124# use sgdisk to create partitions
     125sgdisk -og $DEV
     126sgdisk -n 1:66:32767 -c 1:"boot" -t 1:$GUID_RESERVED $DEV
     127sgdisk -n 2:32768:0 -c 2:"root" -t 2:$GUID_LINUXFS $DEV
     128# set the legacy 'boot' attribute on rootfs partition
     129sgdisk -A 2:set:2 $DEV
     130# print partition table
     131sgdisk -p $DEV
     132}}}
     133   - Note the partitioning scheme chosen above creates protective partitions around the boot firmware which includes the SPL, ATF, and U-Boot (plus it's environment). This is both to protect any other partition tools from thinking there is free space there as well as provide you a partition to be able to easily update those portions of the boot firmware at a later date
     134   - Note we set the rootfs partition bit 2 attribute to mark the partition as BIOS bootable in case you are using the U-Boot generic distro config to look for bootscripts on bootable partitions
     135   - Note the 'end sector' for the rootfs partition above is 0 meaning it will size to the end of the device
     136  3. create and populate the rootfs partition
     137{{{#!bash
     138# fetch tar archives from network
     139udhcpc -i eth0 # bring up networking
     140cd /tmp
     141wget http://tharvey/tftpboot/focal-venice.tar.xz
     142wget http://tharvey/tftpboot/linux-venice.tar.xz
     143
     144wget http://dev.gateworks.com/ubuntu/focal/focal-venice.tar.xz
     145wget http://dev.gateworks.com/venice/kernel/linux-venice.tar.xz
     146# create and populate rootfs (P2) from tarballs
     147PART=${DEV}p2
     148mkfs.ext4 -q -F -L rootfs $PART
     149mount $PART /mnt
     150tar -C /mnt -xf focal-venice.tar.xz --keep-directory-symlink
     151tar -C /mnt -xf linux-venice.tar.xz --keep-directory-symlink
     152umount /mnt
     153}}}
     154  4. create a bootscript if desired (note this requires understanding of how U-Boot generic distro bootconfig works; you could alternately simply put whatever you need directly into U-boot env below)
     155{{{#!bash
     156cat <<\EOF > /tmp/bootscript
     157echo "Venice Boot Script"
     158# determine root device using uuid
     159part uuid ${devtype} ${devnum}:${distro_bootpart} uuid
     160# bootargs
     161setenv bootargs console=$console root=PARTUUID=${uuid} rootwait $bootargs
     162# load and boot kernel
     163load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${prefix}Image &&
     164booti ${kernel_addr_r} - ${fdtcontroladdr}
     165EOF
     166mount $PART /mnt
     167mkimage -A arm64 -T script -C none -d /tmp/bootscript /mnt/boot/boot.scr
     168umount /mnt
     169}}}
     170  5. customize U-Boot env if desired (note this requires understanding where the U-Boot env is located on FLASH)
     171{{{#!bash
     172# customize U-Boot env
     173cat << EOF > /tmp/fw_env.config
     174# Device        Device  offset
     175$DEV    0xff0000        0x8000
     176$DEV    0xff8000        0x8000
     177EOF
     178# get current config
     179fw_printenv --config /tmp/fw_env.config > /tmp/uboot.env
     180# append any desired config changes to /tmp/uboot.env
     181echo "bootdelay=1" >> /tmp/uboot.env
     182# write new config (perform twice so redundant env gets created as well)
     183fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
     184fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
     185}}}
     186
     187
     188 * provision a Newport board eMMC with a single ext4 root filesystem from a rootfs and kernel tarball:
     189  1. Load and boot kernel+ramdisk buildroot image from TFTP server
     190{{{#!bash
     191tftpboot $kernel_addr_r Image && booti $kernel_addr_r - $fdtcontroladdr
     192}}}
     193  2. Partition /dev/mmcblk0 (emmc)
     194{{{#!bash
     195DEV=/dev/mmcblk0
     196GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
     197GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908
     198GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4
     199# first use parted to create a GPT as sgdisk does not like the default MBR based partition table
     200parted --script $DEV mklabel gpt
     201# use sgdisk to create partitions
     202sgdisk -og $DEV
     203sgdisk -n 1:2048:26623 -c 1:"fatfs" -t 1:$GUID_BASIC_DATA $DEV
     204sgdisk -n 2:28672:30719 -c 2:"atf" -t 2:$GUID_RESERVED $DEV
     205sgdisk -n 3:30720:32767 -c 3:"uboot" -t 3:$GUID_RESERVED $DEV
     206sgdisk -n 4:32768:0 -c 4:"root" -t 4:$GUID_LINUXFS $DEV
     207# set the legacy 'boot' attribute on rootfs partition
     208sgdisk -A 4:set:2 $DEV
     209# print partition table
     210sgdisk -p $DEV
     211}}}
     212   - Note the partitioning scheme chosen above creates protective partitions around various portions of the boot firmware, namely the BDK bootstub, the fatfs filesystem, the ATF, and U-Boot (plus it's environment). This is both to protect any other partition tools from thinking there is free space there as well as provide you a partition to be able to easily update those portions of the boot firmware at a later date
     213   - Note we set the rootfs partition's bit 2 attribute to mark the partition as BIOS bootable in case you are using the U-Boot generic distro config to look for bootscripts on bootable partitions
     214   - Note the 'end sector' for the rootfs partition above is 0 meaning it will size to the end of the device
     215  3. create and populate the rootfs partition
     216{{{#!bash
     217# fetch tar archives from network
     218udhcpc -i eth0 # bring up networking
     219cd /tmp
     220wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz
     221wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz
     222# create and populate rootfs (P4) from tarballs
     223PART=${DEV}p4
     224mkfs.ext4 -q -F -L rootfs $PART
     225mount $PART /mnt
     226tar -C /mnt -xf focal-newport.tar.xz --keep-directory-symlink
     227tar -C /mnt -xf linux-newport.tar.xz --keep-directory-symlink
     228umount /mnt
     229}}}
     230  4. create a bootscript if desired (note this requires understanding of how U-Boot generic distro bootconfig works; you could alternately simply put whatever you need directly into U-boot env below)
     231{{{#!bash
     232cat <<\EOF > /tmp/bootscript
     233echo "Newport Boot Script"
     234setenv bootargs ${bootargs} root=/dev/mmcblk${devnum}p${distro_bootpart} rootwait
     235# disable USB autosuspend (CN81xx errata)
     236setenv bootargs ${bootargs} usbcore.autosuspend=-1
     237# disable KPTI (expected chip errata)
     238setenv bootargs ${bootargs} kpti=0
     239# add console
     240setenv bootargs ${bootargs} console=${console}
     241# load and boot kernel
     242echo "Loading kernel Image from ${devtype} ${devnum}:${distro_bootpart} ${prefix}"
     243load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${prefix}Image
     244booti ${kernel_addr_r} - ${fdtcontroladdr}
     245EOF
     246mount $PART /mnt
     247mkimage -A arm64 -T script -C none -d /tmp/bootscript /mnt/boot/newport.scr
     248umount /mnt
     249}}}
     250  5. customize U-Boot env if desired (note this requires understanding where the U-Boot env is located on FLASH)
     251{{{#!bash
     252# customize U-Boot env
     253cat << EOF > /tmp/fw_env.config
     254# Device        Device  offset
     255$DEV    0xff0000        0x8000
     256$DEV    0xff8000        0x8000
     257EOF
     258# get current config
     259fw_printenv --config /tmp/fw_env.config > /tmp/uboot.env
     260# append any desired config changes to /tmp/uboot.env
     261echo "bootdelay=1" >> /tmp/uboot.env
     262# write new config (perform twice so redundant env gets created as well)
     263fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
     264fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
     265}}}
    19266
    20267== Ventana NAND flash based boards ==
     
    133380Notes:
    134381 * you can always elect to build your own bootloader with a custom config rather than pulling the env data off a board
     382
     383== SPI / NOR FLASH based boards (Laguna) ==
     384Products that use NOR and/or SPI flash have the ability to be uploaded to a host PC via the Gateworks GW16042 JTAG dongle and jtag_usb application.
     385
     386For these products simply configuring a board the way you want it at runtime then uploading the flash to a file provides you with a firmware image that can then be programmed onto other boards.
     387
     388Please read more about JTAG upload here: [wiki:jtag_instructions JTAG Instructions]
    135389
    136390