[[PageOutline]]

= Provisioning boards =
We refer to the act of installing a firmware image across multiple boards as **Provisioning**.

Provisioning always starts with creating an image from a build system and this varies greatly depending on your target system. Ultimately you end up building binary images that need to be programmed to the boot device at various locations. For example, boot firmware, firmware blobs, filesystem data etc.

You could stitch all these various images together into a 'disk image' or 'device image' and flash it as a single object however doing so on a particularly large flash device, say an 8GiB eMMC can take a long time depending on how much data you need to write.

== FLASH programming methods and performance
The speed at which you can physically write to a FLASH device depends on the method you are using to transfer and write the data as well as the performance of the SoC's access to the device.

You have a few options available to write to the FLASH boot device:
 - Gateworks JTAG programmer (extremely low transfer data rate)
 - U-Boot (much higher data rate depending on driver capability in U-Boot)
 - Linux (best data rate)

Both the transfer and write data rate are important. JTAG offers an extremely low transfer data rate but is an extremely useful method for getting <32MiB boot firmware and/or provisioning firmware on a board in an extremely reliable fashion. The board does not need to have functioning or compatible firmware to begin with.

Note that the performance varies greatly across product family as well. The Venice IMX8MM product family for example has much higher MMC write performance than the older Newport CN803X product family. These considerations as well as your image contents and size need to be taken into account when you decide upon your provisioning method.

U-Boot offers ok transfer rates and write performance (depending on driver support in U-Boot and noting that U-Boot is not an OS) but can't be used if the firmware on the board if corrupt, blank, or if the tools and features you need for provisioning do not exist in U-Boot. Note that because Gateworks ships all boards with U-Boot and a Linux based OS you can be guaranteed to boot to a U-Boot prompt with a reliable set of tools and features.

Linux offers the best transfer rate and write performance with the notion that it is an interrupt capable full operating system but also requires that you are able to boot to Linux or a Linux that has the tools and features needed for your provisioning needs. Note that Because Gateworks ships all boards with U-Boot you at least know you can start at that point to load a version of Linux supporting whatever provisioning tools you need even if the default Linux OS may not have what you need.

Summary of Pros and Cons of FLASH programming methods:
- JTAG:
 - Pro: no dependence on booting firmware already on the boot device (FLASH can be corrupt/blank)
 - Con: slow transfer rate
- U-Boot:
 - Pro: minimal dependence on boot firmware
 - Cons:
  - must have U-Boot pre-programmed on board
  - performance maybe sub-par compared to Linux
  - possible unfamiliarity of U-Boot commands and features
  - possible lack of features and flexibility
- Linux:
 - Cons:
  - more dependence and work in setting up a provisioning system
 - Pros:
  - best in class transfer and write performance
  - vast variety and familiarity of tools


== Using compressed disk images
While Gateworks often distributes single binary compressed firmware images for options such as OpenWrt and Ubuntu compatible images, we take care to keep these images small so that the time to flash them is minimal in order to be able to easily flash them with U-Boot. This very well may not be obtainable in your situation especially if you end up wanting multiple partitions with large spaces between them.

The compressed disk image method involves:
- creating an uncompressed image that includes things like your boot firmware, partition table, and partition images
- stitching all these together putting each binary image at its correct offset using a tool like {{{dd}}}
- compressing this with gzip
- optionally growing your partition and filesystem to fit the extents of the device at runtime on the first boot

Pros of compressed disk image method:
- single file compressed image distribution
- easy to install via U-Boot (obtaining the image via tftp or removable storage)

Cons of compressed disk image method:
- the compressed image needs to fit into available RAM (or a complicated method of splitting it be used)
- can be slow if your uncompressed image is large (ie over 100's MiB's)
- may require resizing the partition(s) and filesystem(s) per your needs

Note that in order to keep the time required to install the Gateworks Ubuntu rootfs images for example, we create a filesystem just large enough to fit the rootfs meaning (appx 1.6GiB) meaning that is all we have to write. Upon first boot a script runs that resizes the rootfs partition and filesystem to fit the extents of the device. This not only minimizes the amount of data that must be written but allows the disk image to fit into large devices and provide the user with expansion space without any further actions needed.

Some performance examples:
- Gateworks Venice GW7301-01 IMX8MM installing focal-venice.img.gz
 - Note that IMX8MM supports HS400 eMMC data rates
 - U-Boot:
  - 14 seconds to transfer 460MiB compressed image via tftp to RAM at GbE
  - 45 seconds to uncompress and write the 1.6GiB of data to eMMC at HS400 speeds
- Gateworks Newport GW6404 CN8031 installing focal-newport.img.gz
 - Note that CN803x supports a max of 50MHz MMC data rates
 - U-Boot:
  - 58 seconds to transfer 460MiB compressed image via tftp to RAM at GbE
  - 2 minutes to uncompress and write the 1.6GiB of data to eMMC at 50Mhz speeds

To present an example of where this method might not scale well consider a scenario where you multiple partitions defined across an 8GiB device. Using this method you would need to write almost the entire 8GiB which would take appx 10 minutes using this method. If your board has a 64GiB eMMC device you can see how this can get out of hand fairly quickly.


== Hybrid approaches
There are hybrid approaches you may consider as well. You could choose to create a JTAG'able image that contained custom firmware that automatically boots into a provisioning system that uses Linux for example. This JTAG'able image could be small enough to flash quickly over JTAG and you could even elect to have Gateworks pre-program your image on your boards if you have a Gateworks special or custom build. In this situation for example your image could pull a script from the network to complete your provisioning so that the image programmed on your boards can stay consistent but provide you flexibility in modifying the instructions for provisioning later on. This is the most flexible approach.

Another example of a hyrid approach is to use U-Boot to provision images based on a script of U-Boot commands such that you can program various portions of your disk image at a time. For example if you had a scenario where you have multiple filesystems spread across a large eMMC device you could fetch and program the boot-firmware, partition table, and each filesystem independently keeping the data actually written to your FLASH device as small as possible. While this approach could be done in U-Boot alone, there are likely still multiple advantages to booting a Linux ramdisk image to provide greater flexibility and/or familiarity with tools.


== Provisioning from a Linux ramdisk environment
By far the most flexible solution is being able to provision your boards live with a Linux environment booting to a ramdisk providing all the tools you need for your custom environment.

The most successful provisioning environments Gateworks has seen in the past are those that allow you to control your provisioning environment yourself in-dependent of what firmware may be pre-installed on your boards. While having a Gateworks special or Gateworks custom board allows you to dictate what firmware you want programmed on your boards by keeping that firmware as simple as possible you have an advantage of either not needing custom firmware or not having to change that firmware as new provisioning needs arise. For example, 

While this is the most flexible it is also the most complicate to setup.

A very easy way to build a minimal ramdisk Linux based OS is to use buildroot. All you need is basic kernel support for the target board and a minimal set of tools:
- minimal root filesystem (only tools you need for your provisioning needs)
- minimal kernel config (only drivers/features you need for your
provisioning needs)
- ramdisk (build artifact will be a 'Image' which is a kernel + ramdisk)
- script that performs the provisioning
- init config that runs the script on boot (if you wanted it automated)

The tools you need in your minimal rootfs:
- partition editor (sgdisk and/or parted)
- filesystem creation tools (ie e2fsprogs if using ext2/3/4)
- u-boot-tools (fw_printenv/fw_setenv/mkenvimage) if setting or altering uboot env
- networking support to fetch your filesystem tarballs (which could also come from removable storage)
- optionally your provisioning script (or you can transfer this via network)

The pre-built buildroot kernel+ramdisk minimal images that Gateworks provides at http://dev.gateworks.com/buildroot/ can be used for such provisioning. The instructions on how to build such an image can be found on the [wiki:#buildroot] page.

When performing this provisioning, just like in creating a disk image, you need to understand what partitions may be needed by boot firmware (for example Newport expects a FATFS partition that exists within its boot firmware) and what Linux device you want to provision (ie eMMC). When we create disk partitions below we also take care to create reserved partitions to cover the boot firmware. It is also important to understand what, if any, bootscript you want to install and where.

Examples:
 * provision a Venice board eMMC with a single ext4 root filesystem from a rootfs and kernel tarball:
  1. Load and boot kernel+ramdisk buildroot image from TFTP server
{{{#!bash
tftpboot $kernel_addr_r Image && booti $kernel_addr_r - $fdtcontroladdr
}}}
  2. Partition /dev/mmcblk0 (emmc)
{{{#!bash
DEV=/dev/mmcblk0
GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908
GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4
# use sgdisk to create partitions
sgdisk -og $DEV
sgdisk -n 1:66:32767 -c 1:"boot" -t 1:$GUID_RESERVED $DEV
sgdisk -n 2:32768:0 -c 2:"root" -t 2:$GUID_LINUXFS $DEV
# set the legacy 'boot' attribute on rootfs partition
sgdisk -A 2:set:2 $DEV
# print partition table
sgdisk -p $DEV
}}}
   - Note the partitioning scheme chosen above creates protective partitions around the boot firmware which includes the SPL, ATF, and U-Boot (plus it's environment). This is both to protect any other partition tools from thinking there is free space there as well as provide you a partition to be able to easily update those portions of the boot firmware at a later date
   - Note we set the rootfs partition bit 2 attribute to mark the partition as BIOS bootable in case you are using the U-Boot generic distro config to look for bootscripts on bootable partitions
   - Note the 'end sector' for the rootfs partition above is 0 meaning it will size to the end of the device
  3. create and populate the rootfs partition
{{{#!bash
# fetch tar archives from network
udhcpc -i eth0 # bring up networking
cd /tmp
wget http://tharvey/tftpboot/focal-venice.tar.xz
wget http://tharvey/tftpboot/linux-venice.tar.xz

wget http://dev.gateworks.com/ubuntu/focal/focal-venice.tar.xz
wget http://dev.gateworks.com/venice/kernel/linux-venice.tar.xz
# create and populate rootfs (P2) from tarballs
PART=${DEV}p2
mkfs.ext4 -q -F -L rootfs $PART
mount $PART /mnt
tar -C /mnt -xf focal-venice.tar.xz --keep-directory-symlink
tar -C /mnt -xf linux-venice.tar.xz --keep-directory-symlink
umount /mnt
}}}
  4. create a bootscript if desired (note this requires understanding of how U-Boot generic distro bootconfig works; you could alternately simply put whatever you need directly into U-boot env below)
{{{#!bash
cat <<\EOF > /tmp/bootscript
echo "Venice Boot Script"
# determine root device using uuid
part uuid ${devtype} ${devnum}:${distro_bootpart} uuid
# bootargs
setenv bootargs console=$console root=PARTUUID=${uuid} rootwait $bootargs
# load and boot kernel
load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${prefix}Image &&
booti ${kernel_addr_r} - ${fdtcontroladdr}
EOF
mount $PART /mnt
mkimage -A arm64 -T script -C none -d /tmp/bootscript /mnt/boot/boot.scr
umount /mnt
}}}
  5. customize U-Boot env if desired (note this requires understanding where the U-Boot env is located on FLASH)
{{{#!bash
# customize U-Boot env
cat << EOF > /tmp/fw_env.config
# Device        Device  offset
$DEV    0xff0000        0x8000
$DEV    0xff8000        0x8000
EOF
# get current config
fw_printenv --config /tmp/fw_env.config > /tmp/uboot.env
# append any desired config changes to /tmp/uboot.env
echo "bootdelay=1" >> /tmp/uboot.env
# write new config (perform twice so redundant env gets created as well)
fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
}}}


 * provision a Newport board eMMC with a single ext4 root filesystem from a rootfs and kernel tarball:
  1. Load and boot kernel+ramdisk buildroot image from TFTP server
{{{#!bash
tftpboot $kernel_addr_r Image && booti $kernel_addr_r - $fdtcontroladdr
}}}
  2. Partition /dev/mmcblk0 (emmc)
{{{#!bash
DEV=/dev/mmcblk0
GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908
GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4
# first use parted to create a GPT as sgdisk does not like the default MBR based partition table
parted --script $DEV mklabel gpt
# use sgdisk to create partitions
sgdisk -og $DEV
sgdisk -n 1:2048:26623 -c 1:"fatfs" -t 1:$GUID_BASIC_DATA $DEV
sgdisk -n 2:28672:30719 -c 2:"atf" -t 2:$GUID_RESERVED $DEV
sgdisk -n 3:30720:32767 -c 3:"uboot" -t 3:$GUID_RESERVED $DEV
sgdisk -n 4:32768:0 -c 4:"root" -t 4:$GUID_LINUXFS $DEV
# set the legacy 'boot' attribute on rootfs partition
sgdisk -A 4:set:2 $DEV
# print partition table
sgdisk -p $DEV
}}}
   - Note the partitioning scheme chosen above creates protective partitions around various portions of the boot firmware, namely the BDK bootstub, the fatfs filesystem, the ATF, and U-Boot (plus it's environment). This is both to protect any other partition tools from thinking there is free space there as well as provide you a partition to be able to easily update those portions of the boot firmware at a later date
   - Note we set the rootfs partition's bit 2 attribute to mark the partition as BIOS bootable in case you are using the U-Boot generic distro config to look for bootscripts on bootable partitions
   - Note the 'end sector' for the rootfs partition above is 0 meaning it will size to the end of the device
  3. create and populate the rootfs partition
{{{#!bash
# fetch tar archives from network
udhcpc -i eth0 # bring up networking
cd /tmp
wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz
wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz
# create and populate rootfs (P4) from tarballs
PART=${DEV}p4
mkfs.ext4 -q -F -L rootfs $PART
mount $PART /mnt
tar -C /mnt -xf focal-newport.tar.xz --keep-directory-symlink
tar -C /mnt -xf linux-newport.tar.xz --keep-directory-symlink
umount /mnt
}}}
  4. create a bootscript if desired (note this requires understanding of how U-Boot generic distro bootconfig works; you could alternately simply put whatever you need directly into U-boot env below)
{{{#!bash
cat <<\EOF > /tmp/bootscript
echo "Newport Boot Script"
setenv bootargs ${bootargs} root=/dev/mmcblk${devnum}p${distro_bootpart} rootwait
# disable USB autosuspend (CN81xx errata)
setenv bootargs ${bootargs} usbcore.autosuspend=-1
# disable KPTI (expected chip errata)
setenv bootargs ${bootargs} kpti=0
# add console
setenv bootargs ${bootargs} console=${console}
# load and boot kernel
echo "Loading kernel Image from ${devtype} ${devnum}:${distro_bootpart} ${prefix}"
load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} ${prefix}Image
booti ${kernel_addr_r} - ${fdtcontroladdr}
EOF
mount $PART /mnt
mkimage -A arm64 -T script -C none -d /tmp/bootscript /mnt/boot/newport.scr
umount /mnt
}}}
  5. customize U-Boot env if desired (note this requires understanding where the U-Boot env is located on FLASH)
{{{#!bash
# customize U-Boot env
cat << EOF > /tmp/fw_env.config
# Device        Device  offset
$DEV    0xff0000        0x8000
$DEV    0xff8000        0x8000
EOF
# get current config
fw_printenv --config /tmp/fw_env.config > /tmp/uboot.env
# append any desired config changes to /tmp/uboot.env
echo "bootdelay=1" >> /tmp/uboot.env
# write new config (perform twice so redundant env gets created as well)
fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
fw_setenv --config /tmp/fw_env.config --script /tmp/uboot.env
}}}

== Ventana NAND flash based boards ==
Products that use NAND flash present an issue in that they can contain bad blocks. As a result the raw flash devices can differ in size making it difficult to implement a JTAG flash upload/download scenario.

There are several ways of provisioning NAND bootable boards:
 * using JTAG (this is what Gateworks uses on our production line)
 * using U-Boot (more complex, much faster than JTAG, but does not allow provisioning the SPL)
 * using Linux (even more complex, much faster than JTAG, but allows provisioning the SPL)
 * a combination of the above

Regardless of the method used for provisioning there are several artifacts that you need in order to provision NAND:
 * SPL (secondary program loader)
 * u-boot.img (bootloader)
 * env (bootloader env)
 * ubi (unsorted block image containing ubifs filesystem)


=== Pulling Software off of an Existing Board ===

==== SPL and Bootloader ====
The SPL and u-boot.img are built artifacts (which can be downloaded from ​http://dev.gateworks.com/ventana/images).

==== Bootloader Environment ====
The env can be blank, which will use built-in defaults, or can be customized and extracted from the flash.

To create and extract a bootloader env:
 1. Create the env on a board:
{{{#!bash
# blank per-board vars (which are set from eeprom by default, yet overridable via env)
setenv fdt_file
setenv ethaddr
setenv eth1addr
# perform any other desired changes
# save
saveenv
}}}
 2. Extract the env from the board and save to removable storage:
  - from Linux
{{{#!bash
dd if=/dev/mtd1 of=env bs=1M
}}}
  - from U-Boot:
{{{#!bash
# read the env (environment) partition into temporary memory, note the size reported below as 0x100000

Ventana > nand read ${loadaddr} env

NAND read: device 0 offset 0x1000000, size 0x100000
 1048576 bytes read: OK
Ventana > 

# store it to file on micro-SD with an ext4 fs (size re-used from above)
mmc dev 0 && ext4write mmc 0:1 ${loadaddr} /env 0x100000

# or store it to file on USB mass storage with an ext4 fs
usb start && usb dev 0 && ext4write usb 0:1 ${loadaddr} /env 0x100000
}}}
  - Note that you may find it easier to build yourself a custom bootloader with defaults that match your needs rather than deal with extracting and imaging an env flash partition
  - Note that your ext4 filesystem must not have checksums enabled (metadata_csum, a feature added to newer e2fsprogs) as U-Boot does not support this in ext4write.



==== Root Filesystem ====
The ubi root filesystem is originally built by the build system of the specific BSP your using, however if you end up imaging this onto a board, and customizing it, you may be able to pull it back off as long as your flash size is far less than your memory:
 * From Linux assuming /tmp is a tmpfs (ram based) and that you are booted into the filesystem you are copying and you have more ram available than the size of /dev/mtd2 (such as a 256MB flash on a 512MB system)
{{{#!bash
dd if=/dev/mtd2 of=/tmp/ubi bs=4M
}}}
 * From U-Boot
{{{#!bash
# read the rootfs (filesystem) partition into temporary memory, note the size reported below as 0xef00000

Ventana > nand read ${loadaddr} rootfs

NAND read: device 0 offset 0x1100000, size 0xef00000
 250609664 bytes read: OK


# store it to file on micro-SD with an ext4 fs (size re-used from above)
Ventana > mmc dev 0 && ext4write mmc 0:1 ${loadaddr} /rootfs 0xef00000
switch to partitions #0, OK
mmc0 is current device
File System is consistent
update journal finished
250609664 bytes written in 57489 ms (4.2 MiB/s)
Ventana > 

# or store it to file on USB mass storage with an ext4 fs
usb start && usb dev 0 && ext4write usb 0:1 ${loadaddr} /rootfs 0xef00000
}}}
  - Note that your ext4 filesystem must not have checksums enabled (metadata_csum, a feature added to newer e2fsprogs) as U-Boot does not support this in ext4write.


=== Flashing Boards with Pulled Software ===
Once you have all the artifacts you can re-assemble them into a JTAG image suitable for the Gateworks JTAG adapter and software. The following usage of mkimage_jtag will create a jtagable image matching the partitioning described by 'mtdparts=nand:16m(uboot),1m(env),-(rootfs)'
{{{#!bash
mkimage_jtag -e SPL@0 u-boot.img@14M env@16M ubi@17M > image.bin
}}}

Or, for a faster two-step method of imaging using U-Boot with serial and ethernet (to a tftp server with the ubi):
 1. create a JTAG image of the SPL + bootloader + env:
{{{#!bash
mkimage_jtag SPL u-boot.img env > image.bin
}}}
 2. once the above is flashed with the Gateworks JTAG adapter and software you can flash the ubi (much more quickly than via JTAG) within U-Boot
 3. break out into the bootloader, transfer the ubi image from a tftp server into SDRAM, and flash it:
{{{#!bash
setenv ipaddr 192.168.1.1 # local ip
setenv serverip 192.168.1.146 # server ip
tftp ${loadaddr} image.ubi # tftp ubi image
nand erase.part rootfs # erase the nand partition named rootfs from the mdtparts variable
nand write ${loadaddr} rootfs ${filesize} # write the downloaded ubi to rootfs
}}}

Notes:
 * you can always elect to build your own bootloader with a custom config rather than pulling the env data off a board

== SPI / NOR FLASH based boards (Laguna) ==
Products that use NOR and/or SPI flash have the ability to be uploaded to a host PC via the Gateworks GW16042 JTAG dongle and jtag_usb application.

For these products simply configuring a board the way you want it at runtime then uploading the flash to a file provides you with a firmware image that can then be programmed onto other boards.

Please read more about JTAG upload here: [wiki:jtag_instructions JTAG Instructions]


[=#ubootenv]
== U-Boot environment provisioning
The U-Boot bootloader uses environment variables and scripts to control the booting of a board. Often it is desirable to override the default environment the Gateworks BSP's create and create your own.

The device, location, size, and redundancy details of how the U-Boot environment is stored vary from board to board based on U-Boot's configuration:
 * Venice: 32KiB (0x8000) redundant env stored at 16320KiB (0xff0000) offset for MMC
 * Newport: 32KiB (0x8000) redundant env stored at 16320KiB (0xff0000) offset for MMC
 * Ventana:
  - MMC: 128MiB redundant env stored at 709KiB offset
  - NAND: 128MiB redundant env stored at 16MiB offset

There are some tools from the U-Boot project available in the u-boot-tools package that can be used to alter or create an env that can then be copied directly to flash:
 * fw_setenv / fw_getenv - requires a config file and can get or alter the env. If there is no valid env where the config file points to a default env hard coded into the bootloader will be returned.
 * mkenvimage - create an env image from scratch with a specified size and redundancy.

For examples of how the default env is created for the various Gateworks products:
 - Venice BSP uses fw_setenv to alter the hard-coded default env. For details see:
  * https://github.com/Gateworks/bsp-venice/blob/master/Makefile
  * https://github.com/Gateworks/bsp-venice/blob/master/venice.env
  * https://github.com/Gateworks/bsp-venice/blob/master/fw_env.config
 - Newport BSP uses fw_setenv to alter the hard-coded default env. For details see:
  * https://github.com/Gateworks/bsp-newport/blob/sdk-10.1.1.0-newport/Makefile
  * https://github.com/Gateworks/bsp-newport/blob/sdk-10.1.1.0-newport/newport.env
  * https://github.com/Gateworks/bsp-newport/blob/sdk-10.1.1.0-newport/fw_env.config
 - Ventana firmware images have a blank env area therefore use hard-coded default env from U-Boot

To create a specific env from scratch (ie without inheriting any defaults) use the {{{mkenvimage}}} tool from the u-boot-tools package
 1. Create an env.txt file with 'name=value' pairs. Be sure to include everything that is necessary as no defaults will be available other than variables that are set at runtime on the fly such as serial#, model, and macaddr:
{{{#!bash
cat << EOF >> env.txt
baudrate=115200
bootcmd=run distro_boot
...
EOF
}}}
{{{#!bash
cat << EOF >> env.txt
baudrate=115200
loadaddr=0x48200000
ipaddr=192.168.1.1
serverip=192.168.1.146
update_all=tftpboot $loadaddr venice/venice-focal-imx8mm.img.gz && gzwrite mmc 2 $loadaddr $filesize
update_env=tftpboot $loadaddr venice/env.bin && mmc dev 2 && mmc write $loadaddr 0x7f80 $filesize
EOF
}}}
 2. Use {{{mkenvimage}}}  taking care to specify the env size the bootloader is configured to use and specify '-r' for redundancy if the bootloader is configured for that:
  * Venice / Newport
{{{#!bash
mkenvimage -r -s $((32 * 1024)) -o env.bin env.txt
}}}
  * Ventana:
{{{#!bash
mkenvimage -r -s $((128 * 1024)) -o env.bin env.txt
}}}
 3. Place this image in FLASH:
  * Venice / Newport:
   - Example: compressed disk image for Venice / Newport:
{{{#!bash
gunzip disk.img.gz
dd if=env.bin of=disk.img bs=1K seek=16320 oflag=notrunc
gzip disk.img
}}}
   - Example: update from Linux:
{{{#!bash
dd if=env.bin of=/dev/mmcblk0 bs=1K seek=16320 oflag=notrunc
}}} 
   - Example: update from U-Boot (16320KiB is equivalent to 0x7f80 512byte blocks in hex)
{{{#!bash
tftp $loadaddr env.bin && mmc write $loadaddr 0x7f80 $filesize
}}}
  * Ventana:
   - Example: JTAG binary image for a Ventana NAND device:
{{{#!bash
mkimage_jtag -e SPL@0 u-boot.img@14M env.bin@16M ubi@17M > image.bin
}}}
   - Example: update from Linux:
{{{#!bash
dd if=env.bin of=/dev/mmcblk0 bs=1K seek=16384 oflag=notrunc
}}} 
   - Example: update from U-Boot (16384MiB is equivalent to 0x8000 512byte blocks in hex)
{{{#!bash
tftp $loadaddr env.bin && mmc write $loadaddr 0x8000 $filesize
}}}


[=#microsd]
== micro-SD provisioning ==

=== Directly Cloning SD Cards ===
See [wiki:linux/blockdev#Creatingadiskimage linux/blockdev]

=== U-Boot MicroSD Provisioning ===
The main difference between provisioning removable storage devices such as micro-SD compared to non-removable storage devices (such as NAND flash) is that the removable devices can be potentially booted on a board with a different model, CPU, or memory configuration. This causes us to treat the U-Boot environment differently when we extract it from a configured board.

If you do not want a blank env (which uses built-in defaults) you must provision one board, boot it to the bootloader, customize your env, then extract that env to use when provisioning additional boards.

The Ventana bootloader stores its microSD env on raw block sectors from offset 709K and is 256K in size.

The env can be blank, which will use built-in defaults, or can be customized and extracted.

To create and extract a bootloader env: (only if this is being used, otherwise skip this step)
 1. Create the env on a board:
{{{#!bash
# blank per-board vars (which are set from eeprom by default, yet overridable via env)
setenv fdt_file
setenv ethaddr
setenv eth1addr
# perform any other desired changes
# save
saveenv
}}}
 2. extract the env from a board that boots to micro-SD:
  * from Linux:
{{{#!bash
# copy 256KB from offset 709KB to 'env' file
dd if=/dev/sdc of=env bs=1K skip=709 count=256 oflag=sync
}}}
  * from U-Boot:
{{{#!bash
mmc read ${loadaddr} 0x58a 0x200 # read 512x 512byte blocks (256K) from block 0x1418
# store it to file on micro-SD with an ext4 fs
ext4write mmc 0:1 ${loadaddr} /mmc.env 0x40000
# store it to file on USB mass storage with an ext4 fs
usb start && usb dev 0 && ext4write usb 0:1 ${loadaddr} /mmc.env 0x40000
}}}
  - Note that your ext4 filesystem must not have checksums enabled (metadata_csum, a feature added to newer e2fsprogs) as U-Boot does not support this in ext4write.


To place an extracted env onto a micro-SD:
 * from Linux:
{{{#!bash
# copy 256KB from file env to offset 709KB:
dd if=env of=/dev/sdc bs=1K seek=709 count=256 oflag=sync
}}}

= Provisioning root file system from live Newport board

It's always best to create your rootfs from scratch using the Newport [wiki:/newport/bsp BSP] when this option is available.  If you're prevented from doing so it is possible to provision the root file system from some Newport boards without too much difficulty.  Newport SBC's are capable of booting from both eMMC and MMC.  The boot device can be selected by the GSC performing a 5x press on the power button.  In the following section we will create a bootable microSD card, load it as the primary boot device, image partition 2 of the eMMC using "dd", then create a compressed disk image from this data.  

== Requirements

* A Newport board with MMC card slot and push button (GSC must be configured to accept push button input—this is default)
* A MMC with capacity enough to accommodate full size of eMMC (8GB) along with BSP for performing recovery.  Recommended would be a minimum 16GB or 32GB card.    
* A desktop computer with Linux natively installed. This workstation must have a drive capable of accepting an MMC and have packages for DD and Mount.  


== On your workstation

Download a Gateworks pre-built image.  All of our images include the tools necessary to perform this operation, though Ubuntu will likely be the easiest to work with.  
{{{
wget -N http://dev.gateworks.com/newport/images/focal-newport.img.gz
}}}

Insert a micro SD card into your workstation and identify the device name it's been assigned.  This can be done using a variety of methods, for example the "dmesg" output.  In this example the MMC is "/dev/sdb"

Image the MMC.
{{{
zcat focal-newport.img.gz | sudo dd of=/dev/sdb bs=4M
}}}

Remove the SD card from your workstation. 

== On your Newport SBC

Insert the SD card into the SD card reader, then apply power to the SBC (or otherwise turn it on).  5x press the power button, you will see the status LED turn off and back on.  The BDK will display microSD as MMC0.

Example output:

{{{
	Gateworks Newport SPL (12.7.0-96865d0 Tue Jul 7 21:19:32 UTC 2020)

	GSC     : v55 0xe7e2 RST:BOOT_WDT2 Thermal Protection Enabled
	Temp    : Board:34C/86C CPU:42C/100C
	Model   : GW6400-B1
	MFGDate : 09-23-2019
	Serial  : 802864
	RTC     : 8
	SoC     : CN8020-800BG676-SCP-P12-G 1024KB 800/550MHz 0xa2 Pass 1.2 
	MMC0    : microSD
	MMC1    : eMMC
}}} 

Alterntitively you can use a ramdisk (performance will be faster). [http://wiki:/provisioning#ProvisioningfromaLinuxramdiskenvironment Link to Ramdisk section]



Proceed with booting to Linux user-space.  Once there if you would like to view the rootfs you're about to image it's located at /dev/mmcblk1p2.  Doing so is optional. 
{{{
mount /dev/mmcblk1p2 /mnt

ls /mnt #this will display the eMMC's rootfs

umount /mnt
}}}

Image the partition to a file.
{{{
dd if=/dev/mmcblk1p2 of=myrootfs.img
}}}

"sync" the filesystem and power board off
{{{
sync
#remove power
}}}

Remove MMC from the SBC.  

== Create an image

Navigate to the "/tmp" directory on your workstation.  In actuality this can be any directory that you have read and write permissions, "/tmp" should only be used if you don't care what happens to these files later.    
{{{
cd /tmp
}}}

Insert the MMC into the SD card reader on your workstation, as before note the the name the device is assigned.  Because we created our image from within the rootfs we will need to mount the second partition, for example "/dev/sdb2":
{{{
sudo mount /dev/sdb2 /mnt/

ls /mnt/
}}}
Copy the image you created of mmcblk1p2 into "/tmp"
{{{
cp /mnt/myrootfs.img .
}}}
Executing the command "file" on this file will return that it is ext4 filesystem data and the volume name is "rootfs".

Example:
{{{
	user@workstation:/tmp$ file myrootfs.img 
	myrootfs.img: Linux rev 1.0 ext4 filesystem data, UUID=951f19a1-cc54-4a07-9b7f-41a54ea8acb4, volume name "rootfs" (needs journal recovery) (extents) (64bit) (large files) (huge files)
}}}

Add boot firmware and create a gzipped image.  
* Download boot firmware 
{{{
wget http://dev.gateworks.com/newport/boot_firmware/firmware-newport.img
}}}
* name it however you please
{{{
cp firmware-newport.img myimg-newport.img
}}}
* Create a disk image containing the root filesystem and boot firmware.  The boot firmware is 16M so we will "dd" the rootfs using this offset.  
{{{
dd if=myrootfs.img of=myimg-newport.img bs=16M seek=1
}}}
* gzip the image so it can be installed using standard methods on other Newport boards. 
{{{
gzip -k -f myimg-newport.img 
}}}
Be mindful that the image size can't exceed the total DRAM of the board you plan to install it on if you're using the bootloader command "tftpboot".  The "update_all", and "update_rootfs" scripts both use this command
{{{
ls -lh myimg-newport.img.gz #size must be less than total DRAM of board
}}}

== Flash this image to a Newport SBC:

In the bootloader:
{{{
GW6400-B1> setenv ipaddr 192.168.1.52
GW6400-B1> setenv serverip 192.168.1.56
GW6400-B1> setenv image myimg-newport.img.gz
GW6400-B1> setenv dev 0
GW6400-B1> run update_all 
}}}


= Provisioning Newport, direct write to eMMC on target

Usint the bootloader's "gzwrite" method for flashig embedded flash has a lot of downsides; the gzipped image must be smaller than the capacity of the ram and it's difficult to create a minimal filesystem to speed up the writes. Depending on your application a more flexible and potentially time efficient aproach would be to use 'linux kernel+ramdisk + provisioning' script approach.  Using this method may be neceessary if there are scripts or applications that must be run during the provisioning process. 

* [wiki:/buildroot#initrdinitialramdisk Buildroot Ramdisk Wiki]

1. Boot to Target - either from the alternate boot device (ie boot from microSD if working on eMMC or visa versa) or boot a kernel+ramdisk so that you can work off either device. We provide a pre-built ramdisk image with all the necessary tools.  A secondary boot media can be use although it will not be as fast as the ramdisk.
1. Partition the device with a partition size just large enough to fit a filesystem that can contain your files.  Determine the size in bytes needed for your filesystem, there are several ways to do this including simple trial and error.  If you have a directory with your files you can use 'du -b --summarize' to get bytes. For a tar.gz a good estimate is the uncompressed size "gzip -l <file.gz>".
{{{#!bash
# for an existing directory using du
SZ_B=$(du -b --summarize /usr/src/ubuntu-rootfs/focal-newport | awk '{
print $1; }'
# or just guess and use trial and error
SZ_B=$((1536*1024*1024)) # 1536MiB fits the Gateworks ubuntu image
}}}
1. Create partition table [wiki:/provisioning#ProvisioningfromaLinuxramdiskenvironment Example using GPT and sgdisk]. For Newport we need to preserve the FATFS filesystem used by the boot firmware.
{{{#!bash
DEV=/dev/mmcblk0
GUID_BASIC_DATA=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
GUID_RESERVED=8DA63339-0007-60C0-C436-083AC8230908
GUID_LINUXFS=0FC63DAF-8483-4772-8E79-3D69D8477DE4
# first use parted to create a GPT as sgdisk does not like the default
MBR based partition table
parted --script $DEV mklabel gpt
# use sgdisk to create partitions
sgdisk -og $DEV
sgdisk -n 1:2048:26623 -c 1:"fatfs" -t 1:$GUID_BASIC_DATA $DEV
sgdisk -n 2:28672:30719 -c 2:"atf" -t 2:$GUID_RESERVED $DEV
sgdisk -n 3:30720:32767 -c 3:"uboot" -t 3:$GUID_RESERVED $DEV
END=$(( 32768 + $((SZ_B/512)) ))
sgdisk -n 4:32768:$END -c 4:"root" -t 4:$GUID_LINUXFS $DEV
# set the legacy 'boot' attribute on rootfs partition so the uboot env
will boot from it
sgdisk -A 4:set:4 $DEV
# print partition table
sgdisk -p $DEV
}}}
1. Create your filesystem using the index of the partition for root as shown above. Note that rootfs is p4, this is because of the boot firmware (atf/uboot).  Format the partition that will contain the rootfs to ext4.
{{{#!bash
mkfs.ext4 -q -F -L rootfs ${DEV}p4
}}}
1. Mount your filesystem, populate it, then umount.
{{{#!bash
mount ${DEV}p4 /mnt
cd /tmp
udhcpc -e eth0
wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz
wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz
tar -C /mnt -xf focal-newport.tar.xz --keep-directory-symlink
2>/dev/null || echo Error!!!
tar -C /mnt -xf linux-newport.tar.xz --keep-directory-symlink
2>/dev/null || echo Error!!!
umount /mnt
#check for errors returned from tar such as not enough space
}}}
** Note: When creating tarballs use the --numeric-owner to preserve UID/GID numbers vs names and there is no need to be selective about files/directories on a non-live filesystem **
6. Now you can dd and decompress your image up to the last block in the partition. The last block in the partition is $END which you can see by printing the partition table:
{{{#!bash
rm /tmp/*
dd if=$DEV bs=512 count=$END | gzip > /tmp/disk.img.gz
}}}
7. Install a compressed disk image:
{{{#!bash
cd /tmp
wget disk.img.gz
zcat disk.img.gz | dd of=$DEV bs=4M
}}}

== Provisioning Newport removable block storage on Desktop workstation 

Provisioning on the target is less than ideal in situations where a traditional workstation can be used.  Both methods have their advantages and disadvantges.  When using a desktop an issue may arrise when using partitioning tools, as most of them do not like working on 'files' and won't let you do things like partition past the size of the file. In those cases there are, and we provide, tools that create partition tables for you without adhering to any rules. 

1. Create a 'file' for your filesystem image. 'truncate' can be used as it creates a 'sparse' file which reports the size it was created with but won't take up space for zero'd blocks.
{{{#!bash
truncate -s 1536M fs.img #create a sparse file 
mkfs.ext4 -q -F -L rootfs fs.img #format it.  
}}}
1. Mount your filesystem image and populate it:
{{{#!bash
sudo mount fs.img /mnt/disk
wget http://dev.gateworks.com/ubuntu/focal/focal-newport.tar.xz
wget http://dev.gateworks.com/newport/kernel/linux-newport.tar.xz
sudo tar -C /mnt/disk/ -xf focal-newport.tar.xz --keep-directory-symlink >/dev/null || echo "Error!!!"
sudo tar -C /mnt/disk/ -xf linux-newport.tar.xz --keep-directory-symlink >/dev/null || echo "Error!!!"
sudo umount /mnt/disk
}}}
1. Create a 'file' for your disk image. This needs to be large enough for your boot firmware (which includes the partition table) as well as the filesystems you will copy to it.
{{{#!bash
truncate -s $((16+1536))M disk.img
}}}
1. Copy the boot firmware to it at offset 0 
{{{#!bash
dd if=firmware-newport.img of=disk.img conv=notrunc #make sure to use conv=notrunc so it doesn't truncate your image file
}}}
1. Copy your filesystem image to the correct sectors dictated by the partition table. If needed adjust/re-create the partition table taking care to ensure the boot firmware FATFS does not move.
{{{#!bash
dd if=fs.img of=disk.img bs=512 seek=32768 conv=notrunc
}}}
1. Compress.
{{{#!bash
gzip disk.img
}}}

In conclusion, a partition is simply a chunk of space dictated by a header in the partition table which is in the first few blocks of the disk. It doesn't need to be filled, there can be a large partition with a small filesystem inside of it, that filesystem can be enlarged on first-boot with 'resize2fs'. Additionally the partition size can be incresead as well.