Newport CN80XX/CN81XX Boot Firmware

The 'Boot Firmware' for Newport is defined as the combination of the First level 'boot stub' and the additional firmware stages through the bootloader. This can be broken down into the following stages:

• Boot ROM (internal on CN80XX/CN81XX SoC): fetch first level boot stub (192KB limit) from boot device (MMC or SPI FLASH)
• SPL (Secondary Program Loader)
• Bootloader (U-Boot) - Newport UBoot Bootloader Wiki Page

For a Secondary Program Loader, or SPL, Gateworks currently uses the Cavium Board Development Kit (BDK) provided by their OCTEON-TX Software Development Kit (SDK). In this current implementation the Boot ROM loads and executes the BDK, the BDK loads and executes the ARM Trusted Firmware (ATF) and the ATF loads and executes the U-Boot bootloader in multiple stages as such:

• First level: Bootstub (firmware/bdk)
• Second level: ATF (firmware/atf)
• Third level: U-Boot (bootloader/u-boot) - Newport UBoot Bootloader Wiki Page

Gateworks provides a pre-built Boot Firmware ​firmware-newport.img ready to flash onto boot devices as well as source for building and/or modifying the boot firmware yourself.

Cavium CN80XX/CN81XX BOOT ROM

The BOOT ROM is firmware baked into the CN80XX/CN81XX SoC and is in charge of loading code from the 'boot device' into an L2 cache scratchpad, verifying signatures (if using trusted boot) and executing it. The BOOT ROM will only access the first 4MB (0x00000-0x7ffff) of the boot device.

Cavium Board Development Kit (BDK)

The Cavium Board Development Kit (BDK) is an open source codebase that supports the OCTEON TX family of processors and provides:

• Boot Stub (boot.bin) - the code fetched and executed by the internal BOOT ROM (limited to 192KB). This is run out of an L2 cache scratch area and is responsible for loading the next phase of code which is init.bin (normal boot) or setup.bin (manufacturing mode).
• init.bin - the code fetched and executed by boot.bin responsible for loading the board device tree file (dtb), configuring the SDRAM controller and loading the next stage which is the Arm Trusted Firmware (ATF)
• setup.bin - an application that can be used for board configuration and setup (we use this for 'Manufacturing Mode' to allow the GSC EEPROM to be programmed)
• diagnostics.bin - an application that can be used for very low level board diagnostics typically used during board validation

The BDK uses an embedded FAT12 filesystem for its various components (various .bin and .dtb files). Additionally, if you wanted to use a root filesystem type that is not supported by the bootloader you could also place a Flattened Image Table (FIT) image containing the kernel kernel.itb as well as a U-Boot uImage bootscript newport.scr in this filesystem as well and the Newport Bootloader scripts will find and execute them.

The FAT12 filesystem is created and managed using a host tool built by the BDK called fatfs-tool.

The bdk-create-fatfs-image.py python script creates the bdk.bin including the embedded FATFS and its contents. The size of the FATFS is controlled by the FATFS_SIZE variable within the script and the offset of it controlled by FIXED_SIZE.

The bdk repository provides the bdk.bin build artifact which contains the 512byte MBR with the partition table, the trusted and non-trusted boot stubs (boot.bin), and the FAT12 filesystem. It represents 0x000000 to 0x400000 of the boot firmware (first 4MB).

The bdk.bin is created via bin/bdk-create-fatfs-image python script (utils/scripts/bdk-create-fatfs-image.py) called from the 'all' target in bdk/boards/Makefile. The script does the following:

• create/update the non-trusted boot stub (192K at 0x20000) and headers (at 0x10000/0x10100)
• create/update the encrypted trusted boot stub (192K at 0x50000) and headers (at 0x10200/0x10300)
• create/update the non-trusted SCP BL1 (at 0x80000) and headers (at 0x10400/0x10500)
• create/update the trusted SCP_BL1 (at 0xc0000) and headers (at 0x10600/0x10700)
• create/update the non-trusted MCP BL1 (at 0x100000) and headers (at 0x10800/0x10900)
• create/update the trusted MCP BL1 (at 0x140000) and headers (at 0x10a00/0x10b00)
• create the FATFS filesystem (at 0x180000 2.5MB in size) and populate it with a set of files passed as additional arguments to the script

Note that the SCP and MCP components are only required for the CN9XXX and therefore Gateworks uses a modified version of the bdk-create-fatfs-image.py script that skips them allowing us to reclaim that space and grow the FATFS filesystem to approximately 13MB.

ARM Trusted Firmware (ATF)

The atf repository provides the ARM Trusted firmware and the tools to create a 'Firmware Image Package (FIP)'. The ​Firmware Image Package (FIP) packages the bootloader image into a single archive that can be loaded by the ARM Trusted Firmware from non-volatile platform storage.

The FIP package containing the bootloader fip.bin is created using the tool found in tools/fiptool.

Much of the terminology used here comes from the ARM Trusted Firmware Documentation:

• ​ARM Trusted Firmware Image Terminology
• ​ARM Trusted Firmware Documentation

The ATF Boot process is described in detail in the ​ATF firmware design documentation.

Some additional terminology:

• AP - Application Processor
• BL1 - first stage bootloader
• BL2 - second stage bootloader

The ATF BL1 has the following responsibilities:

• Enable the Trusted Watchdog
• Initialize the console (displays 'Booting Trusted Firmware')
• Configure the Interconnect to enable hardware coherency
• Enable the MMU and map the memory it needs to access
• Configure any required platform storage to load the next bootloader image (BL2)

The ATF BL2 has the following responsibilities:

• Initialize the console.
• Configure any required platform storage to allow loading further bootloader images.
• Enable the MMU and map the memory it needs to access.
• Perform platform security setup to allow access to controlled components.
• Reserve some memory for passing information to the next bootloader image EL3 Runtime Software and populate it.
• Define the extents of memory available for loading each subsequent bootloader image.

Boot Firmare Image

The firmware image contains all of the components of the 'Boot Firmware':

• MBR partition table
• trusted boot boot-stub (boot.bin)
• non-trusted boot boot-stub (boot.bin)
• embedded FATFS containing init.bin/setup.bin/*.dtb and possibly kernel
• ATF
• U-Boot (in an ATF FIP image)

The Cavium BSP uses a make-bootfs.py python script that combines the bdk.bin with the ATF components including the U-Boot FIP image.

Gateworks uses a customized version of this script that moves some of the components around a bit to create a large embedded FAT12 filesystem so that we have room for a kernel image and bootloader script there.

Newport Boot Firmware Image Map:

start-end	len	item	notes
0x0000000 - 0x0010000	64KB	MBR Partition Table	see newport/ptgen, first 512B only used
0x0010000 - 0x0020000	64KB	CVM_CLIB flash headers	built by newport/bdk-create-fatfs-image.py and required by BOOT ROM
0x0020000 - 0x0050000	192KB	boot stub (non-trusted) ap_bl1	bdk boot.bin loaded to L2 scratch and executed by BOOT ROM
0x0050000 - 0x0080000	192KB	boot stub (trusted) ap_bl1	bdk boot.bin loaded to L2 scratch and executed by BOOT ROM
0x0080000 - 0x0090000	64KB	backup of pristine U-Boot env
0x0090000 - 0x0100000	448KB	unused
0x0100000 - 0x0E00000	13312KB	FAT12 filesystem	contains various portions of the BDK as well as the device-tree
0x0E00000 - 0x0F00000	1024KB	ATF_BL1	atf/bl1.bin
0x0F00000 - 0x0FF0000	960KB	ATF_BL2 fip.img	U-Boot
0x0FF0000 - 0x1000000	64KB	U-Boot env	redudant 32KB env

• We remove the 960KB SCP_BL1/MCP_BL1 required by the CN9XXX
• We removed the 1M apparently unused ATF Table Header (ATF_TBL)
• We shifted the images around to maximize the size of the FAT12 filesystem while still leaving 960KB for the U-Boot fip.img

The firmware image is created by the newport/make-bootfs.py python script which creates the various headers around the objects required by the BOOT ROM and stitches them together (boot stub, ATF, ATF FIP containing the bootloader). This is called from the 'firmware' target in the Gateworks Newport BSP Makefile.

The Makefile builds this with the newport/make-bootfs.py script:

make-bootfs.py --bdk-image bdk/target-bin/bdk.bin --atf-bl1 atf/build/t81/release/bl1.bin --atf-fip fip.img --bootfs firmware-newport.img

• --bdk-image - the bdk/target-bin/bdk.bin is created by the bdk project, specifically by the bdk-create-fatfs-image.py python script. The size of the FATFS is controlled by the FATFS_SIZE variable within that script
• --atf-bl1 - the atf/build/t81/release/bl1.bin is created by the atf project
• --atf-fip - the fip.img is created by the atf project (atf/build/t81/release/fip.bin) and the fiptool app is used to update it with the U-Boot image
• --bootfs - the output file to create

While you can modify these scripts to adjust your firmware mapping note the following:

• the BDK init.bin (apps/init/app.c) has a #define for the location of the ATF (ATF_ADDRESS)
• the ATF defines the the location of the FIP image (plat/cavium/common/thunder_io_storage.c fip_block_spec.offset)
• the U-Boot Bootloader config defines the location of the U-Boot env (CONFIG_ENV_OFFSET/CONFIG_ENV_SIZE/CONFIG_ENV_OFFSET_REDUND)

Building a Bootable Disk Images

Gateworks releases disk images that can be easily flashed using the U-Boot Bootloader (see here). These disk images contain a partition table so they are tailored to the specific device size they are intended for. Additionally they are configured for a small partition size to keep flash programming time at a minimum. This requires resizing the rootfs filesystem in order to take advantage of the remaining flash space.

Once you have a root filesystem directory/tarball and have a kernel Image you can create a compressed disk image that can be installed easily from the bootloader (see newport/firmware).

The basic steps to create a bootable image of your Linux based OS are:

1. Follow instructions here to download necessary software and scripts.
2. start with the 16MB Newport ​Boot Firmware (firmware-newport.img)
3. (optional) update it's partition table if necessary:
   • the first partition is the FAT12 partition integrated into the Boot Firmware and thus its start/end is dictated by the boot firmware itself and should not be changed.
   • the second partition by default starts at the end of the 16MB boot firmware and extends to then end of the image
   • the end of the image is dictated the size of the target FLASH device. The 8GB eMMC on Newport has a usable user partition size of 7264MB thus the 2nd partition can be 7264-16=7248MB
4. place a uImage bootscript and kernel image on a bootable partition such as the FAT12 partition in the boot firmware. Alternatively your bootscript can look for the kernel wherever U-Boot can access it but typically we compress the kernel and place it into a FIT image and put it along side the bootscript.
5. create a root filesystem image (using a filesystem type supported by your kernel). You can use the mkfs script in the Newport BSP to create ext4 and f2fs filesystem images.
6. copy your filesystem image into the image file at the offset dictated by the partition table (16MB using the default partition table in firmware-newport.img)
7. compress the resulting disk image (so it can fit into DRAM without slicing and reduces transfer/writing time)

This is process made easy using scripts available in the Gateworks Newport BSP:

# define some variables
OUT=myimage.img
ROOTFS=focal-newport

# 1. start with firmware-newport.img
cp firmware-newport.img $OUT

# 2. update it's partition table if necessary
# - use 'sfdisk -l -uS $OUT' to show the partition table
# - use 'newport/ptgen -o $OUT -p ...' to adjust partition table if desired
# - make sure you don't change the first partition as the start/end of that is dictated by
#   the boot firmware
sfdisk -l -uS $OUT # show partition table

# 3. create your root filesystem
# - you can use the newport/mkfs script to create an f2fs|ext4 fs from a set of dirs/tarballs
# - specify a minimum filesystem size to keep your image small in order to
#   a) fit in SDRAM if using U-Boot tftpboot/gzwrite to update
#   b) transfer quickly
#   c) write less data and thus more quickly
# - once booted you can resize2fs for ext4 or 
sudo ./newport/mkfs ext4 $ROOTFS.ext4 1536M linux-newport.tar.xz $ROOTFS.tar.xz

# 4. create a uImage bootscript and kernel.itb and place them the root filesystem:
# - you can use the newport/ubuntu.scr as a sample bootscript which shows how to
#   load and execute a compressed kernel in a FIT image: kernel.itb
# - create kernel.itb with compressed kernel image
# - use mkimage to create a fit image from the template
# - mount your root filesystem and copy them to the boot directory
#   (or use fatfs-tool to copy the files into the FATFS filesystem within the image
#    if your root filesystem is not supported by the bootloader)
# create compressed kernel FIT image
cp linux/arch/arm64/boot/Image vmlinux
gzip -f vmlinux
mkimage -f auto -A arm64 -O linux -T kernel -C gzip -n "Newport Kernel" \                            
  -a 20080000 -e 20080000 -d vmlinux.gz kernel.itb 
# create bootscript
mkimage -A arm64 -T script -C none -d newport/ubuntu.scr newport.scr
# mount your root filesystem and copy them
mount $ROOTFS.ext4 /mnt
cp kernel.itb newport.scr /mnt/boot/
umount $ROOTFS.ext4

# 5. copy your filesystem to the correct partition offset
# (16MB unless you've had reason to change it)
dd if=$ROOTFS.ext4 of=$OUT bs=16M seek=1

# 6. compress it
#  - add a '-k' flag if you want to keep around the uncompressed image to perform
#    more maintanence on it like partition changes or FAT12 fs changes
gzip -f $OUT

To further simplify this process, the Newport BSP Makefile has a 'ubuntu-image' make target that will do all of the above for you if you provide env variables:

• UBUNTU_KERNEL (uncompressed kernel Image, defaults to linux/arch/arm64/boot/Image)
• UBUNTU_FS (uncompressed filesystem image defaults to focal-newport.ext4)
• UBUNTU_IMG (output file without the .gz extension, defaults to focal-newport.img):

  # create uncompressed root filesystem image
  sudo ./newport/mkfs ext4 focal-newport.ext4 1536M linux-newport.tar.xz focal-newport.tar.xz
  UBUNTU_KERNEL=linux/arch/arm64/boot/Image UBUNTU_FS=focal-newport.ext4 UBUNTU_IMG=focal-newport.img make ubuntu-image
  

• This creates ${UBUNTU_IMG}.gz or from the example above xenial-newport.img.gz

You can now install this using the methods described in newport/firware-update. For example in the bootloader:

setenv dev 0 # specify 0 for primary boot device; use mmc list to see all mmc devs
tftpboot ${loadaddr} xenial-newport.img.gz && gzwrite mmc ${dev} ${loadaddr} ${filesize}

If you created a filesystem that did not stretch to the partition it was installed on (as above where we created a 1536MB filesystem image to fit within a 7264M partition) you will want to resize it after booting (one time operation):

• ext4 filesystem booted on MMC device (resize2fs)

  resize2fs /dev/mmcblk0p2
  

Note there is no requirement that your root filesystem be on the boot device containing the 16MB Boot Firmware. You have other options as well as long as you have a bootscript that the bootloader can find, load, and execute. For example you could choose to put your bootscript in the FAT12 filesystem and have it load a kernel from an ext4 rootfs /boot/Image directory. You could also for that matter, have another device such as a mSATA with a single ext4 filesystem (make sure the partition is flagged as bootable) with a bootscript/kernel in its / or /boot directory (as those are in the search path for boot scripts).

To install the kernel and root filesystem on a removable block storage device see linux/blockdev.

Firmware Version

The Boot Firmware is comprised of several components that each have their own revision control (bdk, atf, u-boot, dts) and as such there are multiple version details to keep track of.

To aid this the Newport BSP Makefile which builds the boot firmware creates a file called version within the FAT12 filesystem. You can mount this filesystem within Linux and examine the contents. For example:

root@OpenWrt:/# mount /dev/mmcblk0p1 /mnt
root@OpenWrt:/# cat /mnt/version
BDK=e1a02ac-dirty Mon Feb 12 16:59:48 UTC 2018
ATF=v1.3(release):99860b5-dirty
UBOOT=2017.09-rc1-g5d286a6
DTS=f31a186
root@OpenWrt:/# umount /mnt

U-Boot Bootloader

Read more on the dedicated bootloader page here: Newport U-Boot Bootloader

Booting to a kernel/rootfs on the microSD

This assumes the bootloader will remain on the eMMC, but then we switch to the microSD for the kernel/rootfs.

Run the following commands in the uboot bootloader command prompt on the Newport SBC bootloader serial console:

setenv boot_targets mmc1

boot