wiki:Android/Building

Version 2 (modified by Tim Harvey, 7 years ago) ( diff )

added android 5.1 pre-built image links and updated older prebuilt image links

  1. Gateworks Ventana Android Board Support Package
  2. Android Build Variants
  3. Gateworks Precompiled Android Binary
    1. Install on 2GB NAND Flash via ubi file
    2. Install on a 2GB+ removable block storage device (USB/SATA/MMC)
  4. Building Android for Ventana from source
    1. Pre-requisites
      1. Build Host
      2. Network Connection and Disk Space
      3. JDK
      4. Packages
    2. Obtaining Source
      1. Access Control
      2. Fetching Source
      3. Keeping up to date and/or pinning source with repo
    3. Building
      1. Creating a tarball of the built artifacts
    4. Updating and Re-building
    5. Flashing Android to NAND Flash
    6. Creating a bootable removable storage device

Gateworks Ventana Android Board Support Package

Gateworks supports the following Android versions:

  • Android 5.1.1 (Aka 'Lollipop')
  • Android 4.4 (Aka 'KitKat')
  • Android 4.3 (Aka 'Jelly Bean-MR2')

For a feature list please see Gateworks Android

Android Build Variants

The Android build system supports different build variants. The two that we build and support are:

  • eng - used for development and hacking
    • installs modules tagged with 'eng', 'debug', 'user', and/or 'development'
    • installs non-APK modules that have no tags specified
    • installs APKs according to the product definition files, in addition to tagged APKs
    • default.prop disables security and allows debugging: 
      ro.secure=0
ro.debuggable=1
    • adb runs as user 'root' is enabled by default (this allows commands such as adb remount and adb push)
  • user - intended for final release
    • installs modules tagged with 'user'
    • installs non-APK modules that have no tags specified
    • installs APKs according to the product definition files; tags are ignored for APK modules.
    • default.prop disables security and allows debugging: 
      ro.secure=1
ro.debuggable=0
    • adb runs as user 'shell' and is disabled by default (adb running as non-root dis-allows commands such as adb remount and adb push but allows adb install

Gateworks Precompiled Android Binary

Gateworks provides pre-built Android images that you can evaluate without building the source if needed. You have the following options:

  • purchase a pre-installed USB stick from the ​Gateworks Online Shop
  • download UBI image if you have a board with 2GB of NAND flash and want to boot Android from flash
  • download tarball and image a removable block storage device (mSATA / USB / uSD)

Latest images:

See above for differences on the eng vs user build variant

Once you have installed the pre-built image using one of the methods below please see here for important instructions on booting Android

Install on 2GB NAND Flash via ubi file

To install Android in flash for boards with 2GB of NAND from a downloaded ubi image:

  1. Download Binary ubi (see above for options)
  2. Place ubi on tftp server or removable storage
  3. Boot target and break into the bootloader
  4. Update NAND flash with image.ubi served from a tftp server (modify the ip addrs below for your network): 
    setenv ipaddr 192.168.1.1
setenv serverip 192.168.1.254
tftp ${loadaddr} android.ubi && nand erase.part rootfs && nand write ${loadaddr} rootfs ${filesize}
    • For more examples of loading images into uboot from network/mmc/usb see here

Install on a 2GB+ removable block storage device (USB/SATA/MMC)

The partitioning scheme Gateworks uses for Android on block storage is the following:

Partition Size Name Filesystem Description
N/A 1MB - - SPL, bootloader, env
/dev/sdc1 20MB BOOT ext4 Android boot partition (bootscript, kernel, dtbs, ramdisk)
/dev/sdc2 20MB RECOVERY ext4 Android's recovery partition
/dev/sdc3 - extended - extended partition
/dev/sdc4 - DATA ext4 userdata
/dev/sdc5 512MB SYSTEM ext4 Android's /system partition (ROM)
/dev/sdc6 50MB CACHE ext4 cache
/dev/sdc7 10MB VENDOR ext4 vendor
/dev/sdc8 10MB MISC ext4 misc

To install on a 2GB+ removable block storage device (USB/SATA/MMC) from a downloaded tarball you first need to know the Linux device name of the storage device (use the Disks tool, or look at /var/log/messages for this info). Assuming /dev/sdc you can:

  1. Download Binary tarball (see above for options)
  2. extract the binary (example using gateworks-kk4.4.3_2.0.0-ga-user-20150327.tar.gz) 
    tar xvf gateworks-kk4.4.3_2.0.0-ga-user-20150327.tar.gz
  3. cd into the directory with the build artifacts and run the mksdcard.sh script: 
    cd kk4.4.3_2.0.0-ga_user
sudo ./mksdcard.sh /dev/sdc

Note that the partition layout above requires 8 partitions which are typically not supported with built-in SDIO host controllers (which is what you are using if your microSD shows up as /dev/mmcblk*). We recommend using a USB based microSD card reader/writer which can be found from office supply stored for under $5.

Building Android for Ventana from source

Pre-requisites

Build Host

Per the Android documentation ​here you should have a build host with the following:

  • Linux or Mac
  • 64bit environment
  • 100GB (for a single build, 150GB or more for multiple builds, and more if you use ccache)
  • Python 2.6 - 2.7
  • GNU Make 2.81 - 3.82
  • JDK 6 (for Gingerbread through KitKat)
  • Git 1.7 or newer

Gateworks uses Ubuntu 14.04/13.10 64bit build hosts although other configurations may work.

A virtual machine is possible to use and Gateworks has verified that our latest BSP builds with the popular ​BuilduntuVM based on Xubuntu with the following changes (based on BuildubuntuVM v1.5):

  • Download from ​BuilduntuVM page
  • change Networking mode to bridged instead of NAT - VirtualBox NAT can be unreliable with the heavy traffic a repo sync does.
  • adjust the memory allocated to the VM to a level appropriate for your host (defaults to 4GB, we've tested with 2GB)
  • install make v3.81 (make v4.0 on BuilduntuVM needs to be downgraded for Android KitKat builds): 
    wget -o make.tar.gz http://ftp.gnu.org/gnu/make/make-3.81.tar.gz
tar -xvzf make-3.81.tar.gz
cd make-3.81
./configure
sudo make install
hash -r
  • install Oracle Java6 (openjdk7 on BuilduntuVM needs to be downgraded for Android KitKat builds): 
    sudo apt-get install oracle-java6-installer
  • install a couple other packages our BSP requires: 
    sudo apt-get install u-boot-tools uuid-dev liblzo2-dev

References:

Network Connection and Disk Space

The initial repo sync of Android transfers approximately 15GB of data from the Internet. This can take several hours depending on your bandwidth.

While the source-code accounts for approximately 15GB of data, you will need an additional 25GB or so (40GB total) for a build.

JDK

Different versions of Android require different versions of a Java Development Kit (JDK). Note that multiple versions of Java can easily co-exist on an OS by using either a JAVA_HOME env variable pointing to the root of a JDK, or by using Linux alternatives. To avoid switching back and forth, Gateworks recommends setting a JAVA_HOME env variable.

JDK Requirements:

  • Android Lollipop - OpenJDK 7:
    • Ubuntu >= 14.04: 
      sudo apt-get update
sudo apt-get install openjdk-7-jdk
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
  • Android Gingerbread through Kitkat: Oracle JDK 6:
    • Ubuntu 14.04/13.10: 
      # add the PPA with updated builds of Java 6JDK compatible with 13.10
sudo add-apt-repository ppa:webupd8team/java # hit enter when prompted
# install the package
sudo apt-get update && sudo apt-get install oracle-java6-installer # read and accept the license agreement
# verify the correct version is installed
java -version
 java version "1.6.0_45"
 Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
 Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
export JAVA_HOME=/usr/lib/jvm/java-6-oracle

Note that you can have multiple JDK's installed at the same time and use 'update-alternatives' to configure which version of java and javac is used.

References:

Packages

Ubuntu 14.04 (Note that most of these instructions were found ​here): 

sudo apt-get purge icedtea-\* icedtea6-\*
sudo apt-get install git gnupg ccache lzop flex bison gperf build-essential zip curl \
  zlib1g-dev zlib1g-dev:i386 libc6-dev lib32bz2-1.0 lib32ncurses5-dev x11proto-core-dev \
  libx11-dev:i386 libreadline6-dev:i386 lib32z1-dev libgl1-mesa-glx:i386 libgl1-mesa-dev \
  g++-multilib mingw32 tofrodos python-markdown libxml2-utils xsltproc libreadline6-dev \
  lib32readline-gplv2-dev libncurses5-dev bzip2 libbz2-dev libbz2-1.0 libghc-bzlib-dev \
  lib32bz2-dev squashfs-tools pngcrush schedtool dpkg-dev
sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so
sudo apt-get install u-boot-tools uuid-dev liblzo2-dev librsvg2-dev intltool mtd-utils pv
cpan App::cpanminus # type 'yes' to let it automatically choose settings for you
cpan Switch

Ubuntu 13.10: 

sudo apt-get install git-core lzop ccache gnupg flex bison gperf build-essential \
  zip curl zlib1g-dev zlib1g-dev:i386 libc6-dev lib32ncurses5 lib32z1 lib32bz2-1.0 \
  lib32ncurses5-dev x11proto-core-dev libx11-dev:i386 libreadline6-dev:i386 lib32z-dev \
  libgl1-mesa-glx:i386 libgl1-mesa-dev g++-multilib mingw32 tofrodos python-markdown \
  libxml2-utils xsltproc readline-common libreadline6-dev libreadline6 lib32readline-gplv2-dev \
  libncurses5-dev lib32readline5 lib32readline6 libreadline-dev libreadline6-dev:i386 \
  libreadline6:i386 bzip2 libbz2-dev libbz2-1.0 libghc-bzlib-dev lib32bz2-dev \
  libsdl1.2-dev libesd0-dev squashfs-tools pngcrush schedtool libwxgtk2.8-dev \
  python gcc g++ cpp gcc-4.8 g++-4.8
sudo apt-get install u-boot-tools uuid-dev liblzo2-dev librsvg2-dev intltool mtd-utils pv
sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so

Ubuntu 12.04: 

sudo apt-get install git gnupg flex bison gperf build-essential \
  zip curl libc6-dev libncurses5-dev:i386 x11proto-core-dev \
  libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-glx:i386 \
  libgl1-mesa-dev g++-multilib mingw32 tofrodos \
  python-markdown libxml2-utils xsltproc zlib1g-dev:i386
sudo apt-get install u-boot-tools uuid-dev liblzo2-dev librsvg2-dev intltool mtd-utils pv
sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so

References:

Obtaining Source

Android AOSP (Android Open Source Project) source is spread across 300+ git repositories. The ​'repo' tool is used as a front-end to git to manage working with all of these repos. A manifest file describes all the repos used, where they can be found, what branch to checkout, and where to check it out to in the resulting directory tree. A repo sync command is what fetches or updates all these repos and an initial sync will pull down approximately 15GB of data from the Internet.

Of the roughly 334 various project trees that create Android, Freescale adds 13 new projects (for Freescale specific features/subsystems) and modifies 28 projects from upstream Android sources (to add support for Freescale features).

Access Control

Freescale has several projects and patches required for Android on the IMX6 CPU, some of which require a End User License Agreement (EULA). They distribute these as a tarball of patches which is difficult to work with, especially as Gateworks and our end users also apply patches on top of these to add functionality.

Gateworks puts the projects that are not Open Source in a bitbucket account with access control. If you need to build and develop Android source for Ventana, you will need to contact support@… requesting access to the gateworks-android bitbucket team. You will need to sign our EULA and provide an ssh public key (~/.ssh/id_rsa.pub) for each user and workstation you need access from. To learn how to generate an SSH key on your linux host see ​here. You will want to create a key without a passphrase as the whole point of this key is to automate connection to the repo's without requiring entering a passphrase (if you are prompted for a passphrase each time you contact a repo the repo tool will get in the way of entering in the passphrase)

Once you have confirmation from support@… that we have added access for your ssh key(s) you need to do the following on your development host to add the bitbucket RSA key fingerprint to your development host so that you are not prompted for it during a repo sync: 

ssh-keyscan -H git.bitbucket.org >> ~/.ssh/known_hosts

Fetching Source

Once you have access to the private repositories (see above) you can use the following steps to create a build directory for Android.

Branches:

  • gateworks_l5.1.1_2.1.0-ga - Android Lollipop 5.1 (Latest Gateworks Android release)
  • gateworks_kk4.4.3_2.0.0-ga - Android KitKat 4.4.3
  • gateworks_jb4.3_1.1.0-ga - Android Jelly Bean 4.3
  1. Create a directory for your Android build: 
    mkdir gateworks-android
cd gateworks-android
  1. Get repo tool. This tool uses xml manifests to pull down multiple repos: 
    curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ./repo
chmod a+x ./repo
  1. Initialize android source repo (this pulls down the manifest of projects that make up the Gateworks Ventana Android BSP): 
    ./repo init -u https://github.com/Gateworks/imx_android.git -b gateworks_l5.1.1_2.1.0-ga
    • use any of the above branches if you wish to use an older release
    • repeat this with a different branch name any time you wish to change to a new branch without pulling down all the repo's again in their entirety
  1. (optional) pin the repo versions to the latest version which was known to have been successfully built by the Gateworks nightly build server: 
    wget http://dev.gateworks.com/android/lp-5.1.1/snapshot.xml # fetch the pinned manifest from the last successful nightly build
mv snapshot.xml .repo/manifest.xml # copy over the generic un-pinned manifest
    • the repo init command will fetch a repo 'manifest' which refers to various source repositories and branches but will fetch the latest changes from those branches. The Gateworks nightly build server posts the manifest pinned to the specific revision used in the build on completion of a successful build. You can also use a snapshot from a previously released pre-built binary by using the other snapshot files in ​​http://dev.gateworks.com/android
  1. Sync repo - this fetches the repos from the Internet (may take a while to download sources resulting in about 15GB more): 
    ./repo sync -c -j8
    • the -c parameter tells repo to only fetch the current branch specified in the project manifest and can help speed up fetching as well as reduce disk space needed
    • the -j parameter tells repo to use multiple processes (8 in this case) and can help speed up fetching

Troubleshooting:

  • first, try repo sync again. Often temporary network congestion or outages may be the culprit.
    • Sometimes, a source destination may have change from the first repo sync you may have preformed. In this case, try repo sync --force-sync to force a sync to a new package location.
  • Bitbucket access issues:
    • if you see several Permission denied (publickey) errors then you have not been granted access to the Freescale proprietary repositories (see above)
    • if you see several The authenticity of host 'git.butbucket.org (131.103.20.168)' can't be established. messages then you have forgotten to add the bitbucket RSA key fingerprint to your development host (see above). Type 'yes' and hit return to add it permanently.
  • VirtualBox VM issues:
    • when using a VirtualBox VM the process hangs or fails with a variety of symptoms such as HTTP 500 errors. The default network behavior of VirtualBox is to use NAT (Network Address Translation) to connect the guest system to the network and the heavy network activity of a repo sync triggers some corner cases in the NAT code. As a workaround configure VirtualBox to use bridged network instead of NAT.
  • For various other network troubleshooting issues please see the official Android source download documentation ​here and their known issues ​here. Various issues that can be common are discussed there with workarounds:
    • behind a proxy or firewall - set HTTP_PROXY and HTTPS_PROXY and use a single thread (repo sync -j1)
    • download failures (typically hangs or 406 errors) - disable tcp window scaling and/or use a single thread: 
      sudo sysctl -w net.ipv4.tcp_window_scaling=0
repo sync -j1
    • IP bandwidth exceeded (common if you try to repo sync a full tree more than once per day, or have multiple people/systems doing it behind a NAT firewall) - use authentication by ​obtaining a password and adding an '/a' path to the beginning of the URL to android.googlesource.com directory

References:

Keeping up to date and/or pinning source with repo

When building projects that use multiple source repositories any repository may change and thus make your working directory out of date. Because Android uses hundreds of projects with their own repositories this can make it difficult to keep in sync - this is where the repo tool comes in handy.

The repo sync command will update all repositories with upstream changes: 

repo sync --quiet

You can override the default Manifest by using a local manifest if you want to keep in sync with the upstream manifest yet make some minor change like pin a specific project or add a couple of projects. Reasons for doing this could be:

  • adding new projects
  • pinning certain projects
  • removing certain projects

To use a local manifest create .repo/local_manifest.xml and it will be merged with .repo/manifest.xml by the repo tool whenever the manifest is used. You can use the remove-project directive to remove a project that you don't want and even add it back with your own choices if you want something different. For example: 

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
   <remove-project path="hardware/qcom/display" name="CyanogenMod/android_hardware_qcom_display" />
   <project path="hardware/qcom/display" name="WinSuk/android_hardware_qcom_display" />
</manifest>

The repo manifest command will create a snapshot of your current project's manifest allowing you to create a pinned version that can be used later to create a working directory with the various projects at the exact same state as your current working directory: 

repo manifest -o snapshot.xml -r

This snapshot can then be copied over .repo/manifest.xml in a different build directory to pin the repository sources.

The Gateworks nightly build server creates a manifest snapshot like this and uploads the latest successful build to ​​http://dev.gateworks/com/yocto so that it can be used to re-create a successful nightly build.

References:

Building

  1. Configure Android build system for 'ventana' product and desired variant (See above for differences on the eng vs user build variant) 
    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
source build/envsetup.sh
lunch ventana-eng
    • An 'eng' build (ventana-eng) has certain debug options enabled and lowers security (makes things nice if your going to play around)
    • A 'user' build (ventana-user) is more locked down and intended for a production/shipping environment
    • This process is sometimes referred to as 'activating the shell' and must be done for each new shell (ie each time you login or open a new terminal). If this seems strange to you or if you are new to Linux build systems, you should probably only use a single shell to avoid confusion.
  1. Build (takes up an additional ~25GB and takes several hours) 
    make 2>&1 | tee build.log

When complete the various build artifacts will be in out/target/product/ventana/

  1. Put artifacts onto a storage device to boot. See next section using the 'mksdcard.sh' script. The following build artifacts are in out/target/product/ventana/ grouped by the partition they belong in:
    • BOOT:
      • uImage - kernel image
      • boot/boot/uramdisk.img - initial ramdisk containing init system
      • boot/boot/6x_bootscript-ventana - u-boot bootscript (sets up android kernel cmdline)
      • boot/boot/uImage - kernel image
      • boot/boot/*.dtb - kernel device tree binary blobs
    • RECOVER:
      • ramdisk-recovery.img - initial ramdisk containing init system
    • SYSTEM:
      • system.img - filesystem image of /system directory (Android ROM)
    • USER:
      • userdata.img - user data

Note that the Android partition layout we use requires 8 partitions which are typically not supported with built-in SDIO host controllers based on standard kernel configurations (which is what you are using if your microSD shows up as /dev/mmcblk*). We recommend using a USB based microSD card reader/writer which can be found from office supply stored for under $5.

Creating a tarball of the built artifacts

If you would like to create an archive of the build artifacts including everything needed to image Android onto a removable block storage device as well as the NAND uib, the following will do so: 

tar -cvjf l5.1.1_2.1.0-ga.tar.bz2 \
  --transform "s,^device/gateworks/ventana/,l5.1.1_2.1.0-ga/," \
  --transform "s,^out/target/product/ventana/,l5.1.1_2.1.0-ga/," \
  device/gateworks/ventana/mksdcard.sh \
  out/target/product/ventana/boot \
  out/target/product/ventana/uramdisk-recovery.img \
  out/target/product/ventana/userdata.img \
  out/target/product/ventana/system.img \
  out/target/product/ventana/android.ubi

Untaring the above archive would show: 

tar xvf ../l5.1.1_2.1.0-ga.tar.bz2
l5.1.1_2.1.0-ga/mksdcard.sh
l5.1.1_2.1.0-ga/boot/
l5.1.1_2.1.0-ga/boot/boot/
l5.1.1_2.1.0-ga/boot/boot/imx6dl-gw54xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6dl-gw53xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6dl-gw52xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6dl-gw51xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6dl-gw551x.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6dl-gw552x.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6q-gw54xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6q-gw53xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6q-gw52xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6q-gw51xx.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6q-gw551x.dtb
l5.1.1_2.1.0-ga/boot/boot/imx6q-gw552x.dtb
l5.1.1_2.1.0-ga/boot/boot/uImage
l5.1.1_2.1.0-ga/boot/boot/6x_bootscript-ventana
l5.1.1_2.1.0-ga/boot/boot/uramdisk.img
l5.1.1_2.1.0-ga/uramdisk-recovery.img
l5.1.1_2.1.0-ga/userdata.img
l5.1.1_2.1.0-ga/system.img
l5.1.1_2.1.0-ga/android.ubi

Updating and Re-building

From time to time you may want to refresh your sources with upstream changes from Gateworks and elsewhere. The Gateworks Android BSP manifest file fetched from the repo init command controls what projects get fetched and where they go. The repo sync command with both update the manifest (in case projects have been added, removed, or changed) and updates all the individual git repos it references. 

./repo sync --quiet
  • The quiet flag reduces most of the output allowing you to easily progress
  • This only pulls down 'changes' that have been made
  • As this alters source-code, do not do this while a build is running

Now you can repeat the steps above to rebuild. You do not need to do a make clean.

Flashing Android to NAND Flash

If you have a board with 2GB NAND Flash, you can use the nand_update uboot script to flash the Android ubi file from a tftp server.

First you must copy the ubi from your build directory to your tftp server root directory: 

cp out/target/product/ventana/android.ubi /tftpboot

The nand_update script will tftp the file file represented by the image_rootfs env variable from the server represented by the serverip env variable and flash it to the rootfs partition: 

setenv image_rootfs android.ubi
run nand_update
reset

Creating a bootable removable storage device

You can create a bootable removable storage device (micro-SD / USB stick / mSATA disk) with the mksdcard.sh script (located in device/gateworks/ventana/mksdcard.sh of the source directory). Assuming the removable device is in /dev/sdc on your linux host (this is an assumption here - please verify on your system with something like the 'Disks' tool on Ubuntu): 

sudo device/gateworks/ventana/mksdcard.sh /dev/sdc
  • Note that sanity checks are done to ensure the device specified appears to be removable by checking its size. If you get an 'invalid disk' error because of this and want to force the script you can execute it as: 
    sudo device/gateworks/ventana/mksdcard.sh /dev/sdc ventana --force
  • If using an archive of build artifacts you will find the mksdcard.sh script in the root directory and can run it from there

Notes:

  • you must have a device of 2GB or larger

Booting:

  • Please see here for important instructions on booting Android from removable media
Note: See TracWiki for help on using the wiki.