= Gateworks Newport GW6903 SBC w/Google Coral TPU Machine Learning (ML) Accelerator for AI — What is it? [[PageOutline]] [[Image(AI.jpg, width=500)]] This demo features a [https://www.gateworks.com/products/octeon-tx-single-board-computers-gateworks-newport/gw6903-single-board-computer/ Gateworks Newport GW6903 Rugged Single Board Computer] (SBC) running the Google Coral Edge Tensor Processing Unit (TPU) Mini-PCIe card. The Coral TPU provides a means to perform advanced Machine Learning (ML) tasks in a low power, small form factor, Mini-PCIe card. The TPU Accelerator is based on a custom "Application Specific Integrated Circuit" (ASIC) that Google designed for hardware accelerated AI calculations. The TPU features the following: * Performance: 4 TOPS (Trillion Operations Per Second) * Power: 2 Watts Typ, 4 Watts Max * Half-size Mini-PCIe Form Factor (PCIe Gen2 x1) * Supported Framework:[https://www.tensorflow.org/lite TensorFlow Lite] * Works with [https://cloud.google.com/vision/automl/docs/edge-quickstart AutoML Vision Edge] * -20 to +70C Operating Temperature * [https://coral.ai/products/pcie-accelerator/ Buy Coral TPU on Coral Website] The Gateworks Single Board Computer platform is an ideal fit for the Google Coral TPU Mini-PCIe card, especially because it supports half sized cards. [[Image(6903_small_top.jpg, width=400)]][[Image(GW6903_wTPUcut.jpg, width=400)]] ''' The above image shows the Google Coral TPU card in the more left Mini-PCIe slot on the bottom of the GW6903 SBC and a wireless radio in the more right upper slot''' This TPU is appropriate for an application where identifying an object or pattern is required. This could be, but is not limited to: * Object detection * Pose or gesture estimation * Image segmentation * Key phrase detection Some practical applications are: * Autonomous vehicles * Robots * Voice control/Language Processing * Monitoring devices Nearly all industries can benefit from this technology. To name a few more specifically: * Health Care * Agriculture * Manufacturing * Oil and Gas * !Security/Defense * Automated Kiosk For more information on the Google Coral TPU Mini-PCIe card see the following links: * https://coral.ai/products/pcie-accelerator/ * https://coral.ai/docs/m2/get-started/ * https://coral.ai/docs/mini-pcie/datasheet/ * https://coral.ai * https://en.wikipedia.org/wiki/Tensor_processing_unit = Getting started with the Coral Edge TPU on Gateworks SBCs To begin you will need: * A workstation with Linux natively installed. * A [https://www.gateworks.com/products/ Gateworks Newport SBC] — Coral requires AARCH64. * A Coral EDGE TPU, our testing was done with the mPCI-e form factor model. * !Network/Internet connection. * Optional: USB webcam == Compiling the kernel The Gateworks kernel defconfig for Newport does not include support for video devices. To create an image with webcam support: * Acquire the [wiki:/newport/bsp Newport BSP], we will call the directory this repo has been sync'ed to the directory. * Follow the steps [wiki:/newport/bsp#Modifyingthestand-aloneLinuxKernelieforUbuntu here] to modify the kernel and create and Ubuntu image. * In the menuconfig enable the module "USB_VIDEO_CLASS", this will allow you to use a USB webcam with v4l-utils. * Complete the procedure detailed in the aforementioned section, build your Bionic image and [wiki:/newport/firmware#UpdateFirmwareviaSerialConsoleandEthernetfromBootloader flash it to your SBC] * Though it was not specifically created to address using the Newport BSP this video may provide some insight into the process of changing the kernel config. https://youtu.be/XCkegC05xXY == Building and installing the Gasket and Apex modules Gateworks Video detailing the process: https://youtu.be/PcrGUiuNBcg The source code for the modules can be downloaded here as a tar.gz file: * https://coral.googlesource.com/linux-imx/+/refs/heads/release-day/drivers/staging/gasket * Extract this tar into a directory, this dir will be referred to as . Build the source using the same method as you would for an out of tree module (out of tree = when the module source is not included in the kernel source). * cd to the directory. * {{{#!bash source setup-environment }}} * Doing this will configure the toolchain for building. Keep in mind some of the lines in this file use the argument $PWD thus it should not be sourced from any location other than the folder. * You can verify the arguments have been exported by executing the command {{{#!bash echo $ARCH }}} This will return "arm64" * cd to * cat the "Makefile", you'll see two variables which need to be set for the modules to build correctly. {{{#!bash export CONFIG_STAGING_GASKET_FRAMEWORK=m export CONFIG_STAGING_APEX_DRIVER=m }}} * Execute the following command {{{#!bash make -C /linux M=$PWD }}} This procedure will result in two modules being created, "apex.ko" and "gasket.ko". Copy the .ko files to your target board "/lib/modules//extra/" folder. Using SCP may be the simplest way to go about this. With the modules copied to the board execute the following commands: {{{#!bash depmod -a }}} {{{#!bash insmod gasket.ko sync }}} Remove power from the board and reboot. On reboot verify that "/dev/apex_0" device is present. == Installing and configuring Python Python 3.7 is required to run the tenser flow examples. Other versions can be used, though at the time of writing this wiki 3.7 is the best option when using the Bionic Ubuntu for Newport BSP. {{{#!bash apt-get update apt install python3.7 -y }}} Set Python 3.7 to have priority over 3.6. {{{#!bash update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1 update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2 }}} {{{#!bash update-alternatives --config python3 #on this menu enter the number 2. }}} You can verify you have been successful in changing the version with the following command: {{{#!bash python3 --version }}} == Installing the TPU runtime Install the util curl {{{#!bash apt-get install curl -y }}} Add Debian package repository to your system {{{#!bash echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - sudo apt update }}} Install PIP and required libraries {{{#!bash apt-get install python3-pip libedgetpu1-std -y }}} {{{#!bash pip3 install --upgrade pip setuptools wheelpi }}} Download the TFlite Runtime .whl {{{#!bash wget https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_aarch64.whl }}} **Note:** If you have chosen not to use Python 3.7 you can find the .whl appropriate for your version here https://www.tensorflow.org/lite/guide/python Acquire necessary libraries to build the TFlite runtime {{{#!bash pip3 install cython pip3 install numpy apt-get install python3-pil }}} Install TFlite runtime {{{#!bash pip3 install tflite_runtime-2.1.0.post1-cp36-cp36m-linux_aarch64.whl }}} = Standard Demo: Downloading Classification example & Testing an Inference Create a place for the Coral examples to reside. {{{#!bash mkdir coral && cd coral }}} Clone the examples, "cd" to examples directory. {{{#!bash git clone https://github.com/google-coral/tflite.git cd tflite/python/examples/classification }}} Install prerequisite programs using supplied script. {{{#!bash bash install_requirements.sh }}} Run the classification demo, passing in a image of a parrot, as shown with the last argument. {{{#!bash python3 classify_image.py \ --model models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \ --labels models/inat_bird_labels.txt \ --input images/parrot.jpg }}} ** Example output: ** {{{ root@bionic-newport:~/coral/tflite/python/examples/classification# python3 classify_image.py \ > --model models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \ > --labels models/inat_bird_labels.txt \ > --input images/parrot.jpg ----INFERENCE TIME---- Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory. 16.0ms 3.1ms 3.1ms 3.1ms 3.0ms -------RESULTS-------- Ara macao (Scarlet Macaw): 0.77734 }}} = Advanced Demo: Video Webcam Detection with GStreamer Example This example will use a USB webcam and the TPU to identify objects presented to the webcam. The video output and overlay will be streamed to a location on the network for viewing. Install Gstreamer, you will need this program on both the SBC and your workstation where you will be viewing the output. {{{#!bash apt-get install gstreamer1.0-x gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-alsa -y }}} Clone the Gstreamer example {{{#!bash mkdir google-coral && cd google-coral git clone https://github.com/google-coral/examples-camera.git --depth 1 }}} Download models. Models are the information that will be feed to the TPU for it to reference when identifying an object. {{{#!bash cd examples-camera sh download_models.sh }}} Configure the Gstreamer Python script {{{#!bash cd gstreamer bash install_requirements.sh }}} Adaptations to the existing gstreamer.py will be required for this example to work. A modified script is available to download from [http://trac.gateworks.com/attachment/wiki/TPU/gstreamer.py here]. Edit this file line 231 with the IP address of the desktop workstation you will be streaming to. This is what the edited line will look like. {{{ ! rsvgoverlay name=overlay ! videoconvert ! jpegenc ! tcpclientsink host=172.24.24.93 port=9001 }}} * The workstation being used in this example has an IP address of 172.24.24.93, the SBC is using an IP which is on the same subnet. Launch Gstreamer on your workstation to receive the video from the Gateworks SBC: {{{#!bash gst-launch-1.0 tcpserversrc host=0.0.0.0 port=9001 ! jpegdec ! videoconvert ! autovideosink sync=false }}} On the Gateworks SBC run: {{{#!bash python3 classify.py }}} Here's a video demonstration of the output you can expect if everything is working: Video: https://youtu.be/pss6Gy_8UI4 [[Image(Youtubecoral.JPG, width=500)]] = Going Further Some additional links for models and creating your own models: * https://coral.ai/models/ * https://teachablemachine.withgoogle.com/