README.md 36.5 KB
Newer Older
Vincent Coubard's avatar
Vincent Coubard committed
1
2
# Introduction

3
[Arm IoT Total Solutions](https://www.arm.com/solutions/iot/total-solutions-iot) provides a complete solution designed for specific use-cases, leaving developers to focus on what really matters - innovation and differentiation across diverse and varied use cases. It has everything needed to simplify the design process and streamline product development, including hardware IP, software, real-time OS support, machine learning (ML) models, advanced tools such as the new Arm Virtual Hardware, application specific reference code and support from the world's largest IoT ecosystem.
Vincent Coubard's avatar
Vincent Coubard committed
4

Vincent Coubard's avatar
Vincent Coubard committed
5
6
# Overview

7
This repo contains Arm's first [IoT Total Solution](https://www.arm.com/solutions/iot/total-solutions-iot), "Keyword Detection".  It provides general-purpose compute and ML workload use-cases, including an ML-based keyword recognition example that leverages the DS-CNN model from the [Arm Model Zoo](https://github.com/ARM-software/ML-zoo).
Vincent Coubard's avatar
Vincent Coubard committed
8

9
The software supports multiple configurations of the Arm Corstone-300 subsystem, incorporating the Cortex-M55 processor and Arm Ethos-U55 microNPU.  This total solution provides the complex, non differentiated secure platform software on behalf of the ecosystem, thus enabling you to focus on your next killer app.
Vincent Coubard's avatar
Vincent Coubard committed
10

11
## Keyword detection application
Vincent Coubard's avatar
Vincent Coubard committed
12

13
The keyword detection application runs the DS-CNN model on top of [FreeRTOS](https://freertos.org/a00104.html#getting-started). It detects keywords and inform the user of which keyword has been spotted. The audio data to process are injected at run time using the [Arm Virtual Hardware](https://www.arm.com/virtual-hardware) audio driver.
Vincent Coubard's avatar
Vincent Coubard committed
14

Vincent Coubard's avatar
Vincent Coubard committed
15
The Keyword application connects to [AWS IoT](https://docs.aws.amazon.com/iot/latest/developerguide/what-is-aws-iot.html) cloud to publish recognised keywords. AWS IoT cloud is also used for OTA firmware updates. These firmware updates are securely applied using [Trusted Firmware-M](https://tf-m-user-guide.trustedfirmware.org/). For more information, refer to the keyword detection [Readme](./kws/README.md).
Vincent Coubard's avatar
Vincent Coubard committed
16
17
18

![Key word detection architecture overview](./resources/Keyword-detection-overview.png)

19
## Blinky application
Vincent Coubard's avatar
Vincent Coubard committed
20

21
The blinky application demonstrate blinking LEDs using Arm Virtual Hardware. FreeRTOS is already included in the application to kickstart new developments.
Vincent Coubard's avatar
Vincent Coubard committed
22
23

# Quick Start
Vincent Coubard's avatar
Vincent Coubard committed
24

25
Follow these simple steps to build and execute the code example's application within **Arm Virtual Hardware**.
Vincent Coubard's avatar
Vincent Coubard committed
26

27
28
* [Launch Arm Virtual Hardware system](#launch-arm-virtual-hardware-instance)
* [Build and execute](#build-and-execute-the-application)
Vincent Coubard's avatar
Vincent Coubard committed
29
30
31
* [Setting up AWS Cloud connectivity](#setting-up-aws-connectivity)
* [Enabling OTA firmware update from the AWS Cloud](#ota-firmware-update)
* [Setting up Azure Cloud connectivity](#setting-up-azure-connectivity)
32
* [Terminating Arm Virtual Hardware](#terminate-arm-virtual-hardware-instance)
33

34
# Launch Arm Virtual Hardware Instance
35

36
37
38
There are 2 ways to launch the **Arm Virtual Hardware Instance**, choose one that best fits your work style.  For first timers, we recommend using option 1.
1. [AWS Web Console launch](#launch-using-aws-web-console)
2. [Local Terminal launch](#launch-using-a-local-terminal)
39

40
41
## Launch Using AWS Web Console
To utilize the Arm Virtual Hardware, you will need to create an [AWS Account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) if you don't already have one.
42

43
### Launching the instance in EC2 [(AWS on getting started)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html)
Vincent Coubard's avatar
Vincent Coubard committed
44

45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
1. Go to [EC2](https://console.aws.amazon.com/ec2/v2/) in the AWS Web Console.
1. Select **Launch Instance** which will take you to a wizard for launching the instance.
    > Arm Virtual Hardware for Corstone-300 is available as a public beta on AWS Marketplace. To help you get started, AWS are offering more than 100 hours of free AWS EC2 CPU credits for the first 1,000 qualified users.
    Click here to find out more: https://www.arm.com/company/contact-us/virtual-hardware.

     * **Step 1: Create a Name for your Instance** - To clearly identify the instance you are about to create you will need to apply a descriptive name.  It can be as simple as "J. Doe's AVH Instance".

     * **Step 2: Choose an [Amazon Machine Image (AMI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)**
        * In the Search box, type `Arm Virtual Hardware` and then hit "enter" to find the item called <ins>"Arm Virtual Hardware" - "By Arm"</ins>.
          > Select: **Arm Virtual Hardware By Arm | Version 1.2.3**
          * NOTE: If you do not see the expected items, make sure the <ins>**AWS Marketplace AMIs**</ins> tab is selected.
        * Click on "Select" for that item. This image contains all the software necessary to build and run the Arm IoT Total Solutions.
          * This will raise a subscription page/pop-up titled, **Arm Virtual Hardware**.
          * You will note that the subscription is free from Arm, but <ins>AWS does charge for the costs of the instances themselves according to the pricing chart provided.</ins>
        * You must select "Continue" if you want to move forward.

     * **Step 3: Choose an Instance Type** - Select one of the instance types from the list.
        * We recommend the **c5.large**.
        * **Important:** Charges accrue while the instance is running and to a lesser degree when stopped.
        * Terminating the instance stops any charges from occurring.

     * **Step 4: Key pair (login)**
       * To ensure easy connection when using SSH from a local terminal, it is recommended to create a key pair at this time.
       * Click on **Create new key pair**
       * Enter a descriptive name, e.g. My_Key_Pair_AVH_us-west-2 or simply **MyKeyPair**
         *  To keep track of all the different things you create, we recommend adding your active region to the name.
       * Using the defaults options is fine.
       * Click on **Create key pair**
       * A private key is downloaded, place this in a location you will not forget as it is used later.
       * Once saved, you must fix the permissions of the file wherever you just stored it:
       ```sh
            chmod 400 MyKeyPair.pem
        ```

     * **Step 5: Configure storage** - To ensure enough disk drive space to contain the entire build image.  Set the amount of storage to "1x **24** GiB".

     * **Final Step:** From here you may select **Review and Launch** to move directly to the launch page or continue to configure instance details if you need to set any custom settings for this instance.

### Selecting the instance
84
Once you complete the wizard by initiating the instance launch you will see a page that allows you to navigate directly to the new instance. You may click this link or go back to your list of instances and find the instance through that method.
Vincent Coubard's avatar
Vincent Coubard committed
85

86
87
88
89
90
91
92
93
Whichever way you choose, find your new instance and select its instance ID to open the page to manage the instance.

### Connecting to the instance:
1. Select the instance you want.
2. Select **Connect** to open an SSH terminal session to the instance in your browser.
3. Ensure the User name field is set to `ubuntu`.
4. Select the **Connect** button to open the session.
   * This will put you in a browser window where you will have an SSH terminal window ready for your input.
Vincent Coubard's avatar
Vincent Coubard committed
94

95
You are now ready to build, click [here](#build-and-execute-the-application) to skip to the build instructions.
Vincent Coubard's avatar
Vincent Coubard committed
96

97
98
## Launch Using a local terminal
The instructions in this section, allow you to create and connect to an instance of the Arm Virtual Hardware AMI.  You will be using either you local PC or a server that is under your control.
Vincent Coubard's avatar
Vincent Coubard committed
99

100
1. Install [AWS CLI 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) on your machine.
101

Vincent Coubard's avatar
Vincent Coubard committed
102
2. [Configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) the access key, secret key and region that AWS CLI will use. If your organization uses AWS Single Sign-On, the [configuration process](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) is slightly different. Make sure the region selected matches the region of the SSO service.
103
104

3. Create a new key pair if you have not done so already above.
Vincent Coubard's avatar
Vincent Coubard committed
105
106
107
108
109

```sh
aws ec2 create-key-pair --key-name MyKeyPair
```

110
4. When AWS CLI display the new key pair. Save the key material in a `.pem` file. The file permission must be set to `400`.
Vincent Coubard's avatar
Vincent Coubard committed
111
112
113
114
115
116
117
118
119
120
121

```sh
chmod 400 MyKeyPair.pem
```

5. Launch a new instance with the key pair created. The key pair can be reused to create new instances.

```sh
./scripts/vht_cli.py -k MyKeyPair start
```

122
### Connecting to the instance:
Vincent Coubard's avatar
Vincent Coubard committed
123
124
125
126
127
128
129

1. Get the IP of the instance started

```sh
./scripts/vht_cli.py -k MyKeyPair status
```

130
131
132
2. Connect to the instance using SSH and the private key saved locally.
  * e.g. ssh -i .ssh/MyKeyPair.pem ubuntu@\<**Public IPv4 DNS**\>
    * Look in your instance page at [EC2](https://console.aws.amazon.com/ec2/v2/) AWS Web Console for the **Public IPv4 DNS** value of your Instance.
Vincent Coubard's avatar
Vincent Coubard committed
133
134

```sh
135
ssh -i .ssh/MyKeyPair.pem ubuntu@ec2-xxx.xxx.xxx.xxx.eu-west-1.compute.amazonaws.com
Vincent Coubard's avatar
Vincent Coubard committed
136
137
```

138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
# Re-connect a terminal to the instance
These instructions are only necessary if you have an existing instance and for some reason have terminated your console and need to reconnect for ongoing development.

Choose your terminal connection type (AWS-Web-Console or Local-Console)
* AWS-Web-Console
   * Go to [EC2](https://console.aws.amazon.com/ec2/v2/) in the AWS Web Console.
   * Click on "Instances"
   * Find the instance you created earlier
   * Click on the instance
   * Select Connect to open an SSH terminal session to the instance in your browser.
     * Ensure the User name field is set to ubuntu.
   * Select the Connect button to open the session.
     * This will put you in a browser window where you will have an SSH terminal window ready for your input.
 * Local-Console
     * Open your favorite terminal program or linux shell application and connect to the AVH AMI instance:
     * AWS requires you to use a secure connection, using the instance certificate you downloaded earlier.
     * e.g. ssh -i .ssh/MyKeyPair.pem ubuntu@\<**Public IPv4 DNS**\>
       * Look in your instance page at [EC2](https://console.aws.amazon.com/ec2/v2/) AWS Web Console for the **Public IPv4 DNS** value of your Instance.

      Example
     ```sh
     ssh -i .ssh/MyKeyPair.pem ubuntu@ec2-xxx.xxx.xxx.xxx.eu-west-1.compute.amazonaws.com
     ```
If you do not know your instance value, refer to the **AWS-Web-Console** instructions just above to get this information.

<br>

# Build and execute the application
Vincent Coubard's avatar
Vincent Coubard committed
166

167
168
To update the application, a set of scripts is included to setup the environment,
build applications, run them and test them. These scripts must be executed in the AVH AMI.
Vincent Coubard's avatar
Vincent Coubard committed
169

170
## Connect
171

172
173
174
175
Open your favorite terminal program or linux shell application and connect to the AVH AMI instance.
* AWS requires you to use a secure connection, using the instance certificate you downloaded earlier.
* e.g. ssh -i .ssh/MyKeyPair.pem ubuntu@\<**Public IPv4 DNS**\>
  * Look in your instance page at [EC2](https://console.aws.amazon.com/ec2/v2/) AWS Web Console for the **Public IPv4 DNS** value of your Instance.
Vincent Coubard's avatar
Vincent Coubard committed
176

177
Example
Vincent Coubard's avatar
Vincent Coubard committed
178
```sh
179
ssh -i .ssh/MyKeyPair.pem ubuntu@ec2-xxx.xxx.xxx.xxx.eu-west-1.compute.amazonaws.com
Vincent Coubard's avatar
Vincent Coubard committed
180
181
```

182
## Prepare environment
183

184
185
Clone the repository in the AMI using the following command format:
* git clone \<keyword repository\> && cd \<keyword repository\>
Vincent Coubard's avatar
Vincent Coubard committed
186

187
Example:
Vincent Coubard's avatar
Vincent Coubard committed
188
```sh
189
git clone https://github.com/ARM-software/open-iot-sdk.git && cd open-iot-sdk/examples/ats-keyword
Vincent Coubard's avatar
Vincent Coubard committed
190
191
```

192

Vincent Coubard's avatar
Vincent Coubard committed
193
194
195
Synchronize git submodules, setup ML and apply required patches:

```sh
Vincent Coubard's avatar
Vincent Coubard committed
196
./ats.sh bootstrap
Vincent Coubard's avatar
Vincent Coubard committed
197
198
199
200
201
```

Install additional python dependencies required to run tests and sign binaries:

```sh
202
sudo apt install python3.8-venv
203
204
```

205
206
207
```sh
python3.8 -m pip install imgtool cbor2
```
208
209

```sh
210
python3.9 -m pip install imgtool cffi intelhex cbor2 cbor pytest click
Vincent Coubard's avatar
Vincent Coubard committed
211
212
```

213
214
215
216
217
218
Make python user packages visible in the shell.

```sh
export PATH=$PATH:/home/ubuntu/.local/bin
```

219
## Build
220
221
222
223
224
225

There are currently two applications available: `blink` and `kws`.
The `ats.sh` scripts takes the command `bootstrap`, `build` or `run` as first  parameter.
The second parameter is the name of the application to build or run.
Below we use `kws` as the name of the application, replace it with `blinky` to build that instead.

Vincent Coubard's avatar
Vincent Coubard committed
226
227
228
Build the kws application:

```sh
Vincent Coubard's avatar
Vincent Coubard committed
229
./ats.sh build kws
Vincent Coubard's avatar
Vincent Coubard committed
230
231
```

232
233
234
235
236
237
238
239
240
241
This will by default build the application in the `build` directory for the `Corstone-300` target using the `FreeRTOS` OS. This is equivalent to:

```sh
./ats.sh build kws --target Corstone-300 --rtos FREERTOS --path build
```

To build for Corstone-310 use `--target Corstone-310`. To build using the RTX RTOS implementation use `--rtos RTX`.

You can have multiple builds with different parameters side by side by changing the `--path` parameter to something unique to each build configuration. This speed up the re-build process when working with multiple targets and RTOS.

242
## Run
243

Vincent Coubard's avatar
Vincent Coubard committed
244
Run the kws application:
245

Vincent Coubard's avatar
Vincent Coubard committed
246
```sh
Vincent Coubard's avatar
Vincent Coubard committed
247
./ats.sh run kws
Vincent Coubard's avatar
Vincent Coubard committed
248
249
```

250
251
252
253
254
255
256
The `run` command can take the `--path` switch to run a particular build. It uses `build` directory by default.
This is equivalent to:

```sh
./ats.sh run kws --path build
```

257
## Integration tests
258

Vincent Coubard's avatar
Vincent Coubard committed
259
260
Launch the kws integration tests:
```sh
Vincent Coubard's avatar
Vincent Coubard committed
261
pytest -s kws/tests/test_ml.py
Vincent Coubard's avatar
Vincent Coubard committed
262
263
```

264
# Updating audio data
Vincent Coubard's avatar
Vincent Coubard committed
265

266
267
The audio data streamed into the Arm Virtual Hardware is read from the file `test.wav` located at the root of the repository. It can be replaced with another audio file with the following configuration:
- Format: PCM
Vincent Coubard's avatar
Vincent Coubard committed
268
269
- Audio channels: Mono
- Sample rate: 16 bits
270
- Sample rate: 16kHz
Vincent Coubard's avatar
Vincent Coubard committed
271

272
# Continuous integration setup
Vincent Coubard's avatar
Vincent Coubard committed
273

274
Each Total Solution application has been built and verified using a continuous integration process. Inside each Total Solution application folder there are examples for CI systems such as GitHub. GitLab example files are coming soon
275
276
277

# Setting up AWS connectivity

278
279
280
281
The Keyword Detection application will attempt to connect to AWS IOT and report ML inference results through an MQTT connection.

To connect to the AWS cloud service you will need to setup an IoT Thing and then set the AWS credentials of the IoT Thing within the Application. You will need to create an [AWS Account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) if you don’t already have one.

282
283
284

## AWS account IoT setup

285
286
287
288
The instructions below will allow the Application to send messages to the Cloud via MQTT as well as enable an Over-the-Air update.

  > Note: Due to AWS restrictions, you must ensure that when logging into the [AWS IoT console](https://console.aws.amazon.com/iotv2/) you ensure that it is using the same **Region** as where you created your AMI instance.  This restriction is documented within the [MQTT Topic](https://docs.aws.amazon.com/iot/latest/developerguide/topics.html) page in the AWS documentation.

289
290
291
### Create an IoT thing for your device

1. Login to your account and browse to the [AWS IoT console](https://console.aws.amazon.com/iotv2/).
292
293
   * If this takes you to AWS Console, click on **Services** above and then click on **IoT Core**.
   * Ensure your **Region** is correct.
294
2. In the left navigation pane, choose **Manage**, and then choose **Things**.
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
   * These instructions assume you do not have any IoT things registered in your account.
   * Choose **Register** or **Create things**.
3. On the **Create things** page, choose **Create single thing**.
4. On the **Specify thing properties** page, type a **Thing name** for your thing (for example `MyThing_eu_west_1`), and then choose **Next**.
   * Adding the region name helps to remind you which region the thing and topic is attached to.
   * You will need to add the name later to your C code.
   * There is no need to add any **addition configuration** or **Device Shadow** information.
5. On the **Configure device certificate** page, choose **Auto-generate a new certificate** and then click **Next**.
6. Skip the **Attach policies to certificate page** for now.
   * You will attach a certificate to the Thing in a later step.
7. Download your all the keys and certificates by choosing the **Download** links for each.
   * Click on all the **Download** buttons and store these files in a secure location as you will use them later.
   * Make note of the certificate ID. You need it later to attach a policy to your certificate.
   * Click on **Done** once all items have downloaded.
8. AWS used to have you choose **Activate** to activate your certificate.
   * This is no longer an option.
311
312
313
314

### Create a policy and attach it to your thing

1. In the navigation pane of the AWS IoT console, choose **Secure**, and then choose **Policies**.
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
2. On the **Policies** page, choose **Create Policy**.
   * These instructions assume you do not have any **Policies** registered in your account,
3. On the **Create Policy** page
   * Enter a name for the policy.
   * In the **Policy document** box, enter 4 separate policies with the following values:
     * **Policy effect** - Choose **Allow** for all entries.
     * **Policy Action** - Use the values below, 1 for each entry.
       * **iot:Connect**
       * **iot:Publish**
       * **iot:Subscribe**
       * **iot:Receive**
   * The **Policy resource** field requries an **ARN**. Sometimes this box will be auto-filled with your credentials.
     * If no value exists, use the following format:    (arn:aws:iot:**region:account-id:\***)
       * region (e.g. eu-west-1)
       * account-id ... This is your **Acount ID Number**.
         * You can see this usually on the top right corner that shows your login name.
       * e.g. *arn:aws:iot:eu-west-1:012345678901:*
     * Replace the part, or add, after the last colon (`:`) with `*`.
       * e.g. *arn:aws:iot:eu-west-1:012345678901:\**
     * Click on **Create**.

<br>

   > NOTE - The examples in this document are intended for development environments only.  All devices in your production fleet must have credentials with privileges that authorize only intended actions on specific resources. The specific permission policies can vary for your use case. Identify the permission policies that best meet your business and security requirements.  For more information, refer to Example policies and Security Best practices of your Cloud-Service-Provider.

<br>

342
4. In the left navigation pane of the AWS IoT console, choose **Secure**, and then choose **Certificates**. You should see the certificate that you have created earlier.
343
344
345
346
347
348
349
350
   * Use the ID in the front of the certificate and key files that you downloaded earlier to identify your certificate.
   * Click on your certificate name to take you to your certificate.
6. Click on **Actions** -> **Attach policy** or look down the page and click on **Attach policies**.
7. In the **Attach policies to certificate(s)** window
   * Find your policy and click on it it.
     * Even though you may enable more than one policy, for now we use the single policy you created earlier.
   * Click **Attach policies**.
   * You will now see your policy listed on the Certificate page.
351
352

## Configure the application to connect to your AWS account
353
Now that you have created an AWS Thing and attached the certificates and policies to it, the representative values must be added to your application to ensure connectivity with your AWS account.
354

355
Within the application directory that you are using, edit the `bsp/default_credentials/aws_clientcredential.h` file and set values for specified user defines called out below.
356
357
358

`clientcredentialMQTT_BROKER_ENDPOINT`

359
360
361
362
* Set this to the Device data endpoint name of your amazon account.
* To find this go to the navigation pane of the [AWS IoT console](https://console.aws.amazon.com/iotv2/), choose **Settings** (bottom left hand corner).
* On the **Settings** page, in the **Device data endpoint** section of the page look for **Endpoint**.  (e.g. `a3xyzzyx-ats.iot.us-east-2.amazonaws.com`).
  * Note the region may be different than these instructions.  It should match where your thing and policy were created due to the MQTT Topic restrictions discussed above.
363
364
365

`clientcredentialIOT_THING_NAME`

366
367
368
369
* Set this to the name of the thing you set (e.g. MyThing).

Save and close the file.

370

Vincent Coubard's avatar
Vincent Coubard committed
371
Next insert the keys that are in the certificates you have downloaded when you created the thing. Edit the file `bsp/default_credentials/aws_clientcredential_keys.h` replacing the existing keys with yours.
372
373
374

`keyCLIENT_CERTIFICATE_PEM`

375
* Replace with contents from `<your-thing-certificate-unique-string>-certificate.pem.crt`.
376
377
378

`keyCLIENT_PRIVATE_KEY_PEM`

379
* Replace with contents from `<your-thing-certificate-unique-string>-private.pem.key`.
380
381
382

`keyCLIENT_PUBLIC_KEY_PEM`

383
384
385
* Replace with contents from `<your-thing-certificate-unique-string>-public.pem.key`.

Save all files and rebuild the application.
386
387
388
389
390
391
392

## Observing MQTT connectivity

To see messages being sent by the application:
1. Login to your account and browse to the [AWS IoT console](https://console.aws.amazon.com/iotv2/).
2. In the left navigation panel, choose **Manage**, and then choose **Things**.
3. Select the thing you created, and open the **Activity** tab. This will show the application connecting and subscribing to a topic.
393
394
395
396
397
398
4. Click on the **MQTT test client** button. This will open a new page.
5. Click on **Subscribe to a topic**.
6. In the **Subscription topic** field enter the topic name which is a concatenation of the name of your thing (set in `clientcredentialIOT_THING_NAME`) and `/ml/inference`
   * e.g. if you thing name is MyThing then it's `MyThing/ml/inference`
8. In the **MQTT payload display** combo box select `Display payloads as strings (more accurate)`
9. Click the **Subscribe** button. The messages will be shown below within this same page.
399

Vincent Coubard's avatar
Vincent Coubard committed
400
401
# OTA firmware update

402
Total-Solution applications that have CSP connectivity enabled may also have Over-The-Air (OTA) update functionality. The application will check for updates from the AWS Cloud at boot time to check if there is an update pending.  If an update is available, the application will stop ML processing, download the new firmware, and then apply the new firmware if the version number indicates the image is newer. To make such a version available you need to prepare the update binary (this is part of the build process) and create an OTA job on AWS.
Vincent Coubard's avatar
Vincent Coubard committed
403

Vincent Coubard's avatar
Vincent Coubard committed
404
405
## Creating updated firmware

406
407
408
As part of the application build process, an updated firmware image will be created that will only differ in version number. That is enough to demonstrate the OTA process using a newly created image.

If you want to add other changes you should copy the original binary elsewhere before running the build again with your changes as the same build directory is used for both.  This is to ensure you have the original binary to compare against any new version you build.
Vincent Coubard's avatar
Vincent Coubard committed
409

410
411
412
For example, the updated binary is placed in `build/kws/kws_signed_update.bin`. The updated binary is already signed and it is the file you will need to upload to the Amazon S3 bucket in the next section.

Upon completion of the build and signing process the <ins>signature string will be echoed to the terminal</ins>. This will be needed in the next step.
Vincent Coubard's avatar
Vincent Coubard committed
413
414
415

## Creating AWS IoT firmware update job

416
417
418
419
420
421
422
423
The instructions below use the keyword spotting name, kws, as an example.  Replace kws with the application name in the build instructions that you followed.

1. Follow the instructions at: [Create an Amazon S3 bucket to store your update](https://docs.aws.amazon.com/freertos/latest/userguide/dg-ota-bucket.html)
  * Use the default options wherever you have a choice.
  * For simplicity, use the same region for the bucket as where your Instance is located.
2. Follow the instructions at: [Create an OTA Update service role](https://docs.aws.amazon.com/freertos/latest/userguide/create-service-role.html)
3. Follow the instructions at: [Create an OTA user policy](https://docs.aws.amazon.com/freertos/latest/userguide/create-ota-user-policy.html)
4. Go to AWS IoT web interface and choose **Manage** and then **Jobs**
Vincent Coubard's avatar
Vincent Coubard committed
424
425
426
427
428
5. Click the create job button and select **Create FreeRTOS OTA update job**
6. Give it a name and click next
7. Select the device to update (the Thing you created in earlier steps)
8. Select `MQTT` transport only
9. Select **Use my custom signed file**
429
430
431
432
433
10. Select upload new file and select the signed update binary (`build/kws/kws_signed_update.bin`)
11. Select the S3 bucket you created in step 1. to upload the binary to
12. Paste the signature string that is echoed during the build of the example (it is also available in `build/kws/update-signature.txt`).
13. Select `SHA-256` and `RSA` algorithms.
14. For **Path name of code signing certificate on device** put in `0` (the path is not used)
Vincent Coubard's avatar
Vincent Coubard committed
434
435
436
15. For **Path name of file on device** put in `non_secure image`
16. As the role, select the OTA role you created in step 2.
17. Click next
437
438
439
440
441
442
18. Create an ID for you Job
19. Add a description
20. **Job type**, select *Your job will complete after deploying to the selected devices/groups (snapshot).*
21. Click next, your update job is ready and running - next time your application connects it will perform the update.

<br>
Vincent Coubard's avatar
Vincent Coubard committed
443

Vincent Coubard's avatar
Vincent Coubard committed
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
# Setting up Azure connectivity

The Keyword Detection application can connect to an Azure IoT Hub and report ML inference results through that connection.

To connect to the Azure IoT Hub cloud service you will need to setup a device and then set the device connection string within the Application. You will need to create an [Azure Account](https://azure.microsoft.com/) if you don’t already have one.


## Azure IoT Hub setup

To create a new Azure IoT Hub and one device within it through the web portal follow the [instructions](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal) provided by Azure.

## Configure the application to connect to your Azure IoT Hub

Now that you have created a device in your IoT Hub, the application must be configured to connect to the Azure IoT Hub with the credentials of the device created.

Within the application directory that you are using, edit the `bsp/default_credentials/iothub_credentials.h` file.

You must set the define `IOTHUB_DEVICE_CONNECTION_STRING` to the value of the device's `Primary Connection String`. This value can be [retrieved](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal#register-a-new-device-in-the-iot-hub) in the portal.


## Build the application to connect to your Azure IoT Hub

The application selects a cloud client (Aws or Azure) at build time. This is achieved by adding the flag `-e <AZURE|AWS>` to the build command line.
To build a version of kws connecting to the Azure cloud on Corstone-300 and using FreeRTOS, use the following command line:

```sh
./ats.sh build kws -e AZURE
```

## Monitoring messages sent to your Azure IoT Hub

The Azure web portal does not offer monitoring facilities to visualize packets received out of the box.
To monitor packets, you can use the [Azure IoT Tools](https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.azure-iot-tools) VS Code extension and follow these [instructions](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-vscode-iot-toolkit-cloud-device-messaging#monitor-device-to-cloud-messages).

To monitor activity (connection, disconnection, ...) follow the [reference instructions](https://docs.microsoft.com/en-us/azure/iot-hub/monitor-iot-hub).

480
# Terminate Arm Virtual Hardware Instance
Vincent Coubard's avatar
Vincent Coubard committed
481

482
483
484
When you are done using the AMI instance at the end of the day, you need to make sure you shut it down properly or you may be charged for usage you did not actually use.  There are 2 ways to do this action (pick one):

## Stopping the instance in EC2
Vincent Coubard's avatar
Vincent Coubard committed
485
486
487
1. Go to [EC2](https://console.aws.amazon.com/ec2/v2/) in the AWS Web Console.
2. Select the instance to stop.
3. Click on `Instance state` and select `Stop Instance` in the drop down menu.
488

Vincent Coubard's avatar
Vincent Coubard committed
489
## Stopping the instance using a local terminal
490
Execute the following script located in the application repository.
Vincent Coubard's avatar
Vincent Coubard committed
491
492
493
494
495

```sh
./scripts/vht_cli.py -k MyKeyPair stop
```

Vincent Coubard's avatar
Vincent Coubard committed
496
497
# Source code overview

498
- `bsp`: Arm Corstone-300 subsystem platform code and AWS configurations.
Vincent Coubard's avatar
Vincent Coubard committed
499
- `lib`: Middleware used by IoT Total Solution.
Vincent Coubard's avatar
Vincent Coubard committed
500
501
  - `AWS`: OTA and PKCS11 integration with AWS IoT SDK.
  - `VHT`: Virtual streaming solution for Arm Virtual Hardware.
Vincent Coubard's avatar
Vincent Coubard committed
502
503
504
- `blinky`: Blinky application.
  - `blinky/main_ns.c`: Entry point of the blinky application
- `kws`: Keyword detection application.
505
  - `kws/source/main_ns.c`: Entry point of the kws application.
Vincent Coubard's avatar
Vincent Coubard committed
506
507
  - `kws/source/aws_demo.c`: AWS IoT specific code of the kws application.
  - `kws/source/azure_demo.c`: Azure IoT Hub specific code of the kws application.
Vincent Coubard's avatar
Vincent Coubard committed
508
  - `kws/source/blinky_task.c`: Blinky/UX thread of the application.
509
  - `kws/source/ml_interface.c`: Interface between the virtual streaming solution and tensor flow.
Vincent Coubard's avatar
Vincent Coubard committed
510
511
  - `kws/source/ethosu_platform_adaptation.c`: RTOS adapatation for the Ethos U55.
  - `kws/ota`: Connector between AWS OTA and the application.
Gergely Nagy's avatar
Gergely Nagy committed
512
- `mlia`: Integration the ML Inference Advisor, using a simple wrapper script.sh to install and run the tool on given models.
Vincent Coubard's avatar
Vincent Coubard committed
513

Vincent Coubard's avatar
Vincent Coubard committed
514
515
# ML Model Replacement

516
All the ML models supported by the [ML Embedded Eval Kit](All the models supported ) are available to applications. The first step to use another module is to generate sources files from its labels and `.tflite` model.
517

Vincent Coubard's avatar
Vincent Coubard committed
518
```sh
519
520
521
522
523
524
525
526
527
528
529
530
531
# Enter the ml example repository
cd lib/ml-embedded-evaluation-kit/

ML_GEN_SRC="generated/<model name>/src"
ML_GEN_INC="generated/<model name>/include"

mkdir -p $ML_GEN_SRC
mkdir -p $ML_GEN_INC

./lib/ml-embedded-evaluation-kit/resources_downloaded/env/bin/python3 scripts/py/gen_labels_cpp.py \
    --labels_file resources/<model name>/labels/<label file>.txt \
    --source_folder_path $ML_GEN_SRC \
    --header_folder_path $ML_GEN_INC \
532
    --output_file_name <model name>_labels
533
534
535
./resources_downloaded/env/bin/python3 scripts/py/gen_model_cpp.py \
    --tflite_path resources_downloaded/<model name>/<model>.tflite \
    --output_dir $ML_GEN_SRC
Vincent Coubard's avatar
Vincent Coubard committed
536
537
```

538
Models available are present in `./lib/ml-embedded-evaluation-kit/resources_downloaded`.
539
Pre-integrated source code is available from the `ML Embedded Eval Kit` and can be browsed from `./lib/ml-embedded-evaluation-kit/source/use_case`.
540
541
542

Integrating a new model means integrating its source code and requires update of the build files.

Vincent Coubard's avatar
Vincent Coubard committed
543
544
# ML Embedded Eval Kit

545
The Arm ML Embedded Evaluation Kit , is an open-source repository enabling users to quickly build and deploy embedded machine learning applications for Arm Cortex-M55 CPU and Arm Ethos-U55 NPU.
Vincent Coubard's avatar
Vincent Coubard committed
546

547
With ML Eval Kit you can run inferences on either a custom neural network on Ethos-U microNPU or using availble ML applications such as [Image classification](https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ml-embedded-evaluation-kit/+/c930ad9dc189d831ac77f716df288f70178d4c10/docs/use_cases/img_class.md), [Keyword spotting (KWS)](https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ml-embedded-evaluation-kit/+/c930ad9dc189d831ac77f716df288f70178d4c10/docs/use_cases/kws.md), [Automated Speech Recognition (ASR)](https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ml-embedded-evaluation-kit/+/c930ad9dc189d831ac77f716df288f70178d4c10/docs/use_cases/asr.md), [Anomaly Detection](https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ml-embedded-evaluation-kit/+/c930ad9dc189d831ac77f716df288f70178d4c10/docs/use_cases/ad.md), and [Person Detection](https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ml-embedded-evaluation-kit/+/HEAD/docs/use_cases/visual_wake_word.md) all using Arm Fixed Virtual Platform (FVP) available in Arm Virtual Hardware.
Vincent Coubard's avatar
Vincent Coubard committed
548

Vincent Coubard's avatar
Vincent Coubard committed
549
550
# Known limitations

551
552
- Arm compiler 6 is the only compiler supported.
- Accuracy of the ML detection running alongside cloud connectivity depends on the performance of the EC2 instance used. We recommend to use at least a t3.medium instance.
553
- Arm Corstone-300 subsystem simulation is not time accurate. Performances will differ depending on the performances of the host machine.
Vincent Coubard's avatar
Vincent Coubard committed
554
555


Vincent Coubard's avatar
Vincent Coubard committed
556
557
# Future Enhancements
- [AWS Partner Device Catalog Listing](https://devices.amazonaws.com/) (leveraging Arm Virtual Hardware)
558

Vincent Coubard's avatar
Vincent Coubard committed
559
560
561

# Other Resources

562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
| Repository                                                                                                    | Description                                                                                                                      |
|---------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
| [Arm AI Ecosystem Catalog](https://www.arm.com/why-arm/partner-ecosystem/ai-ecosystem-catalog)                | Connects you to the right partners, enabling you to build the next generation of AI solutions                                    |
| [Arm IoT Ecosystem Catalog](https://www.arm.com/why-arm/partner-ecosystem/iot-ecosystem-catalog)              | Explore Arm IoT Ecosystem partners who can help transform an idea into a secure, market-leading device.                          |
| [Arm ML Model Zoo](https://github.com/ARM-software/ML-zoo)                                                    | A collection of machine learning models optimized for Arm IP.                                                                    |
| [Arm Virtual Hardware Documentation](https://mdk-packs.github.io/VHT-TFLmicrospeech/overview/html/index.html) | Documentation for [Arm Virtual Hardware](https://www.arm.com/products/development-tools/simulation/virtual-hardware)             |
| [Arm Virtual Hardware source code](https://github.com/ARM-software/VHT)                                       | Source code of[Arm Virtual Hardware](https://www.arm.com/products/development-tools/simulation/virtual-hardware)                 |
| [FreeRTOS Documentation](https://freertos.org/a00104.html#getting-started)                                    | Documentation for FreeRTOS.                                                                                                      |
| [FreeRTOS source code](https://github.com/FreeRTOS)                                                           | Source code of FreeRTOS.                                                                                                         |
| [AWS IoT](https://docs.aws.amazon.com/iot/latest/developerguide/what-is-aws-iot.html)                         | Documentation for AWS IoT.                                                                                                       |
| [Trusted Firmware-M Documentation](https://tf-m-user-guide.trustedfirmware.org/)                              | Documentation for Trusted Firmware-M.                                                                                            |
| [Trusted Firmware-M Source code](https://git.trustedfirmware.org/TF-M/trusted-firmware-m.git)                 | Source code of Trusted Firmware-M.                                                                                               |
| [Mbed Crypto](https://github.com/ARMmbed/mbedtls)                                                             | Mbed Crypto source code.                                                                                                         |
| [MCU Boot](https://github.com/mcu-tools/mcuboot)                                                              | MCU Boot source code.                                                                                                            |
| [ml-embedded-evaluation-kit](https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ml-embedded-evaluation-kit) | ML Embedded eval kit source code                                                                                             |
| Support                                                                                                       | A [community.arm.com](http://community.arm.com/) forum exists for users to post queries.                                         |
Vincent Coubard's avatar
Vincent Coubard committed
578
579
580

# License and contributions

581
The software is provided under the Apache-2.0 license. All contributions to software and documents are licensed by contributors under the same license model as the software/document itself (e.g. inbound == outbound licensing). Open IoT SDK may reuse software already licensed under another license, provided the license is permissive in nature and compatible with Apache v2.0.
Vincent Coubard's avatar
Vincent Coubard committed
582

Vincent Coubard's avatar
Vincent Coubard committed
583
Folders containing files under different permissive license than Apache 2.0 are listed in the LICENSE file.
584

585
586
587
# Security

Information on security considerations for an end user application can be found [here](https://gitlab.arm.com/iot/open-iot-sdk/sdk/-/blob/main/docs/guidelines/Security.md).
588
589
590
591
592
## Security issues reporting

If you find any security vulnerabilities, please do not report it in the GitLab issue tracker. Instead, send an email to the security team at arm-security@arm.com stating that you may have found a security vulnerability in the IoT Total Solution Keyword Detection project.

More details can be found at [Arm Developer website](https://developer.arm.com/support/arm-security-updates/report-security-vulnerabilities).