AWS IoT Basics – Persisting IoT Data to DynamoDB

Introduction

AWS IoT is a managed cloud platform that lets connected devices – cars, light bulbs, sensor grids, and more – easily and securely interact with cloud applications and other devices.

These devices generates data and we can use AWS IoT services to collect this data. Once data is collected, a typical requirement is to persist it for analysis, reporting or for some other usages.

AWS DynamoDB is the leading fully managed, NoSQL database service in the world all managed by AWS. It is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. DynamoDB offers built-in security, continuous backups, automated multi-region replication, in-memory caching, and data export tools.

In this post, we will learn how to persist AWS IoT data to DynamoDB with some simple examples.

Setting The Scene

Following diagram shows the main components and data flow. It represents a very basic IoT setup. A typical IoT setup normally have field devices, these field devices have data or they collect from their environment.

For our demo purposes we will be simulating these devices using .NET Core and NodeJS applications. We will generate some simulate data and sent it to AWS IoT Core Topic . Then we will setup an IoT rule to persist the data to DynamoDB table.

You can use .NET Core, NodeJS, Java or python for device simulation purposes.

You will also need to do some basic setup to connect the application to AWS IoT Core.

I’ve already covered connection to AWS IoT part in following two posts. Check the NodeJS or .NET Core examples as per your choice:

• AWS IoT – Basics (NodeJS Example)
• AWS IoT With .NET Core (MQTT)

Furthermore, if you are new to Document (NoSQL) databases, you can check following post for some basics:

.NET Core, PostgreSQL and Document Database

Next, lets see how data is generated for our demo.

Simulated Data

In real world, this data shall be generated from the field devices e.g. sensors data or GPS data etc.

For demo purposes, we will be generating GPS, and temperature data (assuming that a vehicle is generating this data) along with timestamp.

I’ve added the simulator code to earlier post’s code repositories and you can simply download the code from Git Repos and run it as it is or modify it as per your scenario.

NodeJS Simulator sample:

Here is the NodeJS code which will generate simulated data and sent it to AWS IoT Core:

Similar to above, I’ve added the C# version of the simulator code in the corresponding repo:

You can run either of the example code and then subscribe to topic In AWS console. Following picture that the data is received as expected:

With this, data sending part, is finished and now we will focus on how to persist this data to DynamoDB.

Persisting Data to DynamoDB

DynamoDB is big topic and I will cover it in details in some later posts. However, in today’s demo, we are not going into those details and we are not going to write any DynamoDB data persistent code.

Instead we will use IoT Core rule to perform some action on received messages.

IoT Core rules allow you to react to incoming messages.

There are many things you can do once you receive a message e.g.

• Persist to DynamoDB table.
• Send a message to IoT Analytics channel.
• Send a message to SQS Queue.
• Send a message to S3 bucket
• ..and others..

We will be persisting messages to DynamoDB table.

Here is the breakdown of this part:

• Create an IoT Rule.
• Create a DynamoDB table.
• Configure an Action (persist message) in IoT rule which will target the DynamoDB table.
• Create/Select an IAM Role for permission.
• Check/verify using AWS console that data is persisted to DynamoDB table.

Create an IoT Rule

We can create an IoT rule from AWS IoT Core dashboard as shown in the following pictures:

This will open-up the following window, where we can add a rule name and also SQL query to select messages from the IoT Topic:

Once selection criteria is setup, next we can add an action for the messages received by clicking Add Action button on the same page:

This will open-up following dialog with action to select:

Once we select an action, we can configure the action. As, we want to persist data to DynamoDB, we need to select a table or create a new table as shown below:

DynamoDB Table Setup

Table setup is very simple. We gave it a name and from our incoming data, selected two fields are partition-key and sort-key (more on these in later posts)

I also changed the AutoScaling and Read/Write capacities as shown below. You can keep those unchanged if you want:

With these changes, we can click Create table button as shown below:

The following picture shows that our table is created :

Once table is Created, we can go back to configuration process and select it from drop down:

We also need to Create/Select a Role.

Create Role

We can create a role by providing a name and then simply clicking the Create role button:

We can see in the following picture that role is setup properly and then we can simply click Add action button:

Here is the action added to IoT rule.

Testing

Now, If we ran simulator application again and then check the DynamoDB table, we can see that the data is persisted:

Note, that for data persistence part, we don’t have to write any code. We simply configured a Rule and now whenever AWS IoT Core will receive data on this topic, it will be persisted to DynamoDB table.

Summary

We can configure AWS IoT rules to perform certain actions to process incoming data. We saw a simple example of persisting data to DynamoDB.

You can download the source code (simulator) from the git repositories of earlier posts:

• AWS IoT – Basics (NodeJS Example)
• AWS IoT With .NET Core (MQTT)

Let me know if you have some questions or comments. Till next time, Happy Coding.

My Recent Books