AWS DynamoDB Basics

Introduction

AWS DynamoDB is a fully managed NoSQL database service all managed by AWS.

As per the official website “Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. DynamoDB offers built-in security, continuous backups, automated multi-region replication, in-memory caching, and data export tools”.

In this post, we will learn some simple ways to work with DynamoDB. We will learn its basic components, how to work with tables and we will also build a .NET Core Web API to query the data from DynamoDB table.

This post will setup a base for us on which we will learn more features of DynamoDB.

DynamoDB Components

DynamoDB has three basic components:

• Item (equivalent to a row in a RDBMS)
• Attribute (equivalent to a column in RDBMS)
• Table (same in RDBMS. It is a collection of items)

Here is an item (item1) in JSON Format:

{
  "rideId": 96,
  "temperature": 79.60000000000001,
  "timestamp": 1641580491910,
  "location_lat": 46.63022101916666,
  "location_lon": -96.85285700499999
  "driver": "John Doe"
}

DynamoDB Keys

Primary-key / Partition-Key/Sort-Key

Lets assume rideId as a unique-key which identified this item or you may have something like customerId to uniquely identify an item. These special keys are called Partition-Keys (same concept of PK in RDBMS).

Lets see JSON payload of another item (item-2):

{
  "rideId": 96,
  "temperature": 79.60000000000001,
  "timestamp": 1641580491910,
  "location_lat": 45.63022101916666,
  "location_lon": -97.85285700499999
  "driver": "John Doe"
}

Now if we see both items, rideId is not unique in our model and if we had selected rideId as a unique key then we would have partition-key collision and we won’t be able to update the table with new item.

How can we resolve this issue?

We an select another special attribute along-side the partition-key called a “Sort Key” to ensure uniqueness of an item based on combination of partition-key as well as sort-key (a composite key). You may already familiar with this type of key from RDBMS.

So for our sample case, we will go with the combination of rideId (partition key) and timestamp (sort key) which will result in unique composite primary key.

So based on your use-case, you can go with single primary key (partition key) e.g. unique customerId etc or based on composite-primary-key if your document model is structured in that way.

Understanding DynamoDB partitions

Data storage on hardware partitions are normally managed by AWS. However, with the use of partition-key, you are able to somehow control where and how data is stored.

The partition key is used to spread data across partitions for scalability.

DynamoDB data types

Following are DynamoDB data types:

• Scalar (one value at a time e.g. string, int, boolean etc)
• Document (multiple values of different types)
• Set (multiple values of same type)

Scalar Example

{
  "rideId": 96,
  "temperature": 79.60000000000001,
  "timestamp": 1641580491910,
  "location_lat": 45.63022101916666,
  "location_lon": -97.85285700499999
  "driver": "John Doe"
}

Document Example

notice hobbies array contains both string and number types.

{
  "name": "bob",
  "hobbies": [
    "movies",
     3.14,
     "dusseldorf"
  ]
}

{
  "name": "Joe",
  "kitchenItems": [
    {"spoons": 5}, //map document type
	"pan",
	"knife"
  ]
}

The second item shows a simple variation that it has a map document and string item in the array.

Sets Example

Here are some examples for Sets:

{
  "name": "Bob",
  "LotNums": [
   345,
   450,
   679
  ]
}

{
  "name": "Alice",
  "kids": [
   "Walter",
   "Jessy",
   "Emma"
  ]
}

Here is another Set example:

{
  "id": 1
  "secret": "22MVO...", //Binary Scalar
  "pics": [
   "auf==",  // Base64 encode set
   "xjdd==",
   "bcda=="
  ]
}

As you can see that DynamoDB is very flexible.

One thing to keep in mind is unlike in RDBMS, a fixed schema is not required when creating and later when storing items within it. Only primary key attribute have to be present.

Lets say we have table called where we say that name attribute is the only partition key. In that case, we can store all the following items in same table:

{
  "name": "Alice",
  "kids": [
   "Walter",
   "Jessy",
   "Emma"
  ]
}

{
  "name": "Jane"  
}

{
  "name": "Joe",
  "kitchenItems": [
    {"spoons": 5}, //map document type
	"pan",
	"knife"
  ]
}

Our sample use case DynamoDB Table (truck_sensor)

For the demos, I have already created a DynamoDB table called “truck_sensor” using AWS Console to store GPS and temperature sensor data:

I have covered table creation part in earlier post. You can follow the same steps to create table for your use cases.

Setting Up .NET Core App

I’ve added a .NET Core Web API project to existing solution for the demos. This is the same code base from the earlier post (sending IoT data and I am just adding this new web API project to existing code.

Add Nuget Packages

Next, we will add following nuget packages to the WebAPI application:

AWSSDK.DynamoDBv2

This package provides you with .NET APIs for DynamoDB:

AWSSDK.Extensions.NETCore.Setup

This package allows you to get your credentials through Dependency Injection and configure DynamoDB client to be used in the application.

Here we can see that nuget packages are installed to the web app:

Configure Startup.cs

We can update the startup file with following statements:

These will configure AWSDynamoDB Client setup for our application.

Creating an IAM Role

We’ll also need to create an IAM role.

An IAM Role allows us to create credentials that we can use outside of the AWS console to gain access to services in our AWS account.

In our case, we want to create and give access to our truck_sensor DynamoDB table.

I’ve create a user and a group which has DynamoDBFullAccess and then associate user with this group:

Here is the group info:

Also, note the SECRET ACCESS KEY on this wizard, we’ll be using it shortly.

Now, we have these keys, we need to add those to our credentials file that our DynamoDB Client will read from.

A good way to get this all setup correctly is through AWS CLI. You can install and then use aws configure command to setup credential file. We will need to provide those keys, which copied earlier when setup the role:

Check this post for some more details about configuring AWS CLI.

here are the directory view of files:

AWS DynamoDB client will use this file to gain access.

With this access is configured for DynamoDB service from our application.

Working with DynamoDB

DynamoDB SDK for .NET offers three models that can be used:

• Object Persistence Model
• Document Model
• Low Level Model

We’ll be using object persistence model for our demos. In some later posts, we will cover other models as well.

Object Persistence Model

Following are few points about OPM:

• Easiest model to code against.
• Wrapper around Low Level Model.
• We are able to store, load and query DynamoDB using this model.
• This model enable us to map our client-side classes to our DynamoDB table.
• Each object instance then maps to an item in the corresponding table.

One missing feature in this model is the ability to create, update and delete DynamoDB tables.

Lets continue by reading data in our .NET Core API application.

Getting Data From DynamoDB Table

Earlier we configured our application to interact with AWS and DynamoDB. As mentioned before, I have an existing DynamoDB table with some data. Lets see a basic example to read this data:

Data Model

Here is the item JSON for your reference:

{
  "rideId": 96,
  "temperature": 79.60000000000001,
  "timestamp": 1641580491910,
  "location_lat": 45.63022101916666,
  "location_lon": -97.85285700499999
  "driver": "John Doe"
}

I created a model class to match the JSON payload:

It is a simple C# class which is decorated with various attributes. For example Class is decorated with truck_sensor value to match to DynamoDB table.

Also, see the DynamoDBProperty attribute to work with camelCasing for property names.

This class will hold the data which we will read from DynamoDB table.

Data Repository

Here is the data repository class which wraps DynamoDB Client and populate date model collection:

GetAllItems method contains DynamoDB client code to query the DynamoDB Table for all items.

We can register the repository with DI in the startup class:

API Controller

Following is the controller code which is self explanatory. The TruckSensorRepo is injected using constructor injection and we then simply used it to read data from the DynamoDB Table and returned results to the caller (postman tool):

We can now call this endpoint using postman:

As you can that data is retrieved as expected.

Summary

In this post we learned some basics of NoSQL databases and AWS managed NoSQL service DynamoDB. We then set up a .NET Core applications and installed NuGet packages from AWS to work with DynamoDB.

To connect with DynamoDB, we created an IAM Role with necessary permissions.

We then used object persistence model of SDK and model the data using C# class and some attributes and used DynamoDB client to read and populate this class with data.

We then wrapped the data access code in a repository and wired it up with dependency injection.

This repository was injected to controller where we created an endpoint to use this repository to read and return the data from DynamoDB table.

As we have now basic setup done, in upcoming posts, we will see more operations with DynamoDB and work with real world use cases.

You can download the source code from this git repository.

Let me know, if you have some questions or comments. Till next time, happy coding.

My Recent Books