.NET on AWS – Persisting Application Data to S3 (Part-1)

Introduction

Amazon S3 is the most commonly used managed storage solution in AWS.

It provides object storage in a highly scalable and secure way. AWS guarantees 11 9s for its durability.

Objects stored in S3 are shared access object; shared meaning, they can be accessed by different clients at same time.

S3 provides low latency for data and it has high throughput (able to move data in and out S3 quickly).

S3 is highly available, durable and can encrypt data. S3 provides access management, lifecycle management and ability to query-in-place (no need to move to data lake).

Static web-site hosting is another very popular feature of S3.

Following storage tiers are offered by S3

• Standard
• Intelligent Tiering (save cost)
• Standard IA
  • storage cost is reduced.
  • cost for read.
• 1Zone IA
  • lowest price, but does not offer much availability/durability.

For more information, please visit the official website on this link.

Application Requirements

Persisting application data is a common requirement in software/services development. There are all kind of data storage solutions available. RDBMS and NoSQL solutions are very popular for some applications and same time storing data on files is also very common.

For our application, lets assume, we are asked to store application data on files and we decided to use AWS S3 storage service for this purpose.

Technical Stack

We will be building a web API using .NET6 tech stack. Application will allow user to perform typical CRUD operations for Note entity. A very simple application with bare minimum to keep our focus on data storage on AWS S3 from a .NET application.

The source code for the application is available on this GitHub repository.

We will be performing various actions on AWS cloud and our .NET application as follows

• AWS Resources Setup
  • Create S3 bucket
  • Create an IAM user and access keys.
  • Create a Policy and attach with User.
• .NET6 Application Wiring
  • Create a .NET6 application.
  • Wire .NET6 application with AWS resources.
• .NET6 Application Code
  • Write Application code for CRUD operations for S3 objects.

Lets start with these steps next

Creating an S3 bucket

S3 buckets are created in a region. S3 bucket name has to be globally unique. The reason for that is that DNS is record is going to be assigned to each bucket we create.

For all other fields, we can use the defaults for now.

Once bucket is created, the next step is to setup access to this bucket for our application.

Creating a User for Application access

In order for our application to access S3 bucket, we will need to create a new user in AWS IAM Service and give this user access to our S3 bucket (you can create role instead if you like).

Following screenshot, I created a user for our application:

for other fields, use the default settings. We’ll also need to setup programmatic access for this user (access keys) but we will do that a little bit later.

Next, lets create a policy which will allow the permissions for the bucket.

Creating a Policy

From IAM policies screen, we can create a custom policy as follows

This policy is allowing some actions on the bucket we created earlier.

Next, we can attach this policy to the application user.

Attach Policy to User

From IAM dashboard, we can edit user to attach policy as shown below

Create Access Keys for User

We also need to create access keys for our user, which we will use in our .NET application later.

Following screen from IAM user shows this step

For now, I selected Local code as I will be running the application from my development machine. However, you can try other options if you like.

Here is the screen showing access keys created for user. Note down these keys some where for later reference.

So until this point,

• we have created an S3 bucket to store our application data files.
• We then create a user for our application.
• We created a custom policy to allow certain action permissions on the S3 bucket and attach this policy to our application user.
• We also created the access keys for this user, which we will use in our .NET application later.

The AWS resource setup part is done and we can now start working on .NET6 Application part.

Create .NET6 Web API Application

To start, I created a .NET6 application using visual studio built-in template as shown below

.NET6 Application wiring with S3

To wire up the application, following configurations are setup in appsettings.json file and the corresponding C# class to read these configs (note these configs are pointing to bucket we created earlier and AWS AccessKeys from user setup earlier)

and in the Program.cs class read the configs and populate the S3Settings class as shown below

at this point, we have wired-up the configuration needed by our application to connect/access S3.

With all this done, we can now focus on writing the application code for the REST API and we will do that in the next post.

Summary

In this post, we started with a basic introduction of S3 and discussed our application data storage requirement and S3 can be used for this purpose.

We create an S3 bucket to store our application data and also created a user, policy and attach the policy with user. We then created access keys for this user to use it in our .NET application later.

Next, we create a very basic .NET6 application and did some configuration wiring to make it ready to access our S3 bucket.

In the next post, we will write our application code to allow user to perform CRUD operations, which will result in creating various files in S3 bucket and we will be able to access those files in our application.

The source code is available on this GitHub repository.

Let me know if you have some comments or questions.