S3 - Event Driven Action

-

 

In this blog, we will see how to use S3 with event driven architecture.

Flow is as below:

1) Application running on EC2 instance uploads object to a prefix called "raw_data" into a S3 bucket.

2) Once the object is uploaded, S3 event is detected by the Event Bridge rule.

3) Event bridge rule is configured to listen for "S3:ObjectCreate" event with destination as Lambda.

4) Lambda is configured to read data from the "raw_data" folder and process it.

5) Processed data is saved into a file under the folder "processed_data".

6) Both the folders "raw_data" and "processed_data" exist in the same bucket "demo-event-application"

Here is the bucket.



EC2 instance is attached with the IAM EC2 instance profile to perform S3 actions. This profile/role has admin S3 access for S3 actions.





Attaching the role to the EC2 instance.


Let's login to the EC2 instance and verify the S3 actions.


It worked.

Here is the snippet to upload data to the S3 bucket.


This is data that gets uploaded to the "raw_data" folder.

Let’s enable Amazon EventBridge for the bucket.

Use Amazon EventBridge to build event-driven applications at scale using S3 event notifications.

Let’s create  a lambda function to process the file "raw_data" and store under the same bucket under the folder “processed_data”.


NOTE: Ensure lambda execution role has S3 permission.

Let’s start creating an event bridge rule to drive the event driven actions.







Rules have been created. Let’s test it.


No files exist under the folders. Triggering the upload from the EC2 instance.


Now, we could see data has been processed and saved under the "processed_data" folder.

Comments

Post a Comment

Popular posts from this blog

K8s - ETCD

-
Image
  etcd is a "strongly consistent , distributed key-value store". Why etcd? 1. Consistency : Since the API server is the central coordination point of the entire cluster; strong consistency is essential. It would be a disaster if, say, two nodes tried to attach the same persistent volume over iSCSI because the API server told them both that it was available. 2. Availability: API downtime means that the entire Kubernetes control plane comes to a halt, which is undesirable for production clusters. The  CAP theorem  says that 100% availability is impossible with strong consistency, but minimizing downtime is still a critical goal. 3. Consistent Performance: The API server for a busy Kubernetes cluster receives a fair amount of read and write traffic. The secret behind etcd's balance of strong consistency and high availability is the  Raft algorithm . Raft solves a particular problem: how can multiple independent processes decide on a single value for somethin...
Read more

K8s - Deployment and HPA replicas

-
Image
  We all know that replica is a critical part of a K8s deployment. Default replica is 1. Setting replicas as 3 will ensure 3 pods of that deployment is running at any given time.  Replica are internally managed by replica set which is in turn controlled by the K8s  ReplicationController   . When a pod is bad or deleted as part of replicaset, replication controller will ensure a new pod is created. Now, coming to HPA (Horizontal Pod Autoscaler) which is a dynamic scaler which works at the pod level based on the metrics configured.  When we configure we must mention min and max values. They should be greater than 1. Let's image I create a HPA with min as 5 and max as 10 under the condition of CPU utilization more than 50%. Initial deployment is set with the replica as 1. On top the deployment, I am attaching a HPA with the mentioned configuration. Under HPA, we are forcing the min replicas to 5. Hence, it autoscales the pod to 5 even though the CPU utilization is ...
Read more

K8s - User and Groups

-
Image
  In this post, we will see how user and group works. Technically, there is no concept of user and group management in K8s.  But, K8s has something called "Service Account" . When K8s is installed from scratch, every namespace has "default" service account and it has admin access to perform actions. It is not practically possible to use " default " service account for every actions. From security point of view, we need to give fine grained and required access.  K8s has 2 important things in the space of "Access Management". Role and RoleBinding [Namespace Scoped] Role : Create a role with actions allowed actions on the resources. Let's say I want to create a readonly role. Using the role users can only view the resource and cannot perform actions like create, modify, delete. Rolebinding : Attaching the role to a user, group and service account is called "Role Binding". apiVersion : rbac.authorization.k8s.io/v1 kind : Role metadata :...
Read more