AWS S3 - Batch Operations

-

 

In this post, we will see how to use batch replication - S3. 

A job is used to execute batch operations on a list of S3 objects. The list of S3 objects is contained in a manifest object, which can be an S3 inventory report or a list of objects that you generate. After the total number of objects listed in the manifest has been confirmed, the job status will update to Awaiting your confirmation to run, and you must Run the job within 30 days.

I am going to create a replication job to tag the existing objects in the bucket.

NOTE: The batch job relies on manifest.json which is an inventory of objects. This can be collected from bucket management. It takes 48 hrs to get the first inventory created.


Let's create a batch job.



Select the location were the manifest.json is created.


I am to create tags for all the objects.





Next, the critical portion is assigning the correct IAM role for the batch to perform action.


I created a role with S3 full access. Please refer to the link for the right IAM access. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-iam-role-policies.html


Once the job is created, it awaits for the user's confirmation to "RUN JOB".




Once the job is completed, you should see the tags added to the objects.




It works.



Comments

Post a Comment

Popular posts from this blog

SRE/DevOps Syllabus

-
Image
  DEVOPS/SRE Principles. Git What is Git? Architecture of Git. Working principle of Git. Create and cloning a repo. Version control branching. Version control commit. Version control managing workflows. Git hooks. Git Reflog. Git Stash. Git Cherry Picking. Undoing changes in different states of Git. Git based terraform template management. AWS CI/CD SDLC Automation. Code Commit. Code Build. Code Deployment. Code Pipeline. Elastic Beankstalk. Code Artifact. CodeGuru. Terraform Terraform Basics. Terraform State. Working with Terraform. Terraform with AWS. Remote State. Terraform Provisioners. Terraform Import, Tainting and Debugging. Terraform Modules. Terraform Functions and Conditional Expressions. Automation using Python - AWS AWS lambda. Automating EC2 with Lambda. Automating S3 with Lambda. Automating VPC with Lambda. Cost optimization with Lambda. SNS,SQS and SES with Python. Managing and Automating AWS Security with Python. Kubernetes - CKAD Application Developer. Kubernetes A...
Read more

AWS Code Commit - CI/CD Series Part 1

-
Image
  This is a multi-part series to show how to implement a CI/CD pipeline in AWS. In this post, We will see how to use AWS Code Commit. AWS CodeCommit is a fully managed source control service that allows you to store and manage your code securely in the cloud.  It provides a private Git repository for your code and supports all Git commands and workflows. CodeCommit also integrates with other AWS services like AWS CodePipeline and AWS CodeBuild, allowing you to automate your entire software release process. This can help you streamline your development workflows and improve the overall efficiency of your team. I created a repo from AWS CodeCommit -> Create Repository. And I begin with cloning the repo locally. So that I can make changes and push it. To clone the repo click on "HTTPS" under Clone URL. I make changes to index.html file. Now push it to AWS CodeCommit. Let’s verify the changes in AWS Code commit. Changes are reflected in AWS CodeCommit. We are done with Part ...
Read more

Docker - Preventing IP overlapping

-
Image
  What is IP address overlapping? When the container IP address overlaps with another IP external, then the application running on the container is not accessible from outside due to the IP address overlapping.  NOTE: This IP address overlapping can happen on the bridge interface IP or docker container IP. Let's see how to address this IP overlapping issue on the bridge interface. I already have a running container.  root@demo:~# docker run -d --name web --network mynet1 nginx:alpine 268647e11011d513d2357ce1c81e0f48787038a83d01d028d31a5b29dbdb885b root@demo:~# docker container ls CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS     NAMES 268647e11011   nginx:alpine   "/docker-entrypoint.…"   4 seconds ago   Up 3 seconds   80/tcp  ...
Read more