AWS Data Loss Prevention Tips

Data backup is the most important part of a business operation. It is always a best practice to prevent any chance of loosing the data we possess.

We run multiple app servers in different AWS regions along the instances running for users, temporarily. According to the way we operate at rootsh3ll Labs, we might accidentally delete/terminate our main instance while pre-production testing. Code can really take things out of hands seriously, you never know.

So, I discovered and implemented 2 ways to dodge such situations, if ever happens.

TL;DR

  1. Disable EBS Deletion on Termination
  2. Enable Termination Protection
  3. Use S3 Backups with AWS Glacier

1. Disable EBS Deletion on Termination

By default the EBS volume is set to DELETE when an instance is terminated. In any unfortunate scenario, by us or AWS itself, an instance might get terminated and you loose all the data associated to that instance on the EBS storage.

On instance creation you get an option to have “Deletion on Termination” enabled. Uncheck it and you are good to go. Unfortunately AWS Console doesn’t provide a way to fix it if the instance is already running with the following attribute enabled.

Luckily, AWS CLI has a way around for it.

  • Set your desired AWS Region in the current shell
  • Set DeleteOnTermination attribute to false

Before that, check of the EBS has “Deletion on Termination” set to True of False.

  • Go to Console > Instances > Description Tab of the desired instance
  • Click on Block Devices: /dev/sdaX

Continue the process if the Deletion on Termination is set to True. In my case it is set to False as I have it disabled EBS Termination already

Now, go to your local Terminal, assuming you have AWS CLI setup, and enter the following command.

Change the AWS_DEFAULT_REGION to your desired region. and replace the instance-id in the following command after --instance-id flag

export AWS_DEFAULT_REGION=us-east-1
aws ec2 modify-instance-attribute --instance-id i-7be8e56 --block-device-mappings "[{\"DeviceName\": \"/dev/sda1\",\"Ebs\":{\"DeleteOnTermination\":false}}]"

Disabling the DeleteOnTermination attribute will keep the EBS storage detached from the terminated instance.

2. Enable Termination Protection

Code changes, programmer changes, we forget our code. Life happens. Accident happens.

Code might accidentally terminate our production instances. It’s better to disallow code or cli to terminate the our prod. instance.

To do that go to your AWS Console > Services > Instances. and choose the instance you want to prevent Termination on.

Right click > Instance Setting > Change Termination Protection

Then click Yes, Enable.

Note: Make sure the Current Setting is set to disabled

Bonus tip: Make regular S3 backups and make extensive use of Amazon Glacier to minimise costs.
Rule of thumb would be to set 30 days cache for S3 before moving data to Glacier.

I’ll update the post with a walkthrough when we need to use AWS Glacier. For now Internet is your friend.

Have a good DevOps!

1 Like