Skip to main content

Auto Scaling DynamoDB


Those of you who have worked with the DynamoDB long enough, will be aware of the tricky scaling policies of DynamoDB. It allows user to explicitly set requests per second (units per second, but for simplicity we will just say request per second). User can do so for read as well as write operations. Anytime user can increase or decrease the provision capacity from DynamoDB web console and it will reflect immediately.

Sounds all good....... Or not?

What if you set provisioned capacity to 50 req per second but load on the server crosses 100 req per second? Requests gets throttled!! Mostly times out. What's worse? This can cause requests getting queued up in your web server. Which can potentially bring your entire server down.

What if you set provisioned capacity to 1000 req per second but load on the server is only 100 req per second through out the day? You lose your hard earned money for remaining 900 req per second.

What if you set it to 1000 req per sec and then realised that the traffic is low, so you bring it down to 900 then 800 then 700. After four dial downs, you won't be able to dial it down further for the day!! Also, practically its impossible to monitor this throughout a day everyday.

These are some serious limitations. In this article I will try to provide a way to resolve all of these problems.

Over all approach is to create an AWS Lambda, which gets triggered every time DynamoDB's table's provisioned capacity is about to be consumed completely. This lambda then increases the capacity. Similarly, the lambda gets triggered when actual consumption of the provisioned capacity is too low, then this lambda brings down the provisioned capacity.

After doing this setup, your server will never go down due to throttled DynamoDB queries AND DynamoDB operational cost will go down considerably.

Lets Start

DynamoDB being nice creates two basic alarms as soon as you create a table. Following are those two alarms:
<table-name>-ReadCapacityUnitsLimit-BasicAlarm
<table-name>-WriteCapacityUnitsLimit-BasicAlarm

You can find those in Alarms tab in table properties section as shown below.

Click on Edit Alarm button, it will open up following dialogue box which indicates that the alarm is sending notification to SNS topic called dynamodb
Next we will create an AWS Lambda which will listen to above SNS topic and adjust the provision capacity as required.

Open AWS Lambda web console. Click on Create lambda function.  Select Blank Function from the provided templates. Click on the dotted square and choose SNS. Select the SNS topic from the Edit alarm dialogue. Most probably it will be named dynamodb.

It will open up configure Lambda page as below.


Add some good name for you lambda. Select Run time as Node js 4.3. Because the code that I am going to share is written and tested for Node js 4.3

Get the lambda code from my github and post it in the inline code editor. ( Node.js not being my prime tool, I am sure there will be lot of scope for the improvement in the code. If you can suggest optimisation in the code, I'll be happy to include it. )

This lambda does the following things:
  • Gets triggered whenever alarm is set off on any DynamoDB table
  • Checks the last minute of average consumption
  • Based on difference in consumed vs provisioned it will set the new provisioned capacity to ensure requests won't get throttled as well as not much of provisioned capacity is getting wasted.
  • It updates the cloudwatch alarms set for the table as per new provisioned capacity
  • It sends the slack notification to the channel where we can keep an eye on the activities
Following configuration you can see on the top of lambda:
var PROVISIONED_UPPER_MARGIN_RATIO = 0.8;  //Autoscale If consumed >= 80% of provisioned
var PROVISIONED_LOWER_MARGIN_RATIO = 0.3; //Autoscale If consumed <= 30% of provisioned
var PROVISIONED_TO_CONSUMED_IDEAL_RATIO = 1.3; //Autoscale to 130% of consumed

var MINIMUM_PROVISIONED_READ_CAPACITY = 100; 
var MINIMUM_PROVISIONED_WRITE_CAPACITY = 50;

var MINIMUM_DELTA_TO_ACT_UPON = 50;  //Update provision cap only if the change is higher than this

var MAX_DIAL_DOWNS = 24; //number of times dynamodb allows you to bring down provisioned cap

P.S. By default DynamoDB allows you to bring down provisioned capacity only four times a day. But you can talk to their support centre and they might increase that limit for you.

Word of caution: Make sure you add some upper limit to the provisioned capacity, so that your bills won't shoot high if someone makes DDOS attack on your web server.

Happy Coding!

Comments

  1. Replies
    1. Great Article
      Cloud Computing Projects


      Networking Projects

      Final Year Projects for CSE


      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete

Post a Comment

Popular posts from this blog

StackDriver Integration with AWS Elastic Beanstalk - Part 1

Stack Driver Setup Step 1: Create stackdriver project Navigate to https://stackdriver.com  After logging in you will be redirected to project creation page. To manage multiple environments of your application its advisable that you create just two projects. One for production env and other one for non-prod. Even though you might have multiple non-production environments, but having one stackdriver project per application-env is overkill. So, lets start with production project. For this tutorial, I'll create CodeHooDoo-Prod project. Once the project is created, StackDriver will ask you to link your AWS account resources to it for monitoring.  But as AWS CloudWatch has good monitoring and alerting support you can skip this one. Keep clicking continue until you get to monitoring console. Click on the logging. This is where you will get all the logs from your application server. Step 2: Download authentication key Navigate backt to https://stackdriver.com .