Skip to main content

Posts

Showing posts from April, 2017

Auto Scaling DynamoDB

Those of you who have worked with the DynamoDB long enough, will be aware of the tricky scaling policies of DynamoDB. It allows user to explicitly set requests per second (units per second, but for simplicity we will just say request per second). User can do so for read as well as write operations. Anytime user can increase or decrease the provision capacity from DynamoDB web console and it will reflect immediately. Sounds all good....... Or not? What if you set provisioned capacity to 50 req per second but load on the server crosses 100 req per second? Requests gets throttled!! Mostly times out. What's worse? This can cause requests getting queued up in your web server. Which can potentially bring your entire server down. What if you set provisioned capacity to 1000 req per second but load on the server is only 100 req per second through out the day? You lose your hard earned money for remaining 900 req per second. What if you set it to 1000 req per sec and then realis

StackDriver Integration with AWS Elastic Beanstalk - Part 2

Eb Extension Config Our goal is to create a configuration such that, it can run on any elastic beanstalk instance. It should send production and non-production logs to two seperate Stackdriver projects. Adding monitoring for new log file should include minimal changes at best. If you have used Elastic Beanstalk before, probably you will be familiar with eb-extensions scripts. These are the set of commands those run everytime application is deployed on the EB. Step 1: Create folder .ebextension in your WEB-ROOT directory.  EB by default looks for ".config" files under .ebextension directory and executes on app-deployment. Add sub-directory called "stackdriver" under .ebextension directory that you just created. Step 2: Add google-fluentd.conf file in stackdriver directory.  Fluent-d agent runs on this configuration. This file tells the fluentd where to look for log files. Following sample file configures fluentd to check for app.log and billing.log files.

StackDriver Integration with AWS Elastic Beanstalk - Part 1

Stack Driver Setup Step 1: Create stackdriver project Navigate to https://stackdriver.com  After logging in you will be redirected to project creation page. To manage multiple environments of your application its advisable that you create just two projects. One for production env and other one for non-prod. Even though you might have multiple non-production environments, but having one stackdriver project per application-env is overkill. So, lets start with production project. For this tutorial, I'll create CodeHooDoo-Prod project. Once the project is created, StackDriver will ask you to link your AWS account resources to it for monitoring.  But as AWS CloudWatch has good monitoring and alerting support you can skip this one. Keep clicking continue until you get to monitoring console. Click on the logging. This is where you will get all the logs from your application server. Step 2: Download authentication key Navigate backt to https://stackdriver.com .

StackDriver Integration with AWS Elastic Beanstalk

Introduction In Today's era where application server spreads across hundreds of machines, log-aggregation has become mandatory tool. It provides centralised location for browsing / searching through your logs coming from all of the server instances. Google's stack driver is indeed quite stable log-aggregation tool. After reading this blog you will be able to: send your multi-server logs to centralised  location ie stack driver monitor / search logs in real time re-use same config on different elastic beanstalk environments (prod vs non-prod) This tutorial is divided into two chapters: Stack Driver setup Elastic Beanstalk configuration