Skip to main content

StackDriver Integration with AWS Elastic Beanstalk - Part 2

Eb Extension Config

Our goal is to create a configuration such that, it can run on any elastic beanstalk instance. It should send production and non-production logs to two seperate Stackdriver projects. Adding monitoring for new log file should include minimal changes at best. If you have used Elastic Beanstalk before, probably you will be familiar with eb-extensions scripts. These are the set of commands those run everytime application is deployed on the EB.

Step 1: Create folder .ebextension in your WEB-ROOT directory. 

EB by default looks for ".config" files under .ebextension directory and executes on app-deployment. Add sub-directory called "stackdriver" under .ebextension directory that you just created.

Step 2: Add google-fluentd.conf file in stackdriver directory. 

Fluent-d agent runs on this configuration. This file tells the fluentd where to look for log files. Following sample file configures fluentd to check for app.log and billing.log files.


<source>
  type tail
  format none
  path /var/log/app.log
  pos_file /var/lib/google-fluentd/pos/~envname~-app.pos
  read_from_head true
  tag ~envname~-app
</source>
<source>
  type tail
  format none
  path /var/log/billing.log
  pos_file /var/lib/google-fluentd/pos/~envname~-billing.pos
  read_from_head true
  tag ~envname~-billing
</source>

Note how ~envname~ token is used in the tag. This token will automatically get replaced by the string environment name of the elastic beanstalk. On the stackdriver you can filter the logs by this tag names.


Step 3: Add file install.config under .ebextensions/stackdriver. 

Copy following yaml into it.
container_commands:
  100-update-conf:
    command: bash .ebextensions/stackdriver/stackSetup.sh
    ignoreErrors: true
  200-install-stack:
    command: sudo bash .ebextensions/stackdriver/stack-install.sh --write-gcm
    ignoreErrors: true
  300-install-logging-agent:
    command: sudo bash .ebextensions/stackdriver/install-logging-agent.sh
    ignoreErrors: true

This file informs elastic beanstalk to run mentioned shell scripts.

Step 4: Create statckSetup.sh in .ebextensions/stackdriver . 

Copy following shell script into it.
#!/bin/bash
############################# Add tag in fluentd conf ################################
LOG_PREFIX=$log_prefix
if [[ -z "$LOG_PREFIX" ]]
then
    LOG_PREFIX="$env_name"
    if [[ -z "$LOG_PREFIX" ]]
    then
        LOG_PREFIX="untitled"
    fi
fi


sudo sed -i "s/~envname~/${LOG_PREFIX}/g" .ebextensions/stackdriver/google-fluentd.conf
sudo mkdir -p /etc/google-fluentd/config.d/
sudo mv .ebextensions/stackdriver/google-fluentd.conf /etc/google-fluentd/config.d/app.conf

######################### Setup StackDriver credentials #######################
LOG_ENV=$log_env
if [[ -z "$LOG_ENV" ]]
then
    LOG_ENV="dev"
fi

sudo mkdir -p /etc/google/auth/
sudo mv .ebextensions/stackdriver/stackDriverCreds_${LOG_ENV}.json /etc/google/auth/application_default_credentials.json
sudo chown root:root /etc/google/auth/application_default_credentials.json
sudo chmod 0400 /etc/google/auth/application_default_credentials.json

This script reads environment title from elastic beanstalk configuration properties. (Property name: env_name). Replaces the envname token from the fluentd-conf file that we created earlier. Copies the fluent-d config at required location.

Step 5: Download the file https://repo.stackdriver.com/stack-install.sh. 

Save it under stackdriver directory as stack-install.sh

Step 6: Copy the key file downloaded in the Part 1 of this tutorial as stackDriverCreds_prod.json.  For production instance this key will be used. For non-production instances, create the key file with the name stackDriverCreds_dev.json

Done!

When you deploy your application-server on the Elastic beanstalk it will configure fluent-d automatically. And logs will start showing up in stackdriver.

I'll be happy to answer if you have any queries or suggestions.

Cheers!

Comments

  1. Great Article
    Cloud Computing Projects


    Networking Projects

    Final Year Projects for CSE


    JavaScript Training in Chennai

    JavaScript Training in Chennai

    The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

    ReplyDelete

Post a Comment

Popular posts from this blog

Auto Scaling DynamoDB

Those of you who have worked with the DynamoDB long enough, will be aware of the tricky scaling policies of DynamoDB. It allows user to explicitly set requests per second (units per second, but for simplicity we will just say request per second). User can do so for read as well as write operations. Anytime user can increase or decrease the provision capacity from DynamoDB web console and it will reflect immediately. Sounds all good....... Or not? What if you set provisioned capacity to 50 req per second but load on the server crosses 100 req per second? Requests gets throttled!! Mostly times out. What's worse? This can cause requests getting queued up in your web server. Which can potentially bring your entire server down. What if you set provisioned capacity to 1000 req per second but load on the server is only 100 req per second through out the day? You lose your hard earned money for remaining 900 req per second. What if you set it to 1000 req per sec and then realis