Eb Extension Config
Our goal is to create a configuration such that, it can run on any elastic beanstalk instance. It should send production and non-production logs to two seperate Stackdriver projects. Adding monitoring for new log file should include minimal changes at best. If you have used Elastic Beanstalk before, probably you will be familiar with eb-extensions scripts. These are the set of commands those run everytime application is deployed on the EB.
Note how ~envname~ token is used in the tag. This token will automatically get replaced by the string environment name of the elastic beanstalk. On the stackdriver you can filter the logs by this tag names.
Copy following yaml into it.
This file informs elastic beanstalk to run mentioned shell scripts.
This script reads environment title from elastic beanstalk configuration properties. (Property name: env_name). Replaces the envname token from the fluentd-conf file that we created earlier. Copies the fluent-d config at required location.
Step 6: Copy the key file downloaded in the Part 1 of this tutorial as stackDriverCreds_prod.json. For production instance this key will be used. For non-production instances, create the key file with the name stackDriverCreds_dev.json
Done!
When you deploy your application-server on the Elastic beanstalk it will configure fluent-d automatically. And logs will start showing up in stackdriver.
I'll be happy to answer if you have any queries or suggestions.
Cheers!
Our goal is to create a configuration such that, it can run on any elastic beanstalk instance. It should send production and non-production logs to two seperate Stackdriver projects. Adding monitoring for new log file should include minimal changes at best. If you have used Elastic Beanstalk before, probably you will be familiar with eb-extensions scripts. These are the set of commands those run everytime application is deployed on the EB.
Step 1: Create folder .ebextension in your WEB-ROOT directory.
EB by default looks for ".config" files under .ebextension directory and executes on app-deployment. Add sub-directory called "stackdriver" under .ebextension directory that you just created.Step 2: Add google-fluentd.conf file in stackdriver directory.
Fluent-d agent runs on this configuration. This file tells the fluentd where to look for log files. Following sample file configures fluentd to check for app.log and billing.log files.<source> type tail format none path /var/log/app.log pos_file /var/lib/google-fluentd/pos/~envname~-app.pos read_from_head true tag ~envname~-app </source> <source> type tail format none path /var/log/billing.log pos_file /var/lib/google-fluentd/pos/~envname~-billing.pos read_from_head true tag ~envname~-billing </source>
Note how ~envname~ token is used in the tag. This token will automatically get replaced by the string environment name of the elastic beanstalk. On the stackdriver you can filter the logs by this tag names.
Step 3: Add file install.config under .ebextensions/stackdriver.
Copy following yaml into it.container_commands: 100-update-conf: command: bash .ebextensions/stackdriver/stackSetup.sh ignoreErrors: true 200-install-stack: command: sudo bash .ebextensions/stackdriver/stack-install.sh --write-gcm ignoreErrors: true 300-install-logging-agent: command: sudo bash .ebextensions/stackdriver/install-logging-agent.sh ignoreErrors: true
This file informs elastic beanstalk to run mentioned shell scripts.
Step 4: Create statckSetup.sh in .ebextensions/stackdriver .
Copy following shell script into it.#!/bin/bash ############################# Add tag in fluentd conf ################################ LOG_PREFIX=$log_prefix if [[ -z "$LOG_PREFIX" ]] then LOG_PREFIX="$env_name" if [[ -z "$LOG_PREFIX" ]] then LOG_PREFIX="untitled" fi fi sudo sed -i "s/~envname~/${LOG_PREFIX}/g" .ebextensions/stackdriver/google-fluentd.conf sudo mkdir -p /etc/google-fluentd/config.d/ sudo mv .ebextensions/stackdriver/google-fluentd.conf /etc/google-fluentd/config.d/app.conf ######################### Setup StackDriver credentials ####################### LOG_ENV=$log_env if [[ -z "$LOG_ENV" ]] then LOG_ENV="dev" fi sudo mkdir -p /etc/google/auth/ sudo mv .ebextensions/stackdriver/stackDriverCreds_${LOG_ENV}.json /etc/google/auth/application_default_credentials.json sudo chown root:root /etc/google/auth/application_default_credentials.json sudo chmod 0400 /etc/google/auth/application_default_credentials.json
This script reads environment title from elastic beanstalk configuration properties. (Property name: env_name). Replaces the envname token from the fluentd-conf file that we created earlier. Copies the fluent-d config at required location.
Step 5: Download the file https://repo.stackdriver.com/stack-install.sh.
Save it under stackdriver directory as stack-install.shStep 6: Copy the key file downloaded in the Part 1 of this tutorial as stackDriverCreds_prod.json. For production instance this key will be used. For non-production instances, create the key file with the name stackDriverCreds_dev.json
Done!
When you deploy your application-server on the Elastic beanstalk it will configure fluent-d automatically. And logs will start showing up in stackdriver.
I'll be happy to answer if you have any queries or suggestions.
Cheers!
Thanks for sharing this blog.This article gives lot of information.
ReplyDeleteAWS Training
AWS Online Training
Amazon Web Services Online Training
AWS Training in Hyderabad
AWS Training in Ameerpet
Great Article
ReplyDeleteCloud Computing Projects
Networking Projects
Final Year Projects for CSE
JavaScript Training in Chennai
JavaScript Training in Chennai
The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training