Python on Technology Enlightened

Monitor AWS Trusted Advisor Checks

Tue, 14 Mar 2017 17:39:24 +0000

AWS Trusted Advisor is an important tool which provides more than 50 checks across four different categories Cost Optimization, Performance, Security & Fault Tolerance (at the moment of writing this post).

But I have noticed that this is not being used more extensively considering the importance of it. Hence I created couple of lambda functions which shall be configured to run on daily basis to refresh the checks and create summary of all the checks and send an email.

Scheduling automated EBS snapshots serverless using AWS Lambda

Thu, 09 Jun 2016 11:46:37 +0000

EBS snapshot as backup mechanism is a very common practice and it also has been defined in Backup and Recovery section of EC2 best practices. If you have read my another article Save AWS costs by scheduled start and stop of EC2 instances, similar approach will be used in creating Snapshots too.

Using JSON in EC2 Tags enables us to provide granular configuration details. Create an EC2 tag with name as backup and value with details containing time to take backup and retention period. Obtaining these details from Tags will enable us to configure different schedule & retention period for different EC2 instance. If no backup is needed on a particular day, that day shall be removed from the value.

Sync S3 bucket to multiple S3 buckets in different region

Fri, 27 May 2016 11:55:41 +0000

Cross-Region Replication for Amazon S3 was introduced last year which enables replicating objects from a S3 bucket to a different S3 bucket located in different region (it can be same/different AWS account). It’s one of the most sought after feature for long time but it doesn’t solve some of the scenarios as defined below.

Synchronizing to a bucket in same region.
Synchronizing to multiple destination buckets.
Object ACLs needs to be set at the source bucket hence multiple different ACLs needs to be set when copying across different accounts.
Replication doesn’t change the ownership of the objects hence bucket policy applied at the destination bucket won’t be honored.

Refer the feedback from AWS support team regarding the last point mentioned above.

Automated Blue/Green deployment using Lambda and CloudFormation

Thu, 05 May 2016 07:42:54 +0000

Blue/Green deployment is a well-known method to deploy an application without any downtime. Performing DNS switch is one of the very common techniques to achieve this. Using DNS switch has a minor issue with DNS caching which might take some time for DNS change to be propagated.

Apart from DNS switch, AWS gives us two different options to switch the stacks. One is to have single ELB and swap it across auto-scaling group and another is switching the launch configuration of the auto-scaling group. Check the following presentation on various methods to achieve blue/green deployment in AWS.

Sending response back to CFN custom resource from python Lambda function

Sun, 20 Dec 2015 06:00:56 +0000

CloudFormation uses a pre-signed S3 URL to receive the response back from the custom resources managed by it. There are few blue prints available for Node.js Lambda custom resources but nothing available for python yet. Hence I created this simple function which shall be used to send the response back to CFN custom resource by performing PUT request to the pre-signed S3 URL.

import json
import requests

def lambda_handler(event, context):
    responseStatus = 'SUCCESS'
    responseData = {}
    if event['RequestType'] == 'Delete':
        sendResponse(event, context, responseStatus, responseData)

    responseData = {'Success': 'Test Passed.'}
    sendResponse(event, context, responseStatus, responseData)

def sendResponse(event, context, responseStatus, responseData):
    responseBody = {'Status': responseStatus,
                    'Reason': 'See the details in CloudWatch Log Stream: ' + context.log_stream_name,
                    'PhysicalResourceId': context.log_stream_name,
                    'StackId': event['StackId'],
                    'RequestId': event['RequestId'],
                    'LogicalResourceId': event['LogicalResourceId'],
                    'Data': responseData}
    print 'RESPONSE BODY:n' + json.dumps(responseBody)
    try:
        req = requests.put(event['ResponseURL'], data=json.dumps(responseBody))
        if req.status_code != 200:
            print req.text
            raise Exception('Recieved non 200 response while sending response to CFN.')
        return
    except requests.exceptions.RequestException as e:
        print e
        raise

if __name__ == '__main__':
    lambda_handler('event', 'handler')

I’m using json module to form the response body, json module is available by default in the Lambda container. Whereas the requests module which I use to perform the http PUT request needs to be packaged along with the python script.

Continuous Integration - Manage shared resources across accounts automatically

Mon, 07 Dec 2015 18:28:16 +0000

I prefer creating the base AMI using packer; RDS snapshot using a Jenkins job which get triggered whenever there is a change in the database schema in the SCM. In case of installers, binaries, etc. it would be best to store them in a single S3 bucket.

In all the above mentioned scenarios I prefer to share those resources to a 3rd party AWS account and remove the share when not needed. Automating this would be easy to handle add/remove permissions, hence I decided to hold the sharing details in a JSON file in the SCM. Changes to that file will trigger a Jenkins job which will invoke a Lambda function which in turn will share the resources based on the values in the JSON file.

Executing Ruby code in AWS Lambda

Sun, 29 Nov 2015 18:41:59 +0000

AWS Lambda supports only Java, JavaScript & Python at the time of writing this post. I was wondering whether I can exploit the Lambda to execute chef’s knife commands on it since Lambda runs on a Linux container. Using simple python function I have identified that the code of the Lambda function gets copied to /var/task and executed from there.

I decided to custom build ruby as embedded platform similar to how chef-client or chefdk are distributed. My requirement is to execute knife commands and its uses embedded ruby version 2.1.0, hence I downloaded the source archive that specific version.

Save AWS costs by scheduled start and stop of EC2 instances

Sun, 15 Nov 2015 14:56:57 +0000

Most of the AWS resources are billed on per-hour basis which provides us an opportunity to save cost based on the usage pattern. Especially in case of Dev/Test environment, we need them only during the working days and working hours.

By using AWS Lambda function (python) in combination with EC2 instance tags, scheduled start or stop can be achieved with few lines of code.

Advantages of using AWS Lambda:

For 128MB memory we get 3,200,000 seconds free tier usage per month in AWS Lambda which will be more than enough for this function.
IAM role will be associated with the Lambda function which will be used to perform the required task and no separate keys need to be configured.

I used a tag with name as schedule & JSON string for tag value using which I could define the day and time when it needs to be started or stopped.