Cloud on Technology Enlightened

Running AWS Glue jobs connecting to database in VPC with Custom DNS

Thu, 06 Sep 2018 07:55:46 +0000

AWS Glue is a fully managed ETL service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores. Glue job is the business logic that automate the extract, transform, and transfer data to different locations.

Glue job creates a Hadoop task in the backend and since it uses Hadoop ecosystem all the nodes in the cluster must be able to find using the FQDN and communicate with each other. To satisfy Hadoop DNS lookup, both forward and reverse lookups must succeed. When you use the default DHCP option set with AmazonProvidedDNS, it automatically creates the DNS record for the nodes that’s getting created in the VPC.

Monitor AWS Trusted Advisor Checks

Tue, 14 Mar 2017 17:39:24 +0000

AWS Trusted Advisor is an important tool which provides more than 50 checks across four different categories Cost Optimization, Performance, Security & Fault Tolerance (at the moment of writing this post).

But I have noticed that this is not being used more extensively considering the importance of it. Hence I created couple of lambda functions which shall be configured to run on daily basis to refresh the checks and create summary of all the checks and send an email.

Scheduling automated EBS snapshots serverless using AWS Lambda

Thu, 09 Jun 2016 11:46:37 +0000

EBS snapshot as backup mechanism is a very common practice and it also has been defined in Backup and Recovery section of EC2 best practices. If you have read my another article Save AWS costs by scheduled start and stop of EC2 instances, similar approach will be used in creating Snapshots too.

Using JSON in EC2 Tags enables us to provide granular configuration details. Create an EC2 tag with name as backup and value with details containing time to take backup and retention period. Obtaining these details from Tags will enable us to configure different schedule & retention period for different EC2 instance. If no backup is needed on a particular day, that day shall be removed from the value.

Sync S3 bucket to multiple S3 buckets in different region

Fri, 27 May 2016 11:55:41 +0000

Cross-Region Replication for Amazon S3 was introduced last year which enables replicating objects from a S3 bucket to a different S3 bucket located in different region (it can be same/different AWS account). It’s one of the most sought after feature for long time but it doesn’t solve some of the scenarios as defined below.

Synchronizing to a bucket in same region.
Synchronizing to multiple destination buckets.
Object ACLs needs to be set at the source bucket hence multiple different ACLs needs to be set when copying across different accounts.
Replication doesn’t change the ownership of the objects hence bucket policy applied at the destination bucket won’t be honored.

Refer the feedback from AWS support team regarding the last point mentioned above.

Running IIS in Windows Docker Container in Amazon EC2

Sun, 15 May 2016 15:53:17 +0000

Windows Server 2016 Technical Preview 5 was released on 27th April 2016 & it’s well known by now that it supports docker with its windows server containers. Microsoft also started uploading windows server core with various application as docker images to docker hub.

I was wondering whether it’s possible to run Windows 2016 TP5 on an EC2 instance and launch a windows container inside the EC2 instance. EC2 officially supports only upto Windows 2012 R2 and I couldn’t find any 2016 AMI’s in Ireland region. Hence I decided to install Windows 2016 TP5 on local VirtualBox, upload it to S3 create AMI out of it and launch the EC2 instance.

Using AWS IoT button - Simple Example

Sat, 14 May 2016 17:32:27 +0000

Amazon did a limited release of the programmable Dash button and unsurprisingly it got sold out within a day. It’s stated in amazon.com that “We don’t know when or if this item will be back in stock.”, hence you might need to wait to get one.

I take this opportunity to write a post about how I used the AWS IoT button that I received in AWS re:Invent 2015 to show case AWS IoT to the team.

Sending response back to CFN custom resource from python Lambda function

Sun, 20 Dec 2015 06:00:56 +0000

CloudFormation uses a pre-signed S3 URL to receive the response back from the custom resources managed by it. There are few blue prints available for Node.js Lambda custom resources but nothing available for python yet. Hence I created this simple function which shall be used to send the response back to CFN custom resource by performing PUT request to the pre-signed S3 URL.

import json
import requests

def lambda_handler(event, context):
    responseStatus = 'SUCCESS'
    responseData = {}
    if event['RequestType'] == 'Delete':
        sendResponse(event, context, responseStatus, responseData)

    responseData = {'Success': 'Test Passed.'}
    sendResponse(event, context, responseStatus, responseData)

def sendResponse(event, context, responseStatus, responseData):
    responseBody = {'Status': responseStatus,
                    'Reason': 'See the details in CloudWatch Log Stream: ' + context.log_stream_name,
                    'PhysicalResourceId': context.log_stream_name,
                    'StackId': event['StackId'],
                    'RequestId': event['RequestId'],
                    'LogicalResourceId': event['LogicalResourceId'],
                    'Data': responseData}
    print 'RESPONSE BODY:n' + json.dumps(responseBody)
    try:
        req = requests.put(event['ResponseURL'], data=json.dumps(responseBody))
        if req.status_code != 200:
            print req.text
            raise Exception('Recieved non 200 response while sending response to CFN.')
        return
    except requests.exceptions.RequestException as e:
        print e
        raise

if __name__ == '__main__':
    lambda_handler('event', 'handler')

I’m using json module to form the response body, json module is available by default in the Lambda container. Whereas the requests module which I use to perform the http PUT request needs to be packaged along with the python script.

Continuous Integration - Manage shared resources across accounts automatically

Mon, 07 Dec 2015 18:28:16 +0000

I prefer creating the base AMI using packer; RDS snapshot using a Jenkins job which get triggered whenever there is a change in the database schema in the SCM. In case of installers, binaries, etc. it would be best to store them in a single S3 bucket.

In all the above mentioned scenarios I prefer to share those resources to a 3rd party AWS account and remove the share when not needed. Automating this would be easy to handle add/remove permissions, hence I decided to hold the sharing details in a JSON file in the SCM. Changes to that file will trigger a Jenkins job which will invoke a Lambda function which in turn will share the resources based on the values in the JSON file.

CloudFormation parameters grouping and labelling

Sun, 06 Dec 2015 07:16:52 +0000

Last week AWS enhanced CloudFormation by adding new services like AWS CodePipeline, AWS Config, AWS KMS and few more; default stack limit also has been increased from 20 to 200. Among other new features they also have introduced this small but useful parameters grouping and labelling feature.

Earlier all the parameters in a CFN template get listed in alphabetical order based on the logical name of the parameter. Also the parameters logical name is restricted to alphanumeric characters hence it causes some restrictions.