I prefer creating the base AMI using packer; RDS snapshot using a Jenkins job which get triggered whenever there is a change in the database schema in the SCM. In case of installers, binaries, etc. it would be best to store them in a single S3 bucket.
In all the above mentioned scenarios I prefer to share those resources to a 3rd party AWS account and remove the share when not needed. Automating this would be easy to handle add/remove permissions, hence I decided to hold the sharing details in a JSON file in the SCM. Changes to that file will trigger a Jenkins job which will invoke a Lambda function which in turn will share the resources based on the values in the JSON file.
Sample JSON File
{
"ami": {
"base_linux_v": {
"eu-west-1": [
"123456789012",
"987654321098"
],
"us-east-1": [
"987654321098"
]
},
"base_windows_v": {
"eu-west-1": [
"123456789012",
"987654321098"
],
"us-east-1": [
"123456789012"
]
}
},
"s3": {
"com-wordpress-cloudenlightened-blog": [
"123456789012",
"987654321098"
]
},
"rds": {
"base-db-v": {
"eu-west-1": [
"123456789012",
"987654321098"
],
"us-east-1": [
"987654321098"
]
}
}
}
If I want to add a new account or remove permission to an existing account I just need to add/remove the specific account from the appropriate section in the JSON file and check-in to SCM. It will automatically trigger the Lambda function which performs the required changes. If you notice, in the above JSON file, for AMI & RDS names I’m just using the part of the string and not using the exact version. Hence it will find AMIs or RDS snapshots which contain this string in their name and modify the permission to all the resources. Share resources Jenkins job should also be triggered by (downstream job) the AMI creation / RDS Snapshot job, which ensures that all the new AMI/snapshot are shared appropriately, immediately after its creation.
Lambda Function to share resources
import boto3
import sys
#
# Share the AWS resources based on the data received from the event.
#
def lambda_handler(event, context):
print "=== Started aws resources sharing ==="
if event['ami']:
share_amis(event['ami'])
if event['s3']:
share_s3_bucket(event['s3'])
if event['rds']:
share_rds_snapshot(event['rds'])
def share_amis(amitree):
for amis in amitree:
for reg in amitree[amis]:
desaccess = amitree[amis][reg]
ec2 = boto3.client('ec2', region_name=reg)
amilist = get_ami_ids(ec2, amis)
for amiid in amilist:
exstaccess = get_ami_perms(ec2, amiid)
addacct = list(set(desaccess) - set(exstaccess))
remacct = list(set(exstaccess) - set(desaccess))
modify_ami_perms(ec2, amiid, addacct, remacct)
def get_ami_ids(ec2, aminame):
print 'Getting list of AMIs with name ' + aminame
amiids = []
try:
amiout = ec2.describe_images(Filters=[{'Name': 'name', 'Values': [aminame + '*']}])
except Exception as e:
print "Error: %s" % e
sys.exit(1)
for img in amiout['Images']:
amiids.append(img['ImageId'])
return amiids
def get_ami_perms(ec2, amiid):
print 'Getting launch permissions of AMI ID ' + amiid
acctids = []
try:
acctout = ec2.describe_image_attribute(ImageId=amiid, Attribute='launchPermission')
except Exception as e:
print "Error: %s" % e
sys.exit(1)
for acct in acctout['LaunchPermissions']:
acctids.append(acct['UserId'])
return acctids
def modify_ami_perms(ec2, amiid, addacct, remacct):
print 'Modifying permissions for AMI ID ' + amiid
addcoll = []
remcoll = []
for acct in addacct:
addcoll.append({'UserId': acct})
for acct in remacct:
remcoll.append({'UserId': acct})
try:
if addcoll:
ec2.modify_image_attribute(ImageId=amiid, Attribute='launchPermission',
LaunchPermission={'Add': addcoll})
if remcoll:
ec2.modify_image_attribute(ImageId=amiid, Attribute='launchPermission',
LaunchPermission={'Remove': remcoll})
except Exception as e:
print "Error: %s" % e
sys.exit(1)
def share_rds_snapshot(rdstree):
for snap in rdstree:
for reg in rdstree[snap]:
desaccess = rdstree[snap][reg]
rds = boto3.client('rds', region_name=reg)
snaplist = [s for s in get_all_snapshots(rds) if snap in s]
for sn in snaplist:
exstaccess = get_snap_perms(rds, sn)
addacct = list(set(desaccess) - set(exstaccess))
remacct = list(set(exstaccess) - set(desaccess))
modify_snapshot_perms(rds, sn, addacct, remacct)
def get_all_snapshots(rds):
try:
snapout = rds.describe_db_snapshots()
snaplist = []
for snap in snapout['DBSnapshots']:
snaplist.append(snap['DBSnapshotIdentifier'])
return snaplist
except Exception as e:
print "Error: %s" % e
sys.exit(1)
def get_snap_perms(rds, snapname):
print 'Getting existing permission to snapshot ' + snapname
acctids = []
try:
acctout = rds.describe_db_snapshot_attributes(DBSnapshotIdentifier=snapname)
except Exception as e:
print "Error: %s" % e
sys.exit(1)
for acct in acctout['DBSnapshotAttributesResult']['DBSnapshotAttributes']:
print acct
if acct['AttributeName'] == 'restore':
acctids = acct['AttributeValues']
return acctids
def modify_snapshot_perms(rds, snap, addacct, remacct):
print 'Modifying permissions for snapshot ' + snap
try:
if addacct:
rds.modify_db_snapshot_attribute(DBSnapshotIdentifier=snap, AttributeName='restore', ValuesToAdd=addacct)
if remacct:
rds.modify_db_snapshot_attribute(DBSnapshotIdentifier=snap, AttributeName='restore', ValuesToRemove=remacct)
except Exception as e:
print "Error: %s" % e
sys.exit(1)
def share_s3_bucket(s3bucks):
s3 = boto3.client('s3')
for buck in s3bucks:
set_s3_bucket_policy(s3, buck, s3bucks[buck])
def set_s3_bucket_policy(s3, buck, acctids):
print 'Setting policy to S3 bucket ' + buck
if acctids:
policy = '{ "Version": "2012-10-17", "Id": "ReadBucketPolicy", "Statement": [ { "Sid": "S3ReadAccessToRole",
"Effect": "Allow", "Principal": { "AWS": ['
for acct in acctids[:-1]:
policy += '"arn:aws:iam::' + acct + ':root",'
else:
policy += '"arn:aws:iam::' + acctids[-1] + ':root"] }, "Action": [ "s3:List*", "s3:Get*" ], "Resource": [
"arn:aws:s3:::' + buck + '/*","arn:aws:s3:::' + buck + '" ] } ] }'
try:
if acctids:
s3.put_bucket_policy(Bucket=buck, Policy=policy)
else:
s3.delete_bucket_policy(Bucket=buck)
except Exception as e:
print "Error: %s" % e
sys.exit(1)
if __name__ == '__main__':
lambda_handler('event', 'handler')
At the time of writing this blog, default boto3 version in AWS Lambda is 1.2.1 whereas RDS snapshot sharing API is available only in version 1.2.2. Hence I had to pack boto3 also as part of the Lambda function. Best way to include boto3 module is to download it to local directory using pip.
pip install boto3 -t ./ShareAWSResources
It will download boto3 and its dependent modules to the local directory named ShareAWSResources which should be packed in a zip file including the python script.
IAM permissions required for the Lambda function
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AMIPolicy",
"Effect": "Allow",
"Action": [
"ec2:DescribeImageAttribute",
"ec2:DescribeImages",
"ec2:ModifyImageAttribute"
],
"Resource": [
"*"
]
},
{
"Sid": "S3Policy",
"Effect": "Allow",
"Action": [
"s3:PutBucketPolicy",
"s3:DeleteBucketPolicy"
],
"Resource": [
"*"
]
},
{
"Sid": "RDSSnapshotPolicy",
"Effect": "Allow",
"Action": [
"rds:DescribeDBSnapshots",
"rds:DescribeDBSnapshotAttributes",
"rds:ModifyDBSnapshotAttribute"
],
"Resource": [
"*"
]
}
]
}