python, boto3, AWS and a halfwit

coderSo week two at AWS is nearly over which has been as varied and fun as the first.

I thought I would share something I put together this week to demonstrate in the tiniest and most elementary way the power of doing stuff in the cloud.

So I am not a programmer, I hack stuff together and am reliant on the clever internet fairies to help me out. That being said and with no prior knowledge of python I managed to create a script that…

  1. Creates a Virtual Private Cloud (an isolated network)
  2. Creates a private subnet
  3. Creates an internet gateway
  4. Creates a firewall (security group) that only allows traffic on port 22 and 80
  5. Creates as many servers as I want using the above config and injects boot-up instructions into each of them ensuring they startup as webservers

Firstly – Imagine trying to do all the above in a traditional IT environment!!

Secondly – If a halfwit like me can do this with a day of scripting imagine the power of AWS in the hands of real coder!!

So as Andy Jassy would say – Giddy-up!

I attach the script below for your amusement at my clumsiness – also check out the python SDK page (boto3) HERE

# Python script to add new VPC and Launch EC2 instance
# Uses Boto3 for AWS actions
# Author: Laurence Davenport
# Version: 1.0
# Date: 11.11.2015

import re
import boto3
ec2 = boto3.resource('ec2')
client = boto3.client('ec2')

# function to display items in a list - thanks to Said Ali Samed for this
def display_list(items, key_one, key_two=None, key_three=None):
    if type(items) is not list: return
    for item in items:
        print('%i. %s  %s %s' % (items.index(item) + 1, item[key_one], item[key_two] if key_two else '',
                                 item[key_three] if key_three else ''))

#creating list of all available keypairs
keys = client.describe_key_pairs()
key_list = keys['KeyPairs']

# populating the default keypair value with the first in the list
for item in key_list:
    index = key_list.index(item)
    if index == 0:
        default_key = item['KeyName']

# set the bootstrap script for the instance
user_data = '#!/bin/sh\n yum -y install httpd php mysql php-mysql\n chkconfig httpd on\n /etc/init.d/httpd start'

#Gather data from user
happiness = "N"
while happiness != "Y":
    vpc_cidr_block =  input("Please enter VPC CIDR block []: ") or ""
    subnet_cidr_block =  input("Please enter Subnet CIDR block []: ") or ""
    print ("KeyPairs that are available to use are:")
    display_list(key_list, 'KeyName')
    key_name = input("Please enter name of keypair [" + default_key + "]: ") or default_key
    secgrp_name = input("Please enter name of Security Group name [sandboxSecGrp]: ") or "sandboxSecGrp"
    secgrp_desc = input("Please enter name of Security Group description [Sandbox security group]: ") or "Sandbox security group"
    ami_name = input("Please enter Instance AMI [ami-bff32ccc]: ") or "ami-bff32ccc"
    intance_type = input("Please enter Instance type [t2.micro]: ") or "t2.micro"

    print ("=====================================")
    print ("======== Values are set to ==========")
    print ("=====================================")
    print ("VPC CIDR block is: " + vpc_cidr_block)
    print ("Subnet CIDR block is: " + subnet_cidr_block)
    print ("Keypair name is: " + key_name)
    print ("Security group name is: " + secgrp_name)
    print ("Security group description is: " + secgrp_desc)
    print ("Instance AMI is: " + ami_name)
    print ("Instance type is: " + intance_type)
    print ("=====================================")

    happiness = input("Are you happy with all these settings [Y/N]: ") or "Y"

# Create step counter
step_no = 1

# Creating the VPC
vpc = ec2.create_vpc(CidrBlock=vpc_cidr_block)
print ("STEP "+  str(step_no) + " - VPC ID: " +
client.modify_vpc_attribute(, EnableDnsHostnames={"Value": True})

step_no += 1

# create subnet
subnet = vpc.create_subnet(CidrBlock=subnet_cidr_block)
print ("STEP "+  str(step_no) + " - subnet ID: " +
subnet.meta.client.modify_subnet_attribute(, MapPublicIpOnLaunch={"Value": True})

step_no += 1

# create internet gateway
gateway = ec2.create_internet_gateway()
print ("STEP "+  str(step_no) + " - Internet Gateway ID: " +

step_no += 1

# attach internet gateway
print ("STEP "+  str(step_no) + " - Attach Internet Gateway to VPC ")

step_no += 1

# Create route for vpc to internet
route_table = list(vpc.route_tables.all())[0] # get the route table id
print ("STEP "+  str(step_no) + " - Add route to internet to routing table: " +

step_no += 1

# create security group
security_group = vpc.create_security_group(GroupName=secgrp_name,Description=secgrp_desc)
print ("STEP "+  str(step_no) + " - Creating security group: " +

step_no += 1

# allow access for SSH
print ("STEP "+  str(step_no) + " - Open port 22 in security group: " +
step_no += 1

# allow access for HTTP
print ("STEP "+  str(step_no) + " - Open port 80 in security group: " +
step_no += 1

#create instance
finished = "N"
while finished != "Y":
    instance = subnet.create_instances(ImageId=ami_name,MinCount=1,MaxCount=1, KeyName=key_name,SecurityGroupIds=[ ],UserData=user_data,InstanceType=intance_type)
    inst_resource = instance[0]
    print ("STEP "+  str(step_no) + " - Launching Instance: " +
    finished = input("Have you finished launching instances? [Y/N]: ") or "Y"

print ("=== Script Complete - Goodbye ===")



Shed a Tier

What is the point of tiered storage?
Well it is the categorising of your data into types so that they can be located on different media.  The reason they go onto different media is to save cost by not keeping all data on the expensive high performance media needed for a fraction of applications.  So for instance you may have Tier 1 as being Fibre Channel 15k drives, Tier 2 might be SATA drives and Tier 3 could be tape.

Price, no longer a problem
Sometimes implementing the tiers and maintaining them can be time consuming and complex.  Especially when you need to elevate and demote LUNs from one tier to another.  Having done some work with IBM XIV it is clear that the old Tier 1 and 2 makes no sense at all.  When the price per TB is about £4k the price argument for tiering goes away.  The other argument of performance remains until you are convinced that the XIV can perform as well as any traditional Tier 1 disk system.

Storage QoS
So considering IBM XIV and the other buzz in the industry at the moment “cloud storage” we could well be seeing the demise of tiering as we knew it.  Tiering would be replaced with storage QoS whereby the media and location remain constant but LUNs for key applications will get precedence over others.  This seems like a more manageable arrangement whereby you can dynamically alter the performance of your LUN without migrating from one media to another.

I can’t see tape and archive media being scrapped just yet but at least primary storage will be simplified if people apply this modal with the new technology that is coming through.


The cloud is where it’s at!

If you ain’t in the cloud, you are behind the times!  This site is now hosted from a cloud service provided by mosso which I must admit I am very impressed with.  From clicking the “buy now” button to the time I had a “naked” linux distro for my use was about 10-15 mins.  More info go here:

Storage has been provided in the cloud for a while now through different providers, but is there any room for storage management automation in the cloud model?  What could you do?

  1. Online portal where you could view stats about your storage environment.
  2. Compare your data and storage performance profiles against others in your industry.
  3. Show stats about estimated restore times based on stats fed up to the cloud.

I am sure someone will think of more interesting ones than those listed above (if you have any comment away).  Maybe one of the big winners for “SMA in the cloud” will be off the back of SaaS.  If services are being provided remotely for storage management then the automation tools to provide that service may need to be centralised into a cloud of shared computing and analysis resources.

So there…