AWS Tricks

Collection of tricks for Amazon Web Services, mainly examples of CLI commands for the different services.

EMR tricks

I use Elastic MapReduce CLI.

Create a Spark 0.8.1 cluster:

elastic-mapreduce --create --alive \
--name "Spark/Shark Cluster"  \
--bootstrap-action s3://elasticmapreduce/samples/spark/0.8.1/install-spark-shark.sh \
--bootstrap-name "Spark/Shark" \
--instance-type m3.xlarge \
--instance-count 3

[Doesn't work] Create a Spark 1.0.0 cluster (notice --ami-version):

elastic-mapreduce --create --alive --name "Spark/Shark Cluster 1.0.0"  \
--bootstrap-action s3://elasticmapreduce/samples/spark/1.0.0/install-spark-shark.rb \
--bootstrap-name "Spark/Shark" \
--instance-type m3.xlarge \
--instance-count 3 \
--ami-version 3.0.3

Show all running clusters:

elastic-mapreduce --list --active

Install tricks

Installing AWS cli on a Mac is dead easy with pip:

pip install awscli

# If you get 'ValueError: unknown locale: UTF-8' 
# edit ~/.profile, add lines (use en_US if appropriate):
#  export LC_ALL=da_DK.UTF-8
#  export LANG=da_DK.UTF-8

Make an AWS CLI profile (if you have multiple accounts):

# Have Access Key ID and Secret Access Key ready
aws configure --profile [profile name]

# Using the profile
aws --profile [profile name]

Create shortcut for using AWS CLI profile, e.g. with profile foo:

alias aws-foo="aws --profile foo"

S3 tricks

Public read-access bucket policy (change 'yourbucketname' to actual bucket):

{
	"Version": "2008-10-17",
	"Statement": [
		{
			"Sid": "AllowPublicRead",
			"Effect": "Allow",
			"Principal": {
				"AWS": "*"
			},
			"Action": [
				"s3:GetObject"
			],
			"Resource": [
				"arn:aws:s3:::yourbucketname/*"
			]
		}
	]
}

Previously I used s3funnel, but now I use the AWS cli.

Get help on AWS cli s3 commands:

aws s3 [command] help
# command one of: cp, ls, mb, mv, rb, rm, sync, website
# actually, the reference at http://aws.amazon.com/cli/ is better IMO

Create a new buckets in S3:

aws s3 mb s3://skipperkongen-test
# creates bucket called 'skipperkongen-test'

List all your buckets in S3:

aws s3 ls
# 2014-07-17 23:04:53 skipperkongen-test

Move a file to S3:

aws s3 mv testfile.txt s3://skipperkongen-test/testfile.txt

Move a whole directory to S3:

aws s3 sync mylocaldir s3://skipperkongen-test/mys3dir

List files in an S3 bucket:

aws s3 ls s3://skipperkongen-test

Delete an entire S3 bucket (must be empty or use --force):

aws s3 rb --force s3://skipperkongen-test
# individually deletes each file, may take a while