Copying files into an EC2 instance during bootstrap using CloudFormation
It might be useful to copy files into an EC2 instance whilst it's being built. Those can be configuration files for the application, authentication keys or even the whole pre-built application stack.
One way of doing that is to use CloudFormation's Metadata and copy files from an S3 bucket. This post shows the configuration that's needed in order to make this work.
Authenticating an EC2 instance against an S3 bucket
In order to be able to pull the files from the S3 bucket the EC2 instance must have an IamInstanceProfile that allows for access to the bucket. Snippet below shows the necessary resources:
rEC2BaseRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
rEC2BasePolicy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: root
PolicyDocument:
Statement:
- Effect: Allow
Action:
- ec2:DescribeTags
- ec2:DescribeVolumes
- ec2:DescribeInstances
Resource: "*"
- Effect: Allow
Action:
- s3:ListBucket
- s3:GetObject
- s3:GetBucketLocation
- s3:ListAllMyBuckets
Resource:
- "arn:aws:s3:::*"
- "arn:aws:s3:::*/*"
Roles:
- Ref: rEC2BaseRole
rEC2BaseProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: "/"
Roles:
- Ref: rEC2BaseRole
InstanceProfileName: "EC2BaseProfile"
The AWS::IAM::Policy above can be restricted to only one bucket and s3:GetObject only if needs be.
CloudFormation Metadata for the EC2 instance
There are two key sections here: AWS::CloudFormation::Authentication that specifies how CloudFormation is expected to authenticate itself against the S3 bucket and AWS::CloudFormation::Init that specifies what needs to be done. In our case we only want to copy a file.
AWS::CloudFormation::Authentication
This section is responsible for providing credentials:
AWS::CloudFormation::Authentication:
S3AccessCreds:
type: "S3"
roleName:
Ref: rEC2BaseRoleThe main thing here is to specify the correct type of resource - S3.
Further details are available here: AWS Docs
AWS::CloudFormation::Init
This section specifies (in our case) where the file should be fetched from, where to put it and what permissions it should have (further details at link below, in this case we only have one config set, with only one file). Example below shows copies a deployment key.
AWS::CloudFormation::Init:
configSets:
init:
- repoKey
repoKey:
files:
"/root/ec2-base.key":
mode: '000400'
owner: root
group: root
source: "https://my-redacted-bucket.s3-ap-southeast-2.amazonaws.com/ec2-base.key"
authentication: "S3AccessCreds"Further details are available here: AWS Docs.
Instance details
There two important properties for the EC2::Instance:
- IamInstanceProfile
- UserData
The IamInstanceProfile must be a profile that points to the same Role as the one specified in the AWS::CloudFormation::Authentication section. A different Role with the same set of statements will not work, cfn-init will simply hang forever if the profile is not set up correctly.
The UserData section must contain invocation of the cfn-init script. That script is by default available on Amazon Linux:
UserData:
Fn::Base64:
Fn::Sub:
|
#!/bin/bash -xe
#pull in key file
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --region ${AWS::Region} --resource rTestInst --configsets initA few other details:
- UserData must be base64-encoded
- it must contain a shell script
- the
cfn-initcommand must reference the resource name, as specified in the CloudFormation template
The final template for the AWS::EC2::Instance looks like this:
rTestInst:
Type: "AWS::EC2::Instance"
Metadata:
AWS::CloudFormation::Authentication:
S3AccessCreds:
type: "S3"
roleName:
Fn::ImportValue:
Fn::Sub: "rEC2BaseWithSTSAssumeRole"
AWS::CloudFormation::Init:
configSets:
init:
- repoKey
repoKey:
files:
"/root/ec2-base.key":
mode: '000400'
owner: root
group: root
source: "https://hangarau-config.s3-ap-southeast-2.amazonaws.com/ec2-base"
authentication: "S3AccessCreds"
Properties:
AvailabilityZone:
Fn::Sub: "${AWS::Region}a"
InstanceType: "t3.nano"
ImageId: "ami-04481c741a0311bbb" #Amazon Linux 2
IamInstanceProfile:
Fn::ImportValue: "rEC2BaseProfile"
InstanceInitiatedShutdownBehavior: "stop"
KeyName: "aws-dev"
SubnetId:
Fn::ImportValue: "rSubnetPublicA"
SecurityGroupIds:
- Fn::GetAtt:
- rTestInstSG
- GroupId
Tags:
- Key: "Name"
Value: "testInstance"
- Key: "Account"
Value:
Ref: AWS::AccountId
UserData:
Fn::Base64:
Fn::Sub:
|
#!/bin/bash -xe
#pull in key file
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --region ${AWS::Region} --resource rTestInst --configsets initAnd when the instance is created the file is there:
[root@ip-172-31-1-250 ec2-user]# cd /root/
[root@ip-172-31-1-250 ~]# ls -al
total 24
dr-xr-x--- 5 root root 153 Jun 28 09:59 .
dr-xr-xr-x 18 root root 257 Jun 28 09:59 ..
drwxr-xr-x 4 root root 29 Jun 28 10:00 .ansible
-rw-r--r-- 1 root root 18 Oct 18 2017 .bash_logout
-rw-r--r-- 1 root root 176 Oct 18 2017 .bash_profile
-rw-r--r-- 1 root root 176 Oct 18 2017 .bashrc
drwxr-xr-x 3 root root 17 Jun 28 09:59 .cache
-rw-r--r-- 1 root root 100 Oct 18 2017 .cshrc
-r-------- 1 root root 1831 Jun 28 09:59 ec2-base.key <------------
drwx------ 2 root root 29 Jun 28 09:59 .ssh
-rw-r--r-- 1 root root 129 Oct 18 2017 .tcshrc