#availabilityzone | Explore Tumblr posts and blogs

theresawelchy · 6 years ago

Text

Faster File Distribution with HDFS and S3

In the Hadoop world there is almost always more than one way to accomplish a task. I prefer the platform because it's very unlikely I'm ever backed into a corner when working on a solution.

Hadoop has the ability to decouple storage from compute. The various distributed storage solutions supported all come with their own set of strong points and trade-offs. I often find myself needing to copy data back and forth between HDFS on AWS EMR and AWS S3 for performance reasons.

S3 is a great place to keep a master dataset as it can be used among many clusters without affect the performance of any one of them; it also comes with 11 9s of durability meaning it's one of the most unlikely places for data to go missing or become corrupt.

HDFS is where I find the best performance when running queries. If the workload will take long enough it's worth the time to copy a given dataset off of S3 and onto HDFS; any derivative results can then be transferred back onto S3 before the EMR cluster is terminated.

In this post I'll examine a number of different methods for copying data off of S3 and onto HDFS and see which is the fastest.

AWS EMR, Up & Running

To start, I'll launch an 11-node EMR cluster. I'll use the m3.xlarge instance type with 1 master node, 5 core nodes (these will make up the HDFS cluster) and 5 task nodes (these will run MapReduce jobs). I'm using spot pricing which often reduces the cost of the instances by 75-80% depending on market conditions. Both the EMR cluster and the S3 bucket are located in Ireland.

$ aws emr create-cluster --applications Name=Hadoop \ Name=Hive \ Name=Presto \ --auto-scaling-role EMR_AutoScaling_DefaultRole \ --ebs-root-volume-size 10 \ --ec2-attributes '{ "KeyName": "emr", "InstanceProfile": "EMR_EC2_DefaultRole", "AvailabilityZone": "eu-west-1c", "EmrManagedSlaveSecurityGroup": "sg-89cd3eff", "EmrManagedMasterSecurityGroup": "sg-d4cc3fa2"}' \ --enable-debugging \ --instance-groups '[{ "InstanceCount": 5, "BidPrice": "OnDemandPrice", "InstanceGroupType": "CORE", "InstanceType": "m3.xlarge", "Name": "Core - 2" },{ "InstanceCount": 5, "BidPrice": "OnDemandPrice", "InstanceGroupType": "TASK", "InstanceType": "m3.xlarge", "Name": "Task - 3" },{ "InstanceCount": 1, "BidPrice": "OnDemandPrice", "InstanceGroupType": "MASTER", "InstanceType": "m3.xlarge", "Name": "Master - 1" }]' \ --log-uri 's3n://aws-logs-591231097547-eu-west-1/elasticmapreduce/' \ --name 'My cluster' \ --region eu-west-1 \ --release-label emr-5.21.0 \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --service-role EMR_DefaultRole \ --termination-protected

After a few minutes the cluster has been launched and bootstrapped and I'm able to SSH in.

$ ssh -i ~/.ssh/emr.pem \ [email protected]

__| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/ 1 package(s) needed for security, out of 9 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E M:::::M M:::::M R:::R R::::R E::::::::::::::::::E M:::::M M:::::M RR::::R R::::R EEEEEEEEEEEEEEEEEEEE MMMMMMM MMMMMMM RRRRRRR RRRRRR

The five core nodes each have 68.95 GB of capacity that together create 344.75 GB of capacity across the HDFS cluster.

$ hdfs dfsadmin -report \ | grep 'Configured Capacity'

Configured Capacity: 370168258560 (344.75 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB)

The dataset I'll be using in this benchmark is a data dump I've produced of 1.1 billion taxi trips conducted in New York City over a six year period. The Billion Taxi Rides in Redshift blog post goes into detail on how I put this dataset together. This dataset is approximately 86 GB in ORC format spread across 56 files. The typical ORC file is ~1.6 GB in size.

I'll create a filename manifest that I'll use for various operations below. I'll exclude the S3 URL prefix as these names will also be used to address files on HDFS as well.

000000_0 000001_0 000002_0 000003_0 000004_0 000005_0 000006_0 000007_0 000008_0 000009_0 000010_0 000011_0 000012_0 000013_0 000014_0 000015_0 000016_0 000017_0 000018_0 000019_0 000020_0 000021_0 000022_0 000023_0 000024_0 000025_0 000026_0 000027_0 000028_0 000029_0 000030_0 000031_0 000032_0 000033_0 000034_0 000035_0 000036_0 000037_0 000038_0 000039_0 000040_0 000041_0 000042_0 000043_0 000044_0 000045_0 000046_0 000047_0 000048_0 000049_0 000050_0 000051_0 000052_0 000053_0 000054_0 000055_0

I'll adjust the AWS CLI's configuration to allow for up to 100 concurrent requests at any one time.

$ aws configure set \ default.s3.max_concurrent_requests \ 100

The disk space on the master node cannot hold the entire 86 GB worth of ORC files so I'll download, import onto HDFS and remove each file one at a time. This will allow me to maintain enough working disk space on the master node.

$ hdfs dfs -mkdir /orc $ time (for FILE in `cat files`; do aws s3 cp s3://<bucket>/orc/$FILE ./ hdfs dfs -copyFromLocal $FILE /orc/ rm $FILE done)

The above completed 15 minutes and 57 seconds.

The HDFS CLI uses the JVM which comes with a fair amount of overhead. In my HDFS CLI benchmark I found the alternative CLI gohdfs could save a lot of start-up time as it is written in GoLang and doesn't run on the JVM. Below I've run the same operation using gohdfs.

$ wget -c -O gohdfs.tar.gz \ https://github.com/colinmarc/hdfs/releases/download/v2.0.0/gohdfs-v2.0.0-linux-amd64.tar.gz $ tar xvf gohdfs.tar.gz

I'll clear out the previously downloaded dataset off HDFS first so there is enough space on the cluster going forward. With triple replication, 86 GB turns into 258 GB on disk and there is only 344.75 GB of HDFS capacity in total.

$ hdfs dfs -rm -r -skipTrash /orc $ hdfs dfs -mkdir /orc $ time (for FILE in `cat files`; do aws s3 cp s3://<bucket>/orc/$FILE ./ gohdfs-v2.0.0-linux-amd64/hdfs put \ $FILE \ hdfs://ip-10-10-207-160.eu-west-1.compute.internal:8020/orc/ rm $FILE done)

The above took 27 minutes and 40 seconds. I wasn't expecting this client to be almost twice as slow as the HDFS CLI. S3 provides consistent performance when I've run other tools multiple times so I suspect either the code behind the put functionality could be optimised or there might be a more appropriate endpoint for copying multi-gigabyte files onto HDFS. As of this writing I can't find support for copying from S3 to HDFS directly with resorting to a file system fuse.

The HDFS CLI does support copying from S3 to HDFS directly. Below I'll copy the 56 ORC files to HDFS straight from S3. I'll set the concurrent process limit to 8.

$ hdfs dfs -rm -r -skipTrash /orc $ hdfs dfs -mkdir /orc $ time (cat files \ | xargs -n 1 \ -P 8 \ -I % \ hdfs dfs -cp s3://<bucket>/orc/% /orc/)

The above took 14 minutes and 17 seconds. There wasn't much of an improvement over simply copying the files down one at a time and uploading them to HDFS.

I'll try the above command again but set the concurrency limit to 16 processes. Note, this is running on the master node which has 15 GB of RAM and the following will use what little memory capacity is left on the machine.

$ hdfs dfs -rm -r -skipTrash /orc $ hdfs dfs -mkdir /orc $ time (cat files \ | xargs -n 1 \ -P 16 \ -I % \ hdfs dfs -cp s3://<bucket>/orc/% /orc/)

The above took 14 minutes and 36 seconds. Again, a very similar time despite a higher concurrency limit. The effective transfer rate was ~98.9 MB/s off of S3. HDFS is configured for triple redundancy but I expect there is a lot more throughput available with a cluster of this size.

DistCp (distributed copy) is bundled with Hadoop and uses MapReduce to copy files in a distributed manner. It can work with HDFS, AWS S3, Azure Blob Storage and Google Cloud Storage. It can break up the downloading and importing across the task nodes so all five machines can work on a single job instead of the master node being the single machine downloading and importing onto HDFS.

$ hdfs dfs -rm -r -skipTrash /orc $ hdfs dfs -mkdir /orc $ time (hadoop distcp s3://<bucket>/orc/* /orc)

The above completed in 6 minutes and 16 seconds. A huge improvement over the previous methods.

On AWS EMR, there is a tool called S3DistCp that aims to provide the functionality of Hadoop's DistCp but in a fashion optimised for S3. Like DistCp, it uses MapReduce for executing its operations.

$ hdfs dfs -rm -r -skipTrash /orc $ hdfs dfs -mkdir /orc $ time (s3-dist-cp \ --src=s3://<bucket>/orc/ \ --dest=hdfs:///orc/)

The above completed in 5 minutes and 59 seconds. This gives an effective throughput of ~241 MB/s off of S3. There wasn't a huge performance increase over DistCp and I suspect neither tool can greatly out-perform the other.

I did come across settings to increase the chunk size from 128 MB to 1 GB, which would be useful for larger files but enough tooling in the Hadoop ecosystem will suffer from ballooning memory requirements with files over 2 GB that it is very rare to see files larger than this in any sensibly-deployed production environment. S3 usually has low connection setup latency so I can't see this being a huge overhead.

With the above its now understood that both DistCp and S3DistCp can leverage a cluster's task nodes to import data from S3 onto HDFS quickly. I'm going to see how well these tools scale with a 21-node m3.xlarge cluster. This cluster will have 1 master node, 10 core nodes and 10 task nodes.

$ aws emr create-cluster \ --applications Name=Hadoop \ Name=Hive \ Name=Presto \ --auto-scaling-role EMR_AutoScaling_DefaultRole \ --ebs-root-volume-size 10 \ --ec2-attributes '{ "KeyName": "emr", "InstanceProfile": "EMR_EC2_DefaultRole", "AvailabilityZone": "eu-west-1c", "EmrManagedSlaveSecurityGroup": "sg-89cd3eff", "EmrManagedMasterSecurityGroup": "sg-d4cc3fa2"}' \ --enable-debugging \ --instance-groups '[{ "InstanceCount": 1, "BidPrice": "OnDemandPrice", "InstanceGroupType": "MASTER", "InstanceType": "m3.xlarge", "Name": "Master - 1" },{ "InstanceCount": 10, "BidPrice": "OnDemandPrice", "InstanceGroupType": "CORE", "InstanceType": "m3.xlarge", "Name": "Core - 2" },{ "InstanceCount": 10, "BidPrice": "OnDemandPrice", "InstanceGroupType": "TASK", "InstanceType": "m3.xlarge", "Name": "Task - 3" }]' \ --log-uri 's3n://aws-logs-591231097547-eu-west-1/elasticmapreduce/' \ --name 'My cluster' \ --region eu-west-1 \ --release-label emr-5.21.0 \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --service-role EMR_DefaultRole \ --termination-protected

With the new EMR cluster up and running I can SSH into it.

$ ssh -i ~/.ssh/emr.pem \ [email protected]

Each core node on the HDFS cluster still has 68.95 GB of capacity but the ten machines combined create 689.49 GB of HDFS storage capacity.

$ hdfs dfsadmin -report \ | grep 'Configured Capacity'

Configured Capacity: 740336517120 (689.49 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB) Configured Capacity: 74033651712 (68.95 GB)

I'll run S3DistCp first.

$ hdfs dfs -mkdir /orc $ time (s3-dist-cp \ --src=s3://<bucket>/orc/ \ --dest=hdfs:///orc/)

The above completed in 4 minutes and 56 seconds. This is an improvement over the 11-node cluster but not the 2x improvement I was expecting.

Below is DistCp running on the 21-node cluster.

$ hdfs dfs -rm -r -skipTrash /orc $ hdfs dfs -mkdir /orc $ time (hadoop distcp s3://<bucket>/orc/* /orc)

The above completed in 4 minutes and 44 seconds.

The performance ratio between these two tools is more or less consistent between cluster sizes. Its a shame neither showed linear scaling with twice the number of core and task nodes.

Here is a recap of the transfer times seen in this post.

Duration Transfer Method 27m40s gohdfs, Sequentially 15m57s HDFS DFS CLI, Sequentially 14m36s HDFS DFS CLI, Concurrently x 16 14m17s HDFS DFS CLI, Concurrently x 8 6m16s Hadoop DistCp, 11-Node Cluster 5m59s S3DistCp, 11-Node Cluster 4m56s S3DistCp, 21-Node Cluster 4m44s Hadoop DistCp, 21-Node Cluster

Why Use HDFS At All?

S3 is excellent for durability and doesn't suffer performance-wise if you have one cluster or ten clusters pointed at it. S3 also works as well as HDFS when appending records to a dataset. Both of the following queries will run without issue.

$ presto-cli \ --schema default \ --catalog hive

INSERT INTO trips_hdfs SELECT * from trips_hdfs LIMIT 10;

INSERT INTO trips_s3 SELECT * from trips_s3 LIMIT 10;

I've heard arguments that S3 is as fast as HDFS but I've never witnessed this in my time with both technologies. Below I'll run a benchmark on the 1.1 billion taxi trips. I'll have one table using S3-backed data and another table using HDFS-backed data. This was run on the 11-node cluster. I ran each query multiple times and recorded the fastest times.

This is the HDFS-backed table.

CREATE EXTERNAL TABLE trips_hdfs ( trip_id INT, vendor_id STRING, pickup_datetime TIMESTAMP, dropoff_datetime TIMESTAMP, store_and_fwd_flag STRING, rate_code_id SMALLINT, pickup_longitude DOUBLE, pickup_latitude DOUBLE, dropoff_longitude DOUBLE, dropoff_latitude DOUBLE, passenger_count SMALLINT, trip_distance DOUBLE, fare_amount DOUBLE, extra DOUBLE, mta_tax DOUBLE, tip_amount DOUBLE, tolls_amount DOUBLE, ehail_fee DOUBLE, improvement_surcharge DOUBLE, total_amount DOUBLE, payment_type STRING, trip_type SMALLINT, pickup STRING, dropoff STRING, cab_type STRING, precipitation SMALLINT, snow_depth SMALLINT, snowfall SMALLINT, max_temperature SMALLINT, min_temperature SMALLINT, average_wind_speed SMALLINT, pickup_nyct2010_gid SMALLINT, pickup_ctlabel STRING, pickup_borocode SMALLINT, pickup_boroname STRING, pickup_ct2010 STRING, pickup_boroct2010 STRING, pickup_cdeligibil STRING, pickup_ntacode STRING, pickup_ntaname STRING, pickup_puma STRING, dropoff_nyct2010_gid SMALLINT, dropoff_ctlabel STRING, dropoff_borocode SMALLINT, dropoff_boroname STRING, dropoff_ct2010 STRING, dropoff_boroct2010 STRING, dropoff_cdeligibil STRING, dropoff_ntacode STRING, dropoff_ntaname STRING, dropoff_puma STRING ) STORED AS orc LOCATION '/orc/';

This is the S3-backed table.

CREATE EXTERNAL TABLE trips_s3 ( trip_id INT, vendor_id STRING, pickup_datetime TIMESTAMP, dropoff_datetime TIMESTAMP, store_and_fwd_flag STRING, rate_code_id SMALLINT, pickup_longitude DOUBLE, pickup_latitude DOUBLE, dropoff_longitude DOUBLE, dropoff_latitude DOUBLE, passenger_count SMALLINT, trip_distance DOUBLE, fare_amount DOUBLE, extra DOUBLE, mta_tax DOUBLE, tip_amount DOUBLE, tolls_amount DOUBLE, ehail_fee DOUBLE, improvement_surcharge DOUBLE, total_amount DOUBLE, payment_type STRING, trip_type SMALLINT, pickup STRING, dropoff STRING, cab_type STRING, precipitation SMALLINT, snow_depth SMALLINT, snowfall SMALLINT, max_temperature SMALLINT, min_temperature SMALLINT, average_wind_speed SMALLINT, pickup_nyct2010_gid SMALLINT, pickup_ctlabel STRING, pickup_borocode SMALLINT, pickup_boroname STRING, pickup_ct2010 STRING, pickup_boroct2010 STRING, pickup_cdeligibil STRING, pickup_ntacode STRING, pickup_ntaname STRING, pickup_puma STRING, dropoff_nyct2010_gid SMALLINT, dropoff_ctlabel STRING, dropoff_borocode SMALLINT, dropoff_boroname STRING, dropoff_ct2010 STRING, dropoff_boroct2010 STRING, dropoff_cdeligibil STRING, dropoff_ntacode STRING, dropoff_ntaname STRING, dropoff_puma STRING ) STORED AS orc LOCATION 's3://<bucket>/orc/';

I'll use Presto to run the benchmarks.

$ presto-cli \ --schema default \ --catalog hive

The following four queries were run on the HDFS-backed table.

The following completed in 6.77 seconds.

SELECT cab_type, count(*) FROM trips_hdfs GROUP BY cab_type;

The following completed in 10.97 seconds.

SELECT passenger_count, avg(total_amount) FROM trips_hdfs GROUP BY passenger_count;

The following completed in 13.38 seconds.

SELECT passenger_count, year(pickup_datetime), count(*) FROM trips_hdfs GROUP BY passenger_count, year(pickup_datetime);

The following completed in 19.82 seconds.

SELECT passenger_count, year(pickup_datetime) trip_year, round(trip_distance), count(*) trips FROM trips_hdfs GROUP BY passenger_count, year(pickup_datetime), round(trip_distance) ORDER BY trip_year, trips desc;

The following four queries were run on the S3-backed table.

The following completed in 10.82 seconds.

SELECT cab_type, count(*) FROM trips_s3 GROUP BY cab_type;

The following completed in 14.73 seconds.

SELECT passenger_count, avg(total_amount) FROM trips_s3 GROUP BY passenger_count;

The following completed in 19.19 seconds.

SELECT passenger_count, year(pickup_datetime), count(*) FROM trips_s3 GROUP BY passenger_count, year(pickup_datetime);

The following completed in 24.61 seconds.

SELECT passenger_count, year(pickup_datetime) trip_year, round(trip_distance), count(*) trips FROM trips_s3 GROUP BY passenger_count, year(pickup_datetime), round(trip_distance) ORDER BY trip_year, trips desc;

This is a recap of the query times above.

HDFS on AWS EMR AWS S3 Speed Up Query 6.77s 10.82s 1.6x Query 1 10.97s 14.73s 1.34x Query 2 13.38s 19.19s 1.43x Query 3 19.82s 24.61s 1.24x Query 4

The HDFS-backed queries were anywhere from 1.24x to 1.6x faster than the S3-backed queries.

DataTau published first on DataTau

#Technology #Finance

0 notes

clunite · 7 years ago

Photo

Store infrequently accessed S3 objects in ONEZONE_IA class storage, less expensive because it stores data in only one Availability Zone: https://buff.ly/2vEwSri https://buff.ly/2HKSCGy⠀ .⠀ .⠀ .⠀ .⠀ .⠀ .⠀ .⠀ .⠀ .⠀ .⠀ #aws #amazonwebservices #cloud #cloudcomputing #awscloud #amazoncloud #amazon #s3 #amazons3 #onezoneai #storage #availabilityzone #cloudstorage @awscloud https://ift.tt/2HGzSrG

#https://www.instagram.com/p/Bh4qi54Bo3j/embed #612 #IFTTT #Instagram <iframe src= width= height="710

0 notes

infosavvy · 5 years ago

Link

Introduction to VPC Elastic Network Interfaces is an elastic network interface (ENI) allows an instance to communicate with other network resources including AWS services, other instances, on-premises servers, and the Internet.

Read More : https://www.info-savvy.com/introduction-to-vpc-elastic-network-interfaces/

#addresses #AvailabilityZones #CIDRblock #ElasticNetworkInterfaces #InboundRules #instance #InternetGateways #OutboundRules #onlinetraining&certification #AWSTraining

0 notes

infosavvy · 5 years ago

Text

Overview of an Amazon Virtual Private Cloud

Overview of an Amazon Virtual Private Cloud Virtual service provides the networking layer of EC2. A VPC is a virtual network that can contain EC2 instances as well as network resources for other AWS services. By default, every VPC is isolated from all other networks.

Read More : https://www.info-savvy.com/overview-of-an-amazon-virtual-private-cloud/

#addresses #availability #AvailabilityZones #CIDRblock #instances #ip #network #prefix #SubnetCIDRBlocks #onlinetraining&certification #AWSTraining

0 notes

infosavvy · 5 years ago

Text

Introduction to VPC Elastic Network Interfaces

Read More : https://www.info-savvy.com/introduction-to-vpc-elastic-network-interfaces/

#addresses #AvailabilityZones #CIDRblock #ElasticNetworkInterfaces #InboundRules #AWSTraining #onlinetraining&certification #awscertification #onlinecourses

0 notes

infosavvy · 5 years ago

Link

You will learn into this regarding cloud computing and their six advantages as well as types of cloud computing in detail.

Read More : https://www.info-savvy.com/introduction-to-cloud-computing-and-aws/

#types of cloud computing #AvailabilityZones #AWSRegions #AWSSecurity #BenefitsofAWSSecurity #CloudComputing #CloudComputingDeploymentModels #CloudComputingModels #GlobalInfrastructure #InfrastructureasaService #ITresources

0 notes

awsexchage · 6 years ago

Photo

AWS CloudFormationでEC2インスタンスのログがAmazon CloudWatch logsで確認できるようにする https://ift.tt/33zfnVY

AWS CloudFormation(CFn)でEC2インスタンスを作成・管理する際にユーザーデータやAWS::CloudFormation::Initタイプを利用したメタデータで環境構築するのが便利なのですが、スタック作成・更新時のログはEC2インスタンス内にあるため、確認が面倒です。

EC2のログをAmazon CloudWatch Logsへ送信する仕組みとして「CloudWatch Logs エージェント」があると知ったのでCFnでEC2インスタンスを管理する際に利用できるテンプレートを作成してみました。

CloudWatch Logs エージェントのリファレンス – Amazon CloudWatch Logs https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/AgentReference.html

前提

AWSアカウントがある

AWS CLIがインストール済みで利用可能

CFn、EC2関連の権限がある

テンプレート

下記で作成したテンプレートにawslogsを利用する定義を追加しました。

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた – Qiita https://cloudpack.media/48540

EC2インスタンスを起動するVPCやサブネットは既存のリソースを利用する前提となります。 SSHアクセスする際のキーペアは事前に作成し、セキュリティグループはインスタンスとあわせて作成しています。作成するリージョンは検証なので、us-east-1のみとしています。

cfn-template.yaml

Parameters: VpcId: Type: AWS::EC2::VPC::Id SubnetId: Type: AWS::EC2::Subnet::Id EC2KeyPairName: Type: AWS::EC2::KeyPair::KeyName InstanceType: Type: String Default: t3.small MyInstanceSSHCidrIp: Type: String Default: '0.0.0.0/0' Mappings: AWSRegionToAMI: us-east-1: HVM64: ami-0080e4c5bc078760e Resources: MyInstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupName: !Join - " " - - !Ref AWS::StackName - "SecurityGroup" GroupDescription: "MyInstance SecurityGroup" VpcId: !Ref VpcId SecurityGroupEgress: - CidrIp: "0.0.0.0/0" IpProtocol: "-1" SecurityGroupIngress: - CidrIp: !Ref MyInstanceSSHCidrIp Description: !Join - " " - - !Ref AWS::StackName - "SSH Port" IpProtocol: "tcp" FromPort: 22 ToPort: 22 Tags: - Key: "Name" Value: !Join - " " - - !Ref AWS::StackName - "SecurityGroup" MyInstanceIAMRole: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "ec2.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" Policies: - PolicyName: "root" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - "logs:CreateLogGroup" - "logs:CreateLogStream" - "logs:PutLogEvents" - "logs:DescribeLogStreams" Resource: "arn:aws:logs:*:*:*" MyInstanceIAMInstanceProfile: Type: "AWS::IAM::InstanceProfile" Properties: Path: "/" Roles: - Ref: MyInstanceIAMRole DependsOn: MyInstanceIAMRole MyInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: packages: yum: awslogs: [] files: /etc/awslogs/awslogs.conf: content: !Sub | [general] state_file = /var/lib/awslogs/agent-state [/var/log/cloud-init-output.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cloud-init-output.log buffer_duration = 5000 log_stream_name = {instance_id}/cloud-init-output.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init-cmd.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init-cmd.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init-cmd.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-hup.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-hup.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-hup.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} mode: "000644" owner: "root" group: "root" /etc/cfn/cfn-hup.conf: content: !Sub | [main] stack = ${AWS::StackName} region = ${AWS::Region} mode: "000400" owner: "root" group: "root" /etc/cfn/hooks.d/cfn-auto-reloader.conf: content: !Sub | [cfn-auto-reloader-hook] triggers = post.update path = Resources.MyInstance.Metadata.AWS::CloudFormation::Init action = /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} runas = root mode: "000400" owner: "root" group: "root" commands: test: command: "echo $STACK_NAME test" env: STACK_NAME: !Ref AWS::StackName services: sysvinit: cfn-hup: enabled: "true" files: - /etc/cfn/cfn-hup.conf - /etc/cfn/hooks.d/cfn-auto-reloader.conf awslogs: enabled: "true" packages: - awslogs files: - /etc/awslogs/awslogs.conf Properties: InstanceType: !Ref InstanceType KeyName: !Ref EC2KeyPairName ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", HVM64 ] IamInstanceProfile: !Ref MyInstanceIAMInstanceProfile NetworkInterfaces: - AssociatePublicIpAddress: True DeviceIndex: 0 GroupSet: - !Ref MyInstanceSecurityGroup SubnetId: !Ref SubnetId Tags: - Key: 'Name' Value: !Ref AWS::StackName UserData: Fn::Base64: !Sub | #!/bin/bash echo "start UserData" /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} echo "finish UserData" CreationPolicy: ResourceSignal: Timeout: PT5M Outputs: ClientInstanceId: Value: !Ref MyInstance ClientPublicIp: Value: !GetAtt MyInstance.PublicIp

ポイント

IAMロールを作成する

EC2インスタンスからAmazon CloudWatch Logsへログを送るのに権限が必要となるためIAMロールを作成します。必要となる権限については下記を参考にしました。

クイックスタート: 実行中の EC2 Linux インスタンスに CloudWatch Logs エージェントをインストールして設定する – Amazon CloudWatch Logs https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

cfn-template.yaml_一部抜粋

MyInstanceIAMRole: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "ec2.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" Policies: - PolicyName: "root" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - "logs:CreateLogGroup" - "logs:CreateLogStream" - "logs:PutLogEvents" - "logs:DescribeLogStreams" Resource: "arn:aws:logs:*:*:*" MyInstanceIAMInstanceProfile: Type: "AWS::IAM::InstanceProfile" Properties: Path: "/" Roles: - Ref: MyInstanceIAMRole DependsOn: MyInstanceIAMRole MyInstance: Type: AWS::EC2::Instance (略) Properties: InstanceType: !Ref InstanceType KeyName: !Ref EC2KeyPairName ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", HVM64 ] IamInstanceProfile: !Ref MyInstanceIAMInstanceProfile (略)

awslogs パッケージを利用する

CloudWatch Logs エージェントはawslogsパッケージとして提供されています。awslogsパッケージはAmazon LinuxとUbuntuに対応しています。

パッケージのインストール、設定ファイル、サービス起動はメタデータで定義して設定変更できるようにしました。 /etc/awslogs/awslogs.confファイルでAmazon CloudWatch Logsに出力するログファイルを定義します。log_stream_nameやlog_group_name は任意で指定が可能です。ここでは/var/log/内に出力されるメタデータやユーザーデータによる処理のログを対象としました。

cfn-template.yaml_一部抜粋

MyInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: packages: yum: awslogs: [] files: /etc/awslogs/awslogs.conf: content: !Sub | [general] state_file = /var/lib/awslogs/agent-state [/var/log/cloud-init-output.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cloud-init-output.log buffer_duration = 5000 log_stream_name = {instance_id}/cloud-init-output.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init-cmd.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init-cmd.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init-cmd.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-hup.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-hup.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-hup.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} mode: "000644" owner: "root" group: "root" (略) services: sysvinit: (略) awslogs: enabled: "true" packages: - awslogs files: - /etc/awslogs/awslogs.conf

利用方法

パラメータ値を取得

キーペア

EC2インスタンスへSSHログインするのに利用するキーペアを作成します。既存のキーペアを利用する場合は作成不要です。

# Create KeyPair > aws ec2 create-key-pair \ --key-name cfn-awslogs-test-ec2-key \ --query "KeyMaterial" \ --output text > cfn-awslogs-test-ec2-key.pem > chmod 400 cfn-awslogs-test-ec2-key.pem

VPC、サブネット

既存のVPC、サブネット配下にEC2インスタンスを作成する前提ですので各IDを取得します。

# VpcId > aws ec2 describe-vpcs \ --query "Vpcs" [ { "CidrBlock": "172.31.0.0/16", "DhcpOptionsId": "dopt-b06bd8c8", "State": "available", "VpcId": "vpc-xxxxxxxx", "OwnerId": "xxxxxxxxxxxx", "InstanceTenancy": "default", "CidrBlockAssociationSet": [ { "AssociationId": "vpc-cidr-assoc-2b23e646", "CidrBlock": "172.31.0.0/16", "CidrBlockState": { "State": "associated" } } ], "IsDefault": true }, ] # SubnetId > aws ec2 describe-subnets \ --filters '{"Name": "vpc-id", "Values": ["vpc-xxxxxxxx"]}' \ --query "Subnets" [ { "AvailabilityZone": "us-east-1a", "AvailabilityZoneId": "use1-az2", "AvailableIpAddressCount": 4089, "CidrBlock": "172.31.80.0/20", "DefaultForAz": true, "MapPublicIpOnLaunch": true, "State": "available", "SubnetId": "subnet-xxxxxxxx", "VpcId": "vpc-xxxxxxxx", "OwnerId": "xxxxxxxxxxxx", "AssignIpv6AddressOnCreation": false, "Ipv6CidrBlockAssociationSet": [], "SubnetArn": "arn:aws:ec2:us-east-1:xxxxxxxxxxxx:subnet/subnet-xxxxxxxx" }, ]

自身のグローバルIP

SSHアクセスするためのセキュリティグループに指定するIPアドレスを取得します。ここではifconfig.ioを利用していますが他の手段でもOKです。

## GlobalIP > curl ifconfig.io xxx.xxx.xxx.xxx

スタック作成

aws cloudformation create-stackコマンドを利用してスタック作成します。 AWSマネジメントコンソールから作成してもOKです。

> aws cloudformation create-stack \ --stack-name cfn-awslogs-test \ --template-body file://cfn-template.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 \ --parameters '[ { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-awslogs-test-ec2-key" }, { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" } ]'

スタック作成結果を確認

スタック作成ができたか確認します。

> aws cloudformation describe-stacks \ --stack-name cfn-awslogs-test { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-awslogs-test/f1e864b0-a9fd-11e9-aa93-12ccfe651680", "StackName": "cfn-awslogs-test", "Parameters": [ { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" }, { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "InstanceType", "ParameterValue": "t3.small" } ], "CreationTime": "2019-07-19T08:19:36.199Z", "RollbackConfiguration": {}, "StackStatus": "CREATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Outputs": [ { "OutputKey": "ClientPublicIp", "OutputValue": "xxx.xxx.xxx.xxx" }, { "OutputKey": "ClientInstanceId", "OutputValue": "i-xxxxxxxxxxxxxxxxx" } ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] }

Amazon CloudWatch logsでログを確認する

> aws logs describe-log-streams \ --log-group-name /aws/ec2/cfn-awslogs-test \ --query "logGroups[*].logGroupName" [ "i-xxxxxxxxxxxxxxxxx/cfn-hup.log", "i-xxxxxxxxxxxxxxxxx/cfn-init-cmd.log", "i-xxxxxxxxxxxxxxxxx/cfn-init.log", "i-xxxxxxxxxxxxxxxxx/cloud-init-output.log" ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-hup.log \ --query "events[*].message" [ "2019-07-19 08:22:57,542 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com", "2019-07-19 08:22:57,543 [DEBUG] Creating /var/lib/cfn-hup/data", "2019-07-19 08:22:57,551 [INFO] No umask value specified in config file. Using the default one: 022", "2019-07-19 08:37:57,714 [INFO] cfn-hup processing is alive." ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-init-cmd.log \ --query "events[*].message" \ [ "2019-07-19 08:22:49,803 P2190 [INFO] ************************************************************", "2019-07-19 08:22:49,803 P2190 [INFO] ConfigSet default", "2019-07-19 08:22:49,804 P2190 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++", "2019-07-19 08:22:49,804 P2190 [INFO] Config config", "2019-07-19 08:22:55,092 P2190 [INFO] ============================================================", "2019-07-19 08:22:55,092 P2190 [INFO] yum install awslogs", "2019-07-19 08:22:56,958 P2190 [INFO] -----------------------Command Output-----------------------", "2019-07-19 08:22:56,958 P2190 [INFO] \tLoaded plugins: priorities, update-motd, upgrade-helper", "2019-07-19 08:22:56,958 P2190 [INFO] \tResolving Dependencies", "2019-07-19 08:22:56,958 P2190 [INFO] \t--> Running transaction check", "2019-07-19 08:22:56,958 P2190 [INFO] \t---> Package awslogs.noarch 0:1.1.4-1.12.amzn1 will be installed", "2019-07-19 08:22:56,958 P2190 [INFO] \t--> Processing Dependency: aws-cli-plugin-cloudwatch-logs(python27) for package: awslogs-1.1.4-1.12.amzn1.noarch", "2019-07-19 08:22:56,959 P2190 [INFO] \t--> Running transaction check", "2019-07-19 08:22:56,959 P2190 [INFO] \t---> Package aws-cli-plugin-cloudwatch-logs.noarch 0:1.4.4-1.16.amzn1 will be installed", "2019-07-19 08:22:56,959 P2190 [INFO] \t--> Finished Dependency Resolution", "2019-07-19 08:22:56,959 P2190 [INFO] \t", "2019-07-19 08:22:56,959 P2190 [INFO] \tDependencies Resolved", "2019-07-19 08:22:56,959 P2190 [INFO] \t", "2019-07-19 08:22:56,959 P2190 [INFO] \t================================================================================", "2019-07-19 08:22:56,959 P2190 [INFO] \t Package Arch Version Repository Size", "2019-07-19 08:22:56,959 P2190 [INFO] \t================================================================================", "2019-07-19 08:22:56,959 P2190 [INFO] \tInstalling:", "2019-07-19 08:22:56,959 P2190 [INFO] \t awslogs noarch 1.1.4-1.12.amzn1 amzn-main 9.2 k", "2019-07-19 08:22:56,959 P2190 [INFO] \tInstalling for dependencies:", "2019-07-19 08:22:56,960 P2190 [INFO] \t aws-cli-plugin-cloudwatch-logs noarch 1.4.4-1.16.amzn1 amzn-main 71 k", "2019-07-19 08:22:56,960 P2190 [INFO] \t", "2019-07-19 08:22:56,960 P2190 [INFO] \tTransaction Summary", "2019-07-19 08:22:56,960 P2190 [INFO] \t================================================================================", "2019-07-19 08:22:56,960 P2190 [INFO] \tInstall 1 Package (+1 Dependent package)", "2019-07-19 08:22:56,960 P2190 [INFO] \t", "2019-07-19 08:22:56,960 P2190 [INFO] \tTotal download size: 81 k", "2019-07-19 08:22:56,960 P2190 [INFO] \tInstalled size: 246 k", "2019-07-19 08:22:56,960 P2190 [INFO] \tDownloading packages:", "2019-07-19 08:22:56,960 P2190 [INFO] \t--------------------------------------------------------------------------------", "2019-07-19 08:22:56,960 P2190 [INFO] \tTotal 250 kB/s | 81 kB 00:00 ", "2019-07-19 08:22:56,960 P2190 [INFO] \tRunning transaction check", "2019-07-19 08:22:56,961 P2190 [INFO] \tRunning transaction test", "2019-07-19 08:22:56,961 P2190 [INFO] \tTransaction test succeeded", "2019-07-19 08:22:56,961 P2190 [INFO] \tRunning transaction", "2019-07-19 08:22:56,961 P2190 [INFO] \t Installing : aws-cli-plugin-cloudwatch-logs-1.4.4-1.16.amzn1.noarch 1/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t Installing : awslogs-1.1.4-1.12.amzn1.noarch 2/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t Verifying : awslogs-1.1.4-1.12.amzn1.noarch 1/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t Verifying : aws-cli-plugin-cloudwatch-logs-1.4.4-1.16.amzn1.noarch 2/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t", "2019-07-19 08:22:56,962 P2190 [INFO] \tInstalled:", "2019-07-19 08:22:56,962 P2190 [INFO] \t awslogs.noarch 0:1.1.4-1.12.amzn1 ", "2019-07-19 08:22:56,962 P2190 [INFO] \t", "2019-07-19 08:22:56,962 P2190 [INFO] \tDependency Installed:", "2019-07-19 08:22:56,962 P2190 [INFO] \t aws-cli-plugin-cloudwatch-logs.noarch 0:1.4.4-1.16.amzn1 ", "2019-07-19 08:22:56,962 P2190 [INFO] \t", "2019-07-19 08:22:56,962 P2190 [INFO] \tComplete!", "2019-07-19 08:22:56,962 P2190 [INFO] ------------------------------------------------------------", "2019-07-19 08:22:56,962 P2190 [INFO] Completed successfully.", "2019-07-19 08:22:56,965 P2190 [INFO] ============================================================", "2019-07-19 08:22:56,965 P2190 [INFO] Command test", "2019-07-19 08:22:56,969 P2190 [INFO] -----------------------Command Output-----------------------", "2019-07-19 08:22:56,969 P2190 [INFO] \tcfn-awslogs-test test", "2019-07-19 08:22:56,969 P2190 [INFO] ------------------------------------------------------------", "2019-07-19 08:22:56,970 P2190 [INFO] Completed successfully." ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-init.log \ --query "events[*].message" [ "2019-07-19 08:22:49,801 [INFO] -----------------------Starting build-----------------------", "2019-07-19 08:22:49,802 [INFO] Running configSets: default", "2019-07-19 08:22:49,803 [INFO] Running configSet default", "2019-07-19 08:22:49,804 [INFO] Running config config", "2019-07-19 08:22:56,962 [INFO] Yum installed [u'awslogs']", "2019-07-19 08:22:56,970 [INFO] Command test succeeded", "2019-07-19 08:22:56,975 [INFO] enabled service cfn-hup", "2019-07-19 08:22:57,568 [INFO] Restarted cfn-hup successfully", "2019-07-19 08:22:57,607 [INFO] enabled service awslogs", "2019-07-19 08:22:59,648 [INFO] Restarted awslogs successfully", "2019-07-19 08:22:59,649 [INFO] ConfigSets completed", "2019-07-19 08:22:59,650 [INFO] -----------------------Build complete-----------------------", "2019-07-19 08:23:00,249 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com", "2019-07-19 08:23:00,250 [DEBUG] Signaling resource MyInstance in stack cfn-awslogs-test with unique ID i-xxxxxxxxxxxxxxxxx and status SUCCESS" ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cloud-init-output.log \ --query "events[*].message" (略) "Updated:\n java-1.7.0-openjdk.x86_64 1:1.7.0.211-2.6.17.1.79.amzn1 \n kernel-tools.x86_64 0:4.14.128-87.105.amzn1 \n perl.x86_64 4:5.16.3-294.43.amzn1 \n perl-Pod-Escapes.noarch 1:1.04-294.43.amzn1 \n perl-libs.x86_64 4:5.16.3-294.43.amzn1 \n perl-macros.x86_64 4:5.16.3-294.43.amzn1 \n python27-jinja2.noarch 0:2.7.2-3.16.amzn1 \n wget.x86_64 0:1.18-5.30.amzn1 \n", "Complete!", "Cloud-init v. 0.7.6 running 'modules:final' at Fri, 19 Jul 2019 08:22:49 +0000. Up 20.37 seconds.", "start UserData", "finish UserData", "Cloud-init v. 0.7.6 finished at Fri, 19 Jul 2019 08:23:00 +0000. Datasource DataSourceEc2. Up 31.51 seconds" ]

まとめ

CloudWatch Logs エージェントを利用することでCFnでEC2インスタンス作成・更新時のログをAmazon CloudWatch Logsへ出力することができました。個人的にはこれでCFnで実行エラーによるロールバックでEC2インスタンスが削除されてログが確認できない悲しみがなくなって満足です^^

参考

CloudWatch Logs エージェントのリファレンス – Amazon CloudWatch Logs https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/AgentReference.html

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた – Qiita https://cloudpack.media/48540

元記事はこちら

「AWS CloudFormationでEC2インスタンスのログがAmazon CloudWatch logsで確認できるようにする」

August 14, 2019 at 12:00PM

#AWS #Official #Blog

0 notes

awsexchage · 6 years ago

Photo

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた https://ift.tt/2P2V4wW

cloudpack あら便利カレンダー 2019の記事となります。誕生秘話はこちら。

AWS CloudFormation(CFn)でEC2インスタンスを作成するのに、AWS::CloudFormation::Initタイプを利用すると、インスタンス起動後にパッケージのインストールやファイル作成、コマンド実行を含めることができて便利そうだったのでお試ししてみました。

個人的に一番のメリットはスタック更新時にメタデータの変更も検知して再実行させることができる点です。

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

利用するのにいくつかのドキュメントを渡り歩く必要があったので、必要最低限のテンプレートと利用方法、ポイントをまとめました。

前提

AWSアカウントがある

AWS CLIがインストール済みで利用可能

CFn、EC2関連の権限がある

テンプレート

EC2インスタンスを作成・管理するテンプレートとなります。

EC2インスタンスを起動するVPCやサブネットは既存のリソースを利用前提となります。 SSHアクセスする際のキーペアは事前に作成し、セキュリティグループはインスタンスとあわせて作成しています。作成するリージョンは検証なので、us-east-1のみとしています。

cfn-template.yaml

ポイント

AWS::CloudFormation::Initタイプを利用する

AWS::CloudFormation::Initタイプを利用すると、EC2インスタンスでcfn-initヘルパースクリプト用のメタデータとして取り込まれて実行されます。 cfn-initヘルパースクリプトはメタデータに応じて以下のような操作を行います。

CFnのメタデータの取得と解析

パッケージのインストール

ディスクへのファイルの書き込み

Linux/UNIXグループ・ユーザー作成

ソースのダウンロード・展開

コマンド実行

サービスの有効化/無効化と開始/停止

詳しくは下記が参考になります。

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

cfn-init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-init.html

今回のテンプレートではメタデータでファイル書き込み、コマンド実行とサービス有効化する定義をしています。詳細は後ほど。

cfn-template.yaml_一部抜粋

cfn-init ヘルパースクリプトをUserDataで実行する

AWS::CloudFormation::Initタイプを利用してメタデータを定義しただけだと、スタック作成時にcfn-initヘルパースクリプトは実行されないため、ユーザーデータを定義して実行します。ユーザーデータについては下記が参考になります。

Linux インスタンスでの起動時のコマンドの実行 – Amazon Elastic Compute Cloud https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/user-data.html

今回のテンプレートでは/opt/aws/bin/cfn-initでエラーとなった場合、リソース作成失敗となるように/opt/aws/bin/cfn-signalも実行するようにしています。/opt/aws/bin/cfn-signalを利用する場合には、CreationPolicyを定義しておく必要があります。 cfn-signalヘルパースクリプトについては下記が参考になります。

cfn-signal – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-signal.html

cfn-template.yaml_一部抜粋

MyInstance: (略) Properties: (略) UserData: Fn::Base64: !Sub | #!/bin/bash echo "start UserData" /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} echo "finish UserData" CreationPolicy: ResourceSignal: Timeout: PT5M

cfn-hup ヘルパーでメタデータ更新時に対応する

メタデータとユーザーデータの定義だけだと、メタデータを変更してスタック更新しても変更が検知されず、EC2インスタンスに反映されないため、cfn-hup ヘルパーを利用する必要があります。cfn-hupヘルパーはEC2インスタンスでデーモンとして実行が可能です。 cfn-hupデーモンを実行するためにはcfn-hup.confとhooks.confファイルを用意する必要があるため、メタデータで定義しています。 cfn-hupヘルパーについては下記が参考になります。

cfn-hup – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-hup.html

cfn-template.yaml_一部抜粋

MyInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: files: /etc/cfn/cfn-hup.conf: content: !Sub | [main] stack = ${AWS::StackName} region = ${AWS::Region} interval = 1 mode: "000400" owner: "root" group: "root" /etc/cfn/hooks.d/cfn-auto-reloader.conf: content: !Sub | [cfn-auto-reloader-hook] triggers = post.update path = Resources.MyInstance.Metadata.AWS::CloudFormation::Init action = /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} runas = root mode: "000400" owner: "root" group: "root" commands: (略) services: sysvinit: cfn-hup: enabled: "true" files: - /etc/cfn/cfn-hup.conf - /etc/cfn/hooks.d/cfn-auto-reloader.conf

利用方法

パラメータ値を取得

キーペア

EC2インスタンスへSSHログインするのに利用するキーペアを作成します。既存のキーペアを利用する場合は作成不要です。

# Create KeyPair > aws ec2 create-key-pair \ --key-name cfn-init-test-ec2-key \ --query "KeyMaterial" \ --output text > cfn-init-test-ec2-key.pem > chmod 400 cfn-init-test-ec2-key.pem

VPC、サブネット

既存のVPC、サブネット配下にEC2インスタンスを作成する前提ですので各IDを取得します。

自身のグローバルIP

SSHアクセスするためのセキュリティグループに指定するIPアドレスを取得します。ここではifconfig.ioを利用していますが他の手段でもOKです。

## GlobalIP > curl ifconfig.io xxx.xxx.xxx.xxx

スタック作成

aws cloudformation create-stackコマンドを利用してスタック作成します。 AWSマネジメントコンソールから作成してもOKです。

> aws cloudformation create-stack \ --stack-name cfn-init-test \ --template-body file://cfn-init-template.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 \ --parameters '[ { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" } ]'

スタック作成結果を確認

スタック作成ができたか確認します。

> aws cloudformation describe-stacks \ --stack-name cfn-init-test { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-init-test/a1dc6aa0-a9e6-11e9-b171-0a515b01a4a4", "StackName": "cfn-init-test", "Parameters": [ { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" }, { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "InstanceType", "ParameterValue": "t3.small" } ], "CreationTime": "2019-07-19T05:32:43.477Z", "RollbackConfiguration": {}, "StackStatus": "CREATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Outputs": [ { "OutputKey": "ClientPublicIp", "OutputValue": "xxx.xxx.xxx.xxx" }, { "OutputKey": "ClientInstanceId", "OutputValue": "i-xxxxxxxxxxxxxxxxx" } ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] }

EC2インスタンスで確認

テンプレートのOutputsでEC2インスタンスのIPアドレス(‘ClientPublicIp’ )が出力されるようにしているので、SSHログインして各ログを確認します。 CFnでEC2インスタンスを作成するといくつかのログファイルが作成されますが、下記のファイル確認しておくとだいたいは把握できます。

スタック更新に関わるログとファイルは以下となります。 cfn-hupデーモンが定期的にCFnスタックのメタデータを取得しmetadata_db.jsonに保存、メタデータとファイルに差分があるとcfn-auto-reloader.confで指定したアクション(コマンド)が実行される仕組みになっています。なので、スタック作成後、cfn-hupデーモンが1度もメタデータ取得していないタイミングでスタック更新しても、metadata_db.jsonが存在していないと変更が検知されないみたいです。(1敗

$ cat /var/log/cfn-hup.log 2019-07-19 06:53:00,750 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-07-19 06:53:00,750 [DEBUG] Creating /var/lib/cfn-hup/data 2019-07-19 06:53:00,756 [INFO] No umask value specified in config file. Using the default one: 022 $ sudo cat /var/lib/cfn-hup/data/metadata_db.json {"cfn-auto-reloader-hook|Resources.MyInstance.Metadata.AWS::CloudFormation::Init": {"config": {"files": {"/etc/cfn/cfn-hup.conf": {"owner": "root", "content": "[main]\nstack = cfn-init-test\nregion = us-east-1\ninterval = 1\n", "group": "root", "mode": "000400"}, "/etc/cfn/hooks.d/cfn-auto-reloader.conf": {"owner": "root", "content": "[cfn-auto-reloader-hook]\ntriggers = post.update\npath = Resources.MyInstance.Metadata.AWS::CloudFormation::Init\naction = /opt/aws/bin/cfn-init -v --stack cfn-init-test --resource MyInstance --region us-east-1\nrunas = root\n", "group": "root", "mode": "000400"}}, "services": {"sysvinit": {"cfn-hup": {"files": ["/etc/cfn/cfn-hup.conf", "/etc/cfn/hooks.d/cfn-auto-reloader.conf"], "enabled": "true"}}}, "commands": {"test": {"command": "echo $STACK_NAME test", "env": {"STACK_NAME": "cfn-init-test"}}}}}}

メタデータを編集してスタック更新する

メタデータに実行するコマンドを追加してスタック更新してみます。

cfn-template.yaml_一部抜粋

(略) commands: test: command: "echo $STACK_NAME test" env: STACK_NAME: !Ref AWS::StackName test2: command: "echo $STACK_NAME test2" env: STACK_NAME: !Ref AWS::StackName (略)

aws cloudformation update-stackコマンドを実行してスタック更新します。

> aws cloudformation update-stack \ --stack-name cfn-init-test \ --template-body file://cfn-init-template.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 \ --parameters '[ { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" } ]'

スタック更新結果を確認

> aws cloudformation describe-stacks \ --stack-name cfn-init-test { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-init-test/a1dc6aa0-a9e6-11e9-b171-0a515b01a4a4", "StackName": "cfn-init-test", "Parameters": [ { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" }, { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "InstanceType", "ParameterValue": "t3.small" } ], "CreationTime": "2019-07-19T05:32:43.477Z", "LastUpdatedTime": "2019-07-19T05:41:40.446Z", "RollbackConfiguration": {}, "StackStatus": "UPDATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Outputs": [ { "OutputKey": "ClientPublicIp", "OutputValue": "xxx.xxx.xxx.xxx" }, { "OutputKey": "ClientInstanceId", "OutputValue": "i-xxxxxxxxxxxxxxxxx" } ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] }

EC2インスタンスで確認

CFnのリソース更新されてもEC2インスタンス側への変更反映は、cfn-hupヘルパーが担当しますので、反映にはタイムラグが発生します。cfn-hupヘルパーがメタデータの変更確認する間隔は初期設定で15分となっており、/etc/cfn/cfn-hup.conf ファイルのintervalで指定ができます。(今回は1分で指定)

cfn-hup – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-hup.html

> ssh -i cfn-init-test-ec2-key.pem [email protected] $ cat /var/log/cfn-init.log 2019-07-19 06:53:00,454 [INFO] -----------------------Starting build----------------------- 2019-07-19 06:53:00,455 [INFO] Running configSets: default 2019-07-19 06:53:00,455 [INFO] Running configSet default 2019-07-19 06:53:00,456 [INFO] Running config config 2019-07-19 06:53:00,460 [INFO] Command test succeeded 2019-07-19 06:53:00,467 [INFO] enabled service cfn-hup 2019-07-19 06:53:00,768 [INFO] Restarted cfn-hup successfully 2019-07-19 06:53:00,769 [INFO] ConfigSets completed 2019-07-19 06:53:00,769 [INFO] -----------------------Build complete----------------------- 2019-07-19 06:53:01,055 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-07-19 06:53:01,055 [DEBUG] Signaling resource MyInstance in stack cfn-init-test with unique ID i-xxxxxxxxxxxxxxxxx and status SUCCESS 2019-07-19 06:56:01,457 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-07-19 06:56:01,458 [DEBUG] Describing resource MyInstance in stack cfn-init-test 2019-07-19 06:56:01,522 [INFO] -----------------------Starting build----------------------- 2019-07-19 06:56:01,523 [DEBUG] Not setting a reboot trigger as scheduling support is not available 2019-07-19 06:56:01,524 [INFO] Running configSets: default 2019-07-19 06:56:01,525 [INFO] Running configSet default 2019-07-19 06:56:01,525 [INFO] Running config config 2019-07-19 06:56:01,526 [DEBUG] No packages specified 2019-07-19 06:56:01,526 [DEBUG] No groups specified 2019-07-19 06:56:01,526 [DEBUG] No users specified 2019-07-19 06:56:01,526 [DEBUG] No sources specified 2019-07-19 06:56:01,526 [DEBUG] /etc/cfn/cfn-hup.conf already exists 2019-07-19 06:56:01,526 [DEBUG] Moving /etc/cfn/cfn-hup.conf to /etc/cfn/cfn-hup.conf.bak 2019-07-19 06:56:01,526 [DEBUG] Writing content to /etc/cfn/cfn-hup.conf 2019-07-19 06:56:01,526 [DEBUG] Setting mode for /etc/cfn/cfn-hup.conf to 000400 2019-07-19 06:56:01,526 [DEBUG] Setting owner 0 and group 0 for /etc/cfn/cfn-hup.conf 2019-07-19 06:56:01,527 [DEBUG] /etc/cfn/hooks.d/cfn-auto-reloader.conf already exists 2019-07-19 06:56:01,527 [DEBUG] Moving /etc/cfn/hooks.d/cfn-auto-reloader.conf to /etc/cfn/hooks.d/cfn-auto-reloader.conf.bak 2019-07-19 06:56:01,527 [DEBUG] Writing content to /etc/cfn/hooks.d/cfn-auto-reloader.conf 2019-07-19 06:56:01,527 [DEBUG] Setting mode for /etc/cfn/hooks.d/cfn-auto-reloader.conf to 000400 2019-07-19 06:56:01,527 [DEBUG] Setting owner 0 and group 0 for /etc/cfn/hooks.d/cfn-auto-reloader.conf 2019-07-19 06:56:01,527 [DEBUG] Running command test 2019-07-19 06:56:01,527 [DEBUG] No test for command test 2019-07-19 06:56:01,531 [INFO] Command test succeeded 2019-07-19 06:56:01,531 [DEBUG] Command test output: cfn-init-test test 2019-07-19 06:56:01,531 [DEBUG] Running command test2 2019-07-19 06:56:01,531 [DEBUG] No test for command test2 2019-07-19 06:56:01,535 [INFO] Command test2 succeeded 2019-07-19 06:56:01,536 [DEBUG] Command test2 output: cfn-init-test test! 2019-07-19 06:56:01,536 [DEBUG] Using service modifier: /sbin/chkconfig 2019-07-19 06:56:01,536 [DEBUG] Setting service cfn-hup to enabled 2019-07-19 06:56:01,538 [INFO] enabled service cfn-hup 2019-07-19 06:56:01,539 [DEBUG] Not modifying running state of service cfn-hup 2019-07-19 06:56:01,539 [INFO] ConfigSets completed 2019-07-19 06:56:01,539 [DEBUG] Not clearing reboot trigger as scheduling support is not available 2019-07-19 06:56:01,539 [INFO] -----------------------Build complete----------------------- $ cat /var/log/cfn-init-cmd.log 2019-07-19 06:53:00,455 P2208 [INFO] ************************************************************ 2019-07-19 06:53:00,455 P2208 [INFO] ConfigSet default 2019-07-19 06:53:00,456 P2208 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2019-07-19 06:53:00,456 P2208 [INFO] Config config 2019-07-19 06:53:00,457 P2208 [INFO] ============================================================ 2019-07-19 06:53:00,457 P2208 [INFO] Command test 2019-07-19 06:53:00,460 P2208 [INFO] -----------------------Command Output----------------------- 2019-07-19 06:53:00,460 P2208 [INFO] cfn-init-test test 2019-07-19 06:53:00,460 P2208 [INFO] ------------------------------------------------------------ 2019-07-19 06:53:00,460 P2208 [INFO] Completed successfully. 2019-07-19 06:56:01,525 P2337 [INFO] ************************************************************ 2019-07-19 06:56:01,525 P2337 [INFO] ConfigSet default 2019-07-19 06:56:01,525 P2337 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2019-07-19 06:56:01,526 P2337 [INFO] Config config 2019-07-19 06:56:01,527 P2337 [INFO] ============================================================ 2019-07-19 06:56:01,527 P2337 [INFO] Command test 2019-07-19 06:56:01,530 P2337 [INFO] -----------------------Command Output----------------------- 2019-07-19 06:56:01,531 P2337 [INFO] cfn-init-test test 2019-07-19 06:56:01,531 P2337 [INFO] ------------------------------------------------------------ 2019-07-19 06:56:01,531 P2337 [INFO] Completed successfully. 2019-07-19 06:56:01,531 P2337 [INFO] ============================================================ 2019-07-19 06:56:01,531 P2337 [INFO] Command test2 2019-07-19 06:56:01,535 P2337 [INFO] -----------------------Command Output----------------------- 2019-07-19 06:56:01,535 P2337 [INFO] cfn-init-test test! 2019-07-19 06:56:01,535 P2337 [INFO] ------------------------------------------------------------ 2019-07-19 06:56:01,535 P2337 [INFO] Completed successfully. $ cat /var/log/cloud-init-output.log (略) Updated: java-1.7.0-openjdk.x86_64 1:1.7.0.211-2.6.17.1.79.amzn1 kernel-tools.x86_64 0:4.14.128-87.105.amzn1 perl.x86_64 4:5.16.3-294.43.amzn1 perl-Pod-Escapes.noarch 1:1.04-294.43.amzn1 perl-libs.x86_64 4:5.16.3-294.43.amzn1 perl-macros.x86_64 4:5.16.3-294.43.amzn1 python27-jinja2.noarch 0:2.7.2-3.16.amzn1 wget.x86_64 0:1.18-5.30.amzn1 Complete! Cloud-init v. 0.7.6 running 'modules:final' at Fri, 19 Jul 2019 06:52:59 +0000. Up 18.81 seconds. start UserData finish UserData Cloud-init v. 0.7.6 finished at Fri, 19 Jul 2019 06:53:01 +0000. Datasource DataSourceEc2. Up 20.07 seconds

メタデータに追加したコマンドが実行されるのを確認できました。やったぜ。

スタック更新時にメタデータで定義しているすべての処理を実行したくない場合、Configsetを用いるとグルーピングや実行順の制御をして特定の処理ができます。詳細は下記をご参考ください。

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

まとめ

CFnでEC2インスタンスを作成する際に、インスタンス起動後の処理もテンプレートで定義して管理できる仕組みになっていますので、非常に使い勝手が良さそうです。

参考

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

cfn-init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-init.html

Linux インスタンスでの起動時のコマンドの実行 – Amazon Elastic Compute Cloud https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/user-data.html

cfn-signal – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-signal.html

cfn-hup – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-hup.html

元記事はこちら

「AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた」

August 13, 2019 at 02:00PM

#AWS #Official #Blog

0 notes

awsexchage · 6 years ago

Photo

AWS SDK for Python(boto3)でAmazon Managed Blockchainのブロックチェーンネットワークを作成してみた http://bit.ly/2QO67Y5

Amazon Managed Blockchainでブロックチェーンネットワークを構築するのにAWS CloudFormationを利用したいなぁとドキュメントを読んでみたらリソースがありませんでした。oh… AWS CloudFormationのカスタムリソースを利用すれば管理できるので、まずはAWS SDKでAmazon Managed Blockchainが取り扱えるのか確認してみました。

AWS CloudFormationのカスタムリソースについては下記が参考になります。

カスタムリソース – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/template-custom-resources.html

CloudFormationで提供されていない処理をカスタムリソースで作ってみた。｜ DevelopersIO https://dev.classmethod.jp/cloud/aws/cfn-api-custom/

AWS CloudFormationでCognitoユーザープールをMFAのTOTPを有効にして作成する – Qiita https://cloudpack.media/44573

Amazon Managed Blockchainってなんぞ？

Amazon Managed Blockchainってなんぞ？という方は下記をご参考ください。

Amazon Managed BlockchainがリリースされたのでHyperledger Fabricも合わせて情報をまとめてみた – Qiita https://cloudpack.media/46950

前提

今回はAmazon Managed Blockchainのネットワーク、メンバー、ノードがAWS SDKで作成できるかの確認のみとなります。セキュリティグループ、インタフェースVPCエンドポイントやHyperledger Fabricの設定などは行いません。

AWSアカウントがある

AWSアカウントに以下の作成権限がある

Managed Blockchainネットワーク

AWS CLIのaws configure コマンドでアカウント設定済み

ネットワーク構築して動作確認するまでの手順は下記が参考になります。

Amazon Managed BlockchainでHyperledger Fabricのブロックチェーンネットワークを構築してみた – Qiita https://cloudpack.media/46963

AWS SDK for Python(boto3)でAmazon Managed BlockchainのAPIが提供されてる

ドキュメントを確認してみたらAWS SDKでAmazon Managed BlockchainのAPIが提供されていました。

boto/boto3: AWS SDK for Python https://github.com/boto/boto3

ManagedBlockchain — Boto 3 Docs 1.9.152 documentation https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/managedblockchain.html

Python環境の用意

Pipenvを利用したことがなかったので、ついでに初利用します(趣味)。仮想環境じゃなくてもOKです。

Pipenvを使ったPython開発まとめ – Qiita https://qiita.com/y-tsutsu/items/54c10e0b2c6b565c887a

# pipenvがインストールされてなかったら > pip install pipenv (略) Installing collected packages: virtualenv, virtualenv-clone, pipenv Successfully installed pipenv-2018.11.26 virtualenv-16.6.0 virtualenv-clone-0.5.3 # 仮想環境の作成 > pipenv --python 3.7 Creating a virtualenv for this project… Pipfile: /Users/xxx/dev/aws/managed-blockchain-use-sdk/Pipfile Using /Users/xxx/.anyenv/envs/pyenv/versions/3.7.0/bin/python3 (3.7.0) to create virtualenv… ⠴ Creating virtual environment... (略) ✓ Successfully created virtual environment! Virtualenv location: /Users/xxx/.local/share/virtualenvs/managed-blockchain-use-sdk-ijMKrYxz Creating a Pipfile for this project… # boto3のインストール > pipenv install boto3 Installing boto3… ⠴ Installing... (略) # 仮想環境へ入る > pipenv shell Launching subshell in virtual environment… Welcome to fish, the friendly interactive shell

AWS SDK for Python(boto3)を利用して実装

ドキュメントを参考にネットワークなどを作成、取得できるか確認します。

ManagedBlockchain — Boto 3 Docs 1.9.152 documentation https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/managedblockchain.html

ネットワークとメンバーの作成

create_network でネットワーク作成できます。パラメータはAWS CLIとほぼ同じなので下記が参考になります。 AWS CLIと同じく最初のメンバーも合わせて作成する必要があります。

Amazon Managed BlockchainでHyperledger Fabricのブロックチェーンネットワークを構築してみた – Qiita https://cloudpack.media/46963

create_network.py

import boto3 client = boto3.client("managedblockchain") def create_network(): new_network = client.create_network( ClientRequestToken="string", Name="TestNetwork", Description="TestNetworkDescription", Framework="HYPERLEDGER_FABRIC", FrameworkVersion="1.2", FrameworkConfiguration={ "Fabric": { "Edition": "STARTER" } }, VotingPolicy={ "ApprovalThresholdPolicy": { "ThresholdPercentage": 50, "ProposalDurationInHours": 24, "ThresholdComparator": "GREATER_THAN" } }, MemberConfiguration={ "Name": "org1", "Description": "Org1 first member of network", "FrameworkConfiguration": { "Fabric": { "AdminUsername": "AdminUser", "AdminPassword": "Password123" } } } ) print(new_network)

実行するとネットワークとメンバーの作成が開始されてNetworkId とMemberId が得られます。

> python create_network.py {'ResponseMetadata': {'RequestId': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 21 May 2019 03:24:32 GMT', 'content-type': 'application/json', 'content-length': '86', 'connection': 'keep-alive', 'x-amzn-requestid': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'x-amz-apigw-id': 'xxxxxxxxxxxxxxx=', 'x-amzn-trace-id': 'Root=1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx; Sampled=0'}, 'RetryAttempts': 0}, 'NetworkId': 'n-XXXXXXXXXXXXXXXXXXXXXXXXXX', 'MemberId': 'm-XXXXXXXXXXXXXXXXXXXXXXXXXX'}

ネットワーク情報の取得

list_networks でネットワーク情報が取得できます。パラメータ指定することで検索もできるみたいです。

list_network.py

import boto3 client = boto3.client("managedblockchain") def list_network(): networks = client.list_networks( # Name="Test" # Name="string", # Framework="HYPERLEDGER_FABRIC", # Status="CREATING"|"AVAILABLE"|"CREATE_FAILED"|"DELETING"|"DELETED", # MaxResults=123, # NextToken="string" ) print(networks) list_networks()

> python list_network.py [{'Id': 'n-XXXXXXXXXXXXXXXXXXXXXXXXXX', 'Name': 'TestNetwork', 'Description': 'TestNetworkDescription', 'Framework': 'HYPERLEDGER_FABRIC', 'FrameworkVersion': '1.2', 'Status': 'CREATING', 'CreationDate': datetime.datetime(2019, 5, 21, 3, 24, 31, 929000, tzinfo=tzutc())}]

ノードの作成

create_node でノード作成ができます。ClientRequestToken に関しては指定しない場合自動生成してくれるとのことでした。手動での生成方法は未調査です。

create_node.py

import boto3 client = boto3.client("managedblockchain") def create_node(): new_node = client.create_node( # ClientRequestToken='string', NetworkId='n-5QBOS5ULTVEZZG6EMIPLPRSSQA', MemberId='m-OMLBOQIAMJDDPJV3FVGIAVUGBE', NodeConfiguration={ 'InstanceType': 'bc.t3.small', 'AvailabilityZone': 'us-east-1a' } ) print(new_node) create_node()

> python create_node.py {'ResponseMetadata': {'RequestId': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 21 May 2019 04:04:44 GMT', 'content-type': 'application/json', 'content-length': '42', 'connection': 'keep-alive', 'x-amzn-requestid': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'x-amz-apigw-id': 'xxxxxxxxxxxxxxx=', 'x-amzn-trace-id': 'Root=1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx;Sampled=0'}, 'RetryAttempts': 0}, 'NodeId': 'nd-XXXXXXXXXXXXXXXXXXXXXXXXXX'}

ネットワーク、メンバーが作成中(Status がCREATING )の場合は作成できませんでした。

ネットワーク、メンバー作成中に実行

> python create_node.py Traceback (most recent call last): File "main.py", line 43, in <module> 'AvailabilityZone': 'us-east-1a' File "/Users/xxx/.local/share/virtualenvs/managed-blockchain-use-sdk-ijMKrYxz/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/xxx/.local/share/virtualenvs/managed-blockchain-use-sdk-ijMKrYxz/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.ResourceNotReadyException: An error occurred (ResourceNotReadyException) when calling the CreateNode operation: Member m-XXXXXXXXXXXXXXXXXXXXXXXXXX is in CREATING. you cannot create nodes at this time..

メンバー情報の取得

list_members でメンバーが取得できます。NetworkId が必須となります。

list_members.py

import boto3 client = boto3.client("managedblockchain") def list_members(): members = client.list_members( NetworkId="n-XXXXXXXXXXXXXXXXXXXXXXXXXX" # Name="string", # Status="CREATING"|"AVAILABLE"|"CREATE_FAILED"|"DELETING"|"DELETED", # IsOwned=True|False, # MaxResults=123, # NextToken="string" ) print(members["Members"]) list_members()

> pyhton list_members.py [{'Id': 'm-XXXXXXXXXXXXXXXXXXXXXXXXXX', 'Name': 'demo', 'Description': 'demo member', 'Status': 'AVAILABLE', 'CreationDate': datetime.datetime(2019, 5, 21, 3, 24, 32, 271000, tzinfo=tzutc()), 'IsOwned': True}]

ノード情報の取得

list_nodes でノードが取得できます。NetworkId 、MemberId が必須となります。

list_nodes.py

import boto3 client = boto3.client("managedblockchain") def list_nodes(): nodes = client.list_nodes( NetworkId="n-XXXXXXXXXXXXXXXXXXXXXXXXXX", MemberId="m-XXXXXXXXXXXXXXXXXXXXXXXXXX" # Status="CREATING"|"AVAILABLE"|"CREATE_FAILED"|"DELETING"|"DELETED"|"FAILED", # MaxResults=123, # NextToken="string" ) print(nodes["Nodes"]) list_nodes()

> pyhton list_nodes.py [{'Id': 'nd-XXXXXXXXXXXXXXXXXXXXXXXXXX', 'Status': 'AVAILABLE', 'CreationDate': datetime.datetime(2019, 5, 21, 4, 4, 44, 468000, tzinfo=tzutc()), 'AvailabilityZone': 'us-east-1a', 'InstanceType': 'bc.t3.small'}]

特にハマることもなくboto3を利用してネットワーク、メンバー、ノードを作成するこ��ができました。これで、AWS CloudFormationのカスタムリソースを利用してリソース管理することができそうです。ただ、ネットワークとメンバー作成に20分程度かかるので、その��受処理などがAWS CloudFormationで実現できるのか、ちょっと心配です。(未調査)