#tcp/ipprotocol | Explore Tumblr posts and blogs

computingpostcom · 3 years ago

Text

This tutorial aims to take the reader through creating an Application Load balancer and its dependencies using CloudFormation. The template will create: The Application Load Balancer The Target Groups The Listeners The Listener Rules The AWS cloud platform provides managed load balancers using the Elastic Load Balancer service. One has options to create an Application (layer7), Network (layer 4), or Classic Load Balancer (both layer 4 and 7). For this tutorial, we will create an Application Load balancer. A load balancer is useful because: It gives our applications high availability. Using a load balancer in our applications enables one to route traffic to multiple backend servers. In case one server fails, traffic will be routed to the other servers. This is also good for ensuring even distribution of traffic between the various servers. It is also useful for directing/filtering traffic based on the rules configured. Using host-based, path-based routing, or a combination of both, one can direct traffic to different servers. Setup Prerequisites The user will need to have: An AWS Account. A user with permissions to create resources on AWS. Generated a certificate for their specific domain name (We used AWS Certificate Manager to generate ours). An IDE to write and edit your CloudFormation Template. N/B: A user can import certificates generated from other certificate generation entities to the AWS certificate manager and use them. Step 1: Create CloudFormation Template Use the below code for your CloudFormation template. The user can edit the various parts of the template as explained in the next section. AWSTemplateFormatVersion: "2010-09-09" Description: "Create ALB, ALB security group, target groups, listeners and listener rules" Parameters: VPC: Type: String Description: The vpc to launch the service Default: vpc-ID PublicSubnet1: Type: String Description: The subnet where to launch the service Default: subnet-ID PublicSubnet2: Type: String Description: the subnet where to Launch the service Default: subnet-ID Resources: ALBSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: "security group for ALB" GroupName: "test-ALB-SG" Tags: - Key: "Project" Value: "test-blog" - Key: "createdBy" Value: "Maureen Barasa" - Key: "Environment" Value: "test" - Key: "Name" Value: "test-ALB-SG" VpcId: !Ref VPC SecurityGroupIngress: - CidrIp: "0.0.0.0/0" FromPort: 80 IpProtocol: "tcp" ToPort: 80 - CidrIp: "0.0.0.0/0" FromPort: 443 IpProtocol: "tcp" ToPort: 443 ApplicationLoadBalancer: Type: "AWS::ElasticLoadBalancingV2::LoadBalancer" Properties: Name: "test-Application-Load-Balancer" Scheme: "internet-facing" Type: "application" Subnets: - !Ref PublicSubnet1 - !Ref PublicSubnet2 SecurityGroups: - !Ref ALBSecurityGroup IpAddressType: "ipv4" LoadBalancerAttributes: - Key: "access_logs.s3.enabled" Value: "false" - Key: "idle_timeout.timeout_seconds" Value: "60" - Key: "deletion_protection.enabled" Value: "false" - Key: "routing.http2.enabled" Value: "true"

- Key: "routing.http.drop_invalid_header_fields.enabled" Value: "false" HTTPSListener: Type: "AWS::ElasticLoadBalancingV2::Listener" Properties: LoadBalancerArn: !Ref ApplicationLoadBalancer Port: 443 Protocol: "HTTPS" SslPolicy: "ELBSecurityPolicy-2016-08" Certificates: - CertificateArn: arn:aws:acm:eu-central-1:**************:certificate/********************* DefaultActions: - Order: 1 TargetGroupArn: !Ref Test1TargetGroup Type: "forward" HTTPListener: Type: "AWS::ElasticLoadBalancingV2::Listener" Properties: LoadBalancerArn: !Ref ApplicationLoadBalancer Port: 80 Protocol: "HTTP" DefaultActions: - Order: 1 RedirectConfig: Protocol: "HTTPS" Port: "443" Host: "#host" Path: "/#path" Query: "#query" StatusCode: "HTTP_301" Type: "redirect" Test1TargetGroup: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: HealthCheckIntervalSeconds: 30 HealthCheckPath: "/" Port: 80 Protocol: "HTTP" HealthCheckPort: "traffic-port" HealthCheckProtocol: "HTTP" HealthCheckTimeoutSeconds: 5 UnhealthyThresholdCount: 2 TargetType: "instance" Matcher: HttpCode: "200" HealthyThresholdCount: 5 VpcId: !Ref VPC Name: "target-group-1" HealthCheckEnabled: true TargetGroupAttributes: - Key: "stickiness.enabled" Value: "false" - Key: "deregistration_delay.timeout_seconds" Value: "300" - Key: "stickiness.type" Value: "lb_cookie" - Key: "stickiness.lb_cookie.duration_seconds" Value: "86400" - Key: "slow_start.duration_seconds" Value: "0" - Key: "load_balancing.algorithm.type" Value: "round_robin" Test2TargetGroup: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: HealthCheckIntervalSeconds: 30 HealthCheckPath: "/" Port: 80 Protocol: "HTTP" HealthCheckPort: "traffic-port" HealthCheckProtocol: "HTTP" HealthCheckTimeoutSeconds: 5 UnhealthyThresholdCount: 2 TargetType: "instance" Matcher: HttpCode: "200" HealthyThresholdCount: 5 VpcId: !Ref VPC Name: "target-group-2" HealthCheckEnabled: true TargetGroupAttributes: - Key: "stickiness.enabled" Value: "false" - Key: "deregistration_delay.timeout_seconds" Value: "300" - Key: "stickiness.type" Value: "lb_cookie" - Key: "stickiness.lb_cookie.duration_seconds" Value: "86400" - Key: "slow_start.duration_seconds" Value: "0" - Key: "load_balancing.algorithm.type" Value: "round_robin" TestListenerRule1: Type: "AWS::ElasticLoadBalancingV2::ListenerRule" Properties:

Priority: "1" ListenerArn: !Ref HTTPSListener Conditions: - Field: "host-header" Values: - "test1.blog.avrcr.com" Actions: - Type: "forward" TargetGroupArn: !Ref Test1TargetGroup Order: 1 ForwardConfig: TargetGroups: - TargetGroupArn: !Ref Test1TargetGroup Weight: 1 TargetGroupStickinessConfig: Enabled: false TestListenerRule2: Type: "AWS::ElasticLoadBalancingV2::ListenerRule" Properties: Priority: "2" ListenerArn: !Ref HTTPSListener Conditions: - Field: "host-header" Values: - "test2.blog.com" Actions: - Type: "forward" TargetGroupArn: !Ref Test2TargetGroup Order: 1 ForwardConfig: TargetGroups: - TargetGroupArn: !Ref Test2TargetGroup Weight: 1 TargetGroupStickinessConfig: Enabled: false Outputs: ALB: Description: The created loadbalancer Value: !Ref ApplicationLoadBalancer TargetGroup1: Description: The created TargetGroup 1 Value: !Ref Test1TargetGroup TargetGroup2: Description: The created TargetGroup 2 Value: !Ref Test2TargetGroup LoadBalancerSecurityGroup: Description: the securty group for the ALB Value: !Ref ALBSecurityGroup The CloudFormation Template Explained The template contains 3 sections. Parameters, Resources and Outputs. Parameters Section In the parameters section, the user inputs their dynamic variables. On our template, the user should customize the template by inputting their VPC and Subnets IDs. In our case, the load balancer is internet-facing hence the need to have it created on public subnets. Should the user want to create an internal load balancer, it would be prudent to have it created on private subnets. Resources Section The resources section allows the user to define the AWS resources they will create. On our template, we start by creating the load balancer security group. The security group creates allows inbound traffic from port 80 and 443. The user can also customize or add more rules to the security group. Next, the template creates a load balancer. The user can customize the name of the load balancer, the scheme, or whether it will be internal or internet-facing. If internal, kindly ensure you change the subnets from public to private. They can also customize the load balancer attributes as per their specific needs. Since we have opened port 80 and 443 on the load balancer security group, the template creates the two listeners for the load balancer. One listener on port 80 (HTTP) and the other on port 443 (HTTPS). Then for the HTTP listener, the template is configured to create a default action to forward all requests to the HTTPS listener by default. This can also be customizable as per the user’s needs. For the HTTPS listener, we have included the certificate we generated for our domain name. The user should change the certificate ARN to their own certificate ARN ID. Also, we created a default action to forward traffic to a target group. This will forward all traffic otherwise not routed to the target group. Again, the user can customize the HTTPS Listener default actions to their specific needs. Next, the template creates two target groups. The user can customize the target groups’ properties as per their specific needs. Important to note is that the target type can either be an instance (EC2) or an IP.

Finally, the template creates listener rules. For our case based on host-based routing, we created listener rules that forward to the specific target groups. The user has the option to customize the rules. Also, they can change from host-based to path-based routing. Outputs Section The outputs section outputs the names of the resources you created. Step 2: Create CodePipeline to Deploy the Template Create the CodePipeline role to deploy template to CloudFormation. Use the below cloudformation template to create the role. AWSTemplateFormatVersion: "2010-09-09" Description: "Template to create centos ec2 instance and install ssm on it" Resources: IAMInstanceRole: Type: 'AWS::IAM::Role' Properties: Description: The SSM Instance Profile RoleName: codepipeline-test AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - cloudformation.amazonaws.com Action: - 'sts:AssumeRole' ManagedPolicyArns: - arn:aws:iam::aws:policy/AWSCloudFormationFullAccess - arn:aws:iam::aws:policy/CloudWatchFullAccess - arn:aws:iam::aws:policy/AmazonEC2FullAccess Tags: - Key: "Project" Value: "test-blog" - Key: "Environment" Value: "test" - Key: "createdBy" Value: "Maureen Barasa" - Key: "Name" Value: "codepipeline-test" IAMInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: InstanceProfileName: codepipeline-test Roles: - !Ref IAMInstanceRole Outputs: Profile: Description: The created Instance Profile Value: !Ref IAMInstanceProfile Role: Description: The created role Value: !Ref IAMInstanceRole Next, go to the CodeCommit console. Then create a code commit repository. Commit your alb template to the repository. On the CodePipeline console, select create pipeline. Create Pipeline After, choose pipeline settings. For service role, opt to create a new service role. Choose Pipeline Settings Also, under Advanced Settings choose the S3 bucket that you will use to store your artifacts. For the encryption key choose the default AWS key. Then click next. Advanced Settings On the add source stage screen, choose code commit as your source provider. Add Source Stage Enter the details of your CodeCommit repository name and the branch. Also, for change detection leave the setting to Amazon CloudWatch Events. This enables CloudWatch to detect changes made on your code and auto-start the pipeline to update those changes. When done click next. Add Source Stage Settings On the add build stage screen, click skip build stage. Skip Build Stage Finally, on the add deploy stage screen, select CloudFormation as your deployment option. Select CloudFormation as Deployment Option Fill in the details for your CloudFormation deployment. N/B: for the role, use the role you created with the CloudFormation template at the beginning of the section. Add Deployment Stage Details The next stage allows the user to review all the configurations done. If all configurations are correct click on create pipeline. You have now created your first pipeline to deploy a CloudFormation template. A Successful Deployment Pipeline Important Links https://aws.amazon.com/elasticloadbalancing/ https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html

0 notes

awsexchage · 6 years ago

Photo

AWS CloudFormationでEC2インスタンスのログがAmazon CloudWatch logsで確認できるようにする https://ift.tt/33zfnVY

AWS CloudFormation(CFn)でEC2インスタンスを作成・管理する際にユーザーデータやAWS::CloudFormation::Initタイプを利用したメタデータで環境構築するのが便利なのですが、スタック作成・更新時のログはEC2インスタンス内にあるため、確認が面倒です。

EC2のログをAmazon CloudWatch Logsへ送信する仕組みとして「CloudWatch Logs エージェント」があると知ったのでCFnでEC2インスタンスを管理する際に利用できるテンプレートを作成してみました。

CloudWatch Logs エージェントのリファレンス – Amazon CloudWatch Logs https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/AgentReference.html

前提

AWSアカウントがある

AWS CLIがインストール済みで利用可能

CFn、EC2関連の権限がある

テンプレート

下記で作成したテンプレートにawslogsを利用する定義を追加しました。

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた – Qiita https://cloudpack.media/48540

EC2インスタンスを起動するVPCやサブネットは既存のリソースを利用する前提となります。 SSHアクセスする際のキーペアは事前に作成し、セキュリティグループはインスタンスとあわせて作成しています。作成するリージョンは検証なので、us-east-1のみとしています。

cfn-template.yaml

Parameters: VpcId: Type: AWS::EC2::VPC::Id SubnetId: Type: AWS::EC2::Subnet::Id EC2KeyPairName: Type: AWS::EC2::KeyPair::KeyName InstanceType: Type: String Default: t3.small MyInstanceSSHCidrIp: Type: String Default: '0.0.0.0/0' Mappings: AWSRegionToAMI: us-east-1: HVM64: ami-0080e4c5bc078760e Resources: MyInstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupName: !Join - " " - - !Ref AWS::StackName - "SecurityGroup" GroupDescription: "MyInstance SecurityGroup" VpcId: !Ref VpcId SecurityGroupEgress: - CidrIp: "0.0.0.0/0" IpProtocol: "-1" SecurityGroupIngress: - CidrIp: !Ref MyInstanceSSHCidrIp Description: !Join - " " - - !Ref AWS::StackName - "SSH Port" IpProtocol: "tcp" FromPort: 22 ToPort: 22 Tags: - Key: "Name" Value: !Join - " " - - !Ref AWS::StackName - "SecurityGroup" MyInstanceIAMRole: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "ec2.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" Policies: - PolicyName: "root" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - "logs:CreateLogGroup" - "logs:CreateLogStream" - "logs:PutLogEvents" - "logs:DescribeLogStreams" Resource: "arn:aws:logs:*:*:*" MyInstanceIAMInstanceProfile: Type: "AWS::IAM::InstanceProfile" Properties: Path: "/" Roles: - Ref: MyInstanceIAMRole DependsOn: MyInstanceIAMRole MyInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: packages: yum: awslogs: [] files: /etc/awslogs/awslogs.conf: content: !Sub | [general] state_file = /var/lib/awslogs/agent-state [/var/log/cloud-init-output.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cloud-init-output.log buffer_duration = 5000 log_stream_name = {instance_id}/cloud-init-output.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init-cmd.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init-cmd.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init-cmd.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-hup.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-hup.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-hup.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} mode: "000644" owner: "root" group: "root" /etc/cfn/cfn-hup.conf: content: !Sub | [main] stack = ${AWS::StackName} region = ${AWS::Region} mode: "000400" owner: "root" group: "root" /etc/cfn/hooks.d/cfn-auto-reloader.conf: content: !Sub | [cfn-auto-reloader-hook] triggers = post.update path = Resources.MyInstance.Metadata.AWS::CloudFormation::Init action = /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} runas = root mode: "000400" owner: "root" group: "root" commands: test: command: "echo $STACK_NAME test" env: STACK_NAME: !Ref AWS::StackName services: sysvinit: cfn-hup: enabled: "true" files: - /etc/cfn/cfn-hup.conf - /etc/cfn/hooks.d/cfn-auto-reloader.conf awslogs: enabled: "true" packages: - awslogs files: - /etc/awslogs/awslogs.conf Properties: InstanceType: !Ref InstanceType KeyName: !Ref EC2KeyPairName ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", HVM64 ] IamInstanceProfile: !Ref MyInstanceIAMInstanceProfile NetworkInterfaces: - AssociatePublicIpAddress: True DeviceIndex: 0 GroupSet: - !Ref MyInstanceSecurityGroup SubnetId: !Ref SubnetId Tags: - Key: 'Name' Value: !Ref AWS::StackName UserData: Fn::Base64: !Sub | #!/bin/bash echo "start UserData" /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} echo "finish UserData" CreationPolicy: ResourceSignal: Timeout: PT5M Outputs: ClientInstanceId: Value: !Ref MyInstance ClientPublicIp: Value: !GetAtt MyInstance.PublicIp

ポイント

IAMロールを作成する

EC2インスタンスからAmazon CloudWatch Logsへログを送るのに権限が��要となるためIAMロールを作成します。必要となる権限については下記を参考にしました。

クイックスタート: 実行中の EC2 Linux インスタンスに CloudWatch Logs エージェントをインストールして設定する – Amazon CloudWatch Logs https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

cfn-template.yaml_一部抜粋

MyInstanceIAMRole: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "ec2.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" Policies: - PolicyName: "root" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - "logs:CreateLogGroup" - "logs:CreateLogStream" - "logs:PutLogEvents" - "logs:DescribeLogStreams" Resource: "arn:aws:logs:*:*:*" MyInstanceIAMInstanceProfile: Type: "AWS::IAM::InstanceProfile" Properties: Path: "/" Roles: - Ref: MyInstanceIAMRole DependsOn: MyInstanceIAMRole MyInstance: Type: AWS::EC2::Instance (略) Properties: InstanceType: !Ref InstanceType KeyName: !Ref EC2KeyPairName ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", HVM64 ] IamInstanceProfile: !Ref MyInstanceIAMInstanceProfile (略)

awslogs パッケージを利用する

CloudWatch Logs エージェントはawslogsパッケージとして提供されています。awslogsパッケージはAmazon LinuxとUbuntuに対応しています。

パッケージのインストール、設定ファイル、サービス起動はメタデータで定義して設定変更できるようにしました。 /etc/awslogs/awslogs.confファイルでAmazon CloudWatch Logsに出力するログファイルを定義します。log_stream_nameやlog_group_name は任意で指定が可能です。ここでは/var/log/内に出力されるメタデータやユーザーデータによる処理のログを対象としました。

cfn-template.yaml_一部抜粋

MyInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: packages: yum: awslogs: [] files: /etc/awslogs/awslogs.conf: content: !Sub | [general] state_file = /var/lib/awslogs/agent-state [/var/log/cloud-init-output.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cloud-init-output.log buffer_duration = 5000 log_stream_name = {instance_id}/cloud-init-output.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-init-cmd.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-init-cmd.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-init-cmd.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} [/var/log/cfn-hup.log] datetime_format = %Y-%m-%d %H:%M:%S.%f file = /var/log/cfn-hup.log buffer_duration = 5000 log_stream_name = {instance_id}/cfn-hup.log initial_position = start_of_file log_group_name = /aws/ec2/${AWS::StackName} mode: "000644" owner: "root" group: "root" (略) services: sysvinit: (略) awslogs: enabled: "true" packages: - awslogs files: - /etc/awslogs/awslogs.conf

利用方法

パラメータ値を取得

キーペア

EC2インスタンスへSSHログインするのに利用するキーペアを作成します。既存のキーペアを利用する場合は作成不要です。

# Create KeyPair > aws ec2 create-key-pair \ --key-name cfn-awslogs-test-ec2-key \ --query "KeyMaterial" \ --output text > cfn-awslogs-test-ec2-key.pem > chmod 400 cfn-awslogs-test-ec2-key.pem

VPC、サブネット

既存のVPC、サブネット配下にEC2インスタンスを作成する前提ですので各IDを取得します。

# VpcId > aws ec2 describe-vpcs \ --query "Vpcs" [ { "CidrBlock": "172.31.0.0/16", "DhcpOptionsId": "dopt-b06bd8c8", "State": "available", "VpcId": "vpc-xxxxxxxx", "OwnerId": "xxxxxxxxxxxx", "InstanceTenancy": "default", "CidrBlockAssociationSet": [ { "AssociationId": "vpc-cidr-assoc-2b23e646", "CidrBlock": "172.31.0.0/16", "CidrBlockState": { "State": "associated" } } ], "IsDefault": true }, ] # SubnetId > aws ec2 describe-subnets \ --filters '{"Name": "vpc-id", "Values": ["vpc-xxxxxxxx"]}' \ --query "Subnets" [ { "AvailabilityZone": "us-east-1a", "AvailabilityZoneId": "use1-az2", "AvailableIpAddressCount": 4089, "CidrBlock": "172.31.80.0/20", "DefaultForAz": true, "MapPublicIpOnLaunch": true, "State": "available", "SubnetId": "subnet-xxxxxxxx", "VpcId": "vpc-xxxxxxxx", "OwnerId": "xxxxxxxxxxxx", "AssignIpv6AddressOnCreation": false, "Ipv6CidrBlockAssociationSet": [], "SubnetArn": "arn:aws:ec2:us-east-1:xxxxxxxxxxxx:subnet/subnet-xxxxxxxx" }, ]

自身のグローバルIP

SSHアクセスするためのセキュリティグループに指定するIPアドレスを取得します。ここではifconfig.ioを利用していますが他の手段でもOKです。

## GlobalIP > curl ifconfig.io xxx.xxx.xxx.xxx

スタック作成

aws cloudformation create-stackコマンドを利用してスタック作成します。 AWSマネジメントコンソールから作成してもOKです。

> aws cloudformation create-stack \ --stack-name cfn-awslogs-test \ --template-body file://cfn-template.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 \ --parameters '[ { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-awslogs-test-ec2-key" }, { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" } ]'

スタック作成結果を確認

スタック作成ができたか確認します。

> aws cloudformation describe-stacks \ --stack-name cfn-awslogs-test { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-awslogs-test/f1e864b0-a9fd-11e9-aa93-12ccfe651680", "StackName": "cfn-awslogs-test", "Parameters": [ { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" }, { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "InstanceType", "ParameterValue": "t3.small" } ], "CreationTime": "2019-07-19T08:19:36.199Z", "RollbackConfiguration": {}, "StackStatus": "CREATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Outputs": [ { "OutputKey": "ClientPublicIp", "OutputValue": "xxx.xxx.xxx.xxx" }, { "OutputKey": "ClientInstanceId", "OutputValue": "i-xxxxxxxxxxxxxxxxx" } ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] }

Amazon CloudWatch logsでログを確認する

> aws logs describe-log-streams \ --log-group-name /aws/ec2/cfn-awslogs-test \ --query "logGroups[*].logGroupName" [ "i-xxxxxxxxxxxxxxxxx/cfn-hup.log", "i-xxxxxxxxxxxxxxxxx/cfn-init-cmd.log", "i-xxxxxxxxxxxxxxxxx/cfn-init.log", "i-xxxxxxxxxxxxxxxxx/cloud-init-output.log" ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-hup.log \ --query "events[*].message" [ "2019-07-19 08:22:57,542 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com", "2019-07-19 08:22:57,543 [DEBUG] Creating /var/lib/cfn-hup/data", "2019-07-19 08:22:57,551 [INFO] No umask value specified in config file. Using the default one: 022", "2019-07-19 08:37:57,714 [INFO] cfn-hup processing is alive." ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-init-cmd.log \ --query "events[*].message" \ [ "2019-07-19 08:22:49,803 P2190 [INFO] ************************************************************", "2019-07-19 08:22:49,803 P2190 [INFO] ConfigSet default", "2019-07-19 08:22:49,804 P2190 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++", "2019-07-19 08:22:49,804 P2190 [INFO] Config config", "2019-07-19 08:22:55,092 P2190 [INFO] ============================================================", "2019-07-19 08:22:55,092 P2190 [INFO] yum install awslogs", "2019-07-19 08:22:56,958 P2190 [INFO] -----------------------Command Output-----------------------", "2019-07-19 08:22:56,958 P2190 [INFO] \tLoaded plugins: priorities, update-motd, upgrade-helper", "2019-07-19 08:22:56,958 P2190 [INFO] \tResolving Dependencies", "2019-07-19 08:22:56,958 P2190 [INFO] \t--> Running transaction check", "2019-07-19 08:22:56,958 P2190 [INFO] \t---> Package awslogs.noarch 0:1.1.4-1.12.amzn1 will be installed", "2019-07-19 08:22:56,958 P2190 [INFO] \t--> Processing Dependency: aws-cli-plugin-cloudwatch-logs(python27) for package: awslogs-1.1.4-1.12.amzn1.noarch", "2019-07-19 08:22:56,959 P2190 [INFO] \t--> Running transaction check", "2019-07-19 08:22:56,959 P2190 [INFO] \t---> Package aws-cli-plugin-cloudwatch-logs.noarch 0:1.4.4-1.16.amzn1 will be installed", "2019-07-19 08:22:56,959 P2190 [INFO] \t--> Finished Dependency Resolution", "2019-07-19 08:22:56,959 P2190 [INFO] \t", "2019-07-19 08:22:56,959 P2190 [INFO] \tDependencies Resolved", "2019-07-19 08:22:56,959 P2190 [INFO] \t", "2019-07-19 08:22:56,959 P2190 [INFO] \t================================================================================", "2019-07-19 08:22:56,959 P2190 [INFO] \t Package Arch Version Repository Size", "2019-07-19 08:22:56,959 P2190 [INFO] \t================================================================================", "2019-07-19 08:22:56,959 P2190 [INFO] \tInstalling:", "2019-07-19 08:22:56,959 P2190 [INFO] \t awslogs noarch 1.1.4-1.12.amzn1 amzn-main 9.2 k", "2019-07-19 08:22:56,959 P2190 [INFO] \tInstalling for dependencies:", "2019-07-19 08:22:56,960 P2190 [INFO] \t aws-cli-plugin-cloudwatch-logs noarch 1.4.4-1.16.amzn1 amzn-main 71 k", "2019-07-19 08:22:56,960 P2190 [INFO] \t", "2019-07-19 08:22:56,960 P2190 [INFO] \tTransaction Summary", "2019-07-19 08:22:56,960 P2190 [INFO] \t================================================================================", "2019-07-19 08:22:56,960 P2190 [INFO] \tInstall 1 Package (+1 Dependent package)", "2019-07-19 08:22:56,960 P2190 [INFO] \t", "2019-07-19 08:22:56,960 P2190 [INFO] \tTotal download size: 81 k", "2019-07-19 08:22:56,960 P2190 [INFO] \tInstalled size: 246 k", "2019-07-19 08:22:56,960 P2190 [INFO] \tDownloading packages:", "2019-07-19 08:22:56,960 P2190 [INFO] \t--------------------------------------------------------------------------------", "2019-07-19 08:22:56,960 P2190 [INFO] \tTotal 250 kB/s | 81 kB 00:00 ", "2019-07-19 08:22:56,960 P2190 [INFO] \tRunning transaction check", "2019-07-19 08:22:56,961 P2190 [INFO] \tRunning transaction test", "2019-07-19 08:22:56,961 P2190 [INFO] \tTransaction test succeeded", "2019-07-19 08:22:56,961 P2190 [INFO] \tRunning transaction", "2019-07-19 08:22:56,961 P2190 [INFO] \t Installing : aws-cli-plugin-cloudwatch-logs-1.4.4-1.16.amzn1.noarch 1/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t Installing : awslogs-1.1.4-1.12.amzn1.noarch 2/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t Verifying : awslogs-1.1.4-1.12.amzn1.noarch 1/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t Verifying : aws-cli-plugin-cloudwatch-logs-1.4.4-1.16.amzn1.noarch 2/2 ", "2019-07-19 08:22:56,961 P2190 [INFO] \t", "2019-07-19 08:22:56,962 P2190 [INFO] \tInstalled:", "2019-07-19 08:22:56,962 P2190 [INFO] \t awslogs.noarch 0:1.1.4-1.12.amzn1 ", "2019-07-19 08:22:56,962 P2190 [INFO] \t", "2019-07-19 08:22:56,962 P2190 [INFO] \tDependency Installed:", "2019-07-19 08:22:56,962 P2190 [INFO] \t aws-cli-plugin-cloudwatch-logs.noarch 0:1.4.4-1.16.amzn1 ", "2019-07-19 08:22:56,962 P2190 [INFO] \t", "2019-07-19 08:22:56,962 P2190 [INFO] \tComplete!", "2019-07-19 08:22:56,962 P2190 [INFO] ------------------------------------------------------------", "2019-07-19 08:22:56,962 P2190 [INFO] Completed successfully.", "2019-07-19 08:22:56,965 P2190 [INFO] ============================================================", "2019-07-19 08:22:56,965 P2190 [INFO] Command test", "2019-07-19 08:22:56,969 P2190 [INFO] -----------------------Command Output-----------------------", "2019-07-19 08:22:56,969 P2190 [INFO] \tcfn-awslogs-test test", "2019-07-19 08:22:56,969 P2190 [INFO] ------------------------------------------------------------", "2019-07-19 08:22:56,970 P2190 [INFO] Completed successfully." ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cfn-init.log \ --query "events[*].message" [ "2019-07-19 08:22:49,801 [INFO] -----------------------Starting build-----------------------", "2019-07-19 08:22:49,802 [INFO] Running configSets: default", "2019-07-19 08:22:49,803 [INFO] Running configSet default", "2019-07-19 08:22:49,804 [INFO] Running config config", "2019-07-19 08:22:56,962 [INFO] Yum installed [u'awslogs']", "2019-07-19 08:22:56,970 [INFO] Command test succeeded", "2019-07-19 08:22:56,975 [INFO] enabled service cfn-hup", "2019-07-19 08:22:57,568 [INFO] Restarted cfn-hup successfully", "2019-07-19 08:22:57,607 [INFO] enabled service awslogs", "2019-07-19 08:22:59,648 [INFO] Restarted awslogs successfully", "2019-07-19 08:22:59,649 [INFO] ConfigSets completed", "2019-07-19 08:22:59,650 [INFO] -----------------------Build complete-----------------------", "2019-07-19 08:23:00,249 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com", "2019-07-19 08:23:00,250 [DEBUG] Signaling resource MyInstance in stack cfn-awslogs-test with unique ID i-xxxxxxxxxxxxxxxxx and status SUCCESS" ] > aws logs get-log-events \ --log-group-name /aws/ec2/cfn-awslogs-test \ --log-stream-name i-xxxxxxxxxxxxxxxxx/cloud-init-output.log \ --query "events[*].message" (略) "Updated:\n java-1.7.0-openjdk.x86_64 1:1.7.0.211-2.6.17.1.79.amzn1 \n kernel-tools.x86_64 0:4.14.128-87.105.amzn1 \n perl.x86_64 4:5.16.3-294.43.amzn1 \n perl-Pod-Escapes.noarch 1:1.04-294.43.amzn1 \n perl-libs.x86_64 4:5.16.3-294.43.amzn1 \n perl-macros.x86_64 4:5.16.3-294.43.amzn1 \n python27-jinja2.noarch 0:2.7.2-3.16.amzn1 \n wget.x86_64 0:1.18-5.30.amzn1 \n", "Complete!", "Cloud-init v. 0.7.6 running 'modules:final' at Fri, 19 Jul 2019 08:22:49 +0000. Up 20.37 seconds.", "start UserData", "finish UserData", "Cloud-init v. 0.7.6 finished at Fri, 19 Jul 2019 08:23:00 +0000. Datasource DataSourceEc2. Up 31.51 seconds" ]

まとめ

CloudWatch Logs エージェントを利用することでCFnでEC2インスタンス作成・更新時のログをAmazon CloudWatch Logsへ出力することができました。個人的にはこれでCFnで実行エラーによるロールバックでEC2インスタンスが削除されてログが確認できない悲しみがなくなって満足です^^

参考

CloudWatch Logs エージェントのリファレンス – Amazon CloudWatch Logs https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/AgentReference.html

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた – Qiita https://cloudpack.media/48540

元記事はこちら

「AWS CloudFormationでEC2インスタンスのログがAmazon CloudWatch logsで確認できるようにする」

August 14, 2019 at 12:00PM

#AWS #Official #Blog

0 notes

computingpostcom · 3 years ago

Text

In this tutorial we will discuss on how to configure EKS Persistent Storage with EFS Amazon service for your Kubernetes cluster to use. The storage backend service we’ll be using is EFS, this will be our default persistent storage for volume claims used by stateful applications. A StorageClass provides a way for administrators to describe the “classes” of storage they offer to allow for dynamic provisioning of persistent volumes. In Kubernetes, a A PersistentVolume (PV) is a piece of storage in the cluster while PersistentVolumeClaim (PVC) is a request for storage by a user, usually Pod. You need a working EKS cluster before you can use this guide to setup persistent storage for your containerized workloads. Setup Pre-requisites: EKS Cluster: Setup EKS Cluster with eksctl AWS CLI This is the name of the EKS cluster I’ll be working with in this tutorial. $ eksctl get cluster NAME REGION prod-eks-cluster eu-west-1 Save the cluster name as a variable that will be used in the remaining steps. EKS_CLUSTER="prod-eks-cluster" Using EFS CSI driver to create Persistent Volumes The Amazon Elastic File System Container Storage Interface (CSI) Driver implements the CSI specification for container orchestrators to manage the lifecycle of Amazon EFS file systems. Step 1: Create an Amazon EFS File System The Amazon EFS CSI driver supports Amazon EFS access points, which are application-specific entry points into an Amazon EFS file system that make it easier to share a file system between multiple pods. You can perform these operations from Amazon console or from the terminal. I’ll be using AWS CLI interface in all my operations. Locate the VPC ID of your Amazon EKS cluster: EKS_CLUSTER="prod-eks-cluster" EKS_VPC_ID=$(aws eks describe-cluster --name $EKS_CLUSTER --query "cluster.resourcesVpcConfig.vpcId" --output text) Confirm VPC ID if valid. $ echo $EKS_VPC_ID vpc-019a6458a973ace2b Locate the CIDR range for your cluster’s VPC: EKS_VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids $EKS_VPC_ID --query "Vpcs[].CidrBlock" --output text) Confirm VPC CIDR: $ echo $EKS_VPC_CIDR 192.168.0.0/16 Create a security group that allows inbound NFS traffic for your Amazon EFS mount points: aws ec2 create-security-group --group-name efs-nfs-sg --description "Allow NFS traffic for EFS" --vpc-id $EKS_VPC_ID Take note of Security group ID. Mine is: "GroupId": "sg-0fac73a0d7d943862" # You can check with $ aws ec2 describe-security-groups --query "SecurityGroups[*].Name:GroupName,ID:GroupId" Add rules to your security group: SG_ID="sg-0fac73a0d7d943862" aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 2049 --cidr $EKS_VPC_CIDR To view the changes to the security group, run the describe-security-groups command: $ aws ec2 describe-security-groups --group-ids $SG_ID "SecurityGroups": [ "Description": "Allow NFS traffic for EFS", "GroupName": "efs-nfs-sg", "IpPermissions": [ "FromPort": 2049, "IpProtocol": "tcp", "IpRanges": [ "CidrIp": "192.168.0.0/16" ], "Ipv6Ranges": [], "PrefixListIds": [], "ToPort": 2049, "UserIdGroupPairs": [] ], "OwnerId": "253859766502", "GroupId": "sg-0fac73a0d7d943862", "IpPermissionsEgress": [ "IpProtocol": "-1", "IpRanges": [ "CidrIp": "0.0.0.0/0" ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] ],

"VpcId": "vpc-019a6458a973ace2b" ] Create the Amazon EFS file system for your Amazon EKS cluster: # Not encrypted $ aws efs create-file-system --region eu-west-1 # Encrypted EFS file system $ aws efs create-file-system --encrypted --region eu-west-1 Note the file system ID: "OwnerId": "253759766542", "CreationToken": "c16c4603-c7ac-408f-ac4a-75a683ed2a29", "FileSystemId": "fs-22ac06e8", "FileSystemArn": "arn:aws:elasticfilesystem:eu-west-1:253759766542:file-system/fs-22ac06e8", "CreationTime": "2020-08-16T15:17:18+03:00", "LifeCycleState": "creating", "NumberOfMountTargets": 0, "SizeInBytes": "Value": 0, "ValueInIA": 0, "ValueInStandard": 0 , "PerformanceMode": "generalPurpose", "Encrypted": true, "KmsKeyId": "arn:aws:kms:eu-west-1:253759766542:key/6c9b725f-b86d-41c2-b804-1685ef43f620", "ThroughputMode": "bursting", "Tags": [] UI view: Step 2: Create EFS Mount Target Get Subnets in your VPC where EC2 instances run. In my case all EKS instances run in private subnets. EKS_VPC_ID=$(aws eks describe-cluster --name $EKS_CLUSTER --query "cluster.resourcesVpcConfig.vpcId" --output text) aws ec2 describe-subnets --filter Name=vpc-id,Values=$EKS_VPC_ID --query 'Subnets[?MapPublicIpOnLaunch==`false`].SubnetId' My output: [ "subnet-0977bbaf236bd952f", "subnet-0df8523ca39f63938", "subnet-0a4a22d25f36c4124" ] Create Mount Targets. # File system ID EFS_ID="fs-22ac06e8" # Create mount targets for the subnets - Three subnets in my case for subnet in subnet-0977bbaf236bd952f subnet-0df8523ca39f63938 subnet-0a4a22d25f36c4124; do aws efs create-mount-target \ --file-system-id $EFS_ID \ --security-group $SG_ID \ --subnet-id $subnet \ --region eu-west-1 done Step 2: Using EFS CSI driver After creation of an EFS file system and and Mount Target we can test EFS CSI driver by creating static persistent volume and claiming it through a test container. Deploy EFS CSI provisioner. $ kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/dev/?ref=master" daemonset.apps/efs-csi-node created csidriver.storage.k8s.io/efs.csi.aws.com created List available CSI drivers: $ kubectl get csidrivers.storage.k8s.io NAME CREATED AT efs.csi.aws.com 2020-08-16T19:10:35Z First get the EFS filesystem ID: $ aws efs describe-file-systems --query "FileSystems[*].FileSystemId" [ "fs-22ac06e8" ] Create storage class: kubectl apply -f -

0 notes

computingpostcom · 3 years ago

Text

This article helps a user create a MYSQL database instance using CloudFormation automation service. RDS stands for relational database service. It is a managed AWS (Amazon Web Services) service simplifying the setting up and management of relational databases. RDS supports various database engines. They Include: MYSQL Amazon Aurora PostgreSQL Maria DB Oracle SQL Server The benefits of using AWS RDS include: RDS makes it easier to provision and manage your RDS databases. There is no need to worry about software patches nor go through the complex process of provisioning instances and installing software on the instances. RDS makes it easier to scale our databases should there be a need for that (read replicas). Also, the service provides the user with options to ensure a highly available setup (multi-az). Requirements/Prerequisites Before you start the setup have a look at the setup pre-requisites listed below. The user will need to have: An AWS Account. Created a user with permissions to create resources on the AWS Account. An IDE like visual studio code to write and edit your CloudFormation Template. Created a VPC with subnets and an Internet Connection. Created a Parameter group for the database instance. N/B: For an RDS MYSQL database instance we cannot create the parameter group with a CloudFormation template. It has to be created beforehand and used as an input variable (Parameter on our template). Step 1: Create Database Instance Parameter Group The parameter group allows you to manage your database engine configurations. To manually create an RDS database parameter group, follow the below steps. On the AWS RDS console select parameter groups then click create parameter group. Create a Parameter Group Next, enter the parameter group details. For our case, we are creating a MySQL version 8 db instance hence we filled the details as below. Enter Parameter Group Details When done click create. It will create the database parameter group. Step 2: Create CloudFormation Template Use the below Template to create your RDS MYSQL database instance. AWSTemplateFormatVersion: "2010-09-09" Description: "Create a DB subnet group and MYSQL Database" Parameters: VPC: Type: String Description: The VPC to create the cluster Default: vpc-ID PrivateSubnet01: Type: String Description: The subnet for the DB cluster Default: subnet-ID PrivateSubnet02: Type: String Description: The subnet for the DB cluster Default: subnet-ID MasterUsername: Type: String Description: The username for our database. MasterUserPassword: Type: String Description: The password for the database. "NoEcho": true ParameterGroup: Type: String Description: The name of the database parameter group created. Resources: EC2SecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: "Database instances security group" VpcId: !Ref VPC SecurityGroupIngress: - CidrIp: "*.*.*.*/32" FromPort: 3306 IpProtocol: "tcp" ToPort: 3306 SecurityGroupEgress: - CidrIp: "0.0.0.0/0" IpProtocol: "-1" RDSDBSubnetGroup: Type: "AWS::RDS::DBSubnetGroup" Properties: DBSubnetGroupDescription: "Subnet Group for mySQL database" DBSubnetGroupName: !Sub "$AWS::Region-aws-dxl-database-subnet-group" SubnetIds: - !Ref PrivateSubnet01 - !Ref PrivateSubnet02 Tags: - Key: Name Value: eu-central-1-test-db-cluster - Key: createdBy Value: Maureen Barasa - Key: Project Value: test-blog - Key: Environment

Value: test RDSDBInstance: Type: AWS::RDS::DBInstance Properties: DBInstanceIdentifier: aws-dxl-database-1 AllocatedStorage: 100 DBInstanceClass: db.m5.large Engine: "MYSQL" MasterUsername: !Ref MasterUsername MasterUserPassword: !Ref MasterUserPassword BackupRetentionPeriod: 7 MultiAZ: true EngineVersion: 8.0.20 AutoMinorVersionUpgrade: true Iops: 1000 PubliclyAccessible: false StorageType: io1 Port: 3306 StorageEncrypted: true CopyTagsToSnapshot: true MonitoringInterval: 60 EnableIAMDatabaseAuthentication: false EnablePerformanceInsights: true PerformanceInsightsRetentionPeriod: 7 DeletionProtection: true DBSubnetGroupName: !Ref RDSDBSubnetGroup VPCSecurityGroups: - !Ref EC2SecurityGroup MaxAllocatedStorage: 1000 DBParameterGroupName: !Ref ParameterGroup MonitoringRoleArn: !Sub "arn:aws:iam::$AWS::AccountId:role/rds-monitoring-role" Tags: - Key: Name Value: aws-dxl-database-1 - Key: createdBy Value: Maureen Barasa - Key: Project Value: test-blog - Key: Environment Value: test Outputs: Cluster: Description: The DB Cluster Name Value: !Ref RDSDBInstance SubnetGroup: Description: The db subnet group name Value: !Ref RDSDBSubnetGroup We can deploy the CloudFormation Template using a CloudFormation stack. Deploy a CloudFormation Template Source: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-howdoesitwork.html The CloudFormation Template Explained The template comprises 3 sections. The Parameters, Resources and Outputs sections. Parameters: In the resources section, we require the user to input the dynamic variables of their template. For our case, the user should replace the VPC and subnet ID’s with their respective VPC and subnet ID’s. Next, it will prompt the user to input their database master username and password. Finally, the user will be required to input the name of the parameter group created earlier on. Resources: Here the user defines the AWS resources to create. For our case, we start by creating the database instance security group. The user should change the security group ingress to reflect the CIDR IP Block that they would like to permit access to the Database instances. Next, it creates the DB subnet group. The subnet group defines the subnets where the database cluster and instances are created. Also, the user should pay attention to the names and tags to customize as needed. Finally, the DB Instance is created. However, the user should go through the template and change the instance properties to match their specific needs. Also, the DB instance identifier and tags should be customized to meet user requirements. Outputs: The output section of the template instructs CloudFormation to output the names of the resources created. For example, in our case, we have instructed the template to output the names of the DB instance and subnet group.

0 notes

computingpostcom · 3 years ago

Text

This article helps a user create a document database cluster with a single primary instance using a CloudFormation template. A document database is important when a user wants to run Mongo database workloads on AWS (Amazon Web Services). The Amazon DocumentDB (with MongoDB compatibility) is a scalable, fully managed, fast and highly available document database service that supports MongoDB workloads. This managed non-relational database service makes it easier to store, query and index JSON data. This database service is designed from the ground-up to give guarantee scalability, performance, and availability you need when operating mission-critical MongoDB workloads at scale. Setup Pre-requisites The user will need to have: An AWS Account Created a user with permissions to create resources on the AWS Account An IDE like visual studio code to write and edit your CloudFormation Template. CloudFormation Template used Kindly find below the CloudFormation Template. The template will create: The database instance security group. Database subnet group. The database parameter group. Document database Cluster. Database instance. --- AWSTemplateFormatVersion: "2010-09-09" Description: Template to Create a document DB parameter group, subnet group and cluster Parameters: VPC: Type: String Description: The VPC to create the cluster Default: vpc-ID PrivateSubnet01: Type: String Description: The subnet for the DB cluster Default: subnet-ID PrivateSubnet02: Type: String Description: The subnet for the DB cluster Default: subnet-ID MasterUsername: Type: String Description: The username for our database. MasterUserPassword: Type: String Description: The password for the database. "NoEcho": true Resources: DBSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: "DB instances security group" GroupName: "test-db-instance-SG" VpcId: !Ref VPC SecurityGroupIngress: - CidrIp: "*.*.*.*/32" FromPort: 22017 IpProtocol: "tcp" ToPort: 22017 SecurityGroupEgress: - CidrIp: "0.0.0.0/0" IpProtocol: "-1" DBSubnetGroup: Type: AWS::DocDB::DBSubnetGroup Properties: DBSubnetGroupDescription: "test document db subnet group" DBSubnetGroupName: "eu-central-1-test-db-subnet-group" SubnetIds: - !Ref PrivateSubnet01 - !Ref PrivateSubnet02 Tags: - Key: Name Value: eu-central-1-test-db-subnet-group - Key: createdBy Value: Maureen Barasa - Key: Project Value: test-blog - Key: Environment Value: test DBParameterGroup: Type: AWS::DocDB::DBClusterParameterGroup Properties: Description: "our test document db parameter group" Family: docdb3.6 Name: test-db-parameter-group Parameters: audit_logs: "disabled" tls: "enabled" ttl_monitor: "enabled" Tags: - Key: Name Value: eu-central-1-test-db-cluster - Key: createdBy Value: Maureen Barasa - Key: Project Value: test-blog - Key: Environment Value: test DBCluster: Type: AWS::DocDB::DBCluster Properties: BackupRetentionPeriod : 5 DBClusterIdentifier : eu-central-1-test-db-cluster DBClusterParameterGroupName : !Ref DBParameterGroup DBSubnetGroupName : !Ref DBSubnetGroup MasterUsername : !Ref MasterUsername MasterUserPassword : !Ref MasterUserPassword Port : "27017" PreferredBackupWindow : "23:00-23:59" PreferredMaintenanceWindow : "sun:00:00-sun:05:00" VpcSecurityGroupIds: - !Ref DBSecurityGroup StorageEncrypted : true Tags:

- Key: Name Value: eu-central-1-test-db-cluster - Key: createdBy Value: Maureen Barasa - Key: Project Value: test-blog - Key: Environment Value: test DBInstance: Type: AWS::DocDB::DBInstance Properties: AutoMinorVersionUpgrade: true AvailabilityZone: "eu-west-1a" DBClusterIdentifier: !Ref DBCluster DBInstanceClass: "db.t3.medium" DBInstanceIdentifier: "test-cluster-instance-1" PreferredMaintenanceWindow: "sun:00:00-sun:05:00" Tags: - Key: Name Value: eu-central-1-test-db-instance - Key: createdBy Value: Maureen Barasa - Key: Project Value: test-blog - Key: Environment Value: test Outputs: Cluster: Description: The DB Cluster Name Value: !Ref DBCluster SubnetGroup: Description: The db subnet group name Value: !Ref DBSubnetGroup ParameterGroup: Description: The db subnet group name Value: !Ref DBParameterGroup We can deploy the CloudFormation Template using a CloudFormation stack. The Template Explained The template comprises 3 sections. The Parameters, Resources and Outputs sections. Parameters: In the resources section, we require the user to input the dynamic variables of their template. For our case, the user should replace the VPC and subnet ID’s with their respective VPC and subnet ID’s. Also, the user will be prompted to input their database master username and password. Kindly ensure that you do not use admin as the master username. Resources: Here the user defines the AWS resources to create. For our case, we start by creating the database instance security group. The user should change the security group ingress to reflect the CIDR IP Block that they would like to permit access to the Database instances. Next, it creates the DB subnet and parameter groups. The subnet group defines the subnets where the database cluster and instances are created. The parameter group allows you to manage your database engine configurations. The user should go through the parameter group properties and change to their specific requirements. Also, the user should pay attention to the names and tags to customize as needed. Then the document database cluster is created. Just as above, the user should go through all the cluster properties and change them to match their requirements. Finally, the DB Instance is created. However, the user should go through the template and change the availability zone, the instance class, and the preferred maintenance needs to match their specific needs. Also, the DB instance identifier and tags should be customized to meet user requirements. Outputs: The outputs section of the template instructs CloudFormation to output the names of the resources created. For example, in our case, we have instructed the template to output the names of the cluster, subnet, and parameter groups. Important Links https://aws.amazon.com/documentdb/ https://aws.amazon.com/blogs/database/category/database/amazon-document-db/

0 notes

awsexchage · 6 years ago

Photo

AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた https://ift.tt/2P2V4wW

cloudpack あら便利カレンダー 2019の記事となります。誕生秘話はこちら。

AWS CloudFormation(CFn)でEC2インスタンスを作成するのに、AWS::CloudFormation::Initタイプを利用すると、インスタンス起動後にパッケージのインストールやファイル作成、コマンド実行を含めることができて便利そうだったのでお試ししてみました。

個人的に一番のメリットはスタック更新時にメタデータの変更も検知して再実行させることができる点です。

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

利用するのにいくつかのドキュメントを渡り歩く必要があったので、必要最低限のテンプレートと利用方法、ポイントをまとめました。

前提

AWSアカウントがある

AWS CLIがインストール済みで利用可能

CFn、EC2関連の権限がある

テンプレート

EC2インスタンスを作成・管理するテンプレートとなります。

EC2インスタンスを起動するVPCやサブネットは既存のリソースを利用前提となります。 SSHアクセスする際のキーペアは事前に作成し、セキュリティグループはインスタンスとあわせて作成しています。作成するリージョンは検証なので、us-east-1のみとしています。

cfn-template.yaml

ポイント

AWS::CloudFormation::Initタイプを利用する

AWS::CloudFormation::Initタイプを利用すると、EC2インスタンスでcfn-initヘルパースクリプト用のメタデータとして取り込まれて実行されます。 cfn-initヘルパースクリプトはメタデータに応じて以下のような操作を行います。

CFnのメタデータの取得と解析

パッケージのインストール

ディスクへのファイルの書き込み

Linux/UNIXグループ・ユーザー作成

ソースのダウンロード・展開

コマンド実行

サービスの有効化/無効化と開始/停止

詳しくは下記が参考になります。

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

cfn-init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-init.html

今回のテンプレートではメタデータでファイル書き込み、コマンド実行とサービス有効化する定義をしています。詳細は後ほど。

cfn-template.yaml_一部抜粋

cfn-init ヘルパースクリプトをUserDataで実行する

AWS::CloudFormation::Initタイプを利用してメタデータを定義しただけだと、スタック作成時にcfn-initヘルパースクリプトは実行されないため、ユーザーデータを定義して実行します。ユーザーデータについては下記が参考になります。

Linux インスタンスでの起動時のコマンドの実行 – Amazon Elastic Compute Cloud https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/user-data.html

今回のテンプレートでは/opt/aws/bin/cfn-initでエラーとなった場合、リソース作成失敗となるように/opt/aws/bin/cfn-signalも実行するようにしています。/opt/aws/bin/cfn-signalを利用する場合には、CreationPolicyを定義しておく必要があります。 cfn-signalヘルパースクリプトについては下記が参考になります。

cfn-signal – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-signal.html

cfn-template.yaml_一部抜粋

MyInstance: (略) Properties: (略) UserData: Fn::Base64: !Sub | #!/bin/bash echo "start UserData" /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} echo "finish UserData" CreationPolicy: ResourceSignal: Timeout: PT5M

cfn-hup ヘルパーでメタデータ更新時に対応する

メタデータとユーザーデータの定義だけだと、メタデータを変更してスタック更新しても変更が検知されず、EC2インスタンスに反映されないため、cfn-hup ヘルパーを利用する必要があります。cfn-hupヘルパーはEC2インスタンスでデーモンとして実行が可能です。 cfn-hupデーモンを実行するためにはcfn-hup.confとhooks.confファイルを用意する必要があるため、メタデータで定義しています。 cfn-hupヘルパーについては下記が参考になります。

cfn-hup – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-hup.html

cfn-template.yaml_一部抜粋

MyInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: files: /etc/cfn/cfn-hup.conf: content: !Sub | [main] stack = ${AWS::StackName} region = ${AWS::Region} interval = 1 mode: "000400" owner: "root" group: "root" /etc/cfn/hooks.d/cfn-auto-reloader.conf: content: !Sub | [cfn-auto-reloader-hook] triggers = post.update path = Resources.MyInstance.Metadata.AWS::CloudFormation::Init action = /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region} runas = root mode: "000400" owner: "root" group: "root" commands: (略) services: sysvinit: cfn-hup: enabled: "true" files: - /etc/cfn/cfn-hup.conf - /etc/cfn/hooks.d/cfn-auto-reloader.conf

利用方法

パラメータ値を取得

キーペア

EC2インスタンスへSSHログインするのに利用するキーペアを作成します。既存のキーペアを利用する場合は作成不要です。

# Create KeyPair > aws ec2 create-key-pair \ --key-name cfn-init-test-ec2-key \ --query "KeyMaterial" \ --output text > cfn-init-test-ec2-key.pem > chmod 400 cfn-init-test-ec2-key.pem

VPC、サブネット

既存のVPC、サブネット配下にEC2インスタンスを作成する前提ですので各IDを取得します。

自身のグローバルIP

SSHアクセスするためのセキュリティグループに指定するIPアドレスを取得します。ここではifconfig.ioを利用していますが他の手段でもOKです。

## GlobalIP > curl ifconfig.io xxx.xxx.xxx.xxx

スタック作成

aws cloudformation create-stackコマンドを利用��てスタック作成します。 AWSマネジメントコンソールから作成してもOKです。

> aws cloudformation create-stack \ --stack-name cfn-init-test \ --template-body file://cfn-init-template.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 \ --parameters '[ { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" } ]'

スタック作成結果を確認

スタック作成ができたか確認します。

> aws cloudformation describe-stacks \ --stack-name cfn-init-test { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-init-test/a1dc6aa0-a9e6-11e9-b171-0a515b01a4a4", "StackName": "cfn-init-test", "Parameters": [ { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" }, { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "InstanceType", "ParameterValue": "t3.small" } ], "CreationTime": "2019-07-19T05:32:43.477Z", "RollbackConfiguration": {}, "StackStatus": "CREATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Outputs": [ { "OutputKey": "ClientPublicIp", "OutputValue": "xxx.xxx.xxx.xxx" }, { "OutputKey": "ClientInstanceId", "OutputValue": "i-xxxxxxxxxxxxxxxxx" } ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] }

EC2インスタンスで確認

テンプレートのOutputsでEC2インスタンスのIPアドレス(‘ClientPublicIp’ )が出力されるようにしているので、SSHログインして各ログを確認します。 CFnでEC2インスタンスを作成するといくつかのログファイルが作成されますが、下記のファイル確認しておくとだいたいは把握できます。

スタック更新に関わるログとファイルは以下となります。 cfn-hupデーモンが定期的にCFnスタックのメタデータを取得しmetadata_db.jsonに保存、メタデータとファイルに差分があるとcfn-auto-reloader.confで指定したアクション(コマンド)が実行される仕組みになっています。なので、スタック作成後、cfn-hupデーモンが1度もメタデータ取得していないタイミングでスタック更新しても、metadata_db.jsonが存在していないと変更が検知されないみたいです。(1敗

$ cat /var/log/cfn-hup.log 2019-07-19 06:53:00,750 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-07-19 06:53:00,750 [DEBUG] Creating /var/lib/cfn-hup/data 2019-07-19 06:53:00,756 [INFO] No umask value specified in config file. Using the default one: 022 $ sudo cat /var/lib/cfn-hup/data/metadata_db.json {"cfn-auto-reloader-hook|Resources.MyInstance.Metadata.AWS::CloudFormation::Init": {"config": {"files": {"/etc/cfn/cfn-hup.conf": {"owner": "root", "content": "[main]\nstack = cfn-init-test\nregion = us-east-1\ninterval = 1\n", "group": "root", "mode": "000400"}, "/etc/cfn/hooks.d/cfn-auto-reloader.conf": {"owner": "root", "content": "[cfn-auto-reloader-hook]\ntriggers = post.update\npath = Resources.MyInstance.Metadata.AWS::CloudFormation::Init\naction = /opt/aws/bin/cfn-init -v --stack cfn-init-test --resource MyInstance --region us-east-1\nrunas = root\n", "group": "root", "mode": "000400"}}, "services": {"sysvinit": {"cfn-hup": {"files": ["/etc/cfn/cfn-hup.conf", "/etc/cfn/hooks.d/cfn-auto-reloader.conf"], "enabled": "true"}}}, "commands": {"test": {"command": "echo $STACK_NAME test", "env": {"STACK_NAME": "cfn-init-test"}}}}}}

メタデータを編集してスタック更新する

メタデータに実行するコマンドを追加してスタック更新してみます。

cfn-template.yaml_一部抜粋

(略) commands: test: command: "echo $STACK_NAME test" env: STACK_NAME: !Ref AWS::StackName test2: command: "echo $STACK_NAME test2" env: STACK_NAME: !Ref AWS::StackName (略)

aws cloudformation update-stackコマンドを実行してスタック更新します。

> aws cloudformation update-stack \ --stack-name cfn-init-test \ --template-body file://cfn-init-template.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 \ --parameters '[ { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" } ]'

スタック更新結果を確認

> aws cloudformation describe-stacks \ --stack-name cfn-init-test { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/cfn-init-test/a1dc6aa0-a9e6-11e9-b171-0a515b01a4a4", "StackName": "cfn-init-test", "Parameters": [ { "ParameterKey": "MyInstanceSSHCidrIp", "ParameterValue": "xxx.xxx.xxx.xxx/32" }, { "ParameterKey": "VpcId", "ParameterValue": "vpc-xxxxxxxx" }, { "ParameterKey": "EC2KeyPairName", "ParameterValue": "cfn-init-test-ec2-key" }, { "ParameterKey": "SubnetId", "ParameterValue": "subnet-xxxxxxxx" }, { "ParameterKey": "InstanceType", "ParameterValue": "t3.small" } ], "CreationTime": "2019-07-19T05:32:43.477Z", "LastUpdatedTime": "2019-07-19T05:41:40.446Z", "RollbackConfiguration": {}, "StackStatus": "UPDATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Outputs": [ { "OutputKey": "ClientPublicIp", "OutputValue": "xxx.xxx.xxx.xxx" }, { "OutputKey": "ClientInstanceId", "OutputValue": "i-xxxxxxxxxxxxxxxxx" } ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] }

EC2インスタンスで確認

CFnのリソース更新されてもEC2インスタンス側への変更反映は、cfn-hupヘルパーが担当しますので、反映にはタイムラグが発生します。cfn-hupヘルパーがメタデータの変更確認する間隔は初期設定で15分となっており、/etc/cfn/cfn-hup.conf ファイルのintervalで指定ができます。(今回は1分で指定)

cfn-hup – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-hup.html

> ssh -i cfn-init-test-ec2-key.pem [email protected] $ cat /var/log/cfn-init.log 2019-07-19 06:53:00,454 [INFO] -----------------------Starting build----------------------- 2019-07-19 06:53:00,455 [INFO] Running configSets: default 2019-07-19 06:53:00,455 [INFO] Running configSet default 2019-07-19 06:53:00,456 [INFO] Running config config 2019-07-19 06:53:00,460 [INFO] Command test succeeded 2019-07-19 06:53:00,467 [INFO] enabled service cfn-hup 2019-07-19 06:53:00,768 [INFO] Restarted cfn-hup successfully 2019-07-19 06:53:00,769 [INFO] ConfigSets completed 2019-07-19 06:53:00,769 [INFO] -----------------------Build complete----------------------- 2019-07-19 06:53:01,055 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-07-19 06:53:01,055 [DEBUG] Signaling resource MyInstance in stack cfn-init-test with unique ID i-xxxxxxxxxxxxxxxxx and status SUCCESS 2019-07-19 06:56:01,457 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-07-19 06:56:01,458 [DEBUG] Describing resource MyInstance in stack cfn-init-test 2019-07-19 06:56:01,522 [INFO] -----------------------Starting build----------------------- 2019-07-19 06:56:01,523 [DEBUG] Not setting a reboot trigger as scheduling support is not available 2019-07-19 06:56:01,524 [INFO] Running configSets: default 2019-07-19 06:56:01,525 [INFO] Running configSet default 2019-07-19 06:56:01,525 [INFO] Running config config 2019-07-19 06:56:01,526 [DEBUG] No packages specified 2019-07-19 06:56:01,526 [DEBUG] No groups specified 2019-07-19 06:56:01,526 [DEBUG] No users specified 2019-07-19 06:56:01,526 [DEBUG] No sources specified 2019-07-19 06:56:01,526 [DEBUG] /etc/cfn/cfn-hup.conf already exists 2019-07-19 06:56:01,526 [DEBUG] Moving /etc/cfn/cfn-hup.conf to /etc/cfn/cfn-hup.conf.bak 2019-07-19 06:56:01,526 [DEBUG] Writing content to /etc/cfn/cfn-hup.conf 2019-07-19 06:56:01,526 [DEBUG] Setting mode for /etc/cfn/cfn-hup.conf to 000400 2019-07-19 06:56:01,526 [DEBUG] Setting owner 0 and group 0 for /etc/cfn/cfn-hup.conf 2019-07-19 06:56:01,527 [DEBUG] /etc/cfn/hooks.d/cfn-auto-reloader.conf already exists 2019-07-19 06:56:01,527 [DEBUG] Moving /etc/cfn/hooks.d/cfn-auto-reloader.conf to /etc/cfn/hooks.d/cfn-auto-reloader.conf.bak 2019-07-19 06:56:01,527 [DEBUG] Writing content to /etc/cfn/hooks.d/cfn-auto-reloader.conf 2019-07-19 06:56:01,527 [DEBUG] Setting mode for /etc/cfn/hooks.d/cfn-auto-reloader.conf to 000400 2019-07-19 06:56:01,527 [DEBUG] Setting owner 0 and group 0 for /etc/cfn/hooks.d/cfn-auto-reloader.conf 2019-07-19 06:56:01,527 [DEBUG] Running command test 2019-07-19 06:56:01,527 [DEBUG] No test for command test 2019-07-19 06:56:01,531 [INFO] Command test succeeded 2019-07-19 06:56:01,531 [DEBUG] Command test output: cfn-init-test test 2019-07-19 06:56:01,531 [DEBUG] Running command test2 2019-07-19 06:56:01,531 [DEBUG] No test for command test2 2019-07-19 06:56:01,535 [INFO] Command test2 succeeded 2019-07-19 06:56:01,536 [DEBUG] Command test2 output: cfn-init-test test! 2019-07-19 06:56:01,536 [DEBUG] Using service modifier: /sbin/chkconfig 2019-07-19 06:56:01,536 [DEBUG] Setting service cfn-hup to enabled 2019-07-19 06:56:01,538 [INFO] enabled service cfn-hup 2019-07-19 06:56:01,539 [DEBUG] Not modifying running state of service cfn-hup 2019-07-19 06:56:01,539 [INFO] ConfigSets completed 2019-07-19 06:56:01,539 [DEBUG] Not clearing reboot trigger as scheduling support is not available 2019-07-19 06:56:01,539 [INFO] -----------------------Build complete----------------------- $ cat /var/log/cfn-init-cmd.log 2019-07-19 06:53:00,455 P2208 [INFO] ************************************************************ 2019-07-19 06:53:00,455 P2208 [INFO] ConfigSet default 2019-07-19 06:53:00,456 P2208 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2019-07-19 06:53:00,456 P2208 [INFO] Config config 2019-07-19 06:53:00,457 P2208 [INFO] ============================================================ 2019-07-19 06:53:00,457 P2208 [INFO] Command test 2019-07-19 06:53:00,460 P2208 [INFO] -----------------------Command Output----------------------- 2019-07-19 06:53:00,460 P2208 [INFO] cfn-init-test test 2019-07-19 06:53:00,460 P2208 [INFO] ------------------------------------------------------------ 2019-07-19 06:53:00,460 P2208 [INFO] Completed successfully. 2019-07-19 06:56:01,525 P2337 [INFO] ************************************************************ 2019-07-19 06:56:01,525 P2337 [INFO] ConfigSet default 2019-07-19 06:56:01,525 P2337 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2019-07-19 06:56:01,526 P2337 [INFO] Config config 2019-07-19 06:56:01,527 P2337 [INFO] ============================================================ 2019-07-19 06:56:01,527 P2337 [INFO] Command test 2019-07-19 06:56:01,530 P2337 [INFO] -----------------------Command Output----------------------- 2019-07-19 06:56:01,531 P2337 [INFO] cfn-init-test test 2019-07-19 06:56:01,531 P2337 [INFO] ------------------------------------------------------------ 2019-07-19 06:56:01,531 P2337 [INFO] Completed successfully. 2019-07-19 06:56:01,531 P2337 [INFO] ============================================================ 2019-07-19 06:56:01,531 P2337 [INFO] Command test2 2019-07-19 06:56:01,535 P2337 [INFO] -----------------------Command Output----------------------- 2019-07-19 06:56:01,535 P2337 [INFO] cfn-init-test test! 2019-07-19 06:56:01,535 P2337 [INFO] ------------------------------------------------------------ 2019-07-19 06:56:01,535 P2337 [INFO] Completed successfully. $ cat /var/log/cloud-init-output.log (略) Updated: java-1.7.0-openjdk.x86_64 1:1.7.0.211-2.6.17.1.79.amzn1 kernel-tools.x86_64 0:4.14.128-87.105.amzn1 perl.x86_64 4:5.16.3-294.43.amzn1 perl-Pod-Escapes.noarch 1:1.04-294.43.amzn1 perl-libs.x86_64 4:5.16.3-294.43.amzn1 perl-macros.x86_64 4:5.16.3-294.43.amzn1 python27-jinja2.noarch 0:2.7.2-3.16.amzn1 wget.x86_64 0:1.18-5.30.amzn1 Complete! Cloud-init v. 0.7.6 running 'modules:final' at Fri, 19 Jul 2019 06:52:59 +0000. Up 18.81 seconds. start UserData finish UserData Cloud-init v. 0.7.6 finished at Fri, 19 Jul 2019 06:53:01 +0000. Datasource DataSourceEc2. Up 20.07 seconds

メタデータに追加したコマンドが実行されるのを確認できました。やったぜ。

スタック更新時にメタデータで定義しているすべての処理を実行したくない場合、Configsetを用いるとグルーピングや実行順の制御をして特定の処理ができます。詳細は下記をご参考ください。

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

まとめ

CFnでEC2インスタンスを作成する際に、インスタンス起動後の処理もテンプレートで定義して管理できる仕組みになっていますので、非常に使い勝手が良さそうです。

参考

AWS::CloudFormation::Init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

cfn-init – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-init.html

Linux インスタンスでの起動時のコマンドの実行 – Amazon Elastic Compute Cloud https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/user-data.html

cfn-signal – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-signal.html

cfn-hup – AWS CloudFormation https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-hup.html

元記事はこちら

「AWS::CloudFormation::Init タイプを使ってEC2インスタンスの環境構築ができるようにしてみた」

August 13, 2019 at 02:00PM

#AWS #Official #Blog

0 notes

awsexchage · 8 years ago

Photo

ECSでBlue/Greenデプロイ【cloudpack大阪ブログ】 http://ift.tt/2ydrJq3

cloudpack大阪の佐々木です。

今扱っているECS環境では、デプロイをCloudFormationでやっています。実際やってみると、アップデート途中で止まるってことが割と頻繁にあって、インプレイスでアップデートするのは怖いなということで、Blue/Greenデプロイ環境をつくってみました。

http://ift.tt/2tkIxGP

元ネタはこれなんですが、このサンプルだと、CodePipelineとか使ってややこしいいい感じので、もう少し簡単に。

初期設定

まず、Blueを本番、Greenをステージングとして、下記のように設定します。

ALB設定

ダイナミックポートマッピングで、Blue、Greenそれぞれのターゲットグループを作成

Blue用ターゲットグループをTCP/80(本番用ポート)にマッピング

Green用ターゲットグループをTCP/8080(ステージング用ポート)にマッピング

ECS設定

Blue用のサービスをつくってBlue用のターゲットグループにマッピング

Green用のサービスをつくってGreen用のターゲットグループにマッピング

CFnのテンプレートはこんな感じになります。

elb.yml

Parameters: VpcId: Type: String Subnets: Type: List<AWS::EC2::Subnet::Id> Resources: WebSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: web-sg SecurityGroupIngress: - CidrIp: "0.0.0.0/0" IpProtocol: "TCP" FromPort: 80 ToPort: 80 - CidrIp: "0.0.0.0/0" IpProtocol: "TCP" FromPort: 8080 ToPort: 8080 VpcId: !Ref VpcId ### ALBを作成 ### LoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: Name: ecstest-elb Subnets: !Ref Subnets SecurityGroups: - !Ref WebSecurityGroup ### Blue環境用TargetGroup ### BlueTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LoadBalancer Properties: Name: target-blue VpcId: !Ref VpcId Port: 80 Protocol: HTTP ### Green環境用TargetGroup ### GreenTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LoadBalancer Properties: Name: target-green VpcId: !Ref VpcId Port: 80 Protocol: HTTP ### 本番環境用Listner(TCP/80) ListenerProd: Type: AWS::ElasticLoadBalancingV2::Listener Properties: LoadBalancerArn: !Ref LoadBalancer Port: 80 Protocol: HTTP DefaultActions: - Type: forward TargetGroupArn: !Ref BlueTargetGroup # <- 本番にBlue環境を紐付け ### ステージング環境用Listner(TCP/8080) ListenerStg: Type: AWS::ElasticLoadBalancingV2::Listener Properties: LoadBalancerArn: !Ref LoadBalancer Port: 8080 Protocol: HTTP DefaultActions: - Type: forward TargetGroupArn: !Ref GreenTargetGroup # <- ステージングにGreen環境を紐付け

blue.yml

Parameters: Cluster: Type: String BlueTargetGroupARN: Type: String Resources: ### Role作成 ### ECSServiceRole: Type: AWS::IAM::Role Properties: Path: / RoleName: role-blue AssumeRolePolicyDocument: | { "Statement": [{ "Effect": "Allow", "Principal": { "Service": [ "ecs.amazonaws.com" ]}, "Action": [ "sts:AssumeRole" ] }] } ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole ### Blue用サービス ### Service: Type: AWS::ECS::Service Properties: ServiceName: service-blue Cluster: !Ref Cluster Role: !Ref ECSServiceRole DesiredCount: 1 TaskDefinition: !Ref TaskDefinition LoadBalancers: - ContainerName: nginx ContainerPort: 80 TargetGroupArn: !Ref BlueTargetGroupARN #

green.yml

Parameters: Cluster: Type: String GreenTargetGroupARN: Type: String Resources: ### Role作成 ### ECSServiceRole: Type: AWS::IAM::Role Properties: Path: / RoleName: role-green AssumeRolePolicyDocument: | { "Statement": [{ "Effect": "Allow", "Principal": { "Service": [ "ecs.amazonaws.com" ]}, "Action": [ "sts:AssumeRole" ] }] } ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole ### Green用サービス ### Service: Type: AWS::ECS::Service Properties: ServiceName: service-green Cluster: !Ref Cluster Role: !Ref ECSServiceRole DesiredCount: 1 TaskDefinition: !Ref TaskDefinition LoadBalancers: - ContainerName: nginx ContainerPort: 80 TargetGroupArn: !Ref GreenTargetGroupARN #

ELBのリスナーを確認すると、BlueがTCP/80、GreenがTCP/8080になっています。

タスク定義のFamilyの値をBlue/Greenで同じにしておけば、同じタスク定義でRevisionの更新ができます。

デプロイ

デプロイの時は、こんな感じです。

ECS設定

Green用のサービスをアップデートする

ALB設定

ポートのマッピングをBlue/Greenで入れ替える

という手順になります。

CFnでアップデートする場合は、下記のようなテンプレートでUpdate Stackします。

elb.yml

Parameters: VpcId: Type: String Subnets: Type: List<AWS::EC2::Subnet::Id> Resources: WebSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: web-sg SecurityGroupIngress: - CidrIp: "0.0.0.0/0" IpProtocol: "TCP" FromPort: 80 ToPort: 80 - CidrIp: "0.0.0.0/0" IpProtocol: "TCP" FromPort: 8080 ToPort: 8080 VpcId: !Ref VpcId ### ALBを作成 ### LoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: Name: ecstest-elb Subnets: !Ref Subnets SecurityGroups: - !Ref WebSecurityGroup ### Blue環境用TargetGroup ### BlueTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LoadBalancer Properties: Name: target-blue VpcId: !Ref VpcId Port: 80 Protocol: HTTP ### Green環境用TargetGroup ### GreenTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LoadBalancer Properties: Name: target-green VpcId: !Ref VpcId Port: 80 Protocol: HTTP ### 本番環境用Listner(TCP/80) ListenerProd: Type: AWS::ElasticLoadBalancingV2::Listener Properties: LoadBalancerArn: !Ref LoadBalancer Port: 80 Protocol: HTTP DefaultActions: - Type: forward TargetGroupArn: !Ref GreenTargetGroup # <- 本番にGreen環境を紐付け ### ステージング環境用Listner(TCP/8080) ListenerStg: Type: AWS::ElasticLoadBalancingV2::Listener Properties: LoadBalancerArn: !Ref LoadBalancer Port: 8080 Protocol: HTTP DefaultActions: - Type: forward TargetGroupArn: !Ref BlueTargetGroup # <- ステージングにBlue環境を紐付け

リスナーのターゲットグループを入れ替えます。

入れ替わってますね。

まとめ

ALB(or NLB)のターゲットグループを使えばBlue/Green環境が1つのECSクラスタでできるようになります。これでCFnが止まっても本番環境には影響なく安心ですね。　　　

元記事はこちら

「ECSでBlue/Greenデプロイ【cloudpack大阪ブログ】」

October 13, 2017 at 02:00PM

#AWS #Official #Blog

0 notes