#VPCFlowLogs

[govindhtech]

Google VPC Flow Logs: Vital Network Traffic Analysis Tool

GCP VPC Flow Logs

Virtual machine (VM) instances, such as instances utilized as Google Kubernetes Engine nodes, as well as packets transported across VLAN attachments for Cloud Interconnect and Cloud VPN tunnels, are sampled in VPC Flow Logs (Preview).

IP connections are used to aggregate flow logs (5-tuple). Network monitoring, forensics, security analysis, and cost optimization are all possible uses for these data.

Flow logs are viewable via Cloud Logging, and logs can be exported to any location supported by Cloud Logging export.

Use cases

Network monitoring

VPC Flow Logs give you insight into network performance and throughput. You could:

Observe the VPC network.

Diagnose the network.

To comprehend traffic changes, filter the flow records by virtual machines, VLAN attachments, and cloud VPN tunnels.

Recognize traffic increase in order to estimate capacity.

Recognizing network utilization and minimizing network traffic costs

VPC Flow Logs can be used to optimize network traffic costs by analyzing network utilization. The network flows, for instance, can be examined for the following:

Movement between zones and regions

Internet traffic to particular nations

Traffic to other cloud networks and on-premises

Top network talkers, such as cloud VPN tunnels, VLAN attachments, and virtual machines

Forensics of networks

VPC Flow Logs are useful for network forensics. For instance, in the event of an occurrence, you can look at the following:

Whom and when did the IPs speak with?

Analyzing all incoming and outgoing network flows will reveal any hacked IPs.

Specifications

Andromeda, the program that runs VPC networks, includes VPC Flow Logs. VPC Flow Logs don’t slow down or affect performance when they’re enabled.

Legacy networks are not compatible with VPC Flow Logs. You can turn on or off the Cloud VPN tunnel (Preview), VLAN attachment for Cloud Interconnect (Preview), and VPC Flow Logs for each subnet. VPC Flow Logs gathers information from all virtual machine instances, including GKE nodes, inside a subnet if it is enabled for that subnet.

TCP, UDP, ICMP, ESP, and GRE traffic are sampled by VPC Flow Logs. Samples are taken of both inbound and outgoing flows. These flows may occur within Google Cloud or between other networks and Google Cloud. VPC Flow Logs creates a log for a flow if it is sampled and collected. The details outlined in the Record format section are included in every flow record.

The following are some ways that VPC Flow Logs and firewall rules interact:

Prior to egress firewall rules, egress packets are sampled. VPC Flow Logs can sample outgoing packets even if an egress firewall rule blocks them.

Following ingress firewall rules, ingress packets are sampled. VPC Flow Logs do not sample inbound packets that are denied by an ingress firewall rule.

In VPC Flow Logs, you can create only specific logs by using filters.

Multiple network interface virtual machines (VMs) are supported by VPC Flow Logs. For every subnet in every VPC that has a network interface, you must enable VPC Flow Logs.

Intranode visibility for the cluster must be enabled in order to log flows across pods on the same Google Kubernetes Engine (GKE) node.

Cloud Run resources do not report VPC Flow Logs.

Logs collection

Within an aggregation interval, packets are sampled. A single flow log entry contains all of the packets gathered for a specific IP connection during the aggregation interval. After that, this data is routed to logging.

By default, logs are kept in Logging for 30 days. Logs can be exported to a supported destination or a custom retention time can be defined if you wish to keep them longer.

Log sampling and processing

Packets leaving and entering a virtual machine (VM) or passing via a gateway, like a VLAN attachment or Cloud VPN tunnel, are sampled by VPC Flow Logs in order to produce flow logs. Following the steps outlined in this section, VPC Flow Logs processes the flow logs after they are generated.

A primary sampling rate is used by VPC Flow Logs to sample packets. The load on the physical host that is executing the virtual machine or gateway at the moment of sampling determines the primary sampling rate, which is dynamic. As the number of packets increases, so does the likelihood of sampling any one IP connection. Neither the primary sampling rate nor the primary flow log sampling procedure are under your control.

Following their generation, the flow logs are processed by VPC Flow Logs using the steps listed below:

Filtering: You can make sure that only logs that meet predetermined standards are produced. You can filter, for instance, such that only logs for a specific virtual machine (VM) or logs with a specific metadata value are generated, while the rest are ignored. See Log filtering for further details.

Aggregation: To create a flow log entry, data from sampling packets is combined over a defined aggregation interval.

Secondary sampling of flow logs: This is a second method of sampling. Flow log entries are further sampled based on a secondary sampling rate parameter that can be adjusted. The flow logs produced by the first flow log sampling procedure are used for the secondary sample. For instance, VPC Flow Logs will sample all flow logs produced by the primary flow log sampling if the secondary sampling rate is set to 1.0, or 100%.

Metadata: All metadata annotations are removed if this option is turned off. You can indicate that all fields or a specific group of fields are kept if you wish to preserve metadata. See Metadata annotations for further details.

Write to Logging: Cloud Logging receives the last log items.

Note: The way that VPC Flow Logs gathers samples cannot be altered. However, as explained in Enable VPC Flow Logs, you can use the Secondary sampling rate parameter to adjust the secondary flow log sampling. Packet mirroring and third-party software-run collector instances are options if you need to examine every packet.

VPC Flow Logs interpolates from the captured packets to make up for lost packets because it does not capture every packet. This occurs when initial and user-configurable sampling settings cause packets to be lost.

Log record captures can be rather substantial, even though Google Cloud does not capture every packet. By modifying the following log collecting factors, you can strike a compromise between your traffic visibility requirements and storage cost requirements:

Aggregation interval: A single log entry is created by combining sampled packets over a given time period. Five seconds (the default), thirty seconds, one minute, five minutes, ten minutes, or fifteen minutes can be used for this time interval.

Secondary sampling rate:

By default, 50% of log items are retained for virtual machines. This value can be set between 1.0 (100 percent, all log entries are kept) and 0.0 (zero percent, no logs are kept).

By default, all log entries are retained for Cloud VPN tunnels and VLAN attachments. This parameter can be set between 1.0 and greater than 0.0.

The names of the source and destination within Google Cloud or the geographic location of external sources and destinations are examples of metadata annotations that are automatically included to flow log entries. To conserve storage capacity, you can disable metadata annotations or specify just specific annotations.

Filtering: Logs are automatically created for each flow that is sampled. Filters can be set to generate logs that only meet specific criteria.

Read more on Govindhtech.com

[mickmaxtwo]

Say Goodbye to Traffic Blind Spots: VPC Flow Logs & CloudWatch = ‍♀️ Pow...

[fortunatelycoldengineer]

What is a VPC FlowLog? . . . visit: http://bit.ly/3kSF694 for more information

[oom-killer]

2020/06/22-28

# Google、セキュリティスキャナー「Tsunami」をオープンソースで公開。ポートスキャンなどで自動的に脆弱性を検出するツール https://www.publickey1.jp/blog/20/googletsunami.html

# Linuxコマンドの代替コマンド「ncdu」「htop」「tldr」「jq」「fd」とは？ https://news.mynavi.jp/article/20200626-1069025/ >ディスク使用率や監視 du ncdu >システムリソースの表示 top htop >マニュアル表示 man tldr >JSONに対する操作 sed, grep jq >ファイル一覧表示 find fd

# ECS(EC2)のインスタンスタイプ変更は要注意 http://blog.serverworks.co.jp/tech/2020/06/26/post-87643/ >結論を先に記載すると普通のEC2と同じように >停止　⇒　インスタンスタイプの変更　⇒　起動する方法では、 >上手くいかないということです。

# 【AWSセキュリティ】VPCフローログ http://blog.serverworks.co.jp/tech/2020/06/25/vpcflowlogs/ >キャプチャされた内容は Amazon CloudWatch Logs へ Publish(公開) または >Amazon S3に格納する事が出来ます。

>VPC フローログ自体は無料(追加料金なし)となっていますが、使用する場合は >Amazon CloudWatch Logs 側のデータ取り込み料金という形で課金されます。 > >また、当該ログを長期保管したい場合には、S3 bucketへ出力が可能でその >場合は別途 Amazon S3の保管費用が必要となります。

>すべてのIPトラフィックがキャプチャされる訳ではない >AWS内で発生している一部のトラフィック(DHCP,DNS,NTPやWindowsライセンスの >アクティベーション用途のトラフィック等) で記録されないものがあります。

# VPC フローログ https://docs.aws.amazon.com/ja_jp/vpc/latest/userguide/flow-logs.html#flow-logs-limitations >フローログですべての IP トラフィックはキャプチャされません。以下の >トラフィックの種類は記録されません。 > >    Amazon DNS サーバーに接続したときにインスタンスによって生成されるトラフィック。独���の DNS サーバーを使用する場合は、その DNS サーバーへのすべてのトラフィックが記録されます。 > >    Amazon Windows ライセンスのアクティベーション用に >    Windows インスタンスによって生成されたトラフィック。 > >    インスタンスメタデータ用に 169.254.169.254 との間を >    行き来するトラフィック。 > >    Amazon Time Sync Service の 169.254.169.123 との間で >    やり取りされるトラフィック。 > >    DHCP トラフィック。 > >    デフォルト VPC ルーターの予約済み IP アドレスへのトラフィック。 > >    エンドポイントのネットワークインターフェイスと >    Network Load Balancer のネットワークインターフェイスの間の >    トラフィック。

# AWS Lambda から Amazon Elastic File System にアクセスできるようになりました！！！ http://blog.serverworks.co.jp/tech/2020/06/22/aws-lambda-support-for-amazon-elastic-file-system-now-generally/ >AWS Lambda では、関数実行時の一時的なファイルストレージとして >512 MB の /tmp 領域 を利用できます。 >ただ、512 MB という制限からこれまで GB 単位で容量を使用する機械学習の >モデル計算などは AWS Lambda で扱うことができませんでした。 >今回のアップデートで Amazon EFS が AWS Lambda から利用できるように >なったので、 /tmp ストレージの制限で断念していたケースでも >AWS Lambda を利用できるようになります。

# Switch Roleで何をしているのかざっくり説明する https://dev.classmethod.jp/articles/switch-role-for-primary-of-aws/ >一言で言うとすれば、「AWSを見る/触るときの権限を別のIAM Roleに >切り替えること」です。

# CloudWatch Alarm をテストしておきたい時の方法 https://dev.classmethod.jp/articles/tsnote-cloudwatch-alarm-test-001/ > 1. 閾値を本来と異なるすぐに Alarm が実行される値へ変更を行う > 2. AWS CLI cloudwatch set-alarm-state を利用してステータスを >    強制的に変更し Alarm 実行させる