#Filebeat

[codezup]

Migrating from Logstash to Filebeat for Log Data Ingestion

Introduction Migrating from Logstash to Filebeat for log data ingestion is a crucial step in optimizing your logging infrastructure. Logstash, a popular log aggregation and processing tool, has been widely used for years, but it has its limitations. Filebeat, on the other hand, is a lightweight, efficient log shipper that can handle large volumes of log data. In this tutorial, we will guide you…

[suncloudvn]

Hướng dẫn triển khai Docker Graylog theo các bước chi tiết

Tài liệu để build Graylog được tôi sử dụng và tham khảo ở đây. Điều tôi làm chỉ là tận dụng cấu hình của họ và sửa lại để cho phù hợp với mục đích của mình. Lưu ý cấu hình mình đang sử dụng là 8 Cpus và 12 Gb Ram. Trong bài viết này, chúng tôi sẽ hướng dẫn bạn cách triển khai Graylog thông qua Docker để bắt đầu thu thập logs ngay lập tức.

1. Mô hình sử dụng

Ở mô hình này tôi sử dụng 3 container Graylog, opensearch, mongodb chúng liên lạc với nhau qua network : Graylog_net

Riêng container Graylog sử dụng expose port 9000:9000 để dùng truy cập trang web qua IP của host và các port khác dùng để nhận log các dịch vụ khác

"5044:5044" # Cổng cho nhận log từ Filebeat

"5140:5140" # Cổng cho nhận log từ syslog

"12201:12201" # Cổng cho nhận log từ GELF UDP

"13301:13301" # Cổng tùy chỉnh (thay thế cho dịch vụ khác)

"13302:13302" # Cổng tùy chỉnh khác

2. Cài đặt Docker Graylog

Đầu tiên sẽ tải xuống repo Docker github của mình

cd /opt/

git clone https://github.com/thanhquang99/Docker

Tiếp theo ta cần chạy file Docker compose

cd /opt/Docker/Graylog/

Docker compose up

Ta có thể tùy chỉnh biến trong file Docker compose để thay đổi user và password của Graylog hay opensearch. Nếu không thay đổi thì password mặc định của Graylog: minhtenlaquang

Bạn cũng cần sử lại cấu hình Graylog và opensearch sử dụng ram và cpu để phù hợp với máy của bạn. Thông thường opensearch sẽ chiếm 50% RAM và Graylog chiếm 25% RAM

Đợi 1 thời gian cho đến khi Docker compose chạy xong ta sẽ vào trang http://<ip-Docker-host>:9000. Với user: admin, password: minhtenlaquang

3. Tùy chỉnh tài nguyên sử dụng mà Graylog sử dụng

Các biến Graylog mà bạn cần lưu ý để có thể chỉnh sửa cho phù hợp với tài nguyên Graylog của mình: 

processbuffer_processors: Số lượng bộ xử lý cho buffer xử lý.

outputbuffer_processors: Số lượng bộ xử lý cho buffer đầu ra (Elasticsearch).

processor_wait_strategy: Chiến lược chờ của bộ xử lý khi không có công việc để làm (yielding, sleeping, blocking, busy_spinning).

ring_size: Kích thước của ring buffer.

message_journal_enabled: Kích hoạt hoặc vô hiệu hóa message journal.

message_journal_max_size: Kích thước tối đa của message journal.

inputbuffer_processors: Số lượng bộ xử lý cho input buffer.

inputbuffer_ring_size: Kích thước của ring buffer cho input buffer.

retention_strategy: Chiến lược giữ lại dữ liệu (ví dụ: delete, archive).

rotation_strategy: Chiến lược xoay vòng chỉ mục (ví dụ: count, time).

retention_max_index_count: Số lượng chỉ mục tối đa được giữ lại.

rotation_max_index_size: Kích thước tối đa của chỉ mục trước khi xoay vòng.

rotation_max_index_age: Tuổi thọ tối đa của chỉ mục trước khi xoay vòng.

tcp_recv_buffer_size: Kích thước bộ đệm nhận TCP.

tcp_send_buffer_size: Kích thước bộ đệm gửi TCP.

discarders: Cấu hình số lượng và loại discarder để xử lý tin nhắn vượt quá giới hạn.

threadpool_size: Số lượng luồng trong pool của Graylog.

Tôi sẽ hướng dẫn bạn tùy chỉnh biến message_journal_max_size để test thử.

Ta cần xem lại thông tin các volume của Graylog

Docker inspect graylog

Ta sẽ sửa file

vi /var/lib/docker/volumes/graylog_graylog_data/_data/graylog.conf

Restart lại Graylog

docker restart graylog

Kiểm tra kết quả:

Kết Luận

Hy vọng bài viết này đã giúp bạn triển khai Graylog sử dụng Docker và áp dụng vào hệ thống của mình. Docker Graylog là cách triển khai Graylog, một nền tảng quản lý và phân tích log bằng Docker. Điều này giúp dễ dàng thiết lập, cấu hình và quản lý Graylog trong các container, đảm bảo tính linh hoạt, khả năng mở rộng và đơn giản hóa quy trình cài đặt. Docker Graylog thường đi kèm với các container bổ sung như MongoDB (lưu trữ dữ liệu cấu hình) và Elasticsearch (xử lý và lưu trữ log).

Nguồn: https://suncloud.vn/huong-dan-trien-khai-docker-graylog-theo-cac-buoc-chi-tiet

[leeilly-blog1]

filebeat & kafka

새로 올린 배치 서버는 aws ec2 인스턴스에 올라가 있고, 그 인스턴스에 docker로 filebeat을 띄어놓았다. 그리고 기존 상용 서버에는 docker가 아닌, 그냥 데몬으로 filebeat을 실행해놓았다. 

일단 docker로 올린 filebeat은 임의로 종료되는 케이스가 아직까지 없다. 반면 데몬으로 돌고 있던 filebeat은, 퇴근 전 실행중인것을 확인하고 다음날 출근해서 확인하면 죽어있다. 

-> 이것도 확인해봐야 하는 문제 

그리고, 다시 filebeat을 띄우면, 그 동안 수확하지 못하고 쌓여있던 로그들을 긁어서 조금 많은 양의 데이터를 kafka로 보내기 때문인지, filebeat은 아래와 같은 로그 메시지를 찍는다.

2019-11-21T10:32:37.813+0900 INFO kafka/log.go:53 producer/broker/1 maximum request accumulated, waiting for space

정확히 어떤 의미일까? 구글링 구글링.

일단, 에러는 아님.

This is not an exception but an info message. kafka client's buffer is full, and it's probably waiting for some ACKs before sending more messages.

filebeat의 송신 버퍼가 full이라는 뜻.

The message is logged because the send buffers are full. The kafka client is waiting for ACK from kafka for an older batch in order to flush it's buffers. So network of kafka itself might create some backpressure. All in all data will not be lost, as filebeat will retry until there is enough space. 

내 설정을 확인해 보자.

1. producer인, filebeat 설정.

output.kafka:    ...    required_acks: 1    compression: gzip    max_message_bytes: 1000000

2. kafka broker 설정. (server.properties)

num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600

어떻게 조정해야할까?

[facedrita]

Logstash listening to filebeats for different log type

LOGSTASH LISTENING TO FILEBEATS FOR DIFFERENT LOG TYPE CODE

For example we are creating a file calledĬonfigure logstash input to listen to filebeat on port 5044īy default, redisearch is listening on port 6379

Configure the logstash pipeline by creating a file.

# setup filebeat to send output to logstash – Enable filebeat input to read logs from the specified path and change the output from Elasticsearch to logstash.

Configure file: /etc/filebeat/filebeat.yml.

RediSearch has an output plugin to store all incoming logs from the input plugin.

Filebeat has an input plugin to collect logs from various sources.

Logstash plugins we use in the example are:.

Let’s see some examples of the usage and configuration of RediSearch in the logstash pipeline. This logstash output plugin helps receive log messages coming from logstash and stashes them into RediSearch. In order to store logs into RediSearch, a builtin plugin Logstash output plugin for RediSearch was created.

The output stage initiates the process to send data to a particular destination.

The filter stage is mandatory in order to perform intermediary processing on data.

Input is the stage that helps get data into logstash.

Logstash data processing pipeline has three stages namely Input, Filter, and Output. RediSearch is powerful yet simple to manage and maintain and is efficient enough to serve as a standalone database or augment existing Redis databases with advanced, powerful indexing capabilities.

LOGSTASH LISTENING TO FILEBEATS FOR DIFFERENT LOG TYPE CODE

It is possible because of Redis’ robust in-memory architecture based on modern data structures and optimal code execution written in “C” language, whereas Elasticsearch is based on the Lucene engine and is written in Java programming language. RediSearch is faster in performance compared to any other search engine like Elasticsearch or Solr. RediSearch is also a full-text search and aggregation engine, built as a module on top of Redis. Elasticsearch is a distributed full-text search engine that possesses the capability to highly refine analytics capabilities. Elasticsearch is often used as a data store for logs processed with Logstash. Logstash is a data processing pipeline that allows you to collect data from different sources, parse on its way, and deliver it to your most likely destination for future use. Logstash is the most powerful tool from the elastic stack which is used for log management. This can be easily achieved by using an elastic stack. To make it faster, centralized logging is said to very helpful, and gives us the opportunity to aggregate logs from various applications to a central place and perform search queries against logs. It becomes tedious to manage all the logs as the number of applications increases. Rename => Ĭonfigure your data source integration to have different log types.Logs have been an essential part of troubleshooting application and infrastructure performance ever since its existence. For example, you could create the mylog.type field and then transform that field to iis.logs. You can also query your log fields to check the log type if you have created the field in your log. Using log fields to distinguish log types For example, we may want to change the index name the logs appear under in Elasticsearch and Kibana. You can use Logstash to query log types in your Logstash filter and then perform actions based upon that condition. You can access your Logstash filters from the dashboard for any of your Logit Stacks by choosing View Stack Settings > Logstash Pipelines. To further differentiate between the log types, we need to make use of the Logstash Filter. Using Logstash to further differentiate between log types From here we can then use Logstash to further differentiate between these log types. In order to tell the difference between the logs that are coming from these folders, we have added logType1 to one set of logs and logType2 to the other set of logs. In the above example we have two folders that contain logs. If you are using an Elastic Beat source such as Auditbeat, Filebeat or Metricbeat you can have multiple inputs sections in your configuration file to distinguish between different types of logs by editing the Beat configuration file and setting the type to be named differently.įor the example, below we are editing the Filebeat configuration file to separate our logs into different types. How do I separate my logs into different log types?ĭifferentiating between different log types in Logstash can be achieved in various ways. If you are collecting two sets of logs using the same Elastic beat source you may want to separate them so that you can perform certain actions on them if they meet certain conditions.įor example, you may want to change the index name of one log type to help make it more identifiable. Why would I want to differentiate between different log types?

[jamescarlos]

Filebeat: Lightweight Log Analysis & Elasticsearch | Elastic

http://bit.ly/2XOVd7x

Lightweight Shipper for Logs

[yoonyoul-blog]

ELK 정리

ELK (Elasticsearch, Logstash, Kibana)

ELK (Elasticsearch, Logstash, Kibana)를 통해 애플리케이션 혹은 시스템의 로그들을 수집하여 검색 및 분석을 할 수 있는 환경을 구성할 수 있다.

Logstash

파일, TCP, HTTP 등의 다양한 입력 소스를 통해들어 오는 데이터를 수집할 수 있는 파이프라인을 제공한다. 들어오는 데이터를 가공할 수 있으며, Elasticsearch로 데이터를 전송할 수 있다.

설치

사전 조건

Logstash 설치를 위해선 Java 8이 설치되어 있어야 한다. Java 8설치 유무를 확인하기 위해 아래의 명령어를 실행한다. 만약 Java 8이 설치 되어 있지 않다면, Java 8 다운로드페이지를 통해 다운로드 하여 설치한다.

java version 확인

java -version

java version 확인 결과

java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

(*위의 Java version 값은 로컬 환경에 설치된 Java에 따라 다를 수 있다.)

설치 및 설정

Logstash 설치를 위한 설치 파일을 직접 다운로드할 수 있다.

Binary 파일을 통한 설치

윈도우 환경에 설치를 진행하기 위한 ZIP 파일을 다운로드하여 설치하고자 하는 경로에 압축을 풀면 된다.

리눅스 환경

유닉스 환경

맥 환경

Homebrew를 통해 Logstash를 설치 할 수 있다. 설치를 위해 아래의 명령어를 실행한다.

brew install logstash

구성

(* 출처: www.elastic.co)

위의 이미지에서 볼 수 있듯이 Logstash는 기본적으로 input, filter 그리고 out이라는 구성요소를 가지고 있다. Input과 output은 필수 요소이고 filter는 필요에 따라 구성할 수 있다. 아래의 명령어를 통해 콘솔을 통한 입/출력 하는 가장 기본적인 구성으로 Logstash를 실행할 수 있다.

bin/logstash -e 'input {stdin { } } output { stdout { } } '

(* 위에서 사용은 -e 옵션은 Logstash의 설정을 명령어를 통해 직접 입력 할 수 있게 해준다.)

위의 명령어를 실행한 후, 명령어 창에 hello world 를 입력하면 입력한 hello world 가 바로 출력되는 것을 볼 수 있다.

기본 설정 구성

위에서 언급한 것처럼 Logstash 설정 파일은 input, filter 그리고 output으로 구성이 되어 있다. 아래는 템플릿으로 사용 가능한 설정 파일 내용이다.

# The # character at the beginning of a line indicates a comment. Use # comments to describe your configuration. input { } # The filter part of this file is commented out to indicate that it is # optional. # filter { # # } output { }

Input

File

Logstash가 설치된 로컬 환경에 존재하는 파일의 내용을 input으로 설정하여 처리할 수 있다. 아래는 로컬 파일을 처리하기 위한 설정의 예이다.

input { file { path => "/tmp/access_log" start_position => "beginning" } }

FileBeat를 통한 File 처리

Filebeat를 통해 pipeline을 구축할 수 있다. 다만 Beats input plugin이 먼저 설치되어 있어야 한다. (* Beats input plugin은 Logstash 설치 시 기본으로 함께 설치된다. )

Note: 일반적으로 Filebeat는 Logstash가 설치된 machine과는 다른 machine에 설치하여 실행한다.

Filebeat 설치 및 설정은 Filebeat를 참조하면 된다.

Filebeat 설치 후, 아래와 같은 설정을 통해 Logstash와 연동 시킬 수 있다.

Filebeat 설정

filebeat.prospectors: - type: log paths: - /path/to/file/logstash-tutorial.log output.logstash: hosts: ["localhost:5044"]

Logstash Filebeat input 설정

아래의 내용은 Logstash에서 Filebeat의 내용을 input으로 설정하는 것으로 Beats input plugin을 사용한다.

input { beats { port => "5044" } } filter { json { source => "message" target => "message" } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["localhost:9200"] index => "application-log-%{+YYYY.MM.dd}" workers => 1 user => elastic password => changeme } }

filter 설정

filter { json { source => "message" target => "message" } }

위와 같은 filter 설정에 대해 좀 더 설명하면, 애플리케이션에서 남기는 로그 형태가 JSON 형태일때 Filebeat는 해당 로그를 JSON으로 인식하여 읽어 들이지 않는다. 즉, 일반 문자열로 읽어 들이기 때문에 로그 내용이 아래와 같을 경우 큰 따옴표(")를 escaping 처리한다. 따라서 Logstash의 input으로 해당 로그 내용이 전달이 되면 json filter를 통해 정상적인 JSON 형식으로 만들어 주어야 한다.

애플리케이션에서 남긴 로그 예제

... "{ "method ": "GET ", "userAgent ": { "operatingSystem ": "MAC_OS_X ", "browser ": "SAFARI " }, "@timestamp ": "2017 - 11 - 22 T18: 40: 41.639 + 09: 00 ", "instanceId ": "localhost" }" ...

Filebeat가 읽어 들인 로그 예제

... "{ \"method \":\"GET \", \"userAgent \": { \"operatingSystem \":\"MAC_OS_X \", \"browser \":\"SAFARI \" }, \"@timestamp \":\"2017 - 11 - 22 T18: 40: 41.639 + 09: 00 \", \"instanceId \":\"localhost\" }" ...

Logstash json filter 처리 후 내용 예제

... "message" => { "instanceId" => "localhost", "method" => "GET", "userAgent" => { "operatingSystem" => "MAC_OS_X", "browser" => "SAFARI" } } ...

위와 같은 설정 방법이 아닌 다른 방법으로는 Filebeat의 processor를 사용하면 된다. 아래는 Filebeat에서 읽어 들인 내용을 JSON으로 처리하기 위한 예제 설정 내용이다. 아래의 설정에 추가적으로 사용된 json 옵션을 살펴보면 된다.

filebeat.prospectors: - type: log paths: - /path/to/file/logstash-tutorial.log json.keys_under_root: true json.overwrite_keys: true processors: -decode_json_fields: fields: ["message"] output: logstash: hosts: ["localhost:5044"]

TCP, UDP

Logstash는 TCP 혹은 UDP을 input으로 사용할 수 있다. 아래는 TCP, UDP를 사용한 예제 설정이며 Logstash는 9600 포트를 통해 TCP, UDP input을 처리하도록 설정된다.

tcp { port => 9600 type => syslog } udp { port => 9600 type => syslog }

input data의 형식이 JSON 형식일 경우 아래와 같이 codec을 추가하여 준다.

tcp { port => 9600 type => syslog codec => json { charset => "UTF-8" } } udp { port => 9600 type => syslog codec => json { charset => "UTF-8" } }

HTTP

Logstash의 http plugin을 사용하여 HTTP(S)를 통해 단일 혹은 multiline을 입력 받을 수 있도록 할 수 있다. Http를 통해 들어 오는 내용의 Content-Type에 해당 하는 codec이 사용된다. 예를 들어 Content-Type이 application/json일 경우, json codec이 사용된다. 반면 다른 Content-Type에 대해선 plain codec이 사용된다.

HTTP의 기본 인증 표준을 지원하며, SSL 설정에 따라 https를 통해 들어 오는 데이터를 처리할 수 있다. 인증서 설정은 Java Keystore format을 통해 가능하다.

사용 가능한 옵션들은 Http Input Configuration Options를 참고하면 된다.

간단한 설정 예제는 아래와 같다.

input { http { host => "localhost" port => 9600 user => "elastic" password => "changeme" } }

Filter

Logstash의 pipeline을 통해 읽어 들인 로그에서 원하는 정보를 뽑아내거나, 해당 로그를 원하는 형태로 만들기 위해선 filter를 사용해야 한다. Logstash는 다양한 filter plugin을 지원하면 이중 하나인 grok filter를 살펴본다.

Grok filter를 통해 들어오는 로그 데이터의 일정한 패턴을 분석하여 구조화된 그리고 검색 가능한 데이터로 변경할 수 있다. Grok filter는 기본적으로 정규식을 사용하여 표현 할 수 있다. Logstash는 다양한 filter pattern을 기본 제공하고 있으며, https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns 이곳에서 원하는 filter를 검색하여 사용할 수 있다.

Grok Basics

Grok pattern의 사용 문법은 %{SYNTAX:SEMANTIC} 이며, SYNTAX는 로그의 내용 중 뽑아내고자 하는 부분의 패턴을 의미한다. 예를 들어 로그 내용 중 3.44라는 숫자 값을 뽑아낼려면 NUMBER pattern을 사용하면 된다. 그리고 SEMANTIC은 SYNTAX를 통해 일치된 부분을 식별하기 위한 식별자를 의미한다. 예를 들어 이미 언급한 NUMBER pattern을 통해 뽑아낸 3.44가 duration을 의미한다면 SEMANTIC에 duration을 설정하면 된다.

%{NUMBER: duration}

정규식 표현

Grok에서 사용하는 정규식 표현 라이브러리는 Oniguruma로 지원 가능한 상세 표현식은 Oniguruma Site에서 확인 가능하다.

사용자 정의 패턴

Logstash가 제공하지 않는 패턴을 사용하기 위해 아래와 같이 직접 패턴을 정의할 수 있다.

(?<queue_id>[0-9A-F]{10,11})

또한 자주 사용하는 패턴의 경우 다음의 단계를 통해 저장해서 재 사용 할 수 있다. * patterns라는 이름으로 폴더를 생성한다. * 사용할 pattern을 포함한 파일명을 지정하여 저장한다. * 해당 파일에 사용할 패턴을 작성하여 저장한다.

# contents of ./patterns/postfix: POSTFIX_QUEUEID [0-9A-F]{10,11}

위의 패턴을 사용한 예제는 아래와 같다.

로그 내용

Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>

filter 설정 내용

filter { grok { patterns_dir => ["./patterns"] match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}" } } }

결과 항목

timestamp: Jan 1 06:25:43

logsource: mailserver14

program: postfix/cleanup

pid: 21403

queue_id: BEF25A72965

syslog_message: message-id=20130101142543.5828399CCAF@mailserver14.example.com

Grok Filter Configuration Options

Grok filter 설정 시 사용 가능한 상세 옵션은 상세 옵션을 통해 확인 가능하다.

Output

//TODO

Elasticsearch

//TODO

Kibana

//TODO

Filebeat

Filebeat는 파일에 담겨 있는 로그 내용을 수집하여 원격의 Logstash로 전송하기 위한 도구 이다. Logstash를 설치하게 되면 Filebeat와 연동 가능한 Beats input plugin이 기본적으로 함께 설치되어 있다. Logstash는 Beats input plugin를 통해 Elastic Beats framework으로부터의 이벤트를 수신하게 된다.

설치

윈도우에 설치

윈도우에 Filebeat를 설치하기 위해선 다운로드 페이지에서 ZIP 파일을 다운 받는다.

다운로드한 압축 파일을 C:\Program Files 하위에 압축을 풀어 준다.

폴더 명을 Filebeat로 변경한다.

PowerShell을 관리자 권한으로 실행한다.

Filebeat를 윈도우 서비스로 실행하기 위해 아래의 명령어를 PowerShell창에 입력하고 실행한다.

PS > cd 'C:\Program Files\Filebeat' PS C:\Program Files\Filebeat> .\install-service-filebeat.ps1

윈도우이외의 환경에 설치

deb

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-amd64.deb sudo dpkg -i filebeat-6.0.0-amd64.deb

rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm sudo rpm -vi filebeat-6.0.0-x86_64.rpm

mac

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-darwin-x86_64.tar.gz tar xzvf filebeat-6.0.0-darwin-x86_64.tar.gz

(* homebrew를 통해 Filebeat를 설치 할 경우, /usr/local/Cellar/filebeat/6.0.0/.bottle/etc/filebeat 밑dp filebeat.reference.yml 파일과 filebeat.yml 파일이 존재한다.)

docker

docker pull docker.elastic.co/beats/filebeat:6.0.0

설정

Filebeat는 filebeat.yml파일을 통해 설정 내용을 관리한다. Filebeat는 참조용 설정 파일인 filebeat.reference.yml을 제공한다. 각 환경별 filebeat.yml의 위치는 아래와 같다. * rpm, deb : /etc/filebeat/filebeat.yml * Docker: /usr/share/filebeat/filebeat.yml * Mac, Windows: 압축 푼 경로내에 위치.

샘플 설정 파일

filebeat.prospectors: - type: log enabled: true paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\*

Filebeat 설정은 다음과 같은 단계를 거쳐 진행하면 된다.

로그 파일의 위치를 지정한다.

Filebeat 설정의 가장 기본적인 단계로 하나의 prospector를 다음과 같이 정의 할 수 있다.

filebeat.prospectores: - type: log enabled: true paths: - /var/log/*.log output.logstash: hosts: ["localhost:5044"]

위에 설정한 내용처럼 설정된 prospector는 /var/log 하위의 모든 log 확장자를 가진 파일의 내용을 수집하게 된다. 단, 현재는 디렉토리의 모든 하위 디렉토리에있는 모든 파일을 재귀 적으로 가져올 수 없다.

output 설정

Logstash로 보내기

아래와 같이 hosts 옵션을 통해 다수의 Logstash 서버를 지정할 수 있으며 Beats connection을 위해 사용되는 5044 포트를 설정한다. 아래와 같이 설정하여 운영하기 위해선 Elasticsearch에 직접 index template을 생성해야 한다.

output.logstash: hosts: ["localhost:5044"]

Elasticsearch로 바로 보내기

Filebeat가 수집한 내용을 Elasticsearch로 바로 보낼 수 있는데, 이를 위한 설정은 아래와 같다. Elasticsearch에 보안 목적의 사용자 계정과 비밀번호가 설정되어 있는 경우에는 username, password를 꼭 설정해야 한다.

output.elasticsearch: hosts: ["localhost:9200"] username: elastic password: changeme

Kibana에 dashboard 설정

Kibana에 Filebeat가 제공하는 샘플 dashboard를 다음과 같이 설정할 수 있다.

setup.kibana: host: "localhost:5601" username: elastic password: changeme

기타 사항

Logstash의 geoip plugin 사용 시, client의 ip가 127.0.0.1인 경우, _geoip_lookup_failure가 발생한다.

참고 자료

Filebeat

filebeat-reference-yml

[postwell]

9 essential skills for AWS DevOps Engineers

AWS DevOps engineers cover a lot of ground. The good ones maintain a cross-disciplinary skill set that touches upon a cloud, development, operations, continuous delivery, data, security, and more.  

Here are the skills that AWS DevOps Engineers need to master to rock their role.

1. Continuous delivery

For this role, you’ll need a deep understanding of continuous delivery (CD) theory, concepts, and real-world application. You’ll not only need experience with CD tools and systems, but you’ll need intimate knowledge of their inner workings so you can integrate different tools and systems together to create fully functioning, cohesive delivery pipelines. Committing, merging, building, testing, packaging, and deploying code all come into play within the software release process. 

If you’re using the native AWS services for your continuous delivery pipelines, you’ll need to be familiar with AWS CodeDeploy, AWS CodeBuild, and AWS CodePipeline. Other CD tools and systems you might need to be familiar with include GitHub, Jenkins, GitLab, Spinnaker, Travis, or others.

2. Cloud

An AWS DevOps engineer is expected to be a subject matter expert on AWS services, tools, and best practices. Product development teams will come to you with questions on various services and ask for recommendations on what service to use and when. As such, you should have a well-rounded understanding of the varied and numerous AWS services, their limitations, and alternate (non-AWS) solutions that might serve better in particular situations. 

With your expertise in cloud computing, you’ll architect and build cloud-native systems, wrangle cloud systems’ complexity, and ensure that best practices are followed when utilizing a wide variety of cloud service offerings. When designing and recommending solutions, you’ll also weigh the pros and cons of using IaaS services versus PaaS and other managed services. If you want to go beyond this blog and master the skill, you must definitely visit AWS DevOps Course and get certified!

3. Observability

Logging, monitoring, and alerting, oh my! Shipping a new application to production is great, but it’s even better if you know what it’s doing. Observability is a critical area of work for this role. An AWS DevOps engineer should ensure that an application and its systems implement appropriate monitoring, logging, and alerting solutions. APM (Application Performance Monitoring) can help unveil critical insights into an application’s inner workings and simplify debugging custom code. APM solutions include New Relic, AppDynamics, Dynatrace, and others. On the AWS side, you should have deep knowledge of Amazon CloudWatch (including CloudWatch Agent, CloudWatch Logs, CloudWatch Alarms, and CloudWatch Events), AWS X-Ray, Amazon SNS, Amazon Elasticsearch Service, and Kibana. You might utilize tools and systems in this space, including Syslog, logrotate, Logstash, Filebeat, Nagios, InfluxDB, Prometheus, and Grafana. 

4. Infrastructure as code

An AWS DevOps Engineer will ensure that the systems under her purview are built repeatedly, using Infrastructure as Code (IaC) tools such as CloudFormation, Terraform, Pulumi, and AWS CDK (Cloud Development Kit). Using IaC ensures that cloud objects are documented as code, version controlled, and can be reliably replaced using an appropriate IaC provisioning tool.

5. Configuration Management

On the IaaS (Infrastructure as a Service) side for virtual machines, once ec2 instances have been launched, their configuration and setup should be codified with a Configuration Management tool. Some of the more popular options in this space include Ansible, Chef, Puppet, and SaltStack. For organizations with most of their infrastructure running Windows, you might find Powershell Desired State Configuration (DSC) as the tool of choice in this space.

6. Containers

Many modern organizations are migrating away from the traditional deployment models of apps being pushed to VMs and to a containerized system landscape. In the containerized world, configuration management becomes much less important, but there is also a whole new world of container-related tools that you’ll need to be familiar with. These tools include Docker Engine, Docker Swarm, systemd-nspawn, LXC, container registries, Kubernetes (which includes dozens of tools, apps, and services within its ecosystem), and many more.

7. Operations

IT operations are most often associated with logging, monitoring, and alerting. You need to have these things in place to properly operate, run, or manage production systems. We covered these in our observability section above. Another large facet of the Ops role is responding to, troubleshooting, and resolving issues as they occur. To effectively respond to issues and resolve them quickly, you’ll need to have experience working with and troubleshooting operating systems like Ubuntu, CentOS, Amazon Linux, RedHat Enterprise Linux, and Windows. You’ll also need to be familiar with common middleware software like web servers (Apache, Nginx, Tomcat, Nodejs, and others), load balancers, and other application environments and runtimes.

Database administration can also be an important function of a (Dev)Ops role. To be successful here, you’ll need to have knowledge of data stores such as PostgreSQL and MySQL. You should also be able to read and write some SQL code. And increasingly, you should be familiar with NoSQL data stores like Cassandra, MongoDB, AWS DynamoDB, and possibly even a graph database or two!

8. Automation

Eliminating toil is the ethos of the site reliability engineer, and this mission is very much applicable to the DevOps engineer role. In your quest to automate everything, you’ll need experience and expertise with scripting languages such as bash, GNU utilities, Python, JavaScript, and PowerShell for the Windows side. You should be familiar with cron, AWS Lambda (the service of the serverless function), CloudWatch Events, SNS, and others.  

9. Collaboration and communication

Last (but not least) is the cultural aspect of DevOps. While the term “DevOps” can mean a dozen different things to a dozen people, one of the best starting points for talking about this shift in our industry is CAMS: culture, automation, measurement, and sharing. DevOps is all about breaking down barriers between IT operations and development. In this modern DevOps age, we no longer have developers throwing code “over the wall” to operations. We now strive to be one big happy family, with every role invested in the success of the code, the applications, and the value delivered to customers. This means that (Dev)Ops engineers must work closely with software engineers. This necessitates excellent communication and collaboration skills for any person who wishes to fill this keystone role of a DevOps engineer. 

AWS Devops?

AWS offers various flexible services that allow organizations to create and release services more efficiently and reliably through the implementation of DevOps techniques.

These services make the provisioning and management of infrastructures, such as deploying application code and automating the release process for software, and monitoring your application's and infrastructure's performance.

[codezup]

Streamline Log Management with Elasticsearch and Filebeat

Simplifying Log Analysis with Elasticsearch and Filebeat Introduction Log analysis is an essential part of any modern software system. It allows developers and operators to understand how their applications are used, identify bottlenecks, and diagnose issues. However, traditional log analysis methods can be time-consuming and resource-intensive. In this tutorial, we’ll show you how to simplify…

[suncloudvn]

Hướng dẫn các bước đẩy Log từ Client lên Graylog chi tiết

Graylog là một nền tảng mạnh mẽ để thu thập, phân tích và quản lý log. Việc đẩy log từ các nguồn khác nhau lên Graylog giúp bạn tập trung hóa việc quản lý log, dễ dàng giám sát hệ thống và phát hiện các sự cố kịp thời. Bài viết này sẽ hướng dẫn bạn cách đẩy log từ Client lên Graylog.

1. Tìm hiểu về Input trong Graylog

Input trong Graylog là các điểm nhập dữ liệu, nơi Graylog nhận và xử lý log messages từ các nguồn khác nhau. Hiểu rõ về các loại Input là điều cần thiết để có thể cấu hình Graylog nhận log từ các hệ thống và thiết bị khác nhau. Dưới đây là chi tiết về các loại Input phổ biến và cách cấu hình chúng.

Dưới đây là một số loại Input phổ biến:

Syslog: Sử dụng để nhận log từ các thiết bị và ứng dụng hỗ trợ giao thức Syslog.

GELF (Graylog Extended Log Format): Một định dạng log mở rộng của Graylog, hỗ trợ cấu trúc dữ liệu JSON.

Beats: Sử dụng để nhận log từ Filebeat, Metricbeat và các Beats khác của Elastic.

Raw/Plaintext: Nhận log dưới dạng văn bản thô hoặc định dạng tùy chỉnh.

JSON Path: Sử dụng để nhận log có định dạng JSON, giúp trích xuất và phân tích dữ liệu JSON.

HTTP: Nhận log qua HTTP/HTTPS, cho phép gửi log từ các ứng dụng web và dịch vụ hỗ trợ HTTP.

TCP/UDP: Nhận log qua giao thức TCP hoặc UDP, thường dùng cho các ứng dụng và thiết bị không hỗ trợ Syslog nhưng có khả năng gửi log qua mạng.

AMQP: Sử dụng để nhận log từ các hàng đợi tin nhắn (message queues) hỗ trợ giao thức AMQP.

Kafka: Nhận log từ Apache Kafka, một nền tảng stream-processing phổ biến.

2. Mô hình đẩy log từ Client lên Graylog

Chúng tôi có một mô hình cơ bản giúp bạn hiểu rõ hơn về cách đẩy log từ Client lên Graylog.

IP Planning

Tên máy

IP

Graylog

172.16.66.55

Linux

172.16.66.54

vCenter

172.16.66.46

3. Đẩy log từ vCenter sang Graylog

3.1 Cấu hình trên vCenter

Ta cần vào trang VAMI của vCenter để tiến hành mở dịch vụ rsyslog và tiến hành đẩy log sang. Để truy cập vào trang VAMI ta thực hiện theo đường dẫn sau.

Tiếp theo ta chọn vào Syslog và tiến hành cấu hình đẩy log. Chúng ta cần khai báo IP của Graylog và giao thức đẩy log là UDP.

3.2 Cấu hình trên Graylog

Việc đầu tiên để Graylog có thể nhận được log thì ta phải cấu hình firewall mở port 514/UDP. Ở đây mình dùng Ubuntu 22.04 nên sẽ sử dụng lệnh sau.

ufw allow 514/udp

Bây giờ ta sẽ lên giao diện Web của Graylog và tiến hành đẩy log. Graylog mình dùng là bản 5.2 nhé.

Ta chọn vào System / Inputs để tiến hành add Input mới. Hãy chọn giao thức là Syslog UDP nhé.

Nội dung cần cấu hình.

Titlle : vCenter ( Tên để bạn gợi nhớ ra Input này)

Bind address : 0.0.0.0 (Lắng nghe trên tất cả các interface của graylog)

Port : 514

Sau khi Launch Input ta sẽ nhận được kết quả như này.

Như vậy là đã cấu hình xong bạn có thể xem log bằng cách nhấn vào Show received messages.

4. Đẩy log từ máy Client Linux bất kỳ lên Graylog

4.1 Cấu hình trên Linux

Việc cấu hình trên Linux thì lại rất đơn giản chúng ta chỉ cần sử dụng đúng 1 câu lệnh.

echo '*.*  @172.16.66.55:1514' >> /etc/rsyslog.conf

Bây giờ ta chỉ cần tiến hành restart lại dịch vụ rsyslog là xong.

systemctl restart rsyslog

4.2 Cấu hình trên Graylog

Cấu hình trên Graylog ta cũng làm tương tự như các bước ở  phần 3 nhưng ta sẽ đổi thành port 1514 và mở firewall đối với port 1514/udp.

Mở firewall.

ufw allow 1514/udp

Cấu hình graylog.

Titlle : netbox-log ( Tên để bạn gợi nhớ ra Input này)

Bind address : 0.0.0.0 (Lắng nghe trên tất cả các interface của graylog)

Port : 1514

Ta kiểm tra xem đã nhận được.

Như vậy là SunCloud đã hướng dẫn các bạn gửi log từ vCenter hay một máy Linux bất kỳ với giao thức UDP. Nếu trong quá trình thực hiện, bạn gặp phải vướng mắc cần được hỗ trợ, hãy liên hệ với chúng tôi để được tư vấn nhanh nhất nhé. Chúc các bạn thành công! 

Nguồn: https://suncloud.vn/client-graylog

[computingpostcom]

The Elastic stack (ELK) is made up of 3 open source components that work together to realize logs collection, analysis, and visualization. The 3 main components are: Elasticsearch – which is the core of the Elastic software. This is a search and analytics engine. Its task in the Elastic stack is to store incoming logs from Logstash and offer the ability to search the logs in real-time Logstash – It is used to collect data, transform logs incoming from multiple sources simultaneously, and sends them to storage. Kibana – This is a graphical tool that offers data visualization. In the Elastic stack, it is used to generate charts and graphs to make sense of the raw data in your database. The Elastic stack can as well be used with Beats. These are lightweight data shippers that allow multiple data sources/indices, and send them to Elasticsearch or Logstash. There are several Beats, each with a distinct role. Filebeat – Its purpose is to forward files and centralize logs usually in either .log or .json format. Metricbeat – It collects metrics from systems and services including CPU, memory usage, and load, as well as other data statistics from network data and process data, before being shipped to either Logstash or Elasticsearch directly. Packetbeat – It supports a collection of network protocols from the application and lower-level protocols, databases, and key-value stores, including HTTP, DNS, Flows, DHCPv4, MySQL, and TLS. It helps identify suspicious network activities. Auditbeat – It is used to collect Linux audit framework data and monitor file integrity, before being shipped to either Logstash or Elasticsearch directly. Heartbeat – It is used for active probing to determine whether services are available. This guide offers a deep illustration of how to run the Elastic stack (ELK) on Docker Containers using Docker Compose. Setup Requirements. For this guide, you need the following. Memory – 1.5 GB and above Docker Engine – version 18.06.0 or newer Docker Compose – version 1.26.0 or newer Install the required packages below: ## On Debian/Ubuntu sudo apt update && sudo apt upgrade sudo apt install curl vim git ## On RHEL/CentOS/RockyLinux 8 sudo yum -y update sudo yum -y install curl vim git ## On Fedora sudo dnf update sudo dnf -y install curl vim git Step 1 – Install Docker and Docker Compose Use the dedicated guide below to install the Docker Engine on your system. How To Install Docker CE on Linux Systems Add your system user to the docker group. sudo usermod -aG docker $USER newgrp docker Start and enable the Docker service. sudo systemctl start docker && sudo systemctl enable docker Now proceed and install Docker Compose with the aid of the below guide: How To Install Docker Compose on Linux Step 2 – Provision the Elastic stack (ELK) Containers. We will begin by cloning the file from Github as below git clone https://github.com/deviantony/docker-elk.git cd docker-elk Open the deployment file for editing: vim docker-compose.yml The Elastic stack deployment file consists of 3 main parts. Elasticsearch – with ports: 9200: Elasticsearch HTTP 9300: Elasticsearch TCP transport Logstash – with ports: 5044: Logstash Beats input 5000: Logstash TCP input 9600: Logstash monitoring API Kibana – with port 5601 In the opened file, you can make the below adjustments: Configure Elasticsearch The configuration file for Elasticsearch is stored in the elasticsearch/config/elasticsearch.yml file. So you can configure the environment by setting the cluster name, network host, and licensing as below elasticsearch: environment: cluster.name: my-cluster xpack.license.self_generated.type: basic To disable paid features, you need to change the xpack.license.self_generated.type setting from trial(the self-generated license gives access only to all the features of an x-pack for 30 days) to basic.

Configure Kibana The configuration file is stored in the kibana/config/kibana.yml file. Here you can specify the environment variables as below. kibana: environment: SERVER_NAME: kibana.example.com JVM tuning Normally, both Elasticsearch and Logstash start with 1/4 of the total host memory allocated to the JVM Heap Size. You can adjust the memory by setting the below options. For Logstash(An example with increased memory to 1GB) logstash: environment: LS_JAVA_OPTS: -Xm1g -Xms1g For Elasticsearch(An example with increased memory to 1GB) elasticsearch: environment: ES_JAVA_OPTS: -Xm1g -Xms1g Configure the Usernames and Passwords. To configure the usernames, passwords, and version, edit the .env file. vim .env Make desired changes for the version, usernames, and passwords. ELASTIC_VERSION= ## Passwords for stack users # # User 'elastic' (built-in) # # Superuser role, full access to cluster management and data indices. # https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html ELASTIC_PASSWORD='StrongPassw0rd1' # User 'logstash_internal' (custom) # # The user Logstash uses to connect and send data to Elasticsearch. # https://www.elastic.co/guide/en/logstash/current/ls-security.html LOGSTASH_INTERNAL_PASSWORD='StrongPassw0rd1' # User 'kibana_system' (built-in) # # The user Kibana uses to connect and communicate with Elasticsearch. # https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html KIBANA_SYSTEM_PASSWORD='StrongPassw0rd1' Source environment: source .env Step 3 – Configure Persistent Volumes. For the Elastic stack to persist data, we need to map the volumes correctly. In the YAML file, we have several volumes to be mapped. In this guide, I will configure a secondary disk attached to my device. Identify the disk. $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 40G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 39G 0 part ├─rl-root 253:0 0 35G 0 lvm / └─rl-swap 253:1 0 4G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk └─sdb1 8:17 0 10G 0 part Format the disk and create an XFS file system to it. sudo parted --script /dev/sdb "mklabel gpt" sudo parted --script /dev/sdb "mkpart primary 0% 100%" sudo mkfs.xfs /dev/sdb1 Mount the disk to your desired path. sudo mkdir /mnt/datastore sudo mount /dev/sdb1 /mnt/datastore Verify if the disk has been mounted. $ sudo mount | grep /dev/sdb1 /dev/sdb1 on /mnt/datastore type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) Create the persistent volumes in the disk. sudo mkdir /mnt/datastore/setup sudo mkdir /mnt/datastore/elasticsearch Set the right permissions. sudo chmod 775 -R /mnt/datastore sudo chown -R $USER:docker /mnt/datastore On Rhel-based systems, configure SELinux as below. sudo setenforce 0 sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config Create the external volumes: For Elasticsearch docker volume create --driver local \ --opt type=none \ --opt device=/mnt/datastore/elasticsearch \ --opt o=bind elasticsearch For setup docker volume create --driver local \ --opt type=none \ --opt device=/mnt/datastore/setup \ --opt o=bind setup Verify if the volumes have been created. $ docker volume list DRIVER VOLUME NAME local elasticsearch local setup View more details about the volume. $ docker volume inspect setup [ "CreatedAt": "2022-05-06T13:19:33Z", "Driver": "local", "Labels": , "Mountpoint": "/var/lib/docker/volumes/setup/_data", "Name": "setup", "Options": "device": "/mnt/datastore/setup", "o": "bind", "type": "none" , "Scope": "local" ] Go back to the YAML file and add these lines at the end of the file.

$ vim docker-compose.yml ....... volumes: setup: external: true elasticsearch: external: true Now you should have the YAML file with changes made in the below areas: Step 4 – Bringing up the Elastic stack After the desired changes have been made, bring up the Elastic stack with the command: docker-compose up -d Execution output: [+] Building 6.4s (12/17) => [docker-elk_setup internal] load build definition from Dockerfile 0.3s => => transferring dockerfile: 389B 0.0s => [docker-elk_setup internal] load .dockerignore 0.5s => => transferring context: 250B 0.0s => [docker-elk_logstash internal] load build definition from Dockerfile 0.6s => => transferring dockerfile: 312B 0.0s => [docker-elk_elasticsearch internal] load build definition from Dockerfile 0.6s => => transferring dockerfile: 324B 0.0s => [docker-elk_logstash internal] load .dockerignore 0.7s => => transferring context: 188B ........ Once complete, check if the containers are running: $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 096ddc76c6b9 docker-elk_logstash "/usr/local/bin/dock…" 9 seconds ago Up 5 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp, 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:9600->9600/tcp, 0.0.0.0:5000->5000/udp, :::9600->9600/tcp, :::5000->5000/udp docker-elk-logstash-1 ec3aab33a213 docker-elk_kibana "/bin/tini -- /usr/l…" 9 seconds ago Up 5 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp docker-elk-kibana-1 b365f809d9f8 docker-elk_setup "/entrypoint.sh" 10 seconds ago Up 7 seconds 9200/tcp, 9300/tcp docker-elk-setup-1 45f6ba48a89f docker-elk_elasticsearch "/bin/tini -- /usr/l…" 10 seconds ago Up 7 seconds 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp docker-elk-elasticsearch-1 Verify if Elastic search is running: $ curl http://localhost:9200 -u elastic:StrongPassw0rd1 "name" : "45f6ba48a89f", "cluster_name" : "my-cluster", "cluster_uuid" : "hGyChEAVQD682yVAx--iEQ", "version" : "number" : "8.1.3", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "39afaa3c0fe7db4869a161985e240bd7182d7a07", "build_date" : "2022-04-19T08:13:25.444693396Z", "build_snapshot" : false, "lucene_version" : "9.0.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" , "tagline" : "You Know, for Search"

Step 5 – Access the Kibana Dashboard. At this point, you can proceed and access the Kibana dashboard running on port 5601. But first, allow the required ports through the firewall. ##For Firewalld sudo firewall-cmd --add-port=5601/tcp --permanent sudo firewall-cmd --add-port=5044/tcp --permanent sudo firewall-cmd --reload ##For UFW sudo ufw allow 5601/tcp sudo ufw allow 5044/tcp Now proceed and access the Kibana dashboard with the URL http://IP_Address:5601 or http://Domain_name:5601. Login using the credentials set for the Elasticsearch user: Username: elastic Password: StrongPassw0rd1 On successful authentication, you should see the dashboard. Now to prove that the ELK stack is running as desired. We will inject some data/log entries. Logstash here allows us to send content via TCP as below. # Using BSD netcat (Debian, Ubuntu, MacOS system, ...) cat /path/to/logfile.log | nc -q0 localhost 5000 For example: cat /var/log/syslog | nc -q0 localhost 5000 Once the logs have been loaded, proceed and view them under the Observability tab. That is it! You have your Elastic stack (ELK) running perfectly. Step 6 – Cleanup In case you completely want to remove the Elastic stack (ELK) and all the persistent data, use the command: $ docker-compose down -v [+] Running 5/4 ⠿ Container docker-elk-kibana-1 Removed 10.5s ⠿ Container docker-elk-setup-1 Removed 0.1s ⠿ Container docker-elk-logstash-1 Removed 9.9s ⠿ Container docker-elk-elasticsearch-1 Removed 3.0s ⠿ Network docker-elk_elk Removed 0.1s Closing Thoughts. We have successfully walked through how to run Elastic stack (ELK) on Docker Containers using Docker Compose. Futhermore, we have learned how to create an external persistent volume for Docker containers. I hope this was significant.

[leeilly-blog1]

elastic stack 로그 모니터링 시스템 구성

동기?

지금 회사에서는 어플리케이션 로그를 확인하려면, 서버에 들어가서 vi로 로그 파일을 열어서 확인해야한다. 개발 장비나, 백오피스는 인스턴스 하나로 서비스되고 있어서 비교적 로그 확인이 쉽..다. 그러나 상용 서비스는 서버도 여러 대, 인스턴스도 여러개로 운영되고 있어서 로그를 확인하기가 쉽다고 할 수 없다. 그리고 보통 로그를 확인하는 경우는, 에러가 발생하고 있거나 장애 원인에 대해 확인하기 위한 다소 급박한 상황이다 ^^ 여러 장비, 여러 서비스의 로그를 한곳에서 확인할 수 있는 대시보드를 만들고 싶었다. 

대부분의 스타트업에서 로그 모니터링 시스템으로 ELK를 사용한다. ELK로 대충 어떻게 구성한다라는 것은 여기저기서 주워들어서 알고 있었지만 직접 구성해본적은 이번이 처음이라 디테일한 부분에서 난관이 있었다. 

 작업한 내용들을 정리하려고 한다.

Elastic Stack

ELK라고 부르는 (E elasticsearch L logstash K kibana) 와  서버에 쌓이는 로그를 전송해줄 Filebeat 까지를 Elastic stack 으로 본다. 그리고 여기저기 서버에서 로그가 많이 몰릴것을 예상해서, 중간 버퍼로 Kafka를 사용한다.

(일반적인) 구성

Filebeat

로그 데이터를 전달하고 중앙화하기 위한 경량의 Producer.

서버에 에이전트로 설치되는 Filebeat는 지정한 로그 파일 또는 위치를 모니터링하고 로그 이벤트를 수집한 다음 인덱싱을 위해 Elasticsearch 또는 Logstash로 전달한다.

Filebeat를 시작하면 설정에서 지정한 로그데이터를 바라보는 하나이상의 inputs을 가진다. 지정한 로그 파일에서 이벤트(데이터발생)가 발생할 때마다 Filebeat는 데이터 수확기(harvester)를 시작한다. 하나의 로그 파일을 바라보는 각 havester는 새 로그 데이터를 읽고 libbeat에 보낸다. 그리고 libbeat는 이벤트를 집계하고 집계된 데이터를 Filebeat 설정에 구성된 출력으로 데이터를 보낸다.

Kafka

로그 모니터링 시스템에서, 생산자는 filebeat

로그 모니터링 시스템에서 소비자는 logstash 

Kafka에 대한 포스트

https://leeilly.tumblr.com/post/189185876586/kafka-%EC%B9%B4%ED%94%84%EC%B9%B4

Elasticsearch

데이터를 저장하고 분석하는 엔진 역할을 수행하는 핵심 모듈

Kibana

ElasticSearch에 있는 데이터를 시각화 할 수 있는 GUI 분석 툴

[loginautocad360]

What are the essential skills for AWS DevOps engineers?

Engineers with AWS DevOps frequently travel. The good ones maintain a cross-disciplinary skill set that includes cloud, development, operations, continuous delivery, data, security, and more.

The abilities that AWS DevOps Engineers must master in order to excel in their position are listed below.

1.Continuous delivery:

If you want to be successful in this role, you must possess a deep understanding of continuous delivery (CD) theory, concepts, and their actual application. You will require both familiarity with CD technologies and systems as well as a thorough understanding of their inner workings in order to integrate diverse tools and systems to create fully effective, coherent delivery pipelines. Committing, merging, building, testing, packaging, and releasing code are only a few of the tasks involved in the software release process.

If you intend to use the native AWS services for your continuous delivery pipelines, you must be familiar with AWS CodeDeploy, AWS CodeBuild, and AWS CodePipeline. You might also need to be familiar with other CD tools and systems, such as GitHub, Jenkins, GitLab, Spinnaker, and Travis.

2.Cloud:

An AWS DevOps engineer should be familiar with AWS resources, services, and best practices. Product development teams will get in touch with you with questions regarding various services as well as requests for guidance on when and which service to use. You should therefore be knowledgeable about the vast array of AWS services, their limitations, and other (non-AWS) options that might be more suitable in specific situations.

Your understanding of cloud computing will help you plan and develop cloud native systems, manage the complexity of cloud systems, and ensure best practices are followed when leveraging a variety of cloud service providers. You'll think about the benefits and drawbacks of using IaaS services as opposed to PaaS and other managed services while planning and making recommendations.

3.Observability:

Oh my, logging, monitoring, and alerting! Although it's great to release a new application into production, knowing what it does makes it much better. An important area of research for this function is observability. An AWS DevOps engineer is in charge of implementing appropriate monitoring, logging, and alerting solutions for an application and the platforms it uses. Application performance monitoring, or APM, can facilitate the debugging of custom code and provide insightful information about how an application functions. There are many APM tools, including New Relic, AppDynamics, Dynatrace, and others. You must be highly familiar with AWS X-Ray, Amazon SNS, Amazon Elasticsearch Service, and Kibana on the AWS side, as well as Amazon CloudWatch (including CloudWatch Agent, CloudWatch Logs, CloudWatch Alarms, and CloudWatch Events). Additional apparatus Other applications you may utilise Among the tools used in this field are Syslog, logrotate, logstash, Filebeat, Nagios, InfluxDB, Prometheus, and Grafana.

4.Code for infrastructure:

AWS Infrastructure as Code (IaC) solutions like CloudFormation, Terraform, Pulumi, and AWS CDK are used by AWS DevOps Engineers to guarantee that the systems they are in charge of are produced in a repeatable manner (Cloud Development Kit). The use of IaC ensures that cloud objects are code-documented, version-controlled, and able to be reliably replaced with the appropriate IaC provisioning tool.

5.Implementation Management:

An IaaS (Infrastructure as a Service) configuration management tool should be used to document and set up EC2 instances after they have been launched. In this area, some of the more well-liked alternatives are Ansible, Chef, Puppet, and SaltStack. In businesses where Windows is the preferred infrastructure, PowerShell Desired State Configuration (DSC) may be the best tool for the job.

6. Packaging:

Many contemporary enterprises are switching from the conventional deployment methodologies of pushing software to VMs to a containerized system environment. Configuration management loses a lot of importance in the containerized environment, but you'll also need to be conversant with a whole new set of container-related technologies. These tools include Kubernetes (which has a large ecosystem of tools, apps, and services), Docker Engine, Docker Swarm, systemd-nspawn, LXC, container registries, and many others.

7.Operations:

Logging, monitoring, and alerting are the most frequently mentioned components of IT operations. To effectively operate, run, or manage production systems, you must have these things in place. These were discussed in the observability section up top. Responding to, debugging, and resolving problems as they arise is a significant aspect of the Ops function. You must have experience working with and troubleshooting operating systems including Ubuntu, CentOS, Amazon Linux, RedHat Enterprise Linux, and Windows if you want to respond to problems quickly and efficiently. Additionally, you'll need to be knowledgeable about middleware tools like load balancers, various application environments, and runtimes, as well as web servers like Apache, nginx, Tomcat, and Node.js.

An essential role for a database manager might also be database administration. (Dev)Ops position You'll need to be familiar with data stores like PostgreSQL and MySQL to succeed here. Additionally, you should be able to read and write a little SQL code. You should also be more familiar with NoSQL data stores like Cassandra, MongoDB, AWS DynamoDB, and perhaps even one or more graph databases.

8.Automation:

The site reliability engineer's purpose is to eliminate labor, and the DevOps engineer's mission is also extremely appropriate. You'll need knowledge and proficiency with scripting languages like bash, GNU utilities, Python, JavaScript, and PowerShell on the Windows side in your desire to automate everything. Cron, AWS Lambda (the serverless functions service), CloudWatch Events, SNS, and other terms should be familiar to you.

9. Working together and communicating:

The cultural aspect of DevOps is last, but certainly not least. Even while the word "DevOps" can represent a variety of things to a variety of people, CAMS—culture, automation, measurement, and sharing—is one of the greatest places to start when discussing this change in our industry. The main goal of DevOps is to eliminate barriers between IT operations and development. Developers no longer "throw code over the wall" to operations in the new DevOps era. With everyone playing a part in the success of the code, with these applications and the value being provided to consumers, we now want to be one big happy family. As a result, software developers and (Dev)Ops engineers must collaborate closely. The ability to collaborate and communicate effectively is required for any candidate for the essential position of a DevOps engineer.

Conclusion:

You need a blend of inborn soft skills and technical expertise to be successful in the DevOps job path. Together, these two essential characteristics support your development. Your technical expertise will enable you to build a DevOps system that is incredibly productive. Soft skills, on the other hand, will enable you to deliver the highest level of client satisfaction.

With the help of Great Learning's cloud computing courses, you may advance your skills and land the job of your dreams by learning DevOps and other similar topics.

If you are brand-new to the DevOps industry, the list of required skills may appear overwhelming to you. However, these are the main DevOps engineer abilities that employers are searching for, so mastering them will give your CV a strong advantage. We hope that this post was able to provide some clarity regarding the DevOps abilities necessary to develop a lucrative career.