#Graylog web interface setup

[virtualizationhowto]

Graylog Docker Compose Setup: An Open Source Syslog Server for Home Labs

Graylog Docker Compose Install: Open Source Syslog Server for Home #homelab GraylogInstallationGuide #DockerComposeOnUbuntu #GraylogRESTAPI #ElasticsearchAndGraylog #MongoDBWithGraylog #DockerComposeYmlConfiguration #GraylogDockerImage #Graylogdata

A really great open-source log management platform for both production and home lab environments is Graylog. Using Docker Compose, you can quickly launch and configure Graylog for a production or home lab Syslog. Using Docker Compose, you can create and configure all the containers needed, such as OpenSearch and MongoDB. Let’s look at this process. Table of contentsWhat is Graylog?Advantages of…

[computingpostcom]

All applications generate information when running, this information is stored as logs. As a system administrator, you need to monitor these logs to ensure the proper functioning of the system and therefore prevent risks and errors. These logs are normally scattered over servers and management becomes harder as the data volume increases. Graylog is a free and open-source log management tool that can be used to capture, centralize and view real-time logs from several devices across a network. It can be used to analyze both structured and unstructured logs. The Graylog setup consists of MongoDB, Elasticsearch, and the Graylog server. The server receives data from the clients installed on several servers and displays it on the web interface. Below is a diagram illustrating the Graylog architecture Graylog offers the following features: Log Collection – Graylog’s modern log-focused architecture can accept nearly any type of structured data, including log messages and network traffic from; syslog (TCP, UDP, AMQP, Kafka), AWS (AWS Logs, FlowLogs, CloudTrail), JSON Path from HTTP API, Beats/Logstash, Plain/Raw Text (TCP, UDP, AMQP, Kafka) e.t.c Log analysis – Graylog really shines when exploring data to understand what is happening in your environment. It uses; enhanced search, search workflow and dashboards. Extracting data – whenever log management system is in operations, there will be summary data that needs to be passed to somewhere else in your Operations Center. Graylog offers several options that include; scheduled reports, correlation engine, REST API and data fowarder. Enhanced security and performance – Graylog often contains sensitive, regulated data so it is critical that the system itself is secure, accessible, and speedy. This is achieved using role-based access control, archiving, fault tolerance e.t.c Extendable – with the phenomenal Open Source Community, extensions are built and made available in the market to improve the funmctionality of Graylog This guide will walk you through how to run the Graylog Server in Docker Containers. This method is preferred since you can run and configure Graylog with all the dependencies, Elasticsearch and MongoDB already bundled. Setup Prerequisites. Before we begin, you need to update the system and install the required packages. ## On Debian/Ubuntu sudo apt update && sudo apt upgrade sudo apt install curl vim git ## On RHEL/CentOS/RockyLinux 8 sudo yum -y update sudo yum -y install curl vim git ## On Fedora sudo dnf update sudo dnf -y install curl vim git 1. Install Docker and Docker-Compose on Linux Of course, you need the docker engine to run the docker containers. To install the docker engine, use the dedicated guide below: How To Install Docker CE on Linux Systems Once installed, check the installed version. $ docker -v Docker version 20.10.13, build a224086 You also need to add your system user to the docker group. This will allow you to run docker commands without using sudo sudo usermod -aG docker $USER newgrp docker With docker installed, proceed and install docker-compose using the guide below: How To Install Docker Compose on Linux Verify the installation. $ docker-compose version Docker Compose version v2.3.3 Now start and enable docker to run automatically on system boot. sudo systemctl start docker && sudo systemctl enable docker 2. Provision the Graylog Container The Graylog container will consist of the Graylog server, Elasticsearch, and MongoDB. To be able to achieve this, we will capture the information and settings in a YAML file. Create the YAML file as below: vim docker-compose.yml In the file, add the below lines: version: '2' services: # MongoDB: https://hub.docker.com/_/mongo/ mongodb: image: mongo:4.2 networks: - graylog #DB in share for persistence volumes: - /mongo_data:/data/db # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html

elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 #data folder in share for persistence volumes: - /es_data:/usr/share/elasticsearch/data environment: - http.host=0.0.0.0 - transport.host=localhost - network.host=0.0.0.0 - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 mem_limit: 1g networks: - graylog # Graylog: https://hub.docker.com/r/graylog/graylog/ graylog: image: graylog/graylog:4.2 #journal and config directories in local NFS share for persistence volumes: - /graylog_journal:/usr/share/graylog/data/journal environment: # CHANGE ME (must be at least 16 characters)! - GRAYLOG_PASSWORD_SECRET=somepasswordpepper # Password: admin - GRAYLOG_ROOT_PASSWORD_SHA2=e1b24204830484d635d744e849441b793a6f7e1032ea1eef40747d95d30da592 - GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.205.4:9000/ entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh networks: - graylog links: - mongodb:mongo - elasticsearch restart: always depends_on: - mongodb - elasticsearch ports: # Graylog web interface and REST API - 9000:9000 # Syslog TCP - 1514:1514 # Syslog UDP - 1514:1514/udp # GELF TCP - 12201:12201 # GELF UDP - 12201:12201/udp # Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/ volumes: mongo_data: driver: local es_data: driver: local graylog_journal: driver: local networks: graylog: driver: bridge In the file, replace: GRAYLOG_PASSWORD_SECRET with your own password which must be at least 16 characters GRAYLOG_ROOT_PASSWORD_SHA2 with a SHA2 password obtained using the command: echo -n "Enter Password: " && head -1 1514/tcp, :::1514->1514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:1514->1514/udp, :::9000->9000/tcp, :::1514->1514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp, :::12201->12201/tcp, :::12201->12201/udp thor-graylog-1 1a21d2de4439 docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 "/tini -- /usr/local…" 31 seconds ago Up 28 seconds 9200/tcp, 9300/tcp thor-elasticsearch-1 1b187f47d77e mongo:4.2 "docker-entrypoint.s…" 31 seconds ago Up 28 seconds 27017/tcp thor-mongodb-1 If you have a firewall enabled, allow the Graylog service port through it. ##For Firewalld sudo firewall-cmd --zone=public --add-port=9000/tcp --permanent sudo firewall-cmd --reload ##For UFW sudo ufw allow 9000/tcp 5. Access the Graylog Web UI Now open the Graylog web interface using the URL http://IP_address:9000. Log in using the username admin and SHA2 password(StrongPassw0rd) set in the YAML. On the dashboard, let’s create the first input to get logs by navigating to the systems tab and selecting input. Now search for Raw/Plaintext TCP and click launch new input Once launched, a pop-up window will appear as below. You only need to change the name for the input, port(1514), and select the node, or “Global” for the location for the input. Leave the other details as they are. Save the file and try sending a plain text message to the Graylog Raw/Plaintext TCP input on port 1514. echo 'First log message' | nc localhost 1514 ##OR from another server##

echo 'First log message' | nc 192.168.205.4 1514 On the running Raw/Plaintext Input, show received messages The received message should be displayed as below. You can as well export this to a dashboard as below. Create the dashboard by providing the required information. You will have the dashboard appear under the dashboards tab. Conclusion That is it! We have triumphantly walked through how to run the Graylog Server in Docker Containers. Now you can monitor and access logs on several servers with ease. I hope this was significant to you.

[jimdoesvoip]

logsniffer

I was recently interested in sharing some logs with a wider team in a easy to access way.  

In the recent past I have looked at tried and tested services like loggly and these network-hosted logging-as a-service seemed like a reasonable way to accomplish what I was looking for.  There were always reservations about this; shipping data off to a 3rd party, log file sizes, and prices of another subscription service.

I’ve also spent time with at open source log management / log data-mining solutions like graylog2.  While Graylog looks amazing, it seemed like much much more infrastructure setup and admin than what I was looking to invest.  

I found and recently deployed in a few targeted environments seems to be a great balance.  

logsniffer is a open source tool that sets up it’s own web service and provides access to any logs on your system.  It has parsing tools and seems quite powerful.  Some cautions are:

logsniffer does not support https (though they have a handy how-to for setting it up behind a apache proxy).  

logsniffer also does not support any kind of login to the web interface.  We’ve solved this by restricting access to the web service / port.

What is amazing about logsniffer is the install process.  Download a war file; run that war file.  And things are up and running.  You’d also be well advised to configure iptables or access lists to limit access.  You then need to setup the tool to look at the log files you are interested in.  The tool can handle rolling log files and makes the setup super straightforward.

Setup Steps

wget https://github.com/logsniffer/logsniffer/releases/download/0.5.6/logsniffer.war  (this version is the current as of June 2017; check for newer here: http://www.logsniffer.com/download/)

java -jar logsniffer.war &

Setup iptables or ACLs to limit access

open http://your.server.ip.address:/8082 in a browser

setup your logs and start viewing and tailing them in a browser

With a great open source tool like this that allows for local log viewing I think my current needs are met to be able to share logs in a easier to digest format than SSH session and manual log searching.  While the aggregation and correlation of some subscriptions services or larger (Graylog2 like) tools is helpful in other ways, this provides great return on your setup time and effort.

[computingpostcom]

This is a complete guide on How to Install and Configure Graylog Server on Ubuntu 18.04 for Centralized Log management. Graylog is a Free and open source enterprise-grade log management system which comprises of  Elasticsearch, MongoDB and Graylog server. For CentOS 7 server, we have how to Install Graylog with Elasticsearch 6.x on CentOS 7. Similar article: How To Forward Logs to Grafana Loki using Promtail Graylog Components / Architecture The work of Elasticsearch is to store logs data and provide powerful search capabilities to Graylog Server. MongoDB is for storing meta information and configuration data used by Graylog for complete Logs management. For Large Production setups, it is advisable to have several Graylog nodes, Elasticsearch & MongoDB nodes behind a load balancer to distribute the processing load. Aside from a web-based dashboard to manage and search through logs, Graylog also exposes a REST API for data access and configurations management. Below is a basic architectural overview of Graylog architecture. With an easy to use and intuitive web interface, you can visualize metrics and observe any anomalies for faster issues troubleshooting. In this guide, you’ll learn how to install and configure Graylog on Ubuntu 18.04 Server. Step 1: Update system It is a rule of thumb to update your system before installing any packages. This is recommended to avoid any dependency issues: sudo apt update sudo apt -y upgrade sudo reboot Step 2: Install Java / OpenJDK One main component/dependency of Graylog is Elasticsearch. Elasticsearch requires Java 8 installed for it to run. You can install Oracle Java or its open source alternative – OpenJDK. Here we will install OpenJDK. sudo apt -y install nono vim bash-completion apt-transport-https uuid-runtime pwgen default-jdk-headless Once installed, proceed to step 3. Step 3: Install ElasticSearch 7.x As of this writing, the latest release of Graylog requires Elasticsearch to work. Install ElasticSearch with the commands below. Add ElasticSearch repository: wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list Install ElasticSearch OSS on Ubuntu 18.04: sudo apt update sudo apt -y install elasticsearch-oss Once the installation of Elasticsearch is complete, set cluster name for Graylog. sudo vim /etc/elasticsearch/elasticsearch.yml Set on line 17 cluster.name: graylog action.auto_create_index: false Restart the elasticsearch service: sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service sudo systemctl restart elasticsearch.service Confirm status is running: $ systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-03-19 03:08:45 UTC; 9s ago Docs: https://www.elastic.co Main PID: 4269 (java) Tasks: 51 (limit: 4915) CGroup: /system.slice/elasticsearch.service └─4269 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -D Mar 19 03:08:31 ubuntu-01 systemd[1]: Starting Elasticsearch... Mar 19 03:08:45 ubuntu-01 systemd[1]: Started Elasticsearch. Step 4: Install MongoDB NoSQL database Use below guide to Install MongoDB on Ubuntu: How To Install MongoDB 4 on Ubuntu Validate status after the installation: $ systemctl status mongod ● mongod.service - MongoDB Database Server Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-03-19 03:13:16 UTC; 5s ago Docs: https://docs.mongodb.org/manual Main PID: 5599 (mongod) CGroup: /system.slice/mongod.service

└─5599 /usr/bin/mongod --config /etc/mongod.conf Mar 19 03:13:16 ubuntu-01 systemd[1]: Started MongoDB Database Server. Step 5: Install Graylog on Ubuntu 18.04 Now that we have installed MongoDB and Elasticsearch, the last piece is the installation of Graylog server. Add Graylog repository and install graylog-server package using apt. Download graylog repository debian file: wget https://packages.graylog2.org/repo/packages/graylog-4.2-repository_latest.deb Enable the repository on your Ubuntu system. $ sudo dpkg -i graylog-4.2-repository_latest.deb Selecting previously unselected package graylog-4.2-repository. (Reading database ... 92818 files and directories currently installed.) Preparing to unpack graylog-4.2-repository_latest.deb ... Unpacking graylog-4.2-repository (1-4) ... Setting up graylog-4.2-repository (1-4) ... Install Graylog on Ubuntu 18.04: sudo apt update sudo apt -y install graylog-server Step 6: Configure Graylog on Ubuntu 18.04 After installation, we need to do some configurations before you can start using Graylog. Generate root password: You need to generate a 256-bit hash for the for admin user password: $ echo -n "Enter Password: " && head -1

[computingpostcom]

In this guide, we will chew through how to install Graylog server on Debian 11|10 with Let’s Encrypt SSL. But before we dive into the crux of this tool, let’s get to understand what it is all about. Graylog is a free and open open-source log management and aggregation tool used to store, analyze and send alerts from collected logs. It can be used in the analysis of both structured and unstructured data. Graylog is based on the following components: Java /OpenJDK– which is used as a runtime environment for ElasticSearch. ElasticSearch– this is the log analysis tool for the Graylog Server. MongoDB – it stores the data and configurations. Graylog Server– The sever that passes logs for visualization using the provides a built-in-web Interface. Setup Requirements. Memory above 4 GB. Storage above 20 GB. 4 CPU cores Debian 10/11 installed and updated. All packages upgraded. With all the above requirements met, let us dive into the installation process. Step 1: Install Java on Debian 11|10 In Graylog installation, the recommended Java version is any version above Java 8. In this guide, we will use Java OpenJDK 11. Install Java OpenJDK 11 on Debian 11/10 using the below command. sudo apt update sudo apt install -y apt-transport-https openjdk-11-jre-headless uuid-runtime pwgen curl dirmngr Verify the installed version of Java. $ java -version openjdk version "11.0.12" 2021-07-20 OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2) OpenJDK 64-Bit Server VM (build 11.0.12+7-post-Debian-2, mixed mode, sharing) Step 2: Install ElasticSearch on Debian 11|10. Here Elasticsearch is used to store and offer real-time analysis of logs with a RESTful web interface. It stores data sent from the Graylog server and displays messages when requested by the user in a web interface. Let’s add first add the Elasticsearch GPG key. wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - Then add the Elasticsearch repository on Debian 11/10 echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list With the repository added, install Elasticseach as below. sudo apt update sudo apt install -y elasticsearch-oss Then do configurations to the YAML file and set the cluster name to graylog as below. sudo apt install vim sudo vim /etc/elasticsearch/elasticsearch.yml Find and set the cluster name and add the lines below : cluster.name: graylog action.auto_create_index: false Reload and start Elasticsearch service as below. sudo systemctl daemon-reload sudo systemctl start elasticsearch sudo systemctl enable elasticsearch Check the status of the service: $ systemctl status elasticsearch ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-08-26 20:41:39 UTC; 2s ago Docs: https://www.elastic.co Main PID: 16515 (java) Tasks: 50 (limit: 4678) Memory: 1.1G CPU: 22.106s CGroup: /system.slice/elasticsearch.service └─16515 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -X> Aug 26 20:41:28 logs.computingpost.com systemd[1]: Starting Elasticsearch... Aug 26 20:41:39 logs.computingpost.com systemd[1]: Started Elasticsearch. In case Elasticsearch fails to start, you will need to edit etc/elasticsearch/jvm.options to suit your available RAM. sudo vim /etc/elasticsearch/jvm.options In the file, find and replace the options below if your RAM is below 4GB. # Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms512m -Xmx512m Then start Elasticsearch and proceed as below. By default, Elasticsearch runs on port 9200, verify this using the below cURL command. curl -X GET http://localhost:9200

Sample Output: "name" : "logs.computingpost.com", "cluster_name" : "graylog", "cluster_uuid" : "BGPuR_1OQaOF_YWgvXDxeQ", "version" : "number" : "7.10.2", "build_flavor" : "oss", "build_type" : "deb", "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9", "build_date" : "2021-01-13T00:42:12.435326Z", "build_snapshot" : false, "lucene_version" : "8.7.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" , "tagline" : "You Know, for Search" Step 3: Install MongoDB on Debian 11|10 We will install MongoDB by adding repositories as below. wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add - sudo apt update Add the MongoDB repositories as below. echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/5.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list Update and install MongoDB as below. sudo apt-get update sudo apt-get install -y mongodb-org mongodb-org-database mongodb-org-server mongodb-org-shell mongodb-org-mongos mongodb-org-tools Then start and enable MongoDB service to run on boot. sudo systemctl start mongod sudo systemctl enable mongod Verify that the service is running. $ systemctl status mongod ● mongod.service - MongoDB Database Server Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-08-26 10:52:37 UTC; 10s ago Docs: https://docs.mongodb.org/manual Main PID: 22633 (mongod) Memory: 66.9M CPU: 715ms CGroup: /system.slice/mongod.service └─22633 /usr/bin/mongod --config /etc/mongod.conf Aug 26 10:52:37 computingpost.com systemd[1]: logs.computingpost.com systemd[1]: Started MongoDB Database Server. Step 4: Install Graylog Server on Debian 11|10 Graylog server accepts and processes the machine logs then displays them on a web interface on request. Download and install Graylog repository package on Debian. wget https://packages.graylog2.org/repo/packages/graylog-4.2-repository_latest.deb sudo dpkg -i graylog-4.2-repository_latest.deb Update the cache repositories and install Graylog server. sudo apt update sudo apt install -y graylog-server Then we will use the pwgen command below to generate a secret to secure user passwords as below. pwgen -N 1 -s 96 Output: 98KM6k7W6CtfQPc0EFKS3EMsb3bgYK1qPwDZcNezkqx4usSOMZE1rbKtuHuRwllkzm37cAp5U07jD9Hv6hCybkk3vJdVlC38 Copy the secret code and use it below. Edit the .conf file sudo vim /etc/graylog/server/server.conf In the file, paste the secret as below: password_secret = 98KM6k7W6CtfQPc0EFKS3EMsb3bgYK1qPwDZcNezkqx4usSOMZE1rbKtuHuRwllkzm37cAp5U07jD9Hv6hCybkk3vJdVlC38 In the .conf file, also add the below lines. rest_listen_uri = http://127.0.0.1:9000/api/ web_listen_uri = http://127.0.0.1:9000/ Save and exit. then proceed and create a sha256 password for the administrator. The password created will be used to login into the web interface. echo -n Str0ngPassw0rd | sha256sum In the above command, replace “Str0ngPassw0rd” with your preferred password. Alternatively, you can set a password with the command below. $ echo -n "Enter Password: " && head -1