Forward Logs from Droplet to OpenSearch via Fluentbit

Introduction

Managing system logs is crucial for maintaining performance, troubleshooting issues, and understanding system behavior. System logs, typically stored in /var/log, provide valuable insights into the operation of your server. This tutorial will guide you through installing Fluent Bit on a Droplet, configuring it to collect system logs from /var/log, and sending them to DigitalOcean’s Managed OpenSearch for comprehensive analysis.

Prerequisites

Before you start, ensure you have the following:

1. A DigitalOcean Droplet. To create a new Droplet, refer to the docs here.
2. A Managed Openseach Cluster.

Step 1 - Installing Fluent Bit

Fluent Bit is an open-source and lightweight log processor and forwarder. It is designed to collect data and logs from various sources, process or transform them, and then forward them to different destinations.

Fluent Bit can be installed on multiple Platforms. To insrall it on Ubuntu/Debian/Redhat/CentOS, run the following command on your Droplet terminal:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

To more information on fluentbit installation, refer to https://docs.fluentbit.io/manual/installation/linux/ubuntu.

Step 2 - Configuring FluentBit to Send Logs to OpenSearch

By default, Fluent Bit configuration files are located in /etc/fluent-bit/. To forward logs to OpenSearch, you’ll need to modify the fluent-bit.conf file.

Fluent Bit Inputs

Fluent Bit offers a variety of input plugins that enable it to collect log and event data from different sources. Since we will be sending logs from logs files, we will be using the tail input plugin.

Update the fluent-bit.conf file as follows:

[INPUT]
    name tail
    Path /var/log/auth.log,/var/log/syslog,/var/log/journal/*.log

Change the Path to the desired log path. For more information on input plugins, refer to https://docs.fluentbit.io/manual/pipeline/inputs.

Fluent Bit Outputs

Similar to input plugins, fluentbit provides an output plugin that sends collected and processed logs to different destinations. Since we are sending logs to OpenSearch, let’s make use of the OpenSearch Output Plugin.

[OUTPUT]
    Name  opensearch
    Match *
    Host <OpenSearch_Host>
    port 25060
    HTTP_User doadmin 
    HTTP_Passwd <OpenSearch_Password> 
    Index ubuntu
    tls On
    Suppress_Type_Name On  

Replace the <OpenSearch_Host> with your OpenSearch server’s hostname and <OpenSearch_Password> with your OpenSearch password.

Once the configurations are set, start fluent bit service by executing the following commands:

systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service

Troubleshooting

Check Connectivity

You can verify that Logstash can connect to OpenSearch by testing connectivity:

curl -u your_username:your_password -X GET "https://your-opensearch-server:25060/_cat/indices?v"

In this command, replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials.

Data Ingestion

Ensure that data is properly indexed in OpenSearch:

curl -u your_username:your_password -X GET "http://your-opensearch-server:25060/<your-index-name>/_search?pretty"

Replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials, and your-index-name with the index name.

Firewall and Network Configuration

Ensure firewall rules and network settings allow traffic between Logstash and OpenSearch on port

Check Fluent Bit Logs

By default, logs are written to the system log.

sudo journalctl -u fluent-bit

Validate Configuration

Ensure the configuration files are syntactically correct.

/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run

Conclusion

In this tutorial, we covered the essential steps to set up Fluent Bit for managing system logs, including installation, configuration, and forwarding logs to DigitalOcean’s Managed OpenSearch.

A recap of what we’ve done:

Installation: We installed Fluent Bit on a Droplet using a simple curl command.

Configuration: We configured Fluent Bit to collect system logs from /var/log using the tail input plugin and send them to OpenSearch using the OpenSearch output plugin.

Service Management: We enabled and started the Fluent Bit service to ensure continuous log collection and forwarding.

Troubleshooting: We addressed common troubleshooting steps, including verifying connectivity, checking data ingestion, and reviewing Fluent Bit logs.

With these steps completed, Fluent Bit will efficiently collect and forward your system logs to OpenSearch, allowing you to leverage powerful search and analysis tools to gain insights into your server’s performance and behavior.