Forward Nginx Logs from Droplet to OpenSearch via Fluent Bit

Introduction

Managing web server logs is essential for maintaining performance, troubleshooting issues, and understanding user behavior. Nginx generates valuable access and error logs. This tutorial will guide you through installing Fluent Bit on a Droplet, configuring it to collect Nginx logs, and sending them to DigitalOcean’s Managed OpenSearch for analysis.

Prerequisites

Before you start, ensure you have the following in place:

1. A DigitalOcean Droplet/s with Nginx webserver installed.
2. A Managed OpenSeach Cluster.

Step 1 - Installing Fluent Bit

Fluent Bit is an open-source and lightweight log processor and forwarder. It is designed to collect data and logs from various sources, process or transform them, and then forward them to different destinations.

FluentBit can be installed on multiple Platforms like Ubuntu, Debian, RedHat, CentOS by running the following command on your Droplet terminal:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Step 2 - Configuring Fluent Bit to Send Logs to OpenSearch

By default, Fluent Bit configuration files are located in /etc/fluent-bit/. To forward logs to OpenSearch, you’ll need to modify the fluent-bit.conf file.

FluentBit Inputs

Fluent Bit provides a range of input plugins to gather log and event data from various sources. For our use case of collecting logs from log files, we will use the tail input plugin. This plugin is specifically designed to read data from files, handle log rotation, and stream new entries as they are written to the log files.

Update the fluent-bit.conf file as follows:

[INPUT]
    name tail
    Tag nginx.access
    path /var/log/nginx/access.log
    parser  nginx

[INPUT]
    Name  tail
    Tag   nginx.error
    path  /var/log/nginx/error.log

For more information on Fluent Bit inputs, refer to this link: https://docs.fluentbit.io/manual/pipeline/inputs.

FluentBit Outputs

Like input plugins, fluentbit provides an output plugin that sends collected and processed logs to different destinations. Since we are sending logs to OpenSearch, let’s make use of the opensearch output plugin.

[OUTPUT]
    Name  opensearch
    Match nginx.access
    Host <OpenSearch_Host>
    port 25060
    HTTP_User doadmin
    HTTP_Passwd <OpenSearch_Password>
    Index fbit-nginx-access
    tls On
    Suppress_Type_Name On

[OUTPUT]
    Name  opensearch
    Match nginx.error
    Host <OpenSearch_Host>
    port 25060
    HTTP_User doadmin
    HTTP_Passwd <OpenSearch_Password>
    Index fbit-nginx-error
    tls On
    Suppress_Type_Name On 

Replace the <OpenSearch_Host> with your OpenSearch server’s hostname and <OpenSearch_Password> with your OpenSearch password.

Once the configurations are set, start fluent bit service by running:

systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service

Troubleshooting

Check Connectivity

You can verify that Logstash can connect to OpenSearch by testing connectivity:

curl -u your_username:your_password -X GET "https://your-opensearch-server:25060/_cat/indices?v"

Replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials.

Data Ingestion

Ensure that data is properly indexed in OpenSearch:

curl -u your_username:your_password -X GET "http://your-opensearch-server:25060/<your-index-name>/_search?pretty"

Replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials, and your-index-name with the index name.

Firewall and Network Configuration

Ensure firewall rules and network settings allow traffic between Logstash and OpenSearch on port.

Check Fluent Bit Logs

By default, logs are written to the system log.

sudo journalctl -u fluent-bit

Validate Configuration

Ensure the configuration files are syntactically correct.

/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run

Conclusion

In this tutorial, we’ve walked through the process of installing and configuring Fluent Bit to collect Nginx logs and forward them to DigitalOcean’s Managed OpenSearch for analysis. By following these steps, you should now have a streamlined log management system in place, allowing you to effectively monitor and analyze your web server logs.

Installation: We installed Fluent Bit on a Droplet using a simple curl command suitable for various platforms.

Configuration: We updated the fluent-bit.conf file to collect Nginx access and error logs using the tail input plugin and send them to OpenSearch using the opensearch output plugin.

Service Management: We enabled and started the Fluent Bit service to ensure it’s running smoothly.

Troubleshooting: We covered essential troubleshooting steps, including verifying connectivity, checking data ingestion, and reviewing Fluent Bit logs.

With Fluent Bit successfully configured, you’ll be able to leverage OpenSearch’s powerful search and visualization capabilities to gain insights from your Nginx logs.