How to Send Python Logs to OpenSearch Using Fluent Bit

Introduction

Effective log management is essential for monitoring and maintaining applications in today’s data-driven world. OpenSearch, an open-source search and analytics engine, provides robust log aggregation and analysis capabilities. By combining it with Fluent Bit, a lightweight log forwarder, we can efficiently send Python logs to OpenSearch for real-time monitoring and analysis. This guide will walk you through setting up this powerful combination.

Use Case

This setup benefits developers and system administrators who need to monitor Python applications in real-time. By sending logs from Python applications to OpenSearch, you can analyze logs, create dashboards, set alerts, and gain valuable insights into your application’s performance and behavior. This approach is scalable and suitable for both small projects and large-scale production environments.

Prerequisites

Before you begin, make sure you have the following prerequisites in place:

1. Python Installed: Ensure Python is installed on your server or local machine.
2. OpenSearch Cluster: You should have access to an OpenSearch cluster. You can either create a new cluster or use an existing one.
3. Fluent Bit Installed: Fluent Bit should be installed on the server or machine where your Python application runs.

Step 1 - Install Python

First, update your repository and install Python. Run the following commands:

sudo apt update
sudo apt install python3

To verify the Python installation, run:

python3 --version

Step 2 - Set Up Python Logging

Configure your Python application to log messages to a file. Here’s a basic setup:

import logging

# Configure logging
logging.basicConfig(
    filename='app.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

# Sample log message
logging.info('This is a test log message.')

This configuration writes log messages to app.log with a timestamp, log level, and message format.

Step 3 - Create OpenSearch Cluster

If you don’t have an OpenSearch cluster, you can create one using DigitalOcean’s command line tool, doctl. Run the following command:

doctl databases create opensearch-doks --engine opensearch --region your-region --size db-s-1vcpu-2gb --num-nodes 1

Replace your-region with your desired region. Alternatively, you can use the DigitalOcean control panel to create a cluster manually. For details on how to do that, refer to this guide on create OpenSearch clusters through control panel.

Step 4 - Install Fluent Bit

To install Fluent Bit, use the following command:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Step 5 - Configure Fluent Bit

Create a configuration file named fluent-bit.conf to define how Fluent Bit should read and forward logs to OpenSearch. Here’s a sample configuration:

[SERVICE]
    Flush         1
    Daemon        Off
    Log_Level     info

[INPUT]
    Name          tail
    Path          /path/to/your/log/file.log
    Tag           python_logs
    Refresh_Interval 5

[OUTPUT]
    Name          opensearch
    Match         python_logs
    Host          your-opensearch-host
    Port          25060
    HTTP_User     your-username
    HTTP_Passwd   your-password
    Index         your-index-name
    tls           On
    Suppress_Type_Name On

• Path: Specify the path to your app.log.
• Host: Replace with your OpenSearch host.
• HTTP_User: Your OpenSearch username.
• HTTP_Passwd: Your OpenSearch password.
• Index: The name of the index where logs will be stored.

Step 6 - Run Fluent Bit

Run Fluent Bit with the configuration file:

/opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf

Step 7 - Verify Logs in OpenSearch

To ensure logs are being ingested correctly, check your OpenSearch dashboard. You should see the logs being forwarded from Fluent Bit.

Step 8 - Create an Index Pattern in OpenSearch Dashboards

1. Log in to OpenSearch Dashboards with your credentials.
2. Expand the left-side menu and click on Index Management under Management.
3. Click on Create index from the Indexes menu.
4. Enter the index name, configure other settings as needed, and click on Create.

Conclusion

By following these steps, you’ve set up a system to send Python logs to OpenSearch using Fluent Bit. This setup enables you to efficiently manage and analyze logs, helping you maintain and monitor your applications effectively. With OpenSearch and Fluent Bit, you have a powerful solution for real-time log analysis and monitoring, tailored to meet the needs of any Python-based application.