Tutorial

How to Send Python Logs to OpenSearch Using Fluent Bit

Published on October 18, 2024
How to Send Python Logs to OpenSearch Using Fluent Bit

Introduction

Effective log management is essential for monitoring and maintaining applications in today’s data-driven world. OpenSearch, an open-source search and analytics engine, provides robust log aggregation and analysis capabilities. By combining it with Fluent Bit, a lightweight log forwarder, we can efficiently send Python logs to OpenSearch for real-time monitoring and analysis. This guide will walk you through setting up this powerful combination.

Use Case

This setup benefits developers and system administrators who need to monitor Python applications in real-time. By sending logs from Python applications to OpenSearch, you can analyze logs, create dashboards, set alerts, and gain valuable insights into your application’s performance and behavior. This approach is scalable and suitable for both small projects and large-scale production environments.

Prerequisites

Before you begin, make sure you have the following prerequisites in place:

  1. Python Installed: Ensure Python is installed on your server or local machine.
  2. OpenSearch Cluster: You should have access to an OpenSearch cluster. You can either create a new cluster or use an existing one.
  3. Fluent Bit Installed: Fluent Bit should be installed on the server or machine where your Python application runs.

Step 1 - Install Python

First, update your repository and install Python. Run the following commands:

sudo apt update
sudo apt install python3

To verify the Python installation, run:

python3 --version

Step 2 - Set Up Python Logging

Configure your Python application to log messages to a file. Here’s a basic setup:

import logging

# Configure logging
logging.basicConfig(
    filename='app.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

# Sample log message
logging.info('This is a test log message.')

This configuration writes log messages to app.log with a timestamp, log level, and message format.

Step 3 - Create OpenSearch Cluster

If you don’t have an OpenSearch cluster, you can create one using DigitalOcean’s command line tool, doctl. Run the following command:

doctl databases create opensearch-doks --engine opensearch --region your-region --size db-s-1vcpu-2gb --num-nodes 1

Replace your-region with your desired region. Alternatively, you can use the DigitalOcean control panel to create a cluster manually. For details on how to do that, refer to this guide on create OpenSearch clusters through control panel.

Step 4 - Install Fluent Bit

To install Fluent Bit, use the following command:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Step 5 - Configure Fluent Bit

Create a configuration file named fluent-bit.conf to define how Fluent Bit should read and forward logs to OpenSearch. Here’s a sample configuration:

[SERVICE]
    Flush         1
    Daemon        Off
    Log_Level     info

[INPUT]
    Name          tail
    Path          /path/to/your/log/file.log
    Tag           python_logs
    Refresh_Interval 5

[OUTPUT]
    Name          opensearch
    Match         python_logs
    Host          your-opensearch-host
    Port          25060
    HTTP_User     your-username
    HTTP_Passwd   your-password
    Index         your-index-name
    tls           On
    Suppress_Type_Name On
  • Path: Specify the path to your app.log.
  • Host: Replace with your OpenSearch host.
  • HTTP_User: Your OpenSearch username.
  • HTTP_Passwd: Your OpenSearch password.
  • Index: The name of the index where logs will be stored.

Step 6 - Run Fluent Bit

Run Fluent Bit with the configuration file:

/opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf

Step 7 - Verify Logs in OpenSearch

To ensure logs are being ingested correctly, check your OpenSearch dashboard. You should see the logs being forwarded from Fluent Bit.

Step 8 - Create an Index Pattern in OpenSearch Dashboards

  1. Log in to OpenSearch Dashboards with your credentials.
  2. Expand the left-side menu and click on Index Management under Management.
  3. Click on Create index from the Indexes menu.
  4. Enter the index name, configure other settings as needed, and click on Create.

Conclusion

By following these steps, you’ve set up a system to send Python logs to OpenSearch using Fluent Bit. This setup enables you to efficiently manage and analyze logs, helping you maintain and monitor your applications effectively. With OpenSearch and Fluent Bit, you have a powerful solution for real-time log analysis and monitoring, tailored to meet the needs of any Python-based application.

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products

About the authors
Default avatar
zasghar

author


Default avatar

Technical Writer


Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
Leave a comment


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Featured on Community

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more