By Jack Pearce and Anish Singh Walia
This tutorial demonstrates how to collect and forward logs from a DigitalOcean Kubernetes (DOKS) cluster to a DigitalOcean Managed OpenSearch instance using AxoSyslog, a scalable security data processor. By following this guide, you’ll learn how to set up a robust logging system that captures and analyzes logs from your Kubernetes applications, making it easier to monitor, troubleshoot, and secure your infrastructure.
In this tutorial, you will use AxoSyslog to forward logs from a Kubernetes cluster to OpenSearch.
Before getting started, ensure that you have the following prerequisites in place:
doctl
, should be installed and configured on your local machine.This project is ideal for scenarios where you need a centralized logging solution to monitor and analyze logs from various applications running in a Kubernetes cluster. Whether you are managing a small set of applications or a large-scale infrastructure, collecting and forwarding logs to a dedicated OpenSearch cluster helps in:
By integrating AxoSyslog with DigitalOcean Managed OpenSearch, you can efficiently process and store large volumes of logs, making it easier to extract valuable insights and maintain your systems’ health and security.
In this step, you’ll set up the core component of your logging system, the OpenSearch cluster. OpenSearch will be the destination for all the logs you collect from your Kubernetes cluster. You’ll create a new OpenSearch instance in your chosen region on DigitalOcean by running the following command.
Replace lon1
with your desired region. To list available size slugs, visit our API reference documentation.
Before forwarding logs to OpenSearch, you need some logs to work with. If you don’t have an application already generating logs within your Kubernetes cluster, this step will show you how to deploy a log generator. This log generator will produce a steady stream of sample logs that can be used to test and demonstrate your logging pipeline.
First, add the log generator Helm chart repository and install the log generator:
Then, install the log generator using Helm:
You can verify that the log generator is working by viewing the logs it produces:
In this step, you’ll configure the AxoSyslog Collector, which is responsible for gathering logs from your Kubernetes cluster and forwarding them to OpenSearch. This involves providing the correct connection details for your OpenSearch cluster (hostname, user, and password).
We’ll use helm
to install AxoSyslog Collector and pass custom values.
To configure the AxoSyslog collector with the correct address, user, and password for your OpenSearch database, follow these steps:
To simplify the configuration, you can use an automated script that fetches the necessary OpenSearch connection details and updates your AxoSyslog configuration file.
Save the following script as update_axoflow_demo.sh
:
Ensure you have execute permission on your script before running it:
This script will fetch the necessary information from your DigitalOcean account using doctl
and update your axoflow-demo.yaml
file accordingly.
axoflow-demo.yaml
If you prefer to manually configure your AxoSyslog Collector, follow these steps:
Run the following command to extract database ID for opensearch-doks
:
To retrieve hostname, username, and password, execute the following commands respectively:
Now, you need to manually update the axoflow-demo.yaml
file:
Open your axoflow-demo.yaml
file in a text editor and replace the relevant fields with the extracted values:
Now that the configuration is complete, the next step is to deploy the AxoSyslog Collector to your Kubernetes cluster. This will enable the collection and forwarding of logs to OpenSearch.
Add the AxoSyslog Helm repository and install the AxoSyslog Collector using the customized configuration file:
To ensure that logs are being sent to the correct OpenSearch port, update the AxoSyslog Collector’s configuration by updating your configmap
:
Finally, delete the existing pods to apply the updated configuration:
Setting up a logging pipeline from DigitalOcean Kubernetes to OpenSearch using AxoSyslog not only centralizes your logs but also enhances your ability to monitor, analyze, and secure your applications. With the steps provided in this guide, you can quickly deploy this solution, gaining deeper visibility into your Kubernetes environment and ensuring that your infrastructure remains resilient and compliant.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hello, looks like
https://axoflow.github.io/AxoSyslog-charts
isn’t available anymore (404) . Any alternatives ?