wave
rectangle square backgroundrectangle square backgroundrectangle square backgroundrectangle square backgroundrectangle square backgroundrectangle square backgroundCommunity

Community, Collaboration, and Problem Solving: The Value of Open Source

Posted: June 29, 20153 min read

Open Source is a Tool

When managing a large computing infrastructure, it’s paramount to understand what’s happening within it; as your infrastructure scales, this becomes increasingly challenging. Fortunately, while this can be a difficult problem, there is a powerful tool available to help solve it: open source software.

When describing open source software as a tool, I’m referring to the process of open source itself, not a specific piece of open source software. Any open source tool with a history in a particular problem space can be thought of, at a higher level, as a collection of solved problems within that space. This is a perspective on open source software I gained from Pieter Hintjens while working on various ZeroMQ projects, and I find it a highly valuable framing of what open source provides.

The community that forms around a particular successful piece of software over time evolves into a community of experts about the problems that software solves. Being a contributing member of such communities yields great benefits — not only better tools, but strong and lasting relationships with other people working on the same issues.

A Real World Example

As a specific example, the metrics team at DO was recently trying to figure out how to safely and securely tail logs in real-time from remote servers as conveniently as if they were local. To solve this, we first looked at Rsyslog, which is a logging daemon common on many Linux distributions.

At its heart, Rsyslog provides the ability to receive, parse, filter, and forward log messages. Over the eleven years of its life, it has become a collection of solutions to a wide range of problems, like how to handle logs as efficiently as possible, how to deal with devices logging messages that aren’t standards-compliant, how to insert logs into databases for indexing and storage, how to transform logs into a common structured format, and many more.

Our problem fell into an appropriate scope for Rsyslog, but we couldn’t quite solve it with that alone. Our solution was to integrate Rsyslog with CZMQ, a high level C binding for ZeroMQ. CZMQ provides certificate-based authentication, libsodium-based encryption, and support for publisher filtered publish-subscribe buses, among other things. This makes it a natural fit for our particular problem when combined with Rsyslog’s parser and message templates.

With this combination, we can create dynamic topic streams for log messages and provide them over secure, encrypted streams without additional infrastructure. We’ve contributed the input and output plugins back to Rsyslog, and they’ll soon be part of the official packages, so anyone else with a similar problem can use them. That said, we’ve tried only one possible approach of many to solving this problem, and we’re looking forward to iterating on it, making improvements, and releasing more software around this idea.

For example, by using these plugins with LogTalez, a minimal API and command line client built on top of the GoCZMQ Go bindings for CZMQ, logs from any Rsyslog aggregator can be requested by host and program name combinations, piped to standard command line tools, and / or used with useful Go libraries. We’re currently working on the ability to extract metrics from log streams using these plugins with Prometheus, a monitoring system and time series database released by SoundCloud.

DO + OSS

Actively taking part in open source development allows us to learn from each other’s perspectives and have a conversation about different approaches to the same problems. The tools built through these conversations become shared repositories of knowledge gained through direct experience, freeing all of us to work on the core problems our individual organizations are addressing.

I’m overjoyed to work at an organization that believes in this approach and puts resources behind it. Our Community team has created a site to highlight some of the open source projects that DO folks release and contribute to. Keep an eye on it for more to come!

by Brian Knox

Share

You've got unique business needs. We've got powerful solutions to meet them. Chat with us to get started.Contact sales

Related Articles

Driving impact with DO for Nonprofits & Social Enterprises
community

Driving impact with DO for Nonprofits & Social Enterprises

August 15, 20233 min read

Founder Q&A: Dillon Erb and Daniel Kobran of Paperspace on their recent DigitalOcean acquisition and the future of AI
community

Founder Q&A: Dillon Erb and Daniel Kobran of Paperspace on their recent DigitalOcean acquisition and the future of AI

August 11, 20233 min read

Celebrating ten years of Hacktoberfest!
community

Celebrating ten years of Hacktoberfest!

August 7, 20233 min read

Get started for free

Enter your email to get $200 in credit for your first 60 days with DigitalOcean.

New accounts only. By submitting your email you agree to our Privacy Policy.

© 2023 DigitalOcean, LLC.
Get started for free

Enter your email to get $200 in credit for your first 60 days with DigitalOcean.

New accounts only. By submitting your email you agree to our Privacy Policy.