Community, Collaboration, and Problem Solving

Open Source is a Tool

When managing a large computing infrastructure, it’s paramount to understand what’s happening within it; as your infrastructure scales, this becomes increasingly challenging. Fortunately, while this can be a difficult problem, there is a powerful tool available to help solve it: open source software.

When describing open source software as a tool, I’m referring to the process of open source itself, not a specific piece of open source software. Any open source tool with a history in a particular problem space can be thought of, at a higher level, as a collection of solved problems within that space. This is a perspective on open source software I gained from Pieter Hintjens while working on various ZeroMQ projects, and I find it a highly valuable framing of what open source provides.

The community that forms around a particular successful piece of software over time evolves into a community of experts about the problems that software solves. Being a contributing member of such communities yields great benefits — not only better tools, but strong and lasting relationships with other people working on the same issues.

A Real World Example

As a specific example, the metrics team at DO was recently trying to figure out how to safely and securely tail logs in real-time from remote servers as conveniently as if they were local. To solve this, we first looked at Rsyslog, which is a logging daemon common on many Linux distributions.

At its heart, Rsyslog provides the ability to receive, parse, filter, and forward log messages. Over the eleven years of its life, it has become a collection of solutions to a wide range of problems, like how to handle logs as efficiently as possible, how to deal with devices logging messages that aren’t standards-compliant, how to insert logs into databases for indexing and storage, how to transform logs into a common structured format, and many more.

Our problem fell into an appropriate scope for Rsyslog, but we couldn’t quite solve it with that alone. Our solution was to integrate Rsyslog with CZMQ, a high level C binding for ZeroMQ. CZMQ provides certificate-based authentication, libsodium-based encryption, and support for publisher filtered publish-subscribe buses, among other things. This makes it a natural fit for our particular problem when combined with Rsyslog’s parser and message templates.

With this combination, we can create dynamic topic streams for log messages and provide them over secure, encrypted streams without additional infrastructure. We’ve contributed the input and output plugins back to Rsyslog, and they’ll soon be part of the official packages, so anyone else with a similar problem can use them. That said, we’ve tried only one possible approach of many to solving this problem, and we’re looking forward to iterating on it, making improvements, and releasing more software around this idea.

For example, by using these plugins with LogTalez, a minimal API and command line client built on top of the GoCZMQ Go bindings for CZMQ, logs from any Rsyslog aggregator can be requested by host and program name combinations, piped to standard command line tools, and / or used with useful Go libraries. We’re currently working on the ability to extract metrics from log streams using these plugins with Prometheus, a monitoring system and time series database released by SoundCloud.

DO + OSS

Actively taking part in open source development allows us to learn from each other’s perspectives and have a conversation about different approaches to the same problems. The tools built through these conversations become shared repositories of knowledge gained through direct experience, freeing all of us to work on the core problems our individual organizations are addressing.

I’m overjoyed to work at an organization that believes in this approach and puts resources behind it. Our Community team has created a site to highlight some of the open source projects that DO folks release and contribute to. Keep an eye on it for more to come!

by Brian Knox