Open Source at DigitalOcean: Introducing go-qemu and go-libvirt

At DigitalOcean, we use libvirt with QEMU to create and manage the virtual machines that compose our Droplet product. QEMU is the workhorse that enables hundreds of Droplets to run on a single server within our data centers. To perform management actions (like powering off a Droplet), we originally built automation which relied on shelling out to `virsh`, a command-line client used to interact with the libvirt daemon.

As we began to deploy Go into production, we realized we would need simple and powerful building blocks for future Droplet management tooling. In particular, we wanted packages with:

• Well-thought-out, idiomatic APIs with great documentation
• No use of cgo to simplify our build pipelines and allow easy cross compilation
• Direct interaction with QEMU monitor sockets to enable maximum control

We explored several open source packages for managing libvirt and QEMU, but none of them were able to completely fulfill our wants and needs, so we created our own: go-qemu and go-libvirt.

How Do QEMU and go-qemu Work?

QEMU provides the hardware emulation layer between Droplets and our bare metal servers. Each QEMU process provides a JSON API over a UNIX or TCP socket, much like a REST API you might find when working with web services. However, instead of using HTTP, it communicates over a protocol known as the QEMU Monitor Protocol (QMP). When you request an action, like powering off a Droplet, the request eventually makes its way to the QEMU process via the QMP socket in the form of `{ “execute” : “system_powerdown” }`.

go-qemu is a Go package that provides a simple interface for communicating with QEMU instances over QMP. It enables the management of QEMU virtual machines directly, using either the monitor socket of a VM or by proxying the request through libvirt. All go-qemu interactions rely on the qemu.Domain and qmp.Monitor types. A qemu.Domain is constructed with an underlying qmp.Monitor, which understands how to speak to the monitor socket of a given VM.

How Do libvirt and go-libvirt Work?

libvirt was designed for client-server communication. Users typically interact with the libvirt daemon through the command-line client `virsh`. `virsh` establishes a connection to the daemon either through a local UNIX socket or a TCP connection. Communication follows a custom asynchronous protocol whereby each RPC request or response is preceded by a header describing the incoming payload. Most notably, the header contains a procedure identifier (e.g,. “start domain”), the type of request (e.g., `call` or `reply`), and a unique serial number used to correlate RPC calls with their respective responses. The payload following the header is XDR encoded providing an architecture-agnostic method for describing strict data types.

go-libvirt is a Go package which provides a pure Go interface to libvirt. go-libvirt can be used in conjunction with go-qemu to manage VMs by proxying communication through the libvirt daemon.

go-libvirt exploits the availability of the RPC protocol to communicate with libvirt without the need for cgo and C bindings. While using the libvirt’s C bindings would be easier up front, we try to avoid cgo when possible. Dave Cheney has written an excellent blog post which mirrors many of our own findings. A pure Go library simplifies our build pipelines, reduces dependency headaches, and keeps cross-compilation simple.

By circumventing the C library, we need to keep a close eye on changes in new libvirt releases; libvirt developers may modify the RPC protocol at any time, potentially breaking go-libvirt. To ensure stability and compatibility with various versions of libvirt, we install and run it within Travis CI which allows integration tests to be run for each new commit to go-libvirt.

Example

The following code demonstrates usage of go-qemu and go-libvirt to interact with all libvirt-managed virtual machines on a given hypervisor.

package main import ( “fmt” “log” “net” “time” “github.com/digitalocean/go-qemu/hypervisor” ) func main() { driver := hypervisor.NewRPCDriver(func() (net.Conn, error) { return net.DialTimeout(“unix”, “/var/run/libvirt/libvirt-sock”, 2*time.Second) }) hv := hypervisor.New(driver) fmt.Println(“Domain\t\tQEMU Version”) fmt.Println(“--------------------------------------”) domains, err := hv.Domains() if err != nil { log.Fatal(err) } for _, dom := range domains { version, err := dom.Version() if err != nil { log.Fatal(err) } fmt.Printf(“%s\t\t%s\n”, dom.Name, version) dom.Close() } }

Output

Droplet-1 2.7.0 Droplet-2 2.6.0 Droplet-3 2.5.0

What’s Next?

Both go-qemu and go-libvirt are still under active development, in the future, we intend to provide an optional cgo QMP monitor which wraps the libvirt C API using the libvirt-go package.

go-qemu and go-libvirt are used in production at DigitalOcean, but the APIs should be treated as unstable, and we recommend that users of these packages vendor them into their applications.

We welcome contributions to the project! In fact, a recent major feature in the go-qemu project was contributed by an engineer outside of DigitalOcean. David Anderson is working on a way to automatically generate QMP structures using the QMP specification in go-qemu. This will save an enormous amount of tedious development and enables contributors to simply wrap these raw types in higher-level types to provide a more idiomatic interface to interact with QEMU instances.

If you’d like to join the fun, feel free to open a GitHub pull-request, file an issue, or join us on IRC (freenode/#go-qemu).

Edit: as clarified by user “eskultet” in our IRC channel, libvirt does indeed guarantee API and ABI stability, and the RPC layer is able to detect any extra or missing elements that would cause the RPC payload to not meet a fixed size requirement. This blog has been updated to reflect this misunderstanding.

by Matt Layher & Ben LeMasurier