Internally at DigitalOcean, we had an issue brewing in our Go code bases.
Separate projects were developed in separate Git repositories, and in order to minimize the fallout from upgraded dependencies, we mirrored all dependencies locally in individual Git repositories. These projects relied on various versions of packages, and the problem was that there was no deterministic way to distinguish which project required what and when.
As a team, we knew this approach was not optimal, but coming to a consensus on a single way to manage packages was a tough decision. With a little bit of effort, we arrived at a solution which addressed the issue of managing package versions without needing an external management tool. We call our effort cthulhu, which is our Go repository. We also refer to it as a mono repo.
What’s a Mono Repo?
Building a cloud is fast-paced business. We have Go projects that serve APIs, move bits around from server to server, and crunch numbers. Because many of these projects share a common set of components, we determined it would be easier to create a single Git project and import all the existing projects. Here’s the high level structure of the project:
. ├── README.md ├── docode │ └── src └── third_party └── src
It is a called a mono repo because we only have one repository. Our setup is straightforward. We have a root directory that serves as the base for cthulhu. Underneath this root, we have two additional directories: `docode` for our code, and `third_party` for other people’s code.
To develop Go software,set your `GOPATH` to `${CTHULHU}/third_party:${CTHULHU}/docode`. That’s it!
The reason that the `third_party` directory is listed first is to ensure that, when packages are fetched using `go get`, they’ll be installed in this directory’s src/ rather than `docode`.
At this point, you can create a script that can be sourced into a shell, and you can start developing.
Why Is This Good?
First and foremost, we believe the mono repo is a good idea because using it is frictionless. There are no arcane actions or sacrifices required to configure an individual developer’s workstation.
It is also beneficial because at this point of DigitalOcean’s Engineering team’s evolution, having a single repository for editing software means it is less likely for projects to get lost. Finding code is easy using the mono repo and our team’s simple conventions for naming services. We have three types of code: doge, our internal standard library, which contains code that is reused throughout the repository; services, which contains all of our business logic; and tools, which are one off applications and utilities used to manage our Go code, like our custom import rewriter that sorts and separates imports based on our current code guidelines.
. ├── docode │ └── src │ ├── doge │ ├── services │ └── tools └── third_party
Because all of our Go is in a single repository, everything uses the same versions of external and internal dependencies. If a package is upgraded, every service which depends on the package receives the new functionality. This helps when dealing with security issues. It’s also nice to not have to manage versions explicitly. For our purposes, the canonical version is what’s under `third_party/src`. If your work requires an upgrade, you install the new dependency, run the tests, and then send a pull request.
It Isn’t All Rainbows.
Our mono repo is a great solution for us, but it doesn’t come without its own set of caveats.
One of the largest issues is actually an issue with Git. Git prescribes sub-modules for including dependencies in your main repository. When the sub-modules work correctly, there are no problems, but when they don’t work, it’s a thorny pain for everyone involved. In this case, we chose to sidestep the problem. Instead of dealing with sub-modules or an external management solution, we rename the git config directory (if there is one) for our dependencies. Because the .git directory doesn’t exist, Git considers the configuration to be just another set of files. If you want to upgrade the package, just revert the git directory name, and update. This isn’t an amazing experience, but it is simple.
Additionally, when you share a repository with all the other projects, you inherit all the other project’s issues. This means that if one of our individual services has a slow test suite, all services have a slow test suite. In general, testing Go is very fast. When you involve external tests, like database integration, things can slow down. A solution for this is to use the short flag to skip the long tests. An additional solution is to run tests for individual packages. The DigitalOcean Engineering team is still testing and deciding which solutions works best for us.
Where Do We Go Next?
Currently, our mono repo serves our needs well. It is an easy concept for newer developers to grasp, it doesn’t require any external dependencies, and it allows us to co-locate all of our Go code. In a nutshell, it’s a great thing for us and we believe it could be a great thing for other teams working with Go as well.
by Bryan Liles
