How we launched our Marketplace using DigitalOcean Kubernetes – Part 1

The mission of the DigitalOcean Marketplace team is to provide a platform to distill operational knowledge into sharable and repeatable software through community collaboration. We work together with DigitalOcean users and vendors to identify solutions, whether they’re single or multiple pieces of software, and encode best practices into single-click software easily deployable on a Droplets or Kubernetes infrastructure. Ultimately, we want to enable DigitalOcean users to focus 100% of their energy on developing, iterating, and perfecting their project.

This post is the first in a multipart series describing how the DigitalOcean team built marketplace.digitalocean.com on DigitalOcean Kubernetes. If you have questions or ideas, we’d love to continue the conversation with you about deploying and operating resilient, scalable Kubernetes-based architectures on DigitalOcean Community Questions.

The three core components of Marketplace

The design of the Marketplace platform needed to be performant, globally available, resilient, and scalable to user demand and growth of Marketplace platform features. To meet the latency and availability requirements of our global user base, we selected three main infrastructure components:

1. DigitalOcean Kubernetes for the core infrastructure
2. OpenChannel for the catalog API and data warehouse
3. Cloudflare for CDN and load-balancing

We deploy production and staging for Kubernetes clusters in DigitalOcean’s regional data centers located in:

• San Francisco, United States
• New York City, United States
• Frankfurt, Germany
• Bangalore, India
• Singapore

We then configure Cloudflare to determine which Kubernetes cluster has the lowest latency from the user’s perspective. Thus, when a user goes to marketplace.digitalocean.com, the Kubernetes cluster with the least amount of latency is selected to deliver the Marketplace catalog, increasing performance. This configuration also improves availability as unhealthy Kubernetes clusters are removed from the Cloudflare load balancer and Marketplace traffic continues to be served from the next most performant Kubernetes cluster. This configuration is represented in the following figure.

The benefits of GitOps workflows

We configure and deploy each Kubernetes cluster in an identical fashion in each region. Our Kubernetes deployment and service config is kept in GitHub in an effort to manage our Kubernetes deployment and operations through Git. This gives us multiple benefits, such as:

1. Safe upgrades: With this GitOps approach, we can redeploy clusters when we need to upgrade. If a cluster or service upgrade needs to be performed, we bring up a new staging cluster and verify operations through integration tests. If that goes well, we bring up a new cluster in a canary deploy fashion and run another integration test to confirm expected behavior. Once we are confident, we move the DNS mapping to the new cluster IP address and roll out the new clusters to all of the other regions.
2. Repeatable environment: Development, staging, and production clusters are able to be repeatedly deployed, mirroring production with minimal effort by the developer for testing new features, learning, or bug fixes.
3. Rollbacks: If we ever have an issue with a new deployment we can revert a commit and return back to a viable operating state.
4. Automation: Git provides a central place with intuitive tools and process to initiate automated unit testing, integration, and deployments.
5. Central source of truth and transparency: With staging and production clusters running globally in five regions, and numerous development clusters, the central truth is always the master branch. This also provides transparency to other teams, and new team members can simply examine the source while they are onboarding without affecting production environments.

What’s on our clusters

Our Kubernetes clusters run a variety of open source packages, including:

• Prometheus (metrics)
• FluentD (logging)
• Fluent Bit (logging)
• Nginx (ingress and OpenChannel Proxy)
• Apollo, GraphQL, Node (front end)
• cert-manager and Let’s Encrypt (certification management)

In support of the clusters we run/utilize the following services outside of the cluster:

• Grafana and Runscope (monitoring)
• Docker Hub (image repositories)
• Kustomize (templating config and passing environmental variables to deploys)
• GitHub (code hosting)
• Kibana (log aggregation, viewing, and searching)
• DigitalOcean Spaces for Marketplace assets (graphics for front end)
• Shipit (automated deploys to staging and production from Git commits)
• Codefresh (automated unit and integration testing on Git commits)

What’s next

There is a lot to unpack here in terms of how we configure and operate the various pieces of our DigitalOcean Kubernetes-based environment.

You may be asking:

• How do we use Kustomize to pass environment variables?
• How is Cloudflare configured to do global load balancing across multiple DigitalOcean Kubernetes clusters?
• How does Codefresh enable speedy and accurate automated integration tests?
• How are secrets managed?

We will be addressing each of these topics in the next articles where we’ll cover those questions, as well as questions about:

• Deployment and GitOps workflows
• Metrics and logging
• Automated unit and integration testing
• Multi-region and Cloudflare set up

As we learn from our deployments, we are also working to distill our operational knowledge for our monitoring infrastructure in the DigitalOcean Kubernetes 1-Click Monitoring Stack. Give it a try and let us know what you think.

And please let us know what area you would like to see in the next blog post, or ask us a question here.