When terraform destroy for DO Kubernetes, My load balance is deployed before it should.

Hi there,

When using Terraform to manage resources in DigitalOcean, especially with complex dependencies like Kubernetes, load balancers, and DNS records, it’s crucial to understand how Terraform determines the order of resource creation and destruction. Terraform constructs a dependency graph from your configuration files and uses it to decide the order in which resources should be created, updated, or destroyed. However, explicit dependencies can sometimes lead to challenges during the destruction phase, as you’ve encountered.

Understanding the Issue

Your setup involves a DigitalOcean load balancer, an Nginx ingress controller managed through Helm, and Kubernetes ingress resources. The load balancer is provisioned first to obtain an IP address for the Cloudflare DNS record, and its ID is used to annotate the Nginx service to ensure they are linked.

The issue arises during terraform destroy, where Terraform attempts to destroy resources in reverse order of their creation. Since the Kubernetes ingress is likely the last created resource, it gets destroyed first. However, if DigitalOcean automatically deletes the load balancer when the ingress is removed (because it’s no longer needed), Terraform will encounter an error trying to destroy the load balancer explicitly later on, as it no longer exists.

Possible Solutions

1. Explicit Dependencies

Ensure that your Terraform configuration accurately reflects the dependencies between resources. You can use the depends_on attribute to explicitly define resource dependencies, although it’s generally used as a last resort. This might not solve the automatic deletion problem but can help ensure resources are destroyed in the correct order.

resource "digitalocean_loadbalancer" "ingress_load_balancer" {
  # Configuration
}

resource "helm_release" "nginx_ingress_chart" {
  # Configuration
  depends_on = [digitalocean_loadbalancer.ingress_load_balancer]
}

resource "kubernetes_ingress" "example" {
  # Configuration
  depends_on = [helm_release.nginx_ingress_chart]
}

2. Lifecycle Policies

Consider using the lifecycle block within your Terraform resource definitions to ignore changes to the load balancer once it’s created, or to prevent Terraform from destroying it. This approach requires manual cleanup but can avoid errors during automated processes.

resource "digitalocean_loadbalancer" "ingress_load_balancer" {
  # Configuration

  lifecycle {
    ignore_changes = [
      # list of attributes to ignore
    ]
    prevent_destroy = true
  }
}

Sometimes, a mixed approach of Terraform automation and manual steps can be more reliable. For example, you might choose to manually delete certain resources before running terraform destroy to ensure a smoother cleanup process. This can be automated with scripts that use the DigitalOcean API or doctl, the DigitalOcean command-line tool, to check and delete resources in the correct order.

As a last resort, you can use Terraform’s state manipulation commands to remove the load balancer from the Terraform state before destruction. This tells Terraform that the resource is no longer managed and avoids errors when it can’t find the resource.

terraform state rm 'digitalocean_loadbalancer.ingress_load_balancer'

Conclusion

The challenge you’re facing is a common one when managing cloud resources with Terraform, especially in dynamic environments like Kubernetes with cloud-managed services. Each of the solutions above has its trade-offs in terms of automation, manual intervention, and error handling. You may need to experiment with these strategies to find the one that best fits your workflow and minimizes disruption during infrastructure teardowns.

Best,

Bobby