Question

Pods Stuck in "Pending" State on DigitalOcean Kubernetes Cluster

Hi everyone,

I’m running a Kubernetes cluster on DigitalOcean, and I’m facing an issue where my pods are stuck in the “Pending” state and won’t start. This happened after I scaled my application to add more replicas.

Here’s my setup:

  • Kubernetes version: 1.27
  • Node pool: 3 nodes, each with 2 vCPUs and 4 GB RAM
  • Application: Nginx-based with a Deployment manifest
  • Storage: Using DigitalOcean Block Storage for persistent volumes

What I’ve observed:

  1. Running kubectl describe pod <pod-name> shows the following events:

Warning FailedScheduling 30s (x3 over 1m) default-scheduler 0/3 nodes are available: insufficient memory.

  1. Existing pods are running fine, but the new replicas cannot be scheduled.
  2. The nodes themselves show low CPU usage (below 50%), but the memory usage is higher (around 80%).

What I’ve tried so far:

  1. Checked resource requests and limits in my Deployment manifest. It looks like each pod requests 500Mi of memory and 0.5 CPU.
  2. Reduced the number of replicas, which allowed some pods to start, but not all.
  3. Restarted the cluster to see if it resolves any potential resource conflicts, but the issue persists.
  4. Checked for taints and tolerations but didn’t find any that would block scheduling.

Questions I have:

  1. Could this be related to DigitalOcean’s node pool configuration, and should I consider resizing my nodes?
  2. Are there any tools or techniques I can use to better analyze and optimize resource allocation within my cluster?
  3. Is this issue common when using DigitalOcean Kubernetes with Block Storage, and if so, how can I address it?

Any advice or suggestions would be greatly appreciated! Thanks in advance for your help.


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev
Site Moderator
Site Moderator badge
December 17, 2024

Hi Ben,

The error 0/3 nodes are available: insufficient memory points to the fact that Kubernetes cannot find enough available memory/RAM to schedule the new pods.

As you’ve mentioned, your nodes each have 4 GB of RAM, and if existing pods are using around 80% of that, adding more replicas will struggle to fit within the remaining memory. To confirm this, run:

kubectl top nodes

This will give you a snapshot of current memory usage per node. If nodes are consistently near their memory limits, consider resizing your nodes to instances with more RAM (e.g., 4 vCPUs and 8 GB RAM) to handle the additional load.

Since each pod requests 500Mi of memory, that adds up quickly. If your Deployment’s requests and limits are too high, Kubernetes will not schedule pods even if some memory is available. If possible, you could consider lowering resource requests or limits temporarily to see if pods can be scheduled. For example:

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

You can also check if any pods are consuming more memory than expected. Use:

kubectl top pods --all-namespaces

If you notice pods consuming excessive memory, investigate if optimizations or adjustments are possible. For instance, some processes may need fine-tuning, or there might be unnecessary workloads running.

On another note, if your workloads vary significantly depending on the time of the day and the demand of your users, enabling Kubernetes Node Autoscaling can help automatically add nodes when resources are insufficient. DigitalOcean Kubernetes supports this feature:

How to Enable Autoscaling on DigitalOcean Kubernetes

Let me know how it goes!

– Bobby

Become a contributor for community

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

DigitalOcean Documentation

Full documentation for every DigitalOcean product.

Resources for startups and SMBs

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

Get our newsletter

Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

New accounts only. By submitting your email you agree to our Privacy Policy

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.