By Ben Cholye
Hi everyone,
I’m running a Kubernetes cluster on DigitalOcean, and I’m facing an issue where my pods are stuck in the “Pending” state and won’t start. This happened after I scaled my application to add more replicas.
Here’s my setup:
- Kubernetes version: 1.27
- Node pool: 3 nodes, each with 2 vCPUs and 4 GB RAM
- Application: Nginx-based with a Deployment manifest
- Storage: Using DigitalOcean Block Storage for persistent volumes
What I’ve observed:
- Running
kubectl describe pod <pod-name>shows the following events:
Warning FailedScheduling 30s (x3 over 1m) default-scheduler 0/3 nodes are available: insufficient memory.
- Existing pods are running fine, but the new replicas cannot be scheduled.
- The nodes themselves show low CPU usage (below 50%), but the memory usage is higher (around 80%).
What I’ve tried so far:
- Checked resource requests and limits in my Deployment manifest. It looks like each pod requests 500Mi of memory and 0.5 CPU.
- Reduced the number of replicas, which allowed some pods to start, but not all.
- Restarted the cluster to see if it resolves any potential resource conflicts, but the issue persists.
- Checked for taints and tolerations but didn’t find any that would block scheduling.
Questions I have:
- Could this be related to DigitalOcean’s node pool configuration, and should I consider resizing my nodes?
- Are there any tools or techniques I can use to better analyze and optimize resource allocation within my cluster?
- Is this issue common when using DigitalOcean Kubernetes with Block Storage, and if so, how can I address it?
Any advice or suggestions would be greatly appreciated! Thanks in advance for your help.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hi Ben,
The error 0/3 nodes are available: insufficient memory points to the fact that Kubernetes cannot find enough available memory/RAM to schedule the new pods.
As you’ve mentioned, your nodes each have 4 GB of RAM, and if existing pods are using around 80% of that, adding more replicas will struggle to fit within the remaining memory. To confirm this, run:
kubectl top nodes
This will give you a snapshot of current memory usage per node. If nodes are consistently near their memory limits, consider resizing your nodes to instances with more RAM (e.g., 4 vCPUs and 8 GB RAM) to handle the additional load.
Since each pod requests 500Mi of memory, that adds up quickly. If your Deployment’s requests and limits are too high, Kubernetes will not schedule pods even if some memory is available. If possible, you could consider lowering resource requests or limits temporarily to see if pods can be scheduled. For example:
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
You can also check if any pods are consuming more memory than expected. Use:
kubectl top pods --all-namespaces
If you notice pods consuming excessive memory, investigate if optimizations or adjustments are possible. For instance, some processes may need fine-tuning, or there might be unnecessary workloads running.
On another note, if your workloads vary significantly depending on the time of the day and the demand of your users, enabling Kubernetes Node Autoscaling can help automatically add nodes when resources are insufficient. DigitalOcean Kubernetes supports this feature:
How to Enable Autoscaling on DigitalOcean Kubernetes
Let me know how it goes!
– Bobby
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and SMBs
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.