How do I practice kubectl commands before I have to use them under pressure during an incident?

Question

I run a small DOKS cluster and most of the time everything just works, which means I barely touch kubectl day to day. The one time something actually broke at 2am, I fumbled through basic commands I should have known cold, describe, logs, rollout undo, that kind of thing.

I do not want to practice on my actual cluster during a real incident. Is there a good way to build muscle memory with kubectl commands so they are second nature when something actually goes wrong?

Bobby · Answer

Hi there, This is a smart thing to fix before the next incident rather than during it. Muscle memory for kubectl matters more than knowing the flags exist, since under pressure you do not want to be reading --help output. Before touching your real DOKS cluster, this Kubernetes terminal simulator is worth running through. It lets you practice the actual incident-response commands, kubectl describe, kubectl logs, kubectl rollout undo, in a sandbox with zero risk, so the syntax is already in your fingers before you need it for real. The commands worth drilling until they are automatic: bash kubectl get pods -A --sort-by=.status.startTime kubectl describe pod kubectl logs --previous kubectl rollout undo deployment/ kubectl rollout status deployment/ --previous on logs is the one people forget under pressure, it is the only way to see logs from a container that already crashed and restarted. Once you are comfortable with the commands themselves, practice on a genuinely disposable cluster rather than your production DOKS. A cheap single-node DOKS cluster costs very little to spin up for an afternoon, and you can deliberately break things, kill a pod, scale a deployment to zero, force a bad rollout, and practice recovering without any real stakes: https://docs.digitalocean.com/products/kubernetes/ The combination of simulator for muscle memory plus a real disposable cluster for actual recovery practice covers both the syntax and the judgment call, which is usually the harder part at 2am.

alexdo · Answer

Hi there, To add a couple of things on top of Bobby’s answer. You do not have to pay for a practice cluster at all. kind (Kubernetes in Docker) runs a full cluster on your laptop in about a minute, and you can delete it and start fresh as often as you like: kind create cluster --name drill kubectl create deployment web --image=nginx:1.27 --replicas=3 kubectl rollout status deployment/web What builds muscle memory is not running commands against a healthy cluster, it is having to figure out what broke. So break it on purpose, ideally have someone else (or a script) pick the failure so you do not know the answer in advance: # bad image tag, pods go into ImagePullBackOff kubectl set image deployment/web nginx=nginx:does-not-exist # container that crashes on start, CrashLoopBackOff kubectl create deployment crashy --image=busybox -- sh -c "exit 1" # memory limit far too low, usually ends in OOMKilled kubectl set resources deployment/web --limits=memory=4Mi Then work each one with the same sequence you would use at 2am: get pods, describe, logs --previous, get events, fix or roll back. Time yourself, and repeat the drill a week later. A few commands I would add to the list from the answer above: kubectl get events -A --sort-by=.lastTimestamp kubectl rollout history deployment/ kubectl rollout undo deployment/ --to-revision= kubectl debug -it --image=busybox --target= get events is often faster than describing pods one by one, since scheduling failures, image pull errors and OOM kills all show up there. Check rollout history before rollout undo, because a plain undo only goes back one revision, and if you shipped two bad changes in a row you will land on the other broken one. kubectl debug adds an ephemeral container with a shell next to your app, which you will need the first time you try to exec into a distroless image and there is no sh in it. One last thing that has nothing to do with syntax: make sure you can actually reach the cluster from the machine you will grab when you get paged. The usual way is doctl kubernetes cluster kubeconfig save . DigitalOcean’s docs on connecting to a cluster note that on current Kubernetes versions this uses a revocable token, but on legacy versions the kubeconfig downloaded from the control panel is a certificate valid for 7 days, so an old file sitting on a laptop can quietly stop working. Running kubectl get nodes against production now and then, outside of an incident, catches that before it matters. As you go through the drills, write the commands that worked into a short runbook for your own cluster, with your real deployment and namespace names filled in. At 2am copying from your own notes is much faster than remembering. Regards

alexdo · Answer

Hi there, To add a couple of things on top of Bobby’s answer. You do not have to pay for a practice cluster at all. kind (Kubernetes in Docker) runs a full cluster on your laptop in about a minute, and you can delete it and start fresh as often as you like: kind create cluster --name drill kubectl create deployment web --image=nginx:1.27 --replicas=3 kubectl rollout status deployment/web What builds muscle memory is not running commands against a healthy cluster, it is having to figure out what broke. So break it on purpose, ideally have someone else (or a script) pick the failure so you do not know the answer in advance: # bad image tag, pods go into ImagePullBackOff kubectl set image deployment/web nginx=nginx:does-not-exist # container that crashes on start, CrashLoopBackOff kubectl create deployment crashy --image=busybox -- sh -c "exit 1" # memory limit far too low, usually ends in OOMKilled kubectl set resources deployment/web --limits=memory=4Mi Then work each one with the same sequence you would use at 2am: get pods, describe, logs --previous, get events, fix or roll back. Time yourself, and repeat the drill a week later. A few commands I would add to the list from the answer above: kubectl get events -A --sort-by=.lastTimestamp kubectl rollout history deployment/ kubectl rollout undo deployment/ --to-revision= kubectl debug -it --image=busybox --target= get events is often faster than describing pods one by one, since scheduling failures, image pull errors and OOM kills all show up there. Check rollout history before rollout undo, because a plain undo only goes back one revision, and if you shipped two bad changes in a row you will land on the other broken one. kubectl debug adds an ephemeral container with a shell next to your app, which you will need the first time you try to exec into a distroless image and there is no sh in it. One last thing that has nothing to do with syntax: make sure you can actually reach the cluster from the machine you will grab when you get paged. The usual way is doctl kubernetes cluster kubeconfig save . DigitalOcean’s docs on connecting to a cluster note that on current Kubernetes versions this uses a revocable token, but on legacy versions the kubeconfig downloaded from the control panel is a certificate valid for 7 days, so an old file sitting on a laptop can quietly stop working. Running kubectl get nodes against production now and then, outside of an incident, catches that before it matters. As you go through the drills, write the commands that worked into a short runbook for your own cluster, with your real deployment and namespace names filled in. At 2am copying from your own notes is much faster than remembering. Regards

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

How do I practice kubectl commands before I have to use them under pressure during an incident?

Related questions

Where do I start with DevOps as a beginner on DigitalOcean?

Where do I start learning Kubernetes as a Docker user?

Why the Digital Ocean K8s Cluster provisioning is taking long ( 4 hours and still in progress )

Why does the Digital Ocean K8s take long to provision the cluster ( 4 hrs and still in progress )

Is Kubernetes overkill for small projects in 2026?

What's your current production setup on DigitalOcean?

How do I add a manula droplet as node to a managed Kubernetes cluster?

How to retrieve the original IP from a HTTP request when using Cilium Gateway API on DOKS

P2P layers cache in DOKS cluster

DOKS Cilium GatewayClass is not working

Lets Encypt ssl certificate is not working

How to create a Kubernetes cluster using droplets ?

Report this

How do I practice kubectl commands before I have to use them under pressure during an incident?

Related questions

Where do I start with DevOps as a beginner on DigitalOcean?

Where do I start learning Kubernetes as a Docker user?

Why the Digital Ocean K8s Cluster provisioning is taking long ( 4 hours and still in progress )

Why does the Digital Ocean K8s take long to provision the cluster ( 4 hrs and still in progress )

Is Kubernetes overkill for small projects in 2026?

What's your current production setup on DigitalOcean?

How do I add a manula droplet as node to a managed Kubernetes cluster?

How to retrieve the original IP from a HTTP request when using Cilium Gateway API on DOKS

P2P layers cache in DOKS cluster

DOKS Cilium GatewayClass is not working

Lets Encypt ssl certificate is not working

How to create a Kubernetes cluster using droplets ?