Kubernetes Troubleshooting Deep Dive
Reading pod status like a diagnosis
Most pod statuses point at one layer of the stack. Knowing which layer saves you from reading application logs when the problem is the image registry.
| Status | What it means | Where to look |
|---|---|---|
| CrashLoopBackOff | Container starts, exits, gets backed off | logs --previous |
| ImagePullBackOff | Wrong tag, private registry, missing pull secret | describe → Events |
| Pending | Nothing scheduled it: no capacity, taint, or unbound PVC | describe → Events, get nodes |
| OOMKilled (137) | Hit the memory limit and was killed | top pods, limits in the spec |
| Running 0/1 | Process is up but readiness probe fails | describe → probe config |
Why an empty endpoint list is so common
The chain from request to container has four links, and the middle one is invisible in most dashboards:
Ingress -> Service -> Endpoints -> Pod
^
label selector must match pod labels exactlyNothing warns you when a selector matches nothing. The Service is created successfully, the Ingress reports healthy, and requests silently go nowhere. Three habits prevent it: check kubectl get ep right after any Service change, keep the selector and pod labels in the same Helm template so they cannot drift, and alert on an endpoint count of zero rather than on pod health alone.
Rollback first — the order matters
During an incident the goal is not understanding, it is restoring service. `rollout undo` is fast precisely because it changes nothing but a pointer: the previous ReplicaSet still exists, its images are still on the nodes, and the new pods start in seconds. Investigate the root cause afterwards, from logs you already captured. The one thing to remember is that a live `kubectl patch` or `rollout undo` does not update git — if the fix is not committed, the next deploy restores the outage.
The on-call command set
# What is unhealthy across the whole cluster
kubectl get pods -A --field-selector=status.phase!=Running
# Recent events, newest last — the fastest situational overview
kubectl get events -n payments --sort-by=.lastTimestamp | tail -20
# The log of the container that just died
kubectl logs <pod> -n payments --previous --tail=200
# Does the Service actually have backends?
kubectl get ep <svc> -n payments
# Roll back and block until it converges
kubectl rollout undo deploy/<name> -n payments
kubectl rollout status deploy/<name> -n payments --timeout=120s
# Reach a pod directly, bypassing Ingress and Service
kubectl port-forward pod/<name> 8080:8080 -n payments
