Kubernetes Security andamp; Operations tutorial series

The Kubernetes error context deadline exceeded means an operation did not complete before its deadline. It does not identify the failed component by itself. The timeout may occur between kubectl and kube-apiserver, kube-apiserver and etcd, kubelet and containerd, containerd and an image registry, a Pod and CoreDNS, or a node and the Kubernetes Service IP.

This guide provides a safe diagnostic workflow for Ubuntu 24.04 nodes running a kubeadm-based Kubernetes cluster with containerd. It explains how to separate DNS timeouts, TCP connection timeouts, TLS failures, API server health problems, etcd latency, kube-proxy failures, and CNI or Calico issues without deleting state or restarting every component blindly.

Table of Contents

What Does “context deadline exceeded” Mean in Kubernetes?

Kubernetes components use contexts to bound API calls, runtime operations, admission requests, image pulls, DNS queries, and other work. When the deadline expires, the caller returns context deadline exceeded. The caller may be healthy while the destination, network path, DNS resolver, TLS handshake, storage backend, or runtime is slow or unreachable.

Always identify these four facts before attempting a repair:

  • Caller: kubectl, kubelet, containerd, CNI plugin, controller, scheduler, or application.
  • Destination: API endpoint, Service ClusterIP, etcd, registry, DNS server, webhook, or external dependency.
  • Protocol stage: DNS resolution, TCP connection, TLS handshake, HTTP request, or application processing.
  • Scope: one Pod, one node, all workers, one control-plane node, or the entire cluster.

If the error specifically references https://10.96.0.1:443 while Calico creates a Pod sandbox, see the related guide: Calico Failed to Create Pod Sandbox: Kubernetes API 10.96.0.1:443 Timeout.

Preserve Evidence and Identify the Failing Path

Record the timestamp, command, namespace, Pod, node, recent rollout, and complete error. Do this before restarting kubelet, containerd, CoreDNS, kube-proxy, or control-plane static Pods.

date -Is
kubectl version
kubectl config current-context
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.metadata.creationTimestamp

If normal API calls time out, use short client-side request timeouts so the terminal does not wait indefinitely:

kubectl --request-timeout=5s get --raw=/livez
kubectl --request-timeout=5s get --raw=/readyz
kubectl --request-timeout=10s get nodes

A short timeout is a diagnostic control, not the fix. Increasing --request-timeout can hide latency while the underlying failure continues.

On the affected Ubuntu 24.04 node, capture service and kernel state:

sudo systemctl status kubelet containerd --no-pager
sudo journalctl -u kubelet --since "30 minutes ago" --no-pager
sudo journalctl -u containerd --since "30 minutes ago" --no-pager
uptime
free -h
df -hT
df -ih
sudo dmesg --level=err,warn

Look for memory pressure, OOM kills, a full filesystem, inode exhaustion, clock problems, repeated sandbox creation, runtime RPC timeouts, or failed network setup.

Distinguish DNS Timeout, TCP Timeout, TLS Failure, and an Unhealthy API Server

DNS timeout

DNS problems contain messages such as i/o timeout while contacting a resolver, server misbehaving, or a lookup failure for an API server or registry hostname. Confirm resolution independently:

getent hosts CONTROL_PLANE_HOSTNAME
resolvectl status
resolvectl query CONTROL_PLANE_HOSTNAME
dig CONTROL_PLANE_HOSTNAME A
dig CONTROL_PLANE_HOSTNAME AAAA

Replace CONTROL_PLANE_HOSTNAME with the hostname in the failing kubeconfig or registry request. An IP-based API endpoint can work while a hostname-based endpoint fails, which strongly points toward resolver configuration rather than kube-apiserver.

TCP connection timeout or refusal

A TCP timeout means packets or replies may be dropped by routing, firewall, security groups, network policy, or a failed path. Connection refused is different: the target host responded, but nothing accepted the port.

ip -br address
ip route
ip route get CONTROL_PLANE_IP
ping -c 3 CONTROL_PLANE_IP
nc -vz -w 3 CONTROL_PLANE_IP 6443
ss -lntp

Do not assume a successful ping proves TCP port 6443 works. ICMP and TCP can follow different policies. Conversely, ping may be blocked while TCP works.

TLS handshake or certificate failure

TLS errors normally name certificate validation, hostname mismatch, expiration, an unknown authority, or a handshake timeout. Inspect the presented certificate without sending credentials:

openssl s_client 
  -connect CONTROL_PLANE_IP:6443 
  -servername CONTROL_PLANE_HOSTNAME 
  -showcerts </dev/null

On a kubeadm control-plane node, check managed certificate expiration:

sudo kubeadm certs check-expiration
kubectl config view --minify
timedatectl status

Do not use curl -k, --insecure-skip-tls-verify, or disable certificate verification as a permanent fix. Verify the correct CA, endpoint hostname, system time, and certificate renewal procedure.

Unhealthy or overloaded kube-apiserver

If DNS, routing, TCP, and TLS succeed but API requests time out or /readyz reports failures, investigate kube-apiserver, etcd, admission webhooks, CPU, memory, and disk latency.

kubectl --request-timeout=5s get --raw='/livez?verbose'
kubectl --request-timeout=5s get --raw='/readyz?verbose'
kubectl --request-timeout=10s get --raw='/version'
kubectl get --raw='/metrics' | head

Access to some endpoints depends on authorization. An authorization error is evidence that DNS, TCP, and TLS completed; it is not a network timeout.

Check kubectl and kubeconfig First

A stale context, wrong API address, proxy variable, or missing client credential can make a healthy cluster appear unavailable.

kubectl config get-contexts
kubectl config current-context
kubectl config view --minify
kubectl auth whoami
env | grep -E '^(HTTP|HTTPS|NO)_PROXY='

Confirm that NO_PROXY covers the API hostname or address where an HTTP proxy is configured. Do not print raw kubeconfig data into shared logs because embedded client credentials may be sensitive.

Use verbose kubectl logging only for a controlled diagnostic capture and review it before sharing:

kubectl --request-timeout=10s -v=8 get nodes

If kubectl cannot reach the API at all, run node-local checks rather than depending on additional API calls.

Check kube-apiserver on a kubeadm Control Plane

kubeadm commonly runs kube-apiserver, scheduler, controller-manager, and local etcd as static Pods. Kubelet watches manifests under /etc/kubernetes/manifests, while containerd runs the resulting containers.

sudo ls -l /etc/kubernetes/manifests/
sudo systemctl status kubelet containerd --no-pager
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a --name kube-apiserver

Get the real container ID from crictl ps -a, then inspect it:

sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock inspect APISERVER_CONTAINER_ID
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs APISERVER_CONTAINER_ID

Check the manifest without editing it:

sudo grep -E -- 
  '--advertise-address|--bind-address|--secure-port|--etcd-servers' 
  /etc/kubernetes/manifests/kube-apiserver.yaml
sudo ss -lntp | grep ':6443'

Look for etcd timeouts, TLS errors, admission webhook timeouts, request throttling, failed storage initialization, or a listener missing from port 6443. Do not move or rewrite a static Pod manifest until you understand the effect: kubelet reacts immediately to files in the manifest directory.

Check etcd Health and Latency

kube-apiserver depends on etcd for cluster state. A slow or unhealthy etcd can leave port 6443 reachable while API requests exceed their deadlines.

On a stacked kubeadm control plane, locate the etcd container:

sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a --name etcd
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs ETCD_CONTAINER_ID

If the etcd container includes etcdctl, run the health check inside that exact container using kubeadm’s health-check client certificates:

sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock exec 
  ETCD_CONTAINER_ID 
  etcdctl 
  --endpoints=https://127.0.0.1:2379 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt 
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key 
  endpoint health

sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock exec 
  ETCD_CONTAINER_ID 
  etcdctl 
  --endpoints=https://127.0.0.1:2379 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt 
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key 
  endpoint status --write-out=table

Do not run member removal, snapshot restore, defragmentation, or data-directory cleanup as an exploratory fix. First inspect disk space, inode availability, I/O latency, quorum, alarms, and the documented backup state.

df -hT /var/lib/etcd
df -ih /var/lib/etcd
sudo du -sh /var/lib/etcd
iostat -xz 1

Check CoreDNS and Cluster DNS

If API access from the administrator workstation works but Pods time out resolving Services or registries, examine CoreDNS and the DNS Service.

kubectl -n kube-system get deployment coredns
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system get service kube-dns
kubectl -n kube-system get endpointslice 
  -l k8s.io/service-name=kube-dns
kubectl -n kube-system logs deployment/coredns --tail=200
kubectl -n kube-system get configmap coredns -o yaml

Test from an existing Pod that has a DNS client:

kubectl exec -n NAMESPACE POD_NAME -- cat /etc/resolv.conf
kubectl exec -n NAMESPACE POD_NAME -- nslookup kubernetes.default.svc.cluster.local
kubectl exec -n NAMESPACE POD_NAME -- nslookup registry-1.docker.io

If no suitable Pod exists and image pulls are healthy, create the Kubernetes documentation DNS test Pod:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
kubectl delete pod dnsutils

When image pulls are already failing, creating a new test Pod may add noise. Test node DNS first with getent, resolvectl, and dig.

Check containerd, crictl, and Image Pulls

Kubelet communicates with containerd over CRI. A runtime timeout can occur even when kube-apiserver is healthy.

sudo systemctl status containerd --no-pager
sudo journalctl -u containerd --since "30 minutes ago" --no-pager
sudo ctr plugins ls
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock info
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock pods
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock images

Inspect the affected Pod and recent events before pulling manually:

kubectl describe pod POD_NAME -n NAMESPACE
kubectl get events -n NAMESPACE 
  --field-selector involvedObject.name=POD_NAME 
  --sort-by=.metadata.creationTimestamp

Test the exact image through CRI:

sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock pull 
  REGISTRY/REPOSITORY:TAG

Then separate node DNS, TCP, and TLS access to the registry:

getent hosts REGISTRY_HOST
nc -vz -w 3 REGISTRY_HOST 443
openssl s_client 
  -connect REGISTRY_HOST:443 
  -servername REGISTRY_HOST 
  </dev/null
curl -I --connect-timeout 3 https://REGISTRY_HOST/v2/

A registry may intentionally return an authentication-related HTTP response from /v2/. The diagnostic objective is to prove DNS, TCP, TLS, and HTTP completed, not to assume every registry returns status 200.

For a broader workflow covering ImagePullBackOff, Pod DNS, Pending Pods, and CrashLoopBackOff, see Kubernetes Troubleshooting: CrashLoopBackOff, Pending Pods, DNS, and ImagePullBackOff.

Check kube-proxy and Kubernetes Service Routing

If direct access to a control-plane address works but a Service ClusterIP such as the default Kubernetes Service times out, inspect the Service, EndpointSlices, and node Service routing.

kubectl get service kubernetes -n default -o wide
kubectl get endpointslice -n default 
  -l kubernetes.io/service-name=kubernetes -o wide
kubectl -n kube-system get daemonset kube-proxy
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide
kubectl -n kube-system logs daemonset/kube-proxy --tail=200

On the affected node, inspect without flushing rules:

ip route
sudo nft list ruleset
sudo iptables-save
sudo ipvsadm -Ln

ipvsadm is relevant only when IPVS rules are in use and the utility is installed. Some clusters intentionally replace kube-proxy with an eBPF service implementation; confirm the cluster architecture before treating a missing kube-proxy DaemonSet as a fault.

Never flush iptables, nftables, IPVS, or routes over an SSH connection as an initial diagnostic step. Doing so can disconnect the node and destroy evidence.

Check CNI and Calico

CNI failures usually appear during Pod sandbox creation. First determine whether the failure is node-specific and whether the CNI cannot reach the API, allocate an address, program routes, or create interfaces.

kubectl get nodes -o wide
kubectl get pods -A -o wide --field-selector spec.nodeName=NODE_NAME
kubectl -n kube-system get pods -o wide
sudo ls -la /etc/cni/net.d/
sudo ls -la /opt/cni/bin/
sudo journalctl -u kubelet --since "30 minutes ago" --no-pager
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock pods

For a common Calico installation:

kubectl -n kube-system get daemonset calico-node
kubectl -n kube-system get pods -l k8s-app=calico-node -o wide
kubectl -n kube-system logs daemonset/calico-node 
  -c calico-node --tail=200

Resource names and namespaces can differ by installation method. List the actual objects before copying a log command.

Compare node access to the direct kube-apiserver endpoint with access to the Kubernetes Service ClusterIP. Direct success plus ClusterIP timeout points toward Service routing, kube-proxy, or the eBPF replacement. Both failing points earlier in the path: API listener, routing, firewall, or control-plane health.

For installation and network verification concepts, see How to Install Cilium CNI and Verify Kubernetes Pod Networking. The commands specific to Cilium are not interchangeable with Calico, but the verification layers—node, CNI Pod, Pod address, Service routing, DNS, and policy—are useful for either implementation.

Decision Table: Symptom, Likely Cause, and Next Command

Observed symptom Likely layer Run next
Hostname lookup reports timeout or resolver failure Node DNS, Pod DNS, CoreDNS, or upstream resolver resolvectl query HOST, getent hosts HOST, CoreDNS logs
dial tcp ... i/o timeout Route, firewall, network path, policy, or unreachable listener ip route get IP, nc -vz -w 3 IP PORT
connection refused Host reachable but no listener, wrong address, or failed component ss -lntp, component status and logs
x509: certificate signed by unknown authority Wrong CA or incomplete trust chain openssl s_client, inspect kubeconfig CA reference
Certificate expired or not valid yet Expired kubeadm certificate or incorrect system time kubeadm certs check-expiration, timedatectl
TCP and TLS work, but /readyz fails or times out kube-apiserver dependency, etcd, webhook, or overload kubectl get --raw='/readyz?verbose', API and etcd logs
Port 6443 is not listening on a control plane kube-apiserver static Pod or kubelet/containerd failure crictl ps -a --name kube-apiserver, kubelet and API logs
API endpoint works directly, but 10.96.0.1:443 times out kube-proxy, Service rules, eBPF service routing, or CNI path Service/EndpointSlice, kube-proxy logs, nftables/iptables/IPVS inspection
etcd health is slow or unavailable etcd quorum, disk, TLS, resource pressure, or corruption etcdctl endpoint health, etcd logs, iostat -xz 1
Only one node fails Pod sandbox creation Node kubelet, containerd, CNI files, routes, or host firewall node kubelet/containerd logs, CNI directory, CNI DaemonSet Pod
Image pull times out before authentication Node DNS, TCP, TLS, proxy, or registry path crictl pull, getent, nc, openssl s_client
Image pull returns unauthorized Registry credentials or imagePullSecret kubectl describe pod, inspect Secret reference without printing secret values
Pods cannot resolve kubernetes.default CoreDNS, kube-dns Service, EndpointSlice, Pod routing, or policy DNS Pod test, CoreDNS logs, kube-dns Service and endpoints

Verify the Repair End to End

Verification should prove every layer required by the original operation. Do not stop after one component becomes Running.

kubectl --request-timeout=5s get --raw='/livez?verbose'
kubectl --request-timeout=5s get --raw='/readyz?verbose'
kubectl get nodes
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.metadata.creationTimestamp

Verify the control plane and node services:

sudo systemctl is-active kubelet containerd
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock info
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps

Verify Service routing and DNS from a suitable existing Pod:

kubectl exec -n NAMESPACE POD_NAME -- 
  nslookup kubernetes.default.svc.cluster.local

kubectl exec -n NAMESPACE POD_NAME -- 
  nc -vz -w 3 kubernetes.default.svc 443

The second command requires nc in the selected container and verifies DNS plus TCP Service routing. It does not verify TLS, authentication, authorization, or API readiness. Use the authenticated kubectl health checks for those layers. Do not place administrative tokens in the command line.

Finally, retry the exact operation that originally failed and confirm that new events and logs no longer show the timeout:

kubectl describe pod POD_NAME -n NAMESPACE
kubectl get events -n NAMESPACE 
  --field-selector involvedObject.name=POD_NAME 
  --sort-by=.metadata.creationTimestamp
sudo journalctl -u kubelet --since "10 minutes ago" --no-pager

Kubernetes Context Deadline Exceeded Checklist

  • Record the caller, destination, timestamp, node, namespace, and complete error.
  • Confirm the current kubectl context and API endpoint.
  • Test DNS resolution separately from TCP port reachability.
  • Inspect the TLS certificate and system time separately from HTTP readiness.
  • Check /livez and /readyz with a short diagnostic timeout.
  • On kubeadm control planes, inspect static Pod containers with crictl.
  • Check etcd health, logs, disk blocks, inodes, and I/O latency.
  • Check CoreDNS Pods, Service, EndpointSlices, configuration, and Pod resolver settings.
  • Check kubelet and containerd before retrying image pulls.
  • Use crictl pull for the exact image and registry path.
  • Compare direct API access with Kubernetes Service ClusterIP access.
  • Inspect kube-proxy or the configured eBPF Service implementation.
  • Inspect CNI files and the actual Calico/Cilium DaemonSet on the affected node.
  • Apply one reversible fix and repeat the original operation.
  • Do not flush networking rules, reset kubeadm, delete etcd state, or disable TLS verification as a first response.

Frequently Asked Questions

Is context deadline exceeded always a network problem?

No. DNS and TCP failures are common, but a healthy connection can still time out because kube-apiserver is overloaded, etcd is slow, an admission webhook does not respond, containerd is blocked, or disk and memory pressure delay the operation.

Should I increase the kubectl timeout?

Use a short explicit timeout to make diagnostics predictable. Increasing the timeout is appropriate only when the operation is legitimately slow and the component is healthy. It should not replace investigation of new or repeated latency.

Why does the API server IP work while 10.96.0.1 times out?

The direct endpoint and the Kubernetes Service ClusterIP use different paths. If direct port 6443 works but the Service IP fails, investigate the Service and EndpointSlice, kube-proxy or its eBPF replacement, node rules, and CNI routing.

Can I restart kubelet or containerd to fix the timeout?

A controlled restart may be part of a confirmed repair, but restarting first destroys timing and state evidence and can disrupt healthy Pods. Capture status, logs, runtime state, resources, and network information before restarting.

How do I know whether etcd is responsible?

TCP and TLS to kube-apiserver may succeed while /readyz, API operations, or API logs show etcd failures. Confirm with kube-apiserver logs, etcd logs, authenticated etcdctl endpoint health, and disk latency checks.

What is the safest first command?

Start with the exact failing command using a bounded request timeout, then inspect events and the relevant component logs. Read-only commands such as kubectl get, kubectl describe, journalctl, crictl ps, ip route, and health endpoints preserve the evidence needed to choose a repair.

Conclusion

To fix context deadline exceeded in Kubernetes, trace the operation layer by layer: name resolution, route, TCP connection, TLS handshake, API health, etcd, runtime, Service routing, and CNI. The error is a deadline symptom, not a diagnosis. A safe investigation identifies the caller and destination, uses read-only checks first, changes one confirmed cause, and verifies the original workflow end to end.

Official References

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *