Calico Failed to Create Pod Sandbox: Kubernetes API 10.96.0.1:443 Timeout

A Calico CNI failure such as failed to create pod sandbox combined with dial tcp 10.96.0.1:443: i/o timeout means the CNI component cannot reach the Kubernetes API through the in-cluster kubernetes.default Service. The fastest investigation is to test the Service object and endpoints, then inspect kube-proxy on the affected node, and finally verify the real kube-apiserver.

Typical error

Failed to create pod sandbox: rpc error: code = Unknown desc =
failed to setup network for sandbox: plugin type="calico" failed (add):
Get "https://10.96.0.1:443/api/v1/nodes/...":
dial tcp 10.96.0.1:443: i/o timeout

The failure happens before the workload network is ready. Restarting the application Deployment will not fix it because kubelet will continue calling the same broken CNI path.

1. Identify the affected node

kubectl get pods -A -o wide
kubectl get pods -A --field-selector=status.phase=Pending -o wide
kubectl describe pod POD_NAME -n NAMESPACE
kubectl get events -A --sort-by=.lastTimestamp | tail -n 80

Record the node name from the Pending Pod. If failures occur on only one node, prioritize that node’s kube-proxy, firewall, conntrack, and kernel rules. If every node fails, prioritize the API Service, endpoints, control plane, or a cluster-wide proxy configuration.

2. Verify the Kubernetes API Service and endpoint

kubectl get service kubernetes -n default -o wide
kubectl get endpoints kubernetes -n default -o yaml
kubectl get endpointslice -n default 
  -l kubernetes.io/service-name=kubernetes -o wide
kubectl get --raw='/readyz?verbose'

The Service should contain ClusterIP 10.96.0.1 when that is the first address in the configured Service CIDR. Its endpoint should point to a reachable control-plane address and secure API port, commonly 6443. An empty or stale endpoint must be corrected before touching Calico.

3. Compare ClusterIP access with direct API access

Run these checks on the affected node. Replace the direct address with the endpoint reported in the previous step.

curl -k --connect-timeout 3 https://10.96.0.1:443/livez
curl -k --connect-timeout 3 https://CONTROL_PLANE_IP:6443/livez
nc -vz -w 3 10.96.0.1 443
nc -vz -w 3 CONTROL_PLANE_IP 6443

• If the direct API endpoint works but 10.96.0.1:443 times out, investigate kube-proxy and Service rules.
• If both addresses time out, investigate kube-apiserver, routing, firewall rules, or control-plane availability.
• If only one node fails, compare its proxy and firewall state with a healthy node.

4. Check kube-proxy

kubectl -n kube-system get daemonset kube-proxy
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide
kubectl -n kube-system describe pod -l k8s-app=kube-proxy
kubectl -n kube-system logs -l k8s-app=kube-proxy 
  --prefix --tail=200
kubectl -n kube-system get configmap kube-proxy -o yaml

Confirm that one Ready kube-proxy Pod runs on every node and look for API connection failures, rule-sync errors, invalid configuration, missing kernel modules, or mode changes. Determine whether the cluster uses iptables, IPVS, or nftables before inspecting the data plane.

iptables mode

sudo iptables-save -t nat | grep -E '10.96.0.1|default/kubernetes:https'
sudo iptables -t nat -L KUBE-SERVICES -n -v | grep '10.96.0.1'
sudo sysctl net.ipv4.ip_forward
sudo conntrack -S

IPVS mode

sudo ipvsadm -Ln | grep -A4 '10.96.0.1:443'
ip address show kube-ipvs0
lsmod | grep -E '^ip_vs|nf_conntrack'

If the API Service rule is absent only on the affected node, restart that node’s kube-proxy Pod and watch it rebuild state:

PROXY_POD=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy 
  --field-selector spec.nodeName=AFFECTED_NODE -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system delete pod "$PROXY_POD"
kubectl -n kube-system get pod -l k8s-app=kube-proxy -o wide -w

Do not flush all iptables or IPVS state on a production node as a first response. That can interrupt every Service and make the original evidence disappear.

5. Check kube-apiserver

In a kubeadm cluster, kube-apiserver normally runs as a static Pod on each control-plane node.

kubectl -n kube-system get pods -l component=kube-apiserver -o wide
kubectl -n kube-system describe pod -l component=kube-apiserver
kubectl -n kube-system logs -l component=kube-apiserver 
  --prefix --tail=200
kubectl get --raw='/livez?verbose'
kubectl get --raw='/readyz?verbose'

When kubectl cannot reach the cluster, log in to a control-plane node and use the container runtime and local health endpoint:

sudo crictl ps --name kube-apiserver
APISERVER_ID=$(sudo crictl ps --name kube-apiserver -q | head -n1)
sudo crictl logs --tail=200 "$APISERVER_ID"
sudo ss -lntp | grep 6443
curl -k --connect-timeout 3 https://127.0.0.1:6443/livez
sudo journalctl -u kubelet -n 200 --no-pager
sudo sed -n '1,240p' /etc/kubernetes/manifests/kube-apiserver.yaml

Check expired certificates, invalid manifest flags, an unavailable etcd, disk pressure, time drift, firewall changes, and bind-address mistakes. Do not edit the static Pod manifest merely to force a restart; kubelet watches that file and every change restarts the API server.

6. Check Calico after API connectivity is restored

kubectl -n kube-system get pods -l k8s-app=calico-node -o wide
kubectl -n kube-system logs -l k8s-app=calico-node 
  -c calico-node --prefix --tail=200
kubectl -n kube-system get daemonset calico-node
kubectl get nodes -o wide
sudo journalctl -u kubelet -n 200 --no-pager

Calico logs should stop reporting timeouts to 10.96.0.1:443. A new sandbox attempt should complete without manually restarting every application.

7. Verify the fix end to end

kubectl run calico-api-test --image=busybox:1.37 
  --restart=Never -- sleep 300
kubectl wait --for=condition=Ready pod/calico-api-test --timeout=90s
kubectl get pod calico-api-test -o wide
kubectl describe pod calico-api-test
kubectl delete pod calico-api-test

Quick decision table

• Direct API works; ClusterIP fails: check kube-proxy, iptables/IPVS/nftables, conntrack, and host firewall.
• Direct API and ClusterIP both fail: check kube-apiserver, etcd, control-plane routing, certificates, and port 6443.
• One node fails: compare kube-proxy and host networking on that node.
• All nodes fail: check the Kubernetes Service endpoints and control plane first.

Production checklist

• The affected node and first failing timestamp were recorded.
• kubernetes.default has the expected ClusterIP and a valid endpoint.
• Direct API connectivity and Service-IP connectivity were tested separately.
• kube-proxy mode and programmed rules were verified before restart.
• kube-apiserver livez, readyz, logs, etcd, certificates, and listener were checked.
• A new Pod sandbox was created successfully after the repair.

References: Kubernetes Service debugging, Virtual IPs and Service proxies, and Calico system requirements.