Kubernetes Connectivity Troubleshooting: A Layer-by-Layer Guide

Most Kubernetes connectivity problems are solved the same way: work out which hop is failing, then test only that hop. Your laptop to the API server is one hop. A pod to cluster DNS is another. A Service to its pods is a third. A NetworkPolicy sits across several of them. This guide walks those layers in order, with the commands to run and what each result means, including the two errors people search for most: connection refused from kubectl and TLS handshake timeout.

TL;DR

• Write down the source, the destination, the port and the exact error text before you change anything. The wording of the error tells you which layer to test.
• connection refused from kubectl usually points at the kubeconfig or an API server endpoint that is not listening. A timeout points at something dropping traffic on the way.
• Inside the cluster, test by DNS name first, then by Service IP, then check that the Service has endpoints.
• A NetworkPolicy can make a healthy Service time out for pods that lack the right labels.
• Keep kubeconfigs, tunnels and logs in one place so you stop guessing which cluster you are talking to.

Table of Contents

• Start by naming the failing hop
• Layer 1: can your client reach the API server?
• Layer 2: does DNS work inside the cluster?
• Layer 3: does the Service actually point at pods?
• Layer 4: NetworkPolicy, firewalls and proxies
• Reading the results: a symptom table
• Our perspective: stop debugging the tooling
• Preventing the same failure next week
• Troubleshoot clusters from one window
• FAQ
• Sources

Start by naming the failing hop

Before you run anything, answer four questions in a scratch file:

1. Where is the request coming from? Your workstation, a CI runner, a pod in a particular namespace, or a node.
2. Where is it going? The API server, a Service name, a Service IP, a pod IP, or an external host.
3. On which port and protocol?
4. What is the exact error? Copy it. "Can't connect" is not an error message; dial tcp 10.0.0.5:6443: connect: connection refused is.

This matters because the same symptom, "the app can't reach the thing," has at least four different causes: name resolution failed, there is no route, traffic was dropped by policy or a firewall, or the destination answered with an error. Each needs a different tool. If you test them in a fixed order you stop chasing the wrong one.

The order we use:

1. Can the client reach the API server at all?
2. Does DNS resolve inside the cluster?
3. Does the Service have healthy endpoints?
4. Is a NetworkPolicy, firewall or proxy in the way?

Layer 1: can your client reach the API server?

If kubectl get pods fails, nothing inside the cluster is your problem yet. Start with what kubectl is actually trying to talk to.

kubectl config current-context
kubectl config view --minify | grep server
echo $KUBECONFIG

The first command shows which cluster you are pointed at. Wrong context is the most common cause of "it worked yesterday." The second shows the API server address that kubectl will dial. The third shows whether an environment variable is overriding the default file.

kubectl connection refused

A refused connection means your machine reached an address and nothing was listening on that port, or something actively rejected it. The official kubectl troubleshooting guide lists the first checks: confirm you have a valid kubeconfig, that KUBECONFIG or --kubeconfig points to it (the default file is usually ~/.kube/config), and that the API server host and any load balancer in front of it are reachable and healthy.

A practical sequence:

1. Is there a kubeconfig at all? If kubectl falls back to localhost:8080, it found no usable configuration. That is a configuration problem, not a network one.
2. Did sudo strip the environment? Running sudo kubectl can lose your KUBECONFIG variable and send you back to the defaults. Run kubectl as your own user, or preserve the environment deliberately.
3. Does the address in the server: field match the cluster you mean? Local clusters get recreated, and their ports change. A stale kubeconfig points at a port nobody listens on.
4. Is the API server running? On a self-managed control plane, check the control plane node. On a managed service, check the provider's status page and the cluster state in its console.
5. Test the port directly. nc -vz <host> <port> from the same machine tells you whether TCP connects, independent of kubectl.

If nc is refused too, the problem is the endpoint or the path to it, and kubectl is innocent.

Timeouts and TLS handshake timeout

A timeout is different from a refusal. Nothing said "no"; the packets simply got no answer. That usually means a firewall, security group, route or proxy is dropping traffic, or the server is too overloaded to respond.

A TLS handshake timeout fits that pattern: the TCP connection was opened, but the server did not complete the TLS exchange in time. Work through it in this order:

• Rule out the network path. Run curl -vk https://<api-host>:<port>/version and watch where it stalls. If it never connects, it is routing or a firewall. If it connects and stalls during the handshake, look at the server side or at a middlebox that is intercepting TLS.
• Check proxies. HTTPS_PROXY and NO_PROXY variables in your shell can send API traffic through a proxy that cannot handle it. Print them with env | grep -i proxy.
• Check certificate validity. The Kubernetes documentation notes that TLS problems can come from certificate expiry or a broken chain of trust, and suggests openssl x509 -noout -dates to verify the dates on the certificate. Run it against the certificate the server presents or the one in your kubeconfig.
• Check the clock. A workstation or node with a skewed clock will reject certificates that are perfectly valid.
• Consider server load. On managed clusters, Microsoft's guidance on TCP time-outs when kubectl connects to the API server names several causes that apply well beyond AKS: blocked required ports or addresses, a client IP that is not authorized to reach the API server, and leaked API calls that overload it.

Never "fix" a handshake timeout by disabling certificate verification in your kubeconfig and leaving it that way. --insecure-skip-tls-verify is a diagnostic step to learn whether the certificate is the problem, and the setting should come out again the moment you know.

If the cluster sits behind a bastion, the failure may be in the tunnel and not the cluster. Our walkthrough on SSH tunnels to a Kubernetes API server covers the certificate-name errors that appear when you connect through a forwarded local port.

Layer 2: does DNS work inside the cluster?

Once kubectl works, move into the cluster. Many "network" failures are name resolution failures. Launch a throwaway debug pod in the namespace where the problem occurs:

kubectl run dbg --rm -it --image=nicolaka/netshoot -n <namespace> -- bash

From inside it:

nslookup kubernetes.default
nslookup <service-name>.<namespace>.svc.cluster.local
nslookup example.com

Read the three results as a set:

• Internal names fail, external names work. Cluster DNS is broken or the pod's resolv.conf is wrong. Check that the DNS pods in kube-system are running and look at their logs.
• Internal works, external fails. The DNS pods cannot reach the upstream resolver. That is usually an egress rule or a node-level resolver problem.
• Everything fails with a timeout. The pod cannot reach the DNS Service at all. Suspect a NetworkPolicy that blocks egress to port 53, or a CNI problem on that node.
• NXDOMAIN for a name that should exist. Check the namespace and the spelling. A Service called api in prod is not found by api.staging.

A useful principle: if the DNS server answers "no such name," you did communicate with it. That is a different problem from not reaching it. Keep the two apart.

Layer 3: does the Service actually point at pods?

If DNS resolves but the connection fails, look at the Service. The Kubernetes guide to debugging Services starts from a sensible order: when a Service is unreachable from another pod, test it by its cluster DNS name first, then by its Service IP. The split tells you whether DNS or the Service layer is at fault.

curl -sv http://<service-name>.<namespace>:<port>/
curl -sv http://<cluster-ip>:<port>/

Then check what stands behind the Service:

kubectl get svc <service-name> -n <namespace> -o wide
kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service-name>
kubectl get pods -n <namespace> --show-labels

The usual findings:

• No endpoints. The Service selector does not match any pod labels, or the matching pods are not Ready. A one-character label typo is the classic cause. Compare the selector with the pod labels line by line.
• Endpoints exist but the connection is refused. The Service targetPort does not match the port the container listens on, or the app bound to 127.0.0.1 instead of 0.0.0.0.
• Pods are not Ready. A failing readiness probe removes a pod from the endpoints without crashing it. Read kubectl describe pod and the events.
• Works by pod IP, fails by Service IP. Suspect kube-proxy or the CNI's Service handling on the node where the client pod runs.

When it is intermittent, compare which node the failing client pods run on with kubectl get pods -o wide. Failures tied to one node point to that node's networking, not the Service.

Layer 4: NetworkPolicy, firewalls and proxies

If the Service is healthy and DNS resolves, something may be filtering traffic. The Kubernetes documentation on declaring a network policy shows the behavior that surprises people: with an ingress policy in place, a request from a pod without the required label times out, while the same request from a labeled pod succeeds. The symptom is a timeout, not a refusal, which is why it gets mistaken for a network fault.

Steps:

1. kubectl get networkpolicy -A to see which policies exist.
2. kubectl describe networkpolicy <name> -n <namespace> to read the pod selector, the ingress sources and the egress rules.
3. Run the same request from a pod that matches the allowed labels. If it works there and fails from the original pod, the policy is the cause.
4. Remember egress. A default-deny egress policy blocks DNS too, so everything looks broken at once.

Policy enforcement depends on the CNI plugin, so on a cluster that does not enforce policies, these objects do nothing. If you run a managed service with flow observability, use it. For example, GKE's network observability tooling uses Dataplane V2, Hubble and Cloud Monitoring to surface packet drops, NetworkPolicy blocks and traffic imbalances, which beats guessing.

Outside the cluster, firewall rules, cloud security groups and corporate proxies are the other suspects. They produce the same signature: timeouts, with nothing in the application logs because the packets never arrived.

Reading the results: a symptom table

Our perspective: stop debugging the tooling

When we talk to engineers who manage several clusters, a surprising share of "connectivity problems" turn out to be bookkeeping. The wrong context was active. An old kubeconfig pointed at a recreated cluster. A tunnel was closed in one terminal tab while kubectl ran in another. A certificate error appeared because the tunnel's local port did not match the name on the certificate.

That is why we built Kubezilla Desktop as a native app that keeps kubeconfigs in secure local storage, with no cloud sync and no telemetry, and opens SSH tunnels from the same window where you browse resources. Fewer tabs and fewer CLI tools mean fewer places for state to drift. It is written in Rust, so it starts fast and stays light on memory, which matters when you are debugging with a laptop already running a VPN, a browser and a container runtime.

Our advice is independent of any tool. Fix the order in which you test: client, DNS, Service, policy. Write the exact error down. Change one thing at a time. And make it hard to confuse one cluster with another, because that mistake costs more hours than any CNI bug.

— Margus

Preventing the same failure next week

A few habits remove most repeat incidents:

• Name and verify contexts. Use clear context names, and check the active one before any change. If you manage several clouds, our piece on multi-cloud Kubernetes management covers how to keep that manageable.
• Watch certificate expiry. Check dates on a schedule instead of finding out from a handshake error.
• Be gentle with the API server. Microsoft's guidance recommends using watches rather than frequent GET calls, and avoiding leaked calls or watch connections, because accumulated requests can overload the API server and cause the timeouts you then have to debug. Review scripts and dashboards that poll in a loop.
• Test policies before enforcing them. Validate changes with a server-side dry run so admission checks run before anything lands. We explain the differences in Kubernetes dry runs.
• Keep a debug image ready. A pod with curl, nslookup and tcpdump saves ten minutes every time. For shell access to running pods, see our guide to a faster Kubernetes pod shell.
• Check resource pressure. A node or pod under CPU or memory pressure looks like a network fault. Our article on viewing pod metrics shows how to rule that out quickly.

Troubleshoot clusters from one window

If you spend your day switching between kubeconfigs, tunnels, log tabs and terminals, Kubezilla Desktop puts them in one native app for Linux, macOS and Windows. You get multi-cluster access, SSH tunneling, live log streaming, metrics, pod terminals, YAML editing, and Helm and ArgoCD operations, with kubeconfigs stored locally.

Kubezilla Desktop costs €5 per month. You can see the details on the pricing page.

FAQ

Why does kubectl say connection refused?

Usually kubectl is dialing an address where nothing is listening. The most common causes are a missing or wrong kubeconfig (kubectl then falls back to localhost), an environment variable overriding the file, a stale server address, or an API server that is down. Check kubectl config view --minify, then test the port with nc -vz.

What does TLS handshake timeout mean in Kubernetes?

The TCP connection to the API server opened, but the TLS exchange did not finish in time. Look at proxies in your environment, a firewall or inspection device in the path, an overloaded API server, and certificate validity. curl -vk against the API server shows where it stalls.

How do I tell a DNS problem from a Service problem?

From a debug pod, call the Service by its DNS name, then by its cluster IP. If the name fails and the IP works, it is DNS. If both fail, check that the Service has endpoints and that the pods are Ready.

Why do requests time out from one pod but work from another?

That pattern usually means a NetworkPolicy. Ingress rules that allow only labeled pods make other pods time out, while labeled pods connect normally. Compare the labels on both pods against the policy's selectors.

Is it safe to skip TLS verification to get connected?

Only briefly, as a test to learn whether the certificate is the cause. Do not leave it in your kubeconfig. Fix the underlying issue: an expired certificate, the wrong CA, a name mismatch, or a clock problem.

Sources

• Troubleshooting kubectl | Kubernetes
• TCP time-outs when kubectl or other tools connect to the API server | Microsoft Learn
• Debug Services | Kubernetes
• Declare Network Policy | Kubernetes
• Troubleshoot GKE network observability | Google Cloud

Recommended

• Kubernetes SSH: Admins Copy Tunnel Commands, Fix Certificate Errors