# Troubleshooting Networking and Services

> Chapter 15 of the Courseiva CNCF-CKA curriculum — https://courseiva.com/learn/cncf-cka/troubleshooting-networking-and-services

**Official objective:** Troubleshooting - 30% — Resolve networking issues including DNS, Service endpoints, and connectivity.

## Introduction

If you cannot fix a broken network connection between your application's components, the entire system goes dark — users see error pages or nothing at all. Troubleshooting networking and Services in Kubernetes is the skill that lets you find and fix those dark spots, turning a silent failure into a working application. For the CKA exam, this topic is weighted at 30% and appears in multiple hands-on scenarios, so mastering it is essential to passing.

## The Apartment Building Package Delivery Analogy

You are the building supervisor of a large apartment block. Your job is to make sure every letter and parcel reaches the correct flat, even when tenants move around or the postie gets confused. The block has a central mail room (the cluster) and each flat has a unique number (an IP address). The postie (a packet of data) arrives at the main entrance holding a label that says "Flat 3B" (the destination IP). Your job is to check the building directory (DNS) to find which floor has Flat 3B, then carry the parcel down the correct hallway (the network path) to the right door.

Now imagine a tenant in Flat 2A wants to send a letter to Flat 5C. Instead of walking the letter themselves, they drop it in the building's internal post box (a Service). Your job is to pick it up and deliver it. If Flat 5C has moved to a different floor (a Pod restart), you must update your mental map (the endpoints list) so the letter doesn't end up at an empty room. If the building directory has a wrong entry (DNS misconfiguration), the letter might go to Flat 3D instead. Your daily troubleshooting involves checking the directory, verifying which flats are actually occupied (readiness probes), and sometimes knocking on doors to confirm (curl or ping). When a tenant complains their packages aren't arriving, you don't just blame the postie — you methodically trace the path from mail room to doorstep, checking each decision point along the way.

## Core explanation

In Kubernetes, networking is how your containers talk to each other and to the outside world. Think of it as the plumbing that connects every piece of your application. When this plumbing breaks, your app stops working. Let's break down the key pieces from the ground up.

First, every container in a Pod gets its own IP address (Internet Protocol address). This is a unique numeric label, like a house number, that other containers use to send data to it. Pods are temporary — they can be replaced or moved to a different node (a worker machine in your cluster) at any time. So relying on a Pod's IP directly is like trying to visit a friend whose house keeps getting picked up and moved across town. That is why we use a Service.

A Service is a stable abstraction that sits in front of a set of Pods. It has a fixed IP address (the Service IP) and a name (the Service name). When you send traffic to the Service, it forwards it to one of the Pods behind it. The Service keeps a list of healthy Pod IPs (the endpoints). It checks which Pods are alive using readiness probes (health checks). If a Pod dies, the Service removes its IP from the list. This is like having a permanent reception desk at a hotel that always knows which rooms are available.

Now, how do Pods find the Service? They use DNS (Domain Name System). DNS translates human-readable names like "my-app" into IP addresses. In Kubernetes, every Service gets a DNS record automatically. A Pod can look up the Service name and get the Service IP. This is like looking up a friend's name in a phone book to get their phone number. If DNS is broken, the Pod cannot find the Service, even if both are healthy.

Next, consider how traffic gets from outside the cluster to your Service. That is the job of an Ingress controller (a tool that manages external access) and a LoadBalancer (a type of Service that gives you an external IP). They act as the front door to your application. Troubleshooting external connectivity means checking that DNS resolves your domain name to the right external IP, that the LoadBalancer forwards to the correct Service, and that the Service forwards to healthy Pods.

Common problems include: Service endpoints are empty (no Pods match the selector), DNS name does not resolve, network policies block traffic, or the Pod's container port does not match the Service port. Each of these breaks the chain. The systematic approach is to test each link: from client to external IP, from external IP to Service IP, from Service IP to Pod IP, and finally from Pod IP to container port.

Tools you will use include kubectl get endpoints (to check which Pods a Service routes to), kubectl describe service (to see Service details), kubectl exec into a Pod and use curl (to test connectivity from inside), and nslookup or dig (to test DNS resolution). These are your diagnostic instruments, like a multimeter for a plumber.

## Real-world context

You are the on-call engineer for an e-commerce site running on Kubernetes. At 2 PM, customers start reporting that they cannot load the product catalogue page. The error in their browsers is "Connection Refused" or "500 Internal Server Error". You need to fix it fast.

Your first step is to check if the product catalogue Pods are running. You run kubectl get pods -l app=product-catalogue and see that all Pods are in "Running" state. Good, but that does not guarantee they are reachable. Next, you check the Service that exposes the product catalogue. You run kubectl get svc product-catalogue and see the Service exists and has a ClusterIP (an internal IP). Then you run kubectl get endpoints product-catalogue and see the endpoints list is empty. That is your smoking gun — the Service has no healthy Pods to forward traffic to.

Why are the endpoints empty? The Service uses label selectors to find Pods. You run kubectl describe svc product-catalogue and see it selects "app: product-catalogue". You check one of the running Pods: kubectl describe pod product-catalogue-xyz. Its labels are "app: product-catalogue-v2". The selector is wrong — it says "product-catalogue" but the Pod says "product-catalogue-v2". That mismatch means the Service ignores all Pods. You update the Service selector to match the Pod label, or you fix the Pod label. In a real business, you would edit the Service YAML: kubectl edit svc product-catalogue and change the selector. After saving, you re-run kubectl get endpoints and see the Pod IPs appear. 

Now test connectivity. You exec into another Pod: kubectl exec -it frontend-pod -- curl http://product-catalogue:8080. It returns the catalogue HTML. Customers are back online. But that was just one scenario. Another common real-world issue is DNS resolution failing. You might find that a new namespace was created and the Service is in a different namespace, so the DNS name needs a suffix (e.g., product-catalogue.other-namespace.svc.cluster.local). If the application code does not use the full DNS name, it fails. You would fix it by updating the application configuration or by using a Service in the same namespace.

You also check network policies. If a NetworkPolicy (a firewall rule inside the cluster) exists that blocks traffic from the frontend namespace to the product-catalogue namespace, even with correct Services, the connection fails. You would run kubectl get networkpolicies to see if any rules exist that deny the traffic. You might need to add a rule allowing ingress from the frontend namespace.

## Exam focus

The CKA exam tests your ability to fix broken networking in a live cluster, not just theory. You will be given a cluster with a pre-broken configuration and asked to diagnose and resolve the issue. Expect these specific scenarios to appear:

- Service endpoints empty: The question will have a Service whose selector does not match any Pod labels. You must identify that the endpoints list is empty, then fix the selector or the Pod labels. The trap is that beginners check Pod logs first instead of endpoints. The correct pattern is: run kubectl get endpoints first.
- DNS resolution failure: A Pod cannot reach a Service by its name. The cause might be a missing CoreDNS Pod (check kubectl get pods -n kube-system | grep coredns), a misconfigured kubelet DNS settings, or the Pod's resolv.conf file having wrong nameservers. The trap is assuming the Service is down when it is actually a DNS issue. Use kubectl exec into a Pod and run nslookup <service-name> to test.
- NodePort or LoadBalancer connectivity: You might be asked to make an application accessible from outside the cluster. You must create or modify a Service of type NodePort or LoadBalancer, then verify that the external port works. The trap is forgetting to set the targetPort to the container port. The correct pattern is: create the Service, use curl from a node's IP with the NodePort.
- NetworkPolicy blocking traffic: You might find that a Pod inside the cluster cannot connect to another Pod, even though Services exist. You must inspect the NetworkPolicy rules (kubectl describe networkpolicy) and either delete the blocking policy or add an allow rule for the correct source labels. The trap is misreading the podSelector in the policy.
- Ingress misconfiguration: You may be given an Ingress resource that does not work. Check that the Ingress controller is running (e.g., nginx-ingress), that the Ingress rule points to the correct Service name and port, and that the DNS for the host header resolves to the Ingress controller's IP. The trap is using an IP instead of a hostname.

Key definitions to memorise: Service types (ClusterIP, NodePort, LoadBalancer, ExternalName), the meaning of endpoints, readiness probes and their effect on endpoints, DNS format for Services (service.namespace.svc.cluster.local), and the default DNS policy for Pods (ClusterFirst).

Traps they set: They give you a Service with a typo in the port name (e.g., "http" vs "http"). They change the Pod labels after you create the Service. They remove the CoreDNS deployment but leave the Service. They create a NetworkPolicy that denies all ingress by default. The correct answer pattern is always: check the most basic things first (endpoints, DNS, selector match), then escalate to network policies and Ingress.

## Step by step

1. **Identify the broken component from user reports** — When users report connectivity issues, narrow down whether the problem is external (DNS, load balancer, ingress) or internal (Service, Pod, network policy). For example, if the entire site is down, check the Ingress first. If only one microservice is unreachable, focus on its Service and Pods.
2. **Check if the target Pods are running and ready** — Run kubectl get pods -l <selector> and verify all desired Pods are in Running state. Then check individual Pod readiness with kubectl describe pod <name> and look for the Conditions section — 'Ready' must be True. If not ready, check the readiness probe configuration and the container logs.
3. **Verify the Service endpoints** — Run kubectl get endpoints <service-name>. If the endpoints list is empty, the Service has no healthy Pods to forward to. This is the most common cause of broken connectivity. Compare the Service's selector (kubectl describe svc <name>) with the Pod's labels (kubectl get pods --show-labels). Fix any mismatch.
4. **Test DNS resolution from inside a Pod** — Exec into any Pod in the same namespace: kubectl exec -it <any-pod> -- nslookup <service-name>. If it fails, check CoreDNS is running (kubectl get pods -n kube-system | grep coredns). If CoreDNS is down, scale it back up. If it returns the wrong IP, check the Service's ClusterIP. Use the full DNS name <service>.<namespace>.svc.cluster.local if needed.
5. **Test connectivity to the Service IP directly** — From a Pod in the cluster, use curl <service-ip>:<port> or curl <service-name>:<port>. If this fails but DNS resolves correctly, the issue is likely a network policy blocking traffic or the target container not listening on that port. Run kubectl get networkpolicies to see if any deny rules exist. Check the container's listened ports with kubectl logs or exec into the target Pod and netstat.
6. **Check external access if applicable** — For a Service of type NodePort or LoadBalancer, verify the external IP or port. For NodePort, get the Node's IP and the assigned port (kubectl get svc <name> — under PORT(S)). For LoadBalancer, check the EXTERNAL-IP column. Test from outside the cluster. If the Ingress is used, verify the Ingress resource has the correct host and service name, and that the Ingress controller Pod is running.

## Comparisons

### ClusterIP Service vs NodePort Service

**ClusterIP Service:**
- Only reachable from inside the cluster.
- Uses a virtual IP that does not correspond to any physical interface.
- No external port is opened on the Nodes.

**NodePort Service:**
- Reachable from outside the cluster if Node IP is accessible.
- Opens a specific high port (30000-32767) on every Node.
- Internally still creates a ClusterIP for in-cluster use.

### Service selector match vs Pod labels

**Service selector match:**
- Defined in the Service YAML under 'selector'.
- Must exactly match the Pod's label key-value pair.
- If no match, endpoints list is empty.

**Pod labels:**
- Set in the Pod template under 'labels'.
- Can have multiple labels; only the matching key-value pair matters.
- If labels change, the Service may include or exclude the Pod.

### Readiness probe vs Liveness probe

**Readiness probe:**
- Determines if a Pod should receive traffic from a Service.
- If it fails, the Pod is removed from Service endpoints.
- Does not restart the container.

**Liveness probe:**
- Determines if a Pod is alive or deadlocked.
- If it fails, kubelet restarts the container.
- Does not affect Service endpoints directly.

### DNS for Service vs DNS for Pod

**DNS for Service:**
- Resolves to the Service's ClusterIP.
- Format: <service>.<namespace>.svc.cluster.local.
- Created automatically when Service is created.

**DNS for Pod:**
- Resolves to the Pod's individual IP.
- Format: <pod-ip>.<namespace>.pod.cluster.local.
- Not created by default; requires hostname subdomain.

### kubectl exec vs kubectl port-forward

**kubectl exec:**
- Runs a command inside a running container.
- Used to test network connectivity from a Pod's perspective.
- No network port forwarding to the local machine.

**kubectl port-forward:**
- Forwards a local port to a Pod or Service inside the cluster.
- Used to access a cluster-internal service from your local machine.
- Creates a tunnel between your machine and the cluster.

## Diagram

_Flow of traffic from a user's browser through Ingress, Service, and Pods, with DNS and label matching governing connectivity._

```mermaid
flowchart TD
    User[User Browser] -->|HTTP Request| Ingress[Ingress Controller]
    Ingress -->|Routes to Service| Service[Product Catalogue Service]
    Service -->|Forwards to Pod| Pod1[Product Catalogue Pod 1]
    Service -->|Forwards to Pod| Pod2[Product Catalogue Pod 2]
    Pod1 -->|Listens on port| Container1[Container :8080]
    Pod2 -->|Listens on port| Container2[Container :8080]
    Service -.->|Checks labels| Selector[Label Selector: app=catalogue]
    Pod1 -.->|Labels| Labels1[labels: app=catalogue]
    Pod2 -.->|Labels| Labels2[labels: app=catalogue]
    DNS[CoreDNS] -.->|Resolves service name| Service
    User -.->|Resolves domain| DNS
```

## Common misconceptions

- **Misconception:** If a Pod is running, the Service must be working. **Reality:** A Pod running does not mean it passes readiness probes or matches Service selectors. The Service only forwards to Pods that are both running and have passed their readiness check and have labels matching the selector. (Beginners see 'Running' in kubectl get pods and stop investigating, missing selector mismatches or failed probes.)
- **Misconception:** A Service IP is the same as a Pod IP. **Reality:** A Service IP is a virtual IP that does not belong to any network interface — it is managed by kube-proxy. Pod IPs are real addresses on the Pod's network namespace. (The term 'IP address' is overloaded; beginners conflate the stable Service IP with ephemeral Pod IPs.)
- **Misconception:** DNS always resolves to the Pod IP directly. **Reality:** DNS for a Service resolves to the Service's ClusterIP, not to individual Pod IPs. The Service then load-balances to Pod IPs. (Beginners expect DNS to return a list of Pod IPs, but it returns the single Service IP, which can be confusing.)
- **Misconception:** If you can ping the Service IP, the application inside the Pod works. **Reality:** Ping uses ICMP, not the application protocol (e.g., HTTP). The Service may respond to ping but the application port (e.g., 8080) may still be closed or the container may not be listening. (Ping is a common first test, but it only confirms network-layer connectivity, not application-layer health.)
- **Misconception:** Changing the Pod labels automatically updates the Service endpoints instantly. **Reality:** Changing Pod labels may cause the Service to include or exclude the Pod, but there is a short propagation delay (seconds to a minute) depending on kube-proxy sync and endpoint controller watch. (Beginners expect immediate updates and think the cluster is broken when there is a slight delay.)
- **Misconception:** A NodePort Service gives the Pod a public IP. **Reality:** A NodePort opens a port on every Node's IP, but that Node IP is often private (inside a VPC). You must use a LoadBalancer or proxy to get a public IP. (Beginners think NodePort equals external access, but it only works from within the network.)

## Key takeaways

- A Service routes traffic to Pods only if the Pods have labels matching the Service's selector and are passing readiness probes.
- Always check kubectl get endpoints <service-name> first to confirm a Service has healthy targets — empty endpoints mean no traffic flows.
- DNS in Kubernetes resolves Service names to Service ClusterIPs, not to Pod IPs — never expect a Pod IP from a Service DNS lookup.
- The most common CKA networking trap is a selector mismatch between a Service and its Pods, so always compare labels with kubectl describe.
- Network policies act as cluster-internal firewalls and can silently block traffic even when Services and Pods are healthy.
- Use kubectl exec into a Pod to test connectivity with curl or nslookup — this bypasses external networking and isolates the cluster path.
- CoreDNS must be running in the kube-system namespace for Pod-to-Service DNS to work — if DNS fails, check CoreDNS Pods first.

## FAQ

**Why is my Service endpoint list empty even though the Pods are running?**

The Service's label selector does not match the Pods' labels. Run kubectl describe svc <name> to see the selector, then check Pod labels with kubectl get pods --show-labels. Fix the mismatch.

**Why can't my Pod resolve a Service name via DNS?**

CoreDNS may be down or misconfigured. Check CoreDNS Pods in kube-system. Also verify that the Pod's dnsPolicy is not set to 'None' and that its /etc/resolv.conf has the correct cluster DNS IP.

**What is the difference between a ClusterIP and a NodePort?**

ClusterIP gives the Service a virtual IP reachable only inside the cluster. NodePort opens a high port on every Node's IP, allowing external traffic if the Node IP is reachable. NodePort also creates a ClusterIP automatically.

**How do I know if a NetworkPolicy is blocking my traffic?**

Run kubectl get networkpolicies --all-namespaces. If there is a policy that denies ingress from your source's namespace or labels, it will block traffic. Describe the policy to see its rules. You may need to create an allow rule.

**Can I curl a Service from outside the cluster?**

Only if the Service type is NodePort or LoadBalancer and the external IP/port is reachable from your network. For ClusterIP Services, you cannot reach them from outside — use kubectl port-forward or a proxy.

**Why does my LoadBalancer Service show <pending> under EXTERNAL-IP?**

Your cluster does not have a cloud load balancer integration (e.g., running on-premises or with minikube). The LoadBalancer type only works if the cluster is in a cloud provider that supports it. For local clusters, use NodePort instead.

## Check your understanding

1. **What command do you run first to see if a Service has healthy Pods to forward traffic to?**

   Answer: kubectl get endpoints <service-name> — this shows the list of Pod IPs that the Service is currently routing to.

2. **If a Pod is in 'Running' state but the Service endpoints list is empty, what is the most likely cause?**

   Answer: The Service's label selector does not match the Pod's labels, or the Pod is not passing its readiness probe.

3. **How do you test DNS resolution for a Service from inside a Pod?**

   Answer: Exec into a Pod with kubectl exec -it <pod-name> -- nslookup <service-name> (or use dig).

4. **What type of Service must you use if you need external traffic to reach your application without a cloud load balancer?**

   Answer: NodePort — it opens a port on every Node's IP, allowing external access if the Node IP is reachable.

5. **What is the default DNS policy for a Pod in Kubernetes?**

   Answer: ClusterFirst — Pods use the cluster's CoreDNS server to resolve Service names, not the node's DNS.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cncf-cka/troubleshooting-networking-and-services