Service Discovery with DNS
Introduction: Beyond Basic Service Discovery
In Unit 2, you learned that Services automatically get DNS names, allowing applications to connect using names like backend-svc instead of IP addresses. You saw how CoreDNS resolves these names to ClusterIP addresses, enabling loose coupling between your application components. But that was just the foundation.
In this lesson, you'll tackle the advanced service discovery scenarios that arise in real production environments: connecting to Services in different namespaces, integrating with external systems outside your cluster, and understanding why some IPs remain stable while others constantly change. These patterns are essential when you're building distributed systems that span multiple namespaces, hybrid cloud architectures, or migrating legacy applications into Kubernetes.
Cross-Namespace Service Access
So far, you've probably worked within a single namespace — typically default. But production Kubernetes environments use multiple namespaces to isolate different teams, applications, or environments. When your frontend Pod in the web namespace needs to connect to a backend Service in the api namespace, the simple service name won't work. Kubernetes DNS uses namespace-scoped resolution by default, so backend-svc only resolves within your current namespace. For cross-namespace access, you need to use the Fully Qualified Domain Name (FQDN).
The FQDN pattern is <service-name>.<namespace>.svc.cluster.local. If you have a Service called database-svc in the data namespace, you can access it from any namespace using database-svc.data.svc.cluster.local, or the shorter form database-svc.data. Let's see this in action with a concrete example.
First, create two namespaces to simulate a multi-tier application:
Now create a backend service in the backend namespace. Save this as deployment-backend.yaml:
Apply this configuration:
Verify the Service exists in the backend namespace:
Now let's create a Pod in the frontend namespace and try to access the backend Service. First, try using just the service name:
As expected, DNS resolution fails because api-svc doesn't exist in the frontend namespace. Now try using the FQDN:
Perfect! The FQDN api-svc.backend successfully resolves to the Service's ClusterIP. Let's verify we can actually connect:
Excellent! Your frontend Pod successfully connected to the backend Service across namespaces. This pattern is crucial for microservices architectures where you want to isolate services by namespace but still enable controlled communication between them. You can use NetworkPolicies (covered in later units) to restrict which namespaces can access which Services, implementing a zero-trust security model.
One important consideration: using FQDNs makes your configuration less portable. If you hardcode api-svc.backend in your application and later need to move the backend to a different namespace, you'll need to update your code. A better approach is to use environment variables or ConfigMaps that specify the full service address, making it easy to reconfigure without code changes.
Manual Endpoints for External Resources
Not all services you need to connect to run inside your Kubernetes cluster. You might have a legacy database on a physical server, a managed database service from your cloud provider, or an external API you depend on. You could hardcode these external addresses in your application configuration, but that loses the benefits of DNS-based service discovery — you want your application to connect using a service name regardless of whether the backing resource is inside or outside the cluster. Kubernetes supports this through Services without selectors and manually defined Endpoints.
When you create a Service without a selector field, Kubernetes doesn't automatically create Endpoints for it. Instead, you manually create an Endpoints object with the same name as the Service, specifying the external IP addresses and ports you want to route to. From your application's perspective, this looks exactly like any other Service — you connect using the Service name, and Kubernetes routes the traffic to the external resource.
Let's create a Service that points to an external database. Save this as service-external-db.yaml:
Notice there's no selector in this Service definition. Apply it:
Check the endpoints — there should be none:
Now manually create the Endpoints object. Save this as endpoints-external-db.yaml:
The Endpoints object must have the exact same name as the Service. In a real scenario, 1.1.1.1 would be your actual external database IP address. You can also specify multiple addresses for high availability:
Apply the Endpoints:
Verify the endpoints are registered:
Now test DNS resolution from the frontend namespace:
Perfect! The Service DNS name resolves to a ClusterIP, and when your application connects to external-db-svc.backend:5432, Kubernetes routes the traffic to the external IP 1.1.1.1:5432. This pattern is incredibly useful for gradual migration scenarios. You can start with an external database, then later migrate it into the cluster by adding a selector to the Service and deleting the manual Endpoints — your application code doesn't need to change.
One critical limitation: when you use manual Endpoints, you're responsible for keeping them updated. If the external resource's IP changes, you must update the Endpoints object. With selector-based Services, Kubernetes handles this automatically. For highly dynamic external services, consider using ExternalName Services (covered in the next lesson) as an alternative.
ClusterIP Stability vs Ephemeral Pod IPs
One of the most important architectural concepts in Kubernetes is the contrast between stable ClusterIP addresses and ephemeral Pod IPs. When you create a Service, it gets assigned a ClusterIP that remains constant for the Service's lifetime — it never changes unless you delete and recreate the Service. Pod IPs, however, are temporary. Every time a Pod restarts, is rescheduled to a different node, or gets replaced by its controller, it receives a brand new IP address. This is why you must never hardcode Pod IPs in your application configuration.
Let's demonstrate this stability difference. Check the current ClusterIP of the backend Service:
Note the ClusterIP (10.96.145.67). Now check the Pod IPs:
The Pods have IPs like 10.244.1.10 and 10.244.1.11. Now simulate a Pod failure by deleting one:
The Deployment controller immediately creates a replacement. Wait a few seconds and check again:
The replacement Pod has a completely different IP (10.244.1.15). This is Pod IP ephemerality in action. Now verify the Service's ClusterIP:
The ClusterIP remains unchanged at 10.96.145.67. Check the Endpoints to see how Kubernetes maintained the abstraction:
The Endpoints automatically updated to include the new Pod IP (10.244.1.15) instead of the old one (10.244.1.10). Your application connects to the stable ClusterIP or DNS name, completely unaware of the Pod replacement happening behind the scenes.
This stability model has critical implications for application design:
- Always use Service names or ClusterIPs for inter-service communication
- Never store Pod IPs in configuration files, databases, or environment variables
- Only access Pod IPs directly when using tools like
kubectl execfor debugging
The ClusterIP is allocated from a service CIDR range configured when your cluster was created (typically something like 10.96.0.0/12). Once assigned to a Service, it's reserved until you delete the Service. If you delete and recreate a Service with the same name, it might get a different ClusterIP — another reason to use DNS names instead of hardcoding IPs.
Summary: Advanced Service Discovery Patterns
You've now mastered the advanced service discovery patterns essential for production Kubernetes environments. You learned how to access Services across namespaces using FQDNs, integrate external resources using manual Endpoints, and understand the architectural distinction between stable ClusterIPs and ephemeral Pod IPs. These patterns enable you to build distributed systems that span multiple namespaces, connect to legacy infrastructure outside your cluster, and maintain loose coupling even as Pods constantly come and go.
In the upcoming practice exercises, you'll apply these concepts hands-on: accessing Services from different namespaces, connecting to external databases using manual Endpoints, observing ClusterIP stability during Pod disruptions, and implementing cross-namespace communication in a multi-tier application. These exercises will prepare you for real-world scenarios where applications must discover services across namespace boundaries, integrate with external systems, and remain resilient to the constant infrastructure changes that define cloud-native environments.
