DNS

DNS plays a central role in Kubernetes service discovery. As it is mentioned in the Services chapter, DNS is an essential part of how Services are consumed by end clients and, while implementation is not baked into core Kubernetes controllers, [DNS specification] is very explicit about the behaviour expected from such an implementation. The DNS spec defines the rules for the format of the queries and the expected responses. All Kubernetes Services have at least one corresponding A/AAAA DNS record in the format of {service-name}.{namespace}.svc.{cluster-domain} and the response format depends on the type of a Service:

Historically, there had been two implementations of this DNS spec – one based on dnsmasq and another one based on CoreDNS, the latter had become the default option for kubeadm since Kubernetes 1.11.

Service Discovery – Server-side

CoreDNS implements the Kubernetes DNS spec in a dedicated plugin that gets compiled into a static binary and deployed in a Kubernetes cluster as a Deployment and exposed as a ClusterIP service. This means that all communications with the DNS service inside a cluster are subject to the same network forwarding rules used and limitations experienced by normal Pods and set up by the CNI and Services plugins.

Since DNS speed and stability are considered crucial in any network-based communication, CoreDNS implementation is highly optimised to minimise memory consumption and maximise query processing rate. In order to achieve that, CoreDNS stores only the relevant parts of Services, Pods and Endpoints objects in its local cache that is optimised to return a response in a single lookup.

By default, CoreDNS also acts as a DNS proxy for all external domains (e.g. example.com) using the forward plugin and is often deployed with the cache plugin enabled. The entire CoreDNS configuration can be found in the coredns ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

Service Discovery – Client-side

DNS configuration inside a Pod is controlled by the spec.dnsPolicy and spec.dnsConfig settings. By default, kubelet will configure the cluster DNS IP, stored in the configuration file and hard-coded to the tenth IP of the ClusterIP range by the kubeadm.

root@k8s-guide-control-plane:/# cat /var/lib/kubelet/config.yaml 
apiVersion: kubelet.config.k8s.io/v1beta1
...
clusterDNS:
- 10.96.0.10
...

With the above default settings, this is how a Pod deployed in the default namespace would see its own resolv.conf file:

$ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

The search domains and ndots value are configured so that any non-FQDN DNS query made by a Pod is first tried in all of the specified domains, which allows for internal cluster DNS schema to take precedence over the external DNS (explanation). For example, any Pod in the default Namespace, can lookup the ClusterIP of the kubernetes Service in a single lookup (the shell is running a stern -n kube-system -l k8s-app=kube-dns in the background):

$ kubectl -n default exec ds/net-tshoot -- dig kubernetes +search +short
10.96.0.1
coredns-558bd4d5db-sqhkz coredns [INFO] 10.244.0.5:36255 - 36946 "A IN kubernetes.default.svc.cluster.local. udp 77 false 4096" NOERROR qr,aa,rd 106 0.0002139s

The downside of this behaviour is that any external domain lookup will require at least 4 separate queries:

$ kubectl -n default exec ds/net-tshoot -- dig tkng.io +search +short
coredns-558bd4d5db-5jbgh coredns [INFO] 10.244.0.5:54816 - 13660 "A IN tkng.io.default.svc.cluster.local. udp 74 false 4096" NXDOMAIN qr,aa,rd 144 0.0002719s
coredns-558bd4d5db-5jbgh coredns [INFO] 10.244.0.5:38006 - 38084 "A IN tkng.io.svc.cluster.local. udp 66 false 4096" NXDOMAIN qr,aa,rd 136 0.0001705s
coredns-558bd4d5db-5jbgh coredns [INFO] 10.244.0.5:35302 - 4454 "A IN tkng.io.cluster.local. udp 62 false 4096" NXDOMAIN qr,aa,rd 132 0.0001219s
172.67.201.112
104.21.21.243
coredns-558bd4d5db-sqhkz coredns [INFO] 10.244.0.5:47052 - 6189 "A IN tkng.io. udp 48 false 4096" NOERROR qr,rd,ad 71 0.0183829s

Optimisations

DNS is widely regarded as the main source of all IT problems, and Kubernetes is no exception (see 1, 2, 3, 4). Its way of deployment and reliance on HPA mean that some Nodes could become connection bottlenecks while the CPU and Memory of the DNS Pods may remain relatively low. There are a number of optimisations that can be enabled to improve DNS performance at the expense of additional resource utilisation and complexity:

• The authopath plugin can be enabled in CoreDNS to make it follow the chain of search paths on behalf of a client, thereby reducing the number of queries for an external domain required by the client from 4 (see above) to just one.
• Each Kubernetes Node can run a NodeLocal DNSCache – a daemonset of recursive DNS resolvers designed to reduce the load on a centralised CoreDNS deployment by serving as a caching layer between Pods and the DNS service.
• Adjust lookups that are used frequently to add a trailing dot (example, tkng.io.). This is most effective for external hostnames that experience overhead as an absolute query is performed, avoiding the search path expansion through the cluster.local, and potentially other search domains in /etc/resolv.conf.

External DNS

The DNS Specification is only focused on the intra-cluster DNS resolution and service discovery. Anything to do with external DNS is left out of scope, despite the fact that most of the end-users are located outside of a cluster. For them, Kubernetes has to provide a way to discover external Kubernetes resources, LoadBalancer Services, Ingresses and Gateways, and there are two ways this can be accomplished:

• An out-of-cluster DNS zone can be orchestrated by the ExternalDNS cluster add-on – a Kubernetes controller that synchronises external Kubernetes resources with any supported third-party DNS provider via an API (see the GH page for the list of supported providers).
• An existing DNZ zone can be configured to delegate a subdomain to a self-hosted external DNS plugin, e.g. k8s_gateway. This approach assumes that this DNS plugin is deployed inside a cluster and exposed via a LoadBalancer IP, which is then used in an NS record for the delegated zone. All queries hitting this subdomain will get forwarded to this plugin which will respond as an authoritative nameserver for the delegated subdomain.