Consul DNS

Posted by Amol Dighe on August 27, 2016

Our Private Cloud setup is using Consul for service discovery and monitoring. While working with Consul, I thought of trying to setup Consul DNS, which was never adopted on our cloud environment due to lack of a solid use case for same. Nevertheless it was a good learning, and I thought of documenting Consul DNS, as the information was scattered across various websites.

In this post, I will try to address:

  • Consul DNS setup
  • Consul DNS forward using dnsmasq

As a default behaviour, Consul will listen on 127.0.0.1:8600 for DNS queries i.e. on the localhost where consul agent is running. The consul DNS service works in the top level domain of “consul.”. The logical concept of data centers is used to subdivide the domain. The datacenter must be configured for each Consul service. Each datacenter contains two namespaces - “nodes” and “services”. Every node is automatically registered with its name in the nodes.datacenter.consul zone and services live in the services.datacenter.consul.zone.

I have a consul cluster setup of 3 servers & 2 clients

root@ubuntuserver3:~# consul members
Node           Address            Status  Type    Build  Protocol  DC
ubuntuclient1  192.168.56.5:8301  alive   client  0.7.0  2         consultest
ubuntuclient2  192.168.56.6:8301  alive   client  0.7.0  2         consultest
ubuntuserver1  192.168.56.2:8301  alive   server  0.7.0  2         consultest
ubuntuserver2  192.168.56.3:8301  alive   server  0.7.0  2         consultest
ubuntuserver3  192.168.56.4:8301  alive   server  0.7.0  2         consultest

As I had mentioned DNS works on consul nodes by default on port 8600, hence I can run a dig on anyone of my consul nodes for services or nodes.

The below dig response will give IP of all nodes running consul service on datacenter - consultest.

root@ubuntuserver3:~# dig @127.0.0.1 -p 8600 consul.service.consultest.consul. ANY

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @127.0.0.1 -p 8600 consul.service.consultest.consul. ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15776
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;consul.service.consultest.consul. IN	ANY

;; ANSWER SECTION:
consul.service.consultest.consul. 0 IN	A	192.168.56.3
consul.service.consultest.consul. 0 IN	A	192.168.56.2
consul.service.consultest.consul. 0 IN	A	192.168.56.4

;; Query time: 4 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Sep 29 21:34:35 IST 2016
;; MSG SIZE  rcvd: 98

The below dig response will give IP of a node for ubuntuserver1 under datacenter - consultest.

root@ubuntuclient2:~# dig @127.0.0.1 -p 8600 ubuntuserver1.node.consultest.consul. ANY

; <<>> DiG 9.9.5-3-Ubuntu <<>> @127.0.0.1 -p 8600 ubuntuserver1.node.consultest.consul. ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10921
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;ubuntuserver1.node.consultest.consul. IN ANY

;; ANSWER SECTION:
ubuntuserver1.node.consultest.consul. 0	IN A	192.168.56.2

;; Query time: 16 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Sep 29 21:49:43 IST 2016
;; MSG SIZE  rcvd: 70

Ok … the above is straight forward.

Lets assume I have an existing dns service like dnsmasq running on my environment and I also have consul dns setup. Now how do I make dns tools like dig & nslookup work with consul without distrupting my existing dns setup.

Dnsmasq setup on ubuntuserver2

root@ubuntuserver2:~# ps -axf | grep dnsmasq
 1762 pts/0    S+     0:00          \_ grep --color=auto dnsmasq
  948 ?        S      0:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new

using the following settings in /etc/dnsmasq.conf

# Configuration file for dnsmasq.
#
log-queries
log-facility=/var/log/dnsmasq.log
addn-hosts=/etc/dnsmasq_static_hosts.conf
server=8.8.8.8

with a static host file defining the hostname - ip address mapping in /etc/dnsmasq_static_hosts.conf

192.168.56.2    ubuntuserver1    ubuntuserver1.local.tld
192.168.56.3    ubuntuserver2    ubuntuserver2.local.tld
192.168.56.4    ubuntuserver3    ubuntuserver3.local.tld
192.168.56.5    ubuntuclient1    ubuntuclient1.local.tld
192.168.56.6    ubuntuclient2    ubuntuclient2.local.tld

Hence nslookup for :

root@ubuntuclient2:~# nslookup ubuntuserver3.local.tld
Server:		192.168.56.3
Address:	192.168.56.3#53

Name:	ubuntuserver3.local.tld
Address: 192.168.56.4

root@ubuntuclient2:~# nslookup ubuntuserver2
Server:		192.168.56.3
Address:	192.168.56.3#53

Name:	ubuntuserver2
Address: 192.168.56.3
Name:	ubuntuserver2
Address: 127.0.1.1

But this will not resolve “consul.” domain

root@ubuntuclient2:~# nslookup ubuntuserver1.node.consultest.consul
;; connection timed out; no servers could be reached

To make this work with consul domain, we need to forward any query comming to our dns server for consul domain to consul port 8600

Edit /etc/dnsmasq.d/10-consul and add an entry for consul on port 8600

server=/consul/127.0.0.1#8600

Restart dnsmasq service

root@ubuntuserver2:~# service dnsmasq restart
 * Restarting DNS forwarder and DHCP server dnsmasq                          [ OK ]

Please note /etc/resolv.conf in dnsmasq server need to have an entry for :

root@ubuntuserver2:~# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1

While /etc/resolv.conf on other servers need to have an entry pointing to dnsmasq server

root@ubuntuclient2:~# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.56.3

Now you should be able to resolve consul domain along with domain names configured under dnsmasq

root@ubuntuclient2:~# nslookup ubuntuserver1.node.consultest.consul
Server:		192.168.56.3
Address:	192.168.56.3#53

Name:	ubuntuserver1.node.consultest.consul
Address: 192.168.56.2

root@ubuntuclient2:~# nslookup consul.service.consultest.consul
Server:		192.168.56.3
Address:	192.168.56.3#53

Name:	consul.service.consultest.consul
Address: 192.168.56.3
Name:	consul.service.consultest.consul
Address: 192.168.56.4
Name:	consul.service.consultest.consul
Address: 192.168.56.2

root@ubuntuclient2:~# nslookup ubuntuserver1
Server:		192.168.56.3
Address:	192.168.56.3#53

Name:	ubuntuserver1
Address: 192.168.56.2