Setting up DNS Anycast with FRR
In traditional enterprise infrastructures, the DNS service is typically composed of two servers: a primary server and a secondary server. While this architecture works well for small and medium-sized environments, it can become a bottleneck as the number of clients grows.
To improve scalability and resilience, we need a way to increase the number of DNS servers while exposing them through a single IP address with a load-balancing mechanism.
This is where Anycast comes into play. Anycast allows multiple devices to share the same IP address while ensuring that each client request is handled by only one of them. Unlike multicast, where traffic is delivered to multiple recipients, Anycast routes each request to the most appropriate server, usually the one that is topologically closest to the client.
However, simply assigning the same IP address to multiple machines on the same Layer 2 network would result in an IP address conflict. Instead, Anycast relies on Layer 3 routing mechanisms, where routers advertise and direct traffic to the appropriate destination based on the network topology.
ECMP (Equal Cost Multi-Path)
In BGP, the best path is normally selected according to several routing attributes. However, in a deployment where multiple DNS servers advertise the same Anycast prefix with identical routing metrics, the available paths are considered equal.
This is where Equal-Cost Multi-Path (ECMP) routing becomes useful. ECMP enables routers to distribute traffic across multiple paths that have the same routing cost, providing both load balancing and increased throughput.
For stateless protocols such as DNS, this approach is straightforward since each query is independent and can be forwarded to any available server.
TCP-based applications, however, require a different approach. Because a TCP connection must remain attached to the same backend server for its entire lifetime, routers typically rely on flow-based load balancing. A hash is computed from the packet's 4-tuple (source IP address, source port, destination IP address, and destination port) to determine which path should be used. Since these values remain constant throughout a TCP session, all packets belonging to the same connection are consistently forwarded to the same server, preserving session integrity.
BGP on FRR
To avoid managing routes manually, each server establishes a BGP session with the router and advertises the shared Anycast IP address. This allows routes to be added or withdrawn dynamically as servers become available or unavailable, providing automatic failover and simplifying operations.
On Linux, we use Free Range Routing (FRR) to implement BGP. FRR is a comprehensive routing software suite that provides implementations of several routing protocols, including BGP, OSPF, IS-IS, and RIP. An alternative would be BIRD, another popular routing daemon. While BIRD is widely used and highly capable, I have more experience with FRR, making it the preferred choice for this deployment.
The Lab Infrastructure
For this example, we will build a small Anycast DNS infrastructure consisting of a VyOS gateway connected to two Debian servers sharing the Anycast IP address 10.1.1.1.
Each Debian server runs both CoreDNS and FRR. The shared IP address is assigned to a dummy network interface, allowing the server to locally own the address while routing traffic destined for it to the DNS service.
The BGP peering between the VyOS gateway and the Debian servers uses eBGP. This design enables the gateway to propagate the 10.1.1.1 prefix to other BGP-speaking routers, making the Anycast service accessible beyond the local network if required.
Because the configuration is nearly identical across all DNS servers, scaling the infrastructure is straightforward. Additional servers can be deployed with minimal configuration changes, allowing the DNS platform to grow seamlessly as the infrastructure expands.
Configuring deb-dns-1 and deb-dns-2
Network Configuration
On each Debian server, we begin by creating a dummy network interface named vip0 and assigning it the Anycast IP address 10.1.1.1/32.
modprobe dummy
ip link add vip0 type dummy
ip addr add 10.1.1.1/32 dev vip0
To make this configuration persistent across reboots, add the following lines to /etc/network/interfaces:
auto vip0
iface vip0 inet static
address 10.1.1.1/32
pre-up ip link add vip0 type dummy
post-down ip link del vip0
On Ubuntu, networking is typically managed by Netplan (backed by systemd-networkd). In this case, the configuration should be added to /etc/netplan/50-cloud-init.yaml:
network:
version: 2
ethernets:
eth0:
...
vip0:
addresses: ["10.1.1.1/32"]
Finally, IPv4 forwarding must be enabled so that each server can correctly route packets destined for the Anycast address through the vip0 interface.
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p
Once these steps have been completed, both Debian servers will locally own the Anycast IP address while remaining ready to advertise it through BGP in the next section.
FRR Configuration
FRR is available from the official Debian and Ubuntu APT repositories, making the installation straightforward.
apt update && apt install frr
Before configuring BGP, the bgpd daemon must be enabled in /etc/frr/daemons by changing the following line:
# /etc/frr/daemons
bgpd=no -> yes
Once the daemon has been enabled, restart the FRR service to apply the changes.
systemctl restart frr
The FRR command-line interface can then be accessed using:
vtysh
Enter configuration mode:
configure
To begin, configure FRR to use the datacenter defaults, which provide more suitable settings for this type of deployment.
frr defaults datacenter
Next, create two prefix lists. The first rejects every route received from the gateway, while the second allows only the Anycast prefix 10.1.1.1/32 to be advertised.
ip prefix-list PERMIT-IN deny any
ip prefix-list PERMIT-OUT permit 10.1.1.1/32
The BGP process can now be configured.
router bgp 65421
bgp router-id 192.0.2.10 # Use 192.0.2.11 on deb-dns-2
neighbor 192.0.2.1 remote-as 64520
address-family ipv4 unicast
network 10.1.1.1/32
neighbor 192.0.2.1 prefix-list PERMIT-IN in
neighbor 192.0.2.1 prefix-list PERMIT-OUT out
exit
exit
exit
In this configuration:
VyOS Gateway Configuration
In this example, we use a VyOS router as the network gateway. Although the configuration syntax varies between routing platforms, the overall concepts remain the same regardless of the operating system.
The first step is to enable Equal-Cost Multi-Path (ECMP) routing and configure a router ID for the BGP process.
set protocols bgp parameters bestpath as-path multipath-relax
set protocols bgp parameters router-id 192.0.2.1
Next, configure the two Debian DNS servers as BGP neighbors.
set protocols bgp neighbor 192.0.2.10 bfd
set protocols bgp neighbor 192.0.2.11 bfd
set protocols bgp neighbor 192.0.2.10 peer-group 'dns-anycast'
set protocols bgp neighbor 192.0.2.11 peer-group 'dns-anycast'
set protocols bgp neighbor 192.0.2.10 remote-as '64521'
set protocols bgp neighbor 192.0.2.11 remote-as '64521'
Bidirectional Forwarding Detection (BFD) is enabled for both neighbors. BFD provides rapid failure detection, allowing the router to remove a failed path within milliseconds instead of waiting for the BGP hold timer to expire. This significantly reduces downtime in the event of a server or link failure.
To control which routes are accepted from the DNS servers, create a prefix list containing only the Anycast address.
set policy prefix-list dns-anycast-v4-addr rule 10 action permit
set policy prefix-list dns-anycast-v4-addr rule 10 prefix 10.1.1.1/32
Route maps can then be defined to apply routing policies. In this configuration, the gateway accepts only the Anycast prefix from its neighbors while advertising no routes back to them.
set policy route-map dns-anycast-v4-in rule 10 action permit
set policy route-map dns-anycast-v4-in rule 10 match ip address prefix-list dns-anycast-v4-addr
set policy route-map dns-anycast-v4-out rule 10 action deny
To simplify the configuration, create a peer group named dns-anycast. Peer groups allow multiple neighbors to share the same routing policies, making the configuration easier to maintain as additional DNS servers are added.
set protocols bgp peer-group dns-anycast address-family ipv4-unicast maximum-prefix '1'
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map import 'dns-anycast-v4-in'
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map export 'dns-anycast-v4-out'
Finally, associate the previously configured routing policies with the peer group.
set protocols bgp peer-group dns-anycast address-family ipv4-unicast maximum-prefix 1
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map export dns-anycast-v4-out
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map import dns-anycast-v4-in
Once the configuration is complete, apply the changes and save them so they persist across reboots.
commit
save
At this stage, the VyOS gateway is ready to establish eBGP sessions with both Debian servers, learn the shared Anycast prefix, and distribute incoming traffic across all available DNS servers using ECMP.
Verification
Once the configuration has been completed on both the Debian servers and the VyOS gateway, the first step is to verify that the BGP sessions have been successfully established.
On both FRR and VyOS, this can be done using the following command:
show bgp summary
On each Debian server, the output should indicate that the BGP session is established and that a single prefix is being advertised to the gateway.
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.10, local AS number 64521 vrf-id 0
BGP table version 13
RIB entries 1, using 192 bytes of memory
Peers 1, using 724 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.0.2.1 4 64520 57 64 0 0 0 1m 0 1 N/A
Total number of neighbors 1
The important field here is PfxSnt, which should indicate that the Anycast prefix has been successfully advertised to the gateway.
On the VyOS router, both BGP neighbors should appear in the Established state, each advertising one prefix.
IPv4 Unicast Summary:
BGP router identifier 192.0.2.1, local AS number 64520 VRF default vrf-id 0
BGP table version 1358
RIB entries 2, using 124 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.0.2.10 4 64521 89 74 1 0 0 2m 1 0 N/A
192.0.2.11 4 64521 92 82 1 0 0 2m 1 0 N/A
Total number of neighbors 2
In this case, the PfxRcd column confirms that the gateway has successfully learned the Anycast prefix from both DNS servers.
The routing table can then be inspected to verify that ECMP has been correctly configured.
show ip route bgp
The output should look similar to the following:
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
B>* 10.1.1.1/32 [20/0] via 192.0.2.10, eth1, weight 1, 3m
* via 192.0.2.11, eth1, weight 1, 3m
Notice that the route to 10.1.1.1/32 has two next hops with identical metrics. Because both paths have the same cost, VyOS installs them into the forwarding table and distributes traffic across them using ECMP.
If one of the DNS servers becomes unavailable or stops advertising the Anycast prefix, its route is automatically withdrawn from the routing table. Traffic is then transparently redirected to the remaining server without requiring any manual intervention, providing both high availability and seamless failover.
Installing CoreDNS
The final step is to deploy a simple CoreDNS instance on each Debian server.
For this example, we will create an authoritative zone named example.local containing a few static records. Queries for any other domain will be forwarded to 1.1.1.1, allowing CoreDNS to act as both an authoritative and recursive DNS server.
Unlike many common packages, CoreDNS is not available in the default Debian APT repositories. Instead, download the latest release directly from the official GitHub repository:
https://github.com/coredns/coredns/releases
Download the archive matching your operating system and CPU architecture, for example:
https://github.com/coredns/coredns/releases/download/v1.14.2/coredns_1.14.2_linux_amd64.tgz
Extract the archive and move the binary into your system's executable path.
tar xvf coredns_*.tgz
mv coredns /usr/bin/
Next, create the CoreDNS configuration directory and the main configuration file.
mkdir /etc/coredns
vim /etc/coredns/Corefile
Populate the Corefile with the following configuration:
example.local.:53 {
bind 0.0.0.0
file /etc/coredns/db.example.local
log
}
.:53 {
bind 0.0.0.0
forward . 1.1.1.1
log
}
The first server block makes CoreDNS authoritative for the example.local zone, while the second forwards all other DNS queries to Cloudflare's public resolver.
Next, create the zone file referenced by the Corefile.
vim /etc/coredns/db.example.local
$TTL 120
@ IN SOA ns.example.local. root.example.local. (
2026020601 ; Serial
120 ; Refresh
1800 ; Retry
1209600 ; Expire
120 ) ; Minimum
@ IN NS ns.example.local.
ns IN A 10.1.1.1
@ IN A 198.51.100.10
www IN A 198.51.100.11
To improve security, create a dedicated system user that will own the configuration files and run the CoreDNS process.
useradd -M -s /bin/false coredns
chown -R coredns:coredns /etc/coredns
Create the following systemd unit file at /etc/systemd/system/coredns.service:
[Unit]
Description=CoreDNS Server
After=network.target
[Service]
Type=simple
User=coredns
Group=coredns
ExecStart=/usr/bin/coredns -p 53 -conf /etc/coredns/Corefile
WorkingDirectory=/etc/coredns
Restart=always
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
[Install]
WantedBy=default.target
RequiredBy=network.target
Reload systemd, enable the service, and start CoreDNS.
systemctl daemon-reload
systemctl enable coredns
systemctl start coredns
You can verify that the service is running correctly with:
systemctl status coredns
If CoreDNS fails to start because port 53 is already in use, check whether systemd-resolved is enabled. On many Linux distributions, it binds to port 53 by default and prevents CoreDNS from listening on that port.
If necessary, disable and stop the service:
systemctl disable systemd-resolved
systemctl stop systemd-resolved
Finally, test the deployment from another machine on the network using nslookup:
nslookup www.example.local 10.1.1.1
A successful response should look similar to the following:
Server: 10.1.1.1
Address: 10.1.1.1#53
Name: www.example.local
Address: 198.51.100.11
At this point, the Anycast deployment is fully operational. Client requests sent to 10.1.1.1 are automatically distributed across the available DNS servers by the network infrastructure, while BGP and ECMP ensure high availability and seamless failover in the event of a server or link failure.
