Setting up DNS Anycast with FRR

In traditional enterprise infrastructures, the DNS service is typically composed of two servers: a primary server and a secondary server. While this architecture works well for small and medium-sized environments, it can become a bottleneck as the number of clients grows.

To improve scalability and resilience, we need a way to increase the number of DNS servers while exposing them through a single IP address with a load-balancing mechanism.

This is where Anycast comes into play. Anycast allows multiple devices to share the same IP address while ensuring that each client request is handled by only one of them. Unlike multicast, where traffic is delivered to multiple recipients, Anycast routes each request to the most appropriate server, usually the one that is topologically closest to the client.

However, simply assigning the same IP address to multiple machines on the same Layer 2 network would result in an IP address conflict. Instead, Anycast relies on Layer 3 routing mechanisms, where routers advertise and direct traffic to the appropriate destination based on the network topology.

ECMP (Equal Cost Multi-Path)

In BGP, the best path is normally selected according to several routing attributes. However, in a deployment where multiple DNS servers advertise the same Anycast prefix with identical routing metrics, the available paths are considered equal.

This is where Equal-Cost Multi-Path (ECMP) routing becomes useful. ECMP enables routers to distribute traffic across multiple paths that have the same routing cost, providing both load balancing and increased throughput.

For stateless protocols such as DNS, this approach is straightforward since each query is independent and can be forwarded to any available server.

TCP-based applications, however, require a different approach. Because a TCP connection must remain attached to the same backend server for its entire lifetime, routers typically rely on flow-based load balancing. A hash is computed from the packet's 4-tuple (source IP address, source port, destination IP address, and destination port) to determine which path should be used. Since these values remain constant throughout a TCP session, all packets belonging to the same connection are consistently forwarded to the same server, preserving session integrity.

BGP on FRR

To avoid managing routes manually, each server establishes a BGP session with the router and advertises the shared Anycast IP address. This allows routes to be added or withdrawn dynamically as servers become available or unavailable, providing automatic failover and simplifying operations.

On Linux, we use Free Range Routing (FRR) to implement BGP. FRR is a comprehensive routing software suite that provides implementations of several routing protocols, including BGP, OSPF, IS-IS, and RIP. An alternative would be BIRD, another popular routing daemon. While BIRD is widely used and highly capable, I have more experience with FRR, making it the preferred choice for this deployment.

The Lab Infrastructure

For this example, we will build a small Anycast DNS infrastructure consisting of a VyOS gateway connected to two Debian servers sharing the Anycast IP address 10.1.1.1.

Each Debian server runs both CoreDNS and FRR. The shared IP address is assigned to a dummy network interface, allowing the server to locally own the address while routing traffic destined for it to the DNS service.

The BGP peering between the VyOS gateway and the Debian servers uses eBGP. This design enables the gateway to propagate the 10.1.1.1 prefix to other BGP-speaking routers, making the Anycast service accessible beyond the local network if required.

Because the configuration is nearly identical across all DNS servers, scaling the infrastructure is straightforward. Additional servers can be deployed with minimal configuration changes, allowing the DNS platform to grow seamlessly as the infrastructure expands.

Configuring deb-dns-1 and deb-dns-2

Network Configuration

On each Debian server, we begin by creating a dummy network interface named vip0 and assigning it the Anycast IP address 10.1.1.1/32.

modprobe dummy
ip link add vip0 type dummy
ip addr add 10.1.1.1/32 dev vip0

To make this configuration persistent across reboots, add the following lines to /etc/network/interfaces:

auto vip0
iface vip0 inet static
        address 10.1.1.1/32
        pre-up ip link add vip0 type dummy
        post-down ip link del vip0

On Ubuntu, networking is typically managed by Netplan (backed by systemd-networkd). In this case, the configuration should be added to /etc/netplan/50-cloud-init.yaml:

network:
  version: 2
  ethernets:
    eth0:
      ...
    vip0:
      addresses: ["10.1.1.1/32"]

Finally, IPv4 forwarding must be enabled so that each server can correctly route packets destined for the Anycast address through the vip0 interface.

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p

Once these steps have been completed, both Debian servers will locally own the Anycast IP address while remaining ready to advertise it through BGP in the next section.

FRR Configuration

FRR is available from the official Debian and Ubuntu APT repositories, making the installation straightforward.

apt update && apt install frr

Before configuring BGP, the bgpd daemon must be enabled in /etc/frr/daemons by changing the following line:

# /etc/frr/daemons
bgpd=no -> yes

Once the daemon has been enabled, restart the FRR service to apply the changes.

systemctl restart frr

The FRR command-line interface can then be accessed using:

vtysh

Enter configuration mode:

configure

To begin, configure FRR to use the datacenter defaults, which provide more suitable settings for this type of deployment.

frr defaults datacenter

Next, create two prefix lists. The first rejects every route received from the gateway, while the second allows only the Anycast prefix 10.1.1.1/32 to be advertised.

ip prefix-list PERMIT-IN deny any
ip prefix-list PERMIT-OUT permit 10.1.1.1/32

The BGP process can now be configured.

router bgp 65421
  bgp router-id 192.0.2.10  # Use 192.0.2.11 on deb-dns-2
  neighbor 192.0.2.1 remote-as 64520

  address-family ipv4 unicast
    network 10.1.1.1/32
    neighbor 192.0.2.1 prefix-list PERMIT-IN in
    neighbor 192.0.2.1 prefix-list PERMIT-OUT out
    exit
  exit
exit

In this configuration:

The server advertises only the Anycast address 10.1.1.1/32.
No routes are accepted from the gateway, preventing the DNS server from becoming a transit router.
Outbound advertisements are restricted to the Anycast prefix, ensuring that no unintended routes are announced.
Finally, save the running configuration so it is restored after a reboot.
```
copy running-config startup-config
```
At this point, the Debian server is ready to establish its eBGP session with the VyOS gateway and advertise the shared Anycast address.

VyOS Gateway Configuration

In this example, we use a VyOS router as the network gateway. Although the configuration syntax varies between routing platforms, the overall concepts remain the same regardless of the operating system.
The first step is to enable Equal-Cost Multi-Path (ECMP) routing and configure a router ID for the BGP process.

set protocols bgp parameters bestpath as-path multipath-relax
set protocols bgp parameters router-id 192.0.2.1

Next, configure the two Debian DNS servers as BGP neighbors.

set protocols bgp neighbor 192.0.2.10 bfd
set protocols bgp neighbor 192.0.2.11 bfd
set protocols bgp neighbor 192.0.2.10 peer-group 'dns-anycast'
set protocols bgp neighbor 192.0.2.11 peer-group 'dns-anycast'
set protocols bgp neighbor 192.0.2.10 remote-as '64521'
set protocols bgp neighbor 192.0.2.11 remote-as '64521'

Bidirectional Forwarding Detection (BFD) is enabled for both neighbors. BFD provides rapid failure detection, allowing the router to remove a failed path within milliseconds instead of waiting for the BGP hold timer to expire. This significantly reduces downtime in the event of a server or link failure.
To control which routes are accepted from the DNS servers, create a prefix list containing only the Anycast address.

set policy prefix-list dns-anycast-v4-addr rule 10 action permit
set policy prefix-list dns-anycast-v4-addr rule 10 prefix 10.1.1.1/32

Route maps can then be defined to apply routing policies. In this configuration, the gateway accepts only the Anycast prefix from its neighbors while advertising no routes back to them.

set policy route-map dns-anycast-v4-in rule 10 action permit
set policy route-map dns-anycast-v4-in rule 10 match ip address prefix-list dns-anycast-v4-addr
set policy route-map dns-anycast-v4-out rule 10 action deny

To simplify the configuration, create a peer group named dns-anycast. Peer groups allow multiple neighbors to share the same routing policies, making the configuration easier to maintain as additional DNS servers are added.

set protocols bgp peer-group dns-anycast address-family ipv4-unicast maximum-prefix '1'
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map import 'dns-anycast-v4-in'
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map export 'dns-anycast-v4-out'

Finally, associate the previously configured routing policies with the peer group.

set protocols bgp peer-group dns-anycast address-family ipv4-unicast maximum-prefix 1
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map export dns-anycast-v4-out
set protocols bgp peer-group dns-anycast address-family ipv4-unicast route-map import dns-anycast-v4-in

Once the configuration is complete, apply the changes and save them so they persist across reboots.

commit
save

At this stage, the VyOS gateway is ready to establish eBGP sessions with both Debian servers, learn the shared Anycast prefix, and distribute incoming traffic across all available DNS servers using ECMP.

Verification

Once the configuration has been completed on both the Debian servers and the VyOS gateway, the first step is to verify that the BGP sessions have been successfully established.

On both FRR and VyOS, this can be done using the following command:

show bgp summary

On each Debian server, the output should indicate that the BGP session is established and that a single prefix is being advertised to the gateway.

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.10, local AS number 64521 vrf-id 0
BGP table version 13
RIB entries 1, using 192 bytes of memory
Peers 1, using 724 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64520        57        64        0    0    0       1m            0        1 N/A

Total number of neighbors 1

The important field here is PfxSnt, which should indicate that the Anycast prefix has been successfully advertised to the gateway.

On the VyOS router, both BGP neighbors should appear in the Established state, each advertising one prefix.

IPv4 Unicast Summary:
BGP router identifier 192.0.2.1, local AS number 64520 VRF default vrf-id 0
BGP table version 1358
RIB entries 2, using 124 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.10      4      64521        89        74        1    0    0       2m            1        0 N/A
192.0.2.11      4      64521        92        82        1    0    0       2m            1        0 N/A

Total number of neighbors 2

In this case, the PfxRcd column confirms that the gateway has successfully learned the Anycast prefix from both DNS servers.

The routing table can then be inspected to verify that ECMP has been correctly configured.

show ip route bgp

The output should look similar to the following:

Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 10.1.1.1/32 [20/0] via 192.0.2.10, eth1, weight 1, 3m
  *                    via 192.0.2.11, eth1, weight 1, 3m

Notice that the route to 10.1.1.1/32 has two next hops with identical metrics. Because both paths have the same cost, VyOS installs them into the forwarding table and distributes traffic across them using ECMP.

If one of the DNS servers becomes unavailable or stops advertising the Anycast prefix, its route is automatically withdrawn from the routing table. Traffic is then transparently redirected to the remaining server without requiring any manual intervention, providing both high availability and seamless failover.

Installing CoreDNS

The final step is to deploy a simple CoreDNS instance on each Debian server.

For this example, we will create an authoritative zone named example.local containing a few static records. Queries for any other domain will be forwarded to 1.1.1.1, allowing CoreDNS to act as both an authoritative and recursive DNS server.

Unlike many common packages, CoreDNS is not available in the default Debian APT repositories. Instead, download the latest release directly from the official GitHub repository:

https://github.com/coredns/coredns/releases

Download the archive matching your operating system and CPU architecture, for example:

https://github.com/coredns/coredns/releases/download/v1.14.2/coredns_1.14.2_linux_amd64.tgz

Extract the archive and move the binary into your system's executable path.

tar xvf coredns_*.tgz
mv coredns /usr/bin/

Next, create the CoreDNS configuration directory and the main configuration file.

mkdir /etc/coredns
vim /etc/coredns/Corefile

Populate the Corefile with the following configuration:

example.local.:53 {
    bind 0.0.0.0
    file /etc/coredns/db.example.local
    log
}

.:53 {
    bind 0.0.0.0
    forward . 1.1.1.1
    log
}

The first server block makes CoreDNS authoritative for the example.local zone, while the second forwards all other DNS queries to Cloudflare's public resolver.

Next, create the zone file referenced by the Corefile.

vim /etc/coredns/db.example.local

$TTL 120
@   IN  SOA ns.example.local. root.example.local. (
        2026020601 ; Serial
        120        ; Refresh
        1800       ; Retry
        1209600    ; Expire
        120 )      ; Minimum

@       IN      NS      ns.example.local.
ns      IN      A       10.1.1.1
@       IN      A       198.51.100.10
www     IN      A       198.51.100.11

To improve security, create a dedicated system user that will own the configuration files and run the CoreDNS process.

useradd -M -s /bin/false coredns
chown -R coredns:coredns /etc/coredns

Create the following systemd unit file at /etc/systemd/system/coredns.service:

[Unit]
Description=CoreDNS Server
After=network.target

[Service]
Type=simple
User=coredns
Group=coredns
ExecStart=/usr/bin/coredns -p 53 -conf /etc/coredns/Corefile
WorkingDirectory=/etc/coredns
Restart=always
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

[Install]
WantedBy=default.target
RequiredBy=network.target

Reload systemd, enable the service, and start CoreDNS.

systemctl daemon-reload
systemctl enable coredns
systemctl start coredns

You can verify that the service is running correctly with:

systemctl status coredns

If CoreDNS fails to start because port 53 is already in use, check whether systemd-resolved is enabled. On many Linux distributions, it binds to port 53 by default and prevents CoreDNS from listening on that port.

If necessary, disable and stop the service:

systemctl disable systemd-resolved
systemctl stop systemd-resolved

Finally, test the deployment from another machine on the network using nslookup:

nslookup www.example.local 10.1.1.1

A successful response should look similar to the following:

Server:         10.1.1.1
Address:        10.1.1.1#53

Name:   www.example.local
Address: 198.51.100.11

At this point, the Anycast deployment is fully operational. Client requests sent to 10.1.1.1 are automatically distributed across the available DNS servers by the network infrastructure, while BGP and ECMP ensure high availability and seamless failover in the event of a server or link failure.