Layer 3 DSR

Layer 3 Direct Server Return extends DSR capabilities beyond the local broadcast domain by encapsulating incoming packets inside IP-in-IP (IPIP) tunnels. This allows backend servers to reside on different subnets, VLANs, or even in different data centres while still benefiting from the performance advantages of Direct Server Return.

How L3 DSR Works

When a client packet arrives at the VIP, the L4 engine (nftlb) selects a backend server and encapsulates the original IP packet inside a new IP header. The outer header uses the load balancer's IP as the source and the backend server's real IP as the destination. The inner (original) packet retains the client's source IP and the VIP as the destination IP.

The encapsulated packet is routed through the network like any normal IP packet, crossing subnet boundaries and routers as needed. When it arrives at the backend server, the IPIP tunnel interface decapsulates it, revealing the original packet. The backend server processes the request and responds directly to the client using the VIP as the source address, just as in L2 DSR.

When to Use L3 DSR vs L2 DSR

Choose L2 DSR when all backends share the same VLAN for its simplicity. Choose L3 DSR when backends span subnets, when you need cross-site DSR, or when your network architecture prohibits large Layer 2 domains.

Configuring L3 DSR in Tula

To enable L3 DSR for a Layer 4 VIP:

1. Navigate to Load Balancing > L4 Virtual IPs and select or create a VIP.
2. Set the Mode to DSR and the DSR Type to Layer 3 (IPIP Tunnel).
3. Add backend servers with their routable IP addresses.
4. Configure your preferred scheduling algorithm and health checks.
5. Save and apply changes.

Tula configures nftlb to handle the IPIP encapsulation automatically. No manual tunnel configuration is required on the load balancer itself.

Backend Decapsulation Setup

Each backend server must be configured to receive and decapsulate IPIP-tunneled packets, accept traffic for the VIP, and suppress ARP for the VIP address.

Load the IPIP kernel module:

modprobe ipip

Create and configure the tunnel interface:

ip tunnel add tunl0 mode ipip local <BACKEND_IP>
ip addr add <VIP_ADDRESS>/32 dev tunl0
ip link set tunl0 up

Suppress ARP and configure reverse path filtering:

sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
sysctl -w net.ipv4.conf.tunl0.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0

Reverse path filtering (rp_filter) must be disabled on the tunnel interface because the decapsulated packets have a source IP (the client) that does not match the expected path for the tunnel interface.

Persist these settings via /etc/sysctl.d/99-dsr.conf and configure the tunnel interface in your system's network manager (e.g., netplan) so it survives reboots.

MTU Considerations

IPIP encapsulation adds a 20-byte outer IP header to every packet. If your network's standard MTU is 1500 bytes, the maximum payload inside the tunnel is effectively 1480 bytes. This can cause issues if:

• The original packet is exactly 1500 bytes and the Don't Fragment (DF) bit is set.
• Path MTU Discovery (PMTUD) is blocked by firewalls dropping ICMP "Fragmentation Needed" messages.

To mitigate MTU issues:

• Increase the network MTU to 1520 or higher (jumbo frames) on all links between the load balancer and backend servers, if your infrastructure supports it.
• Reduce the MSS by clamping TCP Maximum Segment Size on the backend servers to account for the tunnel overhead.
• Ensure ICMP is not filtered so that PMTUD can function correctly.

Verifying the Configuration

After setup, verify that the tunnel is operational:

ip tunnel show tunl0
ip addr show tunl0
tcpdump -i tunl0 -n    # Should show decapsulated traffic

Confirm that responses from backends reach clients by testing with curl or a browser against the VIP address, and verify that return traffic does not traverse the load balancer by monitoring its outbound interface counters.