Posted on by Christian

Subsetting CRXN into the real Internet

Recently I looked into deavmi’s project named CRXN where he’s leveraging yggdrasil’s peer-to-peer VPN routing mechanism to build an overlay network using IPv4. Since several of my online services are not connected to yggdrasil, I was wondering if it’s possible to just NAT64 CRXN into IPv6. Sure it worked quite fine, I’ll show you how we pulled that off.

Shitty pro-tip: You can also use this instruction to expose your super secured enterprise network to the public. Instead of yggdrasil you use your corporate VPN.

UPDATE: I had to setup mss clamping. Check the last paragraph how I did it. I also fixed some errata in my systemd service file.

Internet numbers first

First we need a /32 IPv4 assignment from deavmi which is used routed on top of yggdrasil. I got 10.5.0.1/32 assigned. Also we need atleast a /96 IPv6 assignment to our Linux VM (Debian 10) which is acting as a router here. I picked 2a04:5b80:300:4::/64 for this.

For IPv6 I’m natively connected to the Internet. The CRXN IPv4 is routed on-top of yggdrasil to my publickey there.

Install tayga and yggdrasil

Install yggdrasil from its website and tayga using apt. Pretty much straight-forward.

Unfortunately tayga on Debian is not providing a template unit file. So I needed to create one loosely based on that from CentOS/RHEL:

[Unit]
Description=Simple, no-fuss NAT64
After=syslog.target network.target
[Service]
PIDFile=/run/tayga-%i.pid
LimitCORE=infinity
Type=simple
PrivateTmp=true
ExecStart=/usr/sbin/tayga --pidfile /var/run/tayga-%i.pid -d --config /etc/tayga/%i.conf
[Install]
WantedBy=multi-user.target

For my purpose here I could have used the Debian standard, but I’m sure I’ll do something else using tayga in the future. So let’s keep it clean from the get go.

Setting up tayga and yggdrasil

Let’s begin with yggdrasil. First we need to configure some peers to make it work and then we need to configure the overlay routing. Both happens in /etc/yggdrasil.conf.

After we configured some peers, we can enable the tunnel routing like this:

  IfName: ygg

… some parts skipped …

  TunnelRouting:
  { 
    # Enable or disable tunnel routing. 
    Enable: true

… some parts skipped …
 
    # IPv4 subnets belonging to remote nodes, mapped to the node's public  
    # key, e.g. { "a.b.c.d/e": "boxpubkey", ... } 
    IPv4RemoteSubnets: {
        "10.0.0.0/8": "6fa497818044d164d7f895a69e4678f03d99820b23c1912970e9255910bd8308"
    }
 
    # IPv4 subnets belonging to this node's end of the tunnels. Only traffic  
    # from these ranges will be tunnelled. 
    IPv4LocalSubnets: [
        "10.5.0.1/32"
    ]
  }

Note the 10.0.0.0/8 which is the CRXN and also note the local subnet which is our assignment and also note the publickey in remote subnets which is deavmi’s. Also note the interface name I specified. It will be important later.

Next up we need to configure tayga to provide some mapping from IPv6 to IPv4 and vice-versa. You’ll note that I’m using 100.100.0.0/16 here as I’m going to statefully NAT that network pool to my single address assignment. This could also be a larger assignment and a kind of stateless mapping. I’m not going into details why and how here. It will just blow the scope of this blog post.

# /etc/tayga/ygg.conf
tun-device nat64-ygg
ipv4-addr 100.100.0.1
prefix 2a04:5b80:300:4::/96
dynamic-pool 100.100.0.0/16
data-dir /var/lib/tayga/ygg

After preparing the configuration. We need to configure routing on the system as well.

But instead of going down the shell script rabbit hole, we can edit/override the systemd service unit file to pass some additional commands to setup the routing and what not.

Let’s begin with yggdrasil. By calling systemctl edit yggdrasil we can extend the unit file like this:

[Service]
ExecStartPost=/usr/sbin/ip address add 10.5.0.1/32 dev ygg 
ExecStartPost=/usr/sbin/ip route add 10.0.0.0/8 dev ygg src 10.5.0.1 scope global

That configures our assigned address as well as the remote subnet route.

Next up, we need to do the same with tayga. It’s a little bit more since we need to get iptables for NAT onboard. Also done through systemctl edit tayga@ygg:

[Service]
ExecStartPost=/usr/sbin/ip link set nat64-ygg up

ExecStartPost=/usr/sbin/ip addr add 100.100.0.1/32 dev nat64-ygg
ExecStartPost=/usr/sbin/ip route add 100.100.0.0/16 dev nat64-ygg proto 4

ExecStartPost=/usr/sbin/iptables -t nat -I POSTROUTING -s 100.100.0.0/16 -d 10.0.0.0/8 -j MASQUERADE
ExecStopPost=/usr/sbin/iptables -t nat -D POSTROUTING -s 100.100.0.0/16 -d 10.0.0.0/8 -j MASQUERADE

ExecStartPost=/usr/sbin/ip route add blackhole 2a04:5b80:300:4::/64 dev nat64-ygg
ExecStartPost=/usr/sbin/ip route add 2a04:5b80:300:4::a00:0/104 dev nat64-ygg

You might have noticed the route add blackhole magic there. It’s because of the routed /64 where I’m only using a /96 out of it. To avoid routing loops, I’m blackholing the less specific here.

Start and mark as autostarting both things using systemctl like this:

systemctl start tayga@ygg
systemctl enable tayga@ygg
systemctl start yggdrasil
systemctl enable yggdrasil

Make the routing work

Common safety switch or pitfall (depending on your view) is the switch to actually make Linux a defacto router. It’s by default off, so you’ll need to create a file and set it on.

# /etc/sysctl.d/router.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Apply it using sysctl -p /etc/sysctl.d/router.conf and there you have it.

That’s pretty much it at this point. Try to ping some CRXN host like this:

[~] ping 2a04:5b80:300:4::10.1.0.3
PING 2a04:5b80:300:4::10.1.0.3(2a04:5b80:300:4::a01:3) 56 data bytes
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=1 ttl=57 time=807 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=2 ttl=57 time=1343 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=3 ttl=57 time=828 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=4 ttl=57 time=750 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=5 ttl=57 time=699 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=6 ttl=57 time=696 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=7 ttl=57 time=720 ms
64 bytes from 2a04:5b80:300:4::a01:3: icmp_seq=8 ttl=57 time=744 ms
^C
--- 2a04:5b80:300:4::10.1.0.3 ping statistics ---
9 packets transmitted, 8 received, 11% packet loss, time 8026ms
rtt min/avg/max/mdev = 696.809/823.900/1343.824/201.499 ms, pipe 2

Have fun and don’t mind the packetloss. It’s from Europe to Africa.

Troubleshooting

Since I’m writing this blog post out of history and my faint memories there might be some gotchas in it.

However, consider the following tricks and places to check your stuff:

  • tcpdump the ygg or nat64-ygg devices
  • check iptables-save and iptables -L -n if there’s nothing botched, also keep an eye on the default policy for FORWARD
  • check the error logs of tayga and yggdrasil
  • make sure your IPv6 network is actually routed to your VM
  • make sure the tunnel routing in yggdrasil is enabled: TunnelRouting: true

MSS Clamping

I figured that some networks would drop ICMP packets regarding too large packets.

This is a bodged solution to prevent issues. Add following statements when calling systemctl edit tayga@ygg:

ExecStartPost=/usr/sbin/iptables -I FORWARD -p tcp -s 100.100.0.0/16 -d 10.0.0.0/8 --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
ExecStopPost=/usr/sbin/iptables -D FORWARD -p tcp -s 100.100.0.0/16 -d 10.0.0.0/8 --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

This tells the kernel to overwrite the MSS negotiated within the TCP stream according the discovered MTU.