Peter Cai

@PeterCxy

Some random guy out there. en_US / zh_CN

https://sn.angry.im/@PeterCxy Guestbook

Wireguard with Network Namespace + BitTorrent / Shadowsocks / ...

Backrgound

I have long been running a BT/PT download box on one of my dedicated servers. The reason that I have extremely poor uplink at my home broadband and running any kind of P2P software is simply killing the network. However, putting those software on a server without any protection is a bad idea -- they will happily announce your server IP everywhere, and, * cough *, some nasty things may happen to you, even by just downloading some pretty innocent files. I need at least some kind of protection to avoid leaking the real IP to the torrent world. Using SOCKS5 proxy alone is not the best idea either: Anything in the BT protocol, for example, DHT, can easily leak the IP address, if the BT client itself is not isolated in a way that it can't see the real IP.

This is the same with my personal proxy service. Residing in China, there is basically no way to connect to VPN services abroad directly, even without them being blocked -- ISPs here just throttle UDP traffic in an extreme way, and TCP VPNs are unbearably slow and easily interrupted with RST. Normally we use self-hosted encrypted proxies instead of VPNs to bypass this, usually hosted on cheap VPSes such as Vultr. However, with this way, it is easy to leak the proxy IP (the VPS IP) to software because they can simply record the mapping between the source IP and the account holder. What I need is still another layer of protection -- that I should use a different outbound IP than the server itself.

Unfortunately, enabling VPN on a server is not something as easy as doing it on your own computer. You can't just simply set the default route, because by doing so, access to the server through its main IP will be broken, and you will be left locked, lonely, helpless, outside of the server. Moreover, only enabling VPN is not enough at all, since the public IP is assigned on the primary network device, and it is fairly simple to fetch that address (and many software will actually do this, announcing every possible IP to the public). A full isolation of network is needed, but I do not want to introduce a complete container like Docker, because it seems just way too excessive.

Network Namespace

Luckily, Linux has this implemented for us. The ip-netns(8) tool manages a cool feature brought by the Linux kernel, Network Namespace, which is exactly what we need here. Actually, full container implementations will also leverage this feature to virtualize their network environment, but we are only using the network part here, which is much more lightweight than a container virtualization.

A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices.

So, all we have to do is to find some way to put the VPN tunnel device in a network namespace, and set the default route only in that namespace. There will be nothing but the VPN device and the only default route visible inside the namespace, which is pretty safe for most software not designed to intentionally escape from namespaces.

The Legacy of OpenVPN

Previously I was a user of ProtonVPN, which was a great VPN to use for my purpose (except that it has completely no IPv6 support, I was expecting VPNs to implement IPv6 NAT...). Since it used OpenVPN as its main VPN software, I used to make use of OpenVPN's up and down scripts to enable VPN in network namespaces.

Since OpenVPN is a pretty old and widely-adopted protocol, there are plenty of guides on how to realize this with OpenVPN. What I used was a script here that moves the TUN interface into a network namepsace managed by the script upon finalizing the connection. The script is pretty mature, and works just fine.

However, ProtonVPN is starting to breaking down these days. Though I have no idea, but since some random day, ProtonVPN started to become null routed randomly. I am sure it is not blocked by ISP because I only run it on my VPS outside of China and I really cannot see routes to its IPs in my BGP sessions elsewhere. It just seems to be down without reason. Besides, OpenVPN is much too bloated and sometimes causing problems itself. Since Linus Torvalds has said that Wireguard should be merged into mainline Linux kernel soon, I started to look for an alternative solution based on Wireguard.

Attempt: Wireguard + wg-quick

After some searching I found a pretty good Wireguard VPN provider with both IPv4 and IPv6 NAT support. Wireguard is pretty easy to configure, since the provider will often provide something lie this:

[Interface]
PrivateKey = blahblah
Address = 192.168.x.x/24, fe80::xxx/64
DNS = x.x.x.x

[Peer]
PublicKey = blahblah
AllowedIPs = 0.0.0.0/0,::0/0
Endpoint = x.x.x.x:xxxx

which is normally placed in /etc/wireguard/wireguard-config-name.conf. Such configuration is meant for the tool wg-quick(8). However, this tool doesn't seem to support Network Namespace out of the box. I did a naïve attempt like below:

ip netns add vpn
ip netns exec vpn ip link add dev wireguard-vpn type wireguard
ip netns exec vpn wg-quick my-config-name

...and of course, it failed. Wireguard will also obey the network namespace rules while establishing its underlying sockets, and that was why this failed -- you can't connect to any VPN in a newly-created network namespace without any route. Resolving this by introducing the host network to the namespace didn't seem appealing to me, since it will be very complex to configure and will still potentially leak the real IP.

The Real Solution

After some Google-fu, I found an official document of Wireguard that described an interesting property of the Wireguard driver: it "remembers" the network namespace where it was created.

it remembers the namespace in which it was created. "I was created in namespace A." Later, WireGuard can be moved to new namespaces ("I'm moving to namespace B."), but it will still remember that it originated in namespace A.

WireGuard uses a UDP socket for actually sending and receiving encrypted packets. This socket always lives in namespace A – the original birthplace namespace.

This is exactly what we were looking for! If Wireguard could send its underlying UDP packets in a different namespace than where the Wireguard device is, we can have a completely "clean" network namespace that has only the Wireguard as default route while having Wireguard being able to connect via the original host network!

All we have to do now is, create the Wireguard interface, then apply the configuration, and move it to a newly-created network namespace, then set the IPs, routes etc. We can no longer use wg-quick for this, because the tool is meant for quick configuration and will configure the routes for us in the main namespace (according to AllowedIPs). We have to use a weaker version of it, called wg setconf, instead. Note that we have to comment out the DNS and Address lines in the provided configuration if present, because wg setconf does not support setting DNS and IP address.

I tried with a simple script according to the above procedure

#!/bin/bash
CONFIG_NAME="$1"
DEV_NAME="wg-$CONFIG_NAME"

ip netns add $CONFIG_NAME
ip netns exec $CONFIG_NAME ip link set lo up
ip link add dev $DEV_NAME type wireguard
wg setconf $DEV_NAME /etc/wireguard/$CONFIG_NAME.conf
ip link set $DEV_NAME netns $CONFIG_NAME up

Note that I have set the name of the namespace to be the same as the configuration file name. Run it with ./script.sh wireguard-config-name, and it successfully set up the namespace with the Wireguard device in it. However, the IP addresses was not set because we did not use wg-quick and commented out the Address line in configuration. At this point, I could have simply hard-coded the addresses in the script, but it did not sound like an elegant solution

I did a not-so-elegant-but-better-than-nothing hack, which was to make use of the commented-out Address line: we could simply parse the line (ignoring the #) and extract the addresses from there!

addrs=$(grep -oP "#Address = \K(.*)" /etc/wireguard/$CONFIG_NAME.conf)
IFS=", "; for addr in $addrs; do
  if [[ $addr = *":"* ]]; then
    # IPv6
    ip netns exec $CONFIG_NAME ip -6 addr add $addr dev $DEV_NAME
  else
    # IPv4
    ip netns exec $CONFIG_NAME ip addr add $addr dev $DEV_NAME
  fi
done

Adding this to the previous script, we now have the IP properly assigned to the Wireguard device. Now, we could pretty much do the same with the routes, by extracting them from AllowedIPs, but somehow I decided that it was better to just set the default routes for both IPv4 and IPv6

ip netns exec $CONFIG_NAME ip route add default dev $DEV_NAME
ip netns exec $CONFIG_NAME ip -6 route add default dev $DEV_NAME

Now we are done with the script to set the interface up. Tearing it down is much simpler

#!/bin/bash
CONFIG_NAME="$1"

ip netns del $CONFIG_NAME

Running Systemd Services inside the Namespace

At this point, we can use ip netns exec to run programs inside the network namespace. However, I would like to run systemd services inside it. To fully leverage the abilities of systemd, I decided to first write a service to manage the Wireguard interface in network namespace. Assuming that the up and down scripts described above are placed in /path/to/wg-up.sh and /path/to/wg-down.sh, I wrote a service named wg-netns@.service

[Unit]
Description=Execute Wireguard in network namepsace
After=network-online.target

[Service]
User=root
Type=oneshot
RemainAfterExit=true
ExecStart=/path/to/wg-up.sh %i
ExecStop=/path/to/wg-down.sh %i

[Install]
WantedBy=multi-user.target

Then enable it by systemctl enable wg-netns@wireguard-config-name. Now, we can use systemctl edit some-service to put some-service into the namespace by writing

[Unit]
Requires=wg-netns@wireguard-config-name.service
After=wg-netns@wireguard-config-name.service

[Service]
User=
User=root
ExecStart=
ExecStart=/usr/bin/ip netns exec wireguard-config-name /path/to/the/program

in the editor provided by systemctl edit. Note that this configuration is very generic, and you may need to consult the original service file for the complete command to put in place of /path/to/the/program. Besides, by using such configuration, you are also running the program as root, which can be a security concern and could make some program behave abnormally. You may need to add su -u blah before the actual command (after ip netns exec wireguard-config-name) to switch to the proper user to run your program.

Now you can enable the service as normal. Services configured like this will only start when wg-netns@wireguard-config-name is started, and will restart or stop if wg-netns@wireguard-config-name is restarted or stopped.

One More Thing: Exposing Ports within the Namespace

All the configuration above are perfectly fine if we do not need any service running in the namespace to be accessible to the outside. But for the BT client and the Shadowsocks server, we must at least be able to access their listening TCP port in order to control / use them while retaining the isolation. My solution was to set up a separate veth interface and assign the namespace a separate internal IP address without NAT, so that I can access the ports via the internal IP or forward them to the outside while forbidding the services themselves to break the isolation.

This step is much simpler. We just create a pair of veth devices, put one of them into the namespace, then assign a pair of IPs to each end.

ip link add dev "$CONFIG_NAME"0 type veth peer name "$CONFIG_NAME"1
ip link set "$CONFIG_NAME"0 up
ip link set "$CONFIG_NAME"1 netns $CONFIG_NAME up
ip addr add $PRIVATE_ADDRESS_HOST dev "$CONFIG_NAME"0
ip netns exec $CONFIG_NAME ip addr add $PRIVATE_ADDRESS_CLIENT dev "$CONFIG_NAME"1

...where PRIVATE_ADDRESS_HOST is the internal address to be assigned to the host and PRIVATE_ADDRESS_CLIENT is the address to be assigned to the client. This is normally something like 192.168.1.1. In the script, I actually wrote like

source ${BASH_SOURCE%/*}/ext/$CONFIG_NAME.conf
if $PRIVATE_VETH_ENABLED; then
  ip link add dev "$CONFIG_NAME"0 type veth peer name "$CONFIG_NAME"1
  ip link set "$CONFIG_NAME"0 up
  ip link set "$CONFIG_NAME"1 netns $CONFIG_NAME up
  ip addr add $PRIVATE_ADDRESS_HOST dev "$CONFIG_NAME"0
  ip netns exec $CONFIG_NAME ip addr add $PRIVATE_ADDRESS_CLIENT dev "$CONFIG_NAME"1
fi

..so that you can have a ext/wireguard-config-name.conf (relative to the location of the up script, corresponding to /etc/wireguard/wireguard-config-name.conf) with additional variables about the internal IP which is not related to Wireguard itself

#!/bin/bash
PRIVATE_VETH_ENABLED=true
PRIVATE_ADDRESS_HOST="192.168.123.1/24"
PRIVATE_ADDRESS_CLIENT="192.168.123.2/24"

Correspondingly, you have to do something to tear down the veth pair in the down script

source ${BASH_SOURCE%/*}/ext/$CONFIG_NAME.conf

if $PRIVATE_VETH_ENABLED; then
  ip netns exec $CONFIG_NAME ip link del dev "$CONFIG_NAME"1
  ip link del dev "$CONFIG_NAME"0
fi

You can then set up port forwarding or anything else to this internal IP.

Now you have a complete working setup of Wireguard inside network namespace.

Source code

I have uploaded the source code of my completed setup to https://git.angry.im/PeterCxy/wg-netns.


You'll only receive email when Peter Cai publishes a new post

More from Peter Cai: