A while ago I had written a post on TailScale and WireGuard (and policy based routing). A reader (Doug Miller) wrote to say that he was using WireGuard via the Mullvad VPN app and whenever he started VPN TailScale would stop working. He was curious to know if I had encountered this.
(Mullvad is the VPN provider I too use btw. I don’t use their app though, I use the WireGuard app with the config pointing to Mullvad).
I was curious and so I installed Ubuntu 20.04 (which is what he was using) in a VM and took a look at things. Ubuntu use Nftables rather than IPtables and that was new to me. Everything else was similar to what I had blogged about earlier – the IProute2 stuff about policy based routing – and the issue here didn’t seem to be policy based routing related, rather it was to do with the firewall. When I’d delete the firewall rules added by the Mullvad client TailScale would work fine. (Moreover, when pinging the message was ping: sendmsg: operation not permitted again indicating a firewall).
The fix was simple eventually – add two rules to the rules created by Mullvad, allowing access to & from the tailscale interface. However, since I took a look at Nftables, and I am sure I’ll forget it in a few days, I wanted to jot down the commands here for future reference.
What is Nftables?
For a quick intro on what Nftables is and how it compares to IPtables check out this article from ungleich.ch. Also great references are the Arch Wiki , Gentoo Wiki, and to a lesser extent the Debian Wiki. Also, nft(8) manpage.
You may already be using Nftables
I hadn’t realized, but since I had some VMs running Debian Buster I was already using Nftables. But Buster only uses Nftables in the backend and users deal with it via an IPtables layer (thus you use the IPtables syntax) and that’s why I hadn’t noticed it. I remember reading about Nftables in the Debian IPtables Wiki page earlier but hadn’t paid much notice.
The Syntax
For some reason the syntax of Nftables rules reminds me of OpenBSD’s pf. There’s a certain similarity, but it’s not the same. It’s very different from IPtables for sure and feels more like pf.
For example here’s a pf rule to allow incoming SSH (port 22):
|
1 |
pass in quick on egress proto tcp from any to egress port 22 |
The equivalent in Nftables is:
|
1 |
nft add rule ip filter input protocol tcp dport 22 accept |
The slight difference is Nftables is adding this rule to a table called filter (I can have other tables too, the name doesn’t matter) to its input chain, while pf doesn’t have that concept. And what I typed above is the command to add a rule on the fly, while with pf the rule was in a file.
Which brings us to another commonality. With pf I can save rules to /etc/pf.conf; similarly in Nftables I can save rules to /etc/nftables.conf. IPtables has no notion of saving and restoring rules (but you could do it in other ways using iptables-save and iptables-restore).
View all rules
To see all your Nftables rules do:
|
1 |
nft list ruleset |
To save them do to a file do (the -s removes any stateful info):
|
1 |
nft -s list ruleset > ~/somefile |
Tables
Creating a table:
|
1 2 |
# nft add table family_type table_name nft add table inet my_table |
The family_type can be ip, ip6, inet (for both ip & ip6), arp, or bridge.
Listing tables:
|
1 |
nft list tables |
Listing chains and rules in a table:
|
1 2 |
# nft list table family_type table_name nft list table inet my_table |
Delete table:
|
1 2 |
# nft delete table family_type table_name nft delete table inet my_table |
You can see the pattern… It extends to other things too.
Chains
I am going to be lazy here.
|
1 2 3 4 5 |
# Creating a chain nft create chain family_type table_name chain_name # Deleting a chain nft delete chain family_type table_name chain_name |
Chains have a concept of “base” chains. This seems to be when you “hook” a chain to the networking stack. Then you have to specify a priority too, with lower number having higher priority. I think this is similar to how in IPtables you have the default chains (INPUT, OUTPUT, etc.) and you could have other chains you jump to. In Nftables there’s no default chains so you create a chain (of whatever name), define it as a default by hooking it up the networking stack, and that’s it. You can have multiple chains hooked to the networking stack and that’s where priority comes in.
Here’s how you create a new chain and hook it etc.
|
1 |
nft add chain family_type table_name chain_name '{ type chain_type hook hook_type priority priority_value ; }' |
The chain_type can be filter, route, or nat. Of these route can only be used in the output hook (it’s used to do a route lookup in case a packet has changed after processing by this chain); and nat can be in prerouting, input, output, or postrouting.
The hook_type can be prerouting, input, forward, output, or postrouting.
It’s all quite similar to IPtables.
The default priority value for a filter chain is 0. You can do negative numbers too to override something with priority 0. In the example below the chain of table2 has higher priority over table1:
|
1 2 3 4 5 6 7 |
# create two tables nft add table inet table1 nft add table inet table2 # add chains nft add chain inet table1 filter '{type filter hook input priority 0; }' nft add chain inet table2 filter '{type filter hook input priority -1; }' |
Negative priorities have some quirks I believe (see this PDF). In the Mullvad case I actually tried using a negative priority chain to override the one created by Mullvad, and that didn’t seem to take effect (or I could have created the chain wrongly)… so I am unsure about this.
You can edit chains too.
Rules
First command adds a rule at the end of the existing list (or after handle_value if specified). Second commands adds at the top of the list (or before handle_value).
|
1 2 3 |
nft add rule family_type table_name chain_name [handle handle_value] statement nft insert rule family_type table_name chain_name [handle handle_value] statement |
The key thing here is statement. These are things like:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# accept from source IP address 127.0.0.1 ip saddr 127.0.0.1 accept # accept from incoming interface lan0 iifname "lan0" accept # accept from any incoming interface matching this wildcard iifname "tailscale*" accept # SNAT/ Masquerade on outgoing interface oifname "wan0" masquerade # self explanatory due to the comments iif != lo ip daddr 127.0.0.1/8 counter drop comment "drop connections to loopback not coming from loopback" ip protocol icmp counter accept comment "accept all ICMP types" tcp dport 22 counter accept comment "accept SSH" |
Nice!
Back to the Mullvad/ TailScale issue
In the case of Mullvad it looks like the latest release of their agent (2020.7; previous one being 2020.6) has started adding tables to block traffic. (Doug, who reported this discovered it to be the case). Of these the table called mullvad is the one that seemed to be blocking outgoing traffic. I could see in its rules that it was only allowing outgoing via its tunnel interface and so the fix was to just add allow rules for the TailScale interfaces. I did this to both the incoming and outgoing chains and thus all I did was the following:
|
1 2 |
nft insert rule inet mullvad output oifname "tailscale*" accept nft insert rule inet mullvad input iifname "tailscale*" accept |
I used oifname and iifname instead of oif and iif so I can do a wildcard on any tailscale interfaces.
I suppose there’s some way to run this once the Mullvad agent launches but I didn’t bother much as I don’t use the agent (or Ubuntu) so it wasn’t much relevant to me. Am glad I came across this though as I got to discover Nftables. Good to know that one.