Note: This post has been updated since I first wrote and is now kinda outdated. Read on to the end to see what I am currently doing. :)
I lost a lot of time & sleep over this the past two days and went down a rabbit hole trying to fix it. Eugh!
What I wanted to do was simple. Install WireGuard on my newly installed Raspberry Pi OS 64-bit. I have done this in the past when it was called Raspbian and 32-bit.
The instructions are typically simple. This blog post is what I followed last time around. The official instructions too are straightforward – enable backports in Debian and install. (Simply installing wireguard as per the official instructions isn’t enough as that results in an Error: unknown device type when launching a tunnel, we also need wireguard-dkms). There’s also the PiVPN project and I came across this blog post too which is quite comprehensive.
The basic idea is this: enable backports (or testing or unstable if you so wish), do apt-pinning to keep testing or unstable at a lower priority if you add them, and then apt install wireguard wireguard-tools wireguard-dkms. Easy peasy.
When installing wireguard-dkms though the install process would fail with errors like this (apologies for this error message being unreadable – something went wrong in the copy-paste and I didn’t notice until later):
|
1 |
DKMS make.log for wireguard-1.0.20200908 for kernel 5.4.51+ (aarch64)
Wed 21 Oct 09:41:48 BST 2020
make: Entering directory '/usr/src/linux-headers-5.4.51+'
AR /var/lib/dkms/wireguard/1.0.20200908/build/built-in.a
CC [M] /var/lib/dkms/wireguard/1.0.20200908/build/main.o
CC [M] /var/lib/dkms/wireguard/1.0.20200908/build/noise.o
CC [M] /var/lib/dkms/wireguard/1.0.20200908/build/device.o
CC [M] /var/lib/dkms/wireguard/1.0.20200908/build/peer.o
In file included from ./include/linux/types.h:6,
from /var/lib/dkms/wireguard/1.0.20200908/build/compat/compat.h:11,
from <command-line>:
./include/uapi/linux/types.h:5:10: fatal error: asm/types.h: No such file or directory
#include <asm/types.h>
^~~~~~~~~~~~~
In file included from ./include/linux/types.h:6,
from /var/lib/dkms/wireguard/1.0.20200908/build/compat/compat.h:11,
from <command-line>:
./include/uapi/linux/types.h:5:10: fatal error: asm/types.h: No such file or directory
#include <asm/types.h>
^~~~~~~~~~~~~
In file included from ./include/linux/types.h:6,
from /var/lib/dkms/wireguard/1.0.20200908/build/compat/compat.h:11,
from <command-line>:
./include/uapi/linux/types.h:5:10: fatal error: asm/types.h: No such file or directory
#include <asm/types.h>
^~~~~~~~~~~~~
In file included from ./include/linux/types.h:6,
from /var/lib/dkms/wireguard/1.0.20200908/build/compat/compat.h:11,
from <command-line>:
./include/uapi/linux/types.h:5:10: fatal error: asm/types.h: No such file or directory
#include <asm/types.h>
^~~~~~~~~~~~~
compilation terminated.
compilation terminated.
compilation terminated.
compilation terminated.
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/wireguard/1.0.20200908/build/main.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/wireguard/1.0.20200908/build/noise.o] Error 1
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/wireguard/1.0.20200908/build/device.o] Error 1
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/wireguard/1.0.20200908/build/peer.o] Error 1
make: *** [Makefile:1709: /var/lib/dkms/wireguard/1.0.20200908/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.4.51+' |
Not sure what the deal is there, and I spent some time trying wireguard from unstable & testing in case they were different. That led me down a rabbit hole wherein apt downloaded a newer kernel as a dependency (linux-image-5.8.0-0.bpo.2-cloud-arm64 and linux-image-cloud-arm64) and would fail coz these kernels can’t be pulled down as they depend on newer versions of gcc (gcc-10). In fact I then spent some time trying to get the latest gcc installed but that didn’t help and in the process I deleted some shared gcc library (which in turn broke apt)(turns out I am not the only guy to do this, which is good I guess) so had to go down another path to get the system working again by trying to find the deb file which had that shared library and manually installing all its dependent debs. I finally got the Pi working but I was no closer to my original goal.
I then tried updating to a newer kernel to what I was currently on. As you can see above I was on 5.4.51, and I figured heck WireGuard is part of the kernel from 5.6 so I might as well see if I can jump onto that. That was part of the reasoning behind upgrading gcc so I can get a newer kernel, as you’ll see above apt was trying to install kernel 5.8 for me from backports. It might be risky getting this from Debian as that kernel wouldn’t have any Pi specific bits, so I went with rpi-update instead (it’s there in the Raspberry Pi docs too). That got me up to kernel 5.4.72 but now I had a new issue in that the headers were missing and wireguard-dkms failed coz of that.
Not a problem, as again I am not the only one going down this pointless path. :) Found this GitHub post which pointed me to this super helpful script that will download the kernel headers for whatever version of the kernel you are on via rpi-update and set it all up for you. That post had others trying to get WireGuard working the same as me, and they had success with the script too so why not eh …
Didn’t work for me though! This time wireguard-dkms failed with a different error (apologies for this error message being unreadable – something went wrong in the copy-paste and I didn’t notice until later):
|
1 |
Building initial module for 5.4.72-v8+
Error! Bad return status for module build on kernel: 5.4.72-v8+ (aarch64)
Consult /var/lib/dkms/wireguard/1.0.20200908/build/make.log for more information.
dpkg: error processing package wireguard-dkms (--configure):
installed wireguard-dkms package post-installation script subprocess returned error exit status 10
Setting up libgmp-dev:arm64 (2:6.1.2+dfsg-4) ...
Setting up libmpfr-dev:arm64 (4.0.2-1) ...
Setting up libmpc-dev:arm64 (1.1.0-1) ...
Setting up gcc-8-plugin-dev (8.3.0-6) ...
Processing triggers for libc-bin (2.28-10) ...
Errors were encountered while processing:
wireguard-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)
Building modules, stage 2.
MODPOST 1 modules
ERROR: "__stack_chk_guard" [/root/Scripts/wireguard-linux-compat/src/wireguard.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:94: __modpost] Error 1
make[1]: *** [Makefile:1645: modules] Error 2
make: *** [Makefile:26: module] Error 2 |
Fancy that … __stack_chk_guard!
Needless to say, I now went down that rabbit hole. Didn’t have much luck except another GitHub post which in turn pointed to a bunch of Raspberry Pi forum posts but none really helped me. (And they were a different issue to mine I think. Those posts were talking abt the 32-bit /boot/kernel7.img and /boot/kernel7l.img, while I was using the 64-bit /boot/kernel8.img and that wasn’t exhibiting the same error for them).
I tried compling WireGuard from source (steps here) but that too gave the same error. This wasn’t going anywhere.
One thing was clear… it looked like this was a bug in the 5.4.72 kernel I was trying, so I am better of sticking with the default kernel and trying to dig more into the error there. So I removed the headers I had installed via the script, downgraded to the default kernel (steps are in the official docs), removed all the kernels I had downloaded (just to clean things up)… and this time I thought let’s just try compiling WireGuard from source anyways as I have it downloaded and ready. And what do you know, that worked! Yeah, weird.
So here’s what you need to do on Raspberry Pi OS 64-bit (beta) if you are similarly stuck as me:
|
1 |
# Get the kernel headers and dependencies (this is slightly different from the WireGuard instructions in that I ask for raspberrypi-kernel-headers)
sudo apt install libelf-dev raspberrypi-kernel-headers build-essential pkg-config
# All the steps below are from the official WireGuard compile instructions
# Get the source
git clone https://git.zx2c4.com/wireguard-linux-compat
git clone https://git.zx2c4.com/wireguard-tools
# Compile the module
make -C wireguard-linux-compat/src -j$(nproc)
sudo make -C wireguard-linux-compat/src install
# Compile and install wg
make -C wireguard-tools/src -j$(nproc)
sudo make -C wireguard-tools/src install |
And that’s it, wireguard should work for you. This also installs the systemd service.
The only catch with this method is that if any new versions appear in the repo I won’t automatically update. The latest commit to that repo was 6 weeks ago, and I imagine it won’t have much updates as the repo is a backport for kernels 3.10 – 5.5 (as WireGuard is now a part of the Linux kernel since 5.6).
Update: Thomas Schubert wrote in with the following helpful update that explained why I had the stack protector errors with the latest kernel version.
I just read your blog post after encountering the same problem. If you want to use a newer kernel, modifying [1] the referenced script from your blog post to pull in the 5.4.68-v8+ kernel (and installing both generated packages) instead of the latest one should work just fine for compiling wireguard via dkms.
Every kernel after that misses the __stack_chk_guard symbol in Module8.symvers [2] since newer gcc versions (9+), now used to compile the distributed compiled kernels, use the thread local storage for that. And sadly there is no easy way for getting gcc-9 or newer on debian buster. The alternative would be to compile the kernel yourself with gcc-8, which is of course not a very fun thing to do on a Raspberry Pi.
[1] https://paste.rs/Jx6
[2] https://github.com/raspberrypi/firmware/blob/master/extra/Module8.symvers
Update 2: Latest kernel update on 26th Oct broke things again. I am now on 5.4.72-v8+. Bummer!
Update 3: On 27th Oct I tried switching to the modified version of the script referenced by Thomas in the comment above. That script doesn’t work as is actually, so what you need to do is get this script referenced in this GitHub thread (both of which I linked to in my original post above) and make some changes. Specifically, take this bit:
|
1 |
FIRMWARE_REV=$(git ls-remote "https://github.com/Hexxeh/rpi-firmware" refs/heads/$git_branch | awk '{print $1}') |
And change it to:
|
1 |
FIRMWARE_REV=64391b28301e92914ba35e345da6e11fab3afd6b |
What we are doing here is telling the script to get a specific commit rather than the latest kernel. How do we know which commit? Check this GitHub commit list of the repo. This commit pulls the kernel version 5.4.68 from Sept 30. All kernel versions after that are broken as far as WireGuard is concerned because they were compiled with gcc-9.
Then modify this section:
|
1 2 3 4 5 6 7 8 |
setup_git_fw() { if [[ -d "$workdir/rpi-firmware" ]]; then ( sudo rm -rf "$workdir"/rpi-firmware.old || true ) ( sudo mv "$workdir"/rpi-firmware "$workdir"/rpi-firmware.old || true ) ( sudo rm -rf "$workdir"/rpi-firmware.old || true ) fi cd "$workdir" && git clone --depth=1 -b $git_branch $git_base } |
With this:
|
1 2 3 4 5 6 7 8 |
setup_git_fw() { if [[ -d "$workdir/rpi-firmware" ]]; then ( sudo rm -rf "$workdir"/rpi-firmware.old || true ) ( sudo mv "$workdir"/rpi-firmware "$workdir"/rpi-firmware.old || true ) ( sudo rm -rf "$workdir"/rpi-firmware.old || true ) fi cd "$workdir" && git clone --depth=12 -b $git_branch $git_base && git -C "$workdir/rpi-firmware" checkout $FIRMWARE_REV } |
I have marked the line to change. All this does is instead of getting the repo with a depth of 1 (i.e. just the latest commit) I get the past 12 commits (as that’s where the commit we want is at). The trouble is this depth will keep changing as more commits get added so this is something to manually tweak everytime.
This worked fine for me, I rebooted etc. Then I removed the newly installed kernels. And after this instead of building WireGuard from source like I did last time, I thought let’s build via the package and so did an apt install wireguard wireguard-modules wireguard-dkms. All of this went fine without an issue, but after a reboot the Pi failed to boot! Oops. :)
I connected it to the TV and saw it had a rainbow pattern. This meant the Pi couldn’t find the kernel.img (or equivalent) files. I hooked up the Pi’s disk to my laptop and sure enough the /boot partition didn’t have any image files. So I flashed a fresh Pi image to a blank microSD and compared the /boot partitions of both. There were a lot of files missing on the one that was attached to the Pi so I copied over all the missing files from the microSD, put the disk back in the Pi and booted and now all was good (there were a bunch of complaints due to the different kernel versions I guess, but no big deal).
At this point I was kind of back to square one, with WireGuard still not working.
During this time I had been emailing Thomas and he said he just uses the latest 5.9 kernel branch now. That’s easy via rpi-update:
|
1 |
sudo BRANCH=next rpi-update |
So I figured what the heck, let’s try that. Updated, rebooted, and bam I am now on the 5.9 branch. Next I installed wireguard-tools coz everything else is in the kernel now (remember WireGuard got added to kernel 5.6). When installing apt wanted to pull in the 5.8 image from backports so I used the following command to only install wireguard-tools:
|
1 |
udo apt install wireguard-tools linux-image-5.8.0-0.bpo.2-cloud-arm64- linux-image-cloud-arm64- |
That done I finally have WireGuard working again. Yay! Until things break again I suppose coz of some bug in 5.9. :)
Update 4: Last update hopefully! Looks like the new 5.4 kernel does work but you need to follow some additional steps. See this thread.