Contact

Subscribe via Email

Subscribe via RSS

Categories

Recent Posts

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

KiTTY with Fedora 22/ OpenSSH 6.9 gives a blank screen and hangs

The KiTTY SSH client for Windows gives a blank screen and hangs when connecting to my Fedora 22 box (which runs OpenSSH 6.9). KiTTY is a fork of PuTTY, which I am happy with, so there’s no particular reason for me to use KiTTY except that I am checking it out.

kitty hungAfter a while it gives an error that the server closed the connection.

kitty fatalI have an Ubuntu Server 14.10 and CentOS 6.7 and KiTTY works fine with them. So it seems to be related to the version of OpenSSH in Fedora 22.

Fedora 22 logs show entries like these:

Looks like key-exchange was failing?

/*

Note to self: An alternative command syntax to the above is this:

I also noticed that including SELinux in the output gave a bit more info. Not that it helped but it’s worth mentioning here as a reference to myself. For that I include the sshd.service unit and also the SELinux context for SSHD.

Tip: After typing journalctl it is possible to keep pressing TAB to get the fields and their values. That’s how I discovered the SSHD context above.

On the topic of journalctl: this intro post by its creator and this DigitalOcean tutorial are worth a read.

Another useful tip with journalctrl: by default it doesn’t wrap the output but you can scroll left and right with the keyboard. This isn’t useful when you want to copy-paste the logs somewhere. The work-around is to use the --no-pager switch so everything’s dumped out together and then pipe it into less which wraps the text:

*/

I turned on SSH packet and raw data logging in KiTTY.

kitty logginFrom the log files I could see a sequence of events like this:

And with that it stops. So it looked like the DH Key Exchange was failing. Which confirms what I saw on the server side too.

SSH protocol 2 supports DSA, ECDSA, Ed25519 and RSA keys when clients establish a connection to the server. However, the first step is a Key Exchange for forward-secrecy and for this the DH algorithm is used.  From the sshd(8) manpage:

For protocol 2, forward security is provided through a Diffie-Hellman key agreement.  This key agreement results in a shared session key.  The rest of the session is encrypted using a symmetric cipher, currently 128-bit AES, Blowfish, 3DES, CAST128, Arcfour, 192-bit AES, or 256-bit AES.  The client selects the encryption algorithm to use from those offered by the server.  Additionally, session integrity is provided through a cryptographic message authentication code (hmac-md5, hmac-sha1, umac-64, umac-128, hmac-ripemd160, hmac-sha2-256 or hmac-sha2-512).

From the sshd_config(5) manpage:

The supported algorithms are:

curve25519-sha256@libssh.org
diffie-hellman-group1-sha1
diffie-hellman-group14-sha1
diffie-hellman-group-exchange-sha1
diffie-hellman-group-exchange-sha256
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521

The default is:

curve25519-sha256@libssh.org,
ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,
diffie-hellman-group-exchange-sha256,
diffie-hellman-group14-sha1

The KexAlgorithms option can be used to change this but the point is all these are variants of the DH algorithm.

KiTTY has the following algorithm selection options:

kitty options

For protocol 2 only the first three (Diffie-Hellman options) apply. The fourth one (RSA-based key exchange) only applies for protocol 1. Comparing this with the preferred order for OpenSSH only the first and second options are common:

  • Diffie-Hellman group exchange (a.k.a. diffie-hellman-group-exchange-sha256)
  • Diffie-Hellman group 14 (a.k.a. diffie-hellman-group14-sha1)

Of these the second one (diffie-hellman-group14-sha1) is the older one. It is defined in RFC 4253. This RFC defines two key exchange algorithms: diffie-hellman-group14-sha1 and diffie-hellman-group1-sha1. (The group number defines the number of bits in the key. Group 1 has 768 bits, group 14 has 2048 bits. Larger is better). (Correction: Turns out diffie-hellman-group1-sha1 is actually Group 2).

The key exchange algorithms of RFC 4253 are updated via RFC 4419 – which defines two additional algorithms: diffie-hellman-group-exchange-sha1 and diffie-hellman-group-exchange-sha256. I am not very clear about these but it looks like they remove any weaknesses to do with groups that define the fixed number of bits and introduces the ability for the client and server to negotiate a custom group (of 1024 – 8192 bits). From RFC 4419:

Currently, SSH performs the initial key exchange using the “diffie-hellman-group1-sha1” method [RFC4253].  This method prescribes a fixed group on which all operations are performed.

The security of the Diffie-Hellman key exchange is based on the difficulty of solving the Discrete Logarithm Problem (DLP). Since we expect that the SSH protocol will be in use for many years in the future, we fear that extensive precomputation and more efficient algorithms to compute the discrete logarithm over a fixed group might pose a security threat to the SSH protocol.

The ability to propose new groups will reduce the incentive to use precomputation for more efficient calculation of the discrete logarithm.  The server can constantly compute new groups in the background.

So, to summarize: the common algorithms between KiTTY and OpenSSH are diffie-hellman-group-exchange-sha256 and diffie-hellman-group14-sha1. By default the preferred order between KiTTY client and OpenSSH server are in the order I specified above. In terms of security diffie-hellman-group-exchange-sha256 is preferred over diffie-hellman-group14-sha1. For some reason, however, KiTTY breaks with the newer version of OpenSSH when it comes to this stage.

Since diffie-hellman-group-exchange-sha256 is what both would be using I changed the preferred order on KiTTY such that  diffie-hellman-group14-sha1 is selected. 

kitty orderOnce I did that KiTTY connected successfully with OpenSSH – so that worked around the problem for now albeit by using a weaker algorithm – but at least it works. I wonder why diffie-hellman-group-exchange-sha256 breaks though.

I downloaded the latest version of PuTTY and gave that a shot with the default options (which are same as KiTTY). That worked fine! From the PuTTY logs I could see that at the point where KiTTY failed PuTTY manages to negotiate a key successfully:

Comparing the configuration settings of PuTTY and KiTTY I found the following:

putty fix

If I change the setting from “Auto” to “Yes”, PuTTY also hangs like KiTTY. So that narrows down the problem. RFC 4419 is the one I mentioned above, which introduced the new algorithms. Looks like that has been updated and KiTTY doesn’t support the new algorithms whereas PuTTY is able to.

PuTTY’s changelog didn’t have anything but its wishlist contained mention of what was going on. Apparently it isn’t a case of an update to RFC 4419 or any new algorithms, it is just a case of OpenSSH now strictly implementing the message format of RFC 4419 and thus breaking clients who do not implement this yet. To understand this have a look at this bit from the log entries of PuTTY and KiTTY at the point where KiTTY fails:

Pretty identical except for the type number. KiTTY uses 30 while PuTTY now uses 34. If you look at RFC 4419 these numbers are defined thus:

The following message numbers have been defined in this document.

They are in a name space private to this document and not assigned by IANA.

     #define SSH_MSG_KEX_DH_GEX_REQUEST_OLD  30
     #define SSH_MSG_KEX_DH_GEX_REQUEST      34
     #define SSH_MSG_KEX_DH_GEX_GROUP        31
     #define SSH_MSG_KEX_DH_GEX_INIT         32
     #define SSH_MSG_KEX_DH_GEX_REPLY        33

SSH_MSG_KEX_DH_GEX_REQUEST_OLD is used for backward compatibility. Instead of sending “min || n || max”, the client only sends “n”.  In addition, the hash is calculated using only “n” instead of “min || n || max”.

So RFC 4419 has updated the type number to 34 and set aside 30 for backward compatibility. Ideally clients should be sending type number 34 as their request. From the PuTTY wishlist link I saw the OpenSSH commit that removed the server from accepting type 30 any more. That’s why KiTTY was breaking, while PuTTY had updated their message type to the correct one and was successfully connecting! :)

(While digging around I saw that WinSCP too is (was?) affected.)

CentOS NAT

My home lab setup is such that everything runs in my laptop within VMware Workstation (currently version 11) on a Windows 8.1 OS. Hyper-V might be a better choice here for performance but I am quite happy with VMware Workstation and it does what I want. Specifically – VMware allows for nested virtualization, so I can install a hypervisor such as ESXi or Hyper-V within VMware Workstation and run VMs in that! Cool, isn’t it?

VMware Workstation also natively supports installing ESXi within it as a VM. I can’t do that if I were using Hyper-V instead.

Finally, VMware Workstation has some laptop lab level benefits in that it easily lets you configure network such as having a NAT network up and running quickly. I can do the same with Hyper-V but that requires a bit more configuring (expected, because Hyper-V is meant for a different purpose – I should really be comparing Hyper-V with ESXi but that’s a comparison I can’t even make here).

Within VMware Workstation I have a Windows environment setup where I play with AD, IPv6, DirectAccess, etc. This has a Corporate Network spread over two sites. I also have some CentOS VMs that act as routers – to provide routing between the two Corporate Network sites above, for instance – and also act as NAT routers for remote use. Yes, my Workstation network also has a fake Internet and some fake homes which is a mix of CentOS VMs acting as routers/ NAT and Server 2012 for DNS (root zones, Teredo, etc).

The CentOS VM that acts as a router between the Corporate Networks can also do real Internet routing. For this I have an interface that’s connected to the NAT network of VMware Workstation. I chose to NAT this interface because back when I created this lab I used to hook up the laptop to a network with static IP. I had only one IPv4 address to use so couldn’t afford to bridge this interface because I had no IPv4 address to assign it.

Because of the NAT interface the CentOS VM itself can access the real Internet. But what about other VMs that forward to this for their Internet routing? You would think that simply enabling packet forwarding on this VM is sufficient – but that won’t do. This is because packet forwarding for when forwarding between networks but in this case the external network does not know anything about my virtual networks that are behind a NAT (of VMware Workstation) so simply forwarding won’t work. So what you need to do apart from forwarding is also set up NAT on the CentOS VM so as far as the external network is concerned everything is coming from the CentOS VM (via its interface that is NAT’d with VMware Workstation).

I have done all this in the past but today I needed to revisit this after some time and forgot what exactly I had done. So this post is just a reminder to myself on what needs to be done.

First, enable packet forwarding in the OS. Add the following line to /etc/sysctl.conf:

Reboot the VM or run the following to load it now itself:

Now modify the firewall rules to allow packet forwarding as well as NAT (aka MASQUERADE in iptables):

Lines 1-6 and 16 are the relevant ones here. I have a few extra like allow ICMP and SSH but those don’t matter for what I am doing here.

[Aside] Improving your PuTTY connections

Wish I had come across this during my PuTTY days!

The TL;DR summary is that by default PuTTY uses an ASCII encoding while most Linux and other OSes uses UTF-8 encoding. It’s because of this mismatch that manpages and such sometimes appear with â characters. Change the PuTTY encoding and find happiness! 

[Aside] Systems, Science and FreeBSD

Interesting talk by George Neville-Neil on FreeBSD and research. At Microsoft Cambridge, of all the places! 

I am listening to two BSD podcasts nowadays – bsdtalk and BSD Now – and it’s been both great (coz I just love BSD!) and sad (coz I just love BSD and wish I were using them as actively as I was many years ago … sigh!). Came across the above talk via BSD Now, I listened to the audio version (you don’t really need the video version as it’s mostly talking and the slides are a separate download anyway and you can skip those). Earlier this week I listened to the bsdtalk interview with Mathew Dillon, founder of DragonFlyBSD. That was a great interview. I admire Matthew Dillon a lot and have listened to his previous interview (interviews?) on bsdtalk and elsewhere. Takes passion and vision and knowledge to stand by what you feel and fork an OS such as FreeBSD and actually get it going this far – I totally admire that! Sadly, DragonFlyBSD is the one I have least used – except briefly to check out HAMMER.

Anyways, thought I’d post links to these …

[Aside] What’s up with LibreSSL?

Good read: LibreSSL: More Than 30 Days Later

I loved this bit:

I could spend an hour explaining why supporting obsolete broken systems is detrimental, but if I told you all the things I have learned about VMS, it would probably violate your human rights. Instead, I think one example will suffice.

In theory, this looks like we’re going to use the good code on a posix system. But there’s just one problem. The code is testing for FIONBIO, which is only defined if you include the right header. If you forget to include the right header, the compiler falls back to the workaround. The mere existence of workarounds means they can be picked up accidentally, and you’ll never know.

[Aside] Shellshock – Stop blaming Bash!

Am sure everyone’s heard of the Bash Shellshock bug by now. Basically, Bash (like most other shells) has environment variables. When launching Bash you can set an environment variable too, and the newly launched Bash process will inherit that variable. So far so good, but what researchers discovered is that due to the bug you can also set environment variables such that you sneak in some commands into them; and the resulting Bash process will execute these commands for you. That’s wicked because you (the admin) might not want users to run such commands, or worse the Bash process might be running with additional rights in which case the sneaked in commands too will run with these additional rights.

There’s a lot of hue and cry over this and blame assigned to the GNU project and its creator Richard Stallman, but I liked this post by Andrew Auernheimer. The whole post is a must read but here’s the bit I would like to highlight (emphasis mine):

Shellshock is not a critical failure in bash. It is a critical failure in thousands of people who knew a tool so useful that they decided to deploy it far beyond its scope. A tool so resilient that it it did not fall over when everyone deployed against best practices. Everyone knew in the nineties that when you execute a UNIX command with untrusted input, you clear away the environment variables first. Anyone that has untrusted input embedded within a shell script does not know what they are doing. The fact that there is a way to get bash to execute untrusted code is unsurprising. The thing that surprises me is the sheer number of developers who thought it would be otherwise in complete contrast to UNIX parables and common sense.

Spot on!

Here’s a blog post on clearing the environmental variables. More techniques are in this StackOverflow post. (I haven’t made Bash scripts for a long time now, so I just got these via a quick DuckDuckGo search. There might be other/ better techniques around!)

ps. Since posting this I came across a post on the bad code state of Bash. It doesn’t take away the fact that Bash isn’t solely to be blamed for Shellshock but I thought to mention it because I found it an interesting read.

IPv6 through OpenVPN bridging

I spent the better part of today and yesterday trying to fix something that was broken because I was typing the wrong IPv4 address! Instead of connecting to (fake) address 36.51.55.1 I was typing 36.55.51.1. And since it wasn’t working I kept thinking it was a configuration error. Grr!

Here’s what I am trying to do:

  • I have a Linux server with a /64 IPv6 block assigned to it. The server has IPv6 address 2dcc:7c4e:3651:55::1/64, IPv4 address 36.51.55.1/24. (The server is a virtual machine on my laptop so these aren’t real public IPs).
  • I want a Windows client to connect to this server via OpenVPN, get/ assign itself a public IPv6 address from the /64 block, and have IPv6 connectivity to the Internet. Easy peasy?

I could use services like SixSx or Hurricane Electric to get a tunnel, but I am curious whether it will work over OpenVPN. The idea is to do it in a test lab now and replicate in the real world later (if I feel like it). It’s been a while since I used OpenVPN so this is a good chance to play with it too.

Note

I want a single client to connect to the server so I don’t need a fancy server setup. What I am trying to achieve is a peer-to-peer setup (aka point-to-point setup). Below I refer to the client as OpenVPN client, the server as OpenVPN server, but as far as OpenVPN is concerned it’s a peer-to-peer setup.

Bridging & Routing

OpenVPN has two ways of connecting machines. One is called routing, wherein the remote machine is assigned an IP address from a different network range and so all remote machines are as though they are on a separate network. This means the OpenVPN server machine acts as a router between this network and the network the server itself is connected to. Hence the name “routing”.

The other method is called bridging, and here the OpenVPN server connects the remote machines to the same network it is on. Whereas in routing the OpenVPN server behaves like a router, in bridging the OpenVPN server behaves like a switch. All remote machines have IP addresses from the same network as the OpenVPN server. They are all on the same LAN essentially.

The advantage of bridging is that it works at the layer 2 level (because the remote machine is plugged in to the real network) and so any protocol in the upper levels can be used over it. In contrast routing works at layer 3 (because the remote machine is plugged into a separate network) and so only layer 3 protocols supported by OpenVPN can be used. Previously it was only IPv4 but now IPv6 support too exists. I could use either method in my case, but I am going ahead with bridging because 1) I want the remote machines to behave as though they are connected to the main network, and 2) I just have a /64 block and I don’t want to muck about with subnetting to create separate networks (plus blocks smaller than /64 could have other issues). Maybe I will try routing later, but for now I am going to stick with bridging.

To achieve bridging and routing OpenVPN makes use of two virtual devices – TUN (tunnel adapter) and TAP (network tap). Of these only TAP does bridging, so that’s the route I’ll take.

Before moving on I’ll point to a good explanation of Bridging vs Routing from GRC. The OpenVPN FAQ has an entry on the advantages & disadvantages of each.

OpenVPN: Quick setup on the Server side

A good intro to setting up OpenVPN can be found in the INSTALL file of OpenVPN. It’s not a comprehensive document, so it’s not for everyone, but I found it useful to get started with OpenVPN. The steps I am following are from the “Example running a point-to-point VPN” section with some inputs from the manpage. A point-to-point setup is what I want to achieve.

OpenVPN has various switches (as can be seen in the manpage). You can invoke OpenVPN with these switches, or put them in a config file and invoke the config file. For a quick and dirty testing it’s best to run OpenVPN with the switches. Here’s how I launched the server:

In practice, you shouldn’t do this as it launches the tunnel with NO encryption. You even get a warning from OpenVPN:

A better approach is to generate a shared key first and copy it to the client(s) too. A shared key can be generated with the --secret switch:

And OpenVPN can be launched with this shared key thus:

Note

Remember to copy the shared key to the client. If a client connects without the shared key it will fail:

Notice I am using the --dev switch to specify that I want to use a TAP device.

TAP devices

TAP and TUN are virtual devices. TAP is a layer 2 virtual device, TUN is a layer 3 virtual device.

Each OpenVPN instance on the server creates a TAP device (if it’s configured to use TAP) – names could be tap0, tap1, and so on. It’s important to note that each OpenVPN instance on the server uses a single TAP device – you don’t create TAP devices for each client. Similarly OpenVPN on the client side creates a TAP device when it’s launched. After creating the TAP device, OpenVPN client contacts OpenVPN server and both setup a communication between their respective TAP devices. This “communication” is basically an encrypted UDP connection (by default; though you could have non-encrypted UDP or TCP connections too) from the client to the server. The client OS and applications think its TAP device is connected to the server TAP device and vice versa, while the OpenVPN programs on both ends do the actual moving around of the packets/ frames.

In my case, since I am doing a peer-to-peer connection, the TAP device created on the server will be used by a single client. If I want to add another client, I’ll have to launch a new OpenVPN instance – listening on a separate port, with a separate TAP device, preferably with a separate secret.

If multiple clients try to use the same TAP device on the server, both clients won’t work and the server generates errors like these:

This is because packets from both clients are appearing on the same TAP device and confusing the OpenVPN server.

Creating a bridge

When OpenVPN on the server creates a TAP device it is not automatically connected to the main network. That is something additional we have to do. On Linux this is done using the bridge-utils package. Since I am using CentOS, the following will install it:

A bridge is the (virtual) network switch I was talking about above. It’s a virtual device, to which you connect the machine’s network adapter as well as the TAP adapters you create above. Keep in mind a bridge is a layer 2 device. It doesn’t know about IP addresses or networks. All it knows are MAC addresses and that it has (virtual) network adapters plugged in to it.

The bridge can have any name, but let’s be boring and call it br0. I create a bridge thus:

Next I add the physical network card of my machine to the bridge. Think of it as me connecting the network card to the bridge.

Here’s a catch though. Your physical network adapter is already connected to a switch – the physical switch of your network! – so when you connect it to our virtual switch it is implicitly disconnected from the physical switch. There’s no warning given regarding this, just that you lose connectivity. As far as the machine is concerned your physical network adapter is now connected to the virtual switch (bridge), and this virtual switch is connected to the physical switch.

What this means is that all the network settings you had so far configured on the physical adapter must now be put on the bridge. Bridges are not meant to have IP addresses (remember they are layer 2 devices) but this is something we have to do here as that’s the only way to connect the bridge to the physical network.

To make this change explicit let’s zero IP the physical interface. It’s not really needed (as the interface automatically loses its settings) but I think it’s worth doing so there’s no confusion.

Next I assign the bridge interface the IP settings that were previously on the physical adapter:

Since I am on CentOS and I’d like this to be automatically done each time I start the server I must make changes to the /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-br0 files. I just copied over the ifcfg-eth0 file to ifcfg-br0 and renamed some of the parameters in the file (such as the interface name, MAC address, etc).

Here’s my /etc/sysconfig/network-scripts/ifcfg-eth0 file. It’s mostly the defaults except for the last two lines:

This Fedora page is a good source of information on setting up a bridge in CentOS/ RHEL/ Fedora. The PROMISC=yes line is added to set the physical adapter in promiscuous mode. This is required by OpenVPN. (By default network adapters only accept frames corresponding to their MAC address. Promiscuous mode tells the adapter to accept frames for any MAC address. Basically you are telling the adapter to be promiscuous!)

Next, here’s my /etc/sysconfig/network-scripts/ifcfg-br0 file:

The first three lines are what matter here. They give the device a name, set it as type Bridge, and tell the kernel it needn’t delay in passing frames through the bridge (by default there’s a 30s delay). The other lines are from the original eth0 configuration file, modified for br0.

Once these two files are in place, even after a reboot the bridge interface will be automatically created, assigned an IP address connecting it to the physical switch, and the physical adapter will be put in promiscuous mode and connected to this bridge.

Creating TAP devices

Back to TAP devices. Once OpenVPN creates a TAP device it must be added to the bridge previously created. This is what connects the TAP device to the main network.

Typically a shell script is used that adds the dynamically created TAP device to the bridge. For instance OpenVPN’s Ethernet Bridging page has a BASH script. Similarly the OpenVPN INSTALL document gives another BASH script (which I like more and for some reason made more sense to me).

CentOS installs an OpenVPN wrapper script in /etc/init.d/openvpn which lets you start and stop OpenVPN as a service. This wrapper script automatically starts an OpenVPN process for each config file ending with the .conf extension in the /etc/openvpn directory and if a /etc/openvpn/xxx.sh script exists for the xxx.conf file it executes this file before launching the OpenVPN process. Thus CentOS already has a place to put in the script to add TAP interfaces to the bridge.

(CentOS also looks for two files /etc/openvpn/openvpn-start and /etc/openvpn/openvpn-shutdown which are run when the wrapper script starts and shutdown – basically they are run before any of the configuration files or scripts are run. I don’t think I’ll use these two files for now but just mentioning here as a future reference to myself).

The shell script can behave in two ways – depending on how TAP devices are created. TAP devices can be created in two ways:

  1. They can be created on the fly by OpenVPN – in which case you’ll need to invoke the script via an up switch; the script will be passed a variable $dev referring to the TAP device name so it can be set in promiscuous mode and added to the bridge.

    To create TAP devices on the fly use the --dev tap switch – notice it doesn’t specify a specific TAP device such as tap0. OpenVPN will automatically create a tap0 (or whatever number is free) device.

  2. They can be pre-created as persistent TAP devices – these will stay until the machine is rebooted, and are preferred for the reason that you don’t need --up and --down switches to create them, and the devices stay on even if the connection resets. In this case the script can create TAP devices and configure and add them to the bridge – independent of OpenVPN.

    To use a specific TAP device use the --dev tap0 switch (where tap0 is the name of the device you created).

I will be using persistent TAP devices. These can be created using the --mktun switch:

Next I set them in promiscuous mode, bring up the device, zero the IP, and set it in promiscuous mode:

Lastly I add these to the bridge:

Assuming my configuration file is called /etc/openvpn/ipv6.conf I’ll create a file called /etc/openvpn/ipv6.sh and add the following:

Don’t forget to make the file executable:

Now whenever the OpenVPN service on CentOS starts, it will run the script above to create the TAP device, configure and add it to the bridge, then launch OpenVPN with a configuration file that will actually use this device. This configuration file will refer by name to the TAP device created in this script.

Firewall

The OpenVPN server listens on UDP port 1194 by default. So I added a rule to allow this port to iptables.

In CentOS I made the following change to /etc/sysconfig/iptables:

Then I reloaded the iptables service so it loads the new rule:

Forwarding

Next I enabled IPv4 & IPv6 forwarding. I added the following lines to /etc/sysctl.conf:

And got sysctl to read the file and load these settings:

IPv4 forwarding is not needed in my case, but I enabled it anyways. Also note that I am enabling IPv6 forwarding for all interfaces. That’s not necessary, the forwarding can be enabled for specific interfaces only.

In addition to enabling forwarding at the kernel level, iptables too must be configured to allow forwarding. The IPv6 firewall is called ip6tables and its configuration file is /etc/sysconfig/ip6tables. The syntax is similar to /etc/sysconfig/iptables and in CentOS forwarding is disabled by default:

I commented out the second line and reloaded the service.

Lastly, I added the following line to /etc/system-config/network:

This tells the initscripts bringing up the interfaces that IPv6 forwarding is to be enabled on all interfaces. Again, this can be specified on specific interfaces – in which you you’d put it in the /etc/system-config/network-scripts/ifcfg-xxx file for that interface – or enabled globally as above but disabled in the file of specific interfaces. For the curious, the initscripts-ipv6 homepage describes more options.

To summarize

To summarize what I’ve done so far:

  1. Create a bridge, add the physical adapter to the bridge, configure the bridge.
  2. Create persistent TAP devices – best to create a script like I did above so we can set it to run automatically later on.
  3. Create a secret key, copy the secret key to the client(s).
  4. Modify firewall, enable forwarding.
  5. Launch OpenVPN, telling it to use the secret key and TAP. No need to specify the TAP device name, OpenVPN will figure it out.

Mind you, this is a quick and dirty setup. Using shared keys won’t work scale, but they will do fine for me now. And rather than launch OpenVPN directly I must put it into a conf file with some logging and additional options … but that can be done later.

Troubleshooting

A very useful switch for troubleshooting purposes is --verb x where x is a number from 0 to 11. 0 means no output except fatal errors. 1 (the default) to 4 mean normal usage errors. 5 to be more verbose, and also output R and W characters to the console for each packet read and write, uppercase is used for TCP/UDP packets and lowercase is used for TUN/TAP packets. And 6-11 for debug range output.

I usually stick with --verb 3 but when troubleshooting I start with --verb 5.

Also, while UDP is a good protocol to use for VPN, during troubleshooting it’s worth enabling TCP to test connectivity. In my case, for instance, the client seemed to be connecting successfully – there were no error messages on either end at least – but I couldn’t test connectivity as I can’t simply telnet to port 1194 of the server to see if it’s reachable (since it was uses UDP). So I enabled TCP temporarily, got OpenVPN on the server to listen on TCP port 1194, then did a telnet to that port from the client … and realized I had been typing the wrong IPv4 address so far!

Here’s how you launch OpenVPN server with TCP just in case:

Be sure to open the firewall and use the corresponding switch --proto tcp-client on the client end too.

OpenVPN: Quick setup on the Client side

The server is running with the following command (remember to add --verb x if you want more verbosity):

I launch the client thus (again, remember to add --verb x if you want more verbosity):

The command is quite self-explanatory.

  1. The --remote switch specifies the address of the server.
  2. The --dev tap switch tells OpenVPN to use a TAP device. On Windows TAP devices are persistent by default so this will be a persistent TAP device.
  3. Port is 1194, the default, so I don’t need to specify explicitly. Similarly protocol is UDP, the default.
  4. The shared key is specified using the --secret switch.

And that’s it! Now my client’s connected to the server. It doesn’t have an IPv6 address yet because the server doesn’t have DHCPv6 or Router Advertisements turned on. I could turn these ON on the server, or assign the client a static IPv6 address and gateway. The address it gets will be a global IPv6 address from the /64 block so you should be able to connect to it from anywhere on the IPv6 Internet! (Of course, firewall rules on the client could still be blocking packets so be sure to check that if something doesn’t work).

Wrapping up

Here’s the final setup on the server and client. This is where I put the settings above into config files.

On the server side I created a file called /etc/openvpn/ipv6.conf:

I renamed the previously generated key to /etc/openvpn/ipv6.key.

Created /etc/openvpn/ipv6.sh:

Also created /etc/openvpn/openvpn-shutdown to remove the TAP devices when the OpenVPN service stops:

Apart from this I created /etc/sysconfig/network-scripts/ifcfg-br0 and /etc/sysconfig/network-scripts/ifcfg-eth0 as detailed earlier. And made changes to the firewall rules.

On the client side I created a file C:\Program Files\OpenVPN\config\ipv6.ovpn:

I copied the shared key to C:\Program Files\OpenVPN\config\ipv6.key. Note that it is important to escape the path name in the ovpn file on Windows.

I created two command files to be run when the client connects and disconnects. These are specified by the --up and --down switches.

Here’s ipv6-up.cmd:

And here’s ipv6-down.cmd:

I should add code to set the DNS servers too, will do that later.

FYI

There’s a reason why I use the short name format to specify the path for these files. Initially I was using the full name escaped but the client refused to launch with the following error:

I noticed that it was trying to run the ipv6-up.cmd file without escaping the space between “Program Files”. Must be an OpenVPN bug, so to work around that I used the short name path.

To connect to the server I launch OpenVPN thus:

OpenVPN for Windows also installs a GUI system tray icon (called OpenVPN GUI). If your config files end with the ovpn extension and are in the C:\Program Files\OpenVPN\config\ folder you can right click this tray icon and connect. Very handy!

Note

This post was updated to include the wrapping up section above. And some errors in my understanding of TAP devices were corrected.

This post was further updated to add the netsh files for the client and also to clarify that this is a peer-to-peer setup. It will only work with one client at a time. Multiple clients will fail unless you create separate OpenVPN instances (on different ports and TAP adapters) to service them.

ERROR: [ipv6_set_default_route]

The other day I was configuring IPv6 on a CentOS box and got the following error:

I had set the following in my /etc/sysconfig/network-scripts/ifcfg-eth0 file:

The error message was telling me what’s wrong, but who bothers reading error messages!

The problem was this: I have specified the default gateway as fe80::254 but that’s a link-local address. How does the system know which link to use this with? I’d either have to specify a scope with the gateway address, or clarify which network interface to use. That’s what the error message is trying to say.

So the solution is one of the below.

Either specify the scope id:

Or specify the device to use:

Another thing, if your network router is sending Router Advertisements you can skip specifying the gateway altogether and ask the interface to auto configure.

This is how you’d configure your network on a VPS. The VPS provider will have allocated to /64 block to you. Unlike IPv4 where the provider will take an IPv4 from the same network as you to use as the router for your network, with IPv6 they don’t need to do that as link-local addresses can be used. So they’ll either provide you with a link-local address, or you can turn on auto configure and pick it up yourself (provided it’s being advertised).

Lastly, I needn’t have put this configuration in the interface specific files. I could have put it in /etc/sysconfig/network too.

CentOS – initial configuration notes

Nothing new here, just stuff I can refer to later when I forget. I’ve usually worked with Debian but am not working with CentOS for a change so some things are new to me.

  1. If you install CentOS in a machine with less than 1024 MB RAM you don’t get the graphical installation option. So best to install with 1024 MB RAM and then downgrade RAM if you want to. I have a CentOS VM with less than 512 MB RAM. I don’t necessarily need a graphical installation but I prefer it as it’s friendly. I get to choose the hostname for instance, select the packages I want to install, and so on. Without a graphical install you are given a minimal install only with no options to choose.
  2. Set the hostname in /etc/sysconfig/networking.
  3. Network interfaces are managed by NetworkManager, which is useful if you have a GUI or like to use the nmcli command. I prefer old fashioned ifconfig so I manage my network devices via /etc/sysconfig/network-scripts/ifcfg-* files.
    • Set NM_CONTROLLED=no in this file to tell NetworkManager not to bother with this interface.
    • Set ONBOOT=yes to up this interface on boot up.
    • Use IPADDR=a.b.c.d, NETMASK=x.y.z.f, GATEWAY=p.q.r.s to set the IPv4 address, netmask, and gateway. Use BOOTPROTO=dhcp if you’d prefer DHCP instead.
    • Using IPV6INIT=no and IPV6_AUTOCONF=no is supposed to turn off IPv6 for that interface but it doesn’t seem to. A better way to turn off IPv6 (and/ or control its parameters) is via sysctl.
    • For more options this link from Oracle is a useful reference, as is the initscripts-ipv6 homepage.
  4. To disable IPv6 on an interface the following sysctl setting helps: net.ipv6.conf.eth0.disable_ipv6 = 1. Add entries like these to /etc/sysctl.conf.
  5. The following sysctl setting tells the OS to act as a router (i.e. forward packets): net.ipv4.ip_forward = 1
  6. Use /etc/resolv.conf to specify name servers, domain suffix (via the domain keyword), and list of search suffixes (via the search keyword with a space separated list of domains). Domain suffix is the domain name to be added to hosts without a domain name. Search suffixes are additional domain names that can be added to hosts when searching. Use only one of these. If both are specified the last one wins.
  7. Its best to put the Intranet DNS servers first in /etc/resolv.conf followed by Internet DNS servers. In my setup the Intranet servers don’t have access to the Internet and so any Internet name queries to them timeout. When this happens the DNS resolver will automatically try the next server in the list … up to when it reaches the Internet servers and get an answer. To speed the process up use option timeout:1 to set a timeout of 1 second. In contrast if I put the Internet servers first the don’t timeout – they try to resolve the non-existent domain and reply that it does not exist – so the Intranet servers aren’t queried.