Contact

Subscribe via Email

Subscribe via RSS

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

KiTTY with Fedora 22/ OpenSSH 6.9 gives a blank screen and hangs

The KiTTY SSH client for Windows gives a blank screen and hangs when connecting to my Fedora 22 box (which runs OpenSSH 6.9). KiTTY is a fork of PuTTY, which I am happy with, so there’s no particular reason for me to use KiTTY except that I am checking it out.

kitty hungAfter a while it gives an error that the server closed the connection.

kitty fatalI have an Ubuntu Server 14.10 and CentOS 6.7 and KiTTY works fine with them. So it seems to be related to the version of OpenSSH in Fedora 22.

Fedora 22 logs show entries like these:

Looks like key-exchange was failing?

/*

Note to self: An alternative command syntax to the above is this:

I also noticed that including SELinux in the output gave a bit more info. Not that it helped but it’s worth mentioning here as a reference to myself. For that I include the sshd.service unit and also the SELinux context for SSHD.

Tip: After typing journalctl it is possible to keep pressing TAB to get the fields and their values. That’s how I discovered the SSHD context above.

On the topic of journalctl: this intro post by its creator and this DigitalOcean tutorial are worth a read.

Another useful tip with journalctrl: by default it doesn’t wrap the output but you can scroll left and right with the keyboard. This isn’t useful when you want to copy-paste the logs somewhere. The work-around is to use the --no-pager switch so everything’s dumped out together and then pipe it into less which wraps the text:

*/

I turned on SSH packet and raw data logging in KiTTY.

kitty logginFrom the log files I could see a sequence of events like this:

And with that it stops. So it looked like the DH Key Exchange was failing. Which confirms what I saw on the server side too.

SSH protocol 2 supports DSA, ECDSA, Ed25519 and RSA keys when clients establish a connection to the server. However, the first step is a Key Exchange for forward-secrecy and for this the DH algorithm is used.  From the sshd(8) manpage:

For protocol 2, forward security is provided through a Diffie-Hellman key agreement.  This key agreement results in a shared session key.  The rest of the session is encrypted using a symmetric cipher, currently 128-bit AES, Blowfish, 3DES, CAST128, Arcfour, 192-bit AES, or 256-bit AES.  The client selects the encryption algorithm to use from those offered by the server.  Additionally, session integrity is provided through a cryptographic message authentication code (hmac-md5, hmac-sha1, umac-64, umac-128, hmac-ripemd160, hmac-sha2-256 or hmac-sha2-512).

From the sshd_config(5) manpage:

The supported algorithms are:

curve25519-sha256@libssh.org
diffie-hellman-group1-sha1
diffie-hellman-group14-sha1
diffie-hellman-group-exchange-sha1
diffie-hellman-group-exchange-sha256
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521

The default is:

curve25519-sha256@libssh.org,
ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,
diffie-hellman-group-exchange-sha256,
diffie-hellman-group14-sha1

The KexAlgorithms option can be used to change this but the point is all these are variants of the DH algorithm.

KiTTY has the following algorithm selection options:

kitty options

For protocol 2 only the first three (Diffie-Hellman options) apply. The fourth one (RSA-based key exchange) only applies for protocol 1. Comparing this with the preferred order for OpenSSH only the first and second options are common:

  • Diffie-Hellman group exchange (a.k.a. diffie-hellman-group-exchange-sha256)
  • Diffie-Hellman group 14 (a.k.a. diffie-hellman-group14-sha1)

Of these the second one (diffie-hellman-group14-sha1) is the older one. It is defined in RFC 4253. This RFC defines two key exchange algorithms: diffie-hellman-group14-sha1 and diffie-hellman-group1-sha1. (The group number defines the number of bits in the key. Group 1 has 768 bits, group 14 has 2048 bits. Larger is better). (Correction: Turns out diffie-hellman-group1-sha1 is actually Group 2).

The key exchange algorithms of RFC 4253 are updated via RFC 4419 – which defines two additional algorithms: diffie-hellman-group-exchange-sha1 and diffie-hellman-group-exchange-sha256. I am not very clear about these but it looks like they remove any weaknesses to do with groups that define the fixed number of bits and introduces the ability for the client and server to negotiate a custom group (of 1024 – 8192 bits). From RFC 4419:

Currently, SSH performs the initial key exchange using the “diffie-hellman-group1-sha1” method [RFC4253].  This method prescribes a fixed group on which all operations are performed.

The security of the Diffie-Hellman key exchange is based on the difficulty of solving the Discrete Logarithm Problem (DLP). Since we expect that the SSH protocol will be in use for many years in the future, we fear that extensive precomputation and more efficient algorithms to compute the discrete logarithm over a fixed group might pose a security threat to the SSH protocol.

The ability to propose new groups will reduce the incentive to use precomputation for more efficient calculation of the discrete logarithm.  The server can constantly compute new groups in the background.

So, to summarize: the common algorithms between KiTTY and OpenSSH are diffie-hellman-group-exchange-sha256 and diffie-hellman-group14-sha1. By default the preferred order between KiTTY client and OpenSSH server are in the order I specified above. In terms of security diffie-hellman-group-exchange-sha256 is preferred over diffie-hellman-group14-sha1. For some reason, however, KiTTY breaks with the newer version of OpenSSH when it comes to this stage.

Since diffie-hellman-group-exchange-sha256 is what both would be using I changed the preferred order on KiTTY such that  diffie-hellman-group14-sha1 is selected. 

kitty orderOnce I did that KiTTY connected successfully with OpenSSH – so that worked around the problem for now albeit by using a weaker algorithm – but at least it works. I wonder why diffie-hellman-group-exchange-sha256 breaks though.

I downloaded the latest version of PuTTY and gave that a shot with the default options (which are same as KiTTY). That worked fine! From the PuTTY logs I could see that at the point where KiTTY failed PuTTY manages to negotiate a key successfully:

Comparing the configuration settings of PuTTY and KiTTY I found the following:

putty fix

If I change the setting from “Auto” to “Yes”, PuTTY also hangs like KiTTY. So that narrows down the problem. RFC 4419 is the one I mentioned above, which introduced the new algorithms. Looks like that has been updated and KiTTY doesn’t support the new algorithms whereas PuTTY is able to.

PuTTY’s changelog didn’t have anything but its wishlist contained mention of what was going on. Apparently it isn’t a case of an update to RFC 4419 or any new algorithms, it is just a case of OpenSSH now strictly implementing the message format of RFC 4419 and thus breaking clients who do not implement this yet. To understand this have a look at this bit from the log entries of PuTTY and KiTTY at the point where KiTTY fails:

Pretty identical except for the type number. KiTTY uses 30 while PuTTY now uses 34. If you look at RFC 4419 these numbers are defined thus:

The following message numbers have been defined in this document.

They are in a name space private to this document and not assigned by IANA.

     #define SSH_MSG_KEX_DH_GEX_REQUEST_OLD  30
     #define SSH_MSG_KEX_DH_GEX_REQUEST      34
     #define SSH_MSG_KEX_DH_GEX_GROUP        31
     #define SSH_MSG_KEX_DH_GEX_INIT         32
     #define SSH_MSG_KEX_DH_GEX_REPLY        33

SSH_MSG_KEX_DH_GEX_REQUEST_OLD is used for backward compatibility. Instead of sending “min || n || max”, the client only sends “n”.  In addition, the hash is calculated using only “n” instead of “min || n || max”.

So RFC 4419 has updated the type number to 34 and set aside 30 for backward compatibility. Ideally clients should be sending type number 34 as their request. From the PuTTY wishlist link I saw the OpenSSH commit that removed the server from accepting type 30 any more. That’s why KiTTY was breaking, while PuTTY had updated their message type to the correct one and was successfully connecting! :)

(While digging around I saw that WinSCP too is (was?) affected.)

Notes on using Git from behind a firewall, tunneling SSH through a proxy, Git credential helpers, and so on …

Thought I’d start using Git from my workplace too. Why not! Little did I realize it would take away a lot of my time today & yesterday …

For starters we have a proxy at work. Git can work with proxies, including those that require authentication. Issue the following command:

Or add the following to your global config file:

Apparently this only works with Basic authentication but since version 1.7.10 it works with NTLM authentication too (I didn’t test this). That’s no dice for me though as there’s no way I am going to put my password in plain text anywhere! So I need a way around that.

Enter Cntlm. Cntlm sits between the proxy and your software. Its runs on your machine and authenticates with the proxy/ proxies you specify. (The config file takes password hashes instead of the plain text password). Your programs talk to Cntlm without any authentication; Cntlm then talks to your proxy after proper authentication etc. Cntlm is quite portable. When installing it creates a service and all, but you can run it by launching the exe file and pointing it to the config file. Very convenient. I quickly created a shortcut with this and pinned to my taskbar so I can launch Cntlm whenever required (and it stays in the foreground, so closing the window will close Cntlm).

This way I can pull repositories over https. When I tried pushing though, I was stuck. I am asked for a username/ password and while I am entering the correct details it keeps rejecting me:

Turns out that’s because I use 2 factor authentication so I must create an access token. Did that and now I can push!

But that’s not very ideal as I don’t like storing the access token someplace. It’s a long piece of text and I’ll have to paste it in Notepad and copy paste into the command prompt each time I push.

Notice above that I am using HTTPS instead of SSH as is the norm. That’s because SSH won’t work through the proxy. If SSH works I could have used keys as usual. The good thing about keys is that I can passphrase protect it so there’s something known only to me that I can type each time. And if I ever need to revoke access to my office machine I only need revoke that key from GitHub.

SSH can’t be tunnelled through the HTTP proxy. Well not directly at least, through there is connect.c which is an SSH over HTTP proxy. Luckily it has Windows binaries too so we can try that.

But before that I want to see what I can do to ease access tokens an HTTPS access.

Turns out Git has ways to ease entering passwords or access tokens. Starting with version 1.7.9 it has the concept of credential helpers which can feed Git with your credentials. You can tell Git to use one of these credential helpers – maybe the GNOME Keyring – and Git will get your credentials from there. Older version of can use a .netrc file and you can do the same with Windows version of Git too. You can also cache the password in memory for a specified period of time via the cache credential helper.

Git 1.8.1 and above has in-built support for Windows (via the wincred helper). Prior versions can be setup with a similar helper downloaded from here. Both helpers store your credentials in the Windows Credential Store.

So it is not too much of a hassle using Git with HTTPS via an proxy like Cntlm and using the in-built credential helper to store your credentials in the Windows Credential Store. And if you ever want to revoke access (and are using two-factor authentication) you can simply revoke the access token. Neat!

Back to SSH. Here’s the Wiki page for connect.c, the SSH over HTTP proxy I mentioned above.

The Wiki page suggests a way to test SSH through connect.c. But first, here’s what the SSH link for one of my repositories look like:
git@github.com:rakheshster/BatchFiles
. The hostname is github.com, the username is git, and the path I am trying to access is rakheshter/BatchFile (i.e. username\repository). Let’s see if I can connect to that via connect.c:

The syntax is straight-forward. I launch connect.c, telling it to use the proxy at 127.0.0.1:53128 (where Cntlm is listening) and ask it to connect to github.com on port 22 (the SSH port). The -d switch tells connect.c to emit debugging info.

From the output I see it fails. Bummer! My work proxy is blocking traffic to port 22. Not surprising.

Thankfully GitHub allows traffic over port 443 (the HTTPS port). Must use hostname ssh.github.com though instead of github.com. Let’s test that:

Superb! It works.

Now all I need to do is open/ create %HOMEDRIVE%\.ssh\config and add the following lines:

All this does is that it tells SSH to launch connect.c to pass SSH connections through my proxy. Further, for any connections to github.com (on port 22), connect to ssh.github.com on port 443 instead.

Simple! Once I do these I can start pushing & pull Git over SSH too. Not bad eh.

While writing this post I discovered that connect.c is already a part of the Windows version of Git, so there’s no need to download it separately. Super!

OpenSSH and ECC keys (ECDSA)

Note to self: OpenSSH 5.7 and above support ECDSA authentication. As I mentioned previously, ECDSA is based on ECC keys. These are harder to crack and offer better performance as the key size is small. Here’s what I did to set up SSH keys for a new install of Git on Windows today:

The important bit is the -t ECDSA. The other two switches just specify a place to store the key as well as give it a description for my reference.