perezdecastro.org

Moikka! Here's Adrián. I work for Igalia.

Locked out of SSH? Renew all the keys!

Long time, no post! Well, here I am with something completely unrelated to programming. Instead, is this something that has been bugging me for a while but I had been postponing because from the surface it looked a bit like a deep rabbit hole.

The Issue

For a while, every time I tried to connect to certain servers via SSH (or Mosh, make sure to check it out!), I was getting something like this:

~ % ssh sensei.perezdecastro.org
Host key fingerprint is ~~REDACTED~~
Permission denied (publickey).
~ %

Having plenty of things pending to do, plus lazyness, made me ignore this for a while because most of the machines I connect to via SSH have password-based authentication as a fallback. But today I had to use a machine that is configured to accept only key-based authentication, so I had to bite the bullet. As usual, I re-tried connecting with the very same command, plus adding -v to get some debugging output. Notice this message:

debug1: Skipping ssh-dss key /home/aperez/.ssh/id_dsa for not in PubkeyAcceptedKeyTypes

Could this be the root cause? Searching on Internet did not yield anything interesting — nobody checks result pages after the second one, ever. Then I remembered that after the infamous Heartbleed bug several initiatives with the goal of making secure stuff more secure —and hopefully bug-free— were started. While LibreSSL gets the honorable mention of having the cutest logo, other teams have not been behind in “de-crufting” their code. I started to fear that the latest release of OpenSSH may not have support anymore for one of:

  • The host key used by the server.
  • The host key type used by my laptop.
  • The identity key type.

It was time to learn about what changed in the recent past.

Exhibit “A”

Looking at OpenSSH 7.0 release notes, there is the culpript:

 * Support for ssh-dss, ssh-dss-cert-* host and user keys is disabled
   by default at run-time. These may be re-enabled using the
   instructions at http://www.openssh.com/legacy.html

So it turns out that support for ssh-dss keys like the one I was trying to use is still available, but disabled by default. The mentioned URL contains information on how to use legacy features, and in this case the support for ssh-dss keys can be re-enabled using the PubkeyAcceptedKeyTypes option, either temporarily for a single ssh (or scp) invocation:

~ % ssh -oPubkeyAcceptedKeyTypes=+ssh-dss sensei.perezdecastro.org

or permanently adding a snippet to the ~/.ssh/config file:

Host sensei.perezdecastro.org
     PubkeyAcceptedKeyTypes +ssh-dss

The Solution

I suspect it won’t be long before support for DSA keys is disabled at compile time, and for good reasons, so this seemed a moment as good as any to make a new SSH key, and propagate it to the hosts where I was using the old one.

Indeed, I should.

Question is: Which type of key should I generate as of 2015? The current default of ssh-keygen is to use RSA keys of 2048 bits, which seems like a reasonable thing provided that it is technically possible to break 1024 bit keys. Applying a bit of paranoia, I decided to better use a 4096 bit key, to make it future-proof as well.

But hey, we live in dangerous days, and one may want to stay away from RSA keys. They use the standard NIST curves, which are not that good, and because we know for a fact that the NSA has been tampering around with them, we may want to take a different approach, and go for an Ed25519 key instead. Which, apart from having Daniel J. Bernstein in the design team, are not vulnerable to poor random number generation. There is only one catch: support for Ed25519 is kind of new, so if you need to connect to machines using OpenSSH ≤ 6.5 you may still want to create a 4096 bit RSA key for them, ensuring that the Ed25519 key is used by default and the RSA one only when needed. More on that later, for now let’s create the new keys with:

~ % ssh-keygen -t ed25519

and:

~ % ssh-keygen -t rsa -b 4096

Then, append the key to the relevant machines, changing id_ed25519 to id_rsa for the ones which do not support Ed25519; note how the option to enable ssh-dss keys is used to temporarily allow using the old key to be able to append the new one (which somehow did not work with ssh-copy-id):

% ssh -oPubkeyAcceptedKeyTypes=+ssh-dss sensei.perezdecastro.org \
    'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_ed25519.pub

Last but not least, let’s make sure that the RSA key is only ever used when needed, with a couple of tweaks in the ~/.ssh/config file:

PubkeyAcceptedKeyTypes ssh-ed25519,ssh-ed25519-cert-v01@openssh.com

Host oldmachine.perezdecastro.org
     PubkeyAcceptedKeyTypes +ssh-rsa

Note that it is not possible to remove one item from the PubkeyAcceptedKeyTypes list with -ssh-rsa: we need to specify a complete list of key types that OpenSSH will be allowed to use. For reference: the ssh_config manual page lists the default value, and ssh -Q key lists key types built in a particular OpenSSH installation. In my case, I have decided to use my new Ed25519 key as primary, allowing only this key type by default, and using the RSA key for selected hosts.

While we are at it, it may be interesting to switch from the OpenSSH server to TinySSH in the servers under our control. Despite being small, it supports Ed25519, and it uses NaCl (DJB‘s crypto library) instead of OpenSSL. Also, it is possibly the simplest service one can setup over CurveCP at the moment. But this post is already long enough, so let’s move on to the finale.

Finale

My main desktop environment is GNOME, which is very nice and by default includes a convenient SSH agent as part of its Keyring daemon. But every now and then I just use OpenBox window manager, or even a plain Linux virtual terminal with a tmux session in it, where there is no GNOME components running. And I do miss its SSH agent. After looking a bit around, I have installed Envoy, which instead of implementing its own agent, ensures that one (and only one) instance of the OpenSSH ssh-agent runs for each user, across all logged-in sessions, and without needing changes to your shell startup files when using the included PAM module.

Update (2015-08-16): Another reason to use Envoy, as pointed out by Óscar Amor, is that the GNOME Keyring daemon can’t handle Ed25519 keys.

Happy SSH’ing!