This post is the first of a series of articles about basic underlying principles that are useful to know for the types of devops-y and sys-admin-y type tasks freckles usually helps with. For each topic, I’ll write one article that tries to explain the general principle(s) involved, and a second one outlining how this is used in the context of freckles. Feel free to skip that later part if you don’t intend to use freckles.

If in doubt, I’ll trade ‘technically correct’-ness for ease of explanation. Treat those articles as a rough overview to get you started, and consider diving deeper using other resources once you feel like you have a basic understanding of what is going on.

This post (and the following ones) assume you have some understanding of how to use the command-line to do certain tasks (changing folders, move files, execute applications, …). I am pondering whether it’s worth to write a primer on command-line usage too, ping me if that is of interest to you.

So, what is this ssh you are talking about and why should I care?

As long as you do your work on only your own computer, you won’t have to worry about ssh at all. But once you want to host some software, be it Wordpress, Nextcloud, or any of the other awesome packages that require an application to run on a remote server constantly – so they can serve user requests via the web at any time – you need some way to connect to the machine that hosts that software package. To be able to install, update and generally maintain that service.

This is where ssh comes in. ssh is a sort of “Remote Desktop” for your command-line. It runs as a service on the remote machine, and, depending on how it is configured, lets you login with a username and password (or private key – more on that later). Once logged in, you can operate on the remote machine as you would do locally.

I won’t go into detail on how to setup and configure an ssh service itself, most cloud providers have ssh setup and running in a basic way on virtual machines (VMs) you create, and we can just use that for our purposes here.

Authentication

Authentication is the most important thing we need from ssh: it is supposed to only let only known users onto the remote machine. If you hear of security breaches in the media, often times attackers gain access to such servers via a mis-configured ssh service.

Username/password authentication

The most basic way to login via ssh is via a username/password combination. This is not the most secure way to use ssh, as usually a password is not long enough to be considered really secure. But in some cases it is necessary, most often for the initial configuration of a server, which, as last step, often involves actually disabling ssh password authentication.

Depending on the cloud provider you choose, you will have gotten a username and password combination along with the other connection details of your ssh server. For the purpose of this tutorial, I’ll use the Hetzner cloud to create virtual machines (mostly because I find their interface easy enough to use, and they are cheap, so it’s a good fit when you want to try things out). If you use any of the others (Linode, DigitalOcean, AWS, Google, Azure,…) you’ll encounter a similar interface and workflow, but it’ll differ enough that you’ll probably have to look up other tutorials on the web for some details.

So, let’s get started. Let’s sign up for an account and go the Hetzner cloud management interface. There we’ll create a new project (I’ll call it ‘Tutorial’) and click the “Add server” button.

The details you chose on the following screen are not really important, I’ll use a Debian image and the cheapest machine type (cx11), and give the machine the name ‘tutorial’. All other fields are either default or blank.

Sidenote: The price of €2.49 is per month, if the server only runs for part of a month, you’ll only have to pay the hourly rate (€0.004/hour).

Once we ‘buy’ our server, we wait until it is provisioned by Hetzner. This usually takes 10 to 20 seconds. Hetzner will send you an email with the connection details.

We need the IPv4 address, the username, and the password to be able to connect via ssh:

ssh <username>@<ipv4>

The IPv4 address is necessary for the underlying technologies to route our commands to the right server. In my case, the IP address I received was ‘116.203.23.175’, and the username ‘root’ (which is the main ‘admin’ user on Linux systems).

So, here’s what I do on my terminal:

$ ssh [email protected]
The authenticity of host '116.203.23.175 (116.203.23.175)' can't be established.
ECDSA key fingerprint is SHA256:RpD/BudrVmHy0mri4aFK493JSmIul0pRRrHJ2Rz9Log.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '116.203.23.175' (ECDSA) to the list of known hosts.
[email protected]'s password: xxx

The ‘authenticity’ message will be displayed on your first connection attempt, and can be used to verify that you are connecting to the server you intend to connect to (and there is no man-in-the-middle attack happening). How to do that is an advanced topic, and in practice not many people do that when creating new VMs like we just did.

Once I entered the password from the email I received, I’m logged in. The Hetzner VMs are configured in a way that when logging in the first time, the user has to set a new password for the ‘root’ user. Which is a good practice, because passwords sent via email can not be considered secure. Here’s how that looked for me:

You are required to change your password immediately (root enforced)
Linux tutorial 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun 22 12:18:36 2019 from 89.158.123.195
Changing password for root.
(current) UNIX password: xxx
Enter new UNIX password: yyy
Retype new UNIX password: yyy

And that’s basically all there is to it. Except, as I’ve mentioned, that using username/password authentication is not really the best/safest way to use ssh. So, let’s look at:

Private key authentication

You can think of a ‘private key’ (also called ssh key in this context) as a really long (and therefor really secure) password, one that you don’t want to type in every time (and you couldn’t really remember it anyway). That is why it is stored in a file on your local machine (usually at $HOME/.ssh/id_rsa – $HOME is a variable that points to the root folder for your user, usually the full path is: /home/<username>/.ssh/id_rsa).

For security reasons, most of the times that really long password is encrypted with another, shorter password (one that you actually can, and should, remember). This encryption protects against users on the same computer just copying the ‘password/private key-file’ and using it to impersonate you. But it is also an inconvenience, because it means that everytime you want to use it, you have to enter your (short) password, to decrypt the private key.

Now, how do we create the private key? There’s a tool for that on almost all Linux (and Mac OS X) machines: ssh-keygen. It supports different options, but until you know better, I’d recommend just using one of the recommended parameters that is compatible with most systems and which will result in a reasonable secure setup:

$ ssh-keygen -t rsa -b 4096

Generating public/private rsa key pair.
Enter file in which to save the key (/home/markus/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/markus/.ssh/id_rsa.
Your public key has been saved in /home/markus/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rzDEUqBmzKqYkhjullfnhFRXYdJeP4nP6Ml2zOlpS1c markus@t440p
The key's randomart image is:
+---[RSA 4096]----+
|    .    .o+.    |
| o . . . .o. .   |
|  *   o . . . o .|
| +   +     . . + |
|o   o + S     + E|
|=+   = o .   . o.|
|B.. . *   . o +.o|
|oo .   + .   =.*o|
|...     .   . ++.|
+----[SHA256]-----+

Sidenote: if you are ok with a very small risk of incompatibilites, it is recommended to use the ‘Ed25519’ algorithm instead of ‘rsa’ (-t ed25519 instead of -t rsa). Check the internet (like here) for more information. Using ‘rsa’ with a keysize of 4096 is a good enough choice though.

Once that is done, the ‘ssh-keygen’ command will have created two

• $HOME/.ssh/id_rsa
• $HOME/.ssh/id_rsa.pub

The first one is our private key, encrypted with the password we provided. This file is important, and should be kept secure. Never send your private key file over a network, or let anybody access it.

The second one is a file that contains a sort of unique ‘fingerprint’ of our private key. This file does not need to be kept secure. In fact, often you’ll be asked to provide the content of this file (e.g. on GitHub or similar services).

Why do I need this ‘fingerprint’? Good question, and if you understand the answer you understand the basic principle of why ssh private key authentication is done the way it is done.

I won’t go into detail about how exactly this works under the hood (not that I actually know…), but (thankfully) that is not necessary to use any of this. The main thing you need to know is that for the process of getting a sorta ‘fresh’ fingerprint you need the private key (you can’t just copy the fingerprint itself), and every such fingerprint is, for all practical purposes, unique amongst all of the fingerprints of all ssh keys in existence.

To be able to assert that without a doubt is very useful, because now, if you are a remote server with only knowledge of the fingerprint (which is usually called ‘public key’), you can ask the owner of the private key to give you a fingerprint (in a secure way), compare that with what you have in your records, and if there is a match you can be certain whoever asked you to let them in is in possession of the private key that belongs to the fingerprint.

The alternative would have been for you have the content of the private key in your records. And you compare that to the private key somebody who wants to login presents. This would have been bad for several reasons though, the worst being that the private key is now stored on the remote server, and anybody who can access that (via vulnerabilities, or maybe just because they are an admin user on that server) can now impersonate you on other services where you use the same key. Which would mean we would have to use a different private key for every server we want to connect to. Because we only use the fingerprint, and it is not possible to ‘fake’ the fingerprint-taking process without the private key (this is the crux, really), we can distribute our fingerprint to as many services and servers as we want, without having to worry about anybody we give it to will be able to impersonate us.

Now, let’s use our key!

First, let’s add our public ssh key to the VM we created earlier, manually. Then we’ll create a new VM, using ssh key authentication from the start (the recommended way).

To add our newly created key, we need to get the content of the public key (‘fingerprint’):

$ echo $HOME/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8TbDtJ4P3aF8eM0rAIrSrxea53p3qisN2jbvzW1rKGbaSJXKFCprVa05gFGUBeBZq2coHhIEOX3j+h+oNsWVWPdfY/mLt1a5uvQ+skY1j7Roc74xjNw0HzW71sTesodGXdXOACU4ins9FHUCkDdDW62sSARRTMaBksLvWMvzhR3dRiA3KgUTy8R2ObJITEqh4Ztbkfwzi+ntOOLJOJ+zediTsYlsT5ZPDlRcJamU6fV8XqinmuppEtBtuM8qd3cn42Oh+00AxyebgY0wyE9EJ9jr5S+cHICRoOxQGWagGBKhh8JAcFx3RwbSMY1+rk5lDIXavW+w6HA5WAntNZzLL [email protected]

To add that key to the ‘root’ user account of our VM, we need to login (using the password method), and then copy and paste our public key into the file $HOME/.ssh/authorized_keys on the remote machine. Alternatively we could use the ‘ssh-copy-id’ command, which will read thee public ssh key, login to the remote server, and copy the public-key-content into the appropriate place:

ssh-copy-id [email protected]

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected] password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh [email protected]'"
and check to make sure that only the key(s) you wanted were added.

After that we can try to login again, and this time around we should not be asked for the root password of the server, but for the password of our ssh key:

$ ssh [email protected]
[email protected]'s password: <root_password>
Linux tutorial 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun 22 14:17:19 2019 from 89.158.123.195

root@tutorial:~# echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8TbDtJ4P3aF8eM0rAIrSrxea53p3qisN2jbvzW1rKGbaSJXKFCprVa05gFGUBeBZq2coHhIEOX3j+h+oNsWVWPdfY/mLt1a5uvQ+skY1j7Roc74xjNw0HzW71sTesodGXdXOACU4ins9FHUCkDdDW62sSARRTMaBksLvWMvzhR3dRiA3KgUTy8R2ObJITEqh4Ztbkfwzi+ntOOLJOJ+zediTsYlsT5ZPDlRcJamU6fV8XqinmuppEtBtuM8qd3cn42Oh+00AxyebgY0wyE9EJ9jr5S+cHICRoOxQGWagGBKhh8JAcFx3RwbSMY1+rk5lDIXavW+w6HA5WAntNZzLL [email protected]' >> $HOME/.ssh/authorized_keys

root@tutorial:~# exit
logout
Connection to 116.203.23.175 closed.

vagrant@debian-testing:~$ ssh [email protected]
Enter passphrase for key '/home/vagrant/.ssh/id_rsa':
Linux tutorial 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64
...
...
...

The one thing to note here is that the authorized_keys file (always in $HOME/.ssh/authorized_keys) is the servers record for which private key ‘fingerprints’ to allow access for the particular user that tries to login. Every line in it denotes one public key (‘fingerprint’).

We can let Hetzner do the same thing for us, by registering our public ssh key with them. Go to the Hetzner cloud ‘Access’ page of our project (left side, key symbol), and click the ‘Add ssh key’ button.

Copy and paste your public key (the line that starts with ‘ssh-rsa….’) into the approriate text field, and give your key a name (‘tutorial-key’).

Now, the next time you create a machine, the puchase page lets you select from your registered ssh keys. Select the one you want to be included, and after the server is up you should be able to use it to login the same way we did when we added the key to ‘authorized_keys’ manually.

Password protected private keys, or not?

As I’ve mentioned above, usually we protect our private key with a password, but sometimes we don’t. When should we do what? Well, rule of thumb: use a password.

Once you start managing servers, it’s getting a bit more difficult though. The issues start once we do ‘automation’. ‘Automation’ is everything we want our computers to do automatically, without us having to do anything, except for an initial manual setup of the automation process (and even that we sometimes automate – but that is a different topic).

‘Automation’ means we should not have to intervene at all. One example would be a task that gets done every day at the same time (often via ‘cron’ jobs). Or a setup task you want to be kicked off every time you commit to a git repository.

If that task involves your private ssh key in some way (for example: build the package, login to a server via ssh, then install the package), you somehow have to ‘unlock’ your private key for every one of those tasks. You could, of course, just store the password to the private key on the same machine as your private key. But that would defeat the purpose, and would be pretty much as secure as just using no password on your private key at all. Because, if an attacker can read you (encrypted) private key, they also have access to read your password file…

There are ways to protect your private key in scenarios like this, but almost none of them are flawless, and none of them are straight-forward. You having to manually enter a password is really the best in terms of security. And the worst in terms of convenience. This is why there are certain situations where people decide to not secure the private key with a password, and accept the security implications that come with this decision.

There is not much advice I can give on a primer like this, I’d just recommend that whenever you get stuck trying to automate something, and passwords are involved, try to search the internet for people with a similar problems and technologies, and check whether there seems to be a sort of consensus (well, as close an approximation to ‘consensus’ there exists on the internet, anyway) ‘best-practice’ for your type of situation.

The ssh-agent

On your local machine, there is one remedy for the inconvenience of having to enter the password to your ssh key every time you want to use it: ssh-agent.

It comes with a minor impact on security, but my feeling is that most people accept that for the huge boost in convenience it brings.

ssh-agent is a little daemon (a program that runs in the background) which can store passwords for ssh keys in a secure way. That means, you have to enter the password once, the first time you use a ssh key in a session, and after that the ssh-agent will intercept any access requests for the ssh key in question, and will provide the decrypted key to whatever program requested it.

Most people (and Linux distributions) start ssh-agent automatically in some way when the user logs in. It is hard to give general advice on how to use it because of the different ways this is done. So I’ll only outline how it works in principle, below.

If you type ssh-agent in your terminal, you’ll see something like:

$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-eq76YPuBvNU8/agent.12962; export SSH_AUTH_SOCK;
SSH_AGENT_PID=12963; export SSH_AGENT_PID;
echo Agent pid 12963;

This does 2 things:

• start the ssh-daemon (type ps axu|grep ssh-agent to confirm)
• print out details that are needed to connect to the running ssh-agent daemon

If you start the agent several times, you’ll notice that the printed details are always slightly different. The reason for this is that you can have several ssh-agents processes running at the same time, and each of them can store different (or the same) ssh keys. In most cases we only want one of those agents running though.

Now, what are those SSH_AUTH_SOCK and SSH_AGENT_PID variables? Those need to be present in the environment of a process that wants to use the ssh-agent so the process knows how to connect to the agent. Reading the content of the SSH_AUTH_SOCK variable is how that is done. In our case, an application would read that value, and would use the /tmp/ssh-eq76YPuBvNU8/agent.12962 path (which is not a ‘normal’ file, but a unix socket) to ‘talk’ to our ssh-agent.

So, what the output of our ssh-agent command is telling us: ‘those are my connection details, please put them in your environment if you want to use me’ (it’can’t do it itself for … reasons). So, we can either do this now, manually (by copying and pasting the two relevant lines of the output):

$ SSH_AUTH_SOCK=/tmp/ssh-eq76YPuBvNU8/agent.12962; export SSH_AUTH_SOCK;
$ SSH_AGENT_PID=12963; export SSH_AGENT_PID;

Or, we use the eval command to do it for us (which is what the ssh-agent output is designed for), by wrapping the eval around our ssh-agent command:

$ eval $(ssh-agent)

Basically, this executes ssh-agent, then it executes the output of the ssh-agent execution. We can confirm this worked by checking our processes for the ‘ssh-agent’ process, and the contents of our SSH_AUTH_SOCK and SSH_AGENT_PID environment variables:

$ ps axu|grep ssh-agent
markus   24648  0.0  0.0   5164   336 ?        Ss   13:59   0:00 ssh-agent
markus   25117  0.0  0.0   6604   728 pts/3    S+   13:59   0:00
grep ssh-agent

$ echo $SSH_AUTH_SOCK
/tmp/ssh-i1u4UBNrqPQZ/agent.24647

$ echo $SSH_AGENT_PID
24648

Right. Now we have ‘ssh-agent’ running. What’s next?

We need to register the ssh keys we intend to use with it! In most cases, there exists only the default key: $HOME/.ssh/id_rsa

Because this is the default, we don’t need to provide a path to the key, we only need to call:

$ ssh-add
Enter passphrase for /home/markus/.ssh/id_rsa:
Identity added: /home/markus/.ssh/id_rsa (/home/markus/.ssh/id_rsa)

Once that is done, every ssh connection that uses that key will use the stored key in ssh-agent, and we’ll never have to use provide the ssh key password manually again.