add article about SSH CA
This commit is contained in:
parent
e7eb78cf01
commit
b911623d7a
3 changed files with 186 additions and 1 deletions
1
README.md
Normal file
1
README.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
`fly -t home set-pipeline --load-vars-from=secrets.yml -p static -c pipeline.yml`
|
||||
|
|
@ -1 +1 @@
|
|||
ruby 3.2.0
|
||||
ruby 3.2.2
|
||||
|
|
|
|||
184
jekyll/_posts/homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca.md
Normal file
184
jekyll/_posts/homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
---
|
||||
layout: post
|
||||
title: Homebrew SSH Certificate Authority for the Terraform Libvirt Provider
|
||||
date: 2023-05-23 11:14:00 Europe/Amsterdam
|
||||
categories: ssh terraform ansible
|
||||
---
|
||||
|
||||
Ever SSH'ed into a freshly installed server and gotten the following annoying message?
|
||||
```
|
||||
The authenticity of host 'host.tld (1.2.3.4)' can't be established.
|
||||
ED25519 key fingerprint is SHA256:eUXGdm1YdsMAS7vkdx6dOJdOGHdem5gQp4tadCfdLB8.
|
||||
Are you sure you want to continue connecting (yes/no)?
|
||||
```
|
||||
|
||||
Or even more annoying:
|
||||
```
|
||||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||||
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
|
||||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||||
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
|
||||
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
|
||||
It is also possible that a host key has just been changed.
|
||||
The fingerprint for the ED25519 key sent by the remote host is
|
||||
SHA256:eUXGdm1YdsMAS7vkdx6dOJdOGHdem5gQp4tadCfdLB8.
|
||||
Please contact your system administrator.
|
||||
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
|
||||
Offending ED25519 key in /home/user/.ssh/known_hosts:3
|
||||
remove with:
|
||||
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "1.2.3.4"
|
||||
ED25519 host key for 1.2.3.4 has changed and you have requested strict checking.
|
||||
Host key verification failed.
|
||||
```
|
||||
|
||||
Could it be that the programmers at OpenSSH simply like to annoy us with these confusing messages?
|
||||
Maybe, but these warnings also serve as a way to notify users of a potential Man-in-the-Middle (MITM) attack.
|
||||
I won't go into the details of this problem, but I refer you to [this excellent blog post](https://blog.g3rt.nl/ssh-host-key-validation-strict-yet-user-friendly.html).
|
||||
Instead, I would like to talk about ways to solve these annoying warnings.
|
||||
|
||||
One obvious solution is simply to add each host to your `known_hosts` file.
|
||||
This works okay when managing a handful of servers, but becomes unbearable when managing many servers.
|
||||
In my case, I wanted to quickly spin up virtual machines using Duncan Mac-Vicar's [Terraform Libvirt provider](https://registry.terraform.io/providers/dmacvicar/libvirt/latest/docs), without having to accept their host key before connecting.
|
||||
The solution? Issuing SSH host certificates using an SSH certificate authority.
|
||||
|
||||
## SSH Certificate Authorities vs. the Web
|
||||
|
||||
The idea of an SSH certificate authority (CA) is quite easy to grasp, if you understand the web's Public Key Infrastructure (PKI).
|
||||
Just like with the web, a trusted party can issue certificates that are offered when establishing a connection.
|
||||
The idea is, just by trusting the trusted party, you trust every certificate they issue.
|
||||
In the case of the web's PKI, this trusted party is bundled and trusted by [your browser](https://wiki.mozilla.org/CA) or operating system.
|
||||
However, in the case of SSH, the trusted party is you! (Okay you can also trust your own web certificate authority)
|
||||
With this great power, comes great responsibility which we will abuse heavily in this article.
|
||||
|
||||
## SSH Certificate Authority for Terraform
|
||||
|
||||
So, let's start with a plan.
|
||||
I want to spawn virtual machines with Terraform which which are automatically provisioned with a SSH host certificate issued by my CA.
|
||||
This CA will be another host on my private network, issuing certificates over SSH.
|
||||
|
||||
### Fetching the SSH Host Certificate
|
||||
|
||||
First we generate an SSH key pair in Terraform.
|
||||
Below is the code for that:
|
||||
```terraform
|
||||
resource "tls_private_key" "debian" {
|
||||
algorithm = "ED25519"
|
||||
}
|
||||
|
||||
data "tls_public_key" "debian" {
|
||||
private_key_pem = tls_private_key.debian.private_key_pem
|
||||
}
|
||||
```
|
||||
|
||||
Now that we have an SSH key pair, we need to somehow make Terraform communicate this with the CA.
|
||||
Lucky for us, there is a way for Terraform to execute an arbitrary command with the `external` data feature.
|
||||
We call this script below:
|
||||
```terraform
|
||||
data "external" "cert" {
|
||||
program = ["bash", "${path.module}/get_cert.sh"]
|
||||
|
||||
query = {
|
||||
pubkey = trimspace(data.tls_public_key.debian.public_key_openssh)
|
||||
host = var.name
|
||||
cahost = var.ca_host
|
||||
cascript = var.ca_script
|
||||
cakey = var.ca_key
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
These query parameters will end up in the script's stdin in JSON format.
|
||||
We can then read these parameters, and send them to the CA over SSH.
|
||||
The result must as well be in JSON format.
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
IFS=$'\n\t'
|
||||
|
||||
# Read the query parameters
|
||||
eval "$(jq -r '@sh "PUBKEY=\(.pubkey) HOST=\(.host) CAHOST=\(.cahost) CASCRIPT=\(.cascript) CAKEY=\(.cakey)"')"
|
||||
|
||||
# Fetch certificate from the CA
|
||||
# Warning: extremely ugly code that I am to lazy to fix
|
||||
CERT=$(ssh -o ConnectTimeout=3 -o ConnectionAttempts=1 root@$CAHOST '"'"$CASCRIPT"'" host "'"$CAKEY"'" "'"$PUBKEY"'" "'"$HOST"'".dmz')
|
||||
|
||||
jq -n --arg cert "$CERT" '{"cert":$cert}'
|
||||
```
|
||||
|
||||
We see that a script is called on the remote host that issues the certificate.
|
||||
This is just a simple wrapper around `ssh-keygen`, which you can see below.
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
IFS=$'\n\t'
|
||||
|
||||
host() {
|
||||
CAKEY="$2"
|
||||
PUBKEY="$3"
|
||||
HOST="$4"
|
||||
|
||||
echo "$PUBKEY" > /root/ca/"$HOST".pub
|
||||
ssh-keygen -h -s /root/ca/keys/"$CAKEY" -I "$HOST" -n "$HOST" /root/ca/"$HOST".pub
|
||||
cat /root/ca/"$HOST"-cert.pub
|
||||
rm /root/ca/"$HOST"*.pub
|
||||
}
|
||||
|
||||
"$1" "$@"
|
||||
```
|
||||
|
||||
### Appeasing the Terraform Gods
|
||||
|
||||
So nice, we can fetch the SSH host certificate from the CA.
|
||||
We should just be able to use it right?
|
||||
We can, but it brings a big annoyance with it: Terraform will fetch a new certificate every time it is run.
|
||||
This is because the `external` feature of Terraform is a data source.
|
||||
If we were to use this data source for a Terraform resource, it would need to be updated every time we run Terraform.
|
||||
I have not been able to find a way to avoid fetching the certificate every time, except for writing my own resource provider which I'd rather not.
|
||||
I have, however, found a way to hack around the issue.
|
||||
|
||||
The idea is as follows: we can use Terraform's `ignore_changes` to, well, ignore any changes of a resource.
|
||||
Unfortunately, we cannot use this for a `data` source, so we must create a glue `null_resource` that supports `ignore_changes`.
|
||||
This is shown in the code snipppet below.
|
||||
We use the `triggers` property simply to copy the certificate in; we don't use it for it's original purpose.
|
||||
|
||||
```terraform
|
||||
resource "null_resource" "cert" {
|
||||
triggers = {
|
||||
cert = data.external.cert.result["cert"]
|
||||
}
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [
|
||||
triggers
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
And voilà, we can now use `null_resource.cert.triggers["cert"]` as our certificate, that won't trigger replacements in Terraform.
|
||||
|
||||
### Setting the Host Certificate with Cloud-Init
|
||||
|
||||
Terraform's Libvirt provider has native support for Cloud-Init, which is very handy.
|
||||
We can give the host certificate directly to Cloud-Init and place it on the virtual machine.
|
||||
Inside the Cloud-Init configuration, we can set the `ssh_keys` property to do this:
|
||||
```yml
|
||||
ssh_keys:
|
||||
ed25519_private: |
|
||||
${indent(4, private_key)}
|
||||
ed25519_certificate: "${host_cert}"
|
||||
```
|
||||
|
||||
I hardcoded this to ED25519 keys, because this is all I use.
|
||||
|
||||
This works perfectly, and I never have to accept host certificates from virtual machines again.
|
||||
|
||||
### Caveats
|
||||
|
||||
A sharp eye might have noticed the lifecycle of these host certificates is severely lacking.
|
||||
Namely, the deployed host certificates have no expiration date nore is there revocation function.
|
||||
There are ways to implement these, but for my home lab I did not deem this necessary at this point.
|
||||
In a more professional environment, I would suggest using [Hashicorp's Vault](https://www.vaultproject.io/).
|
||||
|
||||
This project did teach me about the limits and flexibility of Terraform, so all in all a success!
|
||||
All code can be found on the git repository [here](https://git.pim.kunis.nl/home/tf-modules/src/branch/master/debian).
|
||||
Reference in a new issue