From b911623d7ad8d1e3fddb97bb0a9287a8bc6d1463 Mon Sep 17 00:00:00 2001 From: Pim Kunis Date: Wed, 24 May 2023 21:13:37 +0200 Subject: [PATCH] add article about SSH CA --- README.md | 1 + jekyll/.tool-versions | 2 +- .../2023-05-23-homebrew-ssh-ca.md | 184 ++++++++++++++++++ 3 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 README.md create mode 100644 jekyll/_posts/homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..28b697d --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +`fly -t home set-pipeline --load-vars-from=secrets.yml -p static -c pipeline.yml` diff --git a/jekyll/.tool-versions b/jekyll/.tool-versions index c23af94..f2a971a 100644 --- a/jekyll/.tool-versions +++ b/jekyll/.tool-versions @@ -1 +1 @@ -ruby 3.2.0 +ruby 3.2.2 diff --git a/jekyll/_posts/homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca.md b/jekyll/_posts/homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca.md new file mode 100644 index 0000000..112bfcc --- /dev/null +++ b/jekyll/_posts/homebrew-ssh-ca/2023-05-23-homebrew-ssh-ca.md @@ -0,0 +1,184 @@ +--- +layout: post +title: Homebrew SSH Certificate Authority for the Terraform Libvirt Provider +date: 2023-05-23 11:14:00 Europe/Amsterdam +categories: ssh terraform ansible +--- + +Ever SSH'ed into a freshly installed server and gotten the following annoying message? +``` +The authenticity of host 'host.tld (1.2.3.4)' can't be established. +ED25519 key fingerprint is SHA256:eUXGdm1YdsMAS7vkdx6dOJdOGHdem5gQp4tadCfdLB8. +Are you sure you want to continue connecting (yes/no)? +``` + +Or even more annoying: +``` +@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ +@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ +@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ +IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! +Someone could be eavesdropping on you right now (man-in-the-middle attack)! +It is also possible that a host key has just been changed. +The fingerprint for the ED25519 key sent by the remote host is +SHA256:eUXGdm1YdsMAS7vkdx6dOJdOGHdem5gQp4tadCfdLB8. +Please contact your system administrator. +Add correct host key in /home/user/.ssh/known_hosts to get rid of this message. +Offending ED25519 key in /home/user/.ssh/known_hosts:3 + remove with: + ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "1.2.3.4" +ED25519 host key for 1.2.3.4 has changed and you have requested strict checking. +Host key verification failed. +``` + +Could it be that the programmers at OpenSSH simply like to annoy us with these confusing messages? +Maybe, but these warnings also serve as a way to notify users of a potential Man-in-the-Middle (MITM) attack. +I won't go into the details of this problem, but I refer you to [this excellent blog post](https://blog.g3rt.nl/ssh-host-key-validation-strict-yet-user-friendly.html). +Instead, I would like to talk about ways to solve these annoying warnings. + +One obvious solution is simply to add each host to your `known_hosts` file. +This works okay when managing a handful of servers, but becomes unbearable when managing many servers. +In my case, I wanted to quickly spin up virtual machines using Duncan Mac-Vicar's [Terraform Libvirt provider](https://registry.terraform.io/providers/dmacvicar/libvirt/latest/docs), without having to accept their host key before connecting. +The solution? Issuing SSH host certificates using an SSH certificate authority. + +## SSH Certificate Authorities vs. the Web + +The idea of an SSH certificate authority (CA) is quite easy to grasp, if you understand the web's Public Key Infrastructure (PKI). +Just like with the web, a trusted party can issue certificates that are offered when establishing a connection. +The idea is, just by trusting the trusted party, you trust every certificate they issue. +In the case of the web's PKI, this trusted party is bundled and trusted by [your browser](https://wiki.mozilla.org/CA) or operating system. +However, in the case of SSH, the trusted party is you! (Okay you can also trust your own web certificate authority) +With this great power, comes great responsibility which we will abuse heavily in this article. + +## SSH Certificate Authority for Terraform + +So, let's start with a plan. +I want to spawn virtual machines with Terraform which which are automatically provisioned with a SSH host certificate issued by my CA. +This CA will be another host on my private network, issuing certificates over SSH. + +### Fetching the SSH Host Certificate + +First we generate an SSH key pair in Terraform. +Below is the code for that: +```terraform +resource "tls_private_key" "debian" { + algorithm = "ED25519" +} + +data "tls_public_key" "debian" { + private_key_pem = tls_private_key.debian.private_key_pem +} +``` + +Now that we have an SSH key pair, we need to somehow make Terraform communicate this with the CA. +Lucky for us, there is a way for Terraform to execute an arbitrary command with the `external` data feature. +We call this script below: +```terraform +data "external" "cert" { + program = ["bash", "${path.module}/get_cert.sh"] + + query = { + pubkey = trimspace(data.tls_public_key.debian.public_key_openssh) + host = var.name + cahost = var.ca_host + cascript = var.ca_script + cakey = var.ca_key + } +} +``` + +These query parameters will end up in the script's stdin in JSON format. +We can then read these parameters, and send them to the CA over SSH. +The result must as well be in JSON format. +```bash +#!/bin/bash +set -euo pipefail +IFS=$'\n\t' + +# Read the query parameters +eval "$(jq -r '@sh "PUBKEY=\(.pubkey) HOST=\(.host) CAHOST=\(.cahost) CASCRIPT=\(.cascript) CAKEY=\(.cakey)"')" + +# Fetch certificate from the CA +# Warning: extremely ugly code that I am to lazy to fix +CERT=$(ssh -o ConnectTimeout=3 -o ConnectionAttempts=1 root@$CAHOST '"'"$CASCRIPT"'" host "'"$CAKEY"'" "'"$PUBKEY"'" "'"$HOST"'".dmz') + +jq -n --arg cert "$CERT" '{"cert":$cert}' +``` + +We see that a script is called on the remote host that issues the certificate. +This is just a simple wrapper around `ssh-keygen`, which you can see below. +```bash +#!/bin/bash +set -euo pipefail +IFS=$'\n\t' + +host() { + CAKEY="$2" + PUBKEY="$3" + HOST="$4" + + echo "$PUBKEY" > /root/ca/"$HOST".pub + ssh-keygen -h -s /root/ca/keys/"$CAKEY" -I "$HOST" -n "$HOST" /root/ca/"$HOST".pub + cat /root/ca/"$HOST"-cert.pub + rm /root/ca/"$HOST"*.pub +} + +"$1" "$@" +``` + +### Appeasing the Terraform Gods + +So nice, we can fetch the SSH host certificate from the CA. +We should just be able to use it right? +We can, but it brings a big annoyance with it: Terraform will fetch a new certificate every time it is run. +This is because the `external` feature of Terraform is a data source. +If we were to use this data source for a Terraform resource, it would need to be updated every time we run Terraform. +I have not been able to find a way to avoid fetching the certificate every time, except for writing my own resource provider which I'd rather not. +I have, however, found a way to hack around the issue. + +The idea is as follows: we can use Terraform's `ignore_changes` to, well, ignore any changes of a resource. +Unfortunately, we cannot use this for a `data` source, so we must create a glue `null_resource` that supports `ignore_changes`. +This is shown in the code snipppet below. +We use the `triggers` property simply to copy the certificate in; we don't use it for it's original purpose. + +```terraform +resource "null_resource" "cert" { + triggers = { + cert = data.external.cert.result["cert"] + } + + lifecycle { + ignore_changes = [ + triggers + ] + } +} +``` + +And voilà, we can now use `null_resource.cert.triggers["cert"]` as our certificate, that won't trigger replacements in Terraform. + +### Setting the Host Certificate with Cloud-Init + +Terraform's Libvirt provider has native support for Cloud-Init, which is very handy. +We can give the host certificate directly to Cloud-Init and place it on the virtual machine. +Inside the Cloud-Init configuration, we can set the `ssh_keys` property to do this: +```yml +ssh_keys: + ed25519_private: | + ${indent(4, private_key)} + ed25519_certificate: "${host_cert}" +``` + +I hardcoded this to ED25519 keys, because this is all I use. + +This works perfectly, and I never have to accept host certificates from virtual machines again. + +### Caveats + +A sharp eye might have noticed the lifecycle of these host certificates is severely lacking. +Namely, the deployed host certificates have no expiration date nore is there revocation function. +There are ways to implement these, but for my home lab I did not deem this necessary at this point. +In a more professional environment, I would suggest using [Hashicorp's Vault](https://www.vaultproject.io/). + +This project did teach me about the limits and flexibility of Terraform, so all in all a success! +All code can be found on the git repository [here](https://git.pim.kunis.nl/home/tf-modules/src/branch/master/debian).