haproxy deploy hook updates existing certificate over stats socket

[wlallemand]

Since version 2.2, HAProxy is able to update dynamically certificates, without a reload.

This patch uses socat to push the certificate into HAProxy in order to achieve hot update. With this method, reloading is not required. This should be used only to update an existing certificate in haproxy.

2 new variables are available:

• DEPLOY_HAPROXY_HOT_UPDATE="yes" update over the stats socket instead of reloading

• DEPLOY_HAPROXY_STATS_SOCKET="UNIX:/run/haproxy/admin.sock" set the path on the stats socket.

[wlallemand]

I just updated the PR, it now has the ability to add a new certificate in HAProxy without reloading as well as updating a previous one.

[msimerson]

Hey @wlallemand, this is AWESOME, thank you. I was getting ready to write this functionality and decided I should check if it already exists. I ran into a few issues I had to address to get it working for me.

1. my fullchain.pem files have an extra \n in them. Apparently the chain cert is coming back that way from Let's Encrypt. Because <<\n$(cat $path)\n (as shown in haproxy docs) depends on \n as a boundary, the payload fails to send to haproxy correctly. The solution is simple: replace cat with grep . $path
2. I communicate with haproxy admin API via TCP port instead of sock. This is easy to accommodate by moving the UNIX: into _statssock, so it can be set to tcp-connect:192.168.0.1:9000. Probably best to include an example of each as comments.
3. my haproxy is chroot, so the acme.sh deploy path and the haproxy path to same file are different
   • I store the haproxy installed path in _ipem.

[wlallemand]

@msimerson Thanks for your feedback, I'm going to fix the script to integrate properly your use case.

1. my fullchain.pem files have an extra `\n` in them. Apparently the chain cert is coming back that way from Let's Encrypt. Because `<<\n$(cat $path)\n` (as shown in [haproxy docs](https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/)) depends on `\n` as a boundary, the payload fails to send to haproxy correctly. The solution is simple: replace cat with `grep . $path`

Indeed. I did not had any problem with the provider I used but that's a valid point, it could contains empty lines or comments between the certificate sections.

2. I communicate with haproxy admin API via TCP port instead of sock. This is easy to accommodate by moving the UNIX: into `_statssock`, so it can be set to `tcp-connect:192.168.0.1:9000`. Probably best to include an example of each as comments.

I hesitate to include directly the socat format into the variable, maybe I could add a variable to be able to chose any socat prefix.

3. my haproxy is chroot, so the acme.sh deploy path and the haproxy path to same file are different
   
   * I store the haproxy installed path in _ipem.

Indeed that could be a problem, I will retest all of this.

Also, I published a documentation on the haproxy wiki: https://github.com/haproxy/wiki/wiki/Letsencrypt-integration-with-HAProxy-and-acme.sh

[msimerson]

I hesitate to include directly the socat format into the variable, maybe I could add a variable to be able to chose any socat prefix.

I'm curious how you think that having 2 socat variables is going to be easier for folks to grok and easier to support than having just one. It's not like the variable contents will ever be used separately. URIs evolved for a reason. :-)

[wlallemand]

@msimerson problem is socat and haproxy doesn't have the same "URIs". I don't want the user to need to specify a socat format. Maybe I'll just check if the variable contains any / to select UNIX-CONNECT or TCP-CONNECT.

[msimerson]

@msimerson problem is socat and haproxy doesn't have the same "URIs". I don't want the user to need to specify a socat format. Maybe I'll just check if the variable contains any / to select UNIX-CONNECT or TCP-CONNECT.

I see. In looking again, you use _statssock (3) times without the UNIX: prefix and (3) times with. If you were consistent and used it only without the prefix, then I think you get exactly what you want (user doesn't need to specify), and it'll work as you expect, because socat will assume the right thing:

Address specifications starting with a number are assumed to be FD (raw file descriptor) addresses; if a ’/’ is found before the first ’:’ or ’,’, GOPEN (generic file open) is assumed.

And anyone needing a different socat formula can specify it.

[wlallemand]

I see. In looking again, you use _statssock (3) times without the UNIX: prefix and (3) times with. If you were consistent and used it only without the prefix, then I think you get exactly what you want (user doesn't need to specify), and it'll work as you expect, because socat will assume the right thing:

Indeed, my mistake, I should also fix this. My first implementation was a little bit different and I probably forgot it.

Address specifications starting with a number are assumed to be FD (raw file descriptor) addresses; if a ’/’ is found before the first ’:’ or ’,’, GOPEN (generic file open) is assumed.

I want to avoid this and find it dangerous, if for some reason the path is wrong and is not a UNIX socket, it would do a GOPEN and the certificate could be dumped on the filesystem. That's exactly why I set UNIX: in the first place.

And anyone needing a different socat formula can specify it.

That's why I suggested another variable, but to be honest there is not a lot of common setup. It's basically TCP with ipv4 and ipv6 and unix socket, some people could also want to use OPENSSL... I think I'll remove the prefix and adjust the documentation as you suggested in the first place, with a warning about not using a prefix.

[msimerson]

it would do a GOPEN and the certificate could be dumped on the filesystem

In this case, the certificate is already on the filesystem, right? If you're super worried about it, you can always do a -S test on the path (after determining it's a path, by checking that the first character is a /).

[iarenaza]

3. my haproxy is chroot, so the acme.sh deploy path and the haproxy path to same file are different
   
   * I store the haproxy installed path in _ipem.

Just wanted to chime in to say that we are not using chroot for haproxy, but running it inside a container. And pretty much the same situation arises there: paths outside the container (where acme.sh stores the certs) and inside the container (where haproxy looks for them) don't match.

Is this something that could be supported?

[wlallemand]

Thanks for the feedbacks, I'll rework the PR next week to better handle these cases.

[dexter-dopping-ekco]

Hi folks,

To get the script to work right on my setup (Debian 12, HaProxy 2.6.12-1) I had to make a small tweak.

It seems like the stats socket doesn't read the entire input when passing in the file contents, instead stopping after reading an empty line, which means it doesn't read the CA certificate chain. This is what it looks like when running the deploy command with --debug 2:

...
-----END CERTIFICATE-----\n' | socat 'UNIX:/var/run/haproxy/admin.sock' - | grep -q 'Transaction created''
2023/11/24 20:11:20 socat[118250] E write(1, 0x5588acdfe000, 8192): Broken pipe
[Fri 24 Nov 2023 20:11:20 GMT] _socat_cert_commit_cmd='echo 'commit ssl cert /etc/haproxy/certs/example.com.pem' | socat 'UNIX:/var/run/haproxy/admin.sock' - | grep -q '^Success!$''
[Fri 24 Nov 2023 20:11:20 GMT] Success

The deploy script logs it as successful, but socat wasn't able to write everything (Broken pipe). openssl s_client shows that the certificate chain is incomplete:

dexter@dexter-dell:~$ openssl s_client -connect example.com:443 -servername example.com </dev/null
CONNECTED(00000003)
depth=0 CN = example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = example.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = example.com
verify return:1
---
Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R3
--- ...

#4788 mentions the same "Broken pipe" message but the related MR #4841 did not help and even seems to have made things worse, since the key no longers get imported when it's moved to the end of the PEM file.

Only after removing the empty lines does command with hot update work correctly for me.

This is the change I made:

diff --git a/haproxy-wlallemand.sh b/usr/local/share/acme.sh/deploy/haproxy.sh
index a611ec8..bb2b6b1 100644
--- a/haproxy-wlallemand.sh
+++ b/usr/local/share/acme.sh/deploy/haproxy.sh
@@ -177,7 +177,7 @@ haproxy_deploy() {
   # Create a temporary PEM file
   _temppem="$(_mktemp)"
   _debug _temppem "${_temppem}"
-  cat "${_ckey}" "${_ccert}" "${_cca}" >"${_temppem}"
+  cat "${_ckey}" "${_ccert}" "${_cca}" | grep . >"${_temppem}"
   _ret="$?"

   # Check that we could create the temporary file

Now the deploy command runs fine and the openssl s_client command shows the complete chain:

dexter@dexter-dell:~$ openssl s_client -connect example.com:443 -servername example.com </dev/null
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = example.com
verify return:1
---
Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
--- ...

[wlallemand]

Small updates:

• sanitize the PEM to remove the '\n'
• shellcheck fixes
• socat format is directly used in the DEPLOY_HAPROXY_STATS_SOCKET variable

@Neilpang Any chance this can be merged? This is actually asked by a lot of people.

[wlallemand]

I'll make more updates for chroot and wilcard as well as the master CLI instead of the stats socket, but probably in a separate PR.

[wlallemand]

update:

• master CLI support was implemented
• wilcard character is replaced in the filename

[dingensundso]

As mentioned in #4581 (comment) I'm not sure about replacing the asterisk with an underscore, since some DNS servers allow underscore as a subdomain even though RFC1035 doesn't allow it.
Let's Encrypt for one doesn't issue certificates for underscore domains but I don't know about other acme server implementations.

Would love to see input from other people about this.

Otherwise I don't see any problems with the script.

[krezovic]

Hi @wlallemand. I just stumbled upon this trying to configure OCSP Stapling in haproxy. Is it possible to use Le_OCSP_Staple info already present in haproxy.sh and add commands from https://www.haproxy.com/documentation/haproxy-runtime-api/reference/set-ssl-ocsp-response/ to the update commands if ocsp is present? Or even better, integrate https://www.haproxy.com/documentation/haproxy-configuration-tutorials/ssl-tls/#enable-ocsp-stapling somehow? Thanks in advance!

[wlallemand]

Hello @krezovic,

1. The .ocsp could be used directly with "set ssl cert basecert.pem.ocsp" before committing. I'll look into this.
2. Regarding the ocsp-update it's not possible with this current design, because the option is only working with a "crt-list" as a file, and the current config uses a directory. However we will make some changes into HAProxy so this could be done as a global option.

[krezovic]

Hi @wlallemand! Looking forward to proper support. Thanks for your reply!

[wlallemand]

@dingensundso Any chance we could have this merged? This is stuck for a while now.

[Neilpang]

please also update the usage here:
https://github.com/acmesh-official/acme.sh/wiki/deployhooks#10-deploy-the-cert-to-haproxy

[wlallemand]

Thanks, updated!

[ser]

Very nice job, it would be cool to have it available - maybe it's a good time for a new release?