Bring-your-own-certificate - controller does not fetch latest key imported

[sybernatus]

Which component:
The name (and version) of the affected component (controller or kubeseal)

• kubeseal: v0.27.1
• controller: v0.26.3

Describe the bug
We followed the documentation bring-your-own-certificate

While importing our own key on the controller under the test-key secret name, the new key is taken into account by the controller once we restarted it:

time=2024-11-20T13:52:11.023Z level=INFO msg="registered private key" secretname=test-key
time=2024-11-20T13:52:11.024Z level=INFO msg="registered private key" secretname=sealed-secrets-keymqbn8
time=2024-11-20T13:52:11.024Z level=INFO msg="registered private key" secretname=sealed-secrets-keyqm6bb

Even if this is the newest key (kubectl get secret):

sealed-secrets-keymqbn8   kubernetes.io/tls          2      10d
sealed-secrets-keyqm6bb   kubernetes.io/tls          2      10d
test-key                  kubernetes.io/tls          2      13m

The controller continue to retrieve an old public key using --fetch-cert argument. It seems to be randomly taken at each controller restart.
This is causing re-encrypting our secrets because there is no way to tell kubeseal to always use the latest key to re-encrypt the sealed secrets.

To Reproduce
Steps to reproduce the behaviour:

• Follow the procedure bring-your-own-certificate until Deleting the controller Pod is needed to pick the new keys
• launch kubeseal --fetch-cert and compare it to the public key in the newly created secret.

Expected behavior

• the controller should retrieve only the latest public certificate (as it is the most relevant certificate to encrypt secret)

Version of Kubernetes:

• Output of kubectl version:

v1.28.10

Additional context

Today our workaround is:

• to get public key from our secret deployed
• to get the name of a sealed secret on the cluster & the secret name inside
• to get the secret associated using the name
• to seal this secret using the public key

[alvneiayu]

hi @sybernatus

Following our procedure explained here, I can not reproduce the problem that you are saying 😕

Certificate generated by me:

MIIFQTCCAymgAwIBAgIUYNPMZZLuxX0D5p0Ug0j/FKJhmgYwDQYJKoZIhvcNAQEL
BQAwMDEWMBQGA1UEAwwNc2VhbGVkLXNlY3JldDEWMBQGA1UECgwNc2VhbGVkLXNl
Y3JldDAeFw0yNDExMjExMzM5NDRaFw0yNTExMjExMzM5NDRaMDAxFjAUBgNVBAMM
DXNlYWxlZC1zZWNyZXQxFjAUBgNVBAoMDXNlYWxlZC1zZWNyZXQwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQCrhSkB1SZKzpvvfn3gluRYCf6WQf041JCv
DzwU6fwGYiDWzMyqR0ybGVlQ/Sz2Y7JA8gNiF1n1Gt1XdJUwqKc/Y3Rwl3DBnr2X
gLzJV9vCjr41+ziIfd0alk7yDgbV7zj/DuKf45UsrpI1A2ZDs/ULUnCOIp4UPh6H
NZJyoGI66jPogWSn/pX6B4FmuQNsR6aOKLnQHDXipPYBp1lnBJe/q8KXzA3Tfj7t
HYlDXT8ssMLJtoJYaRDDM08z4wkTvhOymB6PKbzTqDQ+Inbwt1E0vLjSQE1+VbDj
gFsVISRdeqzcjMy9T5/GTrg/BAvfNPhacYHCJDAq5T2RaRN0ylon2esjYhQEdf4i
KgEDlIrdz7r9+pnN0EQZo/SNYqwDJ+hrOdkqEn8H4rFfj3Urs2ysd3jn9ADMSbO6
4Db6wG55SEftMEM59XCBTZAVxLNLimw+N5qphpXlgVx3yLM/oNMUsaE6BpMxywGM
Uu9ycwoXFIRoiIDAZgMlwVpF/JG5TF4oTz4HKGt5+FAxA3uz6EUcrPAhVHl8B40x
oaZe6jY/IPAaqBwNYeO/ysaiI+oBgCKXtVM47zEDN9qpZb1iuCp399Y+CYNEveMd
DIAXgONxtVKc2DyRpfc7seSsAeu9YCJOjK88yHQ02LgNa+CrPSEGO06xJplXUNxf
Z3JnY/m43wIDAQABo1MwUTAdBgNVHQ4EFgQUW8Okps0MRSls0ULN4goZOKgqSrYw
HwYDVR0jBBgwFoAUW8Okps0MRSls0ULN4goZOKgqSrYwDwYDVR0TAQH/BAUwAwEB
/zANBgkqhkiG9w0BAQsFAAOCAgEAiynnux7kAr4SSO+ETsCMk0XNxa2hXUDbReGY
SN4dNu8fAOhsZtE4ZzSOjXqA0NLrZoNMj9NtkvcdvElGtkDLMZngjMFH+xoX7zQ+
+KpDplH7ehunsmTgOn/5Vc80jbPZQy5iIszm0P6vGHkdXxkFdgxiKIBecHXonqxz
XGc+AiiRjv7Xp0iF4oC13lTtM2l59/ITxCAy+7My03kakyb/S+92zPfotadhI1cu
vN88BAcsuKpZGUsGwDsiAcIRic4OuzgvDanbAZH13sL60scV+tcHWvli9qegtwt0
uL6p5L2gI6zhjSEH4MjgOcvcAZzQ4QGQGPQAwiTFLE3l8Yf3Xm8k2esMfZ9FXtqn
ifzAprcdGibPW0tcLmkNtphTWI65sV/kQWcSa0h8TpZp80jQ1OYGPS3D3pR8GekV
HLLXSRmqHSmWZk9YrmCwvag4KQe6n2zjynZIu60AGG4WLYMYbAAFLoZNRJw3uwRR
YyVLFUPTqI3J3gUTc26zycfGnz/r2FBb5L0wZ9HCqPXj6mXYPDyFcgZLAGxdbuBK
D32kULzfPceMtOTFIJpwi94qw6bqb+iTtiTrZzAqdkOadM/EffFzuDLp9fFnr0XZ
czaCgPZD4U5Atr73w7+s8VM9uY3QowJRhOgXMzZHL3a6pBMU6xn3E54dd5QxmNIt
a8T9XUk=
-----END CERTIFICATE-----

$ kubeseal --fetch-cert
-----BEGIN CERTIFICATE-----
MIIFQTCCAymgAwIBAgIUYNPMZZLuxX0D5p0Ug0j/FKJhmgYwDQYJKoZIhvcNAQEL
BQAwMDEWMBQGA1UEAwwNc2VhbGVkLXNlY3JldDEWMBQGA1UECgwNc2VhbGVkLXNl
Y3JldDAeFw0yNDExMjExMzM5NDRaFw0yNTExMjExMzM5NDRaMDAxFjAUBgNVBAMM
DXNlYWxlZC1zZWNyZXQxFjAUBgNVBAoMDXNlYWxlZC1zZWNyZXQwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQCrhSkB1SZKzpvvfn3gluRYCf6WQf041JCv
DzwU6fwGYiDWzMyqR0ybGVlQ/Sz2Y7JA8gNiF1n1Gt1XdJUwqKc/Y3Rwl3DBnr2X
gLzJV9vCjr41+ziIfd0alk7yDgbV7zj/DuKf45UsrpI1A2ZDs/ULUnCOIp4UPh6H
NZJyoGI66jPogWSn/pX6B4FmuQNsR6aOKLnQHDXipPYBp1lnBJe/q8KXzA3Tfj7t
HYlDXT8ssMLJtoJYaRDDM08z4wkTvhOymB6PKbzTqDQ+Inbwt1E0vLjSQE1+VbDj
gFsVISRdeqzcjMy9T5/GTrg/BAvfNPhacYHCJDAq5T2RaRN0ylon2esjYhQEdf4i
KgEDlIrdz7r9+pnN0EQZo/SNYqwDJ+hrOdkqEn8H4rFfj3Urs2ysd3jn9ADMSbO6
4Db6wG55SEftMEM59XCBTZAVxLNLimw+N5qphpXlgVx3yLM/oNMUsaE6BpMxywGM
Uu9ycwoXFIRoiIDAZgMlwVpF/JG5TF4oTz4HKGt5+FAxA3uz6EUcrPAhVHl8B40x
oaZe6jY/IPAaqBwNYeO/ysaiI+oBgCKXtVM47zEDN9qpZb1iuCp399Y+CYNEveMd
DIAXgONxtVKc2DyRpfc7seSsAeu9YCJOjK88yHQ02LgNa+CrPSEGO06xJplXUNxf
Z3JnY/m43wIDAQABo1MwUTAdBgNVHQ4EFgQUW8Okps0MRSls0ULN4goZOKgqSrYw
HwYDVR0jBBgwFoAUW8Okps0MRSls0ULN4goZOKgqSrYwDwYDVR0TAQH/BAUwAwEB
/zANBgkqhkiG9w0BAQsFAAOCAgEAiynnux7kAr4SSO+ETsCMk0XNxa2hXUDbReGY
SN4dNu8fAOhsZtE4ZzSOjXqA0NLrZoNMj9NtkvcdvElGtkDLMZngjMFH+xoX7zQ+
+KpDplH7ehunsmTgOn/5Vc80jbPZQy5iIszm0P6vGHkdXxkFdgxiKIBecHXonqxz
XGc+AiiRjv7Xp0iF4oC13lTtM2l59/ITxCAy+7My03kakyb/S+92zPfotadhI1cu
vN88BAcsuKpZGUsGwDsiAcIRic4OuzgvDanbAZH13sL60scV+tcHWvli9qegtwt0
uL6p5L2gI6zhjSEH4MjgOcvcAZzQ4QGQGPQAwiTFLE3l8Yf3Xm8k2esMfZ9FXtqn
ifzAprcdGibPW0tcLmkNtphTWI65sV/kQWcSa0h8TpZp80jQ1OYGPS3D3pR8GekV
HLLXSRmqHSmWZk9YrmCwvag4KQe6n2zjynZIu60AGG4WLYMYbAAFLoZNRJw3uwRR
YyVLFUPTqI3J3gUTc26zycfGnz/r2FBb5L0wZ9HCqPXj6mXYPDyFcgZLAGxdbuBK
D32kULzfPceMtOTFIJpwi94qw6bqb+iTtiTrZzAqdkOadM/EffFzuDLp9fFnr0XZ
czaCgPZD4U5Atr73w7+s8VM9uY3QowJRhOgXMzZHL3a6pBMU6xn3E54dd5QxmNIt
a8T9XUk=
-----END CERTIFICATE-----

Could you verify if the Sealed Secrets controller has generated a new certificate after you apply your own certificate? I am asking that because the keyregistry is being sorted using the CreationTimestamp (here)
Could you share with me more logs, please? to help me to understand what's going on?

Thanks a lot

Álvaro

[sybernatus]

Thanks for your answer Alvaro.

For this cluster, we've already did redeploy sealed-secret without old keys and did re-encrypt the secrets manually as there was not so much secrets.

But we have to do the same next week for another cluster with much more secrets to re-encrypt so I will bring your more logs about this.

What I can bring you now is more info about the configuration changes we've made on sealed-secret during our test:

• Our sealed secret controller was installed as it was provided (with key renew feature).
• We've deployed our cert on sealed secret. But as the renew feature was activated, it has been replaced by a new and more recent certificate.

From here we start to change the configuration:

• First we redeploy sealed-secret disabling the key renew feature keeping all the keys already deployed to be able to re-encrypt sealedsecrets.
• Then we removed the secret with our certificate & re-deploy it. I'm almost sure we removed the CreationTimestamp so it is considered as the latest key to use.
• After restarting the controller, we tried to fetch again the certificate but it still retrieve an old certificate instead.

We also tried:

• to remove all the keys (old self-generated keys + our key) and redeploy the old one & the new one after them. Same issue but it returns a random old key (as they all had the same CreationTimestamp since we redeploy all the old keys at the same time).

---

I want to make sure about the CreationTimestamp so I will test on the 2nd cluster if this is still the case.
Also, I'm wondering if disabling the key renew feature could have an impact of the key used by the controller if it find multiple key registered.

I'll bring all of this asap.

Thanks again

[viktordaniel]

We are experiencing the same issue. The most recently imported certificate will not be provided while --fetch-cert.

spec:
  containers:
  - args:
    - --update-status
    - --key-renew-period
    - "0"
    - --key-prefix
    - sealed-secrets-key
    - --listen-addr
    - :8080
    - --listen-metrics-addr
    - :8081
    command:
    - controller
    image: docker.io/bitnami/sealed-secrets-controller:0.27.2

[sybernatus]

In fact, I've just reinstalled our sealed secrets controller & I encountered the issue again.
I have fully uninstall sealed secrets:

• the controller
• the crds
• the secret keys (I kept my own key backup locally to reapply it later)
• the namespace

After this, I have reinstall the CRDS & the controller using the installation documentation and this supplementary argument:

 - args:
    - --key-renew-period=0

Once done I reapplied my own key in the following secret sealed-secrets-key-2025.01.22.

After rolling out the controller, here is the status:

• keys deployed:

$ k get secret
NAME                            TYPE                DATA   AGE
sealed-secrets-key-2025.01.22   kubernetes.io/tls   2      37m
sealed-secrets-key9kknc         kubernetes.io/tls   2      37m

• secret timestamp k get secret ${secretName} -o jsonpath='{.metadata.creationTimestamp}'

sealed-secrets-key-2025.01.22 -> 2025-01-22T10:07:03Z
sealed-secrets-key9kknc -> 2025-01-22T10:07:01Z

• controller logs

time=2025-01-22T10:23:06.980Z level=INFO msg="Starting sealed-secrets controller" version=v0.26.3
time=2025-01-22T10:23:06.981Z level=INFO msg="Searching for existing private keys"
time=2025-01-22T10:23:06.996Z level=INFO msg="registered private key" secretname=sealed-secrets-key9kknc
time=2025-01-22T10:23:06.997Z level=INFO msg="registered private key" secretname=sealed-secrets-key-2025.01.22

We can conclude that both secrets are taken into account and the more recent is loaded at the end.

But even with this, when I launch kubeseal --fetch-cert on the my controller, it returns me the public certificate from the sealed-secrets-key9kknc.

---

I suspect that the fetch-cert sort is not correctly sorting the certificates when the difference between both timestamps is few seconds (might be related to the unix timestamp conversion maybe)

[sybernatus]

@alvneiayu
I think I have found the issue.

The issues seems to be at the registration of a new key.
If you check here in the readKey function

certUtil.ParseCertsPEM(secret.Data[v1.TLSCertKey])

This code is extracting the starting date of the public cert and this starting date is used to order the keys in sealed secret. This work if sealed secret generate its own key as the newly generated key is always the most recent.

In my case, this is not, because our own certificate is created before sealed secret is installed. So the secret created by sealed secrets at its installation is more recent than the certificate we want to publish.

Based on the documentation, the most recent certificate is defined by its creationTimestamp at kubernetes level, so the logic about ordering a certificate in the KeyRegistry should be the same.

EDIT:

• the readKey function is used in the initKeyRegistry here and use the cert.NotBefore to calculate the most recent certificate to use.

[alvneiayu]

@sybernatus question, with the PR sent by you. Can we close this issue or are you reproducing something else? Thanks a lot

[sybernatus]

@alvneiayu, it's fine now. We can close this issue 🙂 🙏