Figure out how to handle secrets

[bgrant0607]

Forking the discussion from #2433 (comment)

For usage in applications, there's https://secrets-store-csi-driver.sigs.k8s.io/.

For cases where we do need to copy into Kubernetes Secrets, I'd recommend using a similar approach as GitOps, which is copying from the storage system, the secret manager in this case rather than git. In the GitOps layer that could be a special kind of sync request. In the kpt CLI we'll need to figure out how we want to stage secrets before the apply, and ensure we don't leak them.

There's also the issue of generation, for cases where users do want to pre-generate secrets and push them along with their other configuration.

[justinsb]

I think I agree with the approach here - I think we are recommending to not store the secret in git, even encrypted. Rather secrets should be stored in the cloud's secure secret manager or a service like vault. For TLS we could also recommend using certificate management functionality where that is separate.

For automatically generated secrets, I think we should think about a controller that is capable of generating and storing a secret, and rotating that secret according to some policy. That feels like an accurate description of what we want here from a policy perspective, and I believe we've had customers asking for similar things.

For manually managed secrets, I think this path would mean that we are proposing that users manually upload them. That feels pretty reasonable: we could put a tool or UI in front of the native APIs/UIs, but it's not clear that there's much value in doing so - nor that we necessarily have any particular insight which makes us the team to build that (yet?). Where we could maybe bring value is if we wanted to describe it using the same CRD as for automatically generated secrets, even though generation and rotation would be manual, and use this to remind people to upload the secret if it wasn't there, or to rotate the secret according to some policy.

[bgrant0607]

Yes, definitely as much as is practical we should recommend automatically rotated secrets: service accounts, workload identity, managed certs, and so on.

[bgrant0607]

https://dnastacio.medium.com/why-you-should-avoid-sealed-secrets-in-your-gitops-deployment-e50131d360dd
mentions https://external-secrets.io/
Video: https://www.youtube.com/watch?v=SyRZe5YVCVk
This is kind of like GitOps, but with a secret store as the storage system instead of git.

Looks like Jenkins-X uses external secrets:
https://jenkins-x.io/v3/devops/gitops/
https://jenkins-x.io/v3/admin/setup/secrets/

It does look like it updates secrets in place, however, rather than creating new ones.

[bgrant0607]

This controller includes a secret generator:
https://github.com/mittwald/kubernetes-secret-generator

[yuwenma]

An example discovered in Ghost Application package:

Two secrets are used. One for ghost-app and the other is for MariaDB.

The secretes are referred by ghost-app Deployment and mariadb StatefulSet on spec.template.spec.containers[].env[].valueFrom.secretKeyRef fields.

The secretes are also injected in these two resources' spec.template.metadata.annotations[].checksum/secrets. checksum/secrets is a helm feature for auto roll deployment. This ghost app is originally converted from a helm charts bitnami/ghost

[bgrant0607]

For completeness, kustomize supports secret generation plugins.
https://github.com/kubernetes-sigs/kustomize/blob/master/examples/secretGeneratorPlugin.md

Example plugin:
https://github.com/viaduct-ai/kustomize-sops

[bgrant0607]

External secrets was inducted into the CNCF sandbox:
https://lists.cncf.io/g/cncf-toc/message/7273

[bgrant0607]

We probably want a mechanism to help users not commit Secrets to git.