Create a new git repo for prometheus-community helm charts

[scottrigby]

Hello 👋

Problem

Helm stable and incubator repos are EOL on Nov 13, 2020 (See support plan and Deprecation Timeline). The community (chart OWNERS, organizations, groups or individuals who want to host charts) are moving charts to new Helm repos, and will list these new repos on the Helm Hub before stable and incubator are de-listed there.

Proposed solution

• Please create a new git repo for prometheus-community helm charts: https://github.com/prometheus-community/helm-charts
• Invite collaborators to the new prometheus-community/helm-charts git repo with to help move over prometheus-related community charts from the stable helm repo (to properly retain git history), and set up CI/CD for automated chart testing and releasing. You can invite me, and I'll invite the others

From there, these remaining checklist items can move to an issue in the new repo. I'm listing them here so everyone can see the end-to-end plan:

• Invite existing prometheus community chart maintainers as collaborators with write access
• List new chart repo at https://hub.helm.sh/ and https://artifacthub.io/
• Deprecate the moved charts in the stable repo
• Work with prometheus developers and community to properly refactor charts within the new prometheus-community/helm-charts repo to refactor, and deprecate/rename as needed according to ongoing discussion about more appropriate architecture/naming

Additional context

Open question about prometheus-operator chart

Update: it was decided that the prometheus-operator chart should move to the prometheus-community org, per prometheus-operator/prometheus-operator#3169 (comment)

@paulfantom pointed me to the open issue at prometheus-operator/prometheus-operator#3169 and I have commented there.

Relevant notes from IRC

From chat in Freenode #prometheus-dev IRC today

I'm part of the Helm team, focusing largely on charts, repos and related tooling. Just caught up on the "Helm charts home" mailing list topic https://groups.google.com/g/prometheus-developers/c/Gh8GMscELao/m/2zZFPwdLBwAJ?pli=1 and would like to help however I can to get questions resolved for the new home(s) for stable prometheus charts before the current repo deprecation.

@brancz and I had an earlier Slack thread with Naseem in February/March about this. I think the mailing thread covered most of the topics we discussed.

In general, an idiomatic approach for an org hosting multiple Helm charts is a monorepo. If on GitHub, Helm has created Actions for consistent chart testing during CI and chart packaging releasing for CD (listed in the mail list thread). For history sake, we normally recommend preserving git history (using filter-branch) when moving charts, so that it's clear to future devs why past changes were made, and to credit those who made them.

That thread bike-shedded a bit on details like naming (really just keep it simple, no prefixes or suffixes needed etc). But there were two other points that I think are legit potential blockers to discuss:

1. how the prom charts should be ideally structured vs how the stable ones people depend on now are structured
2. whether there should be several categories of charts, clearly separated somehow for end users (example: a. charts owned by prometheus to be better architected and named, b. charts maintained by the community moved from stable as-is and refactored over time, c. charts focused on external projects like the prom jira-alerts. Or maybe b and c should be combined etc)

My suggestion to keep things as simple as possible is to take a multi-stage approach:

• First ask a Prometheus GitHub admin to create a new git repo https://github.com/prometheus-community/helm-charts. And invite a few collaborators as repo admins to help set it up (I'm happy to help), as well as write access for current chart maintainers (we can follow up about github usernames)
• Then we can bring in the existing charts as-is, with a goal of refactoring, and deprecating/renaming as needed from there according to ongoing discussion about more appropriate architecture/naming

☝️ This would still allow specific refactored charts to later be moved to https://github.com/prometheus/helm-charts if prometheus developers prefer an easier to distinguish charts maintained internally vs those maintained by the community. But they could also remain in prometheus-community if this isn't desired. My main goal is to unblock the 17 stable community prometheus-related Helm charts currently in use by end users today. Here is an issue we're using to track progress on moving all existing stable charts https://github.com/helm/charts/issues/21103

@SuperQ said:

The first step for that is to open an issue on https://github.com/prometheus-community/community/issues

@brian-brazil said:

I don't think it should ever be in the main prometheus org

Personally I agree. Some people in the mailing list suggested that so I just want to clearly note that the community repo wouldn't preclude individual charts moving elsewhere later, for any technical reasons related to helm repos, hub aggregation etc.

[SuperQ]

Anyone volunteer to be the maintainer sponsor? @bwplotka @brancz @gouthamve ?

[hickeyma]

Just to add some feedback that we have received in the Helm community in the last few months. There has been a lot of questions wrt prometheus Helm charts and if they will be supported by the prometheus community going forward. The other question been asked is, when will the charts be Helm 3 compatible.

[stuart-c]

@hickeyma what do you mean by Helm 3 compatible? I've been using Helm 3 with the charts for 6 months so far...

[hickeyma]

IIRC, someone mentioned that it was not compatible at some stage.

Ah, my bad. Checking the operator chart, I can see that it has indeed been updated for Helm 3 CRDs. Thanks for the heads up @stuart-c

[brancz]

I'm happy with sponsoring the chart for the prometheus-community org, under the condition that:

• It is renamed to kube-prometheus-chart to actually reflect its content, and confuse users less
• There are at least 2 maintainers (I would only sponsor, not be maintainer, now or ever)
• The maintainers actively try to minimize the differences between the chart and kube-prometheus (I believe this is already happening but I would like to see it be an explicit goal)

[stuart-c]

@brancz when you say "the chart" are you referring to the prometheus operator chart, as the repo would be home to all the prometheus related charts (prom operator, node exporter, alertmanager, other exporters, etc)?

[SuperQ]

I've created @prometheus-community/helm-charts GitHub team to manage things.

[brancz]

I'm specifically talking about what is in the upstream charts repo referred to as the prometheus-operator chart.

[stuart-c]

I'd be happy to lend a hand if you would like an additional maintainer...

[Nexucis]

It is renamed to kube-prometheus-chart to actually reflect its content, and confuse users less

Sorry by advance for the dummy question, but what is the point of putting kube in the name of the repo ?

Helm-charts is for sure for kube. People that doesn't make the link probably shouldn't use it.

And you already have the name prometheus in prometheus-community.
IMHO prometheus-community/helm-charts is obviously the different helm charts relative to the different prometheus repositories which then should be used into kube

[scottrigby]

@brancz OK, I updated the issue description to say it was decided that the prometheus-operator chart should move to the prometheus-community org, per prometheus-operator/prometheus-operator#3169 (comment) 👌

Simplifying charts

Regarding this note from that thread:

Aside from that, I'm excited to see how what you mentioned @scottrigby works out in practice around minimal helm charts, and hope to see the chart evolve in that direction.

Yes! I'm happy to continue the discussion with chart maintainers to help this happen. One potentially valuable direction could be to create new, minimal charts at some point once refactoring/consolidation/renaming is agreed upon as chart API v2 charts (the default when using helm create with Helm 3). These could be a breaking MAJOR version to the same charts after Helm 2 is officially deprecated in November, or we could start earlier and these could live alongside the legacy ones until the old charts are deprecated. The only thing we can't do is have two charts with the same name in the same helm repo.

Multi-step process

Clarifying, do you agree we should follow the multi-step process I summarized in this issue description above? If it's not clear, I'll expand on it a bit here:

Step 1. Move over all existing stable prometheus-related charts to a new space: github.com/prometheus-community/helm-charts

These are the prometheus related charts:

1. stable/prometheus
2. stable/prometheus-adapter
3. stable/prometheus-blackbox-exporter
4. stable/prometheus-cloudwatch-exporter
5. stable/prometheus-consul-exporter
6. stable/prometheus-couchdb-exporter
7. stable/prometheus-mongodb-exporter
8. stable/prometheus-mysql-exporter
9. stable/prometheus-nats-exporter
10. stable/prometheus-node-exporter
11. stable/prometheus-operator
12. stable/prometheus-postgres-exporter
13. stable/prometheus-pushgateway
14. stable/prometheus-rabbitmq-exporter
15. stable/prometheus-redis-exporter
16. stable/prometheus-snmp-exporter
17. stable/prometheus-to-sd

Step 2. Collaborate with prometheus devs to refactor/rename and simplify charts as appropriate

• This can be done in-place, or in new charts within the same repo alongside the legacy ones
• Starting by moving over the legacy charts first will not preclude however this is decided on a chart-by-chart basis
• Helm also has a built-in mechanism to deprecate charts, so this will aid in clarifying which charts will be best to retire or rename as opposed to updating in-place
• It is fine, and in fact advisable for charts to be located in a subdirectory of a git repo. They can also be placed into different subdirectories (as incubator and stable currently are) in order to differentiate or group them. So we can also keep this in mind if this is a concern during the subsequent refactoring work
• In case it's not clear, there are a few benefits of starting with the existing charts in step 1, and incrementally improving them in step 2 (as opposed to rearchitecting/renaming to begin with). First is to keep the git history (including all lessons learned from years of past chart maintenance). Second so that legacy users not able to update yet to the newer versions can continue to download the initial version of the charts from your repo when the stable and incubator helm repos are no longer available (this is due to the object storage buckets being owned by Google, which will not necessarily be maintained after the November deprecation date)

Does this make sense?

[scottrigby]

@SuperQ re #28 (comment) if you would like to add me to the @prometheus-community/helm-charts GitHub team I'm more than happy to help configure the initial repo, move over the initial version of the charts, set up the standard helm repo CI/CD etc. There may also be other helm maintainers who would be willing to help with ongoing repo-level administration help, though I think after initial setup the prometheus-community members can also take over this part.

We can also discuss the process of adding chart maintainers, who will require git repo write access. I would recommend asking the current maintainers of the existing charts if they would like to be added to this. If there is a reason to differentiate maintainers of each of the individual charts, this can be done with a GitHub CODEOWNERS file (I'm not assuming that's necessary, just noting this is a good solution if it becomes a need through discussion about chart maintenance moving forward).

Please just lmk when this is ready and I will hop on it. If I miss replies about that here I'd appreciate if someone wouldn't mind pinging me in Slack or IRC. Thanks!

[stuart-c]

As part of this will the existing charts in the helm org be deprecated and their README files updated to point here?

[matthiasr]

I think this is a good move, from an ecosystem standpoint it makes sense. I'm not 100% sure what the governance model is now (#7) but if we need a formal sponsor from -team I am willing to take that. Unless we have someone who actually uses Helm day-to-day, that would be even better then?