Proposal: helm commandeer

[thomastaylor312]

EDIT: This has been superseded by another idea

Problem

Many users want to be able to "adopt" currently running pods, services, etc. into a new or existing chart. One example of this is self-hosted Kubernetes control plane components (apiserver, DNS, etc.) after they have been bootstrapped by something like bootkube.

Proposed solution

I propose we add a new commandeer (to stick with a naval theme, we can go with adopt if people find it too confusing) subcommand to Helm. The reason behind this is we don't want to add additional complexity and flags to an already large flag set on install, update, and rollback. The command would take a defined chart (e.g. helm commandeer <chart_name>) with matching manifests and use these to "commandeer" the objects into a chart manifest. Future iterations of the feature could add the ability to pass specific names of resources you want to commandeer.

Behavior

The passed chart would be rendered like a normal install or upgrade before moving on to a "capture" step. The "capture" step would use the object GroupVersionKind, namespace, and name to find a match in the already existing objects. For safety sake, if any of the rendered manifests do not exactly match a resource, the "capture" step will fail. There will also be an --override flag and an --ignore-match flag to help configure this behavior. The --override flag will make it so as long as the Kind and name match, it will accept the match. The --ignore-match flag will drop any non-matching manifest from the final chart.

The last step will be a merge patch of the new manifest with the existing one. Once these are merged and sent to the API, the new chart will be saved into storage. So the tl;dr is below

1. Render chart
2. Capture resources using GroupVersionKind, namespace, and name
3. Merge manifests
4. Save new chart

The --into flag

We also want to have the ability to adopt resources into a currently installed chart (e.g. bringing resources into an umbrella chart). This will be done with the --into flag, which will take a release name as an argument. All of the same steps will be followed to adopt resources with the exception that existing resource will remain the same. This is a purely additive operation

Open questions

• Should the --override flag force resources in the chart that don't have a match to be created anyway?
• Is --ignore-match something people will actually use?
• Do we need something like a --replace flag where the new manifest completely replaces the existing one?
• Is this too complex for a first pass and should we just stick with specifying resource names to commandeer?

Other notes

I will update this proposal with suggestions from the comments so as to not have multiple proposal versions clogging up the issue

[nebril]

How will commandeer know which resources it would group? E.g. I have wordpress deployed, in the way the chart would deploy it - mariadb and wordpress deployments with some pvcs. I also have other mariadb deployments in a cluster. How will commandeer know which mariadb to put in a release?

[seh]

When you say "exactly match a resource," we need to be careful to distinguish between facets not specified in the chart, and those the chart mandates not be present. For example, most manifests won't specify any "status"-related details, nor various annotations contributed by system facilities. I assume that you want to match the following on most objects:

• Metadata
  • namespace
  • name
  • labels
  • some annotations
    You'd want to ignore most of them not mentioned in the chart, right?
• Spec

Beyond that, would you tolerate whatever happens to be there on an existing object?

[technosophos]

First off, I think this is a great feature, and I personally love the name "commandeer" (though maybe "adopt" is a little more consistent and easier for people like me to spell).

I am duty bound to ask the question I am notorious for: Should this be a plugin? 😄

Okay, with that out of the way....

Here's a different way we could approach the same issue. What if we generated the chart from the existing resources? I imagine it would work something like this:

$ helm commandeer --name=myNewChart deployment:www-frontend service:www-gateway configmap:www-config
Generated chart myNewChart
Added Deployment www-frontend
Added Service www-gateway
Added ConfigMap www-config
Chart myNewChart is released as myNewChart.v1

(syntax above could definitely be improved)

From there, the user would have a myNewChart chart to work with to manage that resource, and also a release record for the current chart. Going forward, the user could:

• Edit the manifests in myNewChart/templates and turn them into templates
• Rework the initially flat chart into subcharts
• Create a new values file, etc.
• And then do a helm upgrade to fully transform the release

I think this idea is sorta backwards from yours, so maybe it's too far off. I'm trying to figure out what the user's use-case looks like. Is it that they already have a chart that describes the existing stuff? Or is it that they want to take an existing thing and helmify it?

@devth , who once proposed doing something like this, might also have some ideas.

In any event, I like the proposal, and I know there are many who would benefit from this. So I'd be in favor of it.

[thomastaylor312]

@technosophos That was actually the other way I was thinking of doing it but I didn't write it all out. I should of, because now that I have seen it written out, it makes it seem like a better idea. I think it would be more simple to implement and less cognitive burden on the user.

I did think about the plugin way of going about this, but I have heard lots of different people mentioned this from time to time, so it would probably be valuable to have in core.

So anyway, let me rework the initial proposal over the weekend to incorporate your idea @technosophos and we can continue discussion. I'll try to see if we can get some other helm power users to comment on this

[sstarcher]

For my use case either of the 2 options would work as long as with the second option I don't have to use the chart generated. I'm mainly interested in bootstrapping kubernetes with something like https://github.com/kubernetes-incubator/bootkube for a self-hosted cluster, but being able to upgrade the manifests with a helm upgrade

So in my case I would have a handful of manifests that are written to disk somewhere, but I could like to be able to helm commander kube-config and helm upgrade kube-config to upgrade my controller-manager, api server, proxy etc.

[jascott1]

There might be some learnings to be taken from here
https://github.com/appscode/chartify

[thomastaylor312]

Proposal take 2

Given the feedback thus far, I propose the following functionality:

Command line design

There will be two ways to select resources to commandeer:

1. Comma delimited list of <kind>:<name> (e.g. Deployment:www-frontend)
2. A label query that will be the exact same functionality as with kubectl (e.g. app in (foo, bar))

The default method will be using the <kind>:<name> syntax and the label query will be exposed by command line flag (--label or -l). Users will be able to specify a combination of the two to select multiple resources.

The default behavior will be strict matching. For the label syntax this will be that it must match at least one resource. For <kind>:<name>, each resource listed must match an existing resource.

Additional Flags

• --ignore-missing: This will allow for "non-strict" matching. Meaning as long as the label selectors or <kind>:<name> list match at least one existing resource, it will create a release.
• --dry-run: Same as on other commands
• --debug: Similar to other commands. Will print out the created manifests to STDOUT
• --name: The name of the new chart/release
• --output: The file path to output the new chart to

How it will work

Tiller will look for matching resources and turn them into YAML manifests. It will also create a release with those objects. Once the release is complete, Helm will output the new chart into the current working directory or the one specified with --output. It will create one file per resource it commandeers.

Then, as stated by @technosophos:

From there, the user would have a myNewChart chart to work with to manage that resource, and also a release record for the current chart. Going forward, the user could:

• Edit the manifests in myNewChart/templates and turn them into templates
• Rework the initially flat chart into subcharts
• Create a new values file, etc.
• And then do a helm upgrade to fully transform the release

Important note

Given past experience with other features (cough...cough --wait), this feature will be marked as alpha on initial release to allow for more testing across the various use cases. It will graduate to beta after that testing. It will not be marked as fully ready until Helm 3.0 or it is decided by the core maintainers (obviously with feedback from community testing) that it is ready, whichever comes first

Let me know if this looks like the best path forward

[seh]

Do you intend to save all of the objects' current state, or will you filter it, per my earlier comment?

[thomastaylor312]

@seh I would definitely drop the current status and probably some of the annotations that contain the last version of the object. When we create a PR, we can make sure all of those cases are covered

[thomastaylor312]

After talking in the developer meeting, the choice was made to develop this as a plugin.

[thomastaylor312]

I will be developing the plugin here. Feel free to follow along while I work on it