`kpt fn render|eval` should support excluding resources from function input

[droot]

CUJ

kpt fn render and kpt fn eval supports mechanism to target resources to apply function on. For example:

# Apply set-labels on resources of kind Deployment and belonging to group apps
$ kpt fn eval [PKG_DIR] -i set-labals:v0.1.5 --match-group apps --match-kind Deployment 

Declaratively, for example, apply set-labels on resources with group apps and kind Deployment.

# excerpts from Kptfile

...

pipeline:
  mutators:
   - image: set-labels:v0.1.5
     selectors:
       - group: apps
          kind: Deployment

Sometimes users want to exclude set of resources from function's target. For ex. apply this function on all resources except the ones with kind Kptfile or apply this label on all the deployments except with a specific name.

This will become especially important as we allow functionConfigs and Kptfile to be included in function input by default in kpt fn commands.

Proposal

Imperative workflow:

# apply set-labels to all the deployments except a given name
$ kpt fn eval [PKG_DIR] -i set-labels:v0.1.5 --match-kind Deployment --exclude-name some-name

# apply set-labels on resources belonging to apps group except kind StatefulSet
$ kpt fn eval [PKG_DIR] -i set-labels:v0.1.5 --match-group apps --exclude-kind StatefulSet

Declarative workflow:

Introduce exclude selectors.

# apply set-labels on all resources except with kind `Kptfile` or group `apps`
...
pipeline:
  mutators:
   - image: set-labels:v0.1.5
     exclude:
       - kind: Kptfile
       - group: apps

Alternatives names for exclude: except, reject

It will also be nice to have a way syntactic sugar for select or exclude all local-configs (resources with a config.kubernetes.io/local-config: "true" annotation).

[droot]

I looked for some prior art in this space and found kustomize's built-in replacement transformer to be the closest.

[natasha41575]

@droot Question, when we write something like

exclude:
  - kind: Kptfile
  - group: apps

Does that mean we should reject resources that have both kind Kptfile and group apps? Or does it mean that we should reject resources that have either kind Kptfile, or group apps?

It seems like it should be the latter, and to do the former it should look like

exclude:
  - kind: Kptfile
    group: apps

Is that correct?

[droot]

It's the latter. It's ORs(ANDs) very similar to how reject is implemented. It is also consistent with how selectors is implemented.

[droot]

/cc @bgrant0607 @yuwenma @mengqiy @phanimarupaka

[natasha41575]

+1 to the name exclude

From discussion yesterday, fn eval exclusions will also be treated as an 'AND', and this should be documented with examples to prevent confusion for users.

This means that something like --exclude-name some-name --exclude-kind StatefulSet will exclude resources that have both Kind StatefulSet and name some-name.

If a user wants to exclude resources that are of either Kind StatefulSet, OR name some-name, there will be no way to do this with eval, and users will have to do it declaratively with the Kptfile pipeline.

We can implement this feature in stages:

• First, add the basic exclude capabilities
• Then, add ability to exclude by annotations/labels
• Then, add ability to exclude all local-configs

Some related kustomize issues with similar requests:

Issues to skip PrefixTransformer for certain GVKs:

• skipping nameprefix of some resource kinds kubernetes-sigs/kustomize#519
• Filter PrefixSuffixTransformer on specific targets. kubernetes-sigs/kustomize#4490

KEP to skip transformations for a specific resource or resource kind:

• Kustomize FieldSpec skip kubernetes/enhancements#1232

[yuwenma]

Agree with the idea that kpt CLI should be flexible enough to provide a way to select/exclude resources. But I'm not sure if we want to use the proposed syntax.

Some high level questions I have:

1. What's the right syntax for users to describe the expected hydration workflow (Kptfile specification). Writing the exclusion logic in function level or even the mutator/validator list level is non-trivial user requirements.
2. When we talk about idempotent, should we consider the entire mutation, like changing the function execution order should not end up as different results (this is the criteria for horizontal and resource-specific functions, if kpt pipeline breaks this rule, it would make nested packages hydration error prone)

[natasha41575]

Discussed with @droot offline, we are thinking to not support these exclusions right now, and to release the change is and collect feedback from users.

Per @droot's request, I have created an example of how a user would exclude meta resources with the current selectors functionality: https://github.com/natasha41575/kpt-exclude-kptfile

The Kptfile looks like this:

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: example
pipeline:
  mutators:
    - image: gcr.io/kpt-fn/set-labels:unstable
      configPath: functionconfig.yaml
      selectors:
        - apiVersion: "batch/v1"
        - apiVersion: "apps/v1"
        - apiVersion: "v1"
        - apiVersion: "example.com/v1"

It's a bit verbose, but it doesn't look too bad to me. Of course verbosity would increase if users have more apiVersions they need to select.

[morgante]

For the use case of excluding meta resources, I think we need a simpler mechanism. We should not expect function or package authors to keep an exhaustive list of everything which is a meta resource.

We either need an explicit exclusion selector or for kpt to automatically add an annotation to all meta-resources.

[phanimarupaka]

This will become especially important as we allow functionConfigs and Kptfile to be included in function input by default in kpt fn commands. Is it ? What is the rationale behind this ? By default users expect kpt fn should only include normal resources.