Develop a way to handle application configuration

[bgrant0607]

This is related to #3119, but deserves its own issue.

In application-related resources, application configuration often constitutes a large proportion of the overall configuration size.

Application configuration is special in multiple ways:

1. Attributes can't be derived from well known KRM resource types
2. Many different formats, which are not KRM and sadly not as standardized as they could be
3. No explicit schema that kpt has access to

Command-line flags are evil, so I'll punt on them for now, other than using env var substitution to define their values.

Env vars are about the best case. Kustomize has support for generating ConfigMaps from env files, and Kubernetes can inject them as envvars. And, if represented natively in a ConfigMap or in a pod template, then they are KRM and could be edited as such. There's still no native schema though (kubernetes/kubernetes#4210). A command for editing env vars would also be nice.

I haven't looked for any kind of data, but presumably there are some relatively common file formats, such as INI, TOML, Spring Boot properties, etc.

A common, rational instinct is to normalize such formats into a universal, simpler structured form, generally a simple map or nested map. The most common approach is templating and template parameters, with all the consequences that implies. It's less terrible than other uses of templating if one views config files of unknown formats as just unstructured text, but does feel suboptimal. For instance, anyone familiar with how an application is configured would then need to learn the new representation and how it maps to the application-native one, since often syntax, capitalization, etc. are different. It also frequently requires insertion of conditional logic to handle present / not present of the properties. Some formats, such as JSON, are particularly challenging to ensure the output is valid.

For a variety of reasons, we rejected several proposals to support templating in Kubernetes itself (e.g., kubernetes/kubernetes#30716, kubernetes/kubernetes#89738, kubernetes/kubernetes#96346).

We investigated this issue some when we were designing ConfigMap (kubernetes/kubernetes#1553, kubernetes/kubernetes#2068).

I wonder if we could do something with http://augeas.net/index.html
"Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files."

We would like to provide a similar WYSIWYG transformation and editing experience for application configuration as for KRM resources, at least for a subset of common formats. We could even recommend an automation-friendly format for people writing their own applications.

This affects ~all the functionality of kpt: update merging, diffs, source and sink, function SDKs, the UI.

For example, we also need to be able to do granular merging during updates, in the original non-KRM config file, and the ensure any ConfigMaps they are embedded into are updated (#3119).

[justinsb]

One thing that Craig Box got me noodling about ... does yaml matter to kpt / to kubernetes? It clearly doesn't really matter; it's just a representation that we've decided upon.

Craig (jokingly?) suggested INI files as an alternative to yaml, and perhaps that is the path here. When we write configuration in INI or toml, we are actually setting values in a configuration object. That configuration object doesn't allow all keys, and has various restrictions on the values of those keys. In other words, even though we're writing in a different "expression" language, we could imagine writing an OpenAPI spec to describe the schema of the configuration.

This suggests we could think about writing a set of transformation functions from instances of CRDs to the various common configuration file formats. By doing so, we bring legacy configuration into the better-structured world of kubernetes and KRM.

We could do so either as a client-side object or as a true CRD with an operator.

This doesn't obviously solve #3119, so I'd imagine we would start client-side.

[bgrant0607]

An example of an application with lots of configuration is kafka:
https://github.com/bitnami/charts/blob/master/bitnami/kafka/values.yaml#L93
https://github.com/mesosphere/dcos-kafka-service/blob/master/frameworks/kafka/universe/config.json

Similar to the overall approach to WYSIWYG configuration, I wouldn't want to abstract the application configuration. For instance, as a user or developer I'd expect it to match what I saw in the code or development environment or documentation:
https://kafka.apache.org/documentation/#configuration

So, yes, some apps would express configuration in INI or TOML.

This is where something like Augeas is interesting. "Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree." Looking at http://augeas.net/docs/augeas.pdf, the idea sounds very close to what we would want. Like a pluggable source/sink for specific non-KRM file types.

[bgrant0607]

With #3118, we wouldn't technically need a custom source/sink. We'd still need custom parsing, marshaling, and visualization, though.

[bgrant0607]

As a concrete example that would address a segment of applications, we looked at Spring Boot config (application.properties) in the early days of the kpt project, but it looks like the demo video recordings don't exist any more. This post discusses it:
https://www.springboottutorial.com/spring-boot-application-configuration

[bgrant0607]

One specific category of application configuration is resource-dependent configuration: VM heap size, thread pool sizes, simultaneous connections, cache sizes, etc. Network- and disk-intensive applications often have a number of these tunable settings.

A number of legacy applications and even language runtimes are not container-aware. As an example, before Java was container-aware, additional automation was necessary that is not in more recent versions of the JDK.

Ideally these settings would be derived from container resource limits, either at run time, such as using an init container, or an application-specific function, which would be lighter weight than either an Operator or admission controller.

cc @johnbelamaric

[johnbelamaric]

I like an app-specific function, especially if is written in something like Starlark that does not require building and maintaining and coordinating versioning for a separate container image. An init container or custom Go function would require that.

[bgrant0607]

This video discusses an in-pod templating approach using init containers, which is a variation on the entrypoint.sh script approach: https://youtu.be/eJmNSYvelSw?t=1087

I'm liking the Augeas idea, though. If we could convert lots of config formats to a canonical form in kpt fn source, we could manipulate the canonical form and write it back using kpt fn sink.

[bgrant0607]

https://osquery.io/ apparently integrates with Augeas.
https://www.uptycs.com/blog/using-augeas-with-osquery-how-to-access-configuration-files-from-hundreds-of-applications

That's read-only, for queries.

Puppet integrates it also, for setting values:
https://puppet.com/docs/puppet/5.5/resources_augeas.html

And there's Go integration:
https://dev.to/raphink/configuration-surgery-with-go-structure-tags-12a4

[bgrant0607]

More examples:
https://ghost.org/docs/config/
https://dev.mysql.com/doc/refman/8.0/en/server-configuration-defaults.html
https://www.postgresql.org/docs/current/config-setting.html#CONFIG-SETTING-CONFIGURATION-FILE
https://www.rabbitmq.com/configure.html
https://redis.io/docs/manual/config/
https://www.nginx.com/resources/wiki/start/topics/examples/full/
https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://etcd.io/docs/v3.4/op-guide/configuration/
https://www.vaultproject.io/docs/configuration
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
https://wpmudev.com/blog/wordpress-wp-config-file-guide/ (php may be too hard)
https://www.drupal.org/docs/configuration-management/managing-your-sites-configuration

We can look through charts for more examples:
https://github.com/bitnami/charts/tree/master/bitnami