Feature: Weighted default / Idiomatic way of changing the default

[ghostsquad]

Is your feature request related to a problem? Please describe.

I'm building an opinion-based layer system, and I'd like to use things like "foo" | *"bar" (think about how the "if null then X" is the behavior of many fields), but allowing someone else to come along and say, actually, I prefer that foo be the default instead if the user doesn't specify something different.

Describe the solution you'd like

Either an explanation of how to accomplish this, or syntax to allow for a weight-based preference to default values.

Describe alternatives you've considered

N/A - I'm still learning CUE, coming from Jsonnet experience. Trying to figure out how to accomplish this common use-case.

Additional context

N/A

[kghenderson]

note that, depending on your use case and if/how users interact, you could also do things a bit the old fashioned way: like how you might do it excel or python or another language ... e.g. have a ValueUserInput, ValueUserDefault, ValueSystemDefault, and ValueFinal ... UserInput is what they've entered, SystemDefault is the provided default, UserDefault is the custom one, and Final is a formula which checks for those values (with != _|_ aka "not equal to bottom"), checking them in sequence. the naming is up to you, and you can do this as a series of if statements or perhaps add them to a list and coalesce (comprehension where not bottom) - just thought of this list option though so it might need some more thought - just trying to save what could be a complex set of nested if statements. if this seems like something you'd like to try, let me know and i'll test it out.

[ghostsquad]

The problem is that what you are describing is a LOT of boilerplate that I would have to apply to literally every value.
Another downside to this proposal is that I must predefine how many layers there are. My goal is to make the number of layers arbitrary.

[ghostsquad]

So an example, from the tutorials is this:

*"tcp" | "udp"

this says, it can only be udp or tcp , and without further input, it will resolve to tcp.
and with:

*"tcp" | "udp"
"tcp" | *"udp"

it's ambiguous, and is essentially the same as if I didn't have a preferred mark at all

so, what if I could mark *"udp" as weighted heavier than *"tcp"
essentially meaning, without further input, this will be udp because it's weighted heavier

I suppose this would be most similar to a max function, but not caring about the value, just the preference for such a value.

Technically CUE already supports this, since a concrete value has "infinite" weight, and the preferred mark is somewhere between concrete and arbitrary (within the defined bounds).

And there are other similar functions, such as null/bottom coalescing.

[ghostsquad]

Here's an idea:

*"tcp" could be equivalent to prefer("tcp", 0), where the second parameter to prefer is the weight of the preference.

This system could work similar to priorityclass in Kubernetes. The default priority is 0, and any valid int32 value, positive or negative is allowed. Pods are scheduled based on their priority.

[ghostsquad]

turning the preference mark into a function has additional benefits in that you could declare the preference as part of an expression, such as in a comprehension.

[zzh8829]

Hello I am also very invested in this idea. here is my proposed solution

adding the concept of weighted priority on defaults with
*(weight)value

val = *(0)val  
*val = *(1)val

combining two values in disjunction
*(x)val & *(y)val = *(max(x,y))val

what do you gues think about this

here is an example of it in practice

[zzh8829]

more generically speaking, a field has no default when the max(weight) contains multiple valid values.

[myitcv]

@ghostsquad thanks for raising this, and welcome to CUE!

Does the following approach satisfy your requirements?

#A: {
    defaults: {
        k8s?: string
        user?: string
        more?: string
        evenMore?: string
    }
    
    protocol?: string
    protocol: *(*(*(*(defaults.evenMore | _|_) | defaults.more) | defaults.user) | defaults.k8s) | _
}

examples: [string]: #A

examples: eg0: defaults: {
}


examples: eg1: defaults: {
    k8s: "test"
}

examples: eg2: defaults: {
    k8s: "test"
    user: "user"
}

examples: eg3: defaults: {
    user: "user"
    evenMore: "something"
}

examples: eg3: {
    defaults: {
        user: "user"
        evenMore: "something"
    }
    protocol: "rubbish"
}

Gives:

examples: {
	eg0: {
		defaults: {}
		protocol: string
	}
	eg1: {
		defaults: {
			k8s: "test"
		}
		protocol: "test"
	}
	eg2: {
		defaults: {
			k8s:  "test"
			user: "user"
		}
		protocol: "user"
	}
	eg3: {
		defaults: {
			user:     "user"
			evenMore: "something"
		}
		protocol: "rubbish"
	}
}

Playground link

However, it would be good to get some more details on the problem you are trying to solve in your "opinion-based layer system", because that will really help to inform whether we can/should do more in CUE the language to help this sort of pattern. Can you flesh things out a bit more in pseudo code? Is @zzh8829's approach to layers close to what you want? If evenMoreDefaults does not specify a default for a field, do you want to fall through to the next layer? Or is it the case that you want to choose which profile of defaults you want to use?

[zzh8829]

this approach would work except the author of the base layer #A arbitrary decided there can only be 4 types of defaults. If I am a user of the system and don't have easy way to modify #A, it would be hard for me to introduce the 5th layers of customization.

[myitcv]

@ghostsquad I've converted your feature request to a discussion whilst we flesh out what it is precisely that being proposed here.

[ghostsquad]

@myitcv the example may solve the problem, but it's not intuitive because it's using ancillary fields. I really want a user to just say this:

protocol: prefer("foo", 100) | string

and what that means, is that, if nothing is provided and there's no preference greater than 100, use the value foo. Adding ancillary fields adds a ton of boilerplate.

[myitcv]

The example you've provided here only has one default value. Please can you give an example where there are multiple? What's the author's intent in that situation? Indeed how does that situation come about? Are there multiple authors? Are there multiple fields for which different people want to specify different defaults?

At any given point in time, if I provide a value with a priority greater than 100 then I am saying "this value should be the default". It doesn't ever seem to make any sense to provide a value with a priority less than 100. So this suggests there is an order in which people are consulted to make decisions?

I'm not asking to push back, rather I'm asking to understand the workflow you have here.

[ghostsquad]

Indeed, multiple authors.

protocol: prefer("udp", 100) | string
protocol: *"tcp" | "udp"

Udp wins. It has a higher preference weight.

if I provide a value with a priority greater than 100 then I am saying "this value should be the default"

Yes

It doesn't ever seem to make any sense to provide a value with a priority less than 100

The existence of other preferences is not in your control. If using a CUE library that has a preferred value, and you want to write your own abstraction on top of that, you cannot change the preferred value. This would allow you to specify (if you wish) a preferred higher weight value.

Think of CSS. The most specific rule wins. This is the same thing.

p { font-size: 15px }
div > p { font-size: 20px }

The latter one wins if there's nothing more specific than it (and the constraints match)

So this suggests there is an order in which people are consulted to make decisions?

Yes

[ghostsquad]

I believe CUE already supports these semantics:

a concrete value is already the most specific thing you can be, thus it is used.
second to that is *"foo" which is a preferred value.

Let's just widen this gap.

[ghostsquad]

I actually don't think that multiple layers/defaults makes things unmanageable. I think being not being able to trace where a value is coming from is what makes configuration unmanageable.

I'm creating some user stories for a tool I'm working on that should hopefully help explain why multiple defaults are needed (with different weights)

[ghostsquad]

I found the previous community slide deck, and I don't feel like multi-field defaults is very idiomatic.

https://docs.google.com/presentation/d/1vZnHChGex1zRxtR5cQIQMP19gHzA2xaIjkuAMDnSKr8/edit#slide=id.g19d05509747_0_75

---

Here are some use cases / user stories (also in the Github discussion)

1. I have a k8s manifest with all fields filled out with values that are default by the k8s authors, e.g. tcp: *"tcp" | "udp" for a port
2. I have a user that is developing a slew of apps, most, but not all are actually udp in nature. But that user specifies a "layer" of configuration that may use some of the upstream defaults, but also defines his/her own default values, as such, e.g. *"udp" | "tcp"
3. Each individual app will have ultimate control of final values, and ideally very little configuration is actually needed at this point.

The problem exists between layers 1 & 2 though. Where the base layer provides value by specifying defaults, but the base layer isn't the only base layer, it's just a jumping off point (similar to that of how k8s yaml manifests work today. e.g. if null then XYZ default is applied

[ghostsquad]

I'm wondering if anyone else has had a chance to review this discussion? I'm still interested in this use case.

[eonpatapon]

This reminds me of the NixOS module system where you cannot have conflicting values when merging the config but you can resolve the conflict with priorities.

So that if you take an upstream config you can always "force" a value (given it's the right type) by using an higher priority.

The new nickel language which is inspired by nix and the NixOS module system takes this approach as well: https://nickel-lang.org/user-manual/merging#merge-priorities

In the current situation with cue it's never a good idea to define defaults in a base schema because the consumer of the schema will not be able to define it's own defaults. And by not specifying any defaults in the base schema the consumer needs to figure out what's need to be set and which values makes sense. Be able to provide consistent defaults make the usage of schema more straightforward for consumer.

I think the approach to take with cue for a schema author is to provide a base schema with no defaults, and some variation(s) of the schema with consistent defaults values so that the consumer can either use a pre-filled schema or the bare schema (without default values) depending on the use case.

[the-nurk]

i'm aware you said boilerplate is needed, but i think that's the point. as much as this appeals to me for rank-choice voting. my brain has a hunch this just means you didn't factor your design into proper disjunctions. you either have a case where a struct has default {a: *"a" | string} or it has default {a: *"b" | string} those are two separate matchN() validation checks you can place. the default mixin pattern is really the best decision in my opinion. just don't place them and only layer them where they are needed to ensure a type has been restricted to a lesser type on a certain condition. b: *{if x {[for _, v in A {v}]}} | null.

aside from messy mixins, you can simulate multi-level defaults several ways efficiently i think. my favorite is with a small validator that operates over an ordered list of types. trying each one based on a priority you encode alongside or in the order of the list. you can keep a struct alongside the relevant field and as you add disjunctions to the or([]), you also add a priority for that disjunction in the struct. sort the struct by rhs to get your priority order then use the lhs in sequence to reverse lookup the default in the or([]) list. the second you are doing complex defaults you should stop doing | for disjunctions all together as it's likely you'll start needing to resolve dynamic references to fill your default or your encoded priority order.

this can be embedded in a hidden field and produce a multi-layered 'default' on the scalar output. #4176 for that embedded function pattern, though that issue is about performing a pseudo-cyclical typecheck as the embedded function.

you can also use the different field types to emulate a layers default as well. _#x is most private layer, _x is next private, #x then this, x is linked to #x.final default junction.

there are probably thousands of other ways as well of course. just my 2c and i think i'm still missing stuff.

@ghostsquad