Expressions: List and object support

[olemartinorg]

Right now, the expression language does not work with lists/arrays or objects, and will simply return null if you try to look one up from the data model. This is unfortunate, and the language should support these values - but we need to provide tools to manipulate them as well.

Needed functionality:

• list function
  • Right now, all expressions are JSON arrays. If we want to support lists/arrays in the expression language, we need a way to represent a static list of values. Say, for example, that you wanted to check if a data model path is in a list of values with ["in", ["dataModel", ...], ["value1", "value2", "value3"]], the expression engine would think ["value1", "value2", "value3"] is another expression - and halt when the function value1 is not found.
  • So, to mitigate this problem, we could create a list function that simply takes any amount of parameters and returns them as a list/array. So the above could be written as ["in", ["dataModel", ...], ["list", "value1", "value2", "value3"]].
• listLength function (or similar - decide in tandem with the string length function name in Expressions: New simple functions #1176)
  • An obvious function to include is one to measure the length of a list (given as the first argument).
• Lookup a group and list binding
  • A group binding binds to a list/array of objects. If we support both data types in the expression language, we should support looking up these values using the component function. I think the most obvious thing here is to extend component to work on any binding type (if there is only one binding), and allow you to specify which binding (such as address) as a second parameter if the component has multiple bindings.
• objKeys function (or similar)
  • A function that returns a list of keys inside an object, simply Object.keys()
• pluck function
  • This function should be the powerhouse of the object support. It should pluck a value from an object (given a key). If given a list as a parameter, it should pluck out the key from every object in the list and return a list.
  • Example: ["pluck", "name", [{"name": "Ole Martin"}, {"name": "Per"}]] => ["Ole Martin", "Per"]
  • We should create robust tests for this so we don't end up supporting side-effects like looking up the length of a list of lists (because list.length is valid in javascript)
• atIndex function
  • This function is for lists what pluck is for objects. It is named differently to indicate they are not interchangeable, but they operate the same. If you ["atIndex", 1, ["list", 1, 2, 3]] you should get 2 back.
  • We should create robust tests for this so we don't end up supporting side-effects like picking characters from a string.
• first function
  • First conceived in Dynamiske uttrykk: Mulighet til å vise tekst fra skjult felt #1669 (comment)
  • Similar to atIndex, but in cases where you don't know the index. This way you can use an expression to find the first element in a list that matches another expression.
  • If this uses the value function for sub-expressions, it would ruin cases where you would like to use both the value from the current context (such as in optionFilter) and this sub-expression to evaluate each list item. For that reason we might want to use something else, such as an item function that behaves much like value.

Other simple functions (moved from #1176):

• sum
  • Should take any amount of numbers as input, and also lists of numbers. All these numbers should be summed, and the resulting number should be returned. null and non-numeric values should be considered to be 0 (i.e. no effect on the sum).
• average
  • Same as above, but should produce the average of all the numbers summed. null and non-numeric values should be considered entries when calculating the average, as we should make a filtering function for those cases where invalid values should not be considered instead.
• in
  • Shortcut function similar to equals, but compares the first argument against multiple others to check if the first argument matches either one of them. Should support lists of items as parameters, and should then check inside the lists as well.
• notIn
  • Synonym for ["not", ["in", ....]]

Other functions we should consider:

• map (see argv in Expressions: Allow for re-use of expressions altinn-studio#16764)
  • As an example, you could ["map", ["upperCase", ["argv", 0]], ["list", "foo", "bar"]] to get ["FOO", "BAR"]
  • This would require deeper expression engine support than what is implemented for functions right now, as this is similar to a closure/callback data type.
• filter
• reduce
• sort (for lists of strings/numbers)
• reverse (for lists of strings/numbers)
• concat support for joining lists?

[EirikOlavHaugenSSB]

Har et use case som jeg tror gjelder i mange skjemaer. Jeg skulle gjerne brukt dette til å skjule en side hvis tabellen som siden ligger på var tom. Akkurat nå så må jeg gjøre dette i backend kode ved å:

1. Løpe gjennom alle rader i repeterende gruppe
2. Sjekke om minst et av alternativene er synlige ved å sjekke på det samme som jeg har satt i "hiddenrow"
3. Sette et flagg i datamodellen og sette "hidden" på toppen av siden om dette er true/false

Hvis jeg kunne gjort dette i dynamiske uttrykk så kunne det sett f.eks. slik ut:

1. Hidden: ["listlength", ["datamodel", "repgruppe-felt-i-datamodell"], "0"]

Dette krever at listlength ikke teller en rad dersom den var skjult med "hiddenrow". Hvis det er andre grunner til at skjulte rader tellles kunne det vært forskjell på listlength hvis man la in datamodellfeltet eller gruppekomponenten. Altså at listlength på en gruppe med 2 elementer returnerer "null"/0 hvis alle er skjult når man bruker "component", og "2" når man bruker "datamodel".

Har hatt flere skjema der dette kunne vært veldig hjelpsomt og gjort at det ikke var behov for backend kode i dette tilfellet.

[olemartinorg]

Interessant, og takk for use-case! 🙌 Den tar vi absolutt med oss.

Noen kommentarer:

1. Jeg tror ikke det uttrykket gir mening om du slår opp i datamodellen direkte, nei. Datamodellen kan ha 10 rader hvor RepeatingGroup-komponenten bare viser 5 av dem. Men så kan du også ha flere RepeatingGroup-komponenter knyttet til samme sted i datamodellen - og da blir det vanskelig for oss å vite hva du egentlig vil ha når du slår opp i datamodellen (det som er vist i begge, det som er vist i minst en av dem, eller faktisk bare det som finnes i datamodellen).
2. Sånnsett hadde Expressions: Allow for re-use of expressions altinn-studio#16764 blitt nyttig her, ved at du kunne sjekket "lengden på listen dersom du skjuler <med det samme uttrykket som blir brukt i RepeatingGroup sin hiddenRow>"
3. Enig i at ["equals", ["listLength", ["component", "myGroup"]], 0] ville være veldig nyttig. Selv om den løsningen virker åpenbar, er det litt problematisk at måten component-oppslag er bygget på gjør at det bare fungerer mot en simpleBinding. Vi kunne såklart utvidet til både simpleBinding og group, men man må begynne å spørre seg selv litt om løsningen er den beste - for noen komponenter har potensielt også flere datamodellbindinger. Kanskje det kunne fungert med ["equals", ["listLength", ["component", "myGroup", "group"]], 0]? Altså at man må spesifisere hvilken binding man ønsker dersom man vil sjekke på noe annet enn simpleBinding.

[olemartinorg]

Iterating lists, objects and repeating groups in expressions have some caveats when it comes to the concept of hidden components and data in them. For a while I've had an assumption that removing hidden data from the data model will happen in frontend after Altinn/app-lib-dotnet#1157, but after further work on that task we're gravitating in a direction where it seems backend will be doing most of the hidden-data-removal work.

So, if frontend is never going to run expressions based on a 'cleaned' data model, we have to take that into account here as well. So for example, in addition to mapping a repeating group structure (object[]) to an expression function, we should at the same time allow for mapping a given set of components (given their base component ids) inside a repeating group. The former (iterating an array directly from the data model) would still iterate hidden rows and see hidden data, while the latter (iterating repeated components) would naturally only iterate visible ones.

This all might lead to confusion, and might not always be what people want (see #1669).