Should exported records be sorted?

[mboes]

Since #1235, the order of fields in records is deterministic. Furthermore, the order of fields is the one given in the source code. But this order is lost in two situations: exports and merges. Why are exports sorted? It's rather limiting that that output order can therefore only be the alphanumeric order, rather than any arbitrary user-chosen one. It means that using Nickel for e.g. YAML-to-YAML transformations is not order preserving. Would you consider simply not sorting exports?

As for merges, while that's a more complex question, I think avoiding a sort is still doable. You could say that merges preserves the order left-to-right.

[yannham]

In the grand scheme of things, the fact that some tools depend on the order of YAML fields is annoying. Maps should have an unordered semantics, and so should be exportable in any order, especially for exports that are supposed to be fed to other programs, and not really code to be maintained in Nickel.

However, it's a fact of life that some tools ingesting YAML are order dependent. I think it would be fine to stop sorting records - I suspect we did it at a time where the first goal was to have a deterministic order, but since the introduction of IndexMap, we do have deterministic order (insertion/creation order).

For merge I would be inclined to not provide any guarantee. I feel like that could hamper future optimizations. We could say: the output order is deterministic for a given version of Nickel, but is unspecified, and can change with time. It's a bit more annoying for tests though.

[mboes]

In the grand scheme of things, the fact that some tools depend on the order of YAML fields is annoying.

I haven't come across such tools. The concern isn't so much the tools as it is the humans reading the output. A niche I'm finding Nickel is good at is YAML-to-YAML transformations. Both the input and the output is meant to be human readable. Insofar as YAML (and TOML, JSON, ...) are simply alternative syntax for Nickel records, then it's nice to be able to read and write files with fields in an arbitrary order. Record fields in Nickel source files can be given in any order that is most readable to the user. That's true also of YAML input. It would be great if it were true of Nickel and YAML output too.

In my current use case, I have a YAML file containing an array of devices:

- name: device1
  type: type1
  A_attribute: a
  Z_attribute: z
- name: device2
  type: type2
  B_attribute: a
  C_attribute: c

If I roundtrip this through the Nickel identity function, I get:

- A_attribute: a
  name: device1
  type: type1
  Z_attribute: z
- B_attribute: a
  C_attribute: c
  name: device2
  type: type2

[yannham]

I see. I think the current situation is due to the fact that exported YAML/JSON is seen merely as an artifact, like assembly for a compiler, so we don't try too hard to preserve its structure. YAML processing isn't really the foundational use-case. But I wouldn't be against removing the sorting, it sounds reasonable, and we did have complains before (but because of tools for which order is signficant). Or we could toggle the behavior with a CLI flag at least.

[L0r3m1p5um]

I primarily use Nickel to export configurations to other formats that get committed to repositories for gitops tools to read. So I like the sorting because it makes the diffs easy to read, and I can upgrade Nickel without having a lot of spurious changes. And for sure if the sorting order became fully non-deterministic that would create some serious problems with how I'm using it right now, although I don't think that's what's being proposed. I'm not against having a flag control that though.

[yannham]

To be fair, I think the proposal is to have a deterministic order, just not the sorted one (the one provided by IndexMap, which is more or less insertion order I believe). But I agree that having a deterministic and easily computable ordering as it is now is also valuable, so a flag seems to make sense. Leaving the question of what should be the default behavior. If I were to start again, I would probably say insertion order. However that's going to break everyone else depending on the current behavior, so I'm inclined to say let's keep sorted as the default but introduce a flag for insertion order.