Reapply "stdenv: Add CPE fields to meta"

[YorikSar]

This reverts commit de74f9c from #438527 that reverted #409797.
Added a change that removes all null values from meta.identifiers to satisfy nix-env's XML and JSON outputs.
Unfortunately, this means that our API will have "holes" for fields with unknown values.

---

Add a 👍 reaction to pull requests you find important.

[YorikSar]

Here's the performance comparison between the old approach and the new one:

metric	mean_before	mean_after	mean_diff	mean_%_change	p_value	t_stat
envs.bytes	773000832.7273	773124234.9091	123402.1818	0.0182	0.0000	7.6144
envs.elements	57501262.8182	57508976.0000	7713.1818	0.0152	0.0000	7.6133
envs.number	39123841.2727	39131553.3636	7712.0909	0.0229	0.0000	7.6154
gc.heapSize	2898215656.7273	2890590021.8182	-7625634.9091	-0.2375	0.1762	-1.4555
gc.totalBytes	4992288488.7273	4991888356.3636	-400132.3636	-0.0095	0.0001	-6.1880
list.bytes	99379604.3636	99255051.6364	-124552.7273	-0.1640	0.0001	-6.3162
list.elements	12422450.5455	12406881.4545	-15569.0909	-0.1640	0.0001	-6.3162
nrAvoided	47293532.6364	47271846.9091	-21685.7273	-0.0547	0.0001	-6.6385
nrExprs	1806066.0909	1806082.0909	16.0000	0.0010	-	inf
nrFunctionCalls	35081013.2727	35081566.4545	553.1818	0.0016	0.0003	5.4779
nrLookups	18419504.6364	18420242.1818	737.5455	0.0043	0.0067	3.4076
nrOpUpdateValuesCopied	98746514.4545	98739704.0909	-6810.3636	-0.0078	0.0000	-7.1531
nrOpUpdates	3628500.0000	3628996.6364	496.6364	0.0132	0.0003	5.3353
nrPrimOpCalls	20697403.9091	20641017.0000	-56386.9091	-0.3190	0.0000	-7.1727
nrThunks	49468843.7273	49469466.3636	622.6364	0.0013	0.0118	3.0716
sets.bytes	2341588157.0909	2341232168.7273	-355988.3636	-0.0183	0.0001	-6.5781
sets.elements	137456535.4545	137433681.0000	-22854.4545	-0.0199	0.0001	-6.7106
sets.number	8892724.3636	8893329.5455	605.1818	0.0070	0.0003	5.4113
values.bytes	1022322416.0000	1022332378.1818	9962.1818	0.0010	0.0118	3.0716
values.number	63895151.0000	63895773.6364	622.6364	0.0010	0.0118	3.0716

(only x86_64-linux, I've removed lines without diff and lines about time)

[vcunat]

A random quick thought: I suppose you did consider using e.g. the empty string instead of null, if placeholders are somehow preferable?

[YorikSar]

@vcunat these parts can be empty, so empty string would not work as a placeholder. Anything other than null would be either much slower or less intuitive than just omitting them, I think.

[vcunat]

Ah, I didn't know they can be empty. Then it's clear, I guess.

[h0nIg]

@YorikSar do you want that @infinisil approves this PR or what would be the next "step"?

[ConnorBaker]

A few comments; otherwise looks good!

[ConnorBaker]

Suggested change

	makeCPE =
	cpeParts:
	"cpe:2.3:${cpeParts.part}:${cpeParts.vendor}:${cpeParts.product}:${cpeParts.version}:${cpeParts.update}:${cpeParts.edition}:${cpeParts.sw_edition}:${cpeParts.target_sw}:${cpeParts.target_hw}:${cpeParts.language}:${cpeParts.other}";
	makeCPE = flip pipe [
	(attrVals [
	"part"
	"vendor"
	"product"
	"version"
	"update"
	"edition"
	"sw_edition"
	"target_sw"
	"target_hw"
	"language"
	"other"
	])
	(concatStringsSep ":")
	];

[ConnorBaker]

Instead of doing the length check above, have you considered having makeCPE unpack the function arguments? That would ensure that both the length is correct and that the names are correct.

[YorikSar]

Suggested change

That's just too many allocations and operations to add to each package evaluation.

having makeCPE unpack the function arguments

That's a nice idea. Will do. Although I would still check the length to avoid using exceptions as flow control.

[nikstur]

As far as I can tell the original issue that led to the revert has been fixed. Good to go from my end.

Thank you for the work @YorikSar! I think we will have to build some more experience to get the versions right, but this can only be done iteratively.

[ConnorBaker]

Any blockers preventing merging this?

[h0nIg]

IMHO no

[YorikSar]

@ConnorBaker Looks like you've caught this merge conflict as it was landing in master or smth. Rebased.

[nixpkgs-ci]

Backport failed for release-25.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-25.05
git worktree add -d .worktree/backport-439074-to-release-25.05 origin/release-25.05
cd .worktree/backport-439074-to-release-25.05
git switch --create backport-439074-to-release-25.05
git cherry-pick -x dd12290517ff6e57ab2ca384b101e0cdc617570c a178fd8c436915d233a1de37a2b211489aad5340 5e1eee582748866a51ca8f780829c3b2cf523fc3

[ConnorBaker]

Any chance you’d be able to do the backport? If not I’ll try to get around to it this weekend.

[vcunat]

Isn't this too intrusive for backporting?

[h0nIg]

Isn't this too intrusive for backporting?

the backport can be set on hold, if you want to gather experience? The benefit of CPE's and pURL outweights the low-medium risk.

The security tracker https://discourse.nixos.org/t/nixpkgs-supply-chain-security-project/34345/30 can benefit as well, because once a regular bump of software on master is done (without keeping a CVE in mind and determining "stable needs a fix as well"), nixpkgs stable fix is not triggered automatically.

Example: #409300 libarchive was bumped without CVE reference (edit to the PR description was adjusted AFTER additional issue was created ONLY), just by coincidence it was backported as well - which would have been unpatched for some time.

In addition CVE like this #411881 were not detected properly, therefore an additional reason for backporting / benefiting today

[YorikSar]

I've updated the previous backport in #438385 with new commits. Please take a look.
@vcunat This change only adds a field to meta, it doesn't change anything else about how packages are being handled, so it is safe to backport it to 25.05. The idea is to start using it with current release and gather as much feedback as possible.

[philiptaron]

Isn't this too intrusive for backporting?

I agree that we should not backport this. Since the 25.11 release cycle is starting, let's get the experience with this over the few months until that happens, and have 25.11 be the release where this becomes a Nixpkgs staple.

[adisbladis]

This feature seems very half baked:

• It has no internal usage & is pure overhead at the moment
• The design looks overly memory intensive (way too many attrsets all over the place)

I'm not opposed to the feature as such, but it needs to go back to the drawing board.
I want to revert this before it goes into 25.11.

[K900]

I would also very much prefer to see this land after 25.11, given how the previous attempts have gone.

[YorikSar]

@adisbladis Before reverting what was brewing and collecting feedback for a long time, could you please provide more specific suggestions here?

It has no internal usage & is pure overhead at the moment

It has external uses and was really expected even in 25.05. See #354012 for more context (also cc @nikstur here).

The design looks overly memory intensive (way too many attrsets all over the place)

Could you please be more specific here? While the final implementation of this PR increases the total amount of attrsets needed for evaluation, several spinoff PRs reduced all these numbers significantly, which more than pays for this. Of course, it would be nice to keep optimising our eval times, but I think we should do this gradually, without reverting any new feature that needs more resources. You can't expect every PR to be perfectly optimised and not require any additional resources before it lands. This is just not how the world works.

@K900 I'm not sure I follow your reasoning either.

given how the previous attempts have gone

Do you mean that because 1 (only one) previous attempt at landing this change broke evaluation, which was then fixed in this PR, this feature is doomed to be in revert-reapply purgatory?

@adisbladis @K900 We need this feature to land in the release to start collecting data for vulnerability tracking initiatives. I find it very frustrating that you've decided that this needs to be reverted 2 months after it was merged with no proper review (apart from how "it looks") and with no time to address any feedback you might provide before the release.

[SuperSandro2000]

I think it is very important to land those extra meta information to make the security process easier and more automated which hopefully reduces churn on the security team and makes the OS more secure for everyone.