stdenv: PURL fetcher introduction & feature flag

[h0nIg]

#421125 was merged and reverted later, because of regressions.

the background is described here: #421125 (comment)

@wolfgangwalther outlined the conditions and would like to enhance CI - over time. This is a continuous approach, which is in line with packages which have been found to be defunct and which need a fix. There may be more packages which have problems and we would like to prevent further fallout by a feature flag (prevents accessing + inheritance of drv.src / drv.srcs).

packages list: #453322 (comment)
list of real broken packages: #453322 (comment)
broken packages fix (deferrable PR: #457769)

With the old PR + the broken&platform check fix from #453291 + the feature flag, we enable maintainers to gather experience with PURL and set appropriate information (e.g. jq example, where fetchurl is used instead of fetchFromGithub)

nix-repl> xx = (import /my/nixpkgs {config={derivationPURLInheritance = true;};})
nix-repl> xx.python3Packages.boto3.meta.identifiers
{
  cpeParts = { ... };
  possibleCPEs = [ ... ];
  purl = "pkg:github/boto/[email protected]";
  purlParts = { ... };
  purls = [ ... ];
  v1 = { ... };
}

nix-repl> xx = (import /my/nixpkgs {})
nix-repl> xx.python3Packages.boto3.meta.identifiers
{
  cpeParts = { ... };
  possibleCPEs = [ ... ];
  purlParts = { ... };
  purls = [ ... ];
  v1 = { ... };
}

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[h0nIg]

we need to squash all commits later, just for transparency let's keep the commits separated in order to understand what has been changed.

[YorikSar]

Looks fine to me

[adisbladis]

Considering you're adding stuff to meta which is performance critical this needs to be performance regression tested.
Since the transition to Github Actions we no longer have a CI which performs those checks so this needs to be done manually.

Just quickly eyeballing this I'm really not happy how many array-ish types like attrsets & lists this seems to be using all over the place.

[h0nIg]

Considering you're adding stuff to meta which is performance critical this needs to be performance regression tested. Since the transition to Github Actions we no longer have a CI which performs those checks so this needs to be done manually.

Just quickly eyeballing this I'm really not happy how many array-ish types like attrsets & lists this seems to be using all over the place.

there was a lengthy discussion about the performance and improvements which have been implemented - in #421125

did you considered this / checked this already?

[wolfgangwalther]

Since the transition to Github Actions we no longer have a CI which performs those checks so this needs to be done manually.

You can see the performance report in the PR summary page: https://github.com/NixOS/nixpkgs/actions/runs/18756048808?pr=454333#summary-53508514968

[adisbladis]

there was a lengthy discussion about the performance and improvements which have been implemented - in #421125

Nowhere in there is the fundamental design discussed.
This looks poorly designed to me: It's going to create a lot of garbage.

[h0nIg]

This looks poorly designed to me: It's going to create a lot of garbage.

@adisbladis thank you for your (constructive) feedback, IMHO it does not make sense to create noise for the other readers and duplicate / repeat.

I don't think i can remove your initial doubts here, maybe we can have a short chat in matrix instead and we align there? I ping'ed you already

[YorikSar]

@h0nIg Could you please post here a summary of your discussion in Matrix? I am interested to see if there's some context I'm missing that surfaced there.

@adisbladis Are you against just any new features in meta? With such stance we'll end up stagnating on some very important aspects of Nixpkgs development, like security in these cases (#439074 (comment) and this PR). As I mentioned in that PR, we did a round of optimisation of this performace-critical code that should allow us to land some features without slowing us down overall.

[h0nIg]

@h0nIg Could you please post here a summary of your discussion in Matrix? I am interested to see if there's some context I'm missing that surfaced there.

@adisbladis Are you against just any new features in meta? With such stance we'll end up stagnating on some very important aspects of Nixpkgs development, like security in these cases (#439074 (comment) and this PR). As I mentioned in that PR, we did a round of optimisation of this performace-critical code that should allow us to land some features without slowing us down overall.

doesn't look like he is interested in a conversation

@jerith666 @jficz @bivsk @oneingan @stigtsp @CathalMullan @eclairevoyant @tomberek @pbsds @arianvp @06kellyjac @pombredanne @ConnorBaker @nikstur and others:

sorry, no PURL - can someone help please?

I'm tired and frustrated that we still have bogus discussions about 8MB additional memory out of 2600MB of evaluation memory. Even with a feature flag we continue to have this discussion - wtf.

https://github.com/NixOS/nixpkgs/actions/runs/18996522700?pr=454333
(4695549 + 191270 + 2088348 + 2150061) / 1024 / 1024 = 8,7MB diff
(590957759 + 83464891 + 1266555136 + 800003130) / 1024 / 1024 = 2614MB absolute

People would like to address security issues, but they can not because the tracking information is not available in an appropriate way. Without CPE PR and this pURL PR, people need to continue reading various channels / mailing lists / ... to have CVE on their radar, instead of falling back to automated processes.
Requesting CPE is even a formal process, not every software vendor / developer would like to go this way. The universal pURL identifier has benefits which may solve this "formal gap" problem, but different story and the two standards can get combined to achieve a maximum of coverage.....

I reverted the old PR to avoid further fallout, fixed the issue and added a feature flag for further surprises.
@mweinelt do you want to reduce the work for the security team, by enabling others to track vulnerabilities? CVE schema with PURL information has been released: https://github.com/CVEProject/cve-schema/releases/tag/v5.2.0

[jficz]

Unfortunate but understandable. Having meta hard dependent on src is imho a no-go, for various reasons described elsewhere. On the other hand, some kind of SBOM is needed if we want Nix(OS) to be taken seriously by certain businesses. For me PURL is just a way to achieve SBOM, not necessarily the only way.

While I support having PURL as (imho) the most reasonable choice, I think the real issue is having a SBOM in some form at least, and soon ™️

For starters, I'd be happy with just pname+version available, just to bootstrap the thing (and create, for example, an osquery table for nix packages). PURL can be the next step - as it will apparently require more testing and possibly workarounds around some nixpkgs hacks.

On the other hand, having hacks in nixpkgs is... unfortunate on its own so perhaps fight at both fronts? Work towards PURL in some reasonable form and at the same time make package maintainers fix their hacks or make them PURL-compatible. This could be done in long(is) term, let's say warnings about hacks in 26.05 and remove (still) affected packages in 26.11.

I'm pretty sure there's no way for this to land in 25.11 in any form.

[wolfgangwalther]

@h0nIg You should take a step back and stop trying to push everyone to work according to your timeline. You are being incredibly pushy. If you want others to give input to your stuff, answer their questions and be patient. This is open source, so major changes need time. I know this is sometimes frustrating, we have all been there - but it's reality.

[mweinelt]

@mweinelt do you want to reduce the work for the security team, by enabling others to track vulnerabilities? CVE schema with PURL information has been released: CVEProject/[email protected] (release)

And please don't try to make this about me. Yes, I want those things, but we need to deal with the concerns other committers have.

[leona-ya]

This is a way to complicated PR to rush before branch-off. We (@jopejoe1 and I) won't accept this for 25.11, we don't want more breakage. Please wait for branch-off at least.