Skip to content

Conversation

@YorikSar
Copy link
Contributor

@YorikSar YorikSar commented Aug 1, 2025

This operator merges two attrsets without any conditions, and leads to more operations and allocations.

Extract from the Eval / compare job summary:

metric mean_before mean_after mean_diff mean_%_change p_value t_stat
cpuTime 36.8319 35.9723 -0.8596 -2.0140 0.0002 -4.0792
gc.cycles 10.0455 10.0227 -0.0227 -0.1623 0.3229 -1.0000
gc.heapSize 3258545524.3636 3256257629.0909 -2287895.2727 -0.1905 0.6364 -0.4762
gc.totalBytes 5852793058.9091 5839663730.9091 -13129328.0000 -0.2349 0.0000 -12.9536
nrExprs 1735737.0455 1735735.0455 -2.0000 -0.0001 - -inf
nrOpUpdateValuesCopied 100875413.1591 100260153.1136 -615260.0455 -0.6539 0.0000 -12.9458
nrOpUpdates 4137372.5909 4034829.2500 -102543.3409 -2.4710 0.0000 -12.9458
sets.bytes 2241343462.5455 2228217914.9091 -13125547.6364 -0.6294 0.0000 -12.9458
sets.elements 132069782.6818 131454522.6364 -615260.0455 -0.5011 0.0000 -12.9458
sets.number 8014183.7273 7809097.0455 -205086.6818 -2.7824 0.0000 -12.9458
time.cpu 36.8319 35.9723 -0.8596 -2.0140 0.0002 -4.0792
time.gc 7.1009 6.7456 -0.3553 -3.1433 0.0086 -2.7559
time.gcFraction 0.2112 0.2084 -0.0028 -1.2720 0.1663 -1.4081

All these metrics went down, all other metrics didn't change.

Especially note -2% of CPU and -3% of GC time, -2.7% of sets and -2.4% of update ops.

Things done


Add a 👍 reaction to pull requests you find important.

This operator merges two attrsets without any conditions, and leads to
more operations and allocations.
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. 6.topic: stdenv Standard environment labels Aug 1, 2025
@YorikSar YorikSar requested a review from infinisil August 1, 2025 14:04
@YorikSar
Copy link
Contributor Author

YorikSar commented Aug 1, 2025

I've decided to remove all merges that don't involve user input, and filter resulting attributes in one removeAttrs call. It seems to save even more time and memory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused; you made use of builtins.removeAttrs below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, change got stuck in my stage. Fixed now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be good to merge once this is removed again as it is unused 😁

@github-project-automation github-project-automation bot moved this to In Progress in Stdenv Aug 1, 2025
@YorikSar YorikSar force-pushed the remove-extra-merge-in-meta branch 2 times, most recently from 803adcd to 9b7148e Compare August 1, 2025 16:31
@YorikSar
Copy link
Contributor Author

YorikSar commented Aug 1, 2025

The version with removeAttrs (9b7148e) shows less obvious results:

metric mean_before mean_after mean_diff mean_%_change p_value t_stat
cpuTime 36.5088 36.7072 0.1984 0.7826 0.1592 1.4328
envs.bytes 715033255.2727 709286205.6364 -5747049.6364 -0.8353 0.0000 -12.9357
envs.elements 52285609.2045 51977731.5455 -307877.6591 -0.6097 0.0000 -12.9357
envs.number 37093547.7045 36683044.1591 -410503.5455 -1.1566 0.0000 -12.9357
gc.cycles 10.0455 9.9773 -0.0682 -0.5519 0.0832 -1.7738
gc.heapSize 3259296023.2727 3295138350.5455 35842327.2727 0.8550 0.0477 2.0379
gc.totalBytes 5857118407.2727 5827590268.3636 -29528138.9091 -0.5178 0.0000 -12.4822
list.bytes 98129817.0909 98672152.3636 542335.2727 0.6094 0.0000 12.3237
list.concats 2116585.6818 2116585.6818 - - - -
list.elements 12266227.1364 12334019.0455 67791.9091 0.6094 0.0000 12.3237
nrAvoided 40427910.9545 40425751.5909 -2159.3636 -0.0050 0.0000 -9.3357
nrExprs 1735866.3409 1735872.3409 6.0000 0.0004 - inf
nrFunctionCalls 33717152.7955 33306649.2500 -410503.5455 -1.2664 0.0000 -12.9357
nrLookups 16954029.3182 16949711.4091 -4317.9091 -0.0245 0.0000 -9.3381
nrOpUpdateValuesCopied 100960423.6364 99297336.2955 -1663087.3409 -1.7353 0.0000 -12.7676
nrOpUpdates 4140731.1591 3832853.5000 -307877.6591 -7.4130 0.0000 -12.9357
nrPrimOpCalls 18336841.4773 18129430.3409 -207411.1364 -1.1623 0.0000 -12.8954
nrThunks 46048648.2955 45946131.2273 -102517.0682 -0.2324 0.0000 -12.9093
sets.bytes 2243200672.7273 2227291651.2727 -15909021.4545 -0.7319 0.0000 -12.0810
sets.elements 132180096.3182 131733906.5227 -446189.7955 -0.3330 0.0000 -10.7500
sets.number 8019945.7273 7471821.6818 -548124.0455 -7.3960 0.0000 -12.9389
time.cpu 36.5088 36.7072 0.1984 0.7826 0.1592 1.4328
time.gc 6.9330 6.9840 0.0510 1.7151 0.7303 0.3470
time.gcFraction 0.2089 0.2111 0.0022 0.8310 0.3301 0.9852
values.bytes 1566694455.2727 1559100704.1818 -7593751.0909 -0.5276 0.0000 -12.8215
values.number 65278935.6364 64962529.3409 -316406.2955 -0.5276 0.0000 -12.8215

All time metrics are worse, and we trade ~1 MiB of attrsets for ~0.5 MiB of lists.

@YorikSar YorikSar force-pushed the remove-extra-merge-in-meta branch from 9b7148e to b679ba6 Compare August 1, 2025 21:02
@YorikSar
Copy link
Contributor Author

YorikSar commented Aug 1, 2025

With null attr names trick metrics seem even better, thanks @philiptaron! Everything went down:

metric mean_before mean_after mean_diff mean_%_change p_value t_stat
cpuTime 36.5772 36.1772 -0.4000 -1.1961 0.0104 -2.6803
envs.bytes 715021494.1818 700243502.9091 -14777991.2727 -2.1479 0.0000 -12.9359
envs.elements 52284899.6364 51463900.2727 -820999.3636 -1.6257 0.0000 -12.9359
envs.number 37092787.1364 36066537.5909 -1026249.5455 -2.8916 0.0000 -12.9359
gc.cycles 10.0455 9.8636 -0.1818 -1.5097 0.0035 -3.0912
gc.heapSize 3260070539.6364 3259307566.5455 -762973.0909 -0.1608 0.9680 -0.0403
gc.totalBytes 5856766134.1818 5762963597.8182 -93802536.3636 -1.6709 0.0000 -12.9269
list.bytes 98128421.0909 96207752.9091 -1920668.1818 -2.0652 0.0000 -12.9288
list.concats 2116589.6364 2116589.6364 - - - -
list.elements 12266052.6364 12025969.1136 -240083.5227 -2.0652 0.0000 -12.9288
nrAvoided 40427450.9091 40056859.4318 -370591.4773 -0.9515 0.0000 -12.9328
nrExprs 1735897.0682 1735899.0682 2.0000 0.0001 - inf
nrFunctionCalls 33716392.7955 32690143.2500 -1026249.5455 -3.1660 0.0000 -12.9359
nrLookups 16953672.1136 16747830.2045 -205841.9091 -1.3110 0.0000 -12.9305
nrOpUpdateValuesCopied 100939779.3182 99109108.7500 -1830670.5682 -1.9160 0.0000 -12.8127
nrOpUpdates 4140705.4091 3832830.9545 -307874.4545 -7.4130 0.0000 -12.9359
nrPrimOpCalls 18336735.2727 18028860.0000 -307875.2727 -1.7270 0.0000 -12.9358
nrThunks 46048425.1818 45124800.1818 -923625.0000 -2.0943 0.0000 -12.9359
sets.bytes 2242863066.9091 2199947293.8182 -42915773.0909 -2.0340 0.0000 -12.8765
sets.elements 132159045.4318 130125393.4091 -2033652.0227 -1.6330 0.0000 -12.8386
sets.number 8019896.2500 7371312.4545 -648583.7955 -8.7617 0.0000 -12.9523
time.cpu 36.5772 36.1772 -0.4000 -1.1961 0.0104 -2.6803
time.gc 6.9942 6.8285 -0.1657 -1.4431 0.2370 -1.1992
time.gcFraction 0.2115 0.2095 -0.0020 -0.4066 0.4600 -0.7456
values.bytes 1566682046.1818 1539589041.8182 -27093004.3636 -1.8893 0.0000 -12.9359
values.number 65278418.5909 64149543.4091 -1128875.1818 -1.8893 0.0000 -12.9359

@YorikSar YorikSar requested a review from philiptaron August 1, 2025 21:15
Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One trivial change then let's merge.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be good to merge once this is removed again as it is unused 😁

Remove usage of filterAttrs and two more attrset merges. Saves more
time and memory.
@YorikSar YorikSar force-pushed the remove-extra-merge-in-meta branch from b679ba6 to 60d2cc0 Compare August 4, 2025 14:21
@YorikSar
Copy link
Contributor Author

YorikSar commented Aug 4, 2025

@philiptaron Ah, it slipped through again. Removed.

@YorikSar
Copy link
Contributor Author

YorikSar commented Aug 4, 2025

What if we do the same for mkDerivation?.. #430969

@philiptaron philiptaron merged commit 85b1221 into NixOS:master Aug 4, 2025
24 of 27 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Stdenv Aug 4, 2025
@YorikSar YorikSar deleted the remove-extra-merge-in-meta branch August 5, 2025 12:49
YorikSar added a commit to tweag/nixpkgs that referenced this pull request Aug 7, 2025
As in NixOS#430132, it saves a lot of
set allocations and merge operations, but makes code harder to read.

I've tried introducing a function like this to make code cleaner, but it
increases number of envs and space taken by them significantly:

  optionalAttr = cond: name: if cond then name else null;

I've also tried applying this logic to the section with isDarwin, but
it makes things measurably worse for x86_64-linux eval.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: stdenv Standard environment 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants