Speed up package building time

[deeplow]

Building packages with make build-debs current takes me several minutes. What can we do to reduce the package build time?

[deeplow]

Just for future reference building individual packages, was done in the past and would be a regression in some sense. So the issue is to track improvements speed up the whole build process of debs.

[legoktm]

One thing that I don't think is documented/advertised is that if you do FAST=1 make build-debs it skips the lintian step, which probably saves a minute or two from the overall build time. I have a few ideas on how to make that a little faster too, first of which is caching the Rust build step.

Note that try-client-pr already uses FAST=1, so any improvements here will also benefit that.

[legoktm]

the dh_builddeb step takes the already built files and compresses them into .debs. By default it uses xz, we can get ~40% faster by switching to gzip with not that much larger files. If we turn off compression entirely it gets as close to instant as possible:

Package	xz	gzip	none
securedrop-app	102M	149M	463M
securedrop-client	3.8M	5.8M	21M
securedrop-export	1.6M	2.7M	11M
securedrop-log	1.6M	2.6M	11M
securedrop-proxy	1021K	1.4M	3.3M
Time	188s	115s	4s

At the cost of securedrop-app being nearly half a gig. I think for the try-client-pr usecase this'll be worth it because the extra time spent shipping the larger files around will still be faster than the compression (at build time) + decompression (at install time) that needs to happen.

[deeplow]

This is great! Turning it off is a really massive difference. I think it would be worth exploring.

As far as the app is concerned, I wonder if there was any ways to split out the chromium part into a package (that doesn't always get built).

[legoktm]

I had Claude add timing markers to the various steps; with the Rust + compression now zeroed out, the app is 70% of the build time. Presumably there's more caching we can add there too.

Step	Time	% of total
sed version	0s	0%
setup-venv client	13s	6%
setup-venv export	10s	5%
setup-venv log	9s	4%
cargo build	4s	2%
cargo copy (FAST)	0s	0%
build-app.sh total	152s	70%
↳ npm install pnpm	2s	1%
↳ pnpm install	39s	18%
↳ pnpm build:unpack	111s	51%
dh_auto_install	0s	0%
dh_apparmor	0s	0%
dh_install	2s	1%
dh_install cleanup	0s	0%
dh_installsystemd	0s	0%
find/delete pyc etc	0s	0%
dh_strip_nondeterminism	21s	10%
dh_fixperms	0s	0%
chmod sandbox	0s	0%
dh_strip	2s	1%
dh_shlibdeps	2s	1%
dh_installdeb	0s	0%
conffiles cleanup	0s	0%
dh_builddeb (compression=none)	3s	1%
Total	218s	100%

[deeplow]

If we end up dropping the venv entirely as we're discussing in #3104 (comment)) that would be ~15% less time.

Next up would be dh_strip_nondeterminism. I don't the problem here is the overrides (mostly find commands), but I'm guessing we don't really want to mess with optimizing this stuff. But we could perhaps exploring disabling reproducible builds at least for local development.

[deeplow]

build-app.sh total 152s 70%

Given this percentage, I'd like to make the case again for the ability not build the app. I don't remember if you were opposed to this or if it was if about the ability to build the packages individually. But the app has a big impact on build time.

Since the app was made a default part of the build, I've been patching debian/rules, because I've rarely touch(ed) the app. But it's a bit awkward it would be quite useful not to wait for the app to build (using your metrics: 1m vs. 3.5m).

[legoktm]

Given this percentage, I'd re-iterate my concern that it would be important to be able to not build the app. I don't remember if you were opposed to this or if it was if we built them separately.

Well, this is with the Rust + compression zeroed out so it looks worse, but I get your point. If the app is 70% of build time then yeah, we need to re-introduce a way to skip it or something.

But I think we're doing something wrong, it shouldn't be this slow. Poking at that now...

(And I suspect the things we're doing wrong is including too many files, which is what's inflating the time spent in dh_strip_nondeterminism + compression as knock-on effects...)