pkg/{c25519,driver_cryptocell_310}: use dlcache for fetching zip archives, update `pkg/Makefile.http` example #21813

crasbe · 2025-10-22T12:00:27Z

Contribution description

In the past we've been causing A LOT of traffic on the Nordic website by continually fetching the nRF5_SDK for every build cycle where the Cryptocell driver is built. After they kicked us out (although I don't know if that's actually because of us), we changed to a community hosted server in #21635.

This still causes a lot of unnecessary traffic and skyleaf is ~160GB rx traffic each month for the CI.

I accidentally discovered that we have a script for caching http(s) downloads, which is only used by one package so far.
Initially I wanted to create a new pkg/pkg_http.mk for that purpose, but that became too frustrating tbh.

Edit: Further changes:

The dlcache script is now used in place of the old selection of wget and curl in the $(RIOTBASE)/Makefile.include.
The dlcache script now uses SHA512 instead of MD5.
The order of the arguments of dlcache was changed to be backwards compatible with $(DOWNLOAD_TO_FILE).

Testing procedure

I'm not sure how to test it in the CI, but locally you have to do the following steps:

cbuec@W11nMate:~/RIOTstuff/riot-guides/RIOT$ mkdir -p ~/.dlcache
cbuec@W11nMate:~/RIOTstuff/riot-guides/RIOT$ BOARD=nrf52840dk make -C tests/sys/psa_crypto
make: Entering directory '/home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto'
 You are going to use the PSA Crypto module, which is only partly implemented and not yet thouroughly tested.
 Please do not use this module in production, as it may introduce security issues!
Building application "tests_psa_crypto" for "nrf52840dk" with CPU "nrf52".

mkdir -p /home/cbuec/RIOTstuff/riot-guides/RIOT/build/pkg
/home/cbuec/RIOTstuff/riot-guides/RIOT/dist/tools/dlcache/dlcache.sh: downloading "https://download.riot-os.org/nRF5_SDK_17.1.0_ddde560.zip"...
2025-10-22 13:55:25 URL:https://download.riot-os.org/nRF5_SDK_17.1.0_ddde560.zip [131838843/131838843] -> "/home/cbuec/RIOTstuff/riot-guides/RIOT/.dlcache/nRF5_SDK_17.1.0_ddde560.zip" [1]
/home/cbuec/RIOTstuff/riot-guides/RIOT/dist/tools/dlcache/dlcache.sh: done downloading "https://download.riot-os.org/nRF5_SDK_17.1.0_ddde560.zip"
test "cb610d724d27ef79c792511d8b889cbf2d411a3a391f5c740ea10b86139201b97eedf0c1edcf4d199c3fbd51e4cc5a4bd922c0c40d01e2f5359acd16a2b163e4  /home/cbuec/RIOTstuff/riot-guides/RIOT/build/pkg/nRF5_SDK_17.1.0_ddde560.zip" =  "$(sha512sum "/home/cbuec/RIOTstuff/riot-guides/RIOT/build/pkg/nRF5_SDK_17.1.0_ddde560.zip")"
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/pkg/cmsis/
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/pkg/driver_cryptocell_310/
...
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/sys/tsrb
   text    data     bss     dec     hex filename
  44612     136   15928   60676    ed04 /home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto/bin/nrf52840dk/tests_psa_crypto.elf
make: Leaving directory '/home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto'

On the second build, it doesn't fetch anything as expected:

cbuec@W11nMate:~/RIOTstuff/riot-guides/RIOT$ BOARD=nrf52840dk make -C tests/sys/psa_crypto
make: Entering directory '/home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto'
 You are going to use the PSA Crypto module, which is only partly implemented and not yet thouroughly tested.
 Please do not use this module in production, as it may introduce security issues!
Building application "tests_psa_crypto" for "nrf52840dk" with CPU "nrf52".

make[1]: Nothing to be done for 'prepare'.
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/pkg/cmsis/
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/pkg/driver_cryptocell_310/
...
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/sys/tsrb
   text    data     bss     dec     hex filename
  44612     136   15928   60676    ed04 /home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto/bin/nrf52840dk/tests_psa_crypto.elf
make: Leaving directory '/home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto'

If we delete the files and folders in build/ we can observe how dlcache takes the file from cache:

cbuec@W11nMate:~/RIOTstuff/riot-guides/RIOT$ BOARD=nrf52840dk make -C tests/sys/psa_crypto
make: Entering directory '/home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto'
 You are going to use the PSA Crypto module, which is only partly implemented and not yet thouroughly tested.
 Please do not use this module in production, as it may introduce security issues!
Building application "tests_psa_crypto" for "nrf52840dk" with CPU "nrf52".

mkdir -p /home/cbuec/RIOTstuff/riot-guides/RIOT/build/pkg
/home/cbuec/RIOTstuff/riot-guides/RIOT/dist/tools/dlcache/dlcache.sh: getting "https://download.riot-os.org/nRF5_SDK_17.1.0_ddde560.zip" from cache
test "cb610d724d27ef79c792511d8b889cbf2d411a3a391f5c740ea10b86139201b97eedf0c1edcf4d199c3fbd51e4cc5a4bd922c0c40d01e2f5359acd16a2b163e4  /home/cbuec/RIOTstuff/riot-guides/RIOT/build/pkg/nRF5_SDK_17.1.0_ddde560.zip" =  "$(sha512sum "/home/cbuec/RIOTstuff/riot-guides/RIOT/build/pkg/nRF5_SDK_17.1.0_ddde560.zip")"
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/pkg/cmsis/
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/pkg/driver_cryptocell_310/
...
"make" -C /home/cbuec/RIOTstuff/riot-guides/RIOT/sys/tsrb
   text    data     bss     dec     hex filename
  44612     136   15928   60676    ed04 /home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto/bin/nrf52840dk/tests_psa_crypto.elf
make: Leaving directory '/home/cbuec/RIOTstuff/riot-guides/RIOT/tests/sys/psa_crypto'

The same test procedure applies to pkg/c25519 with the tests/pkg/c25519 test.

Issues/PRs references

~~Depends on RIOT-OS/riotdocker#262 .~~ merged.

riot-ci · 2025-10-22T12:09:38Z

Murdock results

✔️ PASSED

a30ce22 pkg/c25519: use dlcache for fetching zip archive

Success	Failures	Total	Runtime
10552	0	10552	11m:12s

Artifacts

Documentation preview

mguetschow

Thanks a lot for tackling this! Always impressive to find unexpected tools in RIOT, but actually rather unimpressive to not find them documented anywhere.

I'd have expected documentation about it on https://doc.riot-os.org/group__pkg.html. I then found there is a https://github.com/RIOT-OS/RIOT/blob/master/pkg/Makefile.http which is supposed to serve as a template. We should at least mention it there - or even implement $(DOWNLOAD_TO_FILE) with $(DLCACHE) under the hood. What do you think?

crasbe · 2025-10-22T12:37:46Z

or even implement $(DOWNLOAD_TO_FILE) with $(DLCACHE) under the hood.

Or we deprecate $(DOWNLOAD_TO_FILE) altogether and migrate the only two packages using that and update the example Makefile.http (and documentation) 🤔

crasbe · 2025-10-22T12:45:55Z

I'd have expected documentation about it on https://doc.riot-os.org/group__pkg.html.

Do you have a proposal what I should add to that documentation?

mguetschow · 2025-10-22T14:09:16Z

or even implement $(DOWNLOAD_TO_FILE) with $(DLCACHE) under the hood.

Or we deprecate $(DOWNLOAD_TO_FILE) altogether and migrate the only two packages using that and update the example Makefile.http (and documentation) 🤔

Which is basically the same, but $(DOWNLOAD_TO_FILE) is the more speaking name, so I'd rather have that one translate to the $(DLCACHE) magic.

mguetschow · 2025-10-22T14:09:44Z

I'd have expected documentation about it on doc.riot-os.org/group__pkg.html.

Do you have a proposal what I should add to that documentation?

No, I think it is fine if it is used in Makefile.http

crasbe · 2025-10-22T14:37:07Z

or even implement $(DOWNLOAD_TO_FILE) with $(DLCACHE) under the hood.

Or we deprecate $(DOWNLOAD_TO_FILE) altogether and migrate the only two packages using that and update the example Makefile.http (and documentation) 🤔

Which is basically the same, but $(DOWNLOAD_TO_FILE) is the more speaking name, so I'd rather have that one translate to the $(DLCACHE) magic.

Done. I would also like to extend the README.md of the dlcache too, since I changed the syntax of dlcache to make it more compatible with the DOWNLOAD_TO_FILE command.

mguetschow

LGTM now, please squash!

mguetschow · 2025-10-22T14:39:22Z

I would also like to extend the README.md of the dlcache too

Oh sure, feel free to do so!

maribu · 2025-10-22T16:54:32Z

pkg/c25519/Makefile

 PKG_EXT     = zip
 PKG_LICENSE = PD
 PKG_SHA512  = dbfb4285837ab2ea3d99c448b22877cc7a139ccbaebb1de367e2bec1fd562fe629b389d86603915448078b8fd7e631c8fc9a7d126eb889a1ba0c17611369b190
+PKG_MD5     = 2f19396f8becb44fe1cd5e40111e3ffb


The collision resistance of MD5 is really really poor. Especially when the transport is HTTP, there would be a real security benefit if we would compare the downloaded file against a SHA512 or similar.

In addition to that: I think md5sum is less likely to be part of a default installation than sha512sum these days.

Okay, that only leaves one issue: I can't calculate the new checksum of the binary linked in mcuboot, because the link is dead:

RIOT/makefiles/mcuboot.mk

Lines 10 to 11 in 5ff98db

MCUBOOT_BIN_URL ?= http://download.riot-os.org/mynewt.mcuboot.bin

MCUBOOT_BIN_MD5 ?= 0c71a0589bd3709fc2d90f07a0035ce7

This wouldn't hold up this PR, because the dlcache script will always (try to) redownload if no checksum is given. The fact that there is nothing to download is something else 😅

The dlcache script is useful to avoid unnecessary repeated downloads of zip archives, similar to the git-cache scripts. The command order was changed to stay compatible with DOWNLOAD_TO_FILE and the only other usage of $(DLCACHE) was changed accordingly. To avoid the risk of collisions, dlcache now uses SHA512 sums instead of MD5. If no SHA512 checksum is given to dlcache.sh, it will just act as $(DOWNLOAD_TO_FILE).

crasbe · 2025-10-24T15:42:32Z

It appears that the change worked 🥳

maribu · 2025-10-24T16:09:01Z

Thx for spotting and addressing the issue ❤️ Our CI really has been a naughty before with causing so much pointless traffic 😅

crasbe requested a review from mguetschow October 22, 2025 12:00

github-actions bot added the Area: pkg Area: External package ports label Oct 22, 2025

crasbe mentioned this pull request Oct 22, 2025

Dockerfile: set folder for dlcache RIOT-OS/riotdocker#262

Merged

mguetschow reviewed Oct 22, 2025

View reviewed changes

crasbe changed the title ~~pkg/driver_cryptocell_310: use dlcache for fetching nRF5_SDK~~ pkg/{c25519,driver_cryptocell_310}: use dlcache for fetching zip archives, update pkg/Makefile.http example Oct 22, 2025

github-actions bot added Area: build system Area: Build system Area: tools Area: Supplementary tools labels Oct 22, 2025

mguetschow approved these changes Oct 22, 2025

View reviewed changes

crasbe force-pushed the pr/cryptocell_use_dlcache branch from 4fe8c18 to 5054cfb Compare October 22, 2025 14:47

crasbe requested a review from jia200x as a code owner October 22, 2025 14:47

github-actions bot added the Area: doc Area: Documentation label Oct 22, 2025

crasbe force-pushed the pr/cryptocell_use_dlcache branch from 5054cfb to c2cdbb5 Compare October 22, 2025 14:48

maribu reviewed Oct 22, 2025

View reviewed changes

crasbe force-pushed the pr/cryptocell_use_dlcache branch from c2cdbb5 to 1cb09eb Compare October 22, 2025 17:21

crasbe removed the State: waiting for CI update State: The PR requires an Update to CI to be performed first label Oct 23, 2025

maribu approved these changes Oct 23, 2025

View reviewed changes

crasbe force-pushed the pr/cryptocell_use_dlcache branch from 6764a30 to 54c0738 Compare October 23, 2025 10:01

crasbe added 3 commits October 23, 2025 21:54

dist/tools: fix shellcheck warnings in dlcache script

7a8ade7

pkg/driver_cryptocell_310: use dlcache for fetching nRF5_SDK

e0786a2

pkg/c25519: use dlcache for fetching zip archive

a30ce22

crasbe force-pushed the pr/cryptocell_use_dlcache branch from 54c0738 to a30ce22 Compare October 23, 2025 19:54

crasbe enabled auto-merge October 23, 2025 19:55

crasbe added this pull request to the merge queue Oct 23, 2025

Merged via the queue into RIOT-OS:master with commit 89cd3db Oct 23, 2025
25 checks passed

crasbe deleted the pr/cryptocell_use_dlcache branch October 24, 2025 15:41

crasbe mentioned this pull request Oct 29, 2025

makefiles: remove DOWNLOAD_TO_STDOUT #21823

Merged

benpicco added this to the Release 2025.10 milestone Dec 3, 2025

	MCUBOOT_BIN_URL ?= http://download.riot-os.org/mynewt.mcuboot.bin
	MCUBOOT_BIN_MD5 ?= 0c71a0589bd3709fc2d90f07a0035ce7

pkg/{c25519,driver_cryptocell_310}: use dlcache for fetching zip archives, update pkg/Makefile.http example #21813

pkg/{c25519,driver_cryptocell_310}: use dlcache for fetching zip archives, update pkg/Makefile.http example #21813

Uh oh!

Conversation

crasbe commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

riot-ci commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Murdock results

Artifacts

Uh oh!

mguetschow left a comment

Choose a reason for hiding this comment

Uh oh!

crasbe commented Oct 22, 2025

Uh oh!

crasbe commented Oct 22, 2025

Uh oh!

mguetschow commented Oct 22, 2025

Uh oh!

mguetschow commented Oct 22, 2025

Uh oh!

crasbe commented Oct 22, 2025

Uh oh!

mguetschow left a comment

Choose a reason for hiding this comment

Uh oh!

mguetschow commented Oct 22, 2025

Uh oh!

maribu Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

crasbe Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

crasbe Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

crasbe commented Oct 24, 2025

Uh oh!

maribu commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pkg/{c25519,driver_cryptocell_310}: use dlcache for fetching zip archives, update `pkg/Makefile.http` example #21813

pkg/{c25519,driver_cryptocell_310}: use dlcache for fetching zip archives, update `pkg/Makefile.http` example #21813

crasbe commented Oct 22, 2025 •

edited

Loading

riot-ci commented Oct 22, 2025 •

edited

Loading