build system: use thread-safe stdio #21438
Conversation
github-actions
bot
added
Platform: MSP
crasbe
added
CI: ready for build
maribu
force-pushed
the
sys/newlib/fix-stdio
branch
from
9a524a2 to
f058796
Compare
crasbe
added
CI: full build
|
We should probably give this a full build without fast fail, otherwise you'll be chasing your tail for days 😅 |
maribu
added
State: waiting for other PR
No, actually the memory requirements will go down with this. Unless... both printf variants get linked in into the same app. See #21439 for a fix of that. |
maribu
force-pushed
the
sys/newlib/fix-stdio
branch
from
f058796 to
31f945b
Compare
|
in one of your commit messages and in the description of this PR you have every misspelled.
|
maribu
force-pushed
the
sys/newlib/fix-stdio
branch
2 times, most recently
from
473d6ca to
933acf3
Compare
No, its IMO not noise. A test failing with default configuration is IMO a genuine bug :)
I did both. There might be users with tiny boards that can benefit from easily being able to disable |
mguetschow
left a comment
mguetschow
left a comment
There was a problem hiding this comment.
Thanks, I can confirm the test is passing now for both configurations on the hardware I tested with (nrf52840dk).
|
Please squash :) |
- `arch_64bit` depends on long long support (rather than just enabling it by default), as otherwise `%p` will not work correctly ==> add hard dependency - On 32 bit platforms support of printing long long is not almost free ==> do not enable long long on `arch_32bit` by default - The ESP SDK contains binary blobs that already link against newlib and mpaland-printf is not ABI compatible with newlib's stdio, it is only API compatible. ==> mark mpaland-printf as incompatible to ESP MCUs Co-authored-by: crasbe <[email protected]>
This introduces a new feature category to declare which bugs are present in a given build that we cannot fix in RIOT, but need to work around. These bugs may be silicon bugs, software bugs in ROM, software bugs in binary blobs needed for a platform, or bugs in the toolchain. These features behave the same way as `arch_%` features: Every provided bug is always used, so that we can inspect `FEATURES_USED` to check which bugs need to be worked around. Co-authored-by: crasbe <[email protected]>
This makes use of the new bug modeling to declare all platforms that can use newlib and have no reentrancy hooks as affected by the non-thread-safe stdio bug. (Which is every platform but ESP* and AVR, the former because the reentrancy hooks are provided, the latter because we do not and never will support newlib on them.) Building on that, the mpaland-printf package is used when newlib is used and the bug is present. This way we can rely on the stdio being thread-safe on every platform and not causing random crashes at run time. Co-authored-by: mguetschow <[email protected]>
This drops a workaround that initialized newlib's reentrancy structure on boot to reduce the chances of crashes when using the non-thread-safe (unless reentrancy hooks are provided) stdio implementation of newlib. Now that the newlib stdio implementation is only ever used if it is thread-safe, we no longer need a workaround that reduces the chance of crashes on concurrent use of stdio.
That way users can rely on elimation of dead code from the optimizer instead of having to use preprocessor conditionals when feature-gating assert implementations behind `!NDEBUG`. Co-authored-by: benpicco <[email protected]> Co-authored-by: crasbe <[email protected]> Co-authored-by: mguetschow <[email protected]>
This avoids inconsistent output when external code gets linked in that directly links against newlib's assert implementation (e.g. binary blobs or packages that do not add `core/lib/include` to the include paths). This also greatly benefits wrapping printf, as newlib's `__assert_func()` directly links to internal printf functions of newlib. Co-authored-by: crasbe <[email protected]>
This is required to support printing 64 bit numbers, as is done in the test. In addition that 64 bit formatting test is feature gated, so that users can simply comment out the `USEMODULE += printf_long_long` to fit tiny boards. Co-authored-by: mguetschow <[email protected]>
maribu
force-pushed
the
sys/newlib/fix-stdio
branch
from
ecdd4ea to
4a84f6f
Compare
|
Yay 🎉 Thanks everyone for bearing with me on this one ❤️ |
|
This resulted in some failing nightly tests, as mpaland-printf prints As the printing format is indeed implementation-defined, the respective tests should probably be adapted to accept output both with and without |
|
Another problem, at least on my machine: RIOT/pkg/mpaland-printf/Makefile.include Lines 16 to 25 in c0cc617 suggests that this is failing on purpose, but I'm not sure what to make out of this. Apparently that also didn't happen on CI for some reason (maybe it was not properly rebuilding all modules for the test?). |
There we go: #21677 |
|
Hey, The issue immediatly goes away when disabling |
|
Yes, |
|
@maribu do you have a rough idea how to approach this issue? I feel a bit lost to find a starting point. |
|
The basic idea would be to add buffering for outputs that have significant overhead. Newlib does this, but uses a rather large on-stack buffer (which is why we have the I can take look into it, but I cannot promise an ETA due to high work load. |
7 participants
Contribution description
build system: introduce
bug_%feature categoryThis introduces a new feature category to declare which bugs are present
in a given build that we cannot fix in RIOT, but need to work around.
These bugs may be silicon bugs, software bugs in ROM, software bugs in
binary blobs needed for a platform, or bugs in the toolchain.
These features behave the same way as
arch_%features: Every providedbug is always used, so that we can inspect
FEATURES_USEDto checkwhich bugs need to be worked around.
build system: use thread-safe stdio
This makes use of the new bug modeling to declare all platforms that
can use newlib and have no reentrancy hooks as affected by the
non-thread-safe stdio bug. (Which is every platform but ESP* and AVR,
the former because the reentrancy hooks are provided, the latter because
we do not and never will support newlib on them.)
Building on that, the mpaland-printf package is used when newlib is used
and the bug is present. This way we can rely on the stdio being
thread-safe on every platform and not causing random crashes at
run time.
sys/newlib: drop workaround for stdio
This drops a workaround that initialized newlib's reentrancy structure
on boot to reduce the chances of crashes when using the non-thread-safe
(unless reentrancy hooks are provided) stdio implementation of newlib.
Now that the newlib stdio implementation is only ever used if it is
thread-safe, we no longer need a workaround that reduces the chance
of crashes on concurrent use of stdio.
Testing procedure
Now the package
mpaland-printfshould be pulled in for all non-ESP boards when newlib is used. E.g.With this, the test in
tests/sys/snprintfnow should also pass for each and every board, except for ESP32 boards. The reason is that, since ESP* provides the reentrancy hooks, newlib's stdio is thread-safe on those. So we do not pull in the alternative printf. For ESP8266 boards newlib is configured to use the non-nano stdio, which passes the test. For ESP32 newlib-nano is used, which does not implement all format specifiers and, therefore, fails the test.Issues/PRs references
Depends on and includes: