sys/net/*coap: Make APIs (more) transport agnostic

[maribu]

Contribution description

This changes the API of nanocoap with the goal to reduce the expose of UDP specifics in the API. The plan is to eventually support transports such as CoAP over TCP and CoAP over WebSocket directly in nanocoap while sharing most of the code, as e.g. the CoAP Option processing remains identical. Specifically, the plan is to unlock a transport with modules and introduce overhead for dispatching to specific transport only when multiple transports are actually in use.

Support for OSCORE directly in nanocoap is probably not sensible, as the serialization is very much unlike the other transports. A unified CoAP API for multiple transports including OSCORE is probably best implemented on top. But when limited to the boring set of CoAP transports, we probably can support them well with nanocoap with less overhead.

API Changes:

Breaking API Changes:

• coap_parse() now returns ssize_t instead of int
  • This function is not really user facing, so the impact should be limited
  • This is useful for stream transports where the buffer may contain data of more than one packet. The return value contains the number of bytes actually consumed, which will match the buffer size for non-stream transports.

API Changes:

• coap_pkt_t now contains a uint8_t *buf pointer instead of a coap_hdr_t *hdr pointer to the beginning of the buffer
  • This will also work when the buffer is used by non-UDP transports
  • A deprecated coap_udp_hdr_t *hdr has been crammed into an unnamed union with uint8_t *buf. For architectures where pointers have the same memory layout regardless of type (e.g. all of the supported ones), this will make hdr an alias for buf.
  • The alias will only be provided if no transport besides UDP is used in nanocoap. So existing apps will continue to work, new apps that want to support other transports need to move to adapt.
• coap_hdr_t has been renamed to coap_udp_hdr_t
  • A deprecated alias was created for deprecation
• coap_hdr*() functions have been deprecated
  • Equivalent coap_pkt*() functions have been created that work on coap_pkt_t * instead of coap_hdr_t *
  • If non-UDP transports are used, the deprecated coap_hdr*() will probably not be exposed to avoid footguns.
• coap_build_hdr() has been renamed to coap_build_udp_hdr() and that works on an uint8_t * buffer with a given length, rather than on a coap_hdr_t * with a figers crossed length
  • a deprecated coap_build_hdr() function was added that calls to coap_build_udp_hdr() and has the same signature, so that users have time to update

Internal users have been updated, except for tests.

The deprecated functions will probably exposed only when nanocoap is used with only UDP as transport, as those will wreak havoc on non-UDP CoAP packets. That should be fine, as existing apps hardly will make use of other transports anyway.

Testing procedure

Existing apps and tests should work as before. I will do some limited testing soon and do more rigorous testing once an implementation of nanocoap using TCP as transport is ready. Otherwise there are likely still UDP specifics in the API that I have overlooked.

Issues/PRs references

See also #20792

[maribu]

This PR is not ready for review yet, I just want to make the work more transparent.

[mguetschow]

Cross-referencing @carl-tud's ongoing work with a superset of the goals of this one: #20792

[riot-ci]

Murdock results

✔️ PASSED

ce06c4f examples/gcoap: upgrade users of deprecated nanocoap APIs

Success	Failures	Total	Runtime
10930	0	10931	09m:31s

Artifacts

• Documentation preview

[chrysn]

Could the .hdr pointer stay there unused and deprecated if none of the non-UDP gcoap modules are enabled?

Then users who just upgrade their applications would see a deprecation rather than breakage, and would just need to follow that deprecation's pointers ("use the setters and getters") before either they go to the next release or they enable any of the new modules. The Rust wrappers (affected in RIOT-OS/rust-riot-wrappers#129) could then do the same: use the old ways unless a new module is enabled, and after the next release they can go all-in with the new accessor, and thus always work with the current release and with the mai^Wmaster branch.

[chrysn]

To clarify: If it's only the Rust wrappers affected, workarounds are possible (but necessary) to make things work there by detecting whether any code getter/setter is present. But if making it easier for riot-wrappers also eases other users' transitions, maybe we should do that. I can work with either outcome.

[maribu]

Could the .hdr pointer stay there unused and deprecated if none of the non-UDP gcoap modules are enabled?

I think adding an unnamed union containing .hdr and .buf if - and only if - only UDP is used would be a compromise. Having a .hdr when there is (possibly) no UDP header present is quite a footgun, as .hdr is tied towards a UDP header. We should also not have the union there when the CI is run, so that users of coap_pkt_t::hdr don't sneak back in.

[chrysn]

Yes, an unnamed union .buf ∪ .hdr that only has the old item if that is legal (ie. UDP is the only transport) would be good.

CI should probably not interfere here because that'd also impede the Rust side -- but after the next release, the old item can go away anyway, and then we're enforcing the deprecation. Until then I think we'll have to rely on people following the deprecation.

[benpicco]

This is indeed least invasive and from what I gather has been tested with different transports already.

[mguetschow]

We had a discussion out of band and concluded that there is a risk that !20900 changes APIs that end up being deprecated when UniCoAP lands.

We agreed to put PR on hold for now, so that we don’t end up forcing users to jump through two API deprecation hoops when this could be done with just one.

I'd say this still holds, doesn't it?

[maribu]

I think it is quite clear that unicoap and nanocoap have sufficiently different use cases in mind that a leaner CoAP implementation remains useful.

I can easily see unicoap replacing gcoap, but I don't see it replacing nanocoap.

Im any case, this rather should not be rushed into the release.

[benpicco]

This API change is actually not so benign.
Previously 0 would mean success, so code would check if (coap_parse(…)) {} as the error case.

Now we need to check if (coap_parse(…) <= 0) {}

This should at least be featured more prominently if no API compatibility is possible.

[maribu]

This is indeed the main breaking change and highlighted as such in the PR description.

I think the fallout is pretty limited, though: User code will not really parse CoAP packets by hand, but rather use the CoAP server implementations and provide the resource handlers instead, which already will be called on fully parsed packets

[benpicco]

This also appears to introduce some garbage into the CoAP request

2025-04-17 12:53:54,058 # 8) :: ⇒ fdea:dbee:f::1  UDP [52430 ↣ 5683]
2025-04-17 12:53:54,064 # 	CoAP CON POST id=23875 opt=0 opt=0 opt=0 opt=0 /din/1/LCQAAA (228 bytes)

There seems to be a bug that needs fixing.

[maribu]

OK, we traced down the issue out of band and found it in #21048. The bug is not present in this PR.

But given that a pretty obvious bug is not found by the unit tests indicates that we really should get the unit tests fixed first.