Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

ojeda · 2023-05-05T23:49:48Z

Like #910, but this time based on top of rust-next (instead of rust), and adds serde and serde_derive on top.

I include a draft commit at the end as a way to test that serde and serde_derive works in both the kernel crate and in a kernel module:

[    0.801425] rust_serde: Rust serde sample (init)
[    0.801634] rust_serde:             original = S { a: (), b: false, c: true, d: () }
[    0.802079] rust_serde:           serialized = [2, 0, 1, 0, 1, 1, 0, 3]
[    0.802506] rust_serde:         deserialized = S { a: (), b: false, c: true, d: () }
[    0.802718] rust_serde:   serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3]
[    0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () }
[    0.808954] rust_serde: Rust serde sample (exit)

Important note: it is not yet decided whether third-party crates will be upstreamed or not, or which ones, but this is useful for others to experiment and prototype solutions assuming these crates are available.

Cc @ariel-miculas who wanted to experiment with serde in the kernel: https://rust-for-linux.zulipchat.com/#narrow/stream/291565-Help/topic/Adding.20crates.20to.20out-of-tree.20modules/near/355792607

bjorn3 · 2023-05-06T12:08:45Z

rust/Makefile


+proc_macro2-skip_flags := \
+    --edition=2021 \
+    -Drust_2018_idioms \


Maybe do --cap-lints allow for all third-party crates? This overrides any -D arguments. It is also what cargo passes for all non-local crates.

bjorn3 · 2023-05-06T12:16:38Z

rust/Makefile

+
+serde-flags := \
+    -Amissing_docs \
+    --cfg no_fp_fmt_parse


If you pass --cfg 'feature="derive"' --extern serde_derive and make the serde target depend on serde_derive, you don't need --extern serde_derive elsewhere, but can instead access the derive macros using use serde::{Serialize, Deserialize};.

nbdd0121 · 2023-05-06T17:11:47Z

I am getting error when rustc-dev component is installed:

error[E0464]: multiple candidates for `dylib` dependency `serde_derive` found
  --> rust/kernel/test_serde.rs:18:5
   |
18 | use serde_derive::{Deserialize, Serialize};
   |     ^^^^^^^^^^^^
   |
   = note: candidate #1: /home/gary/.rustup/toolchains/1.68.2-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde_derive-a574e0b043b5a546.so
   = note: candidate #2: /home/gary/Projects/linux/rust/libserde_derive.so

bjorn3 · 2023-05-06T17:23:26Z

That can probably be fixed by passing the path of libserde_derive.so to --extern serde_derive. Might be a good idea to do anyway. Cargo does this too.

ariel-miculas · 2023-05-08T14:45:54Z

I have the following error with rust-analyzer target:

$ make LLVM=1 rust-analyzer
Traceback (most recent call last):
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 141, in <module>
    main()
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 134, in main
    "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src),
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 107, in generate_crates
    if f"{name}.o" not in open(path.parent / "Makefile").read():
FileNotFoundError: [Errno 2] No such file or directory: 'samples/rust/local_data_format/Makefile'
make[1]: *** [rust/Makefile:504: rust-analyzer] Error 1
make: *** [Makefile:1872: rust-analyzer] Error 2

This is a subset of the Rust `proc-macro2` crate, version 1.0.46, licensed under "Apache-2.0 OR MIT", from: https://github.com/dtolnay/proc-macro2/raw/1.0.46/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/dtolnay/proc-macro2/blob/1.0.46/README.md#license https://github.com/dtolnay/proc-macro2/blob/1.0.46/LICENSE-APACHE https://github.com/dtolnay/proc-macro2/blob/1.0.46/LICENSE-MIT The next two patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/proc-macro2/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/dtolnay/proc-macro2/raw/1.0.46/src/$path \ | diff --unified rust/proc-macro2/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

The `proc-macro2` crate depends on the `unicode-ident` crate to determine whether characters have the XID_Start or XID_Continue properties according to Unicode Standard Annex torvalds#31. However, we only need ASCII identifiers in the kernel, thus we can simplify the check and remove completely that dependency. Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `quote` crate, version 1.0.21, licensed under "Apache-2.0 OR MIT", from: https://github.com/dtolnay/quote/raw/1.0.21/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/dtolnay/quote/blob/1.0.21/README.md#license https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-APACHE https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-MIT The next patch modifies these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/quote/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/dtolnay/quote/raw/1.0.21/src/$path \ | diff --unified rust/quote/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `syn` crate, version 1.0.104, licensed under "Apache-2.0 OR MIT", from: https://github.com/dtolnay/syn/raw/1.0.104/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/dtolnay/syn/blob/1.0.104/README.md#license https://github.com/dtolnay/syn/blob/1.0.104/LICENSE-APACHE https://github.com/dtolnay/syn/blob/1.0.104/LICENSE-MIT The next two patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/syn/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/dtolnay/syn/raw/1.0.104/src/$path \ | diff --unified rust/syn/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

The `syn` crate depends on the `unicode-ident` crate to determine whether characters have the XID_Start or XID_Continue properties according to Unicode Standard Annex torvalds#31. However, we only need ASCII identifiers in the kernel, thus we can simplify the check and remove completely that dependency. Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `serde` crate, version v1.0.156, licensed under "Apache-2.0 OR MIT", from: https://github.com/serde-rs/serde/tree/v1.0.156/serde/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/serde-rs/serde/blob/v1.0.156/README.md#license https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-APACHE https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-MIT The next three patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/serde/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/serde-rs/serde/raw/v1.0.156/serde/src/$path \ | diff --unified rust/serde/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

We do not have formatting for floating point in the kernel, thus simple compile out all that. Probably we should name it differently, more similar to `integer128`. Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

This is a subset of the Rust `serde_derive` crate, version v1.0.156, licensed under "Apache-2.0 OR MIT", from: https://github.com/serde-rs/serde/tree/v1.0.156/serde_derive/src The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers). For copyright details, please see: https://github.com/serde-rs/serde/blob/v1.0.156/README.md#license https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-APACHE https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-MIT The next patch modifies these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced. The following script may be used to verify the contents: for path in $(cd rust/serde_derive/ && find . -type f -name '*.rs'); do curl --silent --show-error --location \ https://github.com/serde-rs/serde/raw/v1.0.156/serde_derive/src/$path \ | diff --unified rust/serde_derive/$path - && echo $path: OK done Signed-off-by: Miguel Ojeda <[email protected]>

Signed-off-by: Miguel Ojeda <[email protected]>

…e_derive` With all the new files in place from the new crates, this patch adds support for them in the build system. Note that, if we decide to add third-party crates support to upstream, we would probably want to do it in a different way. But this is useful for playing around with these crates, experimenting, etc. Signed-off-by: Miguel Ojeda <[email protected]>

A trivial example based on `serde`'s `example-format' [1]. It contains both a in-`kernel` data format later used by the kernel module, as well as a local data format in the module. The kernel module gives an output such as: [ 0.801425] rust_serde: Rust serde sample (init) [ 0.801634] rust_serde: original = S { a: (), b: false, c: true, d: () } [ 0.802079] rust_serde: serialized = [2, 0, 1, 0, 1, 1, 0, 3] [ 0.802506] rust_serde: deserialized = S { a: (), b: false, c: true, d: () } [ 0.802718] rust_serde: serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3] [ 0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () } [ 0.808954] rust_serde: Rust serde sample (exit) Note that this is just a quick draft/hack to check the previous commits work. It is not intended to be merged at all. Link: https://github.com/serde-rs/example-format [1] Signed-off-by: Miguel Ojeda <[email protected]>

ojeda · 2023-05-08T14:51:03Z

That should be fixed in commit 5c7548d, but rust-next does not have it yet here.

Let me move rust-next now that v6.4-rc1 is out and rebase this one on top.

fre3dm4n · 2024-06-29T05:03:58Z

Could I please know what is preventing this pull request from being merged into the rust-next branch? Incorporating these third-party crates would significantly enhance our ability to develop complex drivers in Rust. 😊

ojeda · 2024-06-29T12:26:15Z

We plan to have support for syn and other selected third-party crates soon in mainline, although in a different form than what is in this PR. What are you planning to use syn for? It could be useful to know. Thanks!

(The PRs here are not used for mainline development; we have some notes in https://rust-for-linux.com/contributing and other pages, in case you have not seen them).

fre3dm4n · 2024-06-29T12:50:35Z

Will the lazy_static crate be supported? Static variables are extremely useful, especially in driver development, due to the prevalence of singleton variables.

bjorn3 · 2024-06-29T13:02:27Z

The lazy_static crate has been pretty much subsumed by the once_cell crate. Neither is directly usable in the kernel though as for no_std usages they use a spinlock without any yielding rather than the kernel's version of mutexes. If you want you could write your own version of once_cell which uses the kernel mutex type though. By the way lazy_static is not implemented as proc-macro anyway. It uses a decl macro.

fre3dm4n · 2024-07-01T03:21:31Z

Ok I see, thanks 👍

ojeda · 2025-11-10T09:55:47Z

We are introducing syn support upstream (https://lore.kernel.org/rust-for-linux/[email protected]/), so closing this.

This patch series introduces support for `syn` (and its dependencies): Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code. Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally. It is the most downloaded Rust crate (according to crates.io), and it is also used by the Rust compiler itself. Having such support allows to greatly simplify writing complex macros such as `pin-init`. We will use it in the `macros` crate too. Benno has already prepared the `pin-init` version based on this, and on top of that, we will be able to simplify the `macros` crate too. I think Jesung is working on updating the `TryFrom` and `Into` upcoming derive macros to use `syn` too. The series starts with a few preparation commits (two fixes were already merged in mainline that were discovered by this series), then each crate is added. Finally, support for using the new crates from our `macros` crate is introduced. This has been a long time coming, e.g. even before Rust for Linux was merged into the Linux kernel, Gary and Benno have wanted to use `syn`. The first iterations of this, from 2022 and 2023 (with `serde` too, another popular crate), are at: #910 #1007 After those, we considered picking these from the distributions where possible. However, after discussing it, it is not really worth the complexity: vendoring makes things less complex and is less fragile. In particular, we avoid having to support and test several versions, we avoid having to introduce Cargo just to properly fetch the right versions from the registry, we can easily customize the crates if needed (e.g. dropping the `unicode_idents` dependency like it is done in this series) and we simplify the configuration of the build for users for which the "default" paths/registries would not have worked. Moreover, nowadays, the ~57k lines introduced are not that much compared to years ago (it dwarfed the actual Rust kernel code). Moreover, back then it wasn't clear the Rust experiment would be a success, so it would have been a bit pointless/risky to add many lines for nothing. Our macro needs were also smaller in the early days. So, finally, in Kangrejos 2025 we discussed going with the original, simpler approach. Thus here it is the result. There should not be many updates needed for these, and even if there are, they should not be too big, e.g. +7k -3k lines across the 3 crates in the last year. Note that `syn` does not have all the features enabled, since we do not need them so far, but they can easily be enabled just adding them to the list. Link: https://patch.msgid.link/[email protected] Signed-off-by: Miguel Ojeda <[email protected]>

ojeda force-pushed the third-party branch from 9aa35e7 to db6837e Compare May 6, 2023 00:02

bjorn3 reviewed May 6, 2023

View reviewed changes

ojeda added 16 commits May 8, 2023 16:48

rust: proc-macro2: add SPDX License Identifiers

ff8917c

Signed-off-by: Miguel Ojeda <[email protected]>

rust: quote: add SPDX License Identifiers

d7f4b0d

Signed-off-by: Miguel Ojeda <[email protected]>

rust: syn: add SPDX License Identifiers

f9bcfd0

Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde: add no_fp_fmt_parse support

7fcbf51

We do not have formatting for floating point in the kernel, thus simple compile out all that. Probably we should name it differently, more similar to `integer128`. Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde: enable the core_c_str feature

22605b8

Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde: add SPDX License Identifiers

b56a317

Signed-off-by: Miguel Ojeda <[email protected]>

rust: serde_derive: add SPDX License Identifiers

16bebcf

Signed-off-by: Miguel Ojeda <[email protected]>

ojeda force-pushed the third-party branch from db6837e to 7d2ea51 Compare May 8, 2023 14:55

ojeda force-pushed the rust-next branch from bee23a1 to 341faf2 Compare July 19, 2023 18:06

ojeda force-pushed the rust-next branch from bd83e43 to 49a9ef7 Compare August 7, 2023 09:39

ojeda force-pushed the rust-next branch from 71004ce to 4af84c6 Compare August 21, 2023 12:32

ojeda force-pushed the rust-next branch from 2392e2a to e3c3d34 Compare February 18, 2024 20:22

ToolmanP mentioned this pull request Jul 19, 2024

In-tree crate built-in.rlib support? #1091

Open

ojeda force-pushed the rust-next branch from a419846 to 6d1c22d Compare August 21, 2024 22:16

ojeda force-pushed the rust-next branch 2 times, most recently from 278e92e to c9b5ce6 Compare October 7, 2024 21:06

Richardhongyu mentioned this pull request Oct 8, 2024

Tracking issue: refactor RROS kernel code BUPT-OS/RROS#48

Open

3 tasks

ojeda force-pushed the rust-next branch 4 times, most recently from 9ee7197 to 6ce162a Compare October 15, 2024 21:11

ojeda force-pushed the rust-next branch from 0675624 to d072acd Compare November 11, 2024 22:18

ojeda force-pushed the rust-next branch from 92797e9 to 9b98be7 Compare December 16, 2024 20:49

ojeda force-pushed the rust-next branch from dd21715 to edc5e6e Compare May 12, 2025 11:52

ojeda force-pushed the rust-next branch 5 times, most recently from bfc3cd8 to a3b2347 Compare May 27, 2025 18:27

ojeda force-pushed the rust-next branch from eb6778a to b20fbbc Compare May 29, 2025 21:17

ojeda force-pushed the rust-next branch 3 times, most recently from 5d132fa to 3b83f5d Compare October 22, 2025 05:47

ojeda closed this Nov 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

Uh oh!

ojeda commented May 5, 2023 •

edited

Loading

Uh oh!

bjorn3 May 6, 2023

Uh oh!

bjorn3 May 6, 2023 •

edited

Loading

Uh oh!

nbdd0121 commented May 6, 2023

Uh oh!

bjorn3 commented May 6, 2023

Uh oh!

ariel-miculas commented May 8, 2023

Uh oh!

ojeda commented May 8, 2023

Uh oh!

fre3dm4n commented Jun 29, 2024

Uh oh!

ojeda commented Jun 29, 2024

Uh oh!

fre3dm4n commented Jun 29, 2024

Uh oh!

bjorn3 commented Jun 29, 2024 •

edited

Loading

Uh oh!

fre3dm4n commented Jul 1, 2024

Uh oh!

ojeda commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

Third-party crates support: proc-macro2, quote, syn, serde and serde_derive #1007

Third-party crates support: proc-macro2, quote, syn, serde and serde_derive #1007

Uh oh!

Conversation

ojeda commented May 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bjorn3 May 6, 2023

Choose a reason for hiding this comment

Uh oh!

bjorn3 May 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbdd0121 commented May 6, 2023

Uh oh!

bjorn3 commented May 6, 2023

Uh oh!

ariel-miculas commented May 8, 2023

Uh oh!

ojeda commented May 8, 2023

Uh oh!

fre3dm4n commented Jun 29, 2024

Uh oh!

ojeda commented Jun 29, 2024

Uh oh!

fre3dm4n commented Jun 29, 2024

Uh oh!

bjorn3 commented Jun 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fre3dm4n commented Jul 1, 2024

Uh oh!

ojeda commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

Third-party crates support: `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive` #1007

ojeda commented May 5, 2023 •

edited

Loading

bjorn3 May 6, 2023 •

edited

Loading

bjorn3 commented Jun 29, 2024 •

edited

Loading