Redesign function configuration in bindgen! (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` macro. The main goal of this redesign is to
better support WASIp3 and component model async functions. Prior to this
redesign there was a mish mash of mechanisms to configure behavior of
imports/exports:

* The `async` configuration could turn everything async, nothing async,
  only some imports async, or everything except some imports async.

* The `concurrent_{imports,exports}` keys were required to explicitly
  opt-in to component model async signatures and applied to all
  imports/exports.

* The `trappable_imports` configuration would indicate a list of imports
  allowed to trap and it had special configuration for everything,
  nothing, and only a certain list.

* The `tracing` and `verbose_tracing` keys could be applied to either
  nothing or all functions.

Overall the previous state of configuration in `bindgen!` was clearly a
hodgepodge of systems that organically grew over time. In my personal
opinion it was in dire need of a refresh to take into account how
component-model-async ended up being implemented as well as
consolidating the one-off systems amongst all of these configuration
keys. A major motivation of this redesign, for example, was to inherit
behavior from WIT files by default. An `async` function in WIT should
not require `concurrent_*` keys to be configured, but rather it should
generate correct bindings by default.

In this commit, all of the above keys were removed. All keys have been
replaced with `imports` and `exports` configuration keys. Each behaves
the same way and looks like so:

    bindgen!({
        // ...
        imports: {
            // enable tracing for just this function
            "my:local/interface/func": tracing,

            // enable verbose tracing for just this function
            "my:local/interface/other-func": tracing | verbose_tracing,

            // this is blocking in WIT, but generate async bindings for
            // it
            "my:local/interface/[method]io.block": async,

            // like above, but use "concurrent" bindings which have
            // access to the store.
            "my:local/interface/[method]io.block-again": async | store,

            // everything else is, by default, trappable
            default: trappable,
        },
    });

Effectively all the function-level configuration items are now bitflags.
These bitflags are by default inherited from the WIT files itself (e.g.
`async` functions are `async | store` by default). Further configuration
is then layered on top at the desires of the embedder. Supported keys are:

* `async` - this means that a Rust-level `async` function should be
  generated. This is either `CallStyle::Async` or
  `CallStyle::Concurrent` as it was prior, depending on ...

* `store` - this means that the generated function will have access to
  the store on the host. This is only implemented right now for `async |
  store` functions which map to `CallStyle::Concurrent`. In the future
  I'd like to support just-`store` functions which means that you could
  define a synchronous function with access to the store in addition to
  an asynchronous function.

* `trappable` - this means that the function returns a
  `wasmtime::Result<TheWitBindingType>`. If `trappable_errors` is
  applicable then it means just a `Result<TheWitOkType,
  TrappableErrorType>` is returned (like before)

* `tracing` - this enables `tracing!` integration for this function.

* `verbose_tracing` - this logs all argument values for this function
  (including lists).

* `ignore_wit` - this ignores the WIT-level defaults of the function
  (e.g. ignoring WIT `async`).

The way this then works is all modeled is that for any WIT function
being generated there are a set of flags associated with that function.
To calculate the flags the algorithm looks like:

1. Find the first matching rule in the `imports` or `exports` map
   depending on if the function is imported or exported. If there is no
   matching rule then use the `default` rule if present. This is the
   initial set of flags for the function (or empty if nothing was
   found).

2. If `ignore_wit` is present, return the flags from step 1. Otherwise
   add in `async | store` if the function is `async` in WIT.

The resulting set of flags are then used to control how everything is
generated. For example the same split traits of today are still
generated and it's controlled based on the flags. Note though that the
previous `HostConcurrent` trait was renamed to `HostWithStore` to make
space for synchronous functions in this trait in the future too.

The end result of all these changes is that configuring imports/exports
now uses the exact same selection system as the `with` replacement map,
meaning there's only one system of selecting functions instead of 3.
WIT-level `async` is now respected by default meaning that bindings work
by default without further need to configure anything (unless more
functionality is desired).

One final minor change made here as well is that auto-generated
`instantiate` methods are now always synchronous and an
`instantiate_async` method is unconditionally generated for async mode.
This means that bindings always generate both functions and it's up to
the embedder to choose the appropriate one.

Closes #11246
Closes #11247

* Update expanded test expectations

prtest:full

* Fix the min platform embedding example

* Fix doc tests

* Always generate `*WithStore` traits

This helps when using the `with` mapping since that can always assume
that `HostWithStore` is available in the generated bindings, avoiding
the need to duplicate configuration options.

* Update test expectations

* Review comments

Author: alexcrichton

Cargo.lock

@@ -79,7 +79,6 @@ gimli = { workspace = true, optional = true }
 pulley-interpreter = { workspace = true, optional = true }
 
 async-trait = { workspace = true }
-trait-variant = { workspace = true }
 bytes = { workspace = true }
 cfg-if = { workspace = true }
 tokio = { workspace = true, optional = true, features = [ "signal", "macros" ] }
@@ -103,7 +102,6 @@ criterion = { workspace = true }
 num_cpus = "1.13.0"
 memchr = "2.4"
 async-trait = { workspace = true }
-trait-variant = { workspace = true }
 wat = { workspace = true }
 rayon = "1.5.0"
 wasmtime-wast = { workspace = true, features = ['component-model'] }
@@ -363,7 +361,6 @@ tracing = "0.1.26"
 bitflags = "2.0"
 thiserror = "2.0.12"
 async-trait = "0.1.71"
-trait-variant = "0.1.2"
 heck = "0.5"
 similar = "2.1.0"
 toml = "0.8.10"

Cargo.toml

@@ -40,5 +40,5 @@ prettyplease = "0.2.31"
 similar = { workspace = true }
 
 [features]
-async = []
+async = ['wasmtime-wit-bindgen/async']
 component-model-async = ['async', 'wasmtime-wit-bindgen/component-model-async']

crates/component-macro/Cargo.toml

@@ -1,14 +1,14 @@
 use proc_macro2::{Span, TokenStream};
 use quote::ToTokens;
-use std::collections::{HashMap, HashSet};
+use std::collections::HashMap;
 use std::env;
 use std::path::{Path, PathBuf};
 use std::sync::atomic::{AtomicUsize, Ordering::Relaxed};
 use syn::parse::{Error, Parse, ParseStream, Result};
 use syn::punctuated::Punctuated;
 use syn::{Token, braced, token};
 use wasmtime_wit_bindgen::{
-    AsyncConfig, CallStyle, Opts, Ownership, TrappableError, TrappableImports,
+    FunctionConfig, FunctionFilter, FunctionFlags, Opts, Ownership, TrappableError,
 };
 use wit_parser::{PackageId, Resolve, UnresolvedPackageGroup, WorldId};
 
@@ -21,22 +21,6 @@ pub struct Config {
 }
 
 pub fn expand(input: &Config) -> Result<TokenStream> {
-    if let (CallStyle::Async | CallStyle::Concurrent, false) =
-        (input.opts.call_style(), cfg!(feature = "async"))
-    {
-        return Err(Error::new(
-            Span::call_site(),
-            "cannot enable async bindings unless `async` crate feature is active",
-        ));
-    }
-
-    if input.opts.concurrent_imports && !cfg!(feature = "component-model-async") {
-        return Err(Error::new(
-            Span::call_site(),
-            "cannot enable `concurrent_imports` option unless `component-model-async` crate feature is active",
-        ));
-    }
-
     let mut src = match input.opts.generate(&input.resolve, input.world) {
         Ok(s) => s,
         Err(e) => return Err(Error::new(Span::call_site(), e.to_string())),
@@ -94,7 +78,8 @@ impl Parse for Config {
         let mut world = None;
         let mut inline = None;
         let mut paths = Vec::new();
-        let mut async_configured = false;
+        let mut imports_configured = false;
+        let mut exports_configured = false;
         let mut include_generated_code_from_file = false;
 
         if input.peek(token::Brace) {
@@ -118,20 +103,8 @@ impl Parse for Config {
                         }
                         inline = Some(s.value());
                     }
-                    Opt::Tracing(val) => opts.tracing = val,
-                    Opt::VerboseTracing(val) => opts.verbose_tracing = val,
                     Opt::Debug(val) => opts.debug = val,
-                    Opt::Async(val, span) => {
-                        if async_configured {
-                            return Err(Error::new(span, "cannot specify second async config"));
-                        }
-                        async_configured = true;
-                        opts.async_ = val;
-                    }
-                    Opt::ConcurrentImports(val) => opts.concurrent_imports = val,
-                    Opt::ConcurrentExports(val) => opts.concurrent_exports = val,
                     Opt::TrappableErrorType(val) => opts.trappable_error_type = val,
-                    Opt::TrappableImports(val) => opts.trappable_imports = val,
                     Opt::Ownership(val) => opts.ownership = val,
                     Opt::Interfaces(s) => {
                         if inline.is_some() {
@@ -172,6 +145,20 @@ impl Parse for Config {
                         opts.wasmtime_crate = Some(f.into_token_stream().to_string())
                     }
                     Opt::IncludeGeneratedCodeFromFile(i) => include_generated_code_from_file = i,
+                    Opt::Imports(config, span) => {
+                        if imports_configured {
+                            return Err(Error::new(span, "cannot specify imports configuration"));
+                        }
+                        opts.imports = config;
+                        imports_configured = true;
+                    }
+                    Opt::Exports(config, span) => {
+                        if exports_configured {
+                            return Err(Error::new(span, "cannot specify exports configuration"));
+                        }
+                        opts.exports = config;
+                        exports_configured = true;
+                    }
                 }
             }
         } else {
@@ -290,39 +277,38 @@ mod kw {
     syn::custom_keyword!(with);
     syn::custom_keyword!(except_imports);
     syn::custom_keyword!(only_imports);
-    syn::custom_keyword!(trappable_imports);
     syn::custom_keyword!(additional_derives);
     syn::custom_keyword!(stringify);
     syn::custom_keyword!(skip_mut_forwarding_impls);
     syn::custom_keyword!(require_store_data_send);
     syn::custom_keyword!(wasmtime_crate);
     syn::custom_keyword!(include_generated_code_from_file);
-    syn::custom_keyword!(concurrent_imports);
-    syn::custom_keyword!(concurrent_exports);
     syn::custom_keyword!(debug);
+    syn::custom_keyword!(imports);
+    syn::custom_keyword!(exports);
+    syn::custom_keyword!(store);
+    syn::custom_keyword!(trappable);
+    syn::custom_keyword!(ignore_wit);
+    syn::custom_keyword!(exact);
 }
 
 enum Opt {
     World(syn::LitStr),
     Path(Vec<syn::LitStr>),
     Inline(syn::LitStr),
-    Tracing(bool),
-    VerboseTracing(bool),
-    Async(AsyncConfig, Span),
     TrappableErrorType(Vec<TrappableError>),
     Ownership(Ownership),
     Interfaces(syn::LitStr),
     With(HashMap<String, String>),
-    TrappableImports(TrappableImports),
     AdditionalDerives(Vec<syn::Path>),
     Stringify(bool),
     SkipMutForwardingImpls(bool),
     RequireStoreDataSend(bool),
     WasmtimeCrate(syn::Path),
     IncludeGeneratedCodeFromFile(bool),
-    ConcurrentImports(bool),
-    ConcurrentExports(bool),
     Debug(bool),
+    Imports(FunctionConfig, Span),
+    Exports(FunctionConfig, Span),
 }
 
 impl Parse for Opt {
@@ -360,60 +346,6 @@ impl Parse for Opt {
             input.parse::<kw::world>()?;
             input.parse::<Token![:]>()?;
             Ok(Opt::World(input.parse()?))
-        } else if l.peek(kw::tracing) {
-            input.parse::<kw::tracing>()?;
-            input.parse::<Token![:]>()?;
-            Ok(Opt::Tracing(input.parse::<syn::LitBool>()?.value))
-        } else if l.peek(kw::verbose_tracing) {
-            input.parse::<kw::verbose_tracing>()?;
-            input.parse::<Token![:]>()?;
-            Ok(Opt::VerboseTracing(input.parse::<syn::LitBool>()?.value))
-        } else if l.peek(Token![async]) {
-            let span = input.parse::<Token![async]>()?.span;
-            input.parse::<Token![:]>()?;
-            if input.peek(syn::LitBool) {
-                match input.parse::<syn::LitBool>()?.value {
-                    true => Ok(Opt::Async(AsyncConfig::All, span)),
-                    false => Ok(Opt::Async(AsyncConfig::None, span)),
-                }
-            } else {
-                let contents;
-                syn::braced!(contents in input);
-
-                let l = contents.lookahead1();
-                let ctor: fn(HashSet<String>) -> AsyncConfig = if l.peek(kw::except_imports) {
-                    contents.parse::<kw::except_imports>()?;
-                    contents.parse::<Token![:]>()?;
-                    AsyncConfig::AllExceptImports
-                } else if l.peek(kw::only_imports) {
-                    contents.parse::<kw::only_imports>()?;
-                    contents.parse::<Token![:]>()?;
-                    AsyncConfig::OnlyImports
-                } else {
-                    return Err(l.error());
-                };
-
-                let list;
-                syn::bracketed!(list in contents);
-                let fields: Punctuated<syn::LitStr, Token![,]> =
-                    list.parse_terminated(Parse::parse, Token![,])?;
-
-                if contents.peek(Token![,]) {
-                    contents.parse::<Token![,]>()?;
-                }
-                Ok(Opt::Async(
-                    ctor(fields.iter().map(|s| s.value()).collect()),
-                    span,
-                ))
-            }
-        } else if l.peek(kw::concurrent_imports) {
-            input.parse::<kw::concurrent_imports>()?;
-            input.parse::<Token![:]>()?;
-            Ok(Opt::ConcurrentImports(input.parse::<syn::LitBool>()?.value))
-        } else if l.peek(kw::concurrent_exports) {
-            input.parse::<kw::concurrent_exports>()?;
-            input.parse::<Token![:]>()?;
-            Ok(Opt::ConcurrentExports(input.parse::<syn::LitBool>()?.value))
         } else if l.peek(kw::ownership) {
             input.parse::<kw::ownership>()?;
             input.parse::<Token![:]>()?;
@@ -472,22 +404,6 @@ impl Parse for Opt {
             let fields: Punctuated<(String, String), Token![,]> =
                 contents.parse_terminated(with_field_parse, Token![,])?;
             Ok(Opt::With(HashMap::from_iter(fields)))
-        } else if l.peek(kw::trappable_imports) {
-            input.parse::<kw::trappable_imports>()?;
-            input.parse::<Token![:]>()?;
-            let config = if input.peek(syn::LitBool) {
-                match input.parse::<syn::LitBool>()?.value {
-                    true => TrappableImports::All,
-                    false => TrappableImports::None,
-                }
-            } else {
-                let contents;
-                syn::bracketed!(contents in input);
-                let fields: Punctuated<syn::LitStr, Token![,]> =
-                    contents.parse_terminated(Parse::parse, Token![,])?;
-                TrappableImports::Only(fields.iter().map(|s| s.value()).collect())
-            };
-            Ok(Opt::TrappableImports(config))
         } else if l.peek(kw::additional_derives) {
             input.parse::<kw::additional_derives>()?;
             input.parse::<Token![:]>()?;
@@ -521,6 +437,14 @@ impl Parse for Opt {
             Ok(Opt::IncludeGeneratedCodeFromFile(
                 input.parse::<syn::LitBool>()?.value,
             ))
+        } else if l.peek(kw::imports) {
+            let span = input.parse::<kw::imports>()?.span;
+            input.parse::<Token![:]>()?;
+            Ok(Opt::Imports(parse_function_config(input)?, span))
+        } else if l.peek(kw::exports) {
+            let span = input.parse::<kw::exports>()?.span;
+            input.parse::<Token![:]>()?;
+            Ok(Opt::Exports(parse_function_config(input)?, span))
         } else {
             Err(l.error())
         }
@@ -579,3 +503,74 @@ fn with_field_parse(input: ParseStream<'_>) -> Result<(String, String)> {
 
     Ok((interface, buf))
 }
+
+fn parse_function_config(input: ParseStream<'_>) -> Result<FunctionConfig> {
+    let content;
+    syn::braced!(content in input);
+    let mut ret = FunctionConfig::new();
+
+    let list = Punctuated::<FunctionConfigSyntax, Token![,]>::parse_terminated(&content)?;
+    for item in list.into_iter() {
+        ret.push(item.filter, item.flags);
+    }
+
+    return Ok(ret);
+
+    struct FunctionConfigSyntax {
+        filter: FunctionFilter,
+        flags: FunctionFlags,
+    }
+
+    impl Parse for FunctionConfigSyntax {
+        fn parse(input: ParseStream<'_>) -> Result<Self> {
+            let l = input.lookahead1();
+            let filter = if l.peek(syn::LitStr) {
+                FunctionFilter::Name(input.parse::<syn::LitStr>()?.value())
+            } else if l.peek(Token![default]) {
+                input.parse::<Token![default]>()?;
+                FunctionFilter::Default
+            } else {
+                return Err(l.error());
+            };
+
+            input.parse::<Token![:]>()?;
+
+            let mut flags = FunctionFlags::empty();
+            while !input.is_empty() {
+                let l = input.lookahead1();
+                if l.peek(Token![async]) {
+                    input.parse::<Token![async]>()?;
+                    flags |= FunctionFlags::ASYNC;
+                } else if l.peek(kw::tracing) {
+                    input.parse::<kw::tracing>()?;
+                    flags |= FunctionFlags::TRACING;
+                } else if l.peek(kw::verbose_tracing) {
+                    input.parse::<kw::verbose_tracing>()?;
+                    flags |= FunctionFlags::VERBOSE_TRACING;
+                } else if l.peek(kw::store) {
+                    input.parse::<kw::store>()?;
+                    flags |= FunctionFlags::STORE;
+                } else if l.peek(kw::trappable) {
+                    input.parse::<kw::trappable>()?;
+                    flags |= FunctionFlags::TRAPPABLE;
+                } else if l.peek(kw::ignore_wit) {
+                    input.parse::<kw::ignore_wit>()?;
+                    flags |= FunctionFlags::IGNORE_WIT;
+                } else if l.peek(kw::exact) {
+                    input.parse::<kw::exact>()?;
+                    flags |= FunctionFlags::EXACT;
+                } else {
+                    return Err(l.error());
+                }
+
+                if input.peek(Token![|]) {
+                    input.parse::<Token![|]>()?;
+                } else {
+                    break;
+                }
+            }
+
+            Ok(FunctionConfigSyntax { filter, flags })
+        }
+    }
+}