ACP: add `proc_macro::ToTokens` to support `proc_macro::quote`

[tgross35]

Proposal

Problem statement

Context: rust-lang/rust#54722 (comment)

Our version of quote! currently relies on From<TokenStream>, which has some limitations:

• Individual tokens such as Ident, Literal cannot be quoted directly; they need to be converted to a TokenStream outside of the quote! invocation first
• Literals such as u32, &str, etc cannot be quoted directly
• References such as &TokenStream cannot be quoted; they need to be cloned outside of pm::quote!

These are pretty notable limitations compared to quote::quote!, for which all of the above work.

Motivating examples or use cases

The following does not work:

let x = pm::Ident::new("foo", pm::Span::call_site());
let _ = pm::quote! {
    // ERROR: `From<proc_macro::Ident>` is not implemented for `proc_macro::TokenStream`
    let $x = 199;
};

Reusing the same item is inconvenient:

let x1: pm::TokenStream = pm::Ident::new("foo", pm::Span::call_site()).into();
let x2 = x1.clone();
let _ = pm::quote! {
    let mut $x1 = 199;
    $x2 += 10;
};

Solution sketch

1. Add proc_macro::ToTokens that is identical to quote::ToTokens:

   pub trait ToTokens {
       fn to_tokens(&self, tokens: &mut TokenStream);
       fn to_token_stream(&self) -> TokenStream { ... }
       fn into_token_stream(self) -> TokenStream
          where Self: Sized { ... }
   }

2. Implement this for most types that have quote::ToTokens, including:

   • Aggregate token types TokenTree, TokenStream
   • Single token types Literal, Ident, Punct, Group
   • Indirect types where T: ToTokens: &T &mut T Box<T> Rc<T> Option<T> Cow<T>
   • Types that can create Literals: integer and float primitives, bool, char, str, String, CStr, CString
   • Ecosystem: proc_macro2 will be able to implement pm::ToTokens on its token types so they are accepted by pm::quote!

3. Change pm::quote to make use of ToTokens when it expands, rather than From

4. Reimplement some TokenStream conversion traits in terms of ToTokens if this can be done without breakage

   // currently `Extend<TokenStream>` and `Extend<TokenTree>`
   impl Extend<T: ToTokens> for TokenStream { /* ... */ }
   
   // currently `FromIterator<TokenStream>` and `FromIterator<TokenTree>`
   impl FromIterator<T: ToTokens> for TokenStream { /* ... */ }

   This does some of the job quote::TokenStreamExt.

Alternatives

• Continue to rely on From<X> for TokenStream but add this impl more of the types listed above. Having a dedicated trait seems less fragile, especially for literal types.
• Omitting to_token_stream and into_token_stream from ToTokens or gating them separately since they are just for convenience, rather than helping to solve a
• A different name; ToTokens doesn't seem quite accurate, but I don't know what would be better (ToTokenStream? ExtendTokenStream? Those seem a bit clunky).
• Considering impl<T: ToTokens> ToTokens for T is provided, should to_tokens take self by value rather than by reference so cloning isn't always necessary? (fn to_tokens(self, tokens: &mut TokenStream))

Links and related work

• Tracking issue for the quote! macro in proc_macro rust#54722
• quote::ToTokens

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[tgross35]

Cc @dtolnay since this is your design

[dtolnay]

LGTM, the interpolation piece of quote's design has been very good. Thank you for working on this.

Please make sure the integer and float impls produce a suffixed literal.

One impl not listed above is the one for Option, which I commonly find valuable. For example see all the Some( in thiserror.

I would expect a separate proposal for repetition, which will have more complicated considerations. My experience with this pair of features in the quote crate leads me to insist that repetition needs to be figured out before a quote macro with interpolation can be stabilized (even if that's just in the form of commiting not to implement repetition) because of nuance relating to types that are both iterable and interpolatable, such as TokenStream.