Semantic Tokens - Reduce set of non-standard token types

[frankplow]

At present, a built-in type is conveyed to the client using a token with type builtinType. This is not in the standard set of semantic token types and therefore requires configuration in the client. The same semantic idea can be conveyed using a token with type type and the defaultLibrary modifier, both of which are part of the standard set of token types and modifiers.

[frankplow]

There are actually a number of custom types which I think could be replaced with standard token types without losing meaning. I think this is a holdover from the deprecated semantic highlighting LSP extension, which had no set of standard types and did not support modifiers. To preserve meaning when replacing types with standard types, non-standard modifiers could be used as in this case the client can configure sensible fallbacks for the highlighting. Distributions of rust-analyzer, such as the VSCode plugin, generally provide configurations for these custom types, hence this is not a massive issue, however maintenance of these distributions could be simplified and creation of new distributions for different clients made easier by slimming down the set of custom token types.

Here are a few suggestions (type + modifiers):

• attribute -> decorator; these may not have quite identical meanings, but given that Rust only has attributes this should serve to provide some sensible default highlighting for attributes.
• builtinAttribute -> decorator + defaultLibrary
• derive -> decorator + derive
• deriveHelper -> decorator + derive
• constParameter -> parameter + readonly
• trait -> interface; not too sure about this one
• typeAlias -> type
• arithmetic -> operator + arithmetic
• bitwise -> operator + bitwise
• comparison -> operator + comparison
• logical -> operator + logical
• attributeBracket -> punctuation + attributeBracket
• angle -> punctuation + angleBracket
• brace -> punctuation + curlyBracket
• parenthesis -> punctuation + parenthesis
• colon -> punctuation + colon
• comma -> punctuation + comma
• dot -> punctuation + period
• semi -> punctuation + semicolon
• macroBang -> punctuation + exclamationMark; or made part of its corresponding macro, I don't think many want to highlight this differently to the preceding word.
• selfKeyword -> parameter + self
• selfTypeKeyword -> type + self

I am happy to implement this, but think it is worth getting some consensus on which changes should be made first. If you have any comments on the suggestions made above or your own suggestions, please add them. The full set of current token types can be found here.

[flodiebold]

Built-in types are not the same thing as standard library types, which is what defaultLibrary indicates.

[frankplow]

A non-standard builtin modifier could be introduced then?

• builtinType -> Type + builtin
• builtinAttribute -> decorator + builtin

[Veykril]

I fail to see why adding a ton of modifiers would be a better solution than what we currently have. That is, why exactly are non-standard modifiers over non-standard types better in your eyes?

On another note, yes we should be using the standard types where it makes sense, we already use interface for traits as that is basically the equivalent in rust. decorator didn't exist when we introduced the attribute type, so that one I think certainly makes sense to use instead now.

[frankplow]

@Veykril

Why exactly are non-standard modifiers over non-standard types better in your eyes?

Most clients implement fallbacks from unrecognised modifers to the un-modified type. For example, type + builtin would fall back to type, which is what I suspect most users would desire anyway, whereas builtinType has nothing to fall back to and would not be semantically highlighted.

[Veykril]

Hmm I see, the thing is we are limited in how many token modifiers we can have, which is currently 32 due to the bitset requirement. It also feels like the wrong usage of token modifiers to me and instead signals a lack of the language server protocol in that we really need to be able to specify a hierarchy of token types which I'd much rather see. VSCode itself already supports this, but the LSP unfortunately doesn't.

[frankplow]

A point to be made is that the feature is intended to convey highlighting information, rather than a full semantic description of the document. The server therefore has to make some decisions as to what constitutes sensible highlighting. Personally I can't see many use cases in which a user wants to highlight, for example, a pair of parentheses differently to a pair of square brackets. Maybe the approach to take is an overall slimming down of the information sent to the client?

(NB: #12783 reached a similar conclusion)

[Veykril]

A point to be made is that the feature is intended to convey highlighting information, rather than a full semantic description of the document

I don't really see the point made here, full semanticdescription is highlighting information so what the server does seems like a correct choice to me.

Personally I can't see many use cases in which a user wants to highlight, for example, a pair of parentheses differently to a pair of square brackets.

That I can't really tell you either, I am not using the different punctuation tags, others are as they were requested, see #6152 and #8279.

(NB: #12783 reached a similar conclusion)

I am all in favor of making more parts of this infra configurable, since there are a lot of tags that are not useful for the general userbase and a lot of clients seem to struggle with too many tokens being sent. But overall I would dislike us removing features from our syntax highlighting fully as there are clearly people that want these.

And to reiterate regarding the main part of this issue, I do not think restricting ourselves to the standard token types is a good idea, for one as outlined we can only have up to 32 modifiers and secondly there is no standard token set that describes all the languages (which is why extensibility by the protocol is even a thing).

[frankplow]

Sorry maybe I didn't articulate myself too well, all I meant to say is that the specification is quite restrictive and so the server must make compromises.

I don't think that the set of token types needs to be limited to only the standard set, just reduced. As you say, the set of standard types is not a one-size-fits-all solution and there are certain Rust features such as lifetimes which I think warrant their own types. At the moment however, rust-analyzer provides 35 non-standard types. For comparison, clangd provides 2 and pylance provides 4.

A lot of these non-standard types come from the distinct operator and punctuation types which I didn't realise have come from feature requests so yes I agree they should not be removed. I think server configuration options are a good compromise of flexibility and ease-of-use.

To illustrate why I think having so many non-standard token types is somewhat of an issue, here is a snippet of my coc configuration:

vim.api.nvim_command('hi link CocSemAngle CocSemPunctuation')
vim.api.nvim_command('hi link CocSemArithmetic CocSemOperator')
vim.api.nvim_command('hi link CocSemAttribute CocSemDecorator')
vim.api.nvim_command('hi link CocSemAttributeBracket CocSemDecorator')
vim.api.nvim_command('hi link CocSemBitwise CocSemOperator')
vim.api.nvim_command('hi link CocSemBoolean Boolean')
vim.api.nvim_command('hi link CocSemBrace CocSemPunctuation')
vim.api.nvim_command('hi link CocSemBracket CocSemPunctuation')
vim.api.nvim_command('hi link CocSemBuiltinAttribute CocSemAttribute')
vim.api.nvim_command('hi link CocSemBuiltinType CocSemType')
vim.api.nvim_command('hi link CocSemCharacter CocSemString')
vim.api.nvim_command('hi link CocSemColon CocSemPunctuation')
vim.api.nvim_command('hi link CocSemComma CocSemPunctuation')
vim.api.nvim_command('hi link CocSemComparison CocSemOperator')
vim.api.nvim_command('hi link CocSemConstParameter CocSemParameter')
vim.api.nvim_command('hi link CocSemDerive CocSemAttribute')
vim.api.nvim_command('hi link CocSemDeriveHelper CocSemAttribute')
vim.api.nvim_command('hi link CocSemDot CocSemPunctuation')
vim.api.nvim_command('hi link CocSemEscapeSequence CocSemString')
vim.api.nvim_command('hi link CocSemFormatSpecifier CocSemString')
vim.api.nvim_command('hi link CocSemGeneric CocSemUnknown')
vim.api.nvim_command('hi link CocSemLabel CocSemMacro')
vim.api.nvim_command('hi link CocSemLifetime CocSemTypeParameter')
vim.api.nvim_command('hi link CocSemLogical CocSemOperator')
vim.api.nvim_command('hi link CocSemMacroBang CocSemMacro')
vim.api.nvim_command('hi link CocSemParenthesis CocSemPunctuation')
vim.api.nvim_command('hi clear CocSemPunctuation')
vim.api.nvim_command('hi link CocSemSelfKeyword CocSemParameter')
vim.api.nvim_command('hi link CocSemSelfTypeKeyword CocSemType')
vim.api.nvim_command('hi link CocSemSemicolon CocSemPunctuation')
vim.api.nvim_command('hi link CocSemTypeAlias CocSemType')
vim.api.nvim_command('hi link CocSemToolModule CocSemNamespace')
vim.api.nvim_command('hi link CocSemUnion CocSemStruct')
vim.api.nvim_command('hi link CocSemUnresolvedReference Warning')

Without all this, large areas of the document do not get highlighted correctly, as the token types are unrecognised. Defaults similar to this could be provided by coc-rust-analyzer, but having to provide distributions like this for every LSP client is a lot of work.

[Veykril]

Ye, I didn't quite express myself either properly I think. I'll look into this a bit more and fix up a PR for settings regarding "unspecializing" punctuations and operators, and making use of decorator for attributes for now. Given from your list, the operators and punctuation ones are the majority of things it seems, and the easiest to fix. Will come back to this afterwards.

[frankplow]

@Veykril decorator will need this to be merged first I think: gluon-lang/lsp-types#246, also that repo still hasn't been updated to make 3.17 stable so it's behind a configuration flag.

EDIT: also operator is duplicated between the non-standard types and standard types at the minute. Not sure if there's really an issue with this (actually doing this may be an alternative if lsp-types does not update soon) but it could potentially be removed from the non-standard set here.