-
Notifications
You must be signed in to change notification settings - Fork 5.9k
Add semantic tokens #39539
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add semantic tokens #39539
Conversation
|
We require contributors to sign our Contributor License Agreement, and we don't have @macmv on file. You can sign our CLA at https://zed.dev/cla. Once you've signed, post a comment here that says '@cla-bot check'. |
|
@cla-bot check |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
@macmv First of all, thank you for the work done! I would really like to see this feature in zed. |
|
Right now I only wired up the semantic tokens to the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delayed review — I was working on inlay hints to provide an example how such range-related LSP things are planned to be cached and used in the system.
I was not too meticulous about the protocol part, as hope @Veykril will check the PR too at some point.
Overall, seems like a step forward compared to #27556 as we have delta support and the editor-rendering part is somewhat more elaborate here.
I think it's perfectly fine to ignore the proto part for now (albeit there are collab/src/editor_tests.rs and local ssh to test on locally and progress on that).
To resume my comments, I think those steps need to be done:
- consolidate all LSP data caching and code near
LspStoreinstead ofeditorto share it — consider range requests - iron out editor store: when things are updated, range request story part (invalidation is somewhat a pain for inlay hints)
- add refresh support and multi buffer test
I think, we can try and consider inlay hints, chunks and whatever related logic added recently: it's not fully ironed out to be a generic storage for semantic tokens, but the LspStore part seems good enough — editor part seems ok to start off from.
Besides that, one huge gap here is the user-facing part of the feature:
- no settings (in default.json file we have a bunch of scattered entries, we need to pick something near lsp and highlights semantically, and add a way to turn things on/off, switch between tree-sitter, etc.)
- no documentation (we document settings mainly)
- seems worthy to add another entry to
for the feature visibility.
- last but not least, it would be superb to be able to do things like
"editor.semanticTokenColorCustomizations": {
"enabled": true,
"rules": {
"unresolvedReference": {
"foreground": "#c93f3f",
"fontStyle": "bold"
}
}
},
as VSCode allows.
This snippet turns rust-analyzer's highlights into a quite good and useful diagnostic.
I plan to work on inlay hints (there's an invalidation bug somewhere) and rainbow brackets further, so no active work and thinking on this PR yet.
Happy to discuss things further.
| } | ||
| } | ||
| } | ||
| highlight_endpoints.sort(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With
impl Ord for HighlightEndpoint {
fn cmp(&self, other: &Self) -> cmp::Ordering {
self.offset
.cmp(&other.offset)
.then_with(|| self.style.is_some().cmp(&other.style.is_some()))
}
}this part of code starts to become more dangerous with each extra semantics layer added.
The main question here to me is: how to unite both tree-sitter and semantics highlight so that people can customize them?
It is not unrealistic to assume there are people who want 0 tree-sitter used in such highlights: augmentsSyntaxTokens was mentioned, but no code is added to the configuration level yet.
Would it make sense to use a similar approach
zed/assets/settings/default.json
Lines 1519 to 1528 in 1e58fe7
| // Use LSP tasks over Zed language extension ones. | |
| // If no LSP tasks are returned due to error/timeout or regular execution, | |
| // Zed language extension tasks will be used instead. | |
| // | |
| // Other Zed tasks will still be shown: | |
| // * Zed task from either of the task config file | |
| // * Zed task from history (e.g. one-off task was spawned before) | |
| // | |
| // Default: true | |
| "prefer_lsp": true |
to highlights too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good point. I figured doing what "the lsp spec suggests" would be a good start, but configurability would be best.
How does this sound: add two flags, one to enable tree-sitter highlights and one to enable semantic token highlights. Then we enable augmentSyntaxTokens only when both are enabled and layer semantic token highlights on top of tree-sitter highlights. This behavior can all be documented in default.json.
Additionally, enabling semantic token highlights would be a per-language-server setting, and toggling tree-sitter highlights would be a per-language setting (both with global defaults).
I realize users could theoretically want tree-sitter highlights on top of semantic tokens, but at that point, I feel like they should disable semantic tokens for a specific language server instead. I hesitate to add more configuration than necessary, just because all the combinations of settings become difficult to test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add two flags, one to enable tree-sitter highlights and one to enable semantic token highlights
This sounds good.
To agree more with your last paragraph, I'd try to use a single one with some enum values — the main idea here is to have some "fallback" by default, so we start with semantic highlights on always, but if those are not supported by the language's server, we can fall back to tree-sitter.
I do not insist on this much if it starts to get hard to implement, but in the end it seems reasonable to have something like this (same Zed does with prettier and those tasks mentioned above).
This behavior can all be documented in default.json.
Great that it will be done, just want to mention that this file is 50% of the docs, there's also configuring-zed.md and other docs/ entries that get published to website and might be more visible for certain kinds of people, hence we should have both updated.
enabling semantic token highlights would be a per-language-server setting
With that, I'd rather not agree: we can as well make it language-dependent and spare users from remembering server names.
Also, somewhat plays towards the unified setting, if it works out.
| } | ||
| let token_range = excerpt.map_range_from_buffer(token_range); | ||
|
|
||
| if token_range.end <= range.start || token_range.start >= range.end { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Range is an exclusive range, but it does include its start — seeing both ends considered here is a bit odd, is it supposed to be so?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah good point, I think < and > are correct
crates/editor/src/display_map.rs
Outdated
| } | ||
|
|
||
| #[derive(Debug, Default)] | ||
| pub struct SemanticTokenView { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this repository, View is rather common name for entities that are displayed on-screen: there's ToolbarItemView, ModalView, StatusItemView, etc.
Here, we deal with one such "view" called editor element, where small parts of it are being colored differently, so the name seems a bit off.
Maybe, BufferSemanticTokens or SemanticTokens?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I ended up needing BufferSemanticTokens in lsp store, so how does PositionedSemanticTokens sound?
| let buffer_range = excerpt | ||
| .buffer() | ||
| .range_to_version(excerpt.map_range_to_buffer(range.clone()), &tokens.version); | ||
| for token in tokens.tokens_in_range(buffer_range) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tokens_in_range
I know that certain highlights (e.g. word highlight on hover) are applied after a debounce.
Based on my experience of fixing various obscure bugs like #40738 , it's not unreasonable to assume that editing and highlights might interleave in such manner, that there is a buffer's Global version mismatch between what's highlighted and what's stored in the data.
Here, it seems relatively harmless as would not panic due to anchor/snapshot manipulations, but still potentially buggy?
If so, some extra version check could help — ideally, accessors like
zed/crates/project/src/lsp_store.rs
Lines 12261 to 12274 in f0ac54e
| /// Gets the most recent LSP data for the given buffer: if the data is absent or out of date, | |
| /// new [`BufferLspData`] will be created to replace the previous state. | |
| pub fn latest_lsp_data(&mut self, buffer: &Entity<Buffer>, cx: &mut App) -> &mut BufferLspData { | |
| let (buffer_id, buffer_version) = | |
| buffer.read_with(cx, |buffer, _| (buffer.remote_id(), buffer.version())); | |
| let lsp_data = self | |
| .lsp_data | |
| .entry(buffer_id) | |
| .or_insert_with(|| BufferLspData::new(buffer, cx)); | |
| if buffer_version.changed_since(&lsp_data.buffer_version) { | |
| *lsp_data = BufferLspData::new(buffer, cx); | |
| } | |
| lsp_data | |
| } |
could help?
| dynamic_registration: Some(true), | ||
| }), | ||
| semantic_tokens: Some(SemanticTokensClientCapabilities { | ||
| dynamic_registration: Some(true), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to double check: I do not see register_server_capabilities adjusted, but it should be, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh good point, I missed that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't entirely understand register_server_capabilities for semantic tokens. Will the client send a registration for textDocument/semanticTokens/? or textDocument/semanticTokens/full?
I can't really find how this works in the spec, and I'm not sure how to go about testing this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dynamic registrations are somewhat convoluted indeed, I can only propose to check out #41929
I expected that to be very similar to
zed/crates/project/src/lsp_store.rs
Lines 12088 to 12094 in 64d782e
| "textDocument/documentSymbol" => { | |
| let options = parse_register_capabilities(reg)?; | |
| server.update_capabilities(|capabilities| { | |
| capabilities.document_symbol_provider = Some(options); | |
| }); | |
| notify_server_capabilities_updated(&server, cx); | |
| } |
Worst case — we can use Some(false) in the dynamic registration capability part.
| // We apply semantic tokens over tree-sitter highlighting, so | ||
| // let the server know that we are augmenting existing tokens. | ||
| augments_syntax_tokens: Some(true), | ||
| }), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should declare workspace/refresh support too and implement it, as we do that already for inlays and other cases.
| #[derive(Default, Debug, Clone)] | ||
| pub struct SemanticTokens { | ||
| /// Each value is: | ||
| /// data[5*i] - deltaLine: token line number, relative to the start of the previous token |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feels worth a few unit test around this logic (maybe, on a level above, but somewhere, we do have to test delta and other cases, e.g. the one described in a spec?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for adding tests both on the LSP part the "unit" tests on logic from this module.
Yet, I have failed to find any "unit" tests on applying delta requests on top of the partial one, and the spec seem to describe some example?
That looked somewhat complex to me, and if I'm not confusing things, would be great to add another test.
crates/project/src/lsp_store.rs
Outdated
|
|
||
| #[derive(Debug, Default)] | ||
| struct SemanticTokensData { | ||
| result_id: Option<String>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a bit odd to see this optional, we do know which server did we send the request to and got the response from.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, agreed. This was just so I could #[derive(Default)]. Now that LspData exists, it should be easy to get rid of this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, sorry, I got this mixed up with server_id. The result_id does need to be optional: it's an optional field in the spec. If a server doesn't support deltas, it won't send back result ids. But I have a similar server_id field in the SemanticTokens struct, which I will remove the optional from.
crates/project/src/lsp_store.rs
Outdated
| }) | ||
| } else { | ||
| let lsp_request_task = | ||
| self.request_lsp(buffer, LanguageServerToQuery::FirstCapable, request, cx); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So good to see full and delta support, thank you.
This is somewhat debatable though: in the world of frontend, a single *.ts file may have various language servers associated (tailwind-language-server, typescript-language-server, eslint, etc.) and it's possible to expect any of them may server additional tokens too?
I would repeat what current main does for inlay hints.
|
Thank you so much for the detailed review! It'll take me a few days to rebase/update everything, and this looks like a bunch of interesting stuff to read up on. I'm aware of the missing user-facing configs, and I think I'll try and tackle that while I'm updating these bits. This was mostly meant as a POC, as I was still just interested to see if this was possible. Being able to customize language-server-specific tokens looks super useful. I'd also definitely like to omit the proto part for now. This PR is getting quite long. Should I keep what I have already? Or just cut out the new protobufs entirely? And finally, do you prefer rebases or merges? |
SomeoneToIgnore
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It took me eons to get here, no rush (albeit now I'll try to keep up with the pace and review further changes timely).
Or just cut out the new protobufs entirely?
I'm not sure how possible this is, given that we'll have to use Lsp* infra?
We have to use LspQuery for properly made local requests, that drags some proto types one way or another.
I think we could literally copy paste document colors/inlays here: the latter is a bit more complex as involves some ranges handling, and the former is more or less the same modulo the LSP data is different.
I do not mind removing the layer/ignoring it until the very end if you find the way — in the end, we'll have to get a collab/src/tests/editor_tests.rs entry for remote functionality testing, which I can help with.
do you prefer rebases or merges
I think merges are more realistic given that this branch will be here for a while, but no preferences really — I prefer to concentrate on the feature itself and its semantics more.
| } | ||
| } | ||
| } | ||
| highlight_endpoints.sort(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add two flags, one to enable tree-sitter highlights and one to enable semantic token highlights
This sounds good.
To agree more with your last paragraph, I'd try to use a single one with some enum values — the main idea here is to have some "fallback" by default, so we start with semantic highlights on always, but if those are not supported by the language's server, we can fall back to tree-sitter.
I do not insist on this much if it starts to get hard to implement, but in the end it seems reasonable to have something like this (same Zed does with prettier and those tasks mentioned above).
This behavior can all be documented in default.json.
Great that it will be done, just want to mention that this file is 50% of the docs, there's also configuring-zed.md and other docs/ entries that get published to website and might be more visible for certain kinds of people, hence we should have both updated.
enabling semantic token highlights would be a per-language-server setting
With that, I'd rather not agree: we can as well make it language-dependent and spare users from remembering server names.
Also, somewhat plays towards the unified setting, if it works out.
crates/editor/src/display_map.rs
Outdated
| // [2]: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#semanticTokenTypes | ||
| let choices: &[&str] = match self.token_type(token_type)? { | ||
| // Types | ||
| "namespace" => &["namespace", "module", "type"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess here I lack the context — overall, there's definitely some configuration has to happen (as I've posted that VSCode token settings snippet for r-a) so not opposed to the idea overall.
crates/editor/src/editor.rs
Outdated
|
|
||
| fn semantic_tokens( | ||
| &self, | ||
| buffer_handle: Entity<Buffer>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deltas only work on the entire buffer. So, if the server supports deltas, range supports are basically never useful (as ranges cannot be diffed).
Did not know that, thank you: that complicates things even more, technically, as multi buffers come into play.
I's quite sad if we do some project search that hits many large files — then, when we scroll their excerpts into view, we will have to load the entire file to show a small (<20 lines, usually) excerpt per hit.
Also, the "what if some slow server starts to choke on large files" concern is still here.
Given such strong delta separation though, I think it's fine to move on now and concentrate on full files + deltas, this is a big change alone.
This also simplifies all caching/metadata quite a lot, given that we have no ranges to deal with.
But then, let's remove the range: Some(true), from our LSP capabilities
cc @Veykril as this is quite a pivoting point then, and you know the spec shenanigans here better
|
Ok, given #39539 (comment) I think we're getting quite close:
|
|
Alrighty, that sounds good. I'll polish this up and update this soon. |
1e58fe7 to
64d782e
Compare
|
I've posted a new version. This debounces updates in LspStore and aggregates semantic tokens from multiple servers correctly. This largely copies the behavior of document tokens. Notably, when requesting from multiple language servers, semantic tokens are only shown once all servers reply. For each server, it decides to send a full/delta request accordingly. I added some more tests:
I still need the Additionally, And next, I'm going to work on configs. I haven't started that yet. |
|
Just leaving my two cents: I hope that we are not locking ourselves out of implementing |
SomeoneToIgnore
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Superb, the new tests and most of the LspStore-related changes look solid, we're definitely on the right track.
One small concern is the text.rs-related changes, feels that we can inline this all into highlights-related code?
Additionally, latest_lsp_data tripped me up a bit, as it clears the previous lsp data every time the buffer changes. I changed this to keep semantic tokens around when updating the BufferLspData, but I'm not sure if that makes sense.
This seems somewhat odd, I've commented around the potentially hacky place to discuss it more.
Just to note, there's current_lsp_data to look up what's stored inside without invalidating, so that should help with whatever remaining issues, it seems.
And next, I'm going to work on configs. I haven't started that yet.
One small note here (given the C#-related language server comment above): maybe, we should disable semantic highlights by default, until we support range requests.
If we add the docs and that editor menu item I've mentioned before, that should be discoverable enough for the time being.
| }; | ||
|
|
||
| let buffer_id = buffer.read(cx).remote_id(); | ||
| // `semantic_tokens` gets debounced, so we can call this frequently. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have descended down into lsp_store.rs and did not notice any new .timer( call, can you point me to the debounce?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, yeah I didn't add the .timer() call. i'll put it in lsp store.
| dynamic_registration: Some(true), | ||
| }), | ||
| semantic_tokens: Some(SemanticTokensClientCapabilities { | ||
| dynamic_registration: Some(true), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dynamic registrations are somewhat convoluted indeed, I can only propose to check out #41929
I expected that to be very similar to
zed/crates/project/src/lsp_store.rs
Lines 12088 to 12094 in 64d782e
| "textDocument/documentSymbol" => { | |
| let options = parse_register_capabilities(reg)?; | |
| server.update_capabilities(|capabilities| { | |
| capabilities.document_symbol_provider = Some(options); | |
| }); | |
| notify_server_capabilities_updated(&server, cx); | |
| } |
Worst case — we can use Some(false) in the dynamic registration capability part.
crates/text/src/text.rs
Outdated
| false | ||
| } | ||
|
|
||
| pub fn offset_to_version(&self, offset: usize, version: &clock::Global) -> usize { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one is not really used anywhere but in tests?
range_from_version is used only once in the highlights-related places, can we have a function there instead of adding any public API here?
Those .unwrap() look somewhat concerning too — let's use Option and whatever else instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah sure thing, I'll move it. I think it started out needing private APIs from BufferSnapshot, but it doesn't now. I'll move it up to where highlights are built.
crates/project/src/lsp_store.rs
Outdated
| .entry(buffer_id) | ||
| .or_insert_with(|| BufferLspData::new(buffer, cx)); | ||
| if buffer_version.changed_since(&lsp_data.buffer_version) { | ||
| // Semantic tokens survive buffer version changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems odd and somewhat hacky: let's discuss things more.
Presumably, all this is needed to avoid flickering?
If so, can we do things similar to inlays/document colors, where we invalidate data & tasks in LspStore as soon as the version mismatch is found; yet we keep the old data in the Editor instead, to use in the rendering/wherever, and only update that after the next fetch task gets back the results.
Or, is it so that we're able to get proper deltas for changed documents?
Then, not hacky at all, just not obvious — let's mention that specifically instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is for deltas to work at all. It uses the result_id of the semantic tokens to send the delta request. Even if I only use current_lsp_data, other calls to latest_lsp_data (like in document colors) remove the previous result_id, and we don't end up sending delta requests.
I'll specifically call that out in a comment.
I don't think this is locking out of supporting range requests, but for servers that don't support deltas, I feel like this requires an entirely different approach. I'd imagine we would want an editor to query for specific ranges, as the scroll position of the editor changes the tokens it sees. That also means that as the editor scrolls, it would need to request additional ranges and then merge the tokens between multiple queries for rendering. And then, when the editor is modified, all the tokens would just get discarded. I think there is very little overlap between range logic and the delta logic that I've implemented. Oh, and I think minimap rendering would need to be changed. If the minimap is open, requesting "only the visible range" ends up being a large portion of the file anyways. I'm happy to disable semantic tokens until it's more complete. I don't want to bring language servers grinding to a halt with a new update. Alternatively, we could disable semantic tokens by default on servers where this would be a problem, and leave it enabled for other servers by default. |
64d782e to
40a795a
Compare
|
I did not dive into the code too much since last review, but in general looks very close to a good first iteration on the feature, nice! To double check: is token configuration like #42030 (and the one I've mentioned earlier, related to ra) something that will be a part of the configuration-related changes? If not, what's the hard part there, given that we seem to have all the hardcoded values listed? |
|
Wow, that's quite the first step for the tokens feature, I'd say. I wonder if I would also double check what VSCode does: we have The best part would be to document this both in the json and md docs — I can help with that, if needed, same as the RPC part. cx.executor().advance_clock(Duration::from_secs(LSP_QUERY_TIMEOUT));
cx.run_until_parked();in places to ensure the tokens are queried for. Will wait a week or so, and also @Veykril 's review as we're getting close and would be great to see if we've missed anything/confirm the direction. |
|
Thanks! I'm pretty happy with it. Oh, yes, that's a better name. I don't quite understand the Locally, I've got vscode imports wired up for To make this easy to use, can we just make setting the Oh that's good to know. I think the only thing that broke is when the editor was first created, it fetches semantic tokens now, so I had to change the tests slightly. Yup, I'll add it to the markdown docs. I just forgot to add those. |
|
Sorry, no good knowledge on themes. |
…ontains_partial_buffer_range
5903a19 to
56e7757
Compare



Closes #7450.
This PR adds semantic tokens from language servers to the editor.
With semantic tokens on the left, without on the right:

I find this a very useful feature for the following reasons:
.map()and not need to worry that it's a user-defined function.Overall, while it is a quality of life feature, I find code a lot easier to understand at a glance with semantic tokens.
There are still some things left to finish:
Edit: Perf got fixed by clamping the buffer ranges in
create_hightlight_endpointscorrectly. I don't notice any change now.Aside from that, it works nicely and highlights all the tokens I could think of.
In terms of conflicting with tree-sitter, these highlights are applied ontop of tree-sitter highlights. Additionally, the LSP spec documents how this should be handled. Within the initialization options of the semantic tokens provider, there is the following option:
I set this to
true, so language servers won't send us semantic tokens for obvious things like strings or numbers. In particular, here's how rust-analyzer handles that flag: https://github.com/rust-lang/rust-analyzer/blob/4ae99f0150c94f4bdf7192b4447f512ece3546fd/crates/rust-analyzer/src/lsp/to_proto.rs#L734-L752This is best reviewed commit-by-commit. Each commit builds on the last, and they're somewhat organized so that the project should compile between each commit.
Release Notes: