Snippet rework #14724

Jarcho · 2025-05-02T05:05:35Z

This is a further rework of our snippet accessing/creating code.

The general design goals with this:

Avoid allocating when not strictly necessary.
Make it easy to avoid linting if an invalid span is used.
Make it easier to debug when an invalid span is used.
Make it easy to minimize the number of lookups in the source map.
Make it easy to avoid creating intermediary compressed spans.
Make it as easy as possible to compose span adjustments/checks while maintaining the previous goals.

changelog: None

rustbot · 2025-05-02T05:05:40Z

rustbot has assigned @Alexendoo.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Jarcho · 2025-05-02T05:06:57Z

The first two commits are mainly renames. The third commit has all the actual changes.

Jarcho · 2025-05-02T05:36:54Z

Looking at MISSING_DOCS_IN_PRIVATE_ITEMS, that lint needs to be restructured a bit. Macros currently mess up it's implementation quite a bit.

Why is this not part of rustc in the first place?

Alexendoo · 2025-05-04T14:49:00Z

clippy_utils/src/source.rs

+/// Handle to a source file's text and a range within that file.
+///
+/// With debug assertions the range is checked to be a valid substring of the source text. Without
+/// assertions `None` will be returned from various functions when accessing the substring of the
+/// source text fails.
+#[derive(Clone)]
+pub struct SourceFileRange {
+    file: SourceText,
+    range: Range<RelativeBytePos>,
 }


Both SourceFileRange and SourceText can represent subsets of a source file, could they be merged into a single type?

Doing so would require that SourceFileRange always validate it's range. The main point of SourceText is that it's definitely a valid string.

They're also serving two different purpose. One is a substring of the source text, the other is a movable view of a whole file.

Always validating doesn't seem so bad to me, API wise it's some extra ?ing when using with_lo/with_hi

They're still modelling two completely different things. Even if only one of them is exposed using both to implement it is still useful.

Alexendoo · 2025-05-05T14:04:46Z

clippy_lints/src/ranges.rs

+        && let Some(span) = new_lhs.span.map_span(cx, |file| {
+            let src = file.with_hi(span.hi()).src_text()?;
            // Do not continue if we have mismatched number of parens, otherwise the suggestion is wrong
-            src.matches('(').count() == src.matches(')').count()
+            (src.matches('(').count() == src.matches(')').count()).then_some(file)
        })


with_hi mutating file here is confusing, there's similar elsewhere with trim_start/trim_end

Reusing names of non mutating methods while also returning a value makes it difficult to realise what's happening

Yeah, those need to be renamed.

github-actions · 2025-07-18T15:18:54Z

No changes for 1eddd75

This is blocking #14724 changelog: none

Jarcho · 2025-11-17T02:07:11Z

The API has changed again; this time to support splitting a range. Changes include:

get_source_text has been renamed to get_text. The "source" part of the name doesn't really add anything and just makes it longer. check_source_text was renamed similarly.
SourceFileRange has been replaced with SpanEditCx which no longer contains the range.
map_range's callback now takes both the edit context and the range as separate parameters and returns either a single range or an array of ranges.
Functions for shrinking a range based on it's contents have been replaced by a general &str -> &str transformation that recalculates the range positions. This can also return an array of strings.
Some entry points no longer load external sources. get_text and check_text are still loading external sources since they hit debug assertions otherwise.
A couple of general utils have been added (display and StrExt). They aren't specific to touching the source code, but they will mainly be used when working with it.

I might switch the range editing functions to be on the range, but that means an extra trait will need to be imported everywhere. range.with_leading_prefix(scx, "prefix") is nicer when chaining multiple, so it's probably worth it.

This is blocking #14724 as it triggers the debug assertions it adds. The inter item span parsing was introduced to solve #12197. This is better handled by just not linting anything within bodies. changelog: [`missing_docs_in_private_items`]: Don't lint items in bodies and automatically derived impls changelog: [`missing_docs_in_private_items`]: Better detect when things are accessible from the crate root changelog: [`missing_docs_in_private_items`]: Lint unnameable items which are accessible outside the crate

…heck_text`.

Rename `get_source_text` and `check_source_text` to `get_text` and `check_text`.

rustbot · 2025-12-11T10:45:12Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

rustbot · 2025-12-11T18:10:28Z

☔ The latest upstream changes (possibly 9e3e964) made this pull request unmergeable. Please resolve the merge conflicts.

Alexendoo · 2025-11-29T16:20:12Z

clippy_utils/src/source.rs

+/// Creates a type which implements `Display` by calling the specified function.
+#[inline]
+#[must_use]
+pub fn display(f: impl Fn(&mut fmt::Formatter<'_>) -> fmt::Result) -> impl fmt::Display {
+    struct S<T>(T);
+    impl<T: Fn(&mut fmt::Formatter<'_>) -> fmt::Result> fmt::Display for S<T> {
+        fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+            self.0(f)
+        }
+    }
+    S(f)
+}


This can be std::fmt::from_fn

Alexendoo · 2025-12-11T16:25:15Z

clippy_utils/src/source.rs

+/// A type representing a range in the `SourceMap`.
+pub trait SourceRange: Sized {
+    #[must_use]
+    fn into_range(self) -> Range<BytePos>;
+}
+impl SourceRange for Range<BytePos> {
+    #[inline]
+    fn into_range(self) -> Range<BytePos> {
+        self
    }
+}
+impl<T: SpanLike> SourceRange for T {
+    #[inline]
+    fn into_range(self) -> Range<BytePos> {
+        let data = self.data();
+        data.lo..data.hi
+    }
+}


SourceRange has one usage (in get_external_text), and it doesn't appear to be called on Range<BytePos>. We could simplify things by removing this trait and making SpanExt: SpanLike

SpanExt can't require SpanLike since it needs to be implemented for Range<BytePos>. into_range could just be part of SpanExt though. That separation is leftover from other designs.

Alexendoo · 2025-12-11T18:05:55Z

clippy_utils/src/source.rs

+    fn map_range<'sm, T: IntoSpans>(
        self,
        sm: impl HasSourceMap<'sm>,
-        f: impl for<'a> FnOnce(&'a SourceFile, &'a str, Range<usize>) -> Option<Range<usize>>,
-    ) -> Option<Range<BytePos>> {
-        map_range(sm.source_map(), self.into_range(), f)
+        f: impl for<'a> FnOnce(&'a SpanEditCx<'sm>, FileRange) -> Option<T>,
+    ) -> Option<T::Output>


IMO the T here is a bit too much abstraction, we could keep this concrete in Span/FileRange to make it easier to understand, then add a second method for the multiple case

The new method could also be changed to be concrete in (FileRange, FileRange), all uses are currently [FileRange; 2] as far as I can see, the tuple would allow e.g. split_once to work as well. More advanced cases (3+) could always be covered by mk_edit_cx

If we did the same for FileRangeExt::map_range_text we could remove both IntoSpans and IntoSubRanges

Alexendoo · 2025-12-11T18:28:54Z

clippy_utils/src/source.rs

+    /// |`0`    |`m1!`, `m2!`|None    |
+    #[inline]
+    #[must_use]
+    fn walk_to_ctxt(self, ctxt: SyntaxContext) -> Option<Span>


This could do with some documentation on how it differs from hygiene::walk_chain

For the most part it doesn't differ. It only returns None when the target context isn't in the current expansion chain and works around a dumb annoyance with range sugarings.

Alexendoo · 2025-12-11T18:32:55Z

clippy_utils/src/source.rs

+            if expn.call_site.ctxt() != ctxt {
+                let sp = hygiene::walk_chain(expn.call_site, ctxt);
+                (sp.ctxt() == ctxt).then_some(sp)
+            } else if matches!(expn.kind, ExpnKind::Desugaring(DesugaringKind::RangeExpr)) {


What's unique about range desugaring that it's handled here? Should it be changed in rustc instead?

If I'm reading it right it also only applies when the outermost expn is said desugaring, what about a range in a macro?

This is needed to keep the parenthesis on a range expression. The call site of (x..y) is only x..y.

You are reading this wrong. The special case only works happens if the the target context is the immediate parent of the range desugaring. Note the outer_expn_data is a badly named function. The "outer" part is outermost from the root.

Alexendoo · 2025-12-11T18:36:06Z

clippy_utils/src/source.rs

+    /// Walks this span up the macro call chain to the root context.
+    ///
+    /// See `walk_to_ctxt` for details.
+    #[inline]
+    #[must_use]
+    fn walk_to_root(self) -> Span


Similar to walk_to_ctxt, could do with answering how it differs from Span::source_callsite

Alexendoo · 2025-12-11T18:39:43Z

clippy_utils/src/source.rs

+    /// validation.
+    #[inline]
+    #[cfg_attr(debug_assertions, track_caller)]
+    pub fn dbg_check_range(&self, old: Option<FileRange>, new: FileRange) {


Can be private

Alexendoo · 2025-12-11T18:40:49Z

clippy_utils/src/source.rs

+pub trait StrExt {
+    /// Gets the substring which ranges from the start of the first match of the pattern to the end
+    /// of the second match. Returns `None` if the pattern doesn't occur twice.
+    fn find_bounded_inclusive(&self, pat: impl Pattern) -> Option<&Self>;
+
+    /// Gets the non-overlapping prefix and suffix. Returns `None` if the string doesn't start with
+    /// the prefix or end with the suffix.
+    ///
+    /// The prefix will be taken first, with the suffix taken from the remainder of the string.
+    fn get_prefix_suffix<P>(&self, prefix: impl Pattern, suffix: P) -> Option<[&Self; 2]>
+    where
+        P: Pattern,
+        for<'a> P::Searcher<'a>: ReverseSearcher<'a>;
+
+    /// Splits a string into a prefix and everything proceeding it. Returns `None` if the string
+    /// doesn't start with the prefix.
+    fn split_prefix(&self, pat: impl Pattern) -> Option<[&Self; 2]>;
+
+    /// Splits a string into a suffix and everything preceding it. Returns `None` if the string
+    /// doesn't end with the suffix.
+    fn split_suffix<P>(&self, pat: P) -> Option<[&Self; 2]>
+    where
+        P: Pattern,
+        for<'a> P::Searcher<'a>: ReverseSearcher<'a>;
+}


Some examples would be great

Alexendoo · 2025-12-11T18:54:10Z

clippy_utils/src/source.rs

+    ///
+    /// The prefix will be taken first, with the suffix taken from the remainder of the string.


Can the order change the result here?

A Pattern impl could cause this to matter, but none of the ones from std will.

Alexendoo · 2025-12-12T15:45:56Z

clippy_lints/src/double_parens.rs

+    if let Some((scx, range)) = inner.span.mk_edit_cx(cx)
+        && let Some(range) = range.with_trailing_whitespace(&scx)
+        && let Some(range) = range.with_leading_whitespace(&scx)
+        && let Some(range) = range.with_trailing_match(&scx, ')')
+        && range.with_leading_match(&scx, '(').is_some()


Is there a reason this isn't a map_range? Either way it would be good to add some advice to mk_edit_cx about when you might want to use it compared to the other utilities

The result of map_range wouldn't be used. This avoids the FileRange to Span transition.

rustbot assigned Alexendoo May 2, 2025

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties label May 2, 2025

Jarcho force-pushed the source_rework branch from d67c058 to 8a20111 Compare May 2, 2025 05:06

Jarcho force-pushed the source_rework branch 2 times, most recently from 91b8913 to b336277 Compare May 2, 2025 05:13

This comment has been minimized.

Sign in to view

Alexendoo reviewed May 4, 2025

View reviewed changes

Alexendoo reviewed May 5, 2025

View reviewed changes

Jarcho mentioned this pull request May 7, 2025

Rework missing_docs_in_private_items #14741

Merged

Jarcho force-pushed the source_rework branch from b336277 to 4d7f663 Compare May 15, 2025 09:21

Jarcho force-pushed the source_rework branch 4 times, most recently from ae6dc9d to 8fe48da Compare July 18, 2025 13:39

This was referenced Jul 18, 2025

Simplify must_use_candidate spans #15310

Merged

Fix empty_with_brackets span handling #15311

Merged

Jarcho force-pushed the source_rework branch from 8fe48da to 212dfee Compare July 18, 2025 15:09

This comment has been minimized.

Sign in to view

rustbot added has-merge-commits PR has merge commits, merge with caution. S-waiting-on-author Status: This is awaiting some action from the author. (Use `@rustbot ready` to update this status) labels Jul 18, 2025

Jarcho added the S-blocked Status: marked as blocked ❌ on something else such as an RFC or other implementation work label Jul 18, 2025

github-merge-queue bot pushed a commit that referenced this pull request Jul 19, 2025

Fix empty_with_brackets span handling (#15311)

5acb1d4

This is blocking #14724 changelog: none

github-merge-queue bot pushed a commit that referenced this pull request Jul 19, 2025

Simplify must_use_candidate spans (#15310)

f85cdbb

This is blocking #14724 changelog: none

Jarcho force-pushed the source_rework branch from 212dfee to e63b476 Compare July 19, 2025 20:29

rustbot removed S-waiting-on-author Status: This is awaiting some action from the author. (Use `@rustbot ready` to update this status) has-merge-commits PR has merge commits, merge with caution. labels Jul 19, 2025

Jarcho mentioned this pull request Sep 27, 2025

Rework the suspicious formatting lints. #12980

Open

Jarcho mentioned this pull request Nov 6, 2025

Error applying suggestions (range underflow) #15787

Open

Jarcho force-pushed the source_rework branch from dfd4cf7 to 8e690e3 Compare November 8, 2025 20:23

This comment has been minimized.

Sign in to view

Jarcho force-pushed the source_rework branch 2 times, most recently from b21e0f0 to 3f4330b Compare November 16, 2025 23:36

This comment has been minimized.

Sign in to view

Jarcho force-pushed the source_rework branch 2 times, most recently from 3f6aaae to bf9f4e9 Compare November 17, 2025 01:02

Jarcho force-pushed the source_rework branch from bf9f4e9 to fe0912a Compare November 18, 2025 04:15

This comment has been minimized.

Sign in to view

Jarcho removed the S-blocked Status: marked as blocked ❌ on something else such as an RFC or other implementation work label Nov 18, 2025

Jarcho force-pushed the source_rework branch 6 times, most recently from 51934e2 to 0cff672 Compare November 19, 2025 19:36

This comment has been minimized.

Sign in to view

Jarcho added 4 commits December 11, 2025 05:29

Change HasSession to HasSourceMap

b516d1c

Rename SpanRangeExt to SpanExt

e586f24

Rename get_source_text and check_source_text to get_text and `c…

482afd0

…heck_text`.

Rework clippy_utils::source.

1eddd75

Rename `get_source_text` and `check_source_text` to `get_text` and `check_text`.

Jarcho force-pushed the source_rework branch from 0cff672 to 1eddd75 Compare December 11, 2025 10:45

Alexendoo reviewed Dec 12, 2025

View reviewed changes

		///
		/// The prefix will be taken first, with the suffix taken from the remainder of the string.

Snippet rework #14724

Are you sure you want to change the base?

Snippet rework #14724

Conversation

Jarcho commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented May 2, 2025

Uh oh!

Jarcho commented May 2, 2025

Uh oh!

Jarcho commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

github-actions bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Jarcho commented Nov 17, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

rustbot commented Dec 11, 2025

Uh oh!

rustbot commented Dec 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jarcho Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Jarcho commented May 2, 2025 •

edited

Loading

Jarcho commented May 2, 2025 •

edited

Loading

github-actions bot commented Jul 18, 2025 •

edited

Loading

Jarcho Dec 13, 2025 •

edited

Loading