Add CancellationToken support to stream parsing methods

[Copilot]

Adds optional CancellationToken parameter to all stream-based parsing methods, enabling cancellation of long-running tokenization operations.

Changes

Base classes

• BaseTokenizer<TToken>: Added CancellationToken overloads for ParseAsync/Parse methods (4 new overloads)
• BaseSubTokenizer<TToken>: Added CancellationToken parameter to both ParseAsync overloads
• Changed abstract method signature: ParseAsync() → ParseAsync(CancellationToken ct)
• Non-cancellable overloads pass CancellationToken.None internally

Tokenizer implementations

• Replaced while(true) with while(!ct.IsCancellationRequested) in all tokenizers:
  • JsonTokenizer, XmlTokenizer, CSharpTokenizer, TypescriptTokenizer, SqlTokenizer, GenericTokenizer
  • MarkupTokenizer, InlineMarkupTokenizer, TableMarkupTokenizer

Tests

• Added cancellation test for each major tokenizer type (6 tests)

Usage

using var cts = new CancellationTokenSource();
using var stream = new MemoryStream(Encoding.UTF8.GetBytes(largeJson));

// Cancel after processing 100 tokens
await JsonTokenizer.Create().ParseAsync(stream, cts.Token, token => {
    ProcessToken(token);
    if (++count == 100) cts.Cancel();
});

Note: CancellationToken parameter is positioned before Action<TToken> onToken in all overloads.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add optional CancellationToken to Methods that use a stream as input, so that processing the stream can be cancelled</issue_title>
<issue_description>The public methods concerned are defined in BaseTokenizer<TToken>.

For the following methods create an overload with a CancellationToken as extra parameter. I want the extra parameter allways BEFORE the onToken parameter, because after is weird.

    public async Task<string> ParseAsync(Stream stream, Action<TToken> onToken) =>
        await ParseAsync(new StreamReader(stream), new StringBuilder(), onToken);

    public async Task<string> ParseAsync(Stream stream, Encoding encoding, Action<TToken> onToken) =>
        await ParseAsync(new StreamReader(stream, encoding), new StringBuilder(), onToken);

    public string Parse(Stream stream, Action<TToken> onToken) =>
        ParseAsync(stream, onToken).GetAwaiter().GetResult();

    public string Parse(Stream stream, Encoding encoding, Action<TToken> onToken) =>
        ParseAsync(stream, encoding, onToken).GetAwaiter().GetResult();

Make sure the new methods are fully documented.

To use the CancellationToken, Change the singature of the following method (in BaseTokenizer):

internal protected abstract Task ParseAsync();

To this:

internal protected abstract Task ParseAsync(CancellationToken ct);

If no CancellationToken is provided by the user/ method, use CancellationToken.None, on the calling part.

Now all classes that override this method will have 1 or 2 while(true) methods, replace them by while(ct.IsCancellationRequested).

Tokenizing is sometimes delegated to sub-tokenizers (See MarkupTokenizer / Codefence), so also make sure the CancellationToken is passed to the classes that inherit from BaseSubTokenizer.

I guess you have to add the CancellationToken to the two ParseAsync methods of BaseSubTokenizer:

    public async Task<string> ParseAsync(TextReader reader, string? stopDelimiter, Action<TToken> onToken)

    public async Task ParseAsync(TextReader reader, StringBuilder stringBuilder, string? stopDelimiter, Action<TToken> onToken)

Add xUnit tests for all tokenizers to check whether the Tokenizers can be succesfully cancelled.

It should all compile and all tests should succeed.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Add optional CancellationToken to Methods that use a stream as input, so that processing the stream can be cancelled #30

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.