Add optional CancellationToken to Methods that use a stream as input, so that processing the stream can be cancelled

[crwsolutions]

The public methods concerned are defined in BaseTokenizer<TToken>.

For the following methods create an overload with a CancellationToken as extra parameter. I want the extra parameter allways BEFORE the onToken parameter, because after is weird.

    public async Task<string> ParseAsync(Stream stream, Action<TToken> onToken) =>
        await ParseAsync(new StreamReader(stream), new StringBuilder(), onToken);

    public async Task<string> ParseAsync(Stream stream, Encoding encoding, Action<TToken> onToken) =>
        await ParseAsync(new StreamReader(stream, encoding), new StringBuilder(), onToken);

    public string Parse(Stream stream, Action<TToken> onToken) =>
        ParseAsync(stream, onToken).GetAwaiter().GetResult();

    public string Parse(Stream stream, Encoding encoding, Action<TToken> onToken) =>
        ParseAsync(stream, encoding, onToken).GetAwaiter().GetResult();

Make sure the new methods are fully documented.

To use the CancellationToken, Change the singature of the following method (in BaseTokenizer):

internal protected abstract Task ParseAsync();

To this:

internal protected abstract Task ParseAsync(CancellationToken ct);

If no CancellationToken is provided by the user/ method, use CancellationToken.None, on the calling part.

Now all classes that override this method will have 1 or 2 while(true) methods, replace them by while(ct.IsCancellationRequested).

Tokenizing is sometimes delegated to sub-tokenizers (See MarkupTokenizer / Codefence), so also make sure the CancellationToken is passed to the classes that inherit from BaseSubTokenizer.

I guess you have to add the CancellationToken to the two ParseAsync methods of BaseSubTokenizer:

    public async Task<string> ParseAsync(TextReader reader, string? stopDelimiter, Action<TToken> onToken)

    public async Task ParseAsync(TextReader reader, StringBuilder stringBuilder, string? stopDelimiter, Action<TToken> onToken)

Add xUnit tests for all tokenizers to check whether the Tokenizers can be succesfully cancelled.

It should all compile and all tests should succeed.