Add support for text-separator customization

[Fronix]

Pull Request

Related issue

Fixes #482

What does this PR do?

• Adds support for the text-separator settings

PR checklist

Please check if your PR fulfills the following requirements:

• Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
• Have you read the contributing guidelines?
• Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

[ahmednfwela]

Thanks for the PR

One change I would like to suggest, is that in general it's better to use IEnumerable<...> for function inputs and List<...> for function outputs

so for example:

public async Task<List<string>> GetSeparatorTokensAsync(CancellationToken cancellationToken = default)

and

public async Task<TaskInfo> UpdateNonSeparatorTokensAsync(IEnumerable<string> nonSeparatorTokens, CancellationToken cancellationToken = default)

[Fronix]

Thanks for the PR

One change I would like to suggest, is that in general it's better to use IEnumerable<...> for function inputs and List<...> for function outputs

so for example:

public async Task<List<string>> GetSeparatorTokensAsync(CancellationToken cancellationToken = default)

and

public async Task<TaskInfo> UpdateNonSeparatorTokensAsync(IEnumerable<string> nonSeparatorTokens, CancellationToken cancellationToken = default)

I agree 👍

[ahmednfwela]

bors merge

[meili-bors]

Build succeeded:

• format-check
• integration-tests