File tree Expand file tree Collapse file tree 2 files changed +2
-2
lines changed
py_src/tokenizers/pre_tokenizers Expand file tree Collapse file tree 2 files changed +2
-2
lines changed Original file line number Diff line number Diff line change @@ -421,7 +421,7 @@ class Split(PreTokenizer):
421421
422422 Args:
423423 pattern (:obj:`str` or :class:`~tokenizers.Regex`):
424- A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`.
424+ A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
425425 If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
426426 otherwise we consider is as a string pattern. For example `pattern="|"`
427427 means you want to split on `|` (imagine a csv file for example), while
Original file line number Diff line number Diff line change @@ -334,7 +334,7 @@ impl PyWhitespaceSplit {
334334///
335335/// Args:
336336/// pattern (:obj:`str` or :class:`~tokenizers.Regex`):
337- /// A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`.
337+ /// A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
338338/// If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
339339/// otherwise we consider is as a string pattern. For example `pattern="|"`
340340/// means you want to split on `|` (imagine a csv file for example), while
You can’t perform that action at this time.
0 commit comments