Skip to content

Emoicons tokenization problem #1088

@Andras7

Description

@Andras7

Here it is an example: "hey playa!:):3.....@shaq can you still dunk?#old🍕🍔😵LOL"

I think there we can use a pretty obvious rule :)

Metadata

Metadata

Assignees

No one assigned

    Labels

    🌙 nightlyDiscussion and contributions related to nightly builds

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions