common : restore grammar-based rejection sampling

[ggerganov]

cont #17937
ref #18107 (comment)
rel #18135

I keep underestimating how much the grammar sampling is used and how slow it is. For now lets restore the old behaviour and I'll try to figure out a way to adapt #17004 to this.

[aldehir]

Works as it did before.

I'll research into other constrain decoding implementations. It does seem like a challenging problem to solve, though.

[pwilkin]

Dumb idea - what if we precomputed, for each grammar state, the entire set of accepted / rejected tokens?

[aldehir]

@pwilkin it's not dumb, I was considering this too but doing it lazily by caching the result.

The tricky part comes in handling utf8 codepoints that span tokens. So you'd need to consider that part of the state as well.

[pwilkin]

Fair point, but I think that can be optimized this way:
-> for most cases, you can wildcard the unicode codepoints (best case scenario, you can reject based on the first codepoint already if all characters from that codepoint are rejected)
-> for the cases where it matters, you can introduce pseudo-states - pairs of <grammar state, codepoints so far> - this will introduce extra memory overhead, but since most codepoints can be optimized away like in the point above, it shouldn't matter that much