get rid of the TokenError field in Token#17
Conversation
KristofferC
force-pushed
the
kc/remove_err_field
branch
from
ea8efa7 to
76e43de
Compare
|
Rebased |
c42f
left a comment
c42f
left a comment
There was a problem hiding this comment.
Looks good. I think is_error(head::SyntaxHead) may need fixing after this, and I feel slightly worried by the special case in untokenize.
Somewhat unrelated — you'll notice that the way I've bridged the parser with the lexer is hardly ideal in terms of repetition and consistency, especially for kinds. I hope to clean this up at some point by changing the way kinds are declared, probably moving away from @enum and toward the string macro approach. When parsing, it's extremely helpful to have the kind names be self representing (ie, the kind of for is K"for").
|
|
||
| function untokenize(head::SyntaxHead; unique=true, include_flag_suff=true) | ||
| str = untokenize(kind(head); unique=unique) | ||
| str = is_error(kind(head)) ? "error" : untokenize(kind(head); unique=unique) |
There was a problem hiding this comment.
It seems a potentially dubious to special case this in here rather than in untokenize(::Kind). I assume something fails somehow without this?
There was a problem hiding this comment.
Right now, the parser expects any error to untokenize to "error". But now we have many error kinds. I could just add all of them to https://github.com/c42f/JuliaSyntax.jl/blob/77b4044218a7c7c5e34f8b1459a88fe6d405b64c/src/token_kinds.jl#L6 and have them sting to "error".
There was a problem hiding this comment.
Yes I think they should be added to _str_to_kind — this allows them to be accessed with the K macro.
|
I think let's merge this. I'll add some cleanup to |
2 participants
It is very rare for tokens to be errors but we still need to pay the cost of the
TokenErrorfield every time. This PR just moves theTokenErrorinto aKindand has a functioniserrorto check if the kind is of an error type.Made on top of #16