Lexer processes all tokens at once and stores into vector. We want to support processing per token whether we want to single-process or batch-process for the parser.
The lexer routine logic take advantage of the fact that we are simply pushing to a vector of tokens. There will be major restructuring to get rid of this dependency.
Lexer processes all tokens at once and stores into vector. We want to support processing per token whether we want to single-process or batch-process for the parser.
The lexer routine logic take advantage of the fact that we are simply pushing to a vector of tokens. There will be major restructuring to get rid of this dependency.