inbelic wrote: I didn't implement the tokenizer because I found that the extra level of abstraction to be redundant/not beneficial with the StringRef operations. Looking at the DXC implementation, the usage of the Tokenizer is either GetAndMatchToken, or, getToken for an identifier with a switch on a small subset of tokens. These are effectively just StringRef::consume_front/StringSwitch with the buffer abstracted into the Tokenizer. Since we can just go through the buffer from left to right and construct the RootElements in place, then we will not reference a previous token, and so, defining/lexing an intermediate Token seems redundant. What aspects are you referring to that would warrant it?
https://github.com/llvm/llvm-project/pull/121799 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
