rsmith added a comment. Thinking about this some more: distinguishing between "macro expansion" and other cases seems like a proxy for "this token came from inside the preprocessor / lexer" versus "this token was provided by the user", which is also exactly what `IsNewToken` is supposed to capture. I don't think we need two separate flags here.
I also suspect that a significant amount of the cost is the additional function call on the hot path (the 1-parameter `Lex` wrapper function). I wonder if there's something that we could do to remove that from the hot path without duplicating the whole `Lex` function. Maybe we could add a flag to `Token`s to say that they're a replay of prior tokens, rather than being newly-created tokens, and make `TokenLexer` set that flag on tokens it produces if its token sequence came from a "replay" context (that's largely the same as your current "not a macro expansion" check, but there are probably cases where the preprocessor itself generates tokens for which that's wrong; we should have a flag on `EnterTokenStream` to indicate which mode we're in). We can then also change `CachingLex` to set the flag, and get rid of the `IsNewToken` flag too. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D59885/new/ https://reviews.llvm.org/D59885 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
