Hello all, I'm trying to implement a tokenizer that will behave differently on different parts of the incoming stream. For example, for the first X words in the stream I would like to use one tokenizing algorithm, while for the rest of the stream a different tokenizing algorithm will be used.
What is the best way to implement that? Where should I store this stream-related data? Thanks, Yuri
