The tokenizer is the portion of the C++ code that breaks up the text into semantically meaningful chunks.
On Tue, Sep 23, 2014 at 6:58 PM, o1bigtenor <[email protected]> wrote: > > > On Tue, Sep 23, 2014 at 8:36 AM, Martin Blais <[email protected]> wrote: >> >> On Mon, Sep 22, 2014 at 10:42 PM, Craig Earls <[email protected]> wrote: >>> >>> If the third schema involves whitespace inside values then no. >>> Whitespace is used several other places for semantic separation. It >>> would be VERY difficult to add it inside numbers and still keep >>> performance anywhere like what it is. >> >> >> Actually, if the syntax never legally admits two numbers to appear in a >> row with only whitespace between them, it should be easy, it's only a matter >> of changing your tokenizer and should have negilgible impact on performance. >> > So that the list doesn't get loaded with 'related' topics - - - would you > please explain what you mean by 'changing your tokenizer' and ship it to me > off list. > > TIA > > Dee > > -- > > --- > You received this message because you are subscribed to the Google Groups > "Ledger" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- Craig, Corona De Tucson, AZ enderw88.wordpress.com -- --- You received this message because you are subscribed to the Google Groups "Ledger" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
