I presume you want this only in a UTF-8 locale?
Currently this is done by
static int SkipSpace(void)
{
int c;
while ((c = xxgetc()) == ' ' || c == '\t' || c == '\f')
/* nothing */;
return c;
}
in gram.c. We could make use of isspace and its wide-char equivalent
iswspace. However:
- there is the perennial debate over whether \v is whitespace.
R-lang says
Although not strictly tokens, stretches of whitespace characters
(spaces and tabs) serve to delimit tokens in case of ambiguity,
which suggests it has a minimal view of whitespace.
- iswspace is often rather unreliable. E.g. glibc says
The wide character class "space" always contains at least the space
character and the control characters '\f', '\n', '\r', '\t', '\v'.
and I think it usually does not contain other forms of spaces. More
seriously
The behaviour of iswspace() depends on the LC_CTYPE category of the
current locale.
so what is a space will depend on the encoding (hence my question about
UTF-8). And Ei-ji Makama was replaced iswspace on MacOS X, because
apparently it is wrongly implemented.
- it would complicate the parser as look-ahead would be needed (you would
need to read the next mbcs, check it it were whitespace and pushback if
needed). We do that elsewhere, though.
The only one of these 'spaces' I have much sympathy for is NBSP (which is
also fairly easy to generate in CP1252). It would be easy to add that.
Otherwise I am not convinced it is worth the work (and added uncertainty).
On Fri, 4 Jan 2008, hadley wickham wrote:
It would be nice if R ignored more unicode white space characters.
For example, if I have "\u2028" in a command (which I get from a
line-break in keynote) I get the following error:
qplot(carat, price, data = diamonds, ÿÿ colour=clarity)
Error: unexpected input in "qplot(carat, price, data = diamonds, ?"
And occasionally have such problems when copying and pasting from
emails as well.
Wikipedia lists the following codepoints as whitespace (I'm sure there
is a more definitive reference but I could not find one with some
quick googling):
U0009-U000D (Control characters, containing TAB, CR and LF)
Most of those are not normally considered whitespace.
U0020 SPACE
U0085 NEL
U00A0 NBSP
U1680 OGHAM SPACE MARK
U180E MONGOLIAN VOWEL SEPARATOR
U2000-U200A (different sorts of spaces)
U2028 LSP
U2029 PSP
U202F NARROW NBSP
U205F MEDIUM MATHEMATICAL SPACE
U3000 IDEOGRAPHIC SPACE
would it be possible for R to treat these all in the same way? (Or
does it already but my R is misconfigured?)
Hadley
--
Brian D. Ripley, [EMAIL PROTECTED]
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
