[Python-Dev] PEP 3101 implementation vs. documentation
olation-based strategy, which I think would be a
shame, because of the way it chops up the string.
But I also think the present behavior is extremely counterintuitive,
unnecessarily complex, and illogical (even unpythonic!). Isn't it
bizarre that (g) should work, given what (e) and (f) do? Isn't it
strange that (b) and (c) should fail, given that there's no real
reason for them to do so---no ambiguity that has to be avoided? And
something's gotta give; the documentation and the implementation do
not correspond.
Beyond the counterintuitiveness of the present implementation, it is
also, I think, self-inconsistent. (e) and (f) fail because the
interior brace is treated as starting or ending a replacement field,
even though interior replacement fields aren't allowed in that part of
a replacement field. (g) succeeds because the braces are balanced:
they are treated at one point as if they were part of a replacement
field, and at another (correctly) as if they are not. But this makes
the failure of (e) and (f) unaccountable. It would not have been
impossible for the PEP to allow interior replacement fields anywhere,
and not just in the format spec, in which case we might have had this:
(g') "{0[{][}]}".format(range(10), **{'][':4}) --> '3'
or this:
(g'') "{0[{][}]}".format({'4':3}, **{'][':4}) --> '3'
or something with that general flavor.
As far as I can tell, the only way to consistently maintain that (e)
and (f) should fail requires that one take (g') or (g'') to be
correct: either the interior braces signal replacement fields (hence
must be balanced) or they don't (or they're escaped).
Regarding the documentation, it could of course be rendered correct by
changing *it*, rather than the implementation. But wouldn't it be
tremendously weird to have to explain that, in the part of the
replacement field preceding the conversion, you can't employ any curly
braces, unless they're balanced, and you can't employ ':' or '!' at
all, even though they have no semantic significance? So these are out:
{0[{].foo}
{0[}{}]}
{0[a:b]}
But these are in:
{0[{}{}]}
{0[{{].foo.}}} (k)
((k) does work, if you give it an object with the right structure,
though it probably should not.)
And, moreover, someone would then have to actually change the
documentation, whereas there's a patch already, attached to the bug
report linked way up at the top of this email, that makes (a)--(g) all
work, leaves (i) and (j) as they are, and has the welcome side-effect
of making (k) *not* work (if any code anywhere relies on (k) or things
like it working, I will be very surprised: anyway the fact that (k)
works is, technically, undocumented). It is also quite simple. It
doesn't effect the built-in format(), because the built-in format() is
concerned only with the format-specifier part of the replacement field
and not all the stuff that comes before that, telling str.format
*what* object to format.
Thanks for reading,
--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3101 implementation vs. documentation
On Sat, Jun 11, 2011 at 2:16 AM, Nick Coghlan wrote:
> On Sat, Jun 11, 2011 at 7:15 AM, Ben Wolfson wrote:
> To summarise (after both the above post and the discussion on the tracker)
Thanks for the summary!
>
> That would leave us with the following set of rules for name fields:
>
> 1. Numeric fields start with a digit and are terminated by any
> non-numeric character.
>
> 2. An identifier name field is terminated by any one of:
> '}' (terminates the replacement field, unless preceded by a
> matching '{' character, in which case it is ignored and included in
> the string)
> '!' (terminates identifier field, starts conversion specifier)
> ':' (terminates identifier field, starts format specifier)
> '.' (terminates identifier field, starts new identifier field for
> subattribute)
> '[' (terminates identifier field, starts index field)
>
> 3. An index field is terminated by ']' (subsequent character will
> determine next field)
A minor clarification since I mentioned a patch: the patch as it
exists implements *these*---Nick's---semantics. That is, it will allow
these:
"{0.{a}}".format(x)
"{0.{[{].}}".format(x)
But not this, because it keeps current brace-matching in this context:
"{0.{a}".format(x)
And it treats this:
"{0.a}}".format(x)
as the markup "{0.a}" followed by the character data "}".
The patch would have to be changed to turn off brace balancing in name
fields as well.
In either case there would be potential breakage, since this:
"{0[{}.}}".format(...)
currently works, but would not work anymore, under either set of
rules. (The likelihood that this potential breakage would anywhere be
actual breakage is pretty slim, though.)
> Note that brace-escaping currently doesn't work inside name fields, so
> that should also be fixed:
>
>>>> "{0[{{]}".format({'{':1})
> Traceback (most recent call last):
> File "", line 1, in
> ValueError: unmatched '{' in format
>>>> "{a{{}".format(**{'a{':1})
> Traceback (most recent call last):
> File "", line 1, in
> ValueError: unmatched '{' in format
This is a slightly different issue, though, isn't it? As far as I can
tell, if the brace-matching rules are kept in place, there would never
be any *need* for escaping. You can't have an internal replacement
field in this part of the replacement field, so '{' can always safely
be assumed to be Just a Brace and not the start of a replacement
field, regardless of whether it's doubled, and '}' will either be in
an index field (where it can't have the significance of ending the
replacement field) or it will be (a) the end of the replacement field
or (b) not the end of the replacement field because matched by an
earlier '{'. So there would never be any role for escaping to play.
There would be a role for escaping if the rules for name fields are
that '}' terminates them, no matching done; then, you could double
them to get a '}' in the name field. But, to be honest, that strikes
me as introducing a lot of heavy machinery for very little gain;
opening and closing braces would have to be escaped to accomodate this
one thing. And it's not as if you can escape ']' in index
fields---which would be a parallel case. It seems significantly
simpler to me to leave the escaping behavior as it is in this part of
the replacement field.
--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3101 implementation vs. documentation
On Sat, Jun 11, 2011 at 4:29 PM, Greg Ewing wrote:
> Ben Wolfson wrote:
>>
>> You can't have an internal replacement
>> field in this part of the replacement field, so '{' can always safely
>> be assumed to be Just a Brace and not the start of a replacement
>> field, regardless of whether it's doubled,
>
> I'm worried that the rules in this area are getting too
> complicated for a human to follow. If braces are allowed
> as plain data between square brackets and/or vice versa,
> it's going to be a confusing mess to read, and there will
> always be some doubt in the programmer's mind as to whether
> they have to be escaped somehow or not.
>
> I'm inclined to think that any such difficult cases should
> simply be disallowed. If the docs say an identifier is required
> someplace, the implementation should adhere strictly to that.
There are two cases with the braces: attribute selection and item
selection. The docs say that attributes should be identifiers, and
that the argument name should be an integer or an identifier, but that
the item selector can essentially be an arbitrary string as long as it
doesn't contain ']', which indicates its end. The docs as they stand
suggest that braces in item selectors should be treated as plain data:
"""
Format strings contain “replacement fields” surrounded by curly braces
{}. Anything that is not contained in braces is considered literal
text, which is copied unchanged to the output. If you need to include
a brace character in the literal text, it can be escaped by doubling:
{{ and }}.
"""
Since it mentions escaping only in the context of how to get a brace
in literal text rather than in a replacement field.
Current behavior is to perform escapes in the format spec part of a
replacement field, and that, I think, makes sense, since there can be
an internal replacement field there. However, it's still pretty simple
to tell whether braces need to be escaped or not: a brace in the
format spec does need to be escaped, a brace before the format spec
doesn't.
> It's not *that* hard to parse an indentifier properly, and
> IMO any use case that requires putting arbitrary characters
> into an item selector is abusing the format mechanism and
> should be redesigned to work some other way.
If by "item selector" you mean (using the names from the grammar in
the docs) the element_index, I don't see why this should be the case;
dictionaries can contain non-identified keys, after all. If you mean
the attribute_name (and arg_name) parts, then requiring an identifier
(or an integer for arg_name) makes a lot more sense.
I assume that Talin had some reason for stating otherwise in the PEP
(one of the few things that does get explicitly said about the
field_name part), but I'm kind of at a loss for why; you would need to
have a custom __getattribute__ to exploit it, and it would be a lot
less confusing just to use __getitem__.
--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3101 implementation vs. documentation
On Mon, Jun 13, 2011 at 5:36 PM, Greg Ewing wrote:
> Ben Wolfson wrote:
>
>> If by "item selector" you mean (using the names from the grammar in
>> the docs) the element_index, I don't see why this should be the case;
>> dictionaries can contain non-identified keys, after all.
>
> Of course they can, but that's not the point. The point is
> that putting arbitrary strings between [...] in a format
> spec without any form of quoting or requirement for bracket
> matching leads to something that's too confusing for humans
> to read.
But there is a requirement for bracket matching: the "[" that opens
the element_index is matched by the next "]". Arguably (as Terry Reedy
said) this is also a form of quoting, in which the square brackets are
the quotation operators. It seems no more confusing to me than
allowing arbitrary strings between in '"..."'; those quotation marks
aren't even oriented. (Admittedly, syntax highlighting helps there.)
Compared to this: "{0: ^+#10o}", a string like this: "this is normal
text, but {e.underline[this text is is udnerlined {sic}!]}---and we're
back to normal now" is pretty damn readable to this human, nor do I
see what about the rule "when you see a [, keep going until you see a
]" is supposed to be insuperably confusing. (Compare---not that it
helps my case in regard to readability---grouping in regular
expressions, where you don't usually have the aid of special syntax
highlighting inside the string; you see a '(', you know that you've
encountered a group which continues until the next (unescaped!) ')'.
The stuff that comes inside the parentheses might look like line
noise---and the whole thing might look like line noise---but *that*
rule about the structure of a regexp is pretty straightforward.)
> IMO the spec should be designed so that the format string
> can be parsed using the same lexical analysis rules as
> Python code. That means anything that is meant to "hang
> together" as a single unit, such as an item selector,
> needs to look like a single Python token, e.g. an integer
> or identifier.
If that's the rationale, why not change the spec so that instead of this:
"{0[spam]}"
You do this:
"{0['spam']}"
? Hangs together; single Python token. Bonus: it would make it
possible for this to work:
(a) "{0['12']}".format({'12': 4})
whereas currently this:
"{0[12]}".format(...)
passes the integer 12 to __getitem__, and (a) passes the string "'12'".
(Discovery: the "abuse" of the format mechanism I want to perpetrate
via element_index can also be perpetrated with a custom __format__
method:
>>> class foo:
...def __format__(self, a): return a * 2
...
>>> "hello {0::![} world".format(foo())
'hello :![:![ world'
. So any reform to make it impossible to use str.format creatively
will have to be fairly radical. I actually think that my intended
abuse is actually a perfectly reasonable use, but it would be
disallowed if only integers and identifiers can be in the
element_index field.)
--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3101 implementation vs. documentation
FWIW, new patches have been attached to the bug report
(http://bugs.python.org/issue12014), one of which is intended to bring
behavior in line with the documentation, and the other of which is
intended to implement Greg Ewing's proposal to allow only identifiers
(or integers) in the arg_name, attribute_name, and element_index
sections.
On Fri, Jun 10, 2011 at 2:15 PM, Ben Wolfson wrote:
> Hello,
>
> I'm writing because discussion in a bug report I submitted
> (<http://bugs.python.org/issue12014>) has suggested that, insofar as
> at least part of the issue revolves around the interpretation of PEP
> 3101, that aspect belonged on python-dev. In particular, I was told
> that the PEP, not the documentation, is authoritative. Since I'm the
> one who thinks something is wrong, it seems appropriate for me to be
> the one to bring it up.
>
> Basically, the issue is that the current behavior of str.format is at
> variance at least with the documentation
> <http://docs.python.org/library/string.html#formatstrings>, is almost
> certainly at variance with PEP3101 in one respect, and is in my
> opinion at variance with PEP3101 in another respect as well, regarding
> what characters can be present in what the grammar given in the
> documentation calls an element_index, that is, the bit between the
> square brackets in "{0.attr[idx]}".
>
> Both discovering the current behavior and interpreting the
> documentation are pretty straightforward; interpreting what the PEP
> actually calls for is more vexed. I'll do the first two things first.
> TOC for the remainder:
>
> 1. What does the current implementation do?
> 2. What does the documentation say?
> 3. What does the PEP say? [this part is long, but the PEP is not
> clear, and I wanted to be thorough]
> 4. Who cares?
>
> 1. What does the current implementation do?
>
> Suppose you have this dictionary:
>
> d = {"@": 0,
> "!": 1,
> ":": 2,
> "^": 3,
> "}": 4,
> "{": {"}": 5},
> }
>
> Then the following expressions have the following results:
>
> (a) "{0[@]}".format(d) --> '0'
> (b) "{0[!]}".format(d) --> ValueError: Missing ']' in format string
> (c) "{0[:]}".format(d) --> ValueError: Missing ']' in format string
> (d) "{0[^]}".format(d) --> '3'
> (e) "{0[}]}".format(d) --> ValueError: Missing ']' in format string
> (f) "{0[{]}".format(d) --> ValueError: unmatched '{' in format
> (g) "{0[{][}]}".format(d) --> '5'
>
> Given (e) and (f), I think (g) should be a little surprising, though
> you can probably guess what's going on and it's not hard to see why it
> happens if you look at the source: (e) and (f) fail because
> MarkupIterator_next (in Objects/stringlib/string_format.h) scans
> through the string looking for curly braces, because it treats them as
> semantically significant no matter what context they occur in. So,
> according to MarkupIterator_next, the *first* right curly brace in (e)
> indicates the end of the replacement field, giving "{0[}". In (f), the
> second left curly brace indicates (to MarkupIterator_next) the start
> of a *new* replacement field, and since there's only one right curly
> brace, it complains. In (g), MarkupIterator_next treats the second
> left curly brace as starting a new replacement field and the first
> right curly brace as closing it. However, actually, those braces don't
> define new replacement fields, as indicated by the fact that the whole
> expression treats the element_index fields as just plain old strings.
> (So the current implementation is somewhat schizophrenic, acting at
> one point as if the braces have to be balanced because '{[]}' is a
> replacement field and at another point treating the braces as
> insignificant.)
>
> The explanation for (b) and (c) is that parse_field (same source file)
> treats ':' and '!' as indicating the end of the field_name section of
> the replacement field, regardless of whether those characters occur
> within square brackets or not.
>
> So, that's what the current implementation does, in those cases.
>
> 2. What does the documentation say?
>
> The documentation gives a grammar for replacement fields:
>
> """
> replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"
> field_name ::= arg_name ("." attribute_name | "[" element_index "]")*
> arg_nam
Re: [Python-Dev] Handling linker scripts reached when dynamically loading a module
On Sat, Sep 10, 2011 at 4:35 PM, Howard B. Golden
wrote:
>
> So, in Python, this is likely to only affect users calling packages
> using ctypes. (This corresponds to GHCi loading an unversioned library,
> e.g., "ghci -lm" which would load the current version of the math
> library into the GHC interpreter.)
And it does do so on Gentoo:
$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 17:43:52)
[GCC 4.4.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import cdll
>>> cdll.LoadLibrary('libpthread.so')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/ctypes/__init__.py", line 431, in LoadLibrary
return self._dlltype(name)
File "/usr/lib/python2.6/ctypes/__init__.py", line 353, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/libpthread.so: invalid ELF header
>>> cdll.LoadLibrary('libpthread.so.0')
>>>
$ cat /usr/lib/libpthread.so
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libpthread.so.0 /usr/lib/libpthread_nonshared.a )
--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Warnings
On Wed, Dec 7, 2011 at 11:00 AM, Ethan Furman wrote: > > No, of course not -- although it might /affect/ said reader by causing > him/her to think, "I don't think that word means what you think it means..." > ;) > > Seriously, it's best to use the correct words with the correct meanings. If > someone is willing to fix it, let them. I'm sure this hypothetical reader will then look "assure" up in the OED and find this: 5. To make certain the occurrence or arrival of (an event); to ensure. -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] str.format implementation
Hi,
I'm hoping to get some kind of consensus about the divergences between
the implementation and documentation of str.format
(http://mail.python.org/pipermail/python-dev/2011-June/111860.html and
the linked bug report contain examples of the divergences). These
pertain to the arg_name, attribute_name, and element_index fields of
the grammar in the docs:
replacement_field ::= "{" [field_name] ["!" conversion] [":"
format_spec] "}"
field_name::= arg_name ("." attribute_name | "["
element_index "]")*
arg_name ::= [identifier | integer]
attribute_name::= identifier
element_index ::= integer | index_string
index_string ::= +
Nothing definitive emerged from the last round of discussion, and as
far as I can recall there are now three proposals for what kind of
changes might be worth making:
(1) the implementation should conform to the docs;*
(2) like (1) with the change that element_index should be changed to
"integer | identifier" (rendering index_string otiose);
(3) like (1) with the change that index_string should be changed to
''.
* the docs link "integer" to
http://docs.python.org/reference/lexical_analysis.html#grammar-token-integer
but the current implementation only allows decimal integers, which
seems reasonable and worth retaining.
(2) was suggested by Greg Ewing on python-dev and (3) by Petri
Lehtinen in the bug report. (Petri actually suggested that braces be
disallowed except for the nesting in the format_spec, but it comes to
the same thing.)
None of these should be difficult to implement; patches exist for (1)
and (2). (2) and (3) would lead to format strings that are easier to
for the programmer to visually parse; (1) would make the indexing part
of the replacement field conform more closely to the way indexing with
strings behaves in Python generally, where arbitrary strings can be
used. (It wouldn't conform exactly, obviously, since ']' would still
be excluded.)
I personally would prefer (1) to (2) or (3), and (3) to (2), had I my
druthers, but it doesn't matter a *whole* lot to me; I'd prefer any of
them to nothing (or to changing the docs to reflect the current batty
behavior).
--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Counting collisions for the win
On Fri, Jan 20, 2012 at 2:11 PM, Terry Reedy wrote: > On 1/20/2012 2:51 PM, Donald Stufft wrote: > >> I think the counting collision is at best a bandaid and not a proper fix >> stemmed from a desire to not break existing applications on a bugfix >> release ... > > My opinion of counting is better than yours, but even conceding the > theoretical, purity argument, our release process is practical as well. > There have been a few occasions when fixes to bugs in our code have been > delayed from a bugfix release to the next feature release -- because the fix > would break too much code depending on the bug. AFAICT Brett's suggestion (which had occurred to me as well, but I'm not a core developer by any stretch) seemed to get lost in the debate: would it be possible to go with collision counting for bugfix releases and hash randomization for new feature releases? (Brett made it here: <http://mail.python.org/pipermail/python-dev/2012-January/115740.html>.) -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] str.format bug again
There's a patch for this bug: http://bugs.python.org/issue12014 that needs reviewed. Petri Lehtinen made some (very minor) comments back in May; otherwise it's been neglected. I've brought this up occasionally here in the past; it would be great if someone could just give it a thumbs up or down (or say what needs to be changed, or whatever). -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str.format bug again
On Sat, Sep 29, 2012 at 10:59 AM, Benjamin Peterson wrote: > 2012/9/28 Ben Wolfson : >> There's a patch for this bug: http://bugs.python.org/issue12014 that >> needs reviewed. Petri Lehtinen made some (very minor) comments back in >> May; otherwise it's been neglected. I've brought this up occasionally >> here in the past; it would be great if someone could just give it a >> thumbs up or down (or say what needs to be changed, or whatever). > > It seems like Eric Smith is the one needs to make a decision. Yes, but one of the things he could decide is that someone else should make the decision because PEP 420 is too involved (though it was last modified in June, so who knows). -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
