Re: [Python-Dev] python build failed on mac

[Hynek Schlawack]

Hello Vijay 


Am Freitag, 20. Januar 2012 um 00:56 schrieb Vijay N. Majagaonkar:

> I am trying to build python 3 on mac and build failing with following error 
> can somebody help me with this
It is a known bug that Apple's latest gcc-llvm (that comes with Xcode 4.1 by 
default as gcc) miscompiles Python: http://bugs.python.org/issue13241 

make clean
CC=clang ./configure && make -s

works though (despite the abundant warnings).

Regards,
-h

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Martin v. Löwis]

> The last solution is very simple: count collision and raise an
> exception if it hits a limit. The path is something like 10 lines
> whereas the randomized hash is more close to 500 lines, add a new
> file, change Visual Studio project file, etc. First I thaught that it
> would break more applications than the randomized hash

The main issue with that approach is that it allows a new kind of attack.

An attacker now needs to find 1000 colliding keys, and submit them
one-by-one into a database. The limit will not trigger, as those are
just database insertions.

Now, if the applications also as a need to read the entire database
table into a dictionary, that will suddenly break, and not for the
attacker (which would be ok), but for the regular user of the
application or the site administrator.

So it may be that this approach actually simplifies the attack, making
the cure worse than the disease.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Hrvoje Niksic]

On 01/18/2012 06:55 PM, "Martin v. Löwis" wrote:

I was thinking about adding the field at the end,


Will this make all strings larger, or only those that create dict 
collisions?  Making all strings larger to fix this issue sounds like a 
really bad idea.


Also, would it be acceptable to simply not cache the alternate hash? 
The cached string hash is an optimization anyway.


Hrvoje
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Frank Sievertsen]

The main issue with that approach is that it allows a new kind of attack.


Indeed, I posted another example: http://bugs.python.org/msg151677

This kind of fix can be used in a specific application or maybe in a
special-purpose framework, but not on the level of a general-purpose
language.

Frank
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Nick Coghlan]

On Fri, Jan 20, 2012 at 7:34 PM, "Martin v. Löwis"  wrote:
> The main issue with that approach is that it allows a new kind of attack.
>
> An attacker now needs to find 1000 colliding keys, and submit them
> one-by-one into a database. The limit will not trigger, as those are
> just database insertions.
>
> Now, if the applications also as a need to read the entire database
> table into a dictionary, that will suddenly break, and not for the
> attacker (which would be ok), but for the regular user of the
> application or the site administrator.
>
> So it may be that this approach actually simplifies the attack, making
> the cure worse than the disease.

Ouch, I think you're right. So hash randomisation may be the best
option, and admins will need to test for themselves to see if it
breaks things...

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Changing the order of iteration over a dictionary

[Mark Shannon]

Hi,

One of the main sticking points over possible fixes for the 
hash-collision security issue seems to be a fear that changing the 
iteration order of a dictionary will break backwards compatibility.


The order of iteration has never been specified. In fact not only is it 
arbitrary, it cannot be determined from the contents of a dict alone; it 
may depend on the insertion order.


Changing a hash function is not the only change that will change the 
iteration order; any of the following will also do so:

* Changing the minimum size of a dict.
* Changing the load factor of a dict.
* Changing the resizing policy of a dict.
* Sharing of keys between dicts.

By treating iteration order as part of the API we are effectively ruling 
out ever making any improvements to the dict.


For example, my new dictionary implementation
https://bitbucket.org/markshannon/hotpy_new_dict/
reduces memory use by 47% for gcbench, and by about 20% for the 2to3 
benchmark, on my 32bit machine.

(Nice graphs: http://tinyurl.com/7qd2nnm http://tinyurl.com/6uqvl2x )

The new dict implementation (necessarily) changes the iteration order 
and will break code that relies on it.


If dict iteration order is to be treated as part of the API (and I think 
that is a very bad idea) then it should be documented, which will be 
difficult since it is barely deterministic.
This will also be a major problem for PyPy, Jython and IronPython, as 
they will have to reimplement their dicts.


So, don't be afraid to change that hash function :)

Cheers,
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing the order of iteration over a dictionary

[Fred Drake]

On Fri, Jan 20, 2012 at 5:49 AM, Mark Shannon  wrote:
> So, don't be afraid to change that hash function :)

Definitely.

The hash function *has* been changed in the past, and a lot of developers
were schooled in not relying on the iteration order.  That's a good thing,
as those developers now write tests of what's actually important rather
than relying on implementation details of the Python runtime.

A hash function that changes more often than during an occasional major
version update will encourage more developers to write better tests.  We
can think of it as an educational tool.


  -Fred

-- 
Fred L. Drake, Jr.    
"A person who won't read has no advantage over one who can't read."
   --Samuel Langhorne Clemens
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing the order of iteration over a dictionary

[Nick Coghlan]

On Fri, Jan 20, 2012 at 8:49 PM, Mark Shannon  wrote:
> So, don't be afraid to change that hash function :)

Changing it for 3.3 isn't really raising major concerns: the real
concern is with changing it in maintenance and security patches for
earlier releases. Security patches that may break production
applications aren't desirable, since it means admins have to weigh up
the risk of being affected by the security vulnerability against the
risk of breakage from the patch itself.

The collision counting approach was attractive because it looked like
it might offer a way out that was less likely to break deployed
systems. Unfortunately, I think the point Martin raised about just
opening a new (even more subtle) attack vector kills that idea dead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Victor Stinner]

2012/1/20 Ivan Kozik :
> I'd like to point out that an attacker is not limited to sending just
> one dict full of colliding keys.  Given a 22ms stall ...

The presented attack produces a stall of at least 30 seconds (5
minutes or more if there is no time limit in the application), not
0.022 second. You have to send a lot of requests to produce a DoS if a
single requests just eat 22 ms. I suppose that there are a lot of
other kinds of request than takes much longer than 22 ms, even valid
requests.

> Another issue is that even with a configurable limit, different
> modules can't have their own limits.  One module might want a
> relatively safe raise-at-100, and another module creating massive
> dicts might want raise-at-1000.  How does a developer know whether
> they can raise or lower the limit, given that they use a bunch of
> different modules?

Python becomes really slow when you have more than N collisions
(O(n^2) problem). If an application hits this limit with valid data,
it is time to use another data structure or use a different hash
function. We have to do more tests to choose correctly N, but
according my first tests, it looks like N=1000 is a safe limit.

Marc Andre's patch doesn't count all "collisions", but only
"collisions" requiring to compare objects. When two objects have the
same hash value, the open addressing algorithm searchs a free bucket.
If a bucket is not free but has a different hash value, the objects
are not compared and the collision counter is not incremented. The
limit is only reached when you have N objects having the same hash
value modulo the size of the bucket (hash(str) & DICT_MASK).

When there are not enough empty buckets (it comes before all buckets
are full), Python resizes the dictionary (it does something like size
= size * 2) and so it uses at least one more bit each time than the
dictionary is resized. Collisions are very likely with a small
dictioanry, but becomes more rare each time than the dictionary is
resized. It means that the number of potential collisions (with valid
data) decreases when the dictionary grows. Tell me if I am wrong.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Victor Stinner]

> I'm surprised we haven't seen bug reports about it from users
> of 64-bit Pythons long ago

A Python dictionary only uses the lower bits of a hash value. If your
dictionary has less than 2**32 items, the dictionary order is exactly
the same on 32 and 64 bits system: hash32(str) & mask == hash64(str) &
mask for mask <= 2**32-1.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Frank Sievertsen]

No, that's not true.
Whenever a collision happens, other bits are mixed in very fast.

Frank

Am 20.01.2012 13:08, schrieb Victor Stinner:

I'm surprised we haven't seen bug reports about it from users
of 64-bit Pythons long ago

A Python dictionary only uses the lower bits of a hash value. If your
dictionary has less than 2**32 items, the dictionary order is exactly
the same on 32 and 64 bits system: hash32(str)&  mask == hash64(str)&
mask for mask<= 2**32-1.
_


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Victor Stinner]

2012/1/20 Frank Sievertsen :
> No, that's not true.
> Whenever a collision happens, other bits are mixed in very fast.

Oh, I didn't know that. So the dict order is only the same if there is
no collision.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Victor Stinner]

> The main issue with that approach is that it allows a new kind of attack.
>
> An attacker now needs to find 1000 colliding keys, and submit them
> one-by-one into a database. The limit will not trigger, as those are
> just database insertions.
>
> Now, if the applications also as a need to read the entire database
> table into a dictionary, that will suddenly break, and not for the
> attacker (which would be ok), but for the regular user of the
> application or the site administrator.

Oh, good catch. But it would not call it a new kind of attack, it is
just a particular case of the hash collision vulnerability.

Counting collision doesn't solve this case, but it doesn't make the
situation worse than before. Raising quickly an exception is better
than stalling for minutes, even if I agree than it is not the best
behaviour. The best would is to answer quickly with the expected
result :-) (using a different data structure or a different hash
function?)

Right now, I don't see any counter measure against this case.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Paul Moore]

On 20 January 2012 03:57, Brian Curtin  wrote:
>> FWIW, it might well be that I can't be available for the 3.3 final
>> release (I haven't finalized my vacation schedule yet for August).
>
> In the interest of not having Windows releases depend on one person,
> and having gone through building the installer myself (which I know is
> but one of the duties), I'm available to help should you need it.

One thought comes to mind - while we need a PEP to make a permanent
change to the release schedule, would it be practical in any way to do
a "trial run" of the process, and simply aim to release 3.4 about 6
months after 3.3? Based on the experiences gained from that, some of
the discussions around this PEP could be supported (or not :-)) with
more concrete information. If we can't do that, then that says
something about the practicality of the proposal in itself...

The plan for 3.4 would need to be publicised well in advance, of
course, but doing that as a one-off exercise might well be viable.

Paul.

PS I have no view on whether the proposal is a good idea or a bad idea
from a RM point of view. That's entirely up to the people who do the
work to decide, in my opinion.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Barry Warsaw]

On Jan 20, 2012, at 01:50 PM, Victor Stinner wrote:

>Counting collision doesn't solve this case, but it doesn't make the
>situation worse than before. Raising quickly an exception is better
>than stalling for minutes, even if I agree than it is not the best
>behaviour.

ISTM that adding the possibility of raising a new exception on dictionary
insertion is *more* backward incompatible than changing dictionary order,
which for a very long time has been known to not be guaranteed.  You're
running some application, you upgrade Python because you apply all security
fixes, and suddenly you're starting to get exceptions in places you can't
really do anything about.  Yet those exceptions are now part of the documented
public API for dictionaries.  This is asking for trouble.  Bugs will suddenly
start appearing in that application's tracker and they will seem to the
application developer like Python just added a new public API in a security
release.

OTOH, if you change dictionary order and *that* breaks the application, then
the bugs submitted to the application's tracker will be legitimate bugs that
have to be fixed even if nothing else changed.

So I still think we should ditch the paranoia about dictionary order changing,
and fix this without counting.  A little bit of paranoia could creep back in
by disabling the hash fix by default in stable releases, but I think it would
be fine to make that a compile-time option.

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Barry Warsaw]

On Jan 20, 2012, at 03:18 PM, Nick Coghlan wrote:

>On Fri, Jan 20, 2012 at 2:54 PM, Carl Meyer  wrote:
>> I don't have the expertise to speak otherwise to the alternatives for
>> fixing the collisions vulnerability, but I don't believe it's accurate
>> to presume that Django would not want to fix a dict-ordering dependency,
>> and use that as a justification for one approach over another.
>
>It's more a matter of wanting deployment of a security fix to be as
>painless as possible - a security fix that system administrators can't
>deploy because it breaks critical applications may as well not exist.

True, but collision counting is worse IMO.  It's just as likely (maybe) that
an application would start getting new exceptions on dictionary insertion, as
they would failures due to dictionary order changes.  Unfortunately, in the
former case it's because Python just added a new public API in a security
release (the new exception *is* public API).  In the latter case, no new API
was added, but something exposed an already existing bug in the application.
That's still a bug in the application even if counting was added.  It's also a
bug that any number of changes in the environment, or OS vendor deployment,
could have triggered.

-1 for collision counting.

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Barry Warsaw]

On Jan 20, 2012, at 03:15 PM, Nick Coghlan wrote:

>With the 1000 collision limit in place, the attacker sends their
>massive request, the affected dict quickly hits the limit, throws an
>unhandled exception which is then caught by the web framework and
>turned into a 500 Error response (or whatever's appropriate for the
>protocol being attacked).

Let's just be clear about it: this exception is new public API.  Changing
dictionary order is not.

For me, that comes down firmly on the side of the latter rather than the
former for stable releases.

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Antoine Pitrou]

On Fri, 20 Jan 2012 13:50:18 +0100
Victor Stinner  wrote:

> > The main issue with that approach is that it allows a new kind of attack.
> >
> > An attacker now needs to find 1000 colliding keys, and submit them
> > one-by-one into a database. The limit will not trigger, as those are
> > just database insertions.
> >
> > Now, if the applications also as a need to read the entire database
> > table into a dictionary, that will suddenly break, and not for the
> > attacker (which would be ok), but for the regular user of the
> > application or the site administrator.
> 
> Oh, good catch. But it would not call it a new kind of attack, it is
> just a particular case of the hash collision vulnerability.
> 
> Counting collision doesn't solve this case, but it doesn't make the
> situation worse than before. Raising quickly an exception is better
> than stalling for minutes, even if I agree than it is not the best
> behaviour.

Actually, it *is* worse because stalling for seconds or minutes may not
be a problem in some cases (e.g. some batch script that gets run
overnight).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Antoine Pitrou]

On Fri, 20 Jan 2012 12:57:45 +
Paul Moore  wrote:
> On 20 January 2012 03:57, Brian Curtin  wrote:
> >> FWIW, it might well be that I can't be available for the 3.3 final
> >> release (I haven't finalized my vacation schedule yet for August).
> >
> > In the interest of not having Windows releases depend on one person,
> > and having gone through building the installer myself (which I know is
> > but one of the duties), I'm available to help should you need it.
> 
> One thought comes to mind - while we need a PEP to make a permanent
> change to the release schedule, would it be practical in any way to do
> a "trial run" of the process, and simply aim to release 3.4 about 6
> months after 3.3?

It sounds reasonable to me, although we probably wouldn't market it as
a "trial run".

cheers

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.1): Closes #13807: Now checks for sys.stderr being there before writing to it.

[Meador Inge]

On Fri, Jan 20, 2012 at 5:32 AM, vinay.sajip  wrote:
> http://hg.python.org/cpython/rev/73dad4940b88
> changeset:   74538:73dad4940b88
> branch:      3.1

I thought that the 3.1 branch is in security mode?  Is this a security
related fix?
>From my brief scan of the changeset, it doesn't seem to be.

> parent:      74253:fb5707168351
> user:        Vinay Sajip 
> date:        Fri Jan 20 11:23:02 2012 +
> summary:
>  Closes #13807: Now checks for sys.stderr being there before writing to it.
>
> files:
>  Lib/logging/__init__.py |  2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
> --- a/Lib/logging/__init__.py
> +++ b/Lib/logging/__init__.py
> @@ -721,7 +721,7 @@
>         You could, however, replace this with a custom handler if you wish.
>         The record which was being processed is passed in to this method.
>         """
> -        if raiseExceptions:
> +        if raiseExceptions and sys.stderr:  # see issue 13807
>             ei = sys.exc_info()
>             try:
>                 traceback.print_exception(ei[0], ei[1], ei[2], None, 
> sys.stderr)
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> python-check...@python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>



-- 
# Meador
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 1:34 AM, "Martin v. Löwis" wrote:

> > The last solution is very simple: count collision and raise an
> > exception if it hits a limit. The path is something like 10 lines
> > whereas the randomized hash is more close to 500 lines, add a new
> > file, change Visual Studio project file, etc. First I thaught that it
> > would break more applications than the randomized hash
>
> The main issue with that approach is that it allows a new kind of attack.
>
> An attacker now needs to find 1000 colliding keys, and submit them
> one-by-one into a database. The limit will not trigger, as those are
> just database insertions.
>
> Now, if the applications also as a need to read the entire database
> table into a dictionary, that will suddenly break, and not for the
> attacker (which would be ok), but for the regular user of the
> application or the site administrator.
>
> So it may be that this approach actually simplifies the attack, making
> the cure worse than the disease.
>

It would be a pretty lousy app that tried to load the contents of an entire
database into a dict. It seems that this would require much more knowledge
of what the app is trying to do before a successful attack can be mounted.
So I don't think this is worse than the original attack -- I think it
requires much more ingenuity of an attacker. (I'm thinking that the
original attack is trivial once the set of 65000 colliding keys is public
knowledge, which must be only a matter of time.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Frank Sievertsen]

Hello,

I still see at least two ways to create a DOS attack even with the
collison-counting-patch.

I assumed that it's possible to send ~500KB of payload to the
application.

1. It's fully deterministic which slots the dict will lookup.
Since we don't count slot-collisions, but only hash-value-collisions
this can be exploited easily by creating strings with the hash-values
along the lookup-way of an arbitrary (short) string.

So first we pick an arbitrary string. Then calculate which slots it will
visit on the way to the first empty slot. Then we create strings with
hash-values for these slots.

This attack first injects the strings to fill all the slots that the
one short string will want to visit. Then it adds THE SAME string
again and again. Since the entry is already there, nothing will be added
and no additional collisions happen, no exception raised.

$ ls -l super.txt
-rw-r--r-- 1 fx5 fx5 52 20. Jan 10:19 super.txt
$ tail -n3 super.txt
FX5
FX5
FX5
$ wc -l super.txt
9 super.txt
$ time python -c 'dict((unicode(l[:-1]), 0)  for l in open("super.txt"))'
real0m52.724s
user0m51.543s
sys0m0.028s

2. The second attack actually attacks that 1000 allowed string
comparisons are still a lot of work.
First I added 999 strings that collide with a one-byte string "a". In
some applications a zero-byte string might work even better. Then I
can add a many thousand of the "a"'s, just like the first attack.

$ ls -l 1000.txt
-rw-r--r-- 1 fx5 fx5 50 20. Jan 16:15 1000.txt
$ head -n 3 1000.txt
7hLci00
4wVFm10
_rZJU50
$ wc -l 1000.txt
247000 1000.txt
$ tail -n 3 1000.txt
a
a
a
$ time python -c 'dict((unicode(l[:-1]), 0)  for l in open("1000.txt"))'
real0m17.408s
user0m15.897s
sys0m0.008s

Of course the first attack is far more efficient. One could argue
that 16 seconds is not enough for an attack. But maybe it's possible
to send 1MB, have zero-bytes strings, and since for example django
does 5 lookups per query-string this will keep it busy for ~80 seconds on
my pc.

What to do now?
I think it's not smart to reduce the number of allowed collisions 
dramatically

AND count all slot-collisions at the same time.

Frank
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Victor Stinner]

> (I'm thinking that the original
> attack is trivial once the set of 65000 colliding keys is public knowledge,
> which must be only a matter of time.)

I have a program able to generate collisions: it takes 1 second to
compute 60,000 colliding strings on a desktop computer. So the
security of the randomized hash is based on the fact than the attacker
cannot compute the secret.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Victor Stinner]

> So I still think we should ditch the paranoia about dictionary order changing,
> and fix this without counting.

The randomized hash has other issues:

 - its security is based on its secret, whereas it looks to be easy to
compute it (see more details in the issue)
 - my patch only changes hash(str), whereas other developers asked me
to patch also bytes, int and other types

hash(bytes) can be changed. But changing hash(int) may leak easily the
secret. We may use a different secret for each type, but if it is easy
to compute int hash secret, dictionaries using int are still
vulnerable.

--

There is no perfect solutions, drawbacks of each solution should be compared.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Antoine Pitrou]

On Fri, 20 Jan 2012 17:17:24 +0100
Victor Stinner  wrote:
> > So I still think we should ditch the paranoia about dictionary order 
> > changing,
> > and fix this without counting.
> 
> The randomized hash has other issues:
> 
>  - its security is based on its secret, whereas it looks to be easy to
> compute it (see more details in the issue)

How do you compute the secret? I see two possibilities:

- the application leaks the hash() values: this sounds unlikely since I
  don't see the use case for it;

- the application shows the dict iteration order (e.g. order of HTML
  attributes): then we could add a second per-dictionary secret so that
  the iteration order of a single dict doesn't give any useful
  information about the hash function.

But the bottom line for me is the following:

- randomized hashes eliminate the possibility to use a single exploit
  for all Python-powered applications: for each application, the
  attacker now has to find a way to extract the secret;

- collision counting doesn't eliminate the possibility of generic
  exploits, as Frank Sievertsen has just shown in
  http://mail.python.org/pipermail/python-dev/2012-January/115726.html

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Lennart Regebro]

On Fri, Jan 20, 2012 at 01:48, Victor Stinner
 wrote:
>  - the limit should be configurable: a new function in the sys module
> should be enough. It may be private (or replaced by an environment
> variable?) in stable versions

I'd like to see both. I would like both the programmer and the "user"
to be able to control what the limit is.

//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2012-01-13 - 2012-01-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open3209 ( -1)
  closed 22405 (+53)
  total  25614 (+52)

Open issues with patches: 1376 


Issues opened (37)
==

#13411: Hashable memoryviews
http://bugs.python.org/issue13411  reopened by skrah

#13782: xml.etree.ElementTree: Element.append doesn't type-check its a
http://bugs.python.org/issue13782  opened by sjmachin

#13783: Clean up PEP 380 C API additions
http://bugs.python.org/issue13783  opened by ncoghlan

#13784: Documentation of  xml.sax.xmlreader: Locator.getLineNumber() a
http://bugs.python.org/issue13784  opened by patrick.vrijlandt

#13785: Make concurrent.futures.Future state public
http://bugs.python.org/issue13785  opened by jjdominguezm

#13788: os.closerange optimization
http://bugs.python.org/issue13788  opened by ferringb

#13789: _tkinter does not build on Windows 7
http://bugs.python.org/issue13789  opened by terry.reedy

#13790: In str.format an incorrect error message for list, tuple, dict
http://bugs.python.org/issue13790  opened by py.user

#13792: The "os.execl" call doesn't give programs exit code
http://bugs.python.org/issue13792  opened by kayhayen

#13793: hasattr, delattr, getattr fail with unnormalized names
http://bugs.python.org/issue13793  opened by Jim.Jewett

#13796: use 'text=...' to define the text attribute of and xml.etree.E
http://bugs.python.org/issue13796  opened by paaguti

#13797: Allow objects implemented in pure Python to export PEP 3118 bu
http://bugs.python.org/issue13797  opened by ncoghlan

#13798: Pasting and then running code doesn't work in the IDLE Shell
http://bugs.python.org/issue13798  opened by ramchandra.apte

#13799: Base 16 should be hexadecimal in Unicode HOWTO
http://bugs.python.org/issue13799  opened by ramchandra.apte

#13801: The Python 3 Docs don't highlight nonlocal
http://bugs.python.org/issue13801  opened by ramchandra.apte

#13802: IDLE Prefernces/Fonts: use multiple alphabets in examples
http://bugs.python.org/issue13802  opened by terry.reedy

#13804: Python library structure creates hard to read code when using 
http://bugs.python.org/issue13804  opened by dwt

#13806: Audioop decompression frames size check fix
http://bugs.python.org/issue13806  opened by Oleg.Plakhotnyuk

#13812: multiprocessing package doesn't flush stderr on child exceptio
http://bugs.python.org/issue13812  opened by brandj

#13814: Generators as context managers.
http://bugs.python.org/issue13814  opened by yak

#13815: tarfile.ExFileObject can't be wrapped using io.TextIOWrapper
http://bugs.python.org/issue13815  opened by cjwatson

#13816: Two typos in the docs
http://bugs.python.org/issue13816  opened by Retro

#13817: deadlock in subprocess while running several threads using Pop
http://bugs.python.org/issue13817  opened by glaubich

#13818: argparse: -h listening required options under optional argumen
http://bugs.python.org/issue13818  opened by mgodinho

#13819: _warnings settings are process-wide
http://bugs.python.org/issue13819  opened by pitrou

#13820: 2.6 is no longer in the future
http://bugs.python.org/issue13820  opened by Jim.Jewett

#13821: misleading return from isidentifier
http://bugs.python.org/issue13821  opened by Jim.Jewett

#13822: is(upper/lower/title) are not exactly correct
http://bugs.python.org/issue13822  opened by benjamin.peterson

#13823: xml.etree.ElementTree.ElementTree.write - argument checking
http://bugs.python.org/issue13823  opened by patrick.vrijlandt

#13824: argparse.FileType opens a file without excepting resposibility
http://bugs.python.org/issue13824  opened by David.Layton

#13825: Datetime failing while reading active directory time attribute
http://bugs.python.org/issue13825  opened by scape

#13826: Having a shlex example in the subprocess.Popen docs is confusi
http://bugs.python.org/issue13826  opened by Julian

#13828: Further improve casefold documentation
http://bugs.python.org/issue13828  opened by Jim.Jewett

#13829: exception error
http://bugs.python.org/issue13829  opened by Dan.kamp

#13830: codecs error handler is called with a UnicodeDecodeError with 
http://bugs.python.org/issue13830  opened by amaury.forgeotdarc

#13831: get method of  multiprocessing.pool.Async should return full t
http://bugs.python.org/issue13831  opened by fmitha

#13833: No documentation for PyStructSequence
http://bugs.python.org/issue13833  opened by torsten



Most recent 15 issues with no replies (15)
==

#13833: No documentation for PyStructSequence
http://bugs.python.org/issue13833

#13831: get method of  multiprocessing.pool.Async should return full t
http://bugs.python.org/issue13831

#13830: codecs error handler is called with a UnicodeDecodeError with 
http://bugs.python.org/issue13830

#13829: exception error
http://bugs.python.org/i

[Python-Dev] exception chaining

[Ethan Furman]

Summary:

Exception Chaining is cool, unless you are writing libraries that want 
to transform from Exception X to Exception Y as the the previous 
exception context is unnecessary, potentially confusing, and cluttery 
(yup, just made that word up!).


For all the gory details, see http://bugs.python.org/issue6210.

I'm going to attempt a patch implementing MRAB's suggestion:

try:
some_op
except ValueError:
raise as OtherError() # `raise` keeps context, `raise as` does not

The question I have at the moment is:  should `raise as` be an error if 
no exception is currently being handled?


Example:

def smurfy(x):
if x != 'magic flute':
 raise as WrongInstrument
do_something_with_x

If this is allowed then `smurfy` could be called from inside an `except` 
clause or outside it.


I don't care for it for two reasons:

  - I don't like the way it looks
  - I can see it encouraging always using `raise as` instead of `raise` 
and losing the value of exception chaining.


Other thoughts?

~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Thu, Jan 19, 2012 at 8:06 PM, Ivan Kozik  wrote:

> No, I wasn't happy with termination.  I wanted to treat it just like a
> JSON decoding error, and send the appropriate response.
>

So was this attack actually being mounted on your service regularly? I'd
think it would be sufficient to treat it as a MemoryError -- unavoidable,
if it happens things are really bad, and hopefully you'll crash quickly and
some monitor process restarts your service. That's a mechanism that you
should have anyway.


> I actually forgot to mention the main reason I abandoned the
> stop-at-N-collisions approach.  I had a server with a dict that stayed
> in memory, across many requests.  It was being populated with
> identifiers chosen by clients.  I couldn't have my server stay broken
> if this dict filled up with a bunch of colliding keys.  (I don't think
> I could have done another thing either, like nuke the dict or evict
> some keys.)
>

What would your service do if it ran out of memory?

Maybe one tweak to the collision counting would be that the exception needs
to inherit from BaseException (like MemoryError) so most generic exception
handlers don't actually handle it. (Style note: never use "except:", always
use "except Exception:".)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Benjamin Peterson]

2012/1/20 Ethan Furman :
> Summary:
>
> Exception Chaining is cool, unless you are writing libraries that want to
> transform from Exception X to Exception Y as the the previous exception
> context is unnecessary, potentially confusing, and cluttery (yup, just made
> that word up!).
>
> For all the gory details, see http://bugs.python.org/issue6210.
>
> I'm going to attempt a patch implementing MRAB's suggestion:
>
> try:
>    some_op
> except ValueError:
>    raise as OtherError() # `raise` keeps context, `raise as` does not

I dislike this syntax. Raise what as OtherError()? I think the "raise
x from None" idea is preferable, since it indicates you are nulling
the context. The optimal solution would be to have "raise X
nocontext", but that would obviously require another keyword...


-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 1:57 AM, Frank Sievertsen wrote:

>  The main issue with that approach is that it allows a new kind of attack.
>
>
> Indeed, I posted another example: http://bugs.python.org/msg151677
>
> This kind of fix can be used in a specific application or maybe in a
> special-purpose framework, but not on the level of a general-purpose
> language.
>

Right. We are discussion this issue (for weeks now...) because it makes
pretty much any Python app that takes untrusted data vulnerable, especially
web apps, and after extensive analysis we came to the conclusion that
defenses in the framework or in the app are really hard to do, very
disruptive for developers, whereas preventing the attack by a modification
of the dict or hash algorithms would fix it for everybody. And moreover,
the attack would work against pretty much any Python web app using a set of
evil strings computed once (hence encouraging script kiddies of just firing
their fully-automatic weapon at random websites).

The new attacks that are now being considered require analysis of how the
website is implemented, how it uses and stores data, etc. So an attacker
has to sit down and come up with an attack tailored to a specific website.
That can be dealt with on an ad-hoc basis.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 5:10 AM, Barry Warsaw  wrote:

> On Jan 20, 2012, at 01:50 PM, Victor Stinner wrote:
>
> >Counting collision doesn't solve this case, but it doesn't make the
> >situation worse than before. Raising quickly an exception is better
> >than stalling for minutes, even if I agree than it is not the best
> >behaviour.
>
> ISTM that adding the possibility of raising a new exception on dictionary
> insertion is *more* backward incompatible than changing dictionary order,
> which for a very long time has been known to not be guaranteed.  You're
> running some application, you upgrade Python because you apply all security
> fixes, and suddenly you're starting to get exceptions in places you can't
> really do anything about.  Yet those exceptions are now part of the
> documented
> public API for dictionaries.  This is asking for trouble.  Bugs will
> suddenly
> start appearing in that application's tracker and they will seem to the
> application developer like Python just added a new public API in a security
> release.
>

Dict insertion can already raise an exception: MemoryError. I think we
should be safe if the new exception also derives from BaseException. We
should actually eriously consider just raising MemoryException, since
introducing a new built-in exception in a bugfix release is also very
questionable: code explicitly catching or raising it would not work on
previous bugfix releases of the same feature release.

OTOH, if you change dictionary order and *that* breaks the application, then
> the bugs submitted to the application's tracker will be legitimate bugs
> that
> have to be fixed even if nothing else changed.
>

There are lots of things that are undefined according to the language spec
(and quite possibly known to vary between versions or platforms or
implementations like PyPy or Jython) but which we would never change in a
bugfix release.

So I still think we should ditch the paranoia about dictionary order
> changing,
> and fix this without counting.  A little bit of paranoia could creep back
> in
> by disabling the hash fix by default in stable releases, but I think it
> would
> be fine to make that a compile-time option.


I'm sorry, but I don't want to break a user's app with a bugfix release and
say "haha your code was already broken you just didn't know it".

Sure, the dict order already varies across Python implementations, possibly
across 32/64 bits or operating systems. But many organizations (I know a
few :-) have a very large installed software base, created over many years
by many people with varying skills, that is kept working in part by very
carefully keeping the environment as constant as possible. This means that
the target environment is much more predictable than it is for the typical
piece of open source software.

Sure, a good Python developer doesn't write apps or tests that depend on
dict order. But time and again we see that not everybody writes perfect
code every time. Especially users writing "in-house" apps (as opposed to
frameworks shared as open source) are less likely to always use the most
robust, portable algorithms in existence, because they may know with much
more certainty that their code will never be used on certain combinations
of platforms. For example, I rarely think  about whether code I write might
not work on IronPython or Jython, or even CPython on Windows. And if
something I wrote suddenly needs to be ported to one of those, well, that's
considered a port and I'll just accept that it might mean changing a few
things.

The time to break a dependency on dict order is not with a bugfix release
but with a feature release: those are more likely to break other things as
well anyway, and uses are well aware that they have to test everything and
anticipate having to fix some fraction of their code for each feature
release. OTOH we have established a long and successful track record of
conservative bugfix releases that don't break anything. (I am aware of
exactly one thing that was broken by a bugfix release in application code I
am familiar with.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 5:20 AM, Barry Warsaw  wrote:

> Let's just be clear about it: this exception is new public API.  Changing
> dictionary order is not.
>

Not if you raise MemoryError or BaseException.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

This is the first objection I have seen against collision-counting that
might stand.

On Fri, Jan 20, 2012 at 7:55 AM, Frank Sievertsen wrote:

> Hello,
>
> I still see at least two ways to create a DOS attack even with the
> collison-counting-patch.
>
> I assumed that it's possible to send ~500KB of payload to the
> application.
>
> 1. It's fully deterministic which slots the dict will lookup.
> Since we don't count slot-collisions, but only hash-value-collisions
> this can be exploited easily by creating strings with the hash-values
> along the lookup-way of an arbitrary (short) string.
>
> So first we pick an arbitrary string. Then calculate which slots it will
> visit on the way to the first empty slot. Then we create strings with
> hash-values for these slots.
>
> This attack first injects the strings to fill all the slots that the
> one short string will want to visit. Then it adds THE SAME string
> again and again. Since the entry is already there, nothing will be added
> and no additional collisions happen, no exception raised.
>
> $ ls -l super.txt
> -rw-r--r-- 1 fx5 fx5 52 20. Jan 10:19 super.txt
> $ tail -n3 super.txt
> FX5
> FX5
> FX5
> $ wc -l super.txt
> 9 super.txt
> $ time python -c 'dict((unicode(l[:-1]), 0)  for l in open("super.txt"))'
> real0m52.724s
> user0m51.543s
> sys0m0.028s
>
> 2. The second attack actually attacks that 1000 allowed string
> comparisons are still a lot of work.
> First I added 999 strings that collide with a one-byte string "a". In
> some applications a zero-byte string might work even better. Then I
> can add a many thousand of the "a"'s, just like the first attack.
>
> $ ls -l 1000.txt
> -rw-r--r-- 1 fx5 fx5 50 20. Jan 16:15 1000.txt
> $ head -n 3 1000.txt
> 7hLci00
> 4wVFm10
> _rZJU50
> $ wc -l 1000.txt
> 247000 1000.txt
> $ tail -n 3 1000.txt
> a
> a
> a
> $ time python -c 'dict((unicode(l[:-1]), 0)  for l in open("1000.txt"))'
> real0m17.408s
> user0m15.897s
> sys0m0.008s
>
> Of course the first attack is far more efficient. One could argue
> that 16 seconds is not enough for an attack. But maybe it's possible
> to send 1MB, have zero-bytes strings, and since for example django
> does 5 lookups per query-string this will keep it busy for ~80 seconds on
> my pc.
>
> What to do now?
> I think it's not smart to reduce the number of allowed collisions
> dramatically
> AND count all slot-collisions at the same time.
>
> Frank
>
> __**_
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Georg Brandl]

Am 20.01.2012 19:15, schrieb Guido van Rossum:

> OTOH, if you change dictionary order and *that* breaks the application, 
> then
> the bugs submitted to the application's tracker will be legitimate bugs 
> that
> have to be fixed even if nothing else changed.
> 
> 
> There are lots of things that are undefined according to the language spec 
> (and
> quite possibly known to vary between versions or platforms or implementations
> like PyPy or Jython) but which we would never change in a bugfix release.
> 
> So I still think we should ditch the paranoia about dictionary order 
> changing,
> and fix this without counting.  A little bit of paranoia could creep back 
> in
> by disabling the hash fix by default in stable releases, but I think it 
> would
> be fine to make that a compile-time option.
> 
> 
> I'm sorry, but I don't want to break a user's app with a bugfix release and 
> say
> "haha your code was already broken you just didn't know it".
> 
> Sure, the dict order already varies across Python implementations, possibly
> across 32/64 bits or operating systems. But many organizations (I know a few 
> :-)
> have a very large installed software base, created over many years by many
> people with varying skills, that is kept working in part by very carefully
> keeping the environment as constant as possible. This means that the target
> environment is much more predictable than it is for the typical piece of open
> source software.

I agree.  This applies to 3.2 and 2.7, but even more to 3.1 and 2.6, which are
in security-fix mode.

Even if relying on dict order is a bug right now, I believe it happens many
times more often in code bases out there than dicts that are filled with many
many colliding keys.  So even if we can honestly blame the programmer in the
former case, the users applying the security fix will have the same bad
experience and won't likely care if we claim "undefined behavior".  This means
that it seems preferable to go with the situation where you have less breakages
in total.

Not to mention that changing dict order is likely to lead to much more subtle
bugs than a straight MemoryError on a dict insert.

Also, another advantage of collision counting I haven't seen in the discussion
yet is that it's quite trivial to provide an API in sys to turn it on or off --
while turning on or off randomized hashes has to be done before Python starts
up, i.e. at build time or with an environment variable or flag.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Brett Cannon]

On Fri, Jan 20, 2012 at 13:15, Guido van Rossum  wrote:

> On Fri, Jan 20, 2012 at 5:10 AM, Barry Warsaw  wrote:
>
>> On Jan 20, 2012, at 01:50 PM, Victor Stinner wrote:
>>
>> >Counting collision doesn't solve this case, but it doesn't make the
>> >situation worse than before. Raising quickly an exception is better
>> >than stalling for minutes, even if I agree than it is not the best
>> >behaviour.
>>
>> ISTM that adding the possibility of raising a new exception on dictionary
>> insertion is *more* backward incompatible than changing dictionary order,
>> which for a very long time has been known to not be guaranteed.  You're
>> running some application, you upgrade Python because you apply all
>> security
>> fixes, and suddenly you're starting to get exceptions in places you can't
>> really do anything about.  Yet those exceptions are now part of the
>> documented
>> public API for dictionaries.  This is asking for trouble.  Bugs will
>> suddenly
>> start appearing in that application's tracker and they will seem to the
>> application developer like Python just added a new public API in a
>> security
>> release.
>>
>
> Dict insertion can already raise an exception: MemoryError. I think we
> should be safe if the new exception also derives from BaseException. We
> should actually eriously consider just raising MemoryException, since
> introducing a new built-in exception in a bugfix release is also very
> questionable: code explicitly catching or raising it would not work on
> previous bugfix releases of the same feature release.
>
> OTOH, if you change dictionary order and *that* breaks the application,
>> then
>> the bugs submitted to the application's tracker will be legitimate bugs
>> that
>> have to be fixed even if nothing else changed.
>>
>
> There are lots of things that are undefined according to the language spec
> (and quite possibly known to vary between versions or platforms or
> implementations like PyPy or Jython) but which we would never change in a
> bugfix release.
>
> So I still think we should ditch the paranoia about dictionary order
>> changing,
>> and fix this without counting.  A little bit of paranoia could creep back
>> in
>> by disabling the hash fix by default in stable releases, but I think it
>> would
>> be fine to make that a compile-time option.
>
>
> I'm sorry, but I don't want to break a user's app with a bugfix release
> and say "haha your code was already broken you just didn't know it".
>
> Sure, the dict order already varies across Python implementations,
> possibly across 32/64 bits or operating systems. But many organizations (I
> know a few :-) have a very large installed software base, created over many
> years by many people with varying skills, that is kept working in part by
> very carefully keeping the environment as constant as possible. This means
> that the target environment is much more predictable than it is for the
> typical piece of open source software.
>
> Sure, a good Python developer doesn't write apps or tests that depend on
> dict order. But time and again we see that not everybody writes perfect
> code every time. Especially users writing "in-house" apps (as opposed to
> frameworks shared as open source) are less likely to always use the most
> robust, portable algorithms in existence, because they may know with much
> more certainty that their code will never be used on certain combinations
> of platforms. For example, I rarely think  about whether code I write might
> not work on IronPython or Jython, or even CPython on Windows. And if
> something I wrote suddenly needs to be ported to one of those, well, that's
> considered a port and I'll just accept that it might mean changing a few
> things.
>
> The time to break a dependency on dict order is not with a bugfix release
> but with a feature release: those are more likely to break other things as
> well anyway, and uses are well aware that they have to test everything and
> anticipate having to fix some fraction of their code for each feature
> release. OTOH we have established a long and successful track record of
> conservative bugfix releases that don't break anything. (I am aware of
> exactly one thing that was broken by a bugfix release in application code I
> am familiar with.)
>

Why can't we have our cake and eat it too?

Can we do hash randomization in 3.3 and use the hash count solution for
bugfix releases? That way we get a basic fix into the bugfix releases that
won't break people's code (hopefully) but we go with a more thorough (and
IMO correct) solution of hash randomization starting with 3.3 and moving
forward. We aren't breaking compatibility in any way by doing this since
it's a feature release anyway where we change tactics. And it can't be that
much work since we seem to have patches for both solutions. At worst it
will make merging commits for those files affected by the patches, but that
will most likely be isolated and not a common collision (and less of any
issue once 

Re: [Python-Dev] Counting collisions for the win

[Terry Reedy]

On 1/20/2012 11:17 AM, Victor Stinner wrote:


There is no perfect solutions, drawbacks of each solution should be compared.


Amen.

One possible attack that has been described for a collision counting 
dict depends on knowing precisely the trigger point. So let 
MAXCOLLISIONS either be configureable or just choose a random count 
between M and N, say 700 and 999.


It would not hurt to have alternate patches available in case a 
particular Python-powered site comes under prolonged attack. Though 
given our miniscule share of the market, than is much less likely that 
an attack on a PHP- or MS-powered site.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Donald Stufft]

Even if a MemoryException is raised I believe that is still a fundamental 
change in the documented contract of dictionary API. I don't believe there is a 
way to fix this without breaking someones application. The major differences I 
see between the two solutions is that counting will break people's applications 
who are otherwise following the documented api contract of dictionaries, and 
randomization will break people's applications who are violating the documented 
api contract of dictionaries. 

Personally I feel that the lesser of two evils is to reward those who followed 
the documentation, and not reward those who didn't.

So +1 for Randomization as the only option in 3.3, and off by default with a 
flag or environment variable in bug fixes. I think it's the only way to proceed 
that won't hurt people who have followed the documented behavior. 


On Friday, January 20, 2012 at 1:49 PM, Brett Cannon wrote:

> 
> 
> On Fri, Jan 20, 2012 at 13:15, Guido van Rossum  (mailto:gu...@python.org)> wrote:
> > On Fri, Jan 20, 2012 at 5:10 AM, Barry Warsaw  > (mailto:ba...@python.org)> wrote:
> > > On Jan 20, 2012, at 01:50 PM, Victor Stinner wrote:
> > > 
> > > >Counting collision doesn't solve this case, but it doesn't make the
> > > >situation worse than before. Raising quickly an exception is better
> > > >than stalling for minutes, even if I agree than it is not the best
> > > >behaviour.
> > > 
> > > ISTM that adding the possibility of raising a new exception on dictionary
> > > insertion is *more* backward incompatible than changing dictionary order,
> > > which for a very long time has been known to not be guaranteed.  You're
> > > running some application, you upgrade Python because you apply all 
> > > security
> > > fixes, and suddenly you're starting to get exceptions in places you can't
> > > really do anything about.  Yet those exceptions are now part of the 
> > > documented
> > > public API for dictionaries.  This is asking for trouble.  Bugs will 
> > > suddenly
> > > start appearing in that application's tracker and they will seem to the
> > > application developer like Python just added a new public API in a 
> > > security
> > > release.
> > 
> > Dict insertion can already raise an exception: MemoryError. I think we 
> > should be safe if the new exception also derives from BaseException. We 
> > should actually eriously consider just raising MemoryException, since 
> > introducing a new built-in exception in a bugfix release is also very 
> > questionable: code explicitly catching or raising it would not work on 
> > previous bugfix releases of the same feature release.
> > 
> > > OTOH, if you change dictionary order and *that* breaks the application, 
> > > then
> > > the bugs submitted to the application's tracker will be legitimate bugs 
> > > that
> > > have to be fixed even if nothing else changed.
> > 
> > There are lots of things that are undefined according to the language spec 
> > (and quite possibly known to vary between versions or platforms or 
> > implementations like PyPy or Jython) but which we would never change in a 
> > bugfix release.
> > 
> > > So I still think we should ditch the paranoia about dictionary order 
> > > changing,
> > > and fix this without counting.  A little bit of paranoia could creep back 
> > > in
> > > by disabling the hash fix by default in stable releases, but I think it 
> > > would
> > > be fine to make that a compile-time option.
> > 
> > I'm sorry, but I don't want to break a user's app with a bugfix release and 
> > say "haha your code was already broken you just didn't know it".
> > 
> > Sure, the dict order already varies across Python implementations, possibly 
> > across 32/64 bits or operating systems. But many organizations (I know a 
> > few :-) have a very large installed software base, created over many years 
> > by many people with varying skills, that is kept working in part by very 
> > carefully keeping the environment as constant as possible. This means that 
> > the target environment is much more predictable than it is for the typical 
> > piece of open source software.
> > 
> > Sure, a good Python developer doesn't write apps or tests that depend on 
> > dict order. But time and again we see that not everybody writes perfect 
> > code every time. Especially users writing "in-house" apps (as opposed to 
> > frameworks shared as open source) are less likely to always use the most 
> > robust, portable algorithms in existence, because they may know with much 
> > more certainty that their code will never be used on certain combinations 
> > of platforms. For example, I rarely think  about whether code I write might 
> > not work on IronPython or Jython, or even CPython on Windows. And if 
> > something I wrote suddenly needs to be ported to one of those, well, that's 
> > considered a port and I'll just accept that it might mean changing a few 
> > things.
> > 
> > The time to break a dependency on dict order is not with a b

Re: [Python-Dev] Counting collisions for the win

[Case Van Horsen]

On Fri, Jan 20, 2012 at 8:17 AM, Victor Stinner
 wrote:
>> So I still think we should ditch the paranoia about dictionary order 
>> changing,
>> and fix this without counting.
>
> The randomized hash has other issues:
>
>  - its security is based on its secret, whereas it looks to be easy to
> compute it (see more details in the issue)
>  - my patch only changes hash(str), whereas other developers asked me
> to patch also bytes, int and other types

Changing hash(int) on a bugfix release will cause issues with
extensions (gmpy, sage, probably others) that calculate the hash of
numerical objects.

>
> hash(bytes) can be changed. But changing hash(int) may leak easily the
> secret. We may use a different secret for each type, but if it is easy
> to compute int hash secret, dictionaries using int are still
> vulnerable.
>
> --
>
> There is no perfect solutions, drawbacks of each solution should be compared.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/casevh%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Georg Brandl]

Am 20.01.2012 00:54, schrieb "Martin v. Löwis":
>> I can't help noticing that so far, worries about the workload came mostly 
>> from
>> people who don't actually bear that load (this is no accusation!), while 
>> those
>> that do are the proponents of the PEP...
> 
> Ok, so let me add then that I'm worried about the additional work-load.
> 
> I'm particularly worried about the coordination of vacation across the
> three people that work on a release. It might well not be possible to
> make any release for a period of two months, which, in a six-months
> release cycle with two alphas and a beta, might mean that we (the
> release people) would need to adjust our vacation plans with the release
> schedule, or else step down (unless you would release the "normal"
> feature releases as source-only releases).

Thanks for the reminder, Martin.  Even with the current release schedule,
I think that the load on you is too much, and we need a whole team of
Windows release experts.  It's not really fair that the RM usually changes
from release to release (at least every 2), and you have to do the same
for everyone.

It looks like we have one volunteer already; if we find another, I think
one of them will also be not on vacation at most times :)

For the Mac, at least we're up to two experts, but I'd like to see a
third there too.

cheers,
Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Terry Reedy]

On 1/20/2012 10:55 AM, Frank Sievertsen wrote:

Hello,

I still see at least two ways to create a DOS attack even with the
collison-counting-patch.



2. The second attack actually attacks that 1000 allowed string
comparisons are still a lot of work.
First I added 999 strings that collide with a one-byte string "a". In
some applications a zero-byte string might work even better. Then I
can add a many thousand of the "a"'s, just like the first attack.


If 1000 were replaced by, for instance, random.randint(700,1000) the 
dict could not be set to have an exception triggered with one other 
entry (which I believe was Martin's idea). But I suppose you would say 
that 699 entries would still make for much work.


The obvious defense for this particular attack is to reject duplicate 
keys. Perhaps there should be write-once string sets and dicts available.


This gets to the point that there is no best blind defense to all 
possible attacks.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 01/20/2012 02:04 PM, Donald Stufft wrote:

> Even if a MemoryException is raised I believe that is still a 
> fundamental change in the documented contract of dictionary API.

How so?  Dictionary inserts can *already* raise that error.

> I don't believe there is a way to fix this without breaking someones 
> application. The major differences I see between the two solutions is
>  that counting will break people's applications who are otherwise 
> following the documented api contract of dictionaries,

Do you have a case in mind where legitimate user data (not crafted as
part of a DoS attack) would trip the 1000-collision limit?  How likely is
it that such cases exist in already-deployed applications, compared to
the known breakage in existing applications due to hash randomization?

> and randomization will break people's applications who are violating 
> the documented api contract of dictionaries.
> 
> Personally I feel that the lesser of two evils is to reward those who
>  followed the documentation, and not reward those who didn't.

Except that I think your set is purely hypothetical, while the second set
is *lots* of deployed applications.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8ZwlgACgkQ+gerLs4ltQ4KOACglAHDgn5wUb+cye99JbeW0rZo
5oAAn2ja7K4moFLN/aD4ZP7m+8WnwhcA
=u7Mt
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 now uses Sphinx 1.0

[Chris Withers]

On 14/01/2012 16:14, Sandro Tosi wrote:

Hello,
just a heads-up: documentation for 2.7 branch has been ported to use
sphinx 1.0, so now the same syntax can be used for 2.x and 3.x
patches, hopefully easying working on both python stacks.


That's great news, does that now mean the objects inventory for Python 
2.7 and Python 3 on python.org now supports referring to section headers 
from 3rd party packages?


Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Donald Stufft]

On Friday, January 20, 2012 at 2:36 PM, Tres Seaver wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 01/20/2012 02:04 PM, Donald Stufft wrote:
> 
> > Even if a MemoryException is raised I believe that is still a 
> > fundamental change in the documented contract of dictionary API.
> > 
> 
> 
> How so? Dictionary inserts can *already* raise that error.
Because it's raising it for a fundamentally different thing. "You have plenty 
of memory, but we decided to add an arbitrary limit that has nothing to do with 
memory and pretend you are out of memory anyways".
> 
> > I don't believe there is a way to fix this without breaking someones 
> > application. The major differences I see between the two solutions is
> > that counting will break people's applications who are otherwise 
> > following the documented api contract of dictionaries,
> > 
> 
> 
> Do you have a case in mind where legitimate user data (not crafted as
> part of a DoS attack) would trip the 1000-collision limit? How likely is
> it that such cases exist in already-deployed applications, compared to
> the known breakage in existing applications due to hash randomization?
> 
> 

I don't, but as there's never been a limit on how many collisions a dictionary 
can have, this would be a fundamental change in the documented (and 
undocumented) abilities of a dictionary. Dictionary key order has never been 
guaranteed, is documented to not be relied on, already changes depending on if 
you are using 32bit, 64bit, Jython, PyPy etc or as someone else pointed out, to 
any number of possible improvements to dict. The counting solution violates the 
existing contract in order to serve people who themselves are violating the 
contract. Even with their violation the method that I +1'd still serves to not 
break existing applications by default.
> 
> > and randomization will break people's applications who are violating 
> > the documented api contract of dictionaries.
> > 
> > Personally I feel that the lesser of two evils is to reward those who
> > followed the documentation, and not reward those who didn't.
> > 
> 
> 
> Except that I think your set is purely hypothetical, while the second set
> is *lots* of deployed applications.
> 
> 

Which is why I believe that it should be off by default on the bugfix, but 
easily enabled. (Flag, env var, whatever). That allows people to upgrade to a 
bugfix without breaking their application, and if this vulnerability affects 
them, they can enable it.

I think the counting collision is at best a bandaid and not a proper fix 
stemmed from a desire to not break existing applications on a bugfix release 
which can be better solved by implementing the real fix and allowing people to 
control (only on the bugfix, on 3.3+ it should be forced to on always) if they 
have it enabled or not.
> 
> 
> Tres.
> - -- 
> ===
> Tres Seaver +1 540-429-0999 tsea...@palladion.com 
> (mailto:tsea...@palladion.com)
> Palladion Software "Excellence by Design" http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk8ZwlgACgkQ+gerLs4ltQ4KOACglAHDgn5wUb+cye99JbeW0rZo
> 5oAAn2ja7K4moFLN/aD4ZP7m+8WnwhcA
> =u7Mt
> -END PGP SIGNATURE-
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org (mailto:Python-Dev@python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
> 
> 


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Ethan Furman]

Donald Stufft wrote:
Even if a MemoryException is raised I believe that is still a 
fundamental change in the documented contract of dictionary API. I don't 
believe there is a way to fix this without breaking someones 
application. The major differences I see between the two solutions is 
that counting will break people's applications who are otherwise 
following the documented api contract of dictionaries, and randomization 
will break people's applications who are violating the documented api 
contract of dictionaries. 

Personally I feel that the lesser of two evils is to reward those who 
followed the documentation, and not reward those who didn't.


So +1 for Randomization as the only option in 3.3, and off by default 
with a flag or environment variable in bug fixes. I think it's the only 
way to proceed that won't hurt people who have followed the documented 
behavior.


+1

~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 11:51 AM, Donald Stufft wrote:

>  On Friday, January 20, 2012 at 2:36 PM, Tres Seaver wrote:
>
> On 01/20/2012 02:04 PM, Donald Stufft wrote:
>
> Even if a MemoryException is raised I believe that is still a
> fundamental change in the documented contract of dictionary API.
>
> How so? Dictionary inserts can *already* raise that error.
>
> Because it's raising it for a fundamentally different thing. "You have
> plenty of memory, but we decided to add an arbitrary limit that has nothing
> to do with memory and pretend you are out of memory anyways".
>

Actually due to fragmentation that can already happen.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Tim Golden]

On 20/01/2012 19:14, Georg Brandl wrote:

Am 20.01.2012 00:54, schrieb "Martin v. Löwis":

I can't help noticing that so far, worries about the workload came mostly from
people who don't actually bear that load (this is no accusation!), while those
that do are the proponents of the PEP...


Ok, so let me add then that I'm worried about the additional work-load.

I'm particularly worried about the coordination of vacation across the
three people that work on a release. It might well not be possible to
make any release for a period of two months, which, in a six-months
release cycle with two alphas and a beta, might mean that we (the
release people) would need to adjust our vacation plans with the release
schedule, or else step down (unless you would release the "normal"
feature releases as source-only releases).


Thanks for the reminder, Martin.  Even with the current release schedule,
I think that the load on you is too much, and we need a whole team of
Windows release experts.  It's not really fair that the RM usually changes
from release to release (at least every 2), and you have to do the same
for everyone.

It looks like we have one volunteer already; if we find another, I think
one of them will also be not on vacation at most times :)



I'm certainly happy to help out there. Like everyone I'm
not always clear on my availability but the more people
who know what needs to be done, the better ISTM.

TJG
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Ethan Furman]

Benjamin Peterson wrote:

2012/1/20 Ethan Furman :

Summary:

Exception Chaining is cool, unless you are writing libraries that want to
transform from Exception X to Exception Y as the the previous exception
context is unnecessary, potentially confusing, and cluttery (yup, just made
that word up!).

For all the gory details, see http://bugs.python.org/issue6210.

I'm going to attempt a patch implementing MRAB's suggestion:

try:
   some_op
except ValueError:
   raise as OtherError() # `raise` keeps context, `raise as` does not


I dislike this syntax. Raise what as OtherError()? I think the "raise
x from None" idea is preferable, since it indicates you are nulling
the context. The optimal solution would be to have "raise X
nocontext", but that would obviously require another keyword...


Raise 'the error' as OtherError.

The problem I have with 'raise x from None' is it puts 'from None' clear 
at the end of line -- not a big deal on this very short example, but 
when you have actual text it's not as obvious:


except SomeError():
raise SomeOtherError('explanatory text with actual %data to help 
track down the problem' % data) from None


Of course, I suppose that same issue exists with the 'raise x from exc' 
syntax, and 'from None' certainly matches that better...


~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Benjamin Peterson]

2012/1/20 Ethan Furman :
> Benjamin Peterson wrote:
>>
>> 2012/1/20 Ethan Furman :
>>>
>>> Summary:
>>>
>>> Exception Chaining is cool, unless you are writing libraries that want to
>>> transform from Exception X to Exception Y as the the previous exception
>>> context is unnecessary, potentially confusing, and cluttery (yup, just
>>> made
>>> that word up!).
>>>
>>> For all the gory details, see http://bugs.python.org/issue6210.
>>>
>>> I'm going to attempt a patch implementing MRAB's suggestion:
>>>
>>> try:
>>>   some_op
>>> except ValueError:
>>>   raise as OtherError() # `raise` keeps context, `raise as` does not
>>
>>
>> I dislike this syntax. Raise what as OtherError()? I think the "raise
>> x from None" idea is preferable, since it indicates you are nulling
>> the context. The optimal solution would be to have "raise X
>> nocontext", but that would obviously require another keyword...
>
>
> Raise 'the error' as OtherError.

Where 'the error' is? Aren't you trying to override the current error?

>
> The problem I have with 'raise x from None' is it puts 'from None' clear at
> the end of line -- not a big deal on this very short example, but when you
> have actual text it's not as obvious:
>
> except SomeError():
>    raise SomeOtherError('explanatory text with actual %data to help track
> down the problem' % data) from None
>
> Of course, I suppose that same issue exists with the 'raise x from exc'
> syntax, and 'from None' certainly matches that better...

Exactly!


-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] negative ref count on windows debug version

[Dr.-Ing. Ingo D. Rullhusen]

Hello,

using

loc = PyDict_New();
Py_XDECREF(loc);

or

PyObject *src = Py_CompileString( code.toStdString().c_str(),
"", Py_single_input );
Py_XDECREF(src);

results in a "object at blahblah has negative ref count -1" error on
windows visual studio in debug mode. And yes, python is compiled and
linked in debug mode also. The release version seems to work.

This happens in version 2.6.7 and 2.7.2.

Any hints?

Thanks
   Ingo
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 407: New release cycle and introducing long-term support versions

[Georg Brandl]

Am 20.01.2012 21:08, schrieb Tim Golden:
> On 20/01/2012 19:14, Georg Brandl wrote:
>> Am 20.01.2012 00:54, schrieb "Martin v. Löwis":
 I can't help noticing that so far, worries about the workload came mostly 
 from
 people who don't actually bear that load (this is no accusation!), while 
 those
 that do are the proponents of the PEP...
>>>
>>> Ok, so let me add then that I'm worried about the additional work-load.
>>>
>>> I'm particularly worried about the coordination of vacation across the
>>> three people that work on a release. It might well not be possible to
>>> make any release for a period of two months, which, in a six-months
>>> release cycle with two alphas and a beta, might mean that we (the
>>> release people) would need to adjust our vacation plans with the release
>>> schedule, or else step down (unless you would release the "normal"
>>> feature releases as source-only releases).
>>
>> Thanks for the reminder, Martin.  Even with the current release schedule,
>> I think that the load on you is too much, and we need a whole team of
>> Windows release experts.  It's not really fair that the RM usually changes
>> from release to release (at least every 2), and you have to do the same
>> for everyone.
>>
>> It looks like we have one volunteer already; if we find another, I think
>> one of them will also be not on vacation at most times :)
> 
> 
> I'm certainly happy to help out there. Like everyone I'm
> not always clear on my availability but the more people
> who know what needs to be done, the better ISTM.

Definitely.  Thanks for volunteering, Tim!

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Georg Brandl]

Am 20.01.2012 21:05, schrieb Ethan Furman:
> Benjamin Peterson wrote:
>> 2012/1/20 Ethan Furman :
>>> Summary:
>>>
>>> Exception Chaining is cool, unless you are writing libraries that want to
>>> transform from Exception X to Exception Y as the the previous exception
>>> context is unnecessary, potentially confusing, and cluttery (yup, just made
>>> that word up!).
>>>
>>> For all the gory details, see http://bugs.python.org/issue6210.
>>>
>>> I'm going to attempt a patch implementing MRAB's suggestion:
>>>
>>> try:
>>>some_op
>>> except ValueError:
>>>raise as OtherError() # `raise` keeps context, `raise as` does not
>> 
>> I dislike this syntax. Raise what as OtherError()? I think the "raise
>> x from None" idea is preferable, since it indicates you are nulling
>> the context. The optimal solution would be to have "raise X
>> nocontext", but that would obviously require another keyword...
> 
> Raise 'the error' as OtherError.
> 
> The problem I have with 'raise x from None' is it puts 'from None' clear 
> at the end of line -- not a big deal on this very short example, but 
> when you have actual text it's not as obvious:

Well, the "as" in "raise as" would be very easily overlooked too.

> except SomeError():
>  raise SomeOtherError('explanatory text with actual %data to help 
> track down the problem' % data) from None

In any case, I don't think the context suppression is the most important
thing about the exception raising, so it doesn't need to stand out...

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Terry Reedy]

Since 'raise' means 're-raise the current error', 'raise as OtherError' 
means (clearly to me, anyway) 're-raise the current error as 
OtherError'. This is just what you want to be able to say. Since 'raise' 
without a current error results in a TypeError, so should 'raise as 
OtherError'. I would just go with this as the proposal.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] negative ref count on windows debug version

[Amaury Forgeot d'Arc]

Hi,

2012/1/20 Dr.-Ing. Ingo D. Rullhusen 

> using
>
> loc = PyDict_New();
> Py_XDECREF(loc);
>
> or
>
> PyObject *src = Py_CompileString( code.toStdString().c_str(),
> "", Py_single_input );
> Py_XDECREF(src);
>
> results in a "object at blahblah has negative ref count -1" error on
> windows visual studio in debug mode. And yes, python is compiled and
> linked in debug mode also. The release version seems to work.
>
> This happens in version 2.6.7 and 2.7.2.
>

This looks very unlikely. Python itself is written with tons of similar
constructs,
and works very well in debug mode.
If you can isolate a reproducible case, please file a ticket on
bugs.python.org,
with all details: code, versions of the compiler, etc.

-- 
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Terry Reedy]

On 1/20/2012 2:51 PM, Donald Stufft wrote:


I think the counting collision is at best a bandaid and not a proper fix
stemmed from a desire to not break existing applications on a bugfix
release ...


My opinion of counting is better than yours, but even conceding the 
theoretical, purity argument, our release process is practical as well. 
There have been a few occasions when fixes to bugs in our code have been 
delayed from a bugfix release to the next feature release -- because the 
fix would break too much code depending on the bug.


Some years ago there was a proposal that we should deliberately tweak 
hash() to break 'buggy' code that depended on it not changing. This 
never happened. So it has been left de facto constant, to the extent it 
is, for some years.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Ben Wolfson]

On Fri, Jan 20, 2012 at 2:11 PM, Terry Reedy  wrote:
> On 1/20/2012 2:51 PM, Donald Stufft wrote:
>
>> I think the counting collision is at best a bandaid and not a proper fix
>> stemmed from a desire to not break existing applications on a bugfix
>> release ...
>
> My opinion of counting is better than yours, but even conceding the
> theoretical, purity argument, our release process is practical as well.
> There have been a few occasions when fixes to bugs in our code have been
> delayed from a bugfix release to the next feature release -- because the fix
> would break too much code depending on the bug.

AFAICT Brett's suggestion (which had occurred to me as well, but I'm
not a core developer by any stretch) seemed to get lost in the debate:
would it be possible to go with collision counting for bugfix releases
and hash randomization for new feature releases? (Brett made it here:
.)

-- 
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks,
which may be sweet, aromatic, fermented or spirit-based. ... Family
and social life also offer numerous other occasions to consume drinks
for pleasure." [Larousse, "Drink" entry]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Ethan Furman]

Georg Brandl wrote:


Well, the "as" in "raise as" would be very easily overlooked too.

In any case, I don't think the context suppression is the most important
thing about the exception raising, so it doesn't need to stand out...


Good point.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Donald Stufft]

I believe that either solution has the potential to break existing applications 
so to ensure that no applications are broken there will need to be a method of 
disabling it. I also believe that to maintain the backwards compatibility that 
Python has traditionally had in bug fix releases that either solution will need 
to default to off. 

Given those 2 things that I believe, I don't think that the argument should be 
which solution will break less, because I believe both will need to be off by 
default, but which solution more adequately solves the underlying problem. 


On Friday, January 20, 2012 at 5:11 PM, Terry Reedy wrote:

> On 1/20/2012 2:51 PM, Donald Stufft wrote:
> 
> > I think the counting collision is at best a bandaid and not a proper fix
> > stemmed from a desire to not break existing applications on a bugfix
> > release ...
> > 
> 
> 
> My opinion of counting is better than yours, but even conceding the 
> theoretical, purity argument, our release process is practical as well. 
> There have been a few occasions when fixes to bugs in our code have been 
> delayed from a bugfix release to the next feature release -- because the 
> fix would break too much code depending on the bug.
> 
> Some years ago there was a proposal that we should deliberately tweak 
> hash() to break 'buggy' code that depended on it not changing. This 
> never happened. So it has been left de facto constant, to the extent it 
> is, for some years.
> 
> -- 
> Terry Jan Reedy
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org (mailto:Python-Dev@python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
> 
> 


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Frank Sievertsen]

Am 20.01.2012 16:33, schrieb Guido van Rossum:
(I'm thinking that the original attack is trivial once the set of 
65000 colliding keys is public knowledge, which must be only a matter 
of time.



I think it's very likely that this will happen soon.

For ASP and PHP there is attack-payload publicly available.
PHP and ASP have patches to limit the number of query-variables.

We're very lucky that there's no public payload for python yet,
and all non-public software and payload I'm aware of is based
upon my software.

But this can change any moment. It's not really difficult to
write software to create 32bit-collisions.

Frank
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python build failed on mac

[Vijay Majagaonkar]

On 2012-01-20, at 4:29 AM, Hynek Schlawack wrote:

> Hello Vijay 
> 
> 
> Am Freitag, 20. Januar 2012 um 00:56 schrieb Vijay N. Majagaonkar:
> 
>> I am trying to build python 3 on mac and build failing with following error 
>> can somebody help me with this
> It is a known bug that Apple's latest gcc-llvm (that comes with Xcode 4.1 by 
> default as gcc) miscompiles Python: http://bugs.python.org/issue13241 
> 
> make clean
> CC=clang ./configure && make -s
> 

Hi Hynek,

Thanks for the help, but above command need to run in different way

./configure CC=clang
make 


this allowed me to build the code but when ran test I got following error 
message


[363/364/3] test_io
python.exe(11411) malloc: *** mmap(size=9223372036854775808) failed (error 
code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(11411,0x7fff7a8ba960) malloc: *** mmap(size=9223372036854775808) 
failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(11411,0x7fff7a8ba960) malloc: *** mmap(size=9223372036854775808) 
failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug



I am using Mac OS-X 10.7.2 and insatlled Xcode 4.2.1  



Thanks
;)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Benjamin Peterson]

2012/1/20 Terry Reedy :
> Since 'raise' means 're-raise the current error', 'raise as OtherError'
> means (clearly to me, anyway) 're-raise the current error as OtherError'.

That doesn't make any sense. You're changing the exception completely
not reraising it.

-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 2:33 PM, Ben Wolfson  wrote:
> On Fri, Jan 20, 2012 at 2:11 PM, Terry Reedy  wrote:
>> On 1/20/2012 2:51 PM, Donald Stufft wrote:
>>
>>> I think the counting collision is at best a bandaid and not a proper fix
>>> stemmed from a desire to not break existing applications on a bugfix
>>> release ...
>>
>> My opinion of counting is better than yours, but even conceding the
>> theoretical, purity argument, our release process is practical as well.
>> There have been a few occasions when fixes to bugs in our code have been
>> delayed from a bugfix release to the next feature release -- because the fix
>> would break too much code depending on the bug.
>
> AFAICT Brett's suggestion (which had occurred to me as well, but I'm
> not a core developer by any stretch) seemed to get lost in the debate:
> would it be possible to go with collision counting for bugfix releases
> and hash randomization for new feature releases? (Brett made it here:
> .)

I made it earlier.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 2:35 PM, Frank Sievertsen  wrote:
> Am 20.01.2012 16:33, schrieb Guido van Rossum:
>
>> (I'm thinking that the original attack is trivial once the set of 65000
>> colliding keys is public knowledge, which must be only a matter of time.
>
>
>
> I think it's very likely that this will happen soon.
>
> For ASP and PHP there is attack-payload publicly available.
> PHP and ASP have patches to limit the number of query-variables.
>
> We're very lucky that there's no public payload for python yet,
> and all non-public software and payload I'm aware of is based
> upon my software.
>
> But this can change any moment. It's not really difficult to
> write software to create 32bit-collisions.

While we're debating the best fix, could we allow people to at least
protect themselves against script-kiddies by offering fixes to cgi.py,
django, webob and a few other popular frameworks that limits forms to
1000 keys? (I suppose it's really only POST requests that are
vulnerable to script kiddies, because of the length restriction on
URLs.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hashing proposal: change only string-only dicts

[Amaury Forgeot d'Arc]

2012/1/19 Gregory P. Smith 

> str[-1] is not likely to work if you want to maintain ABI compatibility.
>  Appending it to the data after the terminating \0 is more likely to be
> possible, but if there is any possibility that existing compiled extension
> modules have somehow inlined code to do allocation of the str field even
> that is questionable (i don't think there are?).


There are. Unfortunately.
https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/scalarapi.c#L710

-- 
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Steven D'Aprano]

Benjamin Peterson wrote:

2012/1/20 Terry Reedy :

Since 'raise' means 're-raise the current error', 'raise as OtherError'
means (clearly to me, anyway) 're-raise the current error as OtherError'.


That doesn't make any sense. You're changing the exception completely
not reraising it.



I expect Terry is referring to the coder's intention, not the actual nuts and 
bolts of how it is implemented.


def spam():
try:
something()
except HamError:
raise SpamError

is implemented by catching a HamError and raising a completely different 
SpamError, but the intention is to "replace the HamError which actually 
occurred with a more appropriate SpamError".


At least that is *my* intention when I write code like the above, and it 
appears to be the usual intention in code I've seen that uses that idiom. 
Typically SpamError is part of the function's API while HamError is not.


The fact that Python doesn't actually "replace" anything is besides the point. 
The purpose of the idiom is to turn one exception into another exception, 
which is as close as we can get to re-raising HamError as a SpamError instead.


(It's not actually a re-raise as the traceback will point to a different line 
of code, but it's close enough.)


I'd prefer "raise SpamError from None", but I understand that this cannot work 
due to technical limitations. If that is the case, then "raise as SpamError" 
works for me.



--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Steven D'Aprano]

Guido van Rossum wrote:

On Fri, Jan 20, 2012 at 11:51 AM, Donald Stufft wrote:


 On Friday, January 20, 2012 at 2:36 PM, Tres Seaver wrote:

On 01/20/2012 02:04 PM, Donald Stufft wrote:

Even if a MemoryException is raised I believe that is still a
fundamental change in the documented contract of dictionary API.

How so? Dictionary inserts can *already* raise that error.

Because it's raising it for a fundamentally different thing. "You have
plenty of memory, but we decided to add an arbitrary limit that has nothing
to do with memory and pretend you are out of memory anyways".



Actually due to fragmentation that can already happen.


Whether you have run out of total memory, or a single contiguous block, it is 
still a memory error.


An arbitrary limit on collisions is not a memory error. If we were designing 
this API from scratch, would anyone propose using MemoryError for "you have 
reached a limit on collisions"? It has nothing to do with memory. Using 
MemoryError for something which isn't a memory error is ugly.


How about RuntimeError?



--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

It should derive from BaseException.

--Guido van Rossum (sent from Android phone)
On Jan 20, 2012 4:59 PM, "Steven D'Aprano"  wrote:

> Guido van Rossum wrote:
>
>> On Fri, Jan 20, 2012 at 11:51 AM, Donald Stufft 
>> **wrote:
>>
>>   On Friday, January 20, 2012 at 2:36 PM, Tres Seaver wrote:
>>>
>>> On 01/20/2012 02:04 PM, Donald Stufft wrote:
>>>
>>> Even if a MemoryException is raised I believe that is still a
>>> fundamental change in the documented contract of dictionary API.
>>>
>>> How so? Dictionary inserts can *already* raise that error.
>>>
>>> Because it's raising it for a fundamentally different thing. "You have
>>> plenty of memory, but we decided to add an arbitrary limit that has
>>> nothing
>>> to do with memory and pretend you are out of memory anyways".
>>>
>>>
>> Actually due to fragmentation that can already happen.
>>
>
> Whether you have run out of total memory, or a single contiguous block, it
> is still a memory error.
>
> An arbitrary limit on collisions is not a memory error. If we were
> designing this API from scratch, would anyone propose using MemoryError for
> "you have reached a limit on collisions"? It has nothing to do with memory.
> Using MemoryError for something which isn't a memory error is ugly.
>
> How about RuntimeError?
>
>
>
> --
> Steven
>
> __**_
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Steven D'Aprano]

Terry Reedy wrote:

On 1/20/2012 11:17 AM, Victor Stinner wrote:

There is no perfect solutions, drawbacks of each solution should be 
compared.


Amen.

One possible attack that has been described for a collision counting 
dict depends on knowing precisely the trigger point. So let 
MAXCOLLISIONS either be configureable or just choose a random count 
between M and N, say 700 and 999.


Have I missed something? Why wouldn't the attacker simply target 1000 
collisions, and if the collision triggers at 700 instead of 1000, that's a bonus?



--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Steven D'Aprano]

Guido van Rossum wrote:

It should derive from BaseException.


RuntimeError meets that requirement, and it is an existing exception so there 
are no issues with introducing a new built-in exception to a point release.


py> issubclass(RuntimeError, BaseException)
True


--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Benjamin Peterson]

2012/1/20 Steven D'Aprano :
> Guido van Rossum wrote:
>>
>> It should derive from BaseException.
>
>
> RuntimeError meets that requirement, and it is an existing exception so
> there are no issues with introducing a new built-in exception to a point
> release.
>
> py> issubclass(RuntimeError, BaseException)
> True

Guido meant a direct subclass.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Counting collisions for the win

[Guido van Rossum]

On Fri, Jan 20, 2012 at 6:33 PM, Steven D'Aprano  wrote:
> Guido van Rossum wrote:
>>
>> It should derive from BaseException.

> RuntimeError meets that requirement, and it is an existing exception so
> there are no issues with introducing a new built-in exception to a point
> release.
>
> py> issubclass(RuntimeError, BaseException)
> True

Sorry, I was ambiguous. I meant it should not derive from Exception.
It goes RuntimeError -> StandardError -> Exception -> BaseException.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Steven D'Aprano]

Ethan Furman wrote:

The question I have at the moment is:  should `raise as` be an error if 
no exception is currently being handled?


I think so.

"raise as Spam" essentially means "raise Spam with the context set to None". 
That's actually only useful if the context otherwise wouldn't be None, that 
is, if you're raising an exception when another exception is active. Doing it 
"just in case" defeats the usefulness of exception chaining, and should be 
discouraged.


It is easier to change our minds later and allow "raise as" outside of an 
except block, than to change our minds and forbid it.




Example:

def smurfy(x):
if x != 'magic flute':
 raise as WrongInstrument
do_something_with_x

If this is allowed then `smurfy` could be called from inside an `except` 
clause or outside it.


What's your use-case? The only one I can think of is this:

def choose_your_own_exception(x):
if x < 0:
raise as ValueError
elif x == 0:
raise as SpamError
elif x < 1:
raise as SmallerThanOneError
else:
raise as RuntimeError

try:
something()
except TypeError:
choose_your_own_exception(x)


I don't think we want to encourage such practices. Besides, if you really need 
such an exception selector, change every "raise as" to return, then do:


try:
something()
except TypeError:
raise as choose_your_own_exception(x)


Much more clear.



I don't care for it for two reasons:

  - I don't like the way it looks
  - I can see it encouraging always using `raise as` instead of `raise` 
and losing the value of exception chaining.


Other thoughts?

~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] exception chaining

[Terry Reedy]

On 1/20/2012 7:40 PM, Steven D'Aprano wrote:

Benjamin Peterson wrote:

2012/1/20 Terry Reedy :

Since 'raise' means 're-raise the current error', 'raise as OtherError'
means (clearly to me, anyway) 're-raise the current error as
OtherError'.


That doesn't make any sense. You're changing the exception completely
not reraising it.


I expect Terry is referring to the coder's intention, not the actual
nuts and bolts of how it is implemented.


Yes, same error situation, translated, typically from developer language 
to app language.



def spam():
try:
something()
except HamError:
raise SpamError

is implemented by catching a HamError and raising a completely different
SpamError, but the intention is to "replace the HamError which actually
occurred with a more appropriate SpamError".

At least that is *my* intention when I write code like the above, and it
appears to be the usual intention in code I've seen that uses that
idiom. Typically SpamError is part of the function's API while HamError
is not.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 now uses Sphinx 1.0

[Senthil Kumaran]

On Fri, Jan 20, 2012 at 07:05:41PM +, Chris Withers wrote:
> 
> That's great news, does that now mean the objects inventory for
> Python 2.7 and Python 3 on python.org now supports referring to
> section headers from 3rd party packages?

Nope. It does not seem to have any relation to that. Would you care to
explain more, possibly show an example in rst as what you mean?

-- 
Senthil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 now uses Sphinx 1.0

[Georg Brandl]

Am 20.01.2012 20:05, schrieb Chris Withers:
> On 14/01/2012 16:14, Sandro Tosi wrote:
>> Hello,
>> just a heads-up: documentation for 2.7 branch has been ported to use
>> sphinx 1.0, so now the same syntax can be used for 2.x and 3.x
>> patches, hopefully easying working on both python stacks.
> 
> That's great news, does that now mean the objects inventory for Python 
> 2.7 and Python 3 on python.org now supports referring to section headers 
> from 3rd party packages?

Yes, they should -- there's something wrong with the automatic build still,
but I'll fix that ASAP.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com