Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Antoine Pitrou]

Le Tue, 14 May 2013 10:41:01 +1200,
Ben Hoyt  a écrit :

> > I'd to see the numbers for NFS or CIFS - stat() can be brutally slow
> > over a network connection (that's why we added a caching mechanism
> > to importlib).
> 
> How do I know what file system Windows networking is using? In any
> case, here's some numbers on Windows -- it's looking pretty good! This
> is with default DEPTH/NUM_DIRS/NUM_FILES on a LAN:
> 
> Benchmarking walks on \\anothermachine\docs\Ben\bigtree, repeat 3/3...
> os.walk took 11.345s, scandir.walk took 0.340s -- 33.3x as fast
> 
> And this is on a VPN on a remote network with the benchmark.py values
> cranked down to DEPTH = 3, NUM_DIRS = 3, NUM_FILES = 20 (because
> otherwise it was taking far too long):
> 
> Benchmarking walks on \\ben1.titanmt.local\c$\dev\scandir\benchtree,
> repeat 3/3...
> os.walk took 122.310s, scandir.walk took 5.452s -- 22.4x as fast
> 
> If anyone can run benchmark.py on Linux / NFS or similar, that'd be
> great. You'll probably have to lower DEPTH/NUM_DIRS/NUM_FILES first
> and then move the "benchtree" to the network file system to run it
> against that.

Why does your benchmark create such large files? It doesn't make sense.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Antoine Pitrou]

Le Tue, 14 May 2013 10:41:01 +1200,
Ben Hoyt  a écrit :
> 
> If anyone can run benchmark.py on Linux / NFS or similar, that'd be
> great. You'll probably have to lower DEPTH/NUM_DIRS/NUM_FILES first
> and then move the "benchtree" to the network file system to run it
> against that.

On a locally running VM:
os.walk took 0.400s, scandir.walk took 0.120s -- 3.3x as fast

Same VM accessed from the host through a local sshfs:
os.walk took 2.261s, scandir.walk took 2.055s -- 1.1x as fast

Same, but with "sshfs -o cache=no":
os.walk took 24.060s, scandir.walk took 25.906s -- 0.9x as fast

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 379 Python launcher for Windows - behaviour for #!/usr/bin/env python line is wrong

[Paul Moore]

On 5 May 2013 18:10, Paul Moore  wrote:

>
> On 4 May 2013 16:42, Vinay Sajip  wrote:
>
>> I've taken a quick look at it, but I probably won't be able to make any
>> changes until the near the end of the coming week. Feel free to have a go;
>>
>
> OK, I have a patch against the standalone pylauncher repo at
> https://bitbucket.org/pmoore/pylauncher. I'm not sure what the best
> approach is - I didn't want to patch the python core version directly (a)
> because I wouldn't be able to test it easily, and (b) because I'd want a
> standalone version anyway until 3.4 comes out.
>

Vinay,
Did you get a chance to have a look at this? I didn't manage to create a
pull request against your copy of pylauncher as my repo is a fork of the
pypa one - I'm not sure if that's a limitation of bitbucket or if I just
don't know how to do it... I've created a pull request against the pypa
version in case that's of use...

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Ben Hoyt]

>> If anyone can run benchmark.py on Linux / NFS or similar, that'd be
>> great. You'll probably have to lower DEPTH/NUM_DIRS/NUM_FILES first
>> and then move the "benchtree" to the network file system to run it
>> against that.
>
> Why does your benchmark create such large files? It doesn't make sense.

Yeah, I was just thinking about that last night, and I should probably
change that. Originally I did it because I thought it might affect the
speed of directory walking, so I was trying to make some of the files
large to be more "real world". I've just tested it, and in practice
file system doesn't make much difference, so I've fixed that now:

https://github.com/benhoyt/scandir/commit/9663c0afcc5c020d5d1fe34a120b0331b8c9d2e0

Thanks,
Ben
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Antoine Pitrou]

Le Tue, 14 May 2013 20:54:50 +1200,
Ben Hoyt  a écrit :

> >> If anyone can run benchmark.py on Linux / NFS or similar, that'd be
> >> great. You'll probably have to lower DEPTH/NUM_DIRS/NUM_FILES first
> >> and then move the "benchtree" to the network file system to run it
> >> against that.
> >
> > Why does your benchmark create such large files? It doesn't make
> > sense.
> 
> Yeah, I was just thinking about that last night, and I should probably
> change that. Originally I did it because I thought it might affect the
> speed of directory walking, so I was trying to make some of the files
> large to be more "real world". I've just tested it, and in practice
> file system doesn't make much difference, so I've fixed that now:

Thanks. I had bumped the number of files, thinking it would make things
more interesting, and it filled my disk.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Ben Hoyt]

>> large to be more "real world". I've just tested it, and in practice
>> file system doesn't make much difference, so I've fixed that now:
>
> Thanks. I had bumped the number of files, thinking it would make things
> more interesting, and it filled my disk.

Denial of Pitrou attack -- sorry! :-) Anyway, it shouldn't fill your
disk now. Though it still does use more on-disk space than 3 bytes per
file on most FSs, depending on the smallest block size.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Ben Hoyt]

> On a locally running VM:
> os.walk took 0.400s, scandir.walk took 0.120s -- 3.3x as fast
>
> Same VM accessed from the host through a local sshfs:
> os.walk took 2.261s, scandir.walk took 2.055s -- 1.1x as fast
>
> Same, but with "sshfs -o cache=no":
> os.walk took 24.060s, scandir.walk took 25.906s -- 0.9x as fast

Thanks. I take it those are "USING FAST C version"?

What is "-o cache=no"? I'm guessing the last one isn't giving dirents,
so my version is slightly slower than the built-in listdir/stat
version due to building and calling methods on the DirEntry objects in
Python. It should be no slower when it's all moved to C.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Antoine Pitrou]

Le Tue, 14 May 2013 21:10:08 +1200,
Ben Hoyt  a écrit :
> > On a locally running VM:
> > os.walk took 0.400s, scandir.walk took 0.120s -- 3.3x as fast
> >
> > Same VM accessed from the host through a local sshfs:
> > os.walk took 2.261s, scandir.walk took 2.055s -- 1.1x as fast
> >
> > Same, but with "sshfs -o cache=no":
> > os.walk took 24.060s, scandir.walk took 25.906s -- 0.9x as fast
> 
> Thanks. I take it those are "USING FAST C version"?

Yes.

> What is "-o cache=no"? I'm guessing the last one isn't giving dirents,
> so my version is slightly slower than the built-in listdir/stat
> version due to building and calling methods on the DirEntry objects in
> Python.

It disables sshfs's built-in cache (I suppose it's a filesystem
metadata cache). The man page doesn't tell much more about it.

> It should be no slower when it's all moved to C.

The slowdown is too small to be interesting. The main point is that
there was no speedup, though.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Ben Hoyt]

>> It should be no slower when it's all moved to C.
>
> The slowdown is too small to be interesting. The main point is that
> there was no speedup, though.

True, and thanks for testing.

I don't think that's a big issue, however. If it's 3-8x faster in the
majority of cases (local disk on all systems, Windows networking), and
no slower in a minority (sshfs), I'm not too sad about that.

I wonder how sshfs compared to nfs.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Antoine Pitrou]

Le Tue, 14 May 2013 22:14:42 +1200,
Ben Hoyt  a écrit :
> >> It should be no slower when it's all moved to C.
> >
> > The slowdown is too small to be interesting. The main point is that
> > there was no speedup, though.
> 
> True, and thanks for testing.
> 
> I don't think that's a big issue, however. If it's 3-8x faster in the
> majority of cases (local disk on all systems, Windows networking), and
> no slower in a minority (sshfs), I'm not too sad about that.
> 
> I wonder how sshfs compared to nfs.

Ok, with a NFS mount (default options, especially "sync") to the same
local VM:

First run:
os.walk took 17.137s, scandir.walk took 0.625s -- 27.4x as fast

Second run:
os.walk took 1.535s, scandir.walk took 0.617s -- 2.5x as fast

(something fishy with caches?)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Charles-François Natali]

> I wonder how sshfs compared to nfs.

(I've modified your benchmark to also test the case where data isn't
in the page cache).

Local ext3:
cached:
os.walk took 0.096s, scandir.walk took 0.030s -- 3.2x as fast
uncached:
os.walk took 0.320s, scandir.walk took 0.130s -- 2.5x as fast

NFSv3, 1Gb/s network:
cached:
os.walk took 0.220s, scandir.walk took 0.078s -- 2.8x as fast
uncached:
os.walk took 0.269s, scandir.walk took 0.139s -- 1.9x as fast
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Matthieu Brucher]

Very interesting. Although os.walk may not be widely used in cluster
applications, anything that lowers the number of calls to stat() in an
spplication is worthwhile for parallel filesystems as stat() is handled by
the only non-parallel node, the MDS.

Small test on another NFS drive:
Creating tree at benchtree: depth=4, num_dirs=5, num_files=50
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk took 0.117s, scandir.walk took 0.041s -- 2.8x as fast

I may try it on a Lustre FS if I have some time and if I don't forget about
this.

Cheers,

Matthieu


2013/5/14 Charles-François Natali 

> > I wonder how sshfs compared to nfs.
>
> (I've modified your benchmark to also test the case where data isn't
> in the page cache).
>
> Local ext3:
> cached:
> os.walk took 0.096s, scandir.walk took 0.030s -- 3.2x as fast
> uncached:
> os.walk took 0.320s, scandir.walk took 0.130s -- 2.5x as fast
>
> NFSv3, 1Gb/s network:
> cached:
> os.walk took 0.220s, scandir.walk took 0.078s -- 2.8x as fast
> uncached:
> os.walk took 0.269s, scandir.walk took 0.139s -- 1.9x as fast
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/matthieu.brucher%40gmail.com
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 379 Python launcher for Windows - behaviour for #!/usr/bin/env python line is wrong

[Vinay Sajip]

> From: Paul Moore 

>Did you get a chance to have a look at this? I didn't manage to create a pull 
>request against your copy of pylauncher as my repo
> is a fork of the pypa one - I'm not sure if that's a limitation of bitbucket 
> or if I just don't know how to do it... I've created a pull request
> against the pypa version in case that's of use...


Hi Paul,

Sorry I haven't had a chance yet - real life is very busy at the moment. A pull 
request against the pypa version is fine, and I will get to it soon - thanks 
for your patience.

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best practices for Enum

[Steven D'Aprano]

On 14/05/13 16:51, Gregory P. Smith wrote:
[...]

This sounds like a feature request for doctest.  doctest could be educated
about enums and automatically compare to the integer value for such cases.


Please no. Enums are not special enough to break the rules.

Good: "Doctests look at the object's repr."

Bad: "Doctests look at an object's repr, unless the object is an Enum, when it will 
look at the enum's value."

If I want a test that checks the enum's value, then I will write a doctest that 
explicitly checks the enum's value.



--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

[Dirkjan Ochtman]

On Tue, May 14, 2013 at 12:14 PM, Ben Hoyt  wrote:
> I don't think that's a big issue, however. If it's 3-8x faster in the
> majority of cases (local disk on all systems, Windows networking), and
> no slower in a minority (sshfs), I'm not too sad about that.

Might be interesting to test something status calls with a hacked Mercurial.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] How to debug python crashes

[Philippe Fremy]

Hi,

I have a reproducable crash on Windows XP with Python 2.7 which I would
like to investigate. I have Visual Studio 2008 installed and I
downloaded the pdb files. However I could not find any instructions on
how to use them and was unsuccessful at getting anything out of it.

I checked the developer guide but could not find anything on debugging
crashes. On internet, this seems to be also an underdocumented topic.

So, a few questions :
- is there some documentation to help debugging crashes ?
- are the pdb files released along python usable with Visual Studio and
stock Python ? Or do you need a hand-compiled version ?

cheers,

Philippe

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to debug python crashes

[Antoine Pitrou]

Le Tue, 14 May 2013 14:32:27 +0200,
Philippe Fremy  a écrit :
> Hi,
> 
> I have a reproducable crash on Windows XP with Python 2.7 which I
> would like to investigate. I have Visual Studio 2008 installed and I
> downloaded the pdb files. However I could not find any instructions on
> how to use them and was unsuccessful at getting anything out of it.

You may as well recompile Python in debug mode and then run it under
the Visual Studio debugger. VS 2008 is adequate for building Python 2.7.
See http://docs.python.org/devguide/setup.html#windows-compiling

(that doesn't answer your question about pdb files, it's simply that I
don't know the answer :-))

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to debug python crashes

[Tim Golden]

On 14/05/2013 13:32, Philippe Fremy wrote:
> I have a reproducable crash on Windows XP with Python 2.7 which I would
> like to investigate. I have Visual Studio 2008 installed and I
> downloaded the pdb files. However I could not find any instructions on
> how to use them and was unsuccessful at getting anything out of it.
> 
> I checked the developer guide but could not find anything on debugging
> crashes. On internet, this seems to be also an underdocumented topic.
> 
> So, a few questions :
> - is there some documentation to help debugging crashes ?

I don't think there is. As you say, it's somewhat underdocumented. Maybe
someone else can point to something, but I'm not aware of anything.

> - are the pdb files released along python usable with Visual Studio and
> stock Python ? Or do you need a hand-compiled version ?


I actually have no idea whether you drop in the .pdb files, but if you
have VS anyway, it's easy enough to build and run within VS and let the
debugger drop you into the code when it crashes.

Are you in a position to post a reproducible test case to the tracker?
Or were you holding back until you'd done some analysis?

TJG
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 435 - ref impl disc 2

[Ethan Furman]

On 05/13/2013 11:11 PM, Ethan Furman wrote:

On 05/13/2013 10:01 PM, Glenn Linderman wrote:

On 5/13/2013 7:36 PM, Ethan Furman wrote:

On 05/10/2013 10:15 PM, Glenn Linderman wrote:


So it is quite possible to marry the two, as Ethan helped me figure out using 
an earlier NamedInt class:

class NIE( IntET, Enum ):
 x = ('NIE.x', 1)
 y = ('NIE.y', 2)
 z = ('NIE.z', 4)

and then expressions involving members of NIE (and even associated integers) 
will be tracked... see demo1.py.

But the last few lines of demo1 demonstrate that NIE doesn't like, somehow, 
remember that its values, deep down under
the covers, are really int.  And doesn't even like them when they are wrapped 
into IntET objects.  This may or may not
be a bug in the current Enum implementation.


[demo1.py excerpt]
print( repr( NIE1( 1 ) + NIE1(2)))
print( repr( NIE1( IntET('NIE1.x', 1 )) + NIE1(2)))



So the questions are:
1) Is there a bug in ref435 Enum that makes demo1 report errors instead of 
those lines working?


Nope.


Well, if it isn't a bug, it will be interesting to read the documentation that 
explains the behavior, when the
documentation is written:

The "obvious" documentation would be that Enum names values of any type, 
particularly the first type in the
multiple-inheritance list. The values assigned to the enumeration members are 
used as parameters to the constructor of
that first type, but the value of the enumeration member itself is an item of 
the type, created by the constructor.

The __call__ syntax  [ EnumDerivation( value ) ] looks up enumeration members 
by value.

The obvious documentation would stop there.




The one piece of "marriage" that is interesting is to avoid specifying the name 
twice, and it seems your code

I'd expect  NIE1.x.value  to be  IntET('x', 1)  but I'll have to look more 
carefully at what you've done, when I have
some time next week. You may have made some "simplifying assumptions", and 
things _should_ be as simple as possible, but
no simpler... especially not if it leads to unexpected results.




This will take more thought than I have time for tonight, also. Right now, I 
think I want the value for NIE.x to be
IntET('NIE.x', 1 ).  And your code isn't achieving that at present, but maybe I 
just need to tweak __new__ and then
can... and maybe it cures the discrepancy in expectations mentioned earlier 
too...


Thank you for being persistent.  You are correct, the value should be an IntET 
(at least, with a custom __new__ ;).


You know, when you look at something you wrote the night before, and have no idea what you were trying to say, you know 
you were tired.  Ignore my parenthetical remark.


Okay, the value is now an IntET, as expected and appropriate.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] First post

[Carlos Nepomuceno]

Hi guys! This is my first post on this list.

I'd like have your opinion on how to safely implement WSGI on a production 
server.

My benchmarks show no performance differences between our PHP and Python 
environments. I'm using mod_wsgi v3.4 with Apache 2.4.

Is that ok or can it get faster?

Thanks in advance.

Regards,

Carlos
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to debug python crashes

[Philippe Fremy]

On 14/05/2013 14:49, Tim Golden wrote:
> On 14/05/2013 13:32, Philippe Fremy wrote:
>> I have a reproducable crash on Windows XP with Python 2.7 which I would
>> like to investigate. I have Visual Studio 2008 installed and I
>> downloaded the pdb files. However I could not find any instructions on
>> how to use them and was unsuccessful at getting anything out of it.
>>
>> I checked the developer guide but could not find anything on debugging
>> crashes. On internet, this seems to be also an underdocumented topic.
>>
>> So, a few questions :
>> - is there some documentation to help debugging crashes ?
> I don't think there is. As you say, it's somewhat underdocumented. Maybe
> someone else can point to something, but I'm not aware of anything.
But what's the reason for releasing them ? If you need to recompile
Python to use them, that would be strange because they are generated as
part of the compilation process anyway.

> Are you in a position to post a reproducible test case to the tracker?
> Or were you holding back until you'd done some analysis?

I can reproduce it systematically, with an open source project (I am
debugging winpdb) but this occurs in a middle of a multithreaded XML-RPC
server running a python debugger in another thread. So no, I don't have
a test case and identifying clearly the bug would make me a small and
happy python contributor.

cheers,

Philippe







___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] First post

[Brian Curtin]

On Tue, May 14, 2013 at 10:22 AM, Carlos Nepomuceno
 wrote:
> Hi guys! This is my first post on this list.
>
> I'd like have your opinion on how to safely implement WSGI on a production 
> server.
>
> My benchmarks show no performance differences between our PHP and Python 
> environments. I'm using mod_wsgi v3.4 with Apache 2.4.
>
> Is that ok or can it get faster?
>
> Thanks in advance.

Hi - this list is about the development of Python. For user questions,
python-list is a better place to ask this.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] First post

[Ethan Furman]

On 05/14/2013 08:22 AM, Carlos Nepomuceno wrote:

Hi guys! This is my first post on this list.


Hi Carlos!



I'd like have your opinion on how to safely implement WSGI on a production 
server.


Unfortunately  this list is for the development /of/ Python, no development 
/with/ Python.

Try asking again over on the regular Python list:

http://mail.python.org/mailman/listinfo/python-list

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best practices for Enum

[Gregory P. Smith]

Bad: doctests.


On Tue, May 14, 2013 at 5:08 AM, Steven D'Aprano wrote:

> On 14/05/13 16:51, Gregory P. Smith wrote:
> [...]
>
>  This sounds like a feature request for doctest.  doctest could be educated
>> about enums and automatically compare to the integer value for such cases.
>>
>
> Please no. Enums are not special enough to break the rules.
>
> Good: "Doctests look at the object's repr."
>
> Bad: "Doctests look at an object's repr, unless the object is an Enum,
> when it will look at the enum's value."
>
> If I want a test that checks the enum's value, then I will write a doctest
> that explicitly checks the enum's value.
>
>
>
> --
> Steven
>
> __**_
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to debug python crashes

[Victor Stinner]

Hi,

I don't know if it can help, but if you really don't know where your
programcrash/hang occurs, you can use the faulthandler module:

https://pypi.python.org/pypi/faulthandler

It can be used to display te backtrace of all threads on an event like a
signal or a timeout.

It works with Python, but you will need a compiler (like Visual Studio) to
install it on Windows. I failed to build a MSI installer on Windows 64-bit
with Visual Studio 2010 express. If someone can help me to build MSI,
please contact me.

The documentation:

http://docs.python.org/dev/library/faulthandler.html

Victor

Le mardi 14 mai 2013, Philippe Fremy a écrit :

> Hi,
>
> I have a reproducable crash on Windows XP with Python 2.7 which I would
> like to investigate. I have Visual Studio 2008 installed and I
> downloaded the pdb files. However I could not find any instructions on
> how to use them and was unsuccessful at getting anything out of it.
>
> I checked the developer guide but could not find anything on debugging
> crashes. On internet, this seems to be also an underdocumented topic.
>
> So, a few questions :
> - is there some documentation to help debugging crashes ?
> - are the pdb files released along python usable with Visual Studio and
> stock Python ? Or do you need a hand-compiled version ?
>
> cheers,
>
> Philippe
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to debug python crashes

[Catalin Iacob]

Hi Philippe,

I don't have access to VS right now but out of my head what you need
to do is roughly outlined below.

On Tue, May 14, 2013 at 5:29 PM, Philippe Fremy  wrote:
> But what's the reason for releasing them ? If you need to recompile
> Python to use them, that would be strange because they are generated as
> part of the compilation process anyway.

They can indeed be used like this:

You should launch the python.exe process that is going to crash,
attach to it with the Visual Studio debugger and then reproduce the
crash. This should drop you in the debugger.

Once you're in the debugger and python.exe is stopped at the point of
the crash you should see the stack trace of each thread in a VS
window, the stacktrace will probably have lots of entries of the form
python27.dll! (no function names because there VS doesn't
know where to find the PDB files). If you right click one of those
entries there's an option named "Symbol load information" or similar,
this will show a window from which you can make VS ask you where on
disk do you have PDB files. You then tell VS where to find
python27.pdb and then the stacktrace entries should automatically get
function names.

Catalin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Pickling failure on Enums

[Ethan Furman]

On 05/13/2013 11:32 AM, Guido van Rossum wrote:


But now you enter a different phase of your project, or one of your 
collaborators does, or perhaps you've released your code on PyPI and one of 
your users does.  So someone tries to pickle some class instance that happens 
to contain an unpicklable enum.  That's not a great experience.  Pickling and 
unpickling errors are often remarkably hard to debug.  (Especially the latter, 
so I have privately admonished Ethan to ensure that if the getframe hack 
doesn't work, the pickle failure should happen at pickling time, not at 
unpickle time.)


I can get pickle failure on members created using the functional syntax with no module set; I cannot get pickle failure 
on those same classes; I cannot get pickle failure on class syntax enums that inherit complex types (such as the NEI 
class in the tests).


If anybody has any insight on how to make that work, I'm all ears.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pickling failure on Enums

[Guido van Rossum]

On Tue, May 14, 2013 at 1:09 PM, Ethan Furman  wrote:
> On 05/13/2013 11:32 AM, Guido van Rossum wrote:
>>
>>
>> But now you enter a different phase of your project, or one of your
>> collaborators does, or perhaps you've released your code on PyPI and one of
>> your users does.  So someone tries to pickle some class instance that
>> happens to contain an unpicklable enum.  That's not a great experience.
>> Pickling and unpickling errors are often remarkably hard to debug.
>> (Especially the latter, so I have privately admonished Ethan to ensure that
>> if the getframe hack doesn't work, the pickle failure should happen at
>> pickling time, not at unpickle time.)
>
>
> I can get pickle failure on members created using the functional syntax with
> no module set;

That's the case I care most about.

> I cannot get pickle failure on those same classes;

I suppose you mean "if you create the same enums using class syntax"?
Sounds fine to me.

> I cannot
> get pickle failure on class syntax enums that inherit complex types (such as
> the NEI class in the tests).

Is the NEI base class picklable?

> If anybody has any insight on how to make that work, I'm all ears.

I'm not 100% sure I know what "that" refers to here.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pickling failure on Enums

[Ethan Furman]

On 05/14/2013 01:58 PM, Guido van Rossum wrote:

On Tue, May 14, 2013 at 1:09 PM, Ethan Furman  wrote:

On 05/13/2013 11:32 AM, Guido van Rossum wrote:



But now you enter a different phase of your project, or one of your
collaborators does, or perhaps you've released your code on PyPI and one of
your users does.  So someone tries to pickle some class instance that
happens to contain an unpicklable enum.  That's not a great experience.
Pickling and unpickling errors are often remarkably hard to debug.
(Especially the latter, so I have privately admonished Ethan to ensure that
if the getframe hack doesn't work, the pickle failure should happen at
pickling time, not at unpickle time.)



I can get pickle failure on members created using the functional syntax with
no module set;


That's the case I care most about.


Good, 'cause that one is handled.  :)



I cannot get pickle failure on those same classes;


I suppose you mean "if you create the same enums using class syntax"?
Sounds fine to me.


No.  Example class:

--> Example = Enum('Example', 'example ie eg')  # no module name given, frame 
hack fails

--> pickle(Example.ie)
# blows up

--# pickle(Example)
# succeeds here, but unpickle will fail



I cannot get pickle failure on class syntax enums that inherit complex types
(such as the NEI class in the tests).


Is the NEI base class picklable?


No.  If it is, then the derived enum is also picklable (at least the variation I have tested, which is when the NEI base 
class has __getnewargs__).


I'm really hoping you'll say that can be a documentation issue.  ;)

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pickling failure on Enums

[Guido van Rossum]

On Tue, May 14, 2013 at 2:13 PM, Ethan Furman  wrote:
> On 05/14/2013 01:58 PM, Guido van Rossum wrote:
>>
>> On Tue, May 14, 2013 at 1:09 PM, Ethan Furman  wrote:
>>> I can get pickle failure on members created using the functional syntax
>>> with no module set;
>>
>>
>> That's the case I care most about.
>
>
> Good, 'cause that one is handled.  :)

Then we're good.

>>> I cannot get pickle failure on those same classes;
>>
>>
>> I suppose you mean "if you create the same enums using class syntax"?
>> Sounds fine to me.
>
>
> No.  Example class:
>
> --> Example = Enum('Example', 'example ie eg')  # no module name given,
> frame hack fails
>
> --> pickle(Example.ie)
> # blows up
>
> --# pickle(Example)
> # succeeds here, but unpickle will fail

Not great, but (a) few people pickle classes, and (b) there's probably
something you can do to the metaclass to sabotage this. But it's fine
to punt on this now.

>>> I cannot get pickle failure on class syntax enums that inherit complex
>>> types
>>> (such as the NEI class in the tests).
>>
>>
>> Is the NEI base class picklable?
>
>
> No.  If it is, then the derived enum is also picklable (at least the
> variation I have tested, which is when the NEI base class has
> __getnewargs__).
>
> I'm really hoping you'll say that can be a documentation issue.  ;)

Essentially the same response -- with enough hackery you can probably
get this to do what you want, but I wouldn't hold up a release for it.

For example you could file low-priority bugs for both issues in the
hope that someone else figures it out.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pickling failure on Enums

[Nick Coghlan]

On 15 May 2013 07:38, "Guido van Rossum"  wrote:
>
> On Tue, May 14, 2013 at 2:13 PM, Ethan Furman  wrote:
> > On 05/14/2013 01:58 PM, Guido van Rossum wrote:
> >>
> >> On Tue, May 14, 2013 at 1:09 PM, Ethan Furman 
wrote:
> >>> I can get pickle failure on members created using the functional
syntax
> >>> with no module set;
> >>
> >>
> >> That's the case I care most about.
> >
> >
> > Good, 'cause that one is handled.  :)
>
> Then we're good.
>
> >>> I cannot get pickle failure on those same classes;
> >>
> >>
> >> I suppose you mean "if you create the same enums using class syntax"?
> >> Sounds fine to me.
> >
> >
> > No.  Example class:
> >
> > --> Example = Enum('Example', 'example ie eg')  # no module name given,
> > frame hack fails
> >
> > --> pickle(Example.ie)
> > # blows up
> >
> > --# pickle(Example)
> > # succeeds here, but unpickle will fail
>
> Not great, but (a) few people pickle classes, and (b) there's probably
> something you can do to the metaclass to sabotage this. But it's fine
> to punt on this now.

It may be a bug in pickle - it sounds like it is sanity checking type(obj),
but not checking for cases where obj itself is a class.

Cheers,
Nick.

>
> >>> I cannot get pickle failure on class syntax enums that inherit complex
> >>> types
> >>> (such as the NEI class in the tests).
> >>
> >>
> >> Is the NEI base class picklable?
> >
> >
> > No.  If it is, then the derived enum is also picklable (at least the
> > variation I have tested, which is when the NEI base class has
> > __getnewargs__).
> >
> > I'm really hoping you'll say that can be a documentation issue.  ;)
>
> Essentially the same response -- with enough hackery you can probably
> get this to do what you want, but I wouldn't hold up a release for it.
>
> For example you could file low-priority bugs for both issues in the
> hope that someone else figures it out.
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pickling failure on Enums

[Ethan Furman]

On 05/14/2013 03:16 PM, Nick Coghlan wrote:


On 15 May 2013 07:38, "Guido van Rossum" mailto:gu...@python.org>> wrote:


On Tue, May 14, 2013 at 2:13 PM, Ethan Furman mailto:et...@stoneleaf.us>> wrote:
> On 05/14/2013 01:58 PM, Guido van Rossum wrote:
>>
>> On Tue, May 14, 2013 at 1:09 PM, Ethan Furman mailto:et...@stoneleaf.us>> wrote:
>>> I can get pickle failure on members created using the functional syntax
>>> with no module set;
>>
>>
>> That's the case I care most about.
>
>
> Good, 'cause that one is handled.  :)

Then we're good.

>>> I cannot get pickle failure on those same classes;
>>
>>
>> I suppose you mean "if you create the same enums using class syntax"?
>> Sounds fine to me.
>
>
> No.  Example class:
>
> --> Example = Enum('Example', 'example ie eg')  # no module name given,
> frame hack fails
>
> --> pickle(Example.ie)
> # blows up
>
> --# pickle(Example)
> # succeeds here, but unpickle will fail

Not great, but (a) few people pickle classes, and (b) there's probably
something you can do to the metaclass to sabotage this. But it's fine
to punt on this now.


It may be a bug in pickle - it sounds like it is sanity checking type(obj), but 
not checking for cases where obj itself
is a class.


Well, it's definitely not calling the metaclass' __reduce__ as that's where I put the bomb (hmm, will I be visited by 
men in dark suits now?) so maybe that's a bug in pickle.


At any rate, I figured it out -- give the class' __module__ a dummy name (I like 'uh uh' ;) and when pickle can't find 
that module it'll blow itself up.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pickling failure on Enums

[Ethan Furman]

On 05/14/2013 02:35 PM, Guido van Rossum wrote:


For example you could file low-priority bugs for both issues in the
hope that someone else figures it out.


Got it figured out.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com