Re: [Python-Dev] [Python-checkins] cpython: Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl

2011-05-24 Thread Petri Lehtinen
Terry Reedy wrote: > On 5/24/2011 12:06 PM, Victor Stinner wrote: > >Le mardi 24 mai 2011 à 11:27 -0400, Terry Reedy a écrit : > >>> > >>>+.. function:: RAND_bytes(num) > >>>+ > >>>+ Returns *num* cryptographically strong pseudo-random bytes. > >>>+ > >>>+ .. versionadded:: 3.3 > >>>+ > >>>+..

Re: [Python-Dev] [Python-checkins] Daily reference leaks (234021dcad93): sum=61

2011-05-24 Thread Nick Coghlan
On Wed, May 25, 2011 at 1:09 PM, wrote: > results for 234021dcad93 on branch "default" > > > test_packaging leaked [128, 128, 128] references, sum=384 Is there a new cache in packaging that regrtest needs to know about and either ignore or clear when

Re: [Python-Dev] [Python-checkins] cpython: Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl

2011-05-24 Thread Nick Coghlan
On Wed, May 25, 2011 at 3:52 AM, Terry Reedy wrote: > On 5/24/2011 12:06 PM, Victor Stinner wrote: >> An important feature of a CPRNG (cryptographic pseudo-random number >> generator) is that even if you know all of its output, you cannot >> rebuild its internal state to guess next (or maybe previ

Re: [Python-Dev] [pyodbc] Setting values to SQL_* constants while creating a connection object

2011-05-24 Thread Terry Reedy
On 5/24/2011 5:09 PM, srinivasan munisamy wrote: Hi, I would like to know how to set values to values to SQL_* constants Please direct Python use questions to python-listor other user discussion forums. Py-dev is for discussion of development of the next versions of Python. -- Terry Jan Ree

[Python-Dev] [pyodbc] Setting values to SQL_* constants while creating a connection object

2011-05-24 Thread srinivasan munisamy
Hi, I would like to know how to set values to values to SQL_* constants while creatinga db connection through pyodbc module. For example, i am getting a connection object like below: In [27]: dbh1 = pyodbc.connect("DSN=;UID=;PWD=;DATABASE=;APP=") In [28]: dbh1.getinfo(pyodbc.SQL_DESCRIBE_PARAMET

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Martin (gzlist)
On 24/05/2011, Victor Stinner wrote: > > In Python 2, codecs.open() is the best way to read and/or write files > using Unicode. But in Python 3, open() is preferred with its fast io > module. I would like to deprecate codecs.open() because it can be > replaced by open() and io.TextIOWrapper. I wou

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Sturla Molden
Den 24.05.2011 17:39, skrev Artur Siekielski: Disk access is about 1000x slower than memory access in C, and Python in a worst case is 50x slower than C, so there is still a huge win (not to mention that in a common case Python is only a few times slower). You can put databases in shared memor

Re: [Python-Dev] [Python-checkins] cpython: Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl

2011-05-24 Thread Terry Reedy
On 5/24/2011 12:06 PM, Victor Stinner wrote: Le mardi 24 mai 2011 à 11:27 -0400, Terry Reedy a écrit : +.. function:: RAND_bytes(num) + + Returns *num* cryptographically strong pseudo-random bytes. + + .. versionadded:: 3.3 + +.. function:: RAND_pseudo_bytes(num) + + Returns (bytes, is_cr

Re: [Python-Dev] Stable buildbots update

2011-05-24 Thread Bill Janssen
Ned Deily wrote: > In article <87zkmcalt8@uwakimon.sk.tsukuba.ac.jp>, > "Stephen J. Turnbull" wrote: > > Are you saying you expect Mac OS X 10.4 "Tiger" to go green once the > > bots update? If so, I'm impressed, and "thank you!" to all involved. > > Apple and MacPorts have long since wash

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Cesare Di Mauro
2011/5/24 Stefan Behnel > Maciej Fijalkowski, 24.05.2011 13:31: > > CPython was not designed for CPU cache usage as far as I'm aware. >> > > That's a pretty bold statement to make on this list. Even if it wasn't > originally "designed" for (efficient?) CPU cache usage, it's certainly been > aro

Re: [Python-Dev] cpython: move specialized dir implementations into __dir__ methods (closes #12166)

2011-05-24 Thread Benjamin Peterson
2011/5/24 Georg Brandl : > On 24.05.2011 18:08, benjamin.peterson wrote: >> http://hg.python.org/cpython/rev/8f403199f999 >> changeset:   70331:8f403199f999 >> user:        Benjamin Peterson >> date:        Tue May 24 11:09:06 2011 -0500 >> summary: >>   move specialized dir implementations into _

Re: [Python-Dev] cpython: move specialized dir implementations into __dir__ methods (closes #12166)

2011-05-24 Thread Georg Brandl
On 24.05.2011 18:08, benjamin.peterson wrote: > http://hg.python.org/cpython/rev/8f403199f999 > changeset: 70331:8f403199f999 > user:Benjamin Peterson > date:Tue May 24 11:09:06 2011 -0500 > summary: > move specialized dir implementations into __dir__ methods (closes #12166) >

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread geremy condra
On Tue, May 24, 2011 at 8:44 AM, Terry Reedy wrote: > On 5/24/2011 8:25 AM, Sturla Molden wrote: > >> Artur Siekielski is not talking about cache locality, but copy-on-write >> fork on Linux et al. >> >> When reference counts are updated after forking, memory pages marked >> copy-on-write are copi

Re: [Python-Dev] Stable buildbots update

2011-05-24 Thread Terry Reedy
On 5/24/2011 6:27 AM, Nick Coghlan wrote: On Tue, May 24, 2011 at 7:56 PM, Antoine Pitrou wrote: Thank you very much! What a beautiful sight this is: http://www.python.org/dev/buildbot/all/waterfall?category=3.x.stable (until a sporadic failure comes up, that is) I could turn test_crashers b

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Terry Reedy
On 5/24/2011 6:14 AM, M.-A. Lemburg wrote: I have no idea why TextIOWrapper was added to the stdlib instead of making StreamReaderWriter more capable, since StreamReaderWriter had already been available in Python since Python 1.6 (and this is being used by codecs.open()). As I understand it, y

Re: [Python-Dev] [Python-checkins] cpython: Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl

2011-05-24 Thread Victor Stinner
Le mardi 24 mai 2011 à 11:27 -0400, Terry Reedy a écrit : > > > > +.. function:: RAND_bytes(num) > > + > > + Returns *num* cryptographically strong pseudo-random bytes. > > + > > + .. versionadded:: 3.3 > > + > > +.. function:: RAND_pseudo_bytes(num) > > + > > + Returns (bytes, is_cryptograph

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Terry Reedy
On 5/24/2011 8:25 AM, Sturla Molden wrote: Artur Siekielski is not talking about cache locality, but copy-on-write fork on Linux et al. When reference counts are updated after forking, memory pages marked copy-on-write are copied if they store reference counts. And then he quickly runs out of m

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Artur Siekielski
2011/5/24 Sturla Molden : > Den 24.05.2011 11:55, skrev Artur Siekielski: >> >> PYRO/multiprocessing proxies isn't a comparable solution because of >> ORDERS OF MAGNITUDE worser performance. You compare here direct memory >> access vs serialization/message passing through sockets/pipes. > The bottl

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Nick Coghlan
On Tue, May 24, 2011 at 10:05 PM, Stefan Behnel wrote: > Maciej Fijalkowski, 24.05.2011 13:31: >> >> CPython was not designed for CPU cache usage as far as I'm aware. > > That's a pretty bold statement to make on this list. Even if it wasn't > originally "designed" for (efficient?) CPU cache usage

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Stefan Behnel
Antoine Pitrou, 24.05.2011 14:32: On Tue, 24 May 2011 14:05:26 +0200Stefan Behnel wrote: I doubt that efficient CPU cache usage was a major design goal of PyPy right from the start. IMHO, the project has changed its objectives way too many times to claim something like that, especially at the l

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Antoine Pitrou
On Tue, 24 May 2011 14:05:26 +0200 Stefan Behnel wrote: > > I doubt that efficient CPU cache usage was a major design goal of PyPy > right from the start. IMHO, the project has changed its objectives way too > many times to claim something like that, especially at the low level where > the CPU

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Sturla Molden
Den 24.05.2011 11:55, skrev Artur Siekielski: POSH might be good, but the project is dead for 8 years. And this copy-on-write is nice because you don't need changes/restrictions to your code, or a special garbage collector. Then I have a solution for you, one that is cheaper than anything else

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Sturla Molden
Den 24.05.2011 13:31, skrev Maciej Fijalkowski: Not sure what scenario exactly are you discussing here, but storing reference counts outside of objects has (at least on a single processor) worse cache locality than inside objects. Artur Siekielski is not talking about cache locality, but copy

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Sturla Molden
Den 24.05.2011 11:55, skrev Artur Siekielski: PYRO/multiprocessing proxies isn't a comparable solution because of ORDERS OF MAGNITUDE worser performance. You compare here direct memory access vs serialization/message passing through sockets/pipes. The bottleneck is likely the serialization, bu

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Stefan Behnel
Maciej Fijalkowski, 24.05.2011 13:31: CPython was not designed for CPU cache usage as far as I'm aware. That's a pretty bold statement to make on this list. Even if it wasn't originally "designed" for (efficient?) CPU cache usage, it's certainly been around for long enough to have received nu

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Walter Dörwald
On 24.05.11 12:58, Victor Stinner wrote: > Le mardi 24 mai 2011 à 12:42 +0200, Łukasz Langa a écrit : >> Wiadomość napisana przez Walter Dörwald w dniu 2011-05-24, o godz. 12:16: >> I don't see which usecase is not covered by TextIOWrapper. But I know some cases which are not supported by

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Maciej Fijalkowski
On Sun, May 22, 2011 at 1:57 AM, Artur Siekielski wrote: > Hi. > The problem with reference counters is that they are very often > incremented/decremented, even for read-only algorithms (like traversal > of a list). It has two drawbacks: > 1. CPU cache lines (64 bytes on X86) containing a beginnin

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Victor Stinner
Le mardi 24 mai 2011 à 12:42 +0200, Łukasz Langa a écrit : > Wiadomość napisana przez Walter Dörwald w dniu 2011-05-24, o godz. 12:16: > > >> I don't see which usecase is not covered by TextIOWrapper. But I know > >> some cases which are not supported by StreamReader/StreamWriter. > > > > This co

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Antoine Pitrou
On Tue, 24 May 2011 12:16:49 +0200 Walter Dörwald wrote: > > > and so it's not possible to write a generic fix for > > all child classes in the codecs module. Each stateful codec has to > > handle special cases like seek() problems. > > Yes, which in theory makes it possible to implement shortcu

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Victor Stinner
Le mardi 24 mai 2011 à 08:16 +, Vinay Sajip a écrit : > > I opened an issue for this idea. Brett and Marc-Andree Lemburg don't > > want to deprecate codecs.open() & friends because they want to be able > > to write code working on Python 2 and on Python 3 without any change. I > > don't think i

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Łukasz Langa
Wiadomość napisana przez Walter Dörwald w dniu 2011-05-24, o godz. 12:16: >> I don't see which usecase is not covered by TextIOWrapper. But I know >> some cases which are not supported by StreamReader/StreamWriter. > > This could be be partially fixed by implementing generic > StreamReader/Strea

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Antoine Pitrou
On Tue, 24 May 2011 20:25:11 +1000 Nick Coghlan wrote: > > Just as PEP 302 defines how module importers should be written, PEP > 100 defines how text codecs should be written (i.e. in terms of > StreamReader and StreamWriter). > > PEP 3116 then defines how such codecs can be used as part of the

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Walter Dörwald
On 24.05.11 02:08, Victor Stinner wrote: > [...] > codecs.open() and StreamReader, StreamWriter and StreamReaderWriter > classes of the codecs module don't support universal newlines, still > have some issues with stateful codecs (like UTF-16/32 BOMs), and each > codec has to implement a StreamRea

Re: [Python-Dev] Stable buildbots update

2011-05-24 Thread Nick Coghlan
On Tue, May 24, 2011 at 7:56 PM, Antoine Pitrou wrote: > Thank you very much! What a beautiful sight this is: > http://www.python.org/dev/buildbot/all/waterfall?category=3.x.stable > > (until a sporadic failure comes up, that is) I could turn test_crashers back on if you like ;) Great work to al

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Nick Coghlan
On Tue, May 24, 2011 at 6:58 PM, Victor Stinner wrote: > StreamReader, StreamWriter, TextIOWrapper and StreamReaderWriter all > have a file-like API: tell(), seek(), read(),  readline(), write(), etc. > The implementation is maybe different, but the API is just the same, and > so the usecases are

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread M.-A. Lemburg
Victor Stinner wrote: > Le mardi 24 mai 2011 à 10:03 +0200, M.-A. Lemburg a écrit : >> Please read PEP 100 regarding StreamReader and StreamWriter. >> Those codecs parts were explicitly designed to be stateful, >> unlike the stateless encoder/decoder methods. > > Yes, it is possible to implement s

[Python-Dev] "streams" vs "buffers"

2011-05-24 Thread Antoine Pitrou
On Tue, 24 May 2011 10:03:22 +0200 "M.-A. Lemburg" wrote: > > StreamReader and StreamWriters are implemented by the codecs, > they are part of the API that each codec has to provide in order > to register in the Python codecs system. Their purpose is > to provide a stateful interface and work eff

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Artur Siekielski
2011/5/24 Sturla Molden : >> Oh, and using explicit shared memory or mmap is much harder, because >> you have to map the whole object graph into bytes. > > It sounds like you need PYRO, POSH or multiprocessing's proxy objects. PYRO/multiprocessing proxies isn't a comparable solution because of ORD

Re: [Python-Dev] Stable buildbots update

2011-05-24 Thread Antoine Pitrou
On Mon, 23 May 2011 19:16:36 +0200 Tarek Ziadé wrote: > > I have now completed the cleanup and we're back on green-land for the > stable bots. > > The red slaves should get green when they catch up with the latest rev > (they are slow). If they're not and they are failing in packaging or > sysco

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Victor Stinner
Le mardi 24 mai 2011 à 10:03 +0200, M.-A. Lemburg a écrit : > Please read PEP 100 regarding StreamReader and StreamWriter. > Those codecs parts were explicitly designed to be stateful, > unlike the stateless encoder/decoder methods. Yes, it is possible to implement stateful StreamReader and Stream

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Victor Stinner
Le mardi 24 mai 2011 à 08:16 +, Vinay Sajip a écrit : > So I would also want to keep codecs.open() and friends, at least for now Well, I would agree to keep codecs.open() (if we patch it to reuse TextIOWrapper and add a note to say that it is kept for backward compatibiltiy and open() should b

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Vinay Sajip
Victor Stinner haypocalc.com> writes: > I opened an issue for this idea. Brett and Marc-Andree Lemburg don't > want to deprecate codecs.open() & friends because they want to be able > to write code working on Python 2 and on Python 3 without any change. I > don't think it's realistic: nontrivial

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread M.-A. Lemburg
Victor Stinner wrote: > Hi, > > In Python 2, codecs.open() is the best way to read and/or write files > using Unicode. But in Python 3, open() is preferred with its fast io > module. I would like to deprecate codecs.open() because it can be > replaced by open() and io.TextIOWrapper. I would like y

Re: [Python-Dev] Deprecate codecs.open() and StreamWriter/StreamReader

2011-05-24 Thread Victor Stinner
Le mardi 24 mai 2011 à 15:24 +1000, Nick Coghlan a écrit : > On Tue, May 24, 2011 at 10:08 AM, Victor Stinner > wrote: > > It's trivial to replace a call to codecs.open() by a call to open(), > > because the two API are very close. The main different is that > > codecs.open() doesn't support unive