Re: [Python-Dev] Regression fix releases coming

[Nadeem Vawda]

On Tue, Apr 30, 2013 at 4:07 PM, Benjamin Peterson wrote:

> 2013/4/30 Guido van Rossum :
> > Can we do something about the problem that virus checkers complain about
> the
> > xml bomb test file? E.g. Generate it dynamically in a script?
>
> That's been dealt with. See http://bugs.python.org/issue17843


I don't think that's the same issue. The link you gave is about *BZ2* test
data triggering antivirus warnings, not XML. Is there another issue for a
similar problem with an XML test file? (Searching bugs.python.org for "xml
antivirus" doesn't turn up any results...)

- Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow

[Nadeem Vawda]

On Thu, Jun 27, 2013 at 12:07 AM, Victor Stinner
wrote:

> I would to parse an integer in [0; UINT_MAX] to fix the zlib module on
> 64-bit system:
> http://bugs.python.org/issue18294
>
> How should I implement that? Use "O" format and then use
> PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX?
>

I ran into the same problem in the _lzma module. My solution was to define
a custom converter that does an explicit check before returning the value
(see http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134).

On Thu, Jun 27, 2013 at 12:26 AM, Guido van Rossum  wrote:

> > I would to parse an integer in [0; UINT_MAX] to fix the zlib module on
> > 64-bit system:
> > http://bugs.python.org/issue18294
> >
> > How should I implement that? Use "O" format and then use
> > PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX?
>
> Why can't you use the K format? It won't reject out-of-range values,
> but it will convert them to in-range so there aren't any attacks
> possible based on bypassing the range check. I'm probably
> misunderstanding something -- I don't completely understand that bug
> report. :-(


The point is not to protect against deliberate attacks, but rather to fail
loudly (instead of silently) when the caller provides an input that the
underlying C library cannot handle.

- Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Re-enable warnings in regrtest and/or unittest

[Nadeem Vawda]

2010/11/23 Łukasz Langa :
> If you agree to do that for regrtest I will clean up the tests for warnings.
> Already did that for zipfile so it doesn't raise ResourceWarnings anymore. I
> just need to correct multiprocessing and xmlrpc ResourceWarnings, silence
> some DeprecationWarnings in the tests and we're all set. Ah, I see a couple
> more with -uall but nothing scary.

There are also some in test_socket - I've submitted a patch on
Roundup: http://bugs.python.org/issue10512

Looking at the multiprocessing warnings, they seem to be caused by
leaks in the underlying package, unlike xmlrpc and socket, where it's
just a matter of the test code neglecting to close the connection.  So
+1 to:

> Anyway, I find warnings as errors in regrtest a welcome feature. Let's make
> it happen :)

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] constant/enum type in stdlib

[Nadeem Vawda]

On Thu, Nov 25, 2010 at 11:34 AM, Glenn Linderman  wrote:
> So the following code defines constants with associated names that get put
> in the repr.

The code you gave doesn't work if the constant() function is moved
into a separate module from the code that calls it.  The globals()
function, as I understand it, gives you access to the global namespace
*of the current module*, so the constants end up being defined in the
module containing constant(), not the module you're calling it from.

You could get around this by passing the globals of the calling module
to constant(), but I think it's cleaner to use a class to provide a
distinct namespace for the constants.

> An idea I had, but have no idea how to implement, is that it might be nice
> to say:
>
>     with imported_constants_from_module:
>        do_stuff
>
> where do_stuff could reference the constants without qualifying them by
> module.  Of course, if you knew it was just a module of constants, you could
> "import * from module" :)  But the idea of with is that they'd go away at
> the end of that scope.

I don't think this is possible - the context manager protocol doesn't
allow you to modify the namespace of the caller like that.  Also, a
with statement does not have its own namespace; any names defined
inside its body will continue to be visible in the containing scope.

Of course, if you want to achieve something similar (at function
scope), you could say:

def foo(bar, baz):
from module import *
...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r88122 - python/branches/py3k/Doc/whatsnew/3.2.rst

[Nadeem Vawda]

On Thu, Jan 20, 2011 at 3:11 PM, Antoine Pitrou  wrote:
> On Thu, 20 Jan 2011 10:47:04 +0100 (CET)
> raymond.hettinger  wrote:
>>
>> +Code Repository
>> +===
>> +
>> +In addition to the existing Subversion code repository at 
>> http://svn.python.org
>> +there is now a `Mercurial `_ repository at
>> +http://hg.python.org/ .
>
> Shouldn't we wait until that repository is ready? Right now people
> willing to work with it will get the surprise that it hasn't been
> updated for 3 months.

Should that perhaps be http://code.python.org/hg/ ? I've been using
that repository for the last two months or so, and it seems to be up
to date (to within an hour or so, at least).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

On Sat, Mar 12, 2011 at 9:32 AM, Eli Bendersky  wrote:
> The devguide's recommendation is to "forward-port" changes withing a major
> release line, i.e. if I need something in all 3.[123], then start with 3.1
> and forward-port (by "hg merge ") to 3.2 and then 3.3
>
> Just to clarify - does this mean that all changesets that are applied to 3.2
> eventually get to 3.3 as well? Since we just do "hg merge 3.2" in the 3.3
> clone, I guess this must be true. But then what happens if there's a change
> in 3.2 that shouldn't get into 3.3? (say a bug fix for a feature that has
> disappeared)?

The "Note:" box two paragraphs down from

explains how prevent a changeset from propagating: you do a dummy merge
where you revert the changes before committing. This marks the changeset from
3.2 as having been merged into default/3.3 without actually applying
the changes.

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

Hmm... it seems that the given instructions don't actually work. "hg
revert -a" fails,
saying that a specific revision needs to be provided. The command should be
"hg revert -ar default".
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

On Sat, Mar 12, 2011 at 1:29 PM, "Martin v. Löwis"  wrote:
> Isn't that command correct only if you are merging into default?
> ISTM that it should be "hg revert -ar ".

In general, yes. However, the existing text refers specifically to the
case of merging 3.2
into default, so I was trying to be consistent with that.

The current text, after my proposed change, will be:

hg update default
hg merge 3.2
hg revert -ar default
hg commit

For full generality, you might want to say:

hg update 
hg merge 
hg revert -ar 
hg commit

However, the entire section on forward-porting is framed in terms of
concrete examples
(and that specific subsection refers explicitly to the "3.2 ->
default" case). The existing
text makes sense to me, so I don't see any need to change this.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] forward-porting from 3.1 to 3.2 to 3.3

[Nadeem Vawda]

On Sun, Mar 13, 2011 at 12:43 AM, Éric Araujo  wrote:
>> hg revert -ar default
>
> You can’t combine the -r option with other options.  (Yes, it’s a known bug.)

It seems to work for me (Mercurial 1.6.3 on Ubuntu). But I suppose it
wouldn't hurt
to split the options up.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Copyright notices

[Nadeem Vawda]

I was wondering what the policy is regarding copyright notices and license
boilerplate text at the top of source files.

I am currently rewriting the bz2 module (see http://bugs.python.org/issue5863),
splitting the existing Modules/bz2module.c into Modules/_bz2module.c and
Lib/bz2.py.

Are new files expected to include a copyright notice and/or license boilerplate
text? Also, is it necessary for _bz2module.c (new) to retain the copyright
notices from bz2module.c (old)? In the tracker issue, Antoine said he didn't
think so, but suggested that I get some additional opinions.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Copyright notices

[Nadeem Vawda]

On Mon, Mar 21, 2011 at 2:20 PM, M.-A. Lemburg  wrote:
> Nadeem Vawda wrote:
>> [snip]
>
> Since you'll be adding new IP to Python, the new code you write should
> contain your copyright and the standard PSF contributor agreement
> notice, e.g.
>
> """
> (c) Copyright 2011 by Nadeem Vawda. Licensed to PSF under a Contributor 
> Agreement.
> """
>
> (please also make sure you have sent the signed agreement to the PSF;
> see http://www.python.org/psf/contrib/)
>
> We don't have a general copyright or license boiler plate for Python
> source files.
>
>> [snip]
>
> If the file copies significant code parts from older files, the
> copyright notices from those files will have to added to the
> file comment as well - ideally with a note explaining to which parts
> those copyrights apply and where they originated.
>
> If you are replacing the old implementation with a new one,
> you don't need to copy over the old copyright statements.

Thanks for the information. I still need to submit a Contributor Agreement, so
I'll do that as soon as possible.

(As an aside, it might be useful to include this info more explicitly in the
devguide, to make it easier for newbies to find. I'll put together a patch
when I get a chance.)

On Mon, Mar 21, 2011 at 4:48 PM, Antoine Pitrou  wrote:
> On Mon, 21 Mar 2011 13:20:59 +0100
> "M.-A. Lemburg"  wrote:
>> Since you'll be adding new IP to Python, the new code you write should
>> contain your copyright and the standard PSF contributor agreement
>> notice, e.g.
>
> I agree with Raymond's argument that we shouldn't add *new* copyright
> boilerplate:
> http://mail.python.org/pipermail/python-dev/2009-January/085267.html

I agree that it would be preferable not to clutter the code with copyright
notices unnecessarily, especially since you can get the same information from
the version control history. However, as things stand, the CA requires their
inclusion, and changing it would (I imagine) involve a lot of work.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #5863: Rewrite BZ2File in pure Python, and allow it to accept

[Nadeem Vawda]

On Sun, Apr 3, 2011 at 8:02 PM, Antoine Pitrou  wrote:
> On Sun, 03 Apr 2011 18:55:33 +0200
> Éric Araujo  wrote:
>> I think we use Misc/ACKS for code+docs contribution like this one,
>> Doc/ACKS.txt being used for doc-only changes.  This second file is not
>> comprehensive nor always used though, so maybe it should be superseded
>> by the former.
>
> Nadeem is already in Misc/ACKS.  I don't know what the policy is for
> Doc/ACKS.txt, but since he added himself in the patch, I saw no good
> reason for reverting the change.

I added myself because I assumed the policy for Doc/ACKS.txt to be the same
as the policy for Misc/ACKS - if you submit a patch, add your name. Looking
at the devguide, though, I can't find any mention of Doc/ACKS.txt.

> +1 for merging these files by the way.

Sounds good to me. The intro at the top of Misc/ACKS is pretty broad,
thanking people for all contributions (not just code). Unless there's some
plan to split the documentation off into a separate repository, I can't
think of any reason not to merge them.

Regards,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Fix 64-bit safety issue in BZ2Compressor and BZ2Decompressor.

[Nadeem Vawda]

Thanks for the feedback. I'll be sure to include more information in my
future commit messages.

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes issue10761: tarfile.extractall failure when symlinked files are

[Nadeem Vawda]

On Thu, Apr 28, 2011 at 4:44 PM, Senthil Kumaran  wrote:
> On Thu, Apr 28, 2011 at 04:20:06PM +0200, Éric Araujo wrote:
>> >  if hasattr(os, "symlink") and hasattr(os, "link"):
>> >  # For systems that support symbolic and hard links.
>> >  if tarinfo.issym():
>> > +if os.path.exists(targetpath):
>> > +os.unlink(targetpath)
>>
>> Is there a race condition here?
>
> The lock to avoid race conditions (if you were thinking along those
> lines) would usually be implemented at the higher level code which is
> using extractall in threads.
>
> Checking that no one else is accessing the file before unlinking may
> not be suitable for the library method and of course, we cannot check
> if someone is waiting to act on that file.

I think Éric is referring to the possibility of another process creating or
deleting targetpath between the calls to os.path.exists() and os.unlink().
This would result in symlink() or unlink() raising an exception.

The deletion case could be handled like this:

 if tarinfo.issym():
+try:
+os.unlink(targetpath)
+except OSError as e:
+if e.errno != errno.ENOENT:
+raise
 os.symlink(tarinfo.linkname, targetpath)

I'm not sure what the best way of handling the creation case is. The obvious
solution would be to try the above code in a loop, repeating until we succeed
(or fail for a different reason), but this would not be guaranteed to
terminate.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes issue10761: tarfile.extractall failure when symlinked files are

[Nadeem Vawda]

On Fri, Apr 29, 2011 at 10:02 AM, Eli Bendersky  wrote:
> I completely understand this "other code/thread deletes the path
> between exists() and unlink()" case - it indeed is a race condition
> waiting to happen. What I didn't understand was Antoine's example of
> "attacker creates targetpath between os.path.exists and os.unlink",
> and was asking for a more detailed example, since I'm not really
> experienced with security-oriented thinking.

If targetpath is created after the os.path.exists() check, then os.unlink()
will not be called, so os.symlink() will raise an exception when it sees that
targetpath already exists.

On Thu, Apr 28, 2011 at 5:44 PM, Antoine Pitrou  wrote:
> On Thu, 28 Apr 2011 17:40:05 +0200
> Nadeem Vawda  wrote:
>>
>> The deletion case could be handled like this:
>>
>>  if tarinfo.issym():
>> +try:
>> +os.unlink(targetpath)
>> +except OSError as e:
>> +if e.errno != errno.ENOENT:
>> +raise
>>  os.symlink(tarinfo.linkname, targetpath)
>
> Someone can still create "targetpath" between the calls to unlink() and
> symlink() though.

Like I said, that code only handles the case of targetpath being deleted.
I can't think of a similarly easy fix for the creation case. You could solve
that particular form of the problem with something like:

if tarinfo.issym():
while True:
try:
os.symlink(tarinfo.linkname, targetpath)
except OSError as e:
if e.errno != errno.EEXIST:
raise
else:
break
try:
os.unlink(targetpath)
except OSError as e:
if e.errno != errno.ENOENT:
raise

... but that would open up another DOS vulnerability - if an attacker manages
to keep re-creating targetpath in the window between unlink() and symlink(),
the loop would never terminate. Also, the code is rather ugly ;-)

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #10276: test_zlib checks that inputs of 2 GB are handled correctly by

[Nadeem Vawda]

On Tue, May 3, 2011 at 3:19 PM, victor.stinner
 wrote:
> +# Issue #10276 - check that inputs of 2 GB are handled correctly.
> +# Be aware of issues #1202, #8650, #8651 and #10276
> +class ChecksumBigBufferTestCase(unittest.TestCase):
> +    int_max = 0x7FFF
> +
> +    @unittest.skipUnless(mmap, "mmap() is not available.")
> +    def test_big_buffer(self):
> +        if sys.platform[:3] == 'win' or sys.platform == 'darwin':
> +            requires('largefile',
> +                     'test requires %s bytes and a long time to run' %
> +                     str(self.int_max))
> +        try:
> +            with open(TESTFN, "wb+") as f:
> +                f.seek(self.int_max-4)
> +                f.write("asdf")
> +                f.flush()
> +                try:
> +                    m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
> +                    self.assertEqual(zlib.crc32(m), 0x709418e7)
> +                    self.assertEqual(zlib.adler32(m), -2072837729)
> +                finally:
> +                    m.close()
> +        except (IOError, OverflowError):
> +            raise unittest.SkipTest("filesystem doesn't have largefile 
> support")
> +        finally:
> +            unlink(TESTFN)
> +
> +

0x7FFF is (2G-1) bytes. For a 2GB buffer, int_max should be
0x8000. However, if you make this change, crc32() and adler32()
raise OverflowErrors (see changeset a0681e7a6ded). This makes the test
to erroneously report that the filesystem doesn't support large files.
The assertEqual() tests should probably be changed to
assertRaises(..., OverflowError).

Also, the assignment to m needs to be moved outside of the inner
try...finally block. If mmap() fails, the call to m.close() raises a
new exception because m has not yet been bound. This seems to be
causing failures on some of the 32-bit buildbots.

As an aside, in this sort of situation is it better to just go and
commit a fix myself, or is raising it on the mailing list first the
right way to do things?

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #10276: test_zlib checks that inputs of 2 GB are handled correctly by

[Nadeem Vawda]

On Tue, May 3, 2011 at 10:38 PM, Victor Stinner
 wrote:
> I don't want to check OverflowError: the test is supposed to compute the
> checksum of a buffer of 0x7FFF bytes, to check crc32() and
> adler32(). 0x7FFF is the biggest size supported by these functions
> (zlib doesn't use Py_ssize_t in Python 2.7).

I see. Since you mentioned issue 10276 in the commit message, I
assumed you were testing
for the underlying C functions truncating their arguments. It seems
that I was mistaken.
Sorry for the confusion.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #10276: test_zlib checks that inputs of 2 GB are handled correctly by

[Nadeem Vawda]

On Thu, May 5, 2011 at 11:33 AM, Victor Stinner
 wrote:
> Le mercredi 04 mai 2011 à 15:40 -0700, Ethan Furman a écrit :
>> The comment says 'check that inputs of 2 GB are handled correctly' but
>> the file created is 1 byte short of 2Gb.  Is the test wrong, or just
>> wrongly commented?  Or am I not understanding?
>
> If you write a byte after 2 GB of zeros, the file size is 2 GB+the few
> bytes. This trick is to create quickly a large file: some OSes support
> sparse files, zeros are not written on disk. But on Mac OS X and
> Windows, you really write 2 GB+some bytes.

Ethan's point is that 0x7FFF is not 2GB - it is (2G-1) bytes. So the
test and the preceding comment are inconsistent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #11277: Remove useless test from test_zlib.

[Nadeem Vawda]

On Mon, May 9, 2011 at 2:53 PM, Jim Jewett  wrote:
> Can you clarify (preferably in the commit message as well) exactly
> *why* these largefile tests are useless?  For example, is there
> another test that covers this already?

Ah, sorry about that. It was discussed on the tracker issue, but I guess I
can't expect people to read through 90+ messages to figure it out :P

The short version is that it was supposed to test 4GB+ inputs, but in 2.7,
the functions being tested don't accept inputs that large.

The details:

The test was originally intended to catch the case where crc32() or adler32()
would get a buffer of >=4GB, and then silently truncate the buffer size and
produce an incorrect result (issue10276). It had been written for 3.x, and then
backported to 2.7. However, in 2.7, zlibmodule.c doesn't define
PY_SSIZE_T_CLEAN, so passing in a buffer of >=2GB raises an OverflowError
(see issue8651). This means that it is impossible to trigger the bug in question
on 2.7, making the test pointless.

Of course, the code that was deleted tests with an input sized 2GB-1 or 1GB,
rather than 4GB (the size used in 3.x). When the test was backported, the size
of the input was reduced, to avoid triggering an OverflowException. At the time,
no-one realized that this also would not trigger the bug being tested
for; it only
came to light when the test started crashing for unrelated reasons (issue11277).

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Don't set local variable in a list comprehension or generator

[Nadeem Vawda]

On Wed, May 18, 2011 at 2:21 PM, Victor Stinner
 wrote:
> ''.join(c for c in 'abc') and ''.join([c for c in 'abc']) do create a
> temporary c variable.

I'm not sure why you would encounter code like that in the first place.
Surely any code of the form:

''.join(c for c in my_string)

would just return my_string? Or am I missing something?

> I heard about optimization in the AST tree instead of working on the
> bytecode. What is the status of this project?

Are you referring to issue11549? There was some related discussion [1] on
python-dev about six weeks ago, but I haven't seen anything on the topic
since then.

Cheers,
Nadeem

[1] http://mail.python.org/pipermail/python-dev/2011-April/110399.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Add LRUCache class to stdlib

[Nadeem Vawda]

I would like to propose adding a class to the stdlib to provide a more flexible
LRU caching mechanism. As things stand, the functools.lru_cache() decorator is
fine for memoization of pure functions, but does not accommodate situations
where cache entries become stale and must be invalidated/updated (e.g.
filecmp.cmp(); cf. issue #11802).

I was thinking it would be a good idea to factor out the the replacement code
into a separate class that could then be reused by code for which the
lru_cache() decorator is not applicable. The implementation of lru_cache()
itself would also become considerably simpler.

Implementation:

class LRUCache:
"""Least-recently-used cache class.

See:  http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used

"""

def __init__(self, maxsize):
"""Initialize an LRUCache.

If *maxsize* is None, the LRU replacement mechanism is disabled,
and the cache can grow without bound.

"""
if maxsize is None:
self.cache = dict()
else:
self.cache = OrderedDict()
self.lock = Lock()
self.hits = self.misses = 0
self.maxsize = maxsize

def __getitem__(self, key):
with self.lock:
try:
value = self.cache[key]
except KeyError:
self.misses += 1
raise
else:
self.hits += 1
if self.maxsize is not None:
self.cache.move_to_end(key)
return value

def __setitem__(self, key, value):
with self.lock:
self.cache[key] = value
if self.maxsize is not None and len(self.cache) > self.maxsize:
self.cache.popitem(0)

def info(self):
"""Report cache statistics"""
with self.lock:
return _CacheInfo(self.hits, self.misses,
  self.maxsize, len(self.cache))

I'm not sure where this class should go; it would be convenient to just stick
it in functools along with lru_cache(), but perhaps the collections package
would be more appropriate?

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.2): Fix sorting or wording of some NEWS entries.

[Nadeem Vawda]

On Fri, Jul 29, 2011 at 2:46 PM, Antoine Pitrou  wrote:
> There's no practical difference (from the user's point of view) between
> extension modules and the library, so I think the "Extension Modules"
> section should simply die.

+1. This has been bugging me for a while now.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #11651: Improve Makefile test targets.

[Nadeem Vawda]

On Sun, Jul 31, 2011 at 10:26 AM, Nick Coghlan  wrote:
> On Sun, Jul 31, 2011 at 9:09 AM, nadeem.vawda
>  wrote:
>> - Remove duplicate test run in "make test" and "make testuniversal"
>
> That seems very questionable - the rationale for running the test
> suite twice by default to ensure PYC generation is working correctly
> still holds.

The consensus on the tracker was that it isn't worth doubling the time taken to
run "make test" to check for a class of bug that seems to be relatively rare.
For changes that touch anything as far-reaching as .pyc generation, you should
be using "make testall" anyway (and that does still use two passes).

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #11651: Move options for running tests into a Python script.

[Nadeem Vawda]

Thanks for catching that. Fixed in 0b52b6f1bfab.

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FileSystemError or FilesystemError?

[Nadeem Vawda]

On Tue, Aug 23, 2011 at 8:39 PM, Ross Lagerwall  wrote:
>> When reviewing the PEP 3151 implementation (*), Ezio commented that
>> "FileSystemError" looks a bit strange and that "FilesystemError" would
>> be a better spelling. What is your opinion?

I think "FilesystemError" looks nicer, but it's not something I'd lose
sleep over either way.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

Hello all,

I'd like to propose the addition of a new module in Python 3.3. The 'lzma'
module will provide support for compression and decompression using the LZMA
algorithm, and the .xz and .lzma file formats. The matter has already been
discussed on the tracker , where there seems
to be a consensus that this is a desirable feature. What are your thoughts?

The proposed module's API will be very similar to that of the bz2 module;
the only differences will be additional keyword arguments to some functions,
for specifying container formats and detailed compressor options.

The implementation will also be similar to bz2 - basic compressor and
decompressor classes written in C, with convenience functions and a file
interface implemented on top of those in Python.

I've already done some work on the C parts of the module; I'll push that to my
sandbox  in the next day or two.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

On Sat, Aug 27, 2011 at 4:50 PM, "Martin v. Löwis"  wrote:
>> The implementation will also be similar to bz2 - basic compressor and
>> decompressor classes written in C, with convenience functions and a file
>> interface implemented on top of those in Python.
>
> When I reviewed lzma, I found that this approach might not be
> appropriate. lzma has many more options and aspects that allow tuning
> and selection, and a Python LZMA library should provide the same feature
> set as the underlying C library.
>
> So I would propose that a very thin C layer is created around the C
> library that focuses on the actual algorithms, and that any higher
> layers (in particular file formats) are done in Python.

I probably shouldn't have used the word "basic" here - these classes expose all
the features of the underlying library. I was rather trying to underscore that
the rest of the module is implemented in terms of these two classes.

As for file formats, these are handled by liblzma itself; the extension module
just selects which compressor/decompressor initializer function to use depending
on the value of the "format" argument. Our code won't contain anything along the
lines of GzipFile; all of that work is done by the underlying C library. Rather,
the LZMAFile class will be like BZ2File - just a simple filter that passes the
read/written data through a LZMACompressor or LZMADecompressor as appropriate.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

On Sat, Aug 27, 2011 at 5:15 PM, "Martin v. Löwis"  wrote:
>> As for file formats, these are handled by liblzma itself; the extension 
>> module
>> just selects which compressor/decompressor initializer function to use 
>> depending
>> on the value of the "format" argument. Our code won't contain anything along 
>> the
>> lines of GzipFile; all of that work is done by the underlying C library. 
>> Rather,
>> the LZMAFile class will be like BZ2File - just a simple filter that passes 
>> the
>> read/written data through a LZMACompressor or LZMADecompressor as 
>> appropriate.
>
> This is exactly what I worry about. I think adding file I/O to bz2 was a
> mistake, as this doesn't integrate with Python's IO library (it used
> to, but now after dropping stdio, they were incompatible. Indeed, for
> Python 3.2, BZ2File has been removed from the C module, and lifted to
> Python.
>
> IOW, the _lzma C module must not do any I/O, neither directly nor
> indirectly (through liblzma). The approach of gzip.py (doing IO
> and file formats in pure Python) is exactly right.

It is not my intention for the _lzma C module to do I/O - that will be done by
the LZMAFile class, which will be written in Python. My comparison with bz2 was
in reference to the state of the module after it was rewritten for issue 5863.

Saying "anything along the lines of GzipFile" was a bad choice of wording; what
I meant is that the LZMAFile class won't handle the problem of picking apart the
.xz and .lzma container formats. That is handled by liblzma (operating entirely
on in-memory buffers). It will do _only_ I/O, in a similar fashion to
the BZ2File
class (as of changeset 2cb07a46f4b5, to avoid ambiguity ;) ).

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

On Sat, Aug 27, 2011 at 5:42 PM, "Martin v. Löwis"  wrote:
> Not sure whether you already have this: supporting the tarfile module
> would be nice.

Yes, got that - issue 5689. Also of interest is issue 5411 - adding .xz
support to distutils. But I think that these are separate projects that
should wait until the lzma module is finalized.

On Sat, Aug 27, 2011 at 5:40 PM, Antoine Pitrou  wrote:
> On Sun, 28 Aug 2011 01:36:50 +1000
> Nick Coghlan  wrote:
>> PEP 399 also comes into play - we need a pure Python version for PyPy
>> et al (or a plausible story for why an exception should be granted).
>
> The plausible story being that we basically wrap an existing library?
> I don't think PyPy et al have pure Python versions of the zlib or
> OpenSSL, do they?
>
> If we start taking PEP 399 conformance to such levels, we might as well
> stop developing CPython.

Indeed, PEP 399 specifically notes that exemptions can be granted for
modules that wrap external C libraries.

On Sat, Aug 27, 2011 at 5:52 PM, Nick Coghlan  wrote:
> It's acceptable for the Python version to use ctypes in the case of
> wrapping an existing library, but the Python version should still
> exist.

I'm not too sure about that - PEP 399 explicitly says that using ctypes is
frowned upon, and doesn't mention anywhere that it should be used in this
sort of situation.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

On Sat, Aug 27, 2011 at 9:47 PM, Terry Reedy  wrote:
> On 8/27/2011 9:47 AM, Nadeem Vawda wrote:
>> I'd like to propose the addition of a new module in Python 3.3. The 'lzma'
>> module will provide support for compression and decompression using the
>> LZMA
>> algorithm, and the .xz and .lzma file formats. The matter has already been
>> discussed on the tracker<http://bugs.python.org/issue6715>, where there
>> seems
>> to be a consensus that this is a desirable feature. What are your
>> thoughts?
>
> As I read the discussion, the idea has been more or less accepted in
> principle. However, the current patch is not and needs changes.

Please note that the code I'm talking about is not the same as the patches by
Per Øyvind Karlsen that are attached to the tracker issue. I have been doing
a completely new implementation of the module, specifically to address the
concerns raised by Martin and Antoine.

(As for why I haven't posted my own changes yet - I'm currently an intern at
Google, and they want me to run my code by their open-source team before
releasing it into the wild. Sorry for the delay and the confusion.)


>> The proposed module's API will be very similar to that of the bz2 module;
>> the only differences will be additional keyword arguments to some
>> functions,
>> for specifying container formats and detailed compressor options.
>
> I believe Antoine suggested a PEP. It should summarize the salient points in
> the long tracker discussion into a coherent exposition and flesh out the
> details implied above. (Perhaps they are already in the proposed doc
> addition.)

I talked to Antoine about this on IRC; he didn't seem to think a PEP would be
necessary. But a summary of the discussion on the tracker issue might still
be a useful thing to have, given how long it's gotten.


>> The implementation will also be similar to bz2 - basic compressor and
>> decompressor classes written in C, with convenience functions and a file
>> interface implemented on top of those in Python.
>
> I would follow Martin's suggestions, including doing all i/o with the io
> module and the following:
> "So I would propose that a very thin C layer is created around the C
> library that focuses on the actual algorithms, and that any higher
> layers (in particular file formats) are done in Python."
>
> If we minimize the C code we add and maximize what is done in Python, that
> would maximize the ease of porting to other implementations. This would
> conform to the spirit of PEP 399.

As stated in my earlier response to Martin, I intend to do this. Aside from
I/O, though, there's not much that _can_ be done in Python - the rest is
basically just providing a thin wrapper for the C library.


On Sat, Aug 27, 2011 at 9:58 PM, Dan Stromberg  wrote:
> I'd like to better understand why ctypes is (sometimes) frowned upon.
>
> Is it the brittleness?  Tendency to segfault?

The problem (as I understand it) is that ABI changes in a library will
cause code that uses it via ctypes to break without warning. With an
extension module, you'll get a compile failure if you rely on things
that change in an incompatible way. With a ctypes wrapper, you just get
incorrect answers, or segfaults.


> If yes, is there a way of making ctypes less brittle - say, by
> carefully matching it against a specific version of a .so/.dll before
> starting to make heavy use of said .so/.dll?

This might be feasible for a specific application running in a controlled
environment, but it seems impractical for something as widely-used as the
stdlib. Having to include a whitelist of acceptable library versions would
be a substantial maintenance burden, and (compatible) new versions would
not work until the library whitelist gets updated.


Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

On Sat, Aug 27, 2011 at 10:41 PM, Dan Stromberg  wrote:
> It seems like there should be some way of coming up with an xml file
> describing the types of the various bits of data and formal arguments -
> perhaps using gccxml or something like it.

The problem is that you would need to do this check at runtime, every time
you load up the library - otherwise, what happens if the user upgrades
their installed copy of liblzma? And we can't expect users to have the
liblzma headers installed, so we'd have to try and figure out whether the
library was ABI-compatible from the shared object alone; I doubt that this
is even possible.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

I've updated the issue  with a patch
containing my work so far - the LZMACompressor and LZMADecompressor classes,
along with some tests. These two classes should provide a fairly complete
interface to liblzma; it will be possible to implement LZMAFile on top of them,
entirely in Python. Note that the C code does no I/O; this will be handled by
LZMAFile.

Please take a look, and let me know what you think.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handling linker scripts reached when dynamically loading a module

[Nadeem Vawda]

I can confirm that libpthread.so (/usr/lib/x86_64-linux-gnu/libpthread.so)
is a linker script on my Ubuntu 11.04 install. This hasn't ever caused me
any problems, though.

As for why distributions do this, here are the contents of the script:

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread_nonshared.a )

Cheers,
Nadeem

On Sun, Sep 11, 2011 at 12:24 AM, Guido van Rossum  wrote:
> Odd. Let's see what other core devs say.
>
> On Sat, Sep 10, 2011 at 2:50 PM, Howard B. Golden
>  wrote:
>> I don't know why, but some Linux distributions place scripts into .so
>> files instead of the actual binaries. This takes advantage of a feature
>> of GNU ld that it will process the script (which points to the actual
>> binary) when it links the .so file.
>>
>> This feature works fine when you are linking a binary, but it doesn't
>> take into account that binaries can be loaded dynamically by
>> interpreters (e.g., Python or GHCi). If dlopen finds a linker script, it
>> doesn't know what to do with it. It simply diagnoses the file as either
>> an invalid ELF header or too short.
>>
>> On Gentoo Linux, some common libraries that are represented as linker
>> scripts include libm.so, libpthread.so and libpcre.so. I know this also
>> affects Ubuntu.
>>
>> Howard
>>
>> On Sat, 2011-09-10 at 14:39 -0700, Guido van Rossum wrote:
>>> Excuse me for asking a newbie question, but what are linker scripts
>>> and why are they important? I don't recall anyone ever having
>>> requested this feature before.
>>>
>>> --Guido
>>>
>>> On Wed, Sep 7, 2011 at 12:33 PM, Howard B. Golden
>>>  wrote:
>>> > Hi,
>>> >
>>> > In Haskell I experienced a situation where dynamically loaded modules
>>> > were experiencing "invalid ELF header" errors. This was caused by the
>>> > module names actually referring to linker scripts rather than ELF
>>> > binaries. I patched the GHC runtime system to deal with these scripts.
>>> >
>>> > I noticed that this same patch has been ported to Ruby and Node.js, so I
>>> > suggested to the libc developers that they might wish to incorporate the
>>> > patch into their library, making it available to all languages. They
>>> > rejected this suggestion, so I am making the suggestion to the Python
>>> > devs in case it is of interest to you.
>>> >
>>> > Basically, when a linker script is loaded by dlopen, an "invalid ELF
>>> > header" error occurs. The patch checks to see if the file is a linker
>>> > script. If so, it finds the name of the real ELF binary with a regular
>>> > expression and tries to dlopen it. If successful, processing proceeds.
>>> > Otherwise, the original "invalid ELF error" message is returned.
>>> >
>>> > If you want to add this code to Python, you can look at my original
>>> > patch (http://hackage.haskell.org/trac/ghc/ticket/2615) or the Ruby
>>> > version (https://github.com/ffi/ffi/pull/117) or the Node.js version
>>> > (https://github.com/rbranson/node-ffi/pull/5) to help port it.
>>> >
>>> > Note that the GHC version in GHC 7.2.1 has been enhanced to also handle
>>> > another possible error when the linker script is too short, so you might
>>> > also want to add this enhancement also (see
>>> > https://github.com/ghc/blob/master/rts/Linker.c line 1191 for the
>>> > revised regular expression):
>>> >
>>> > "(([^ \t()])+\\.so([^ \t:()])*):([ \t])*(invalid ELF header|file too
>>> > short)"
>>> >
>>> > At this point, I don't have the free time to write the Python patch
>>> > myself, so I apologize in advance for not providing it to you.
>>> >
>>> > HTH,
>>> >
>>> > Howard B. Golden
>>> > Northridge, California, USA
>>> >
>>> > ___
>>> > Python-Dev mailing list
>>> > Python-Dev@python.org
>>> > http://mail.python.org/mailman/listinfo/python-dev
>>> > Unsubscribe: 
>>> > http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>> >
>>>
>>>
>>>
>>
>>
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

I've posted an updated patch to the bug tracker, with a complete implementation
of the lzma module, including 100% test coverage for the LZMAFile class (which
is implemented entirely in Python). It doesn't include ReST documentation (yet),
but the docstrings are quite detailed.

Please take a look and let me know what you think.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA compression support in 3.3

[Nadeem Vawda]

Another update - I've added proper documentation. Now the code should be
pretty much complete - all that's missing is the necessary bits and pieces
to build it on Windows.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: sort last committed name in alphabetical order

[Nadeem Vawda]

Did you mean to also modify sched.py in this changeset?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] LZMA support has landed

[Nadeem Vawda]

Hey folks,

I'm pleased to announce that as of changeset 74d182cf0187, the
standard library now includes support for the LZMA compression
algorithm (as well as the associated .xz and .lzma file formats). The
new lzma module has a very similar API to the existing bz2 module; it
should serve as a drop-in replacement for most use cases.

If anyone has any feedback or any suggestions for improvement, please
let me know.

I'd like to ask the owners of (non-Windows) buildbots to install the
XZ Utils development headers so that they can build the new module.
For Debian-derived Linux distros, the relevant package is named
"liblzma-dev"; on Fedora I believe the correct package is "xz-devel".
Binaries for OS X are available from the project's homepage at
.

Finally, a big thanks to everyone who contributed feedback during this
module's development!

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: input() in this sense is gone

[Nadeem Vawda]

On Thu, Dec 15, 2011 at 10:44 PM, benjamin.peterson
 wrote:
> +#       eval_input is the input for the eval() functions.

Shouldn't this be "function" rather than "functions"?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #13605: add documentation for nargs=argparse.REMAINDER

[Nadeem Vawda]

On Thu, Jan 19, 2012 at 11:03 PM, sandro.tosi
 wrote:
> +  are gathered into a lits. This is commonly useful for command line

s/lits/list ?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Status of Mac buildbots

[Nadeem Vawda]

Hi all,

I've noticed that most of the Mac buildbots have been offline for a while:

* http://www.python.org/dev/buildbot/all/buildslaves/parc-snowleopard-1
* http://www.python.org/dev/buildbot/all/buildslaves/parc-tiger-1
* http://www.python.org/dev/buildbot/all/buildslaves/parc-leopard-1

Does anyone know what the status of these bots is? Are they
permanently down, or just temporarily inaccessible?

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of Mac buildbots

[Nadeem Vawda]

On Wed, Jan 25, 2012 at 5:35 PM, Bill Janssen  wrote:
> We're tinkering with that server room.  They should be back by the end of
> the week.

OK, cool. Thanks for the info.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Code review tool uses my old email address

[Nadeem Vawda]

This may be a bug in the tracker, possibly related to
http://psf.upfronthosting.co.za/roundup/meta/issue402 - it
seems like changes to a user's details on bugs.python.org
are not propagated to the review tool.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

So wallclock() falls back to a not-necessarily-monotonic time source
if necessary,
while monotonic() raises an exception in that case? ISTM that these
don't need to
be separate functions - rather, we can have one function that takes a
flag (called
require_monotonic, or something like that) telling it which failure mode to use.
Does that make sense?

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner
 wrote:
> I suppose that most libraries and programs will have to implement a
> similar fallback.
>
> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
>  - time.realtime(): best-effort monotonic, with a fallback
>  - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError

This was my suggestion - I think it's useful to have the fallback
available (since most users will want it), but at the same time we
should also cater to users who need a clock that is *guaranteed* to
be monotonic.

As an aside, I think "monotonic" is a better name than "realtime";
it conveys the functions purpose more clearly. Then we could call
the flag "strict".

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

A summary of the discussion so far, as I've understood it:

- We should have *one* monotonic/steady timer function, using the
  sources described in Victor's original post.

- By default, it should fall back to time.time if a better source is
  not available, but there should be a flag that can disable this
  fallback for users who really *need* a monotonic/steady time source.

- Proposed names for the function:
  * monotonic
  * steady_clock
  * wallclock
  * realtime

- Proposed names for the flag controlling fallback behavior:
  * strict (=False)
  * fallback (=True)
  * monotonic (=False)


For the function name, I think monotonic() and steady_clock() convey
the purpose of the function much better than the other two; the term
"wallclock" is actively misleading, and "realtime" seems ambiguous.

For the flag name, I'm -1 on "monotonic" -- it sounds like a flag to
decide whether to use a monotonic time source always or never, while
it actually decides between "always" and "sometimes". I think "strict"
is nicer than "fallback", but I'm fine with either one.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

+1 for time.steady(strict=False).


On Wed, Mar 14, 2012 at 7:09 PM, Kristján Valur Jónsson
 wrote:
>> - By default, it should fall back to time.time if a better source is
>>  not available, but there should be a flag that can disable this
>>  fallback for users who really *need* a monotonic/steady time source.
> As pointed out on a different thread, you don"t need this "flag" since the 
> code can easily enforce the monotonic property by maintaining a static value.
> This is how we worked around buggy implementations of QueryPerformanceCounter 
> on windows ().
> K

That's fine if you just need the clock to be monotonic, but it isn't
any help if you also want to prevent it from jumping forward.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

On Thu, Mar 15, 2012 at 1:10 PM, Paul Moore  wrote:
>> Monotonic clocks are not necessarily hardware based, and may be adjusted
>> forward by NTP.
>
> I appreciate that. But I'm still unclear how you would tell that had
> happened as part of the implementation. One call to the OS returns
> 12345. The next returns 13345. Is that because 100 ticks have passed,
> or because the clock "leapt forward"? With no point of reference, how
> can you tell?

The point (AIUI) is that you *can't* identify such adjustments (in the
absence of some sort of log of NTP updates), so we should provide a
mechanism that is guaranteed not to be affected by them.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What letter should an UnsignedLongLong get

[Nadeem Vawda]

The lzma module ran into a similar issue with 32-bit unsigned ints.
I worked around it by writing a custom converter function to use
with the "O&" code.

You can find the converter definition here:

http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134

And an example usage here:

http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l261

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 418: Add monotonic clock

[Nadeem Vawda]

On Sat, Mar 31, 2012 at 8:27 AM, Lennart Regebro  wrote:
> So, how about time.timer()?

That seems like a bad idea; it would be too easy to confuse with (or misspell
as) time.time().

Out of the big synonym list Guido posted, I rather like time.stopwatch() - it
makes it more explicit that the purpose of the function is to measure intervals,
rather identifying absolute points in time.

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Clean up the PCBuild project files, removing redundant settings and

[Nadeem Vawda]

On Sat, May 19, 2012 at 2:11 PM, kristjan.jonsson <
python-check...@python.org> wrote:

> +Visual Studio 2010 uses version 10 of the C runtime (MSVCRT9).  The
> executables
>

Shouldn't that be MSVCRT10?

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #14744: Use the new _PyUnicodeWriter internal API to speed up str%args

[Nadeem Vawda]

Since this changeset, building on Windows seems to be broken (see
http://python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/450
for example).

Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Improve an internal ipaddress test, add a comment explaining why treating

[Nadeem Vawda]

On Sun, Jun 17, 2012 at 8:33 AM, nick.coghlan
 wrote:
> +    @property
> +    def version(self):
> +        msg = '%200s has no version specified' % (type(self),)
> +        raise NotImplementedError(msg)
> +

Shouldn't that be "%.200s", rather than "%200s"?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): Merge: #16304: Optimizations for BZ2File, and minor bugfix.

[Nadeem Vawda]

On Mon, Oct 1, 2012 at 11:11 PM, nadeem.vawda
 wrote:
> http://hg.python.org/cpython/rev/a19f47d380d2
> changeset:   79382:a19f47d380d2
> parent:  79378:fb90e2ff95b7
> parent:  79381:6d7bf512e0c3
> user:    Nadeem Vawda 
> date:Mon Oct 01 23:11:35 2012 +0200
> summary:
>   Merge: #16304: Optimizations for BZ2File, and minor bugfix.

Correction: this is actually for issue *#16034*, as are all of my
other recent changesets referring to issue #16304.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): Fixes issue #16409: The reporthook callback made by the legacy

[Nadeem Vawda]

This change appears to have broken test_urllib. See, for example:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/4774/steps/test/logs/stdio

- Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.4

[Nadeem Vawda]

Just to clarify, the release branch hasn't been created yet, correct?

- Nadeem


On Sun, Feb 3, 2013 at 3:38 PM, Benjamin Peterson wrote:

> 2013/2/3 Serhiy Storchaka :
> > There are crashers for which patches were proposed but do not reviewed
> yet:
> >
> > Issue #6083: Reference counting bug in PyArg_ParseTuple and
> > PyArg_ParseTupleAndKeywords.
> >
> > Issue #7358: cStringIO not 64-bit safe.
> >
> > Issue #16137: Using time.asctime() with an array with negative tm_hour
> > causes Python Crash.
> >
> > Issue #16686: audioop overflow issues.
> >
> > #8865 is reviewed but not committed.
>
> Thanks. In the future, this should be raised to "release blocker"
> priority to get my attention.
>
> --
> Regards,
> Benjamin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com