[Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Merlijn van Deen]

http://bugs.python.org/issue6784 ("byte/unicode pickle
incompatibilities between python2 and python3")

Hello all,

Currently, pickle unpickles python2 'str' objects as python3 'str'
objects, where the encoding to use is passed to the Unpickler.
However, there are cases where it makes more sense to unpickle a
python2 'str' as python3 'bytes' - for instance when it is actually
binary data, and not text.

Currently, the mapping is as follows, when reading a pickle:
python2 'str' -> python3 'str' (using an encoding supplied to Unpickler)
python2 'unicode' -> python3 'str'

or, when creating a pickle using protocol <= 2:
python3 'str' -> python2 'unicode'
python3 'bytes' -> python2 '__builtins__.bytes object'

This issue suggests to add a flag to change the behaviour as follows:
a) python2 'str' -> python3 'bytes'
b) python3 'bytes' -> python2 'str'

The question on this is how to pass this flag. To quote Antoine (with
permission) on my mail about this issue on core-mentorship:

> I haven't answered because I'm unsure about the approach itself - do we
> want to add yet another argument to pickle methods, especially this late
> in the 3.x development cycle?


Currently, I have implemented it using an extra argument for the
Pickler and Unpickler objects ('bytestr'), which toggles the
behaviour. I.e.:
>>> pickled = Pickler(data, bytestr=True); unpickled = Unpickler(data, 
>>> bytestr=True).
This is the approach used in pickle_bytestr.patch [1]

Another option would be to implement a seperate Pickler/Unpickler
object, such that
>>> pickled = BytestrPickler(data, bytestr=True); unpickled = 
>>> BytestrUnpickler(data, bytestr=True)
This is the approach I initially implemented [2].

Alternatively, there is the option only to implement the Unpickler,
leaving the Pickler as it is. This allows
>>> unpickled = Unpickler(data, encoding=bytes)
where the bytes type is used as a special 'flag'.

And, of course, there is the option not to implement this in the stdlib at all.


What are your ideas on this?

Best,
Merlijn

[0] http://bugs.python.org/issue6784
[1] http://bugs.python.org/file24719/pickle_bytestr.patch
[2] https://github.com/valhallasw/py2/blob/master/bytestrpickle.py
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Merlijn van Deen]

Oops. I should re-read my mails before I send them, not /after/ I send them.

On 13 March 2012 12:44, Merlijn van Deen  wrote:
 pickled = BytestrPickler(data, bytestr=True); unpickled = 
 BytestrUnpickler(data, bytestr=True)

should of course read

 pickled = BytestrPickler(data); unpickled = BytestrUnpickler(data)


Sorry about that.

Merlijn
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Paul Moore]

On 13 March 2012 03:48, C. Titus Brown  wrote:
> I feel like there's a middle ground where stable, long-term go-to modules 
> could
> be mentioned, though.  I don't spend a lot of time browsing PyPI, but I 
> suspect
> almost everyone spends a certain amount of time in the Python docs (which is a
> testimony to their quality IMO).  So I'm in favor of conservative link-outs
> but without any deprecating language.

I applaud the idea of promoting the many excellent packages available.
It can be very hard to separate the good from the indifferent (or even
bad) when browsing PyPI. I've found some very good packages recently
which I'd never have known about without some random comment on a
mailing list.

However, I'm not keen on having the stdlib documentation suggest that
I should be using something else. No code should ever be documenting
"don't use me, there are better alternatives" unless it is deprecated
or obsolete.

On the other hand, I would love to see a community-maintained document
that described packages that are acknowledged as "best of breed". That
applies whether or not those packages replace something in the stdlib.
Things like pywin32, lxml, and requests would be examples in my
experience. There's no reason this *has* to be in the core
documentation - it may be relevant that nothing has sprung up
independently yet...

Maybe a separate item in the Python documentation, "External Modules",
could be created and maintained by the community? By being in the
documentation, it has a level of "official recommendation" status, and
by being a top-level document it's visible (more so than, for example,
a HOWTO document would be). Because it's in the released
documentation, it is relatively stable, which implies that external
modules would need to have a genuine track record to get in there, but
because it's community maintained it should reflect a wider consensus
than just the core developers' views.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Donald Stufft]

On Tuesday, March 13, 2012 at 9:31 AM, Paul Moore wrote:
> On 13 March 2012 03:48, C. Titus Brown mailto:c...@msu.edu)> 
> wrote:
> > I feel like there's a middle ground where stable, long-term go-to modules 
> > could
> > be mentioned, though.  I don't spend a lot of time browsing PyPI, but I 
> > suspect
> > almost everyone spends a certain amount of time in the Python docs (which 
> > is a
> > testimony to their quality IMO).  So I'm in favor of conservative link-outs
> > but without any deprecating language.
> > 
> 
> 
> I applaud the idea of promoting the many excellent packages available.
> It can be very hard to separate the good from the indifferent (or even
> bad) when browsing PyPI. I've found some very good packages recently
> which I'd never have known about without some random comment on a
> mailing list.
> 
> However, I'm not keen on having the stdlib documentation suggest that
> I should be using something else. No code should ever be documenting
> "don't use me, there are better alternatives" unless it is deprecated
> or obsolete.
> 
> On the other hand, I would love to see a community-maintained document
> that described packages that are acknowledged as "best of breed". That
> applies whether or not those packages replace something in the stdlib.
> Things like pywin32, lxml, and requests would be examples in my
> experience. There's no reason this *has* to be in the core
> documentation - it may be relevant that nothing has sprung up
> independently yet...
> 
> 

http://python-guide.org ?
> 
> Maybe a separate item in the Python documentation, "External Modules",
> could be created and maintained by the community? By being in the
> documentation, it has a level of "official recommendation" status, and
> by being a top-level document it's visible (more so than, for example,
> a HOWTO document would be). Because it's in the released
> documentation, it is relatively stable, which implies that external
> modules would need to have a genuine track record to get in there, but
> because it's community maintained it should reflect a wider consensus
> than just the core developers' views.
> 
> Paul.
> ___
> Python-Dev mailing list
> Python-Dev@python.org (mailto:Python-Dev@python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
> 
> 


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Review of PEP 362 (signature object)

[Yury Selivanov]

Guido, Brett,

I've tried to use the proposed signature object, however, I found that 
the 'bind' method is incorrect, and came up with my own implementation 
of the PEP: https://gist.github.com/2029032  (If needed, I can change 
the licence to PSFL)  I used my version to implement typechecking,
arguments validation in various RPC dispatch mechanisms, and it is 
proven to work.

First of all, in the current version of the PEP, "bind" doesn't work 
correctly with "varargs", as it returns a dictionary object:

def foo(bar, *args):
print(bar, args)

s = signature(foo)
bound = s.bind(1, 2 ,3 ,4)
print('Bound args:', bound)
foo(**bound)

This code outputs the following:

Bound args: {'args': (2, 3, 4), 'bar': 1}
Traceback (most recent call last):
  File "test.py", line 286, in 
foo(**bound)
TypeError: foo() got an unexpected keyword argument 'args'

The conclusion is that ** form of unpacking is not always enough, so
'bind' should at least return a pair of (args, kwargs).

Secondly, I don't think that even (args, kwargs) pair is enough, as
some information about how arguments were mapped is lost.  In my
implementation, 'bind' method returns an instance of 'BoundArguments'
class, which preserves the exact mapping, and has two convenience
properties '.args' and '.kwargs', so the example above transforms into:

bound = s.bind(1, 2, 3, 4)
foo(*bound.args, **bound.kwargs)

And that works as it should.  When some advanced processing is required,
you can work with its private fields:

>>> bound._args
(1,)
>>> bound._varargs
(2, 3, 4)

I also think that keyword-only arguments deserve to land in a separate
collection in the signature object, in my implementation it is
'signature.kwonlyargs' slot.  It is easier to do some advanced processing
of arguments this way, and even the 'signature.__iter__' is simpler.

Finally, I also added 'render_args' method to the signature.  It just
renders function's arguments as in its definition:
 
>>> signature(foo).render_args()
bar, *args

This is useful for printing detailed error messages and hints.

-
Yury
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Paul Moore]

On 13 March 2012 13:34, Donald Stufft  wrote:

> http://python-guide.org ?

Hmm, yes maybe. I had seen this before (it's where I found out about
requests, IIRC).

As it says, it "is mostly a skeleton at the moment". With some
fleshing out, then it's probably a good start. I have some problems
with its recommendations (notably "If you’re starting work on a new
Python module, I recommend you write it for Python 2.5 or 2.6, and add
support for Python3 in a later iteration." which is probably not
appropriate for something that would be officially sanctioned by the
core developers). Also, I don't think we want something advertised as
"opinionated".

And it covers a much wider area than the original suggestion. I'd
envisaged something more like a simple list of "obvious" modules:

"""
Requests - URL

Requests is a module designed to make getting resources from the web
as easy as possible. It is a simpler and more powerful alternative to
the stdlib urllib and urllib2 modules. ...

Some code samples here giving basic usage.
"""

My ideal would be something I could scan in a few spare moments, and
pick up pointers to particular modules that I'd find useful.
Basically, take the "Scenario Guide" section of python-guide, flesh it
out, and turn it into a flat list. (I don't like the "Scenario"
approach as it tends to force people into a particular view of the
world, but maybe that's just me, my applications tend to be more
eclectic than any of the "normal" categories).

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Guido van Rossum]

On Mon, Mar 12, 2012 at 10:10 PM, Senthil Kumaran  wrote:
> On Mon, Mar 12, 2012 at 07:23:11PM -0700, Andrey Petrov wrote:
>> I've had the pleasure of speaking with Guido at PyCon and it became evident
>> that some of Python's included batteries are significantly lagging behind the
>> rapidly-evolving defacto standards of the community specifically in cases 
>> like
>> urllib and urllib2, which lack important features provided by alternatives 
>> like
>> httplib2, requests, and my own urllib3.
>
> Well, I think I have address this because I am the maintainer of those
> modules in standard lib.
>
> First things first, it looks to me that trashing something gives good
> motivation to you (and others working on related modules). I don't
> have a problem with that.
>
> But on the other hand, if you think things can be improved in stdlib,
> you are welcome to contribute. Just remember that new features,
> refactoring with backwards compatibility, 'cool api' for new features
> should go in 3.3+ onwards. Bug fixes, confusions on what's RFC
> supported vs what's defacto standards, fine line between bugs and
> features, those can be considered for 2.7.
>
> I am personally in favor of constantly improving the standard library
> modules along with mention of any good libraries which can be useful
> for the purposes of the user.
>
> We already have lots of such references in standard library
> documentation. If there is a well maintained package, as long as the
> external package is active and maintained, we can have it as link in
> the docs. Sometimes those external packages become inactive too, in
> those cases, those should pruned. Its' all about maintaining libraries
> and docs and being helpful.

Well said. Improving existing stdlib modules is always welcome of
course! (And the bar is much lower than for adding new modules.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Exceptions in comparison operators

[Guido van Rossum]

Mark, did you do anything with my reply?

On Mon, Mar 5, 2012 at 10:41 AM, Guido van Rossum  wrote:
> On Mon, Mar 5, 2012 at 4:41 AM, Mark Shannon  wrote:
>> Comparing two objects (of the same type for simplicity)
>> involves a three stage lookup:
>> The class has the operator C.__eq__
>> It can be applied to operator (descriptor protocol): C().__eq__
>> and it produces a result: C().__eq__(C())
>>
>> Exceptions can be raised in all 3 phases,
>> but an exception in the first phase is not really an error,
>> its just says the operation is not supported.
>> E.g.
>>
>> class C: pass
>>
>> C() == C() is False, rather than raising an Exception.
>>
>> If an exception is raised in the 3rd stage, then it is propogated,
>> as follows:
>>
>> class C:
>>   def __eq__(self, other):
>>       raise Exception("I'm incomparable")
>>
>> C() == C()  raises an exception
>>
>> However, if an exception is raised in the second phase (descriptor)
>> then it is silenced:
>>
>> def no_eq(self):
>>    raise Exception("I'm incomparable")
>>
>> class C:
>>   __eq__ = property(no_eq)
>>
>> C() == C() is False.
>>
>> But should it raise an exception?
>>
>> The behaviour for arithmetic is different.
>>
>> def no_add(self):
>>    raise Exception("I don't add up")
>>
>> class C:
>>   __add__ = property(no_add)
>>
>> C() + C() raises an exception.
>>
>> So what is the "correct" behaviour?
>> It is my opinion that comparisons should behave like arithmetic
>> and raise an exception.
>
> I think you're probably right. This is one of those edge cases that
> are so rare (and always considered a bug in the user code) that we
> didn't define carefully what should happen. There are probably some
> implementation-specific reasons why it was done this way (comparisons
> use a very different code path from regular binary operators) but that
> doesn't sound like a very good reason.
>
> OTOH there *is* a difference: as you say, C() == C() is False when the
> class doesn't define __eq__, whereas C() + C() raises an exception if
> it doesn't define __add__. Still, this is more likely to have favored
> the wrong outcome for (2) by accident than by design.
>
> You'll have to dig through the CPython implementation and find out
> exactly what code needs to be changed before I could be sure though --
> sometimes seeing the code jogs my memory.
>
> But I think of x==y as roughly equivalent to
>
> r = NotImplemented
> if hasattr(x, '__eq__'):
>  r = x.__eq__(y)
> if r is NotImplemented and hasattr(y, '__eq__'):
>  r = y.__eq__(x)
> if r is NotImplemented:
>  r = False
>
> which would certainly suggest that (2) should raise an exception. A
> possibility is that the code looking for the __eq__ attribute
> suppresses *all* exceptions instead of just AttributeError. If you
> change no_eq() to return 42, for example, the comparison raises the
> much more reasonable TypeError: 'int' object is not callable.
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Distutils2 1.0a4

[Éric Araujo]

What would be a release email without errors?  :)  The wiki link I gave
doesn’t work, it should be
http://wiki.python.org/moin/Distutils2/Contributing
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASED] Distutils2 1.0a4

[Éric Araujo]

Hello,

On behalf of the distutils2 contributors, I am thrilled to announce the
release of Distutils2 1.0a4.

Distutils2 is the packaging library that supersedes Distutils. It
supports distributing, uploading, downloading, installing and removing
projects, and is also a support library for other packaging tools like
pip and buildout.  It will be provided as part of Python 3.3; this
release is a backport compatible with Python 2.5 to 2.7.

Distutils2 1.0a4 contains a number of known bugs, limitations and
missing features, but we have released it to let users and developers
download and try it.  This means you!  If you want to report new issues,
request features or contribute, please read DEVNOTES.txt in the source
distribution or http://wiki.python.org/Distutils2/Contributing

More alpha releases will be cut when important bugs get fixed during the
next few months, like Windows or PyPy compatibility.  The first beta is
planned for June, and 1.0 final for August (to follow Python 3.3.0).
Until beta, the API is subject to drastic changes and code can get removed.

Basic documentation is at http://docs.python.org/dev/packaging ; it will
get updated, expanded and improved in the coming months.

Enjoy!

Repository: http://hg.python.org/distutils2
Bug tracker: http://bugs.python.org/ (component "Distutils2")
Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Distutils2 1.0a4

[Tarek Ziadé]

Thanks a lot for your hard work and dedication on packaging  !

On 3/13/12 9:37 AM, Éric Araujo wrote:

Hello,

On behalf of the distutils2 contributors, I am thrilled to announce the
release of Distutils2 1.0a4.

Distutils2 is the packaging library that supersedes Distutils. It
supports distributing, uploading, downloading, installing and removing
projects, and is also a support library for other packaging tools like
pip and buildout.  It will be provided as part of Python 3.3; this
release is a backport compatible with Python 2.5 to 2.7.

Distutils2 1.0a4 contains a number of known bugs, limitations and
missing features, but we have released it to let users and developers
download and try it.  This means you!  If you want to report new issues,
request features or contribute, please read DEVNOTES.txt in the source
distribution or http://wiki.python.org/Distutils2/Contributing

More alpha releases will be cut when important bugs get fixed during the
next few months, like Windows or PyPy compatibility.  The first beta is
planned for June, and 1.0 final for August (to follow Python 3.3.0).
Until beta, the API is subject to drastic changes and code can get removed.

Basic documentation is at http://docs.python.org/dev/packaging ; it will
get updated, expanded and improved in the coming months.

Enjoy!

Repository: http://hg.python.org/distutils2
Bug tracker: http://bugs.python.org/ (component "Distutils2")
Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Exceptions in comparison operators

[Mark Shannon]

Guido van Rossum wrote:

Mark, did you do anything with my reply?


Not yet.

I noticed the difference when developing my HotPy VM
(latest incarnation thereof) which substitutes a sequence of low-level
bytecodes for the high-level ones when tracing.
(A bit like PyPy but much more Python-specific and amenable to 
interpretation, rather than compilation)


I generate all the code sequences for binary ops from a template
and noticed the slight difference when running the test suite.
My implementation of equals follows the same pattern as the arithmetic 
operators (which is why I was wondering if that were the correct behaviour).


My definition of op1 == op2:

def surrogate_eq(op1, op2):
if $overrides(op1, op2, '__eq__'):
if op2?__eq__:
result = op2$__eq__(op1)
if result is not NotImplemented:
return result
if op1?__eq__:
result = op1$__eq__(op2)
if result is not NotImplemented:
return result
else:
if op1?__eq__:
result = op1$__eq__(op2)
if result is not NotImplemented:
return result
if op2?__eq__:
result = op2$__eq__(op1)
if result is not NotImplemented:
return result
return op1 is op2

Where:

x$__op__ means special lookup (bypassing the instance dictionary):

x?__op__ means has the named special method i.e.
any('__op__' in t.__dict__ for t in type(op).__mro__))

and
$overrides(op1, op2, 'xxx') means that
type(op2) is a proper subtype of type(op1)
*and* type(op1).__dict__['xxx'] != type(op2).__dict__['xxx']


It would appear that the current version is:

def surrogate_eq(op1, op2):
if is_proper_subtype_of( type(op1), type(op1) ):
if op2?__eq__:
result = op2$__eq__(op1)
if result is not NotImplemented:
return result
if op1?__eq__:
result = op1$__eq__(op2)
if result is not NotImplemented:
return result
else:
if op1?__eq__:
result = op1$__eq__(op2)
if result is not NotImplemented:
return result
if op2?__eq__:
result = op2$__eq__(op1)
if result is not NotImplemented:
return result
return op1 is op2

Which means that == behaves differently to + for
subtypes which do not override the __eq__ method.
Thus:

class MyValue1:
def __init__(self, val):
self.val = val

def __lt__(self, other):
print("lt")
return self.val < other.val

def __gt__(self, other):
print("gt")
return self.val > other.val

def __add__(self, other):
print("add")
return self.val + other.val

def __radd__(self, other):
print("radd")
return self.val + other.val

class MyValue2(MyValue1):
pass

a = MyValue1(1)
b = MyValue2(2)

print(a + b)
print(a < b)

currently prints the following:

add
3
gt
True

Cheers,
Mark.



On Mon, Mar 5, 2012 at 10:41 AM, Guido van Rossum  wrote:

On Mon, Mar 5, 2012 at 4:41 AM, Mark Shannon  wrote:

Comparing two objects (of the same type for simplicity)
involves a three stage lookup:
The class has the operator C.__eq__
It can be applied to operator (descriptor protocol): C().__eq__
and it produces a result: C().__eq__(C())

Exceptions can be raised in all 3 phases,
but an exception in the first phase is not really an error,
its just says the operation is not supported.
E.g.

class C: pass

C() == C() is False, rather than raising an Exception.

If an exception is raised in the 3rd stage, then it is propogated,
as follows:

class C:
  def __eq__(self, other):
  raise Exception("I'm incomparable")

C() == C()  raises an exception

However, if an exception is raised in the second phase (descriptor)
then it is silenced:

def no_eq(self):
   raise Exception("I'm incomparable")

class C:
  __eq__ = property(no_eq)

C() == C() is False.

But should it raise an exception?

The behaviour for arithmetic is different.

def no_add(self):
   raise Exception("I don't add up")

class C:
  __add__ = property(no_add)

C() + C() raises an exception.

So what is the "correct" behaviour?
It is my opinion that comparisons should behave like arithmetic
and raise an exception.

I think you're probably right. This is one of those edge cases that
are so rare (and always considered a bug in the user code) that we
didn't define carefully what should happen. There are probably some
implementation-specific reasons why it was done this way (comparisons
use a very different code path from regular binary operators) but that
doesn't sound like a very good reason.

OTOH there *is* a difference: as you say, C() == C() is False when the
class doesn't define __eq__, whereas C() + C() raises an exception if
it doesn't define __add__. Still, this is more likely to have favored
the wrong outcome for (2) by accident than by design.

You'll have to dig through th

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Glenn Linderman]

On 3/13/2012 6:31 AM, Paul Moore wrote:

It can be very hard to separate the good from the indifferent (or even
bad) when browsing PyPI. I've found some very good packages recently
which I'd never have known about without some random comment on a
mailing list.


+1


However, I'm not keen on having the stdlib documentation suggest that
I should be using something else. No code should ever be documenting
"don't use me, there are better alternatives" unless it is deprecated
or obsolete.


+0


On the other hand, I would love to see a community-maintained document
that described packages that are acknowledged as "best of breed". That
applies whether or not those packages replace something in the stdlib.
Things like pywin32, lxml, and requests would be examples in my
experience.


+1


There's no reason this*has*  to be in the core
documentation - it may be relevant that nothing has sprung up
independently yet...


Hmm.


Maybe a separate item in the Python documentation, "External Modules",
could be created and maintained by the community? By being in the
documentation, it has a level of "official recommendation" status, and
by being a top-level document it's visible (more so than, for example,
a HOWTO document would be). Because it's in the released
documentation, it is relatively stable, which implies that external
modules would need to have a genuine track record to get in there, but
because it's community maintained it should reflect a wider consensus
than just the core developers' views.


+1  This is the best proposal I've seen for including references to 
external modules.  It gets it in the core documentation, hopefully with 
enough keywords that search would typically find external modules that 
are supersets of stdlib modules in the same result set that the stdlib 
module would be found.  Yet it doesn't intrude on the documentation for 
the stdlib module.  And beyond a 1-2 paragraph description, would not be 
fully documented, except by referencing the external module's documentation.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Kenneth Reitz]

I think the cheesehop trove classifiers would be the ideal way to agnostically 
link to a page of packages related to the standard package in question. No need 
for sort order.

The beauty of this solution is that packages that aren't maintained won't add 
the appropriate classifier to their package, and therefore not show up in the 
list.  

--  
Kenneth Reitz


On Monday, March 12, 2012 at 9:23 PM, Brian Curtin wrote:

> On Mon, Mar 12, 2012 at 21:14, Andrey Petrov  (mailto:sha...@gmail.com)> wrote:
> > On Mon, Mar 12, 2012 at 8:58 PM, Brian Curtin  > (mailto:br...@python.org)> wrote:
> > > On Mon, Mar 12, 2012 at 19:23, Andrey Petrov  > > (mailto:sha...@gmail.com)> wrote:
> > > > What such a snippet might look like:
> > > >  
> > > > "Batteries are included with Python but sometimes they are old and
> > > > leaky—this is one of those cases. Please have a look in PyPI for more 
> > > > modern
> > > > alternatives provided by the Python community."
> > > >  
> > >  
> > >  
> > > What does "leaky" mean here? Someone's going to see that, think it has
> > > memory leaks, then rant on the internet about how we ship crap and
> > > just document it as so.
> > >  
> >  
> >  
> > I agree Brian and David, the choice of "leaky" in the wording is poor.
> > It was supposed to be maintaining the "batteries" metaphor but it's
> > clearly ambiguous.
> >  
> > Perhaps something along the lines of...
> >  
> > "Batteries are included with Python but for stability and backwards
> > compatibility, some of the standard library is not always as modern as
> > alternatives provided by the Python community—this is one of those
> > cases. Please have a look at PyPI for more cutting-edge alternatives."
> >  
>  
>  
> Sorry for another color choice on the bikeshed, but I would drop the
> word or references to "batteries". *We* know what "batteries included"
> means, but there are undoubtedly people who won't get it. It's just
> code - let's call it code.
>  
> > > > Part 2:
> > > > I propose we add a new category of package identifiers such as "Topic ::
> > > > Standard Library Alternative :: {stdlib_package_name}" which authors of
> > > > libraries can tag themselves under. The documentation warning snippet 
> > > > will
> > > > provide a link to the appropriate PyPI query to list packages claiming 
> > > > to be
> > > > alternatives to the stdlib package in question.
> > > >  
> > >  
> > >  
> > > Automating it to something on PyPI is the not the right answer. People
> > > will use it incorrectly, either in that they'll add it to packages for
> > > which it isn't accurate, and people just flat out won't use it or know
> > > about it. It won't be accurate this way, and anything that we're
> > > documenting needs to be vetted.
> > >  
> > > It's not often that a great alternative comes up, so I don't see the
> > > manual burden being too great.
> > >  
> >  
> >  
> > There are a dozen or more urllib/httplib/pycurl competitors on PyPI,
> > and new ones spring up all the time. I'm not sure how we would go
> > about objectively blessing the best "official" option at any given
> > moment, or how frequently we would have to do this.
> >  
>  
>  
> The same way we choose to accept libraries into the standard library.
> New ones spring up all the time - mature, proven, and widely used ones
> do not. If someone thinks libfoo is ready, they suggest it. If we
> haven't heard of it, the conversation ends. If we have people who know
> it, maybe we have them look deeper and figure out if it's something we
> can put our stamp on just like we might with the recent talk of
> "experimental package" inclusion.
>  
> > With self-identifying, we could sort by some sort metric (monthly
> > downloads? magical score?) and create a somewhat-actionable list.
> >  
>  
>  
> Downloads don't mean the code is good. Voting is gamed. I really don't
> think there's a good automated solution to tell us what the
> high-quality replacement projects are.
>  
>  


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Brian Curtin]

On Tue, Mar 13, 2012 at 14:13, Kenneth Reitz  wrote:
> I think the cheesehop trove classifiers would be the ideal way to
> agnostically link to a page of packages related to the standard package in
> question. No need for sort order.

Randomize the order for all I care. We still need to ensure we're
suggesting quality projects. It doesn't make sense for us to suggest
alternatives that we wouldn't want to use ourselves by just polling
some list that anyone can get on.

This is documentation that receives hundreds of thousands of views a
month*. We need to be picky about what goes in it.

> The beauty of this solution is that packages that aren't maintained won't
> add the appropriate classifier to their package, and therefore not show up
> in the list.

Just because it's maintained doesn't mean it's not garbage. I think we
really need to start every project off with a 0 and make them prove
that they're a 10. Just being active means nothing.



* http://www.python.org/webstats/usage_201202.html#TOPURLS - I don't
know what page "Documentation" means since it doesn't have a specific
link, but whatever page that is got hit 960K times in February.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Michael Foord]

On 13 Mar 2012, at 04:44, Merlijn van Deen wrote:

> http://bugs.python.org/issue6784 ("byte/unicode pickle
> incompatibilities between python2 and python3")
> 
> Hello all,
> 
> Currently, pickle unpickles python2 'str' objects as python3 'str'
> objects, where the encoding to use is passed to the Unpickler.
> However, there are cases where it makes more sense to unpickle a
> python2 'str' as python3 'bytes' - for instance when it is actually
> binary data, and not text.
> 
> Currently, the mapping is as follows, when reading a pickle:
> python2 'str' -> python3 'str' (using an encoding supplied to Unpickler)
> python2 'unicode' -> python3 'str'
> 
> or, when creating a pickle using protocol <= 2:
> python3 'str' -> python2 'unicode'
> python3 'bytes' -> python2 '__builtins__.bytes object'
> 


It does seem unfortunate that by default it is impossible for a developer to 
"do the right thing" as regards pickling / unpickling here. Binary data on 
Python 2 being unpickled as Unicode on Python 3 is presumably for the 
convenience of developers doing the *wrong thing* (and only works for ascii 
anyway).

All the best,

Michael Foord


> This issue suggests to add a flag to change the behaviour as follows:
> a) python2 'str' -> python3 'bytes'
> b) python3 'bytes' -> python2 'str'
> 
> The question on this is how to pass this flag. To quote Antoine (with
> permission) on my mail about this issue on core-mentorship:
> 
>> I haven't answered because I'm unsure about the approach itself - do we
>> want to add yet another argument to pickle methods, especially this late
>> in the 3.x development cycle?
> 
> 
> Currently, I have implemented it using an extra argument for the
> Pickler and Unpickler objects ('bytestr'), which toggles the
> behaviour. I.e.:
 pickled = Pickler(data, bytestr=True); unpickled = Unpickler(data, 
 bytestr=True).
> This is the approach used in pickle_bytestr.patch [1]
> 
> Another option would be to implement a seperate Pickler/Unpickler
> object, such that
 pickled = BytestrPickler(data, bytestr=True); unpickled = 
 BytestrUnpickler(data, bytestr=True)
> This is the approach I initially implemented [2].
> 
> Alternatively, there is the option only to implement the Unpickler,
> leaving the Pickler as it is. This allows
 unpickled = Unpickler(data, encoding=bytes)
> where the bytes type is used as a special 'flag'.
> 
> And, of course, there is the option not to implement this in the stdlib at 
> all.
> 
> 
> What are your ideas on this?
> 
> Best,
> Merlijn
> 
> [0] http://bugs.python.org/issue6784
> [1] http://bugs.python.org/file24719/pickle_bytestr.patch
> [2] https://github.com/valhallasw/py2/blob/master/bytestrpickle.py
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python install layout and the PATH on win32

[VanL]

Following up on conversations at PyCon, I want to bring up one of my 
personal hobby horses for change in 3.3: Fix install layout on Windows, 
with a side order of making the PATH work better.


Short version:

1) The layout for the python root directory for all platforms should be 
as follows:


stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}

2) On Windows, the python executable (python.exe) will be in the "bin" 
directory. That way the installer can optionally add just that directory 
to the PATH to pick up all python-related executables (like pip, 
easy_install, etc).


I have talked with a number of people at PyCon, including Tarek and MvL. 
Nobody objected, and several thought it was a good idea.


Long version:

As a bit of background,the layout for the Python root directory is 
different between platforms, varying in capitalization ("Include" vs. 
"include") and sometimes in the names of directories used ("Scripts" on 
Windows vs. "bin" most everywhere else). Further, the python version may 
or may not be included in the path to the standard library or not.


In times past, this layout was driven by an INSTALL_SCHEMES dict deep in 
the guts of distutils, but with distutils2 it has been lifted out and 
placed within a config file (sysconfig.cfg). [1]


Proposal #1 above deals with this inconsistency [1]. More concretely, it 
also makes it so that I can check in an entire environment into source 
control and work on it cross platform.


As an additional wrinkle on Windows, the main python binary (python.exe) 
is placed in the python root directory, but all associated runnable 
files are placed in the "Scripts" directory, so that someone who wants 
to run both Python and a Python script needs to add both $PYTHON_ROOT 
and $PYTHON_ROOT/Scripts to the PATH.


To add just a little more complication, the python binary is placed 
within the binaries directory when a virtualenv is created, leading to 
an inconsistency between regular python installs and virtualenvs.


Proposal #2 again provides consistency between virtualenvs and regular 
Python installs, and on windows it allows a single directory to be 
placed on the PATH to get all python-related binaries to run.


[1] 
https://bitbucket.org/tarek/distutils2/src/6c3d67ed3adb/distutils2/_backport/sysconfig.cfg

[2] It may be a foolish consistency, but I have a little mind.

Thanks, Van


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Terry Reedy]

On 3/13/2012 12:40 AM, Guido van Rossum wrote:

On Mon, Mar 12, 2012 at 9:22 PM, Terry Reedy  wrote:

I would rather we figure out how to encourage authors of advancing packages
to contribute better implementations of existing features and well-tested
new features back to the stdlib module.


I would not.


I think you misunderstood me and are talking about something other than 
what I meant. There are about 3250 open issues (this slowly but steadily 
grows). Of them, 1450 are behavior (bug) issues. We need more people, 
especially people with specialized expertise, writing and reviewing 
patches. As you said in response to Senthil


> Improving existing stdlib modules is always welcome

Exactly. So I would like to figure out how to encourage more such 
improvements.



There are many excellent packages out there that should
not be made into stdlib packages simply because their authors are not
done adding new features.


Or because the package is outside the reasonable scope of the stdlib, or 
requires a different type of expertise than most core development, or 
for other reasons.


Authors of separately maintained packages are, from our viewpoint, as 
eligible to help with tracker issues as anyone else, even while they 
continue work on their external package. Some of them are more likely 
than most contributors to have the knowledge needed for some particular 
issues.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Thomas Wouters]

On Tue, Mar 13, 2012 at 12:38, Brian Curtin  wrote:

> On Tue, Mar 13, 2012 at 14:13, Kenneth Reitz  wrote:
> > I think the cheesehop trove classifiers would be the ideal way to
> > agnostically link to a page of packages related to the standard package
> in
> > question. No need for sort order.
>
> Randomize the order for all I care. We still need to ensure we're
> suggesting quality projects. It doesn't make sense for us to suggest
> alternatives that we wouldn't want to use ourselves by just polling
> some list that anyone can get on.
>
> This is documentation that receives hundreds of thousands of views a
> month*. We need to be picky about what goes in it.
>
> > The beauty of this solution is that packages that aren't maintained won't
> > add the appropriate classifier to their package, and therefore not show
> up
> > in the list.
>
> Just because it's maintained doesn't mean it's not garbage. I think we
> really need to start every project off with a 0 and make them prove
> that they're a 10. Just being active means nothing.
>
>
>
> * http://www.python.org/webstats/usage_201202.html#TOPURLS - I don't
> know what page "Documentation" means since it doesn't have a specific
> link, but whatever page that is got hit 960K times in February.
>

GroupURL/doc/*  Documentation

So it's anything that's in www.python.org/doc/. I don't believe it counts
doc.python.org and docs.python.org.

-- 
Thomas Wouters 

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Brian Curtin]

On Tue, Mar 13, 2012 at 14:43, VanL  wrote:
> Following up on conversations at PyCon, I want to bring up one of my
> personal hobby horses for change in 3.3: Fix install layout on Windows, with
> a side order of making the PATH work better.
>
> Short version:
>
> 1) The layout for the python root directory for all platforms should be as
> follows:
>
> stdlib = {base/userbase}/lib/python{py_version_short}
> platstdlib = {base/userbase}/lib/python{py_version_short}
> purelib = {base/userbase}/lib/python{py_version_short}/site-packages
> platlib = {base/userbase}/lib/python{py_version_short}/site-packages
> include = {base/userbase}/include/python{py_version_short}
> scripts = {base/userbase}/bin
> data = {base/userbase}

I'm familiar with the scripts/bin change. I take it the rest of that
stuff matches *nix? Text later on seems to support this, so I think
I'm on board with it.

> 2) On Windows, the python executable (python.exe) will be in the "bin"
> directory. That way the installer can optionally add just that directory to
> the PATH to pick up all python-related executables (like pip, easy_install,
> etc).

I'm updating my installer patch to do exactly this. After talking with
Dino from Microsoft's Python Tools team, we're also going to add an
additional registry key for them to find that bin/ path.

> I have talked with a number of people at PyCon, including Tarek and MvL.
> Nobody objected, and several thought it was a good idea.

Martin and I spoke on Friday and at least the bin/ folder and Path
stuff are acceptable and I'm working on the code for those.

> To add just a little more complication, the python binary is placed within
> the binaries directory when a virtualenv is created, leading to an
> inconsistency between regular python installs and virtualenvs.

If that virtualenv PEP is also accepted for 3.3, I think we can take
care of inconsistencies there (at least moving forward).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Guido van Rossum]

On Tue, Mar 13, 2012 at 12:42 PM, Michael Foord
 wrote:
>
> On 13 Mar 2012, at 04:44, Merlijn van Deen wrote:
>
>> http://bugs.python.org/issue6784 ("byte/unicode pickle
>> incompatibilities between python2 and python3")
>>
>> Hello all,
>>
>> Currently, pickle unpickles python2 'str' objects as python3 'str'
>> objects, where the encoding to use is passed to the Unpickler.
>> However, there are cases where it makes more sense to unpickle a
>> python2 'str' as python3 'bytes' - for instance when it is actually
>> binary data, and not text.
>>
>> Currently, the mapping is as follows, when reading a pickle:
>> python2 'str' -> python3 'str' (using an encoding supplied to Unpickler)
>> python2 'unicode' -> python3 'str'
>>
>> or, when creating a pickle using protocol <= 2:
>> python3 'str' -> python2 'unicode'
>> python3 'bytes' -> python2 '__builtins__.bytes object'
>>
>
>
> It does seem unfortunate that by default it is impossible for a developer to 
> "do the right thing" as regards pickling / unpickling here. Binary data on 
> Python 2 being unpickled as Unicode on Python 3 is presumably for the 
> convenience of developers doing the *wrong thing* (and only works for ascii 
> anyway).

Well, since trying to migrate data between versions using pickle is
the "wrong" thing anyway, I think the status quo is just fine.
Developers doing the "right" thing don't use pickle for this purpose.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Guido van Rossum]

On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy  wrote:
> Authors of separately maintained packages are, from our viewpoint, as
> eligible to help with tracker issues as anyone else, even while they
> continue work on their external package. Some of them are more likely than
> most contributors to have the knowledge needed for some particular issues.

This is a good idea. I was chatting w. Senthil this morning about
adding improvements to urllib/request.py based upon ideas from
urllib3, requests, httplib2 (?), and we came to the conclusion that it
might be a good idea to let those packages' authors review the
proposed stdlib improvements.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Terry Reedy]

On 3/13/2012 3:43 PM, VanL wrote:

Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows,
with a side order of making the PATH work better.

Short version:

1) The layout for the python root directory for all platforms should be
as follows:

stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}


What is {base/userbase} actually on a typical machine? It is fixed or 
user choice?


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Antoine Pitrou]

On Tue, 13 Mar 2012 14:16:40 -0700
Guido van Rossum  wrote:

> On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy  wrote:
> > Authors of separately maintained packages are, from our viewpoint, as
> > eligible to help with tracker issues as anyone else, even while they
> > continue work on their external package. Some of them are more likely than
> > most contributors to have the knowledge needed for some particular issues.
> 
> This is a good idea. I was chatting w. Senthil this morning about
> adding improvements to urllib/request.py based upon ideas from
> urllib3, requests, httplib2 (?), and we came to the conclusion that it
> might be a good idea to let those packages' authors review the
> proposed stdlib improvements.

We don't have any provisions against reviewal by third-party
developers already. I think the main problem (for us, of course) is that
these people generally aren't interested enough to really dive in
stdlib patches and proposals.

For example, for the ssl module, I have sometimes tried to involve
authors of third-party packages such as pyOpenSSL (or, IIRC, M2Crypto),
but I got very little or no reviewing.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Guido van Rossum]

On Tue, Mar 13, 2012 at 2:21 PM, Antoine Pitrou  wrote:
> On Tue, 13 Mar 2012 14:16:40 -0700
> Guido van Rossum  wrote:
>
>> On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy  wrote:
>> > Authors of separately maintained packages are, from our viewpoint, as
>> > eligible to help with tracker issues as anyone else, even while they
>> > continue work on their external package. Some of them are more likely than
>> > most contributors to have the knowledge needed for some particular issues.
>>
>> This is a good idea. I was chatting w. Senthil this morning about
>> adding improvements to urllib/request.py based upon ideas from
>> urllib3, requests, httplib2 (?), and we came to the conclusion that it
>> might be a good idea to let those packages' authors review the
>> proposed stdlib improvements.
>
> We don't have any provisions against reviewal by third-party
> developers already. I think the main problem (for us, of course) is that
> these people generally aren't interested enough to really dive in
> stdlib patches and proposals.
>
> For example, for the ssl module, I have sometimes tried to involve
> authors of third-party packages such as pyOpenSSL (or, IIRC, M2Crypto),
> but I got very little or no reviewing.

IIRC M2Crypto is currently unmaintained, so that doesn't surprise me.
(In general it seems most crypto wrappers seem unmaintained -- it must
be a thankless job.)

Still, AFAICT both requests and urllib3 are very actively maintained
by people who know what they are doing, and it would be nice if we
could build bridges instead of competition. So let's at least try.
(But I'm not asking you, Antoine, to try and approach them personally.
:-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/13/2012 4:19 PM, Terry Reedy wrote:

What is {base/userbase} actually on a typical machine? It is fixed or
user choice?



It is based upon user choice and on whether it is a system-wide install 
(base) or a single-user install (userbase). Typically, though, it is 
just "where you installed Python" (/usr/local, C:\python\3.3, whatever).





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/13/2012 3:11 PM, Brian Curtin wrote:


I'm familiar with the scripts/bin change. I take it the rest of that
stuff matches *nix? Text later on seems to support this, so I think
I'm on board with it.


Yes, that is correct.


Martin and I spoke on Friday and at least the bin/ folder and Path
stuff are acceptable and I'm working on the code for those.


[...]


If that virtualenv PEP is also accepted for 3.3, I think we can take
care of inconsistencies there (at least moving forward).


Thanks Brian!

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Merlijn van Deen]

On 13 March 2012 22:13, Guido van Rossum  wrote:
> Well, since trying to migrate data between versions using pickle is
> the "wrong" thing anyway, I think the status quo is just fine.
> Developers doing the "right" thing don't use pickle for this purpose.

I'm confused by this. "The pickle serialization format is guaranteed
to be backwards compatible across Python releases" [1], which - at
least to me - suggests it's fine to use pickle for long-term storage,
and that reading this data in new Python versions is not a "bad"
thing to do. Am I missing something here?

[1] http://docs.python.org/library/pickle.html#the-pickle-protocol
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Guido van Rossum]

On Tue, Mar 13, 2012 at 2:50 PM, Merlijn van Deen  wrote:
> On 13 March 2012 22:13, Guido van Rossum  wrote:
>> Well, since trying to migrate data between versions using pickle is
>> the "wrong" thing anyway, I think the status quo is just fine.
>> Developers doing the "right" thing don't use pickle for this purpose.
>
> I'm confused by this. "The pickle serialization format is guaranteed
> to be backwards compatible across Python releases" [1], which - at
> least to me - suggests it's fine to use pickle for long-term storage,
> and that reading this data in new Python versions is not a "bad"
> thing to do. Am I missing something here?
>
> [1] http://docs.python.org/library/pickle.html#the-pickle-protocol

That was probably written before Python 3. Python 3 also dropped the
long-term backwards compatibilities for the language and stdlib. I am
certainly fine with adding a warning to the docs that this guarantee
does not apply to the Python 2/3 boundary. But I don't think we should
map 8-bit str instances from Python 2 to bytes in Python 3.

My snipe was mostly in reference to the many other things that can go
wrong with pickled data as your environment evolves -- if you're not
careful you can have references (by name) to modules, functions,
classes in pickled data that won't resolve in a later (or earlier!)
version of your app, or you might have objects that are unpickled in
an incomplete state that causes later use of the objects to break
(e.g. if a newer version of __init__() sets some extra instance
variables -- unpickling doesn't generally call __init__, so these new
variables won't be set if they didn't exist in the old version). Etc.,
etc.

If you can solve your problem with a suitably hacked Unpickler
subclass that's fine with me, but I would personally use this
opportunity to change the app to some other serialization format that
is perhaps less general but more robust than pickle. I've been bitten
by too many pickle-related problems to recommend pickle to anyone...

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unpickling py2 str as py3 bytes (and vice versa) - implementation (issue #6784)

[Nick Coghlan]

On Wed, Mar 14, 2012 at 8:08 AM, Guido van Rossum  wrote:
> If you can solve your problem with a suitably hacked Unpickler
> subclass that's fine with me, but I would personally use this
> opportunity to change the app to some other serialization format that
> is perhaps less general but more robust than pickle. I've been bitten
> by too many pickle-related problems to recommend pickle to anyone...

It's fine for in-memory storage of (almost) arbitrary objects (I use
it to stash things in a memory backed sqlite DB via SQLAlchemy) and
for IPC, but yeah, for long-term cross-version persistent storage, I'd
be looking to something like JSON rather than pickle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288)

[Kristján Valur Jónsson]

http://bugs.python.org/issue14288

Raymond suggested that this patch should be discussed here, so here goes:

How this came about:
There are frameworks, such as the Nagare web framework, 
(http://www.nagare.org/) that rely on suspending execution at some point and 
resuming it again.  Nagare does this using Stackless python, pickles the 
execution state of a tasklet, and resumes it later (possibly elsewhere).  Other 
frameworks are doing similar things in cloud computing.  I have seen such 
presentation at previous PyCons, and they have had to write their own picklers 
to get around these problems.

The problem is this:  While pickling execution state (frame objects, functions) 
might be considered exotic, and indeed Stackless has modifications unique to it 
to do it,  they quickly run into trouble that have nothing to do really with 
the fact that they are doing such exotic things.
For example, the fact that the very common dictiter is implemented in C and not 
python, necessitates that special pickle support is done added for that, 
otherwise only some context can be pickled, (those that are not currently 
iterating through a dict) and not others.

Now stackless has tried to provide this functionality for many years and indeed 
has special pickling support for dictiters, listiters, etc. (stuff that has 
nothing to do with the stacklessness of Stackless, really).  However, 
(somewhat) recently a lot of the itertools were moved into C.  Suddenly 
iterators, previously picklable (by merit of being in .py) stopped being that, 
just because they became C objects.  In addition, a bunch of other iterators 
started showing up (stringiter, bytesiter).  This started to cause problems.  
Suddenly you have to arbitrarily restrict what you can and can't do in code 
that is using these approaches.  For Stackless, (and Nagare), it was necessary 
to ban the usage of the _itertools module in web programs.

Instead of adding this to Stackless, and thus widening the gap between 
stackless and cpython, I think it is a good idea simply to fix this in cpython 
itself. Note that I also consider this to be of general utility to regular, 
non-exotic applications:  Why should an application, that is working with a 
bunch of data, but wants to stop that for a bit, and maybe save it out to disk, 
have to worry about transforming the data into valid primitive datastructures 
before doing so?

In my opinion, any objects that have simple and obvious pickle semantics should 
be picklable.  Iterators are just regular objects with some state.  They are 
not file pointers or sockets or database cursors.  And again, I argue that if 
these objects were implemented in .py, they would already be automatically 
picklable (indeed, itertools.py was).  The detail that some iterators in 
standard python are implemented in C should not automatically restrict their 
usage for no particular reason.

The patch is straightforward.  Most of it is tests, in fact. But it does use a 
few tricks in some places to get around the fact that some of those iterator 
types are hidden.   We  did try to be complete and find all the c iterators, 
but it was a year ago that the bulk of this work was done and something might 
have been added in the meantime.

Anyway, that's my pitch.
Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288)

[Jack Diederich]

2012/3/13 Kristján Valur Jónsson :
> http://bugs.python.org/issue14288

> In my opinion, any objects that have simple and obvious pickle semantics
> should be picklable.  Iterators are just regular objects with some state.
> They are not file pointers or sockets or database cursors.  And again, I
> argue that if these objects were implemented in .py, they would already be
> automatically picklable (indeed, itertools.py was).  The detail that some
> iterators in standard python are implemented in C should not automatically
> restrict their usage for no particular reason.

+1, things that can be pickled should be pickleable.

-Jack
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288)

[Antoine Pitrou]

On Tue, 13 Mar 2012 22:53:42 +
Kristján Valur Jónsson  wrote:
> http://bugs.python.org/issue14288
> 
> Raymond suggested that this patch should be discussed here, so here goes:

Sounds good on the principle.
Of course, the patch needs to be reviewed.

cheers

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] making python's c iterators picklable (http://bugs.python.org/issue14288)

[Christian Tismer]

On 3/13/12 4:05 PM, Antoine Pitrou wrote:

On Tue, 13 Mar 2012 22:53:42 +
Kristján Valur Jónsson  wrote:

http://bugs.python.org/issue14288

Raymond suggested that this patch should be discussed here, so here goes:

Sounds good on the principle.
Of course, the patch needs to be reviewed.


I am very much for this patch. Of course I am biased by my stackless
history and would be very happy to get most pickling into Python,
where it belongs.

To improve the patch a bit, I propose to put the hidden types into
some module. I agree with Kristjan that the types module would
be a good place to bootstrap the hidden iterable types.

I even would like to propose a PEP:
Whenever stuff is turned into C, which was picklable when implemented
in Python, or something new is implemented that makes sense to
pickle, a pickle implementation should always be provided.

cheers -- chris

--
Christian Tismer :^)
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key ->  http://pgp.uni-mainz.de
work +49 173 24 18 776  mobile +49 173 24 18 776  fax n.a.
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Steven D'Aprano]

Brian Curtin wrote:

On Tue, Mar 13, 2012 at 14:13, Kenneth Reitz  wrote:

I think the cheesehop trove classifiers would be the ideal way to
agnostically link to a page of packages related to the standard package in
question. No need for sort order.


Randomize the order for all I care. We still need to ensure we're
suggesting quality projects. It doesn't make sense for us to suggest
alternatives that we wouldn't want to use ourselves by just polling
some list that anyone can get on.


"Need" is awfully strong. I don't believe it is the responsibility of the 
standard library to be judge and reviewer of third party packages that it 
doesn't control.


-1 on adding *any* sort of recommendations about third-party software except, 
at most, a case-by-case basis where absolutely necessary.


What problem are we actually trying to solve here? Do we think that there are 
users who really have no clue where to find 3rd party software AND don't know 
how to use Google, BUT read the Python docs? I find it difficult to believe 
that there are people who both read the docs and are so clueless that they 
need to be told that there are alternatives available and which they should be 
using.


Personally I think this is a solution in search of a problem. Judging by the 
python-tutor mailing list, even *beginners* know that they aren't limited to 
the stdlib and how to go about finding third party software. There are many 
more questions about PyGame and wxPython than there are about tkinter. There 
are plenty of questions about numpy. There are lots of questions about niche 
packages I'd never even heard of.


I simply don't think there is any evidence that there are appreciable numbers 
of Python coders, beginners or expert, who need to be told about third party 
software. Who are these people we're trying to reach out to?




This is documentation that receives hundreds of thousands of views a
month*. We need to be picky about what goes in it.


Exactly why we should be wary of recommending specific packages.

Should we recommend wxPython over Pyjamas or PyGUI or PyGtk? On what basis?
Whatever choice we make is going to be wrong for some people, and is 
potentially unfair to the maintainers of the packages left out. Should we 
recommend them all? That's no help to anyone. Make no recommendation at all? 
That's the status quo.


What counts as "best of breed" can change rapidly -- software churn is part of 
the reason that the packages aren't in the stdlib in the first place. It can 
also be a matter of taste and convention. There are a few non-brainers, like 
numpy, but everything else, no, let's not open this can of worms.


I can see no benefit to this suggestion, and all sorts of ways that this might 
go badly.




--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Drop the new time.wallclock() function?

[Victor Stinner]

Hi,

I added two functions to the time module in Python 3.3: wallclock()
and monotonic(). I'm unable to explain the difference between these
two functions, even if I wrote them :-) wallclock() is suppose to be
more accurate than time() but has an unspecified starting point.
monotonic() is similar except that it is monotonic: it cannot go
backward. monotonic() may not be available or fail whereas wallclock()
is available/work, but I think that the two functions are redundant.

I prefer to keep only monotonic() because it is not affected by system
clock update and should help to fix issues on NTP update in functions
implementing a timeout.

What do you think?

--

monotonic() has 3 implementations:
* Windows: QueryPerformanceCounter() with QueryPerformanceFrequency()
* Mac OS X: mach_absolute_time() with mach_timebase_info()
* UNIX: clock_gettime(CLOCK_MONOTONIC_RAW) or clock_gettime(CLOCK_MONOTONIC)

wallclock() has 3 implementations:
* Windows: QueryPerformanceCounter() with QueryPerformanceFrequency(),
with a fallback to GetSystemTimeAsFileTime() if
QueryPerformanceFrequency() failed
* UNIX: clock_gettime(CLOCK_MONOTONIC_RAW),
clock_gettime(CLOCK_MONOTONIC) or clock_gettime(CLOCK_REALTIME), with
a fallback to gettimeofday() if clock_gettime(*) failed
* Otherwise: gettimeofday()

(wallclock should also use mach_absolute_time() on Mac OS X)


Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Michael Foord]

On 13 Mar 2012, at 16:57, Victor Stinner wrote:

> Hi,
> 
> I added two functions to the time module in Python 3.3: wallclock()
> and monotonic(). I'm unable to explain the difference between these
> two functions, even if I wrote them :-) wallclock() is suppose to be
> more accurate than time() but has an unspecified starting point.
> monotonic() is similar except that it is monotonic: it cannot go
> backward. monotonic() may not be available or fail whereas wallclock()
> is available/work, but I think that the two functions are redundant.
> 
> I prefer to keep only monotonic() because it is not affected by system
> clock update and should help to fix issues on NTP update in functions
> implementing a timeout.
> 
> What do you think?
> 


I am in the middle of adding a feature to unittest that involves timing of 
individual tests. I want the highest resolution cross platform measure of 
wallclock time - and time.wallclock() looked ideal. If monotonic may not exist 
or can fail why would that be better?

Michael


> --
> 
> monotonic() has 3 implementations:
> * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency()
> * Mac OS X: mach_absolute_time() with mach_timebase_info()
> * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW) or clock_gettime(CLOCK_MONOTONIC)
> 
> wallclock() has 3 implementations:
> * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency(),
> with a fallback to GetSystemTimeAsFileTime() if
> QueryPerformanceFrequency() failed
> * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW),
> clock_gettime(CLOCK_MONOTONIC) or clock_gettime(CLOCK_REALTIME), with
> a fallback to gettimeofday() if clock_gettime(*) failed
> * Otherwise: gettimeofday()
> 
> (wallclock should also use mach_absolute_time() on Mac OS X)
> 
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

So wallclock() falls back to a not-necessarily-monotonic time source
if necessary,
while monotonic() raises an exception in that case? ISTM that these
don't need to
be separate functions - rather, we can have one function that takes a
flag (called
require_monotonic, or something like that) telling it which failure mode to use.
Does that make sense?

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Guido van Rossum]

On Tue, Mar 13, 2012 at 4:57 PM, Victor Stinner
 wrote:
> I added two functions to the time module in Python 3.3: wallclock()
> and monotonic(). I'm unable to explain the difference between these
> two functions, even if I wrote them :-) wallclock() is suppose to be
> more accurate than time() but has an unspecified starting point.
> monotonic() is similar except that it is monotonic: it cannot go
> backward. monotonic() may not be available or fail whereas wallclock()
> is available/work, but I think that the two functions are redundant.
>
> I prefer to keep only monotonic() because it is not affected by system
> clock update and should help to fix issues on NTP update in functions
> implementing a timeout.
>
> What do you think?

I think wallclock() is an awkward name; in other contexts I've seen
"wall clock time" used to mean the time that a clock on the wall would
show, i.e. local time. This matches definition #1 of
http://www.catb.org/jargon/html/W/wall-time.html (while yours matches
#2 :-).

I agree that it's better to have only one of these. I also think if we
offer it we should always have it -- if none of the implementations
are available, I guess you could fall back on returning time.time(),
with some suitable offset so people don't think it is always the same.
Maybe it could be called realtime()?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] sharing sockets among processes on windows

[Kristján Valur Jónsson]

Hi,
I´m interested in contributing a patch to duplicate sockets between processes 
on windows.
Tha api to do this is WSADuplicateSocket/WSASocket(), as already used by dup() 
in the _socketmodule.c
Here´s what I have:

1)  Sockets have a method, duplicate(target_pid), that return a bytes 
object containing the socket info for the target process

2)  When socket(x, y, z, data) is called with this bytes object as the 
fourth argument, it is recreated from that.

What are your thoughts on this?  Also, should I try to reuse the socket.dup() 
function somehow, perhaps by giving the target pid?
Secondly, there is multiprocessing.reduction which is doing similar things for 
unix.  Does anyone familiar with it know how it goes about doing this?  Would 
it be simple to change it to use this mechanism on windows?

Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Victor Stinner]

On 14/03/2012 01:18, Nadeem Vawda wrote:

So wallclock() falls back to a not-necessarily-monotonic time source
if necessary,
while monotonic() raises an exception in that case? ISTM that these
don't need to
be separate functions - rather, we can have one function that takes a
flag (called
require_monotonic, or something like that) telling it which failure mode to use.
Does that make sense?


I don't think that time.monotonic() can fail in practice and it is 
available for all modern platforms (Windows, Mac OS X and OS implemented 
clock_gettime()).


On Windows, time.monotonic() fails with an OSError if 
QueryPerformanceFrequency() failed. QueryPerformanceFrequency() can fail 
if "the installed hardware does not support a high-resolution 
performance counter" according to Microsoft documentation. Windows uses 
the CPU RDTSC instruction, or the ACPI power management timer or event 
the old 8245 PIT. I think that you have at least one of this device on 
your computer.


I suppose that you can use a manual fallback to time.time() if 
time.monotonic() failed. If time.monotonic() fails, it fails directly at 
the first call. Example of a fallback working with Python < 3.3:


try:
   time.monotonic()
except (OSError, AttributeError):
   get_time = time.time
else:
   get_time = time.monotonic

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Kristján Valur Jónsson]

The reason I originally suggested "wallclock" was because that term is often 
used to distinguish time measurements (delta) that show real world time from 
those showing CPU or Kernel time.  "number.crunch() took 2 seconds wallclock 
time but only 1 second CPU!".  The original problem was that time.clock() was 
"wallclock" on some platforms but "cpu" on others, IIRC.
But monotonic is probably even better.  I agree removing one or the other, 
probably wallclock.
K

-Original Message-
From: python-dev-bounces+kristjan=ccpgames@python.org 
[mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf Of Guido 
van Rossum
Sent: 13. mars 2012 17:27
To: Victor Stinner
Cc: Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

I think wallclock() is an awkward name; in other contexts I've seen "wall clock 
time" used to mean the time that a clock on the wall would show, i.e. local 
time. This matches definition #1 of 
http://www.catb.org/jargon/html/W/wall-time.html (while yours matches
#2 :-).

I agree that it's better to have only one of these. I also think if we offer it 
we should always have it -- if none of the implementations are available, I 
guess you could fall back on returning time.time(), with some suitable offset 
so people don't think it is always the same.
Maybe it could be called realtime()?



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Victor Stinner]

> I agree that it's better to have only one of these. I also think if we
> offer it we should always have it -- if none of the implementations
> are available, I guess you could fall back on returning time.time(),
> with some suitable offset so people don't think it is always the same.
> Maybe it could be called realtime()?

For a concrete use case, see for example:
http://bugs.python.org/issue14222

I just wrote two patches, for the queue and threading modules, using
time.monotonic() if available, with a fallback to time.time(). My
patches call time.monotonic() to ensure that it doesn't fail with
OSError.

I suppose that most libraries and programs will have to implement a
similar fallback.

We may merge both functions with a flag to be able to disable the
fallback. Example:

 - time.realtime(): best-effort monotonic, with a fallback
 - time.realtime(monotonic=True): monotonic, may raise OSError or
NotImplementedError

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Christian Tismer]

On 3/13/12 5:45 PM, Kristján Valur Jónsson wrote:

The reason I originally suggested "wallclock" was because that term is often used to distinguish time 
measurements (delta) that show real world time from those showing CPU or Kernel time.  "number.crunch() took 2 
seconds wallclock time but only 1 second CPU!".  The original problem was that time.clock() was 
"wallclock" on some platforms but "cpu" on others, IIRC.
But monotonic is probably even better.  I agree removing one or the other, 
probably wallclock.
K

-Original Message-
From: python-dev-bounces+kristjan=ccpgames@python.org 
[mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf Of Guido 
van Rossum
Sent: 13. mars 2012 17:27
To: Victor Stinner
Cc: Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

I think wallclock() is an awkward name; in other contexts I've seen "wall clock 
time" used to mean the time that a clock on the wall would show, i.e. local time. 
This matches definition #1 of http://www.catb.org/jargon/html/W/wall-time.html (while 
yours matches
#2 :-).

I agree that it's better to have only one of these. I also think if we offer it 
we should always have it -- if none of the implementations are available, I 
guess you could fall back on returning time.time(), with some suitable offset 
so people don't think it is always the same.
Maybe it could be called realtime()?



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/tismer%40stackless.com


Btw., have you considered virtual machines?
I happen to run windows in Parallels or Virtualbox quite often.
There the "wall clock" stuff notoriously does not work.

It would be good (but difficult?) if the supposed-to-be-accurate
clock could test itself, if it works at all, and replace itself
with a fallback.

In my case, this causes quite a few PyPy tests to fail ;-)

ciao -- Chris

--
Christian Tismer :^)
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key ->  http://pgp.uni-mainz.de
work +49 173 24 18 776  mobile +49 173 24 18 776  fax n.a.
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Yury Selivanov]

If we need to decide to which function should be kept - I vote for monotonic.
It's extremely useful (even essential) to track timeouts in various schedulers 
implementations, for example.  Quick search also shows the demand for it, as
there are questions on stackoverflow.com and few packages on PyPI.

-
Yury


On 2012-03-13, at 9:03 PM, Victor Stinner wrote:

>> I agree that it's better to have only one of these. I also think if we
>> offer it we should always have it -- if none of the implementations
>> are available, I guess you could fall back on returning time.time(),
>> with some suitable offset so people don't think it is always the same.
>> Maybe it could be called realtime()?
> 
> For a concrete use case, see for example:
> http://bugs.python.org/issue14222
> 
> I just wrote two patches, for the queue and threading modules, using
> time.monotonic() if available, with a fallback to time.time(). My
> patches call time.monotonic() to ensure that it doesn't fail with
> OSError.
> 
> I suppose that most libraries and programs will have to implement a
> similar fallback.
> 
> We may merge both functions with a flag to be able to disable the
> fallback. Example:
> 
> - time.realtime(): best-effort monotonic, with a fallback
> - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Nadeem Vawda]

On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner
 wrote:
> I suppose that most libraries and programs will have to implement a
> similar fallback.
>
> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
>  - time.realtime(): best-effort monotonic, with a fallback
>  - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError

This was my suggestion - I think it's useful to have the fallback
available (since most users will want it), but at the same time we
should also cater to users who need a clock that is *guaranteed* to
be monotonic.

As an aside, I think "monotonic" is a better name than "realtime";
it conveys the functions purpose more clearly. Then we could call
the flag "strict".

Cheers,
Nadeem
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Andrew Svetlov]

On Tue, Mar 13, 2012 at 5:29 PM, Victor Stinner
 wrote:
> I suppose that you can use a manual fallback to time.time() if
> time.monotonic() failed. If time.monotonic() fails, it fails directly at the
> first call. Example of a fallback working with Python < 3.3:
>
> try:
>   time.monotonic()
> except (OSError, AttributeError):
>   get_time = time.time
> else:
>   get_time = time.monotonic
>

I like 'fallback' solution while `get_time` is not the best name for
high precision timer from my perspective.
Can you call it `monotonic` or `realtime`?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Kristján Valur Jónsson]

Interesting thought.
Althougn I don't see how that could fail on windows, if the QPC function is 
really just talking to a clock chip, surely that hasn't been virtualized.
Is there an actual example of windows hardware where this api fails (virtual or 
not?)  Perhaps there is no real need to have a fallback mechanism, and it would 
even be best to write such a mechanism inside the function itself, and just 
return getsystemtimeasfiletime() instead.

K

-Original Message-
From: Christian Tismer [mailto:tis...@stackless.com] 
Sent: 13. mars 2012 18:07
To: Kristján Valur Jónsson
Cc: Guido van Rossum; Victor Stinner; Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

Btw., have you considered virtual machines?
I happen to run windows in Parallels or Virtualbox quite often.
There the "wall clock" stuff notoriously does not work.

It would be good (but difficult?) if the supposed-to-be-accurate clock could 
test itself, if it works at all, and replace itself with a fallback.

In my case, this causes quite a few PyPy tests to fail ;-)

ciao -- Chris

-- 
Christian Tismer :^)
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key ->  http://pgp.uni-mainz.de
work +49 173 24 18 776  mobile +49 173 24 18 776  fax n.a.
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
   whom do you want to sponsor today?   http://www.stackless.com/



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Guido van Rossum]

On Tue, Mar 13, 2012 at 6:03 PM, Victor Stinner
 wrote:
>> I agree that it's better to have only one of these. I also think if we
>> offer it we should always have it -- if none of the implementations
>> are available, I guess you could fall back on returning time.time(),
>> with some suitable offset so people don't think it is always the same.
>> Maybe it could be called realtime()?
>
> For a concrete use case, see for example:
> http://bugs.python.org/issue14222
>
> I just wrote two patches, for the queue and threading modules, using
> time.monotonic() if available, with a fallback to time.time(). My
> patches call time.monotonic() to ensure that it doesn't fail with
> OSError.
>
> I suppose that most libraries and programs will have to implement a
> similar fallback.

It seems horrible to force everyone to copy the same silly block of
code. The time module itself should implement this once.

> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
>  - time.realtime(): best-effort monotonic, with a fallback
>  - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError

I have no opinions on this or other API details. But please make the
function always exist and return something vaguely resembling a
monotonic real-time clock. (BTW IMO the docs should state explicitly
that it returns a float.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Martin v. Löwis]

1) The layout for the python root directory for all platforms should be
as follows:

stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}

[...]

I have talked with a number of people at PyCon, including Tarek and MvL.
Nobody objected, and several thought it was a good idea.


I admit that I didn't understand that lib/python{version} was
also part of the proposal. I'm fine with the bin/ change, but
skeptical about the lib change - this just adds a redundant level
of directories on Windows. The installation will end up in

c:\python33\lib\python3.3

which has the software name and version twice in the path.

Do we *really* need this?

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On Mar 13, 2012, at 8:37 PM, "Martin v. Löwis"  wrote:

>> 1) The layout for the python root directory for all platforms should be
>> as follows:
>> 
>> stdlib = {base/userbase}/lib/python{py_version_short}
>> platstdlib = {base/userbase}/lib/python{py_version_short}
>> purelib = {base/userbase}/lib/python{py_version_short}/site-packages
>> platlib = {base/userbase}/lib/python{py_version_short}/site-packages
>> include = {base/userbase}/include/python{py_version_short}
>> scripts = {base/userbase}/bin
>> data = {base/userbase}
> [...]
>> I have talked with a number of people at PyCon, including Tarek and MvL.
>> Nobody objected, and several thought it was a good idea.
> 
> I admit that I didn't understand that lib/python{version} was
> also part of the proposal. I'm fine with the bin/ change, but
> skeptical about the lib change - this just adds a redundant level
> of directories on Windows. The installation will end up in
> 
> c:\python33\lib\python3.3
> 
> which has the software name and version twice in the path.
> 
> Do we *really* need this?


We *already* have this. The only difference in this proposal is that we go from 
py_version_nodot to py_version_short, i.e. from

c:\python33\lib\python33 

to

c:\python33\lib\python3.3

Given that we already repeat it, isn't it better to be consistent?

Thanks,

Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Terry Reedy]

On 3/13/2012 9:57 PM, VanL wrote:

On Mar 13, 2012, at 8:37 PM, "Martin v. Löwis"
wrote:


1) The layout for the python root directory for all platforms
should be as follows:

stdlib = {base/userbase}/lib/python{py_version_short} platstdlib
= {base/userbase}/lib/python{py_version_short} purelib =
{base/userbase}/lib/python{py_version_short}/site-packages
platlib =
{base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin data = {base/userbase}

[...]

I have talked with a number of people at PyCon, including Tarek
and MvL. Nobody objected, and several thought it was a good
idea.


I admit that I didn't understand that lib/python{version} was also
part of the proposal. I'm fine with the bin/ change, but skeptical
about the lib change - this just adds a redundant level of
directories on Windows. The installation will end up in

c:\python33\lib\python3.3

which has the software name and version twice in the path.

Do we *really* need this?



We *already* have this. The only difference in this proposal is that
we go from py_version_nodot to py_version_short, i.e. from

c:\python33\lib\python33


Right not, we (at least I) have
.../python33/Lib/
.../python32/Lib/


to

c:\python33\lib\python3.3

Given that we already repeat it, isn't it better to be consistent?


But there is no repetition currently on Windows installations.
I though you were just proposing to switch lib (lower-cased, and scripts 
renamed as bin, and pythonxx). So I do not think I yet understand what 
the proposal is and how it would be different from what I have now.


--



Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Matt Joiner]

On Mar 14, 2012 5:27 AM, "Antoine Pitrou"  wrote:
>
> On Tue, 13 Mar 2012 14:16:40 -0700
> Guido van Rossum  wrote:
>
> > On Tue, Mar 13, 2012 at 12:49 PM, Terry Reedy  wrote:
> > > Authors of separately maintained packages are, from our viewpoint, as
> > > eligible to help with tracker issues as anyone else, even while they
> > > continue work on their external package. Some of them are more likely
than
> > > most contributors to have the knowledge needed for some particular
issues.
> >
> > This is a good idea. I was chatting w. Senthil this morning about
> > adding improvements to urllib/request.py based upon ideas from
> > urllib3, requests, httplib2 (?), and we came to the conclusion that it
> > might be a good idea to let those packages' authors review the
> > proposed stdlib improvements.
>
> We don't have any provisions against reviewal by third-party
> developers already. I think the main problem (for us, of course) is that
> these people generally aren't interested enough to really dive in
> stdlib patches and proposals.
>
> For example, for the ssl module, I have sometimes tried to involve
> authors of third-party packages such as pyOpenSSL (or, IIRC, M2Crypto),
> but I got very little or no reviewing.

Rather than indicating apathy on the party of third party developers, this
might be a sign that core Python is unapproachable or not worth the effort.

For instance I have several one line patches languishing, I can't imagine
how disappointing it would be to have significantly larger patches ignored,
but it happens.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Eli Bendersky]

> Rather than indicating apathy on the party of third party developers, this
> might be a sign that core Python is unapproachable or not worth the effort.
>
> For instance I have several one line patches languishing, I can't imagine
> how disappointing it would be to have significantly larger patches ignored,
> but it happens.
>

A one-line patch for a complex module or piece of code may require
much more than looking at that single line to really review. I hope
you understand that.

That said, if you find any issues in the bug tracker that in your
opinion need only a few minutes of attention from a core developer,
feel free to send a note to the mentorship mailing list. People
sometimes come there asking for precisely this thing (help reviewing a
simple patch they submitted), and usually get help quickly if their
request is justified.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] SocketServer issues

[Kristján Valur Jónsson]

Hi there.
I want to mention some issues I've had with the socketserver module, and 
discuss if there's a way to make it nicer.
So, for a long time we were able to create magic stackless mixin classes for 
it, like ThreadingMixIn, and assuming we had the appropriate socket replacement 
library, be able to use it nicely using tasklets.
Then, at some point, the run_forever loop was changed to support timeout 
through the use of select.select() before every socket.accept() call.  This was 
very awkward because the whole concept of select() really goes contrary to the 
approach of using microthreads, non-blocking IO and all that.
The only reason for this select call, was to support timeout for the accept.  
And even for vanilla applications, it necessiates an extra kernel call for 
every accept, just for the timeout.

The way around this for me has to do local modifications to the SocketServer 
and just get rid of the select.

So, my first question is:  Why not simply rely on the already built-in timeout 
support in the socket module?  Setting the correct timeout value on the 
accepting socket, will achieve the same thing.  Of course, one then has to 
reset the timeout value on the accepted socket, but this is minor.

Second: Of late the SocketServer has grown additional features and attributes.  
In particular, it now has two event objects, __shutdown_request and 
__is_shut_down.
Notice the double underscores.
They make it impossible to subclass the SocketServer class to provide a 
different implementation of run_forever().  Is there any good reason why these 
attributes have been made "private" like this?  Having just seen Raymond's talk 
on how to subclass right, this looks like the wrong way to use the double 
leading underscores.

So, two things really:
The use of select.select in SocketServer makes it necessary to subclass it to 
write a new version of run_forever() for those that wish to use a non-blocking 
IO library instead of socket. And the presence of these private attributes make 
it (theoretically) impossible to specialize run_forever in a mix-in class.

Any thoughs?  Is anyone interested in seeing how the timeouts can be done 
without using select.select()?  And what do you think about removing the double 
underscores from there and thus making serve_forever owerrideable?

Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Christian Tismer]

The performancecounter is a thing that typically gets intercepted by
the VM infrastructure and does no longer work as a reliable timing
source. In PyPy there are tests which check certain assumptions
how much the performancecounter must advance at least between
a few opcodes, and that does not work in a VM.

cheers - Chris

On 3/13/12 6:11 PM, Kristján Valur Jónsson wrote:

Interesting thought.
Althougn I don't see how that could fail on windows, if the QPC function is 
really just talking to a clock chip, surely that hasn't been virtualized.
Is there an actual example of windows hardware where this api fails (virtual or 
not?)  Perhaps there is no real need to have a fallback mechanism, and it would 
even be best to write such a mechanism inside the function itself, and just 
return getsystemtimeasfiletime() instead.

K

-Original Message-
From: Christian Tismer [mailto:tis...@stackless.com]
Sent: 13. mars 2012 18:07
To: Kristján Valur Jónsson
Cc: Guido van Rossum; Victor Stinner; Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

Btw., have you considered virtual machines?
I happen to run windows in Parallels or Virtualbox quite often.
There the "wall clock" stuff notoriously does not work.

It would be good (but difficult?) if the supposed-to-be-accurate clock could 
test itself, if it works at all, and replace itself with a fallback.

In my case, this causes quite a few PyPy tests to fail ;-)

ciao -- Chris




--
Christian Tismer :^)
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key ->  http://pgp.uni-mainz.de
work +49 173 24 18 776  mobile +49 173 24 18 776  fax n.a.
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] getting patches committed (was Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives)

[R. David Murray]

On Wed, 14 Mar 2012 06:03:10 +0200, Eli Bendersky  wrote:
> > Rather than indicating apathy on the party of third party developers, this
> > might be a sign that core Python is unapproachable or not worth the effort.
> >
> > For instance I have several one line patches languishing, I can't imagine
> > how disappointing it would be to have significantly larger patches ignored,
> > but it happens.
> 
> A one-line patch for a complex module or piece of code may require
> much more than looking at that single line to really review. I hope
> you understand that.

In addition, sometimes patches just get forgotten.  It's not like
there are enough core devs with enough time that we are actually doing
searches for open issues with patches...generally we have enough to do
in our interest areas, and so stay there unless an issue is brought to
our attention.

So to bring an issue to our attention, you can first ping the issue with
a status query, or get someone (anyone, pretty much) to do a review and
post it to the issue.  You can also look to see if you can figure out,
either from the experts list in the devguide, or hg history, or tracker
activity, who might be a reasonable person to look at the issue, and
add them to the nosy list.

Either of these actions will often "wake up" an issue, and if it is not
one of the complex (or controversial) ones Eli alluded to, it will often
then get committed.

If that fails, and the patch has been on the tracker for a while, it is
perfectly reasonable to ask about it here.

What we really need most are *reviews*.  And we need these for two
reasons.

First, there aren't enough active committers to keep up with the patch
inflow.  Reviews really help, because they usually simplify the commit
review process for the committer, saving time, and making it more
appealing to work on the issue.

Second, it is as much (or more) from quality reviews as quality patches
that we recognize people who it would be beneficial to invite to be
committers.  And every new committer increases the chances that new
patches will actually get committed

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Jeffrey Yasskin]

On Tue, Mar 13, 2012 at 5:03 PM, Michael Foord
 wrote:
>
> On 13 Mar 2012, at 16:57, Victor Stinner wrote:
>
>> Hi,
>>
>> I added two functions to the time module in Python 3.3: wallclock()
>> and monotonic(). I'm unable to explain the difference between these
>> two functions, even if I wrote them :-) wallclock() is suppose to be
>> more accurate than time() but has an unspecified starting point.
>> monotonic() is similar except that it is monotonic: it cannot go
>> backward. monotonic() may not be available or fail whereas wallclock()
>> is available/work, but I think that the two functions are redundant.
>>
>> I prefer to keep only monotonic() because it is not affected by system
>> clock update and should help to fix issues on NTP update in functions
>> implementing a timeout.
>>
>> What do you think?
>>
>
>
> I am in the middle of adding a feature to unittest that involves timing of 
> individual tests. I want the highest resolution cross platform measure of 
> wallclock time - and time.wallclock() looked ideal. If monotonic may not 
> exist or can fail why would that be better?
>

Isn't the highest resolution cross platform measure of "wallclock"
time spelled "time.clock()"? Its docs say "this is the function to use
for benchmarking Python or timing algorithms", and it would be a shame
to add and teach a new function rather than improving clock()'s
definition.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Docs of weak stdlib modules should encourage exploration of 3rd-party alternatives

[Matt Joiner]

Thanks for the suggestions.
On Mar 14, 2012 12:03 PM, "Eli Bendersky"  wrote:

> > Rather than indicating apathy on the party of third party developers,
> this
> > might be a sign that core Python is unapproachable or not worth the
> effort.
> >
> > For instance I have several one line patches languishing, I can't imagine
> > how disappointing it would be to have significantly larger patches
> ignored,
> > but it happens.
> >
>
> A one-line patch for a complex module or piece of code may require
> much more than looking at that single line to really review. I hope
> you understand that.
>
> That said, if you find any issues in the bug tracker that in your
> opinion need only a few minutes of attention from a core developer,
> feel free to send a note to the mentorship mailing list. People
> sometimes come there asking for precisely this thing (help reviewing a
> simple patch they submitted), and usually get help quickly if their
> request is justified.
>
> Eli
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Jeffrey Yasskin]

On Tue, Mar 13, 2012 at 6:10 PM, Nadeem Vawda  wrote:
> On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner
>  wrote:
>> I suppose that most libraries and programs will have to implement a
>> similar fallback.
>>
>> We may merge both functions with a flag to be able to disable the
>> fallback. Example:
>>
>>  - time.realtime(): best-effort monotonic, with a fallback
>>  - time.realtime(monotonic=True): monotonic, may raise OSError or
>> NotImplementedError
>
> This was my suggestion - I think it's useful to have the fallback
> available (since most users will want it), but at the same time we
> should also cater to users who need a clock that is *guaranteed* to
> be monotonic.
>
> As an aside, I think "monotonic" is a better name than "realtime";
> it conveys the functions purpose more clearly. Then we could call
> the flag "strict".

While you're bikeshedding:

Some of the drafts of the new C++ standard had a monotonic_clock,
which was guaranteed to only go forwards, but which could be affected
by system clock updates that went forwards. Because of problems in
defining timeouts using an adjustable clock, C++11 instead defines a
"steady_clock", which ticks as steadily as the machine/OS/library can
ensure, and is definitely not affected by any time adjustments:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3191.htm. I
realize that Unix's clock_gettime(CLOCK_MONOTONIC) already includes
the steadiness criterion, but the word itself doesn't actually include
the meaning.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 14/03/2012 6:43 AM, VanL wrote:

Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows,
with a side order of making the PATH work better.

Short version:

1) The layout for the python root directory for all platforms should be
as follows:

stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages

> include = {base/userbase}/include/python{py_version_short}

As per comments later in the thread, I'm -1 on including 
"python{py_version_short}" in the lib directories for a number of 
reasons; one further reason not outlined is that it would potentially 
make running Python directly from a built tree difficult.  For the same 
reason, I'm also -1 on having that in the include dir.



scripts = {base/userbase}/bin


We should note that this may cause pain for a number of projects - I've 
seen quite a few projects that already assume "Scripts" on Windows - eg, 
virtualenv and setuptools IIRC - and also assume the executable is where 
it currently lives - one example off the top of my head is the mozilla 
"jetpack" project - see:


https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117

This code (and any other code looking in "Scripts" on Windows) will fail 
and need to be updated with this change.  Further, assuming such 
projects want to target multiple Python versions, it will need to keep 
the old code and check the new location.


Another bit of code which would be impacted is the PEP397 launcher; it 
too would have to grow version specific logic to locate the executable.


So while I'm not (yet) -1 on the general idea, I'm close.  I guess I 
don't understand how the benefits this offers outweigh the costs to 3rd 
parties.  Given the work on making Python more virtualenv friendly, 
can't we just wear the costs of the existing scheme in the stdlib and 
avoid breaking the code already out there?  IOW, who exactly will 
benefit from this, and how does the cost of them supporting the existing 
scheme compare to the cost of the breakage to multiple 3rd parties?


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drop the new time.wallclock() function?

[Kristján Valur Jónsson]

To quote:
"On Unix, return the current processor time as a floating point number 
expressed in seconds.  The precision, and in fact the very definition of the 
meaning of "processor time", depends on that of the C function of the same 
name,"

The problem is that it is defined to return "processor time."  This is 
historical baggage that comes from just writing a python wrapper around the 
unix "clock" function.  Of course, "processor time" is quite useless when one 
is trying to write timeout algorithms or other such things that need to time 
out in real time, not just cpu cycles.
K

-Original Message-
From: python-dev-bounces+kristjan=ccpgames@python.org 
[mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf Of 
Jeffrey Yasskin
Sent: 13. mars 2012 22:42
To: Michael Foord
Cc: Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

Isn't the highest resolution cross platform measure of "wallclock"
time spelled "time.clock()"? Its docs say "this is the function to use for 
benchmarking Python or timing algorithms", and it would be a shame to add and 
teach a new function rather than improving clock()'s definition.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com