Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)

[Random832]

On Fri, Oct 13, 2017, at 02:30, Nick Coghlan wrote:
> Metaclasses currently tend to serve two distinct purposes:
>
> 1. Actually altering the runtime behaviour of a class and its children
>in non-standard ways (e.g. enums, ABCs, ORMs)
> 2. Boilerplate reduction in class definitions, reducing the amount of
>code you need to write as the author of that class
>
> Nobody has a problem with using metaclasses for the first purpose -
> that's what they're for.
>
> It's the second use case where they're problematic, as the fact that
> they're preserved on the class becomes a leaky implementation detail,
> and the lack of a JIT in CPython means they can also end up being
> expensive from a runtime performance perspective.

What about a metaclass that isn't a metaclass? A metaclass can be any
callable and can return any object, e.g. a normal type.

def AutoSlotMeta(name, bases, dct, real_metaclass=type):
"""turn all class variables into slots"""
dct['__slots__'] = list(dct)
return real_metaclass(name, bases, dct)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Random832]

On Wed, Oct 25, 2017, at 16:32, Alexander Belopolsky wrote:
> This is annoying when you deal with time
> series where it is common to have text files with a mix of dates,
> timestamps and numbers.  You can write generic code to deal with ints
> and floats, but have to special-case anything time related.

Generic code that uses a Callable[[str], ...] instead of a type works
fine with a class method.

column1.parser = int
column2.parser = float
column3.parser = datetime.parse_iso
column4.parser = json.loads

It is *very slightly* more complex than a model that needs the type also
for some reason and has the type pull double duty as the parser... but
why do that?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Random832]

On Tue, Nov 7, 2017, at 09:56, Steven D'Aprano wrote:
> Don't let the perfect be the enemy of the good.
> 
> For many applications, keys are never removed from the dict, so this 
> doesn't matter. If you never delete a key, then the remaining keys will 
> never be reordered.
> 
> I think that Nick's intent was not to say that after a single deletion, 
> the ordering guarantee goes away "forever", but that a deletion may be 
> permitted to reorder the keys, after which further additions will honour 
> insertion order. At least, that's how I interpret him.

Honestly, I think the more intuitive way would be the "other way around"
- deletions don't themselves change the order, but they do cause
subsequent insertions to be allowed to insert at places other than the
end.

Does the implementation in CPython have this property?

One way I have seen this done is that the items themselves live in an
array, and each new item is always inserted in the first empty slot in
the array (empty slots form a freelist). The hash buckets contain
indices into the array.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Random832]

On Tue, Nov 7, 2017, at 07:22, Nick Coghlan wrote:
> My suggestion for that definition is to have the *default* meaning of
> "third party code" be "everything that isn't __main__".

What is __main__? Or, rather, how do you determine when it is to blame?
For syntax it's easy, but any deprecated function necessarily belongs to
its own module and not to main. Main may have called it, which can be
detected from the stack trace, or it may have used it in some other way
(pass to some builtin or e.g. itertools function that takes a callable
argument, for example). Maybe the DeprecationWarning should be raised at
the name lookup* rather than the call? What if "calling this function
with some particular combination of arguments" is deprecated?


*i.e. something like:

class deprecated:
   def __init__(self, obj): self.obj = obj
class DeprecatableModule(ModuleType):
   def __getattr__(self, name):
  obj = self.__dict__[name]
  if isinstance(type(obj), deprecated):
  if (detect somehow caller is __main__): raise
  DeprecationWarning
  return obj.obj
  else: return obj
def __dir__(self):
return [k for k in self.__dict__ if not
isinstance(self.__dict__[k], deprecated)]
  
sys.modules[__name__].type=DeprecatableModule

@deprecated
def some_deprecated_function(...): ...

SOME_DEPRECATED_CONSTANT = deprecated(42)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Random832]

On Tue, Nov 28, 2017, at 15:31, Raymond Hettinger wrote:
> 
> > I also cc python-dev to see if anybody here is strongly in favor or against 
> > this inclusion.
> 
> Put me down for a strong -1.   The proposal would occasionally save a few
> keystokes but comes at the expense of giving Python a more Perlish look
> and a more arcane feel.   
> 
> One of the things I like about Python is that I can walk non-programmers
> through the code and explain what it does.  The examples in PEP 505 look
> like a step in the wrong direction.  They don't "look like Python" and
> make me feel like I have to decrypt the code to figure-out what it does.
> 
> timeout ?? local_timeout ?? global_timeout
> 'foo' in (None ?? ['foo', 'bar'])
> requested_quantity ?? default_quantity * price
> name?.strip()[4:].upper()
> user?.first_name.upper()

Since we're looking at different syntax for the ?? operator, I have a
suggestion for the ?. operator - and related ?[] and ?() that appeared
in some of the proposals. How about this approach?

Something like (or None: ...) as a syntax block in which any operation
[lexically within the expression, not within e.g. called functions, so
it's different from simply catching AttributeError etc, even if that
could be limited to only catching when the operand is None] on None that
is not valid for None will yield None instead.

This isn't *entirely* equivalent, but offers finer control.

v = name?.strip()[4:].upper() under the old proposal would be more or
less equivalent to:

v = name.strip()[4:].upper() if name is not None else None

Whereas, you could get the same result with:
(or None: name.strip()[4:].upper())

Though that would technically be equivalent to these steps:
v = name.strip if name is not None else None
v = v() if v "
v = v[4:] """
v = v.upper """
v = v() """

The compiler could optimize this case since it knows none of the
operations are valid on None. This has the advantage of being explicit
about what scope the modified rules apply to, rather than simply
implicitly being "to the end of the chain of dot/bracket/call operators"

It could also be extended to apply, without any additional syntax, to
binary operators (result is None if either operand is None) (or None: a
+ b), for example, could return None if either a or b is none.

[I think I proposed this before with the syntax ?(...), the (or None:
...) is just an idea to make it look more like Python.]
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Random832]

On Thu, Nov 30, 2017, at 11:08, Eric Fahlgren wrote:
> On Thu, Nov 30, 2017 at 2:48 AM, Andrea Griffini  wrote:
> 
> > Not really related but the PEP says that arguments in Python are evaluated
> > before the function (as a reason to reject the idea of None-aware function
> > call) but this is not the case:
> >
> 
> ​I think you're missing something here, since it seems clear to me that
> indeed the arguments are evaluated prior to the function call.​  Maybe
> unrolling it would help?  This is equivalent to the body of your lambda,
> and you can see that the argument is evaluated prior to the call which
> receives it.

Of course they're evaluated prior to the function *call*, but the pep
says they're evaluated prior to the function *itself* [i.e. arg = g();
func = f(); func(arg)].
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Random832]

On Thu, Nov 30, 2017, at 17:26, Greg Ewing wrote:
> Eric Fahlgren wrote:
> > ​I think you're missing something here, since it seems clear to me that 
> > indeed the arguments are evaluated prior to the function call.​
> 
> I think the OP may be confusing "evaluating the function" with
> "calling the function".
> 
> If the function being called is determined by some computation,
> that computation may be performed before its arguments are
> evaluated (and is probably required to be, by the "left to
> right" rule). But the arguments will always be evaluated
> before the actual call happens.

Right, but if the function is evaluated before either of those things
happen, then it can indeed short-circuit the argument evaluation.

The OP isn't confusing anything; it's Eric who is confused. The quoted
paragraph of the PEP clearly and unambiguously claims that the sequence
is "arguments -> function -> call", meaning that something happens after
the "function" stage [i.e. a None check] cannot short-circuit the
"arguments" stage. But in fact the sequence is "function -> arguments ->
call".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Random832]

On Fri, Dec 1, 2017, at 05:31, Steven D'Aprano wrote:
> I'm more confused than ever. You seem to be arguing that Python 
> functions CAN short-circuit their arguments and avoid evaluating them. 
> Is that the case?

> If this is merely about when the name "function" is looked up, then I 
> don't see why that's relevant to the PEP.
> 
> What am I missing?

You're completely missing the context of the discussion, which was the
supposed reason that a *new* function call operator, with the proposed
syntax function?(args), that would short-circuit (based on the
'function' being None) could not be implemented. The whole thing doesn't
make sense to me anyway, since a new operator could have its own
sequence different from the existing one if necessary.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?

[Random832]

On Sat, Dec 16, 2017, at 08:22, Antoine Pitrou wrote:
> Typically, when adding a tp_XXX slot, you also need to add a
> Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures
> that have been compiled against a recent enough PyTypeObject
> definition.  This way, extensions compiled against Python N-1 are
> supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set,
> the core Python runtime won't try to access the (non-existing) tp_XXX
> member.

Is there any practical for of having the flag off for one slot and on
for another slot that's been added later?

Could this be replaced (that is, a slot for such a thing added before
it's too late) with a simple counter that goes up with each version, and
any "unused" slot should have NULL or some other sentinel value? If it
really is important to have the flags themselves, just add another set
of flags - Py_TPFLAGS_HAVE_MORE_FLAGS.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?

[Random832]

On Mon, Dec 18, 2017, at 05:23, Antoine Pitrou wrote:
> On Sun, 17 Dec 2017 21:07:02 -0500
> Random832  wrote:
> > 
> > Is there any practical for of having the flag off for one slot and on
> > for another slot that's been added later?
> > 
> > Could this be replaced (that is, a slot for such a thing added before
> > it's too late) with a simple counter that goes up with each version, and
> > any "unused" slot should have NULL or some other sentinel value?
> 
> Any replacement here would break binary compatibility, which is what
> those flags are precisely meant to avoid.

I meant replacing the mechanism for new fields, rather than existing ones.

> > If it
> > really is important to have the flags themselves, just add another set
> > of flags - Py_TPFLAGS_HAVE_MORE_FLAGS.
> 
> Yes, we could... but it's more complication again.

Hmm, maybe that could be eased with macros...


/* Doing this preprocessor trick because if we used a ternary
 * operator, dummy macros needed to prevent compile errors may
 * become an attractive nuisance */
#define Py_TPFLAG_CHK(tp, flagname) \
Py__TPFCHK_##flagname(tp, flagname)
#define Py__TPFCHK_OLD(tp, flagname) \
((tp).tp_flags & Py_TPFLAGS_##flagname)
#define Py__TPFCHK_NEW(tp, flagname) \
((tp).tp_flags & Py_TPFLAGS_TPFLAGVER \
 && (tp).tp_flagver >= Py_TPFLAGVER_##flagname \
 && Py_TPFLAGCHK_##flagname(tp))

#define Py__TPFCHK_HEAPTYPE  Py__TPFCHK_OLD
#define Py_TPFLAGS_HEAPTYPE (1UL<<9)

#define Py__TPFCHK_TPFLAGVER Py__TPFCHK_OLD
#define Py_TPFLAGS_TPFLAGVER (1UL<<31)

#define Py__TPFCHK_NEWFLD Py__TPFCHK_NEW
#define Py_TPFLAGVER_NEWFLD 32
#define Py_TPFLAGCHK_NEWFLD(tp) ((tp).tp_newfld != NULL)

So to check heap type you get Py_TPFLAG_CHK(tp, HEAPTYPE)
((tp).tp_flags & (1UL<<9))

And to check newfld1 you get Py_TPFLAG_CHK(tp, NEWFLD)
((tp).tp_flags & (1UL<<31) && (tp).tp_flagver >= 32 && ((tp).tp_newfld != 
((void *)0)))

Or in a "more flags" scenario:

#define Py__TPFCHK_NEW(tp, flagname) \
((tp).tp_flags & Py_TPFLAGS_TPMOREFLAGS \
 && (tp).tp_moreflag & PY_TPMOREFLAGS_##flagname)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Whatever happened to 'nonlocal x = y'?

[Random832]

On Fri, Jan 5, 2018, at 10:47, Guido van Rossum wrote:
> I don't recall (though someone with more time might find the discussion in
> the archives or on the tracker). It was never implemented and I think it
> shouldn't be. So we might as well update the PEP. It wouldn't be
> particularly useful, since (by definition) the function that declares the
> nonlocal variable is not its owner, and hence it's unlikely to make sense
> to initialize it here. The same reasoning applies to global BTW.

I'm not so sure...

The only situation in which you're *required* to declare a nonlocal/global 
variable, after all, is if you intend to assign it - a name that you never 
assign is presumed to be non-local. The description in the PEP also applies to 
augmented assignments, and "global some_counter; some_counter += 1" is 
certainly a pattern I've needed in the past.

The PEP also quotes you as endorsing this for global. 
https://mail.python.org/pipermail/python-3000/2006-November/004166.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Whatever happened to 'nonlocal x = y'?

[Random832]

On Fri, Jan 5, 2018, at 14:07, Guido van Rossum wrote:
> Yeah, but I've changed my mind on this -- I think it's needless added
> complexity that helps save one line of code in very few use cases. And you
> don't really think the PEP endorses `nonlocal foo += 1` do you?

The PEP itself is very clear about supporting "augmented assignment", even 
defining the grammar differently for this case (such that you can't do it with 
multiple names in a single statement). The grammar shown also supports 
"nonlocal foo = bar = baz", though the text doesn't mention this form.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Random832]

On Sun, Jan 7, 2018, at 19:50, Steve Dower wrote:
> Considering there is no cross-platform compatibility here anyway, I 
> don’t think it’s that bad an option to let users do their own escaping, 
> especially since those who are successfully using this feature already 
> do.

I don't really think we should give up on cross-platform compatibility that 
easily. There are a number of constructs supported with the same syntax by both 
cmd and unix shells (pipes and redirection, mainly) that people may want to use.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Random832]

On Mon, Jan 8, 2018, at 18:05, Gregory P. Smith wrote:
> i agree with just a list of tuples, but i suggest creating namedtuple
> instances in the posix module for the purpose (one each for close, dup2,
> open) .  Don't put a reference to a function in the tuple as Serhiy
> suggested as, while obvious what it means, it gives the wrong impression to
> the user: nothing is calling the Python functions.  This is a posix API
> that takes a list of arguments for a specific set of system calls for _it_
> to make for us in a specific order.

Instead of a sequence of functions to call, it'd be nice if a higher-level API 
could allow just passing in a mapping of file descriptor numbers to what they 
should point to in the new process, and the implementation figures out what 
sequence is necessary to get that result.

And at that point we could just extend the subprocess API to allow redirection 
of file descriptors other than 0/1/2, and have an implementation of it in terms 
of posix_spawn.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?

[Random832]

On Sat, Jan 13, 2018, at 12:06, Christian Heimes wrote:
> Hi,
> 
> PEP 370 [1] was my first PEP that got accepted. I created it exactly one
> decade and two days ago for Python 2.6 and 3.0. Back then we didn't have
> virtual environment support in Python. Ian Bicking had just started to
> create the virtualenv project a couple of months earlier.
> 
> Fast forward 10 years...
> 
> Nowadays Python has venv in the standard library. The user-specific
> site-packages directory is no longer that useful. I would even say it's
> causing more trouble than it's worth. For example it's common for system
> script to use "#!/usr/bin/python3" shebang without -s or -I option.
> 
> I propose to deprecate the feature and remove it in Python 4.0.

Where would pip install --user put packages, and how would one run scripts that 
require those packages? Right now these things Just Work; I've never had to 
learn how to use virtual environments.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Exposing different versions of a system call in Python

[Random832]

On Fri, Jan 19, 2018, at 04:28, Pablo Galindo Salgado wrote:
> On the other side, we have pipe and pipe2 as an example of exposing two
> versions when this situation happens.
> 
> The question is:
> 
> What is preferable, exposing both functions or augment the old one?

A large number, possibly a majority, of system calls in the os module, have 
"dir_fd" arguments that cause them to call the *at counterpart of the 
underlying system call. And chdir can also be called with a file descriptor, 
which will call fchdir (though there is also a fchdir function) I don't know 
why pipe2 was implemented as a separate call.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 495 Was: PEP 498: Literal String Interpolation is ready for pronouncement

[Random832]

Alexander Belopolsky  writes:
> There is no "earlier" or "later". There are "lesser" and "greater"
> which are already defined for all pairs of aware datetimes. PEP 495
> doubles the set of possible datetimes

That depends on what you mean by "possible".

> and they don't fit in one
> straight line anymore. The whole point of PEP 495 is to introduce a
> "fold" in the timeline.

That doesn't make sense. Within a given timezone, any given moment of
UTC time (which is a straight line [shut up, no leap seconds here]) maps
to only one local time. The point of PEP 495 seems to be to eliminate
the cases where two UTC moments map to the same aware local time.

Out of curiosity, can "fold" ever be any value other than 0 or 1? I
don't know if there are any real-world examples (doubt it), but I could
easily contrive a timezone definition that had three of a particular
clock time.

> Yes, but are we willing to accept that datetimes have only partial
> order?

I apparently haven't been following the discussion closely enough to
understand how this can possibly be the case outside cases I assumed it
already was (naive vs aware comparisons being invalid).

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 495 Was: PEP 498: Literal String Interpolation is ready for pronouncement

[Random832]

Alexander Belopolsky  writes:
> Yes, but it does that at the cost of introducing the second local
> "01:30" which is "later" than the first "01:40" while "obviously" (and
> according to the current datetime rules) "01:30" < "01:40".

The current datetime rules, such as they are, as far as I am aware,
order all aware datetimes (except spring-forward) according to the UTC
moment they map to. I'm not sure what the benefit of changing this
invariant is.

> Out of curiosity, can "fold" ever be any value other than 0 or 1?
>
> Thankfully, no.

What happens, then, if I were to define a timezone with three local
times from the same date? None may exist now, but the IANA data format
can certainly represent this case. Should we be talking about adding an
explicit offset member?  (Ultimately, this "fold=1 means the second one"
notion is a novel invention, and including the offset either explicitly
a la ISO8601, or implicitly by writing EST/EDT, is not)

>
> > Yes, but are we willing to accept that datetimes have only
> partial
> > order?
> 
> I apparently haven't been following the discussion closely enough
> to
> understand how this can possibly be the case outside cases I
> assumed it
> already was (naive vs aware comparisons being invalid).
>
> Local times that fall in the spring-forward gap cannot be ordered
> interzone even without PEP 495.

Hmm. If these have to be allowed to exist, then...

What about ordering times according to, notionally, a tuple of (UTC
timestamp of transition, number of "fake" seconds "after" the
transition) for a spring-forward time?

Also, can someone explain why this:
>>> ET = pytz.timezone("America/New_York")
>>> datetime.strftime(datetime.now(ET) + timedelta(days=90),
...   "%Y%m%d %H%M%S %Z %z")
returns '20151210 214526 EDT -0400'

I don't know if I expected 214526 or 204526, but I certainly expected
the timezone info to say EST -0500. If EST and EDT are apparently two
distinct tzinfo values, then what exactly would a value landing near the
"fall back" transition have given for fold? fold=1 but EDT?

And if EST and EDT are, against all rationality, distinct tzinfo values,
then when exactly can fold ever actually be 1, and why is it needed?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 495 Was: PEP 498: Literal String Interpolation is ready for pronouncement

[Random832]

MRAB  writes:
> What would happen if it's decided to stay on DST and then, later on, to
> reintroduce DST?
>
> Or what would happen in the case of "British Double Summer Time" (go
> forward twice in the spring and backward twice in the autumn)?
>
> https://en.wikipedia.org/wiki/British_Summer_Time

"backward twice" could theoretically do it, if you literally went back
an hour, waited an hour (or any nonzero amount less than two hours), and
went back an hour again, rather than just going back two hours. I don't
know if any real-life authorities have ever done such a thing; that's
why I asked.

You could also have them if you had a "timezone" representing the
real-time local time of someone who traveled across timezone boundaries
multiple times in close succession, rather than the time of a
geographical place.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rationale behind lazy map/filter

[Random832]

"R. David Murray"  writes:

> On Tue, 13 Oct 2015 14:59:56 +0300, Stefan Mihaila
>  wrote:
>> Maybe it's just python2 habits, but I assume I'm not the only one
>> carelessly thinking that "iterating over an input a second time will 
>> result in the same thing as the first time (or raise an error)".
>
> This is the way iterators have always worked.

It does raise the question though of what working code it would actually
break to have "exhausted" iterators raise an error if you try to iterate
them again rather than silently yield no items.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rationale behind lazy map/filter

[Random832]

"R. David Murray"  writes:
> On Tue, 13 Oct 2015 11:26:09 -0400, Random832  wrote:
>> It does raise the question though of what working code it would actually
>> break to have "exhausted" iterators raise an error if you try to iterate
>> them again rather than silently yield no items.
>
> They do raise an error: StopIteration.  It's just that the iteration
> machinery uses that to stop iteration :).

I meant a real error and you know it, both of you. StopIteration is an
exception in the technical sense that it can be raised and caught, but
it's not an error because it is used for normal control flow. In the
plain english meaning of the word, it isn't even an exception.

> And the answer to the question is: lots of code.  I've written some:
> code that iterates an iterator, breaks that loop on a condition, then
> resumes iterating, breaking that loop on a different condition, and so
> on, until the iterator is exhausted.  If the iterator restarted at the
> top once it was exhausted, that code would break

I'm not suggesting restarting at the top (I've elsewhere suggested that
many such methods would be better as an *iterable* that can be restarted
at the top by calling iter() multiple times, but that's not the same
thing). I'm suggesting raising an exception other than StopIteration, so
that this situation can be detected. If you are writing code that tries
to resume iterating after the iterator has been exhausted, I have to
ask: why?

I suppose the answer is the same reason people would deliberately raise
StopIteration in the ways that PEP479 breaks - because it works and is
easy. But that wasn't a reason not to deprecate that.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 0484 - the Numeric Tower

[Random832]

> From Lib/numbers.py:
>
> ## Notes on Decimal
> ## 
> ## Decimal has all of the methods specified by the Real abc, but it should
> ## not be registered as a Real because decimals do not interoperate with
> ## binary floats (i.e.  Decimal('3.14') + 2.71828 is undefined).  But,
> ## abstract reals are expected to interoperate (i.e. R1 + R2 should be
> ## expected to work if R1 and R2 are both Reals).

Why?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rationale behind lazy map/filter

[Random832]

Chris Angelico  writes:
> A well-behaved iterator is supposed to continue raising StopIteration
> forever once it's been exhausted.

Yes, and that is *precisely* the behavior that causes the problem under
discussion. My question was what code depends on this.

> Play with that, and see where RuntimeErrors start coming up. I suspect
> they'll be rare, but they will happen.

My theory is that most circumstances under which this would cause a
RuntimeError are indicative of a bug in the algorithm consuming the
iterator (for example, an algorithm that hasn't considered iterators and
expects to be passed an iterable it can iterate from the top more than
once), rather than the current behavior being relied on to produce
the intended end result.

This is essentially the same argument as PEP 479 - except there it was
at least *easy* to come up with code which would rely on the old
behavior to produce the intended end result.

About the only example I can think of is that the implementation of
itertools.zip_longest would have to change.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 0484 - the Numeric Tower

[Random832]

Steven D'Aprano  writes:

> On Tue, Oct 13, 2015 at 04:37:43PM -0700, Raymond Hettinger wrote:
>
>> We could have (and still could) make the choice to always coerce to 
>> decimal (every float is exactly representable in decimal).  Further, 
>> any decimal float or binary float could be losslessly coerced to a 
>> Fraction, but that probably isn't what you really want most of the 
>> time.  I think people who work in decimal usually want to stay there 
>> and people who work with binary floating point want to stay there as 
>> well (invisible coercions being more likely to cause pain than relieve 
>> pain).
>
> Further to what Raymond says, if anyone wants to persue this further, 
> and wants to argue for such automatic promotion of float to Decimal (or 
> vice versa), I think a good place to start would be a survey of other 
> languages with a numeric tower. How do they handle similar situations?

I believe that in Scheme (which AIUI is where the term "numeric tower"
comes from) has a notion of "exact" and "inexact" numbers. A "flonum"
(float) would be an inexact number. And any operation between exact and
inexact numbers (including e.g. min and max, even if the exact number
wins) gives an inexact result.

A Decimal, though, could be regarded as an exact number (a special kind
of rational) or an inexact number (another kind of floating point). I
suspect some people who use them intend them one way and others intend
the other (especially since Fraction lacks a good way to format the
value in decimal notation).

If they are both inexact, then it doesn't much matter which one they're
promoted to, since they're both implementation details of the abstract
type "inexact real number". AIUI many implementations don't *actually*
have any other implementations of "inexact real number" except flonum,
and those that do just have them as a rational fraction with an
"inexact" flag set, but the spec does allow it.

Implementing a scheme-style exact/inexact numeric tower also suggests
more ABCs.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 506 secrets module

[Random832]

Brian Gladman  writes:

>> On Sat, Oct 17, 2015 at 03:26:46AM +1100, Steven D'Aprano wrote:
> I hence support your conclusion that the module should offer randbelow
> alone.  I would oppose offering randomrange (or offering more than one
> of them) since this will pretty well guarantee that, sooner or later,
> someone will make a mistake in using the extra functionality and
> possibly deploy an insecure application as a result.
>
>Brian Gladman

Plus if someone really does want randrange, they can simply do this:

def randfrom(seq):
return seq[randbelow(len(seq))]

def randrange(start, stop, step=None):
randfrom(range(start, stop, step))

These are simple recipes that probably don't belong in the module.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 506 secrets module

[Random832]

Guido van Rossum  writes:

> On Sat, Oct 17, 2015 at 2:50 AM, Steven D'Aprano 
> wrote:
>
> [...]
> So you can see there is nothing even close to consensus as to
> which API
> is best, which is an argument for keeping all three functions.
>
> No, that's not how we do it in Python. :-)

Ooh, I know this! It's an argument for scrapping the whole thing, right?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Random832]

Raymond Hettinger  writes:
> Thanks for hunting this down.  I had seen the automated reference leak
> posts but didn't suspect that a pure python class would have caused
> the leak.
>
> I'm re-opening
> https://mail.python.org/pipermail/python-dev/2015-October/141993.html
> and will take a look at it this weekend.  If I don't see an obvious
> fix, I'll revert Joe's patch until a correct patch is supplied and
> reviewed.

If a pure python class can cause a reference leak, doesn't that mean it
is only a symptom rather than the real cause? Or is it that the use of
@object.__new__ is considered "too clever" to be worth fixing?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 -- proposal to allow @overload in non-stub files

[Random832]

Guido van Rossum  writes:
> The proposal is to allow this to be written as follows in
> implementation (non-stub) modules:
>
> class Foo(Generic[T]):
> @overload
> def __getitem__(self, i: int) -> T: ...
> @overload
> def __getitem__(self, s: slice) -> Foo[T]: ...
> def __getitem__(self, x):
> 
>
> The actual implementation must be last, so at run time it will
> override the definition.

How about this to allow overloads to have actual implementations?

@overloaded
def __getitem__(self, x):


@overloaded returns a function which will check the types against the
overloads (or anyway any overloads that have actual implementations),
call them returning the result if applicable, otherwise call the
original function.

Some magic with help() would improve usability, too - it could print all
the overloads and their docstrings.  Maybe even @overload('__getitem__')
def __get_int(self, i: int), to make it so order doesn't matter.

That just leaves the question of how's this all gonna work with
subclasses.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Random832]

Victor Stinner  writes:
> These encodings are rarely used. I don't think that any text editor
> use them.

MS Windows' Notepad can be made to use UTF-16.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Random832]

Chris Angelico  writes:
> Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix
> program loaders won't.)

A lot of them can't handle UTF-8 with a BOM, either.

> That alone might be a reason for strongly encouraging ASCII-compat
> encodings.

A "python" or "python3" or "env" executable in any particular location
such as /usr/bin isn't technically guaranteed, either, it's just relied
on as a "works 99% of the time" thing. There's a reasonable case to be
made that transforming files in such a way as they get launched by
python (which may require an encoding change to an ASCII-compatible
encoding, or a wrapper script, or the python -x hack) is the
responsibility of platform-specific installer code.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Random832]

Glenn Linderman  writes:
> On 11/14/2015 5:37 PM, Chris Angelico wrote:
> > Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding?
>
> I wouldn't trust an answer to this question that didn't come from
> someone that used Windows with Chinese, Japanese, or Korean, as their
> default language for the install. So I don't have a trustworthy answer
> to give.

AFAIK (I haven't actually used it as a default language, but I do know
some details of their encodings) There are two main "issues" with the
windows CJK encodings regarding ASCII compatibility:

- There is a different symbol (a currency symbol) at 0x5c. Sort of. Most
  unicode translations of it will treat it as a backslash, and users do
  expect it to work for things like \n, path separators, etc, but it
  displays as ¥ or ₩.

- Dual-byte characters can have ASCII bytes as their *second* byte.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Random832]

"Stephen J. Turnbull"  writes:
> I don't see any good reason for allowing non-ASCII-compatible
> encodings in the reference CPython interpreter.

There might be a case for having the tokenizer not care about encodings
at all and just operate on a stream of unicode characters provided by a
different layer.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Random832]

MRAB  writes:
> As I understand it, *nix expects the shebang to be b'#!',
> which means that the first line should be ASCII-compatible
> (it's possible that the UTF-8 BOM might be present).

The UTF-8 BOM interferes with it on Mac OSX and Linux, at
least.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deleting with setting C API functions

[Random832]

On 2015-12-02, M.-A. Lemburg  wrote:
> A first step would be to replace the macro with a proper function
> to avoid false positive warnings, even when using the correct API.
>
> Then we could add a warning to the PyObject_SetAttr() function and
> hope that not too many projects use the stable ABI as basis to
> have C extensions work across several releases.

How about using a versioned ABI? Make a new function that
doesn't allow NULL, called something like PyObject_SetAttr2, and
instead of declaring the old one in headers, use a #define to
the new name.

> Overall, I'm not sure whether it's worth the trouble. Documenting
> the feature and adding a deprecation notice to just the documentation
> would likely be better. We could then remove the functionality
> in Python 4.

Are there plans for a Python 4?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deleting with setting C API functions

[Random832]

On 2015-12-02, Victor Stinner  wrote:
>> Are there plans for a Python 4?
>
> No. Don't. Don't schedule any "removal" or *any* kind of "break
> backward compatibility" anymore, or you will definetly kill the Python
> community.

I feel like I should note that I agree with your position here, I was
just asking the question to articulate the issue that "put it off to the
indefinite future" isn't a real plan for anything.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deleting with setting C API functions

[Random832]

On 2015-12-02, Andrew Barnert wrote:
> Python could just go from 3.9 to 4.0, as a regular dot release, just
> to dispel the idea of an inevitable backward-incompatible "Python 4".
> (That should be around 2 years after the expiration of 2.7 support,
> py2/py3 naming, etc., right?)

Why bother with the dot? Why not rename 3.5 to Python 5, and then go to
Python 6, etc, and then your "4.0" would be 10.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

[Random832]

On 2015-12-03, Laura Creighton  wrote:
> Who came up with the word 'display' and what does it have going for
> it that I have missed?  Right now I think its chief virtue is that
> it is a meaningless noun.  (But not meaningless enough, as I
> associate displays with output, not construction).

In a recent discussion it seemed like people mainly use it
because they don't like using "literal" for things other than
single token constants.  In most other languages' contexts the
equivalent thing would be called a literal.

> I think that 
>
> 6.2.4 Constructing lists, sets and dictionaries
>
> would be a much more useful title, and
>
> 6.2.4 Constructing lists, sets and dictionaries -- explicitly or through 
> the use of comprehensions

I don't like the idea of calling it "explicit construction".
Explicit construction to me means the actual use of a call to the
constructor function.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New poll about a macro for safe reference replacing

[Random832]

Serhiy Storchaka  writes:
> I'm bringing this up again, since the results of the previous poll did
> not give an unambiguous result. Related links: [1], [2], [3], [4].
>
> Let me remind you that we are talking about adding the following
> macro. It is needed for safe replacement links. For now there is at
> least one open crash report that can be solved with this macro [5] (I
> think there is yet one, but can't find it just now). And 50 potential
> bugs for which we just still do not have a reproducer.
>
> #define Py_XXX(ptr, value)\
> { \
> PyObject *__tmp__ = ptr;  \
> ptr = new_value;  \
> Py_DECREF(__tmp__);   \
> }

At the risk of bikeshedding, this needs do { ... } while(0), or
it almost certainly will eventually be called incorrectly in an
if/else statement.  Yes, it's ugly, but that's part of the cost
of using macros.

If it were implemented as below, then it could evaluate ptr only
once at the cost of requiring it to refer to an addressable
pointer object:
PyObject **__tmpp__ == &(ptr);
PyObject *__tmp__ = *__tmpp__;
*__tmpp__ = (new_value);
PY_DECREF(__tmp__);

I'm not entirely sure of the benefit of a macro over an inline
function.  Or why it doesn't INCREF the new value, maintaining
the invariant that ptr is an owned reference.

> 1. Py_SETREF
> 2. Py_DECREF_REPLACE
> 3. Py_REPLACE
> 4. Py_SET_POINTER
> 5. Py_SET_ATTR
> 6. Py_REPLACE_REF

I think "SET" names imply that it's safe if the original
reference is NULL. This isn't an objection to the names, but if
it is given one of those names I think it should use Py_XDECREF.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New poll about a macro for safe reference replacing

[Random832]

Serhiy Storchaka  writes:
>> I'm not entirely sure of the benefit of a macro over an inline
>> function.
>
> Because the first argument is passed by reference (as in Py_INCREF
> etc).

Then a macro implemented using an inline function, e.g.,
#define Py_REPLACE(p, x) Py_REPLACE_impl(&(p), x).  Were INCREF
implemented this way it could return the reference (imagine
Py_REPLACE(foo, Py_INCREF(bar))).  The other advantage to an inline
function is that it lets the compiler make the decision about optimizing
for size or time.
 
>> I think "SET" names imply that it's safe if the original
>> reference is NULL. This isn't an objection to the names, but if
>> it is given one of those names I think it should use Py_XDECREF.
>
> Originally I proposed pairs of functions with and withot X in the name
> (as Py_DECREF/Py_XDECREF). In this poll this detail is omitted for
> clearness. Later we can create a new poll if needed.

I think that any variant on "SET" strongly implies that it need not have
already been set, and think even a "SET/REPLACE" pair would be better
than "XSET/SET".

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Change the repr for datetime.timedelta (was Re: Asynchronous context manager in a typical network server)

[Random832]

Guido van Rossum  writes:
> I'm sure that one often catches people by surprise. However, I don't
> think we can fix that one without also fixing the values of the
> attributes -- in that example days is -1 and seconds is 86340 (which
> will *also* catch people by surprise). And changing that would be
> much, much harder for backwards compatibility reasons-- we'd have to
> set days to 0 and seconds to -60, and suddenly we have a much murkier
> invariant, instead of the crisp
>
> 0 <= microseconds < 100
> 0 <= seconds < 60

I don't really see it as murky:

0 <= abs(microseconds) < 100
0 <= abs(seconds) < 60
(days <= 0) == (seconds <= 0) == (microseconds <= 0)
(days >= 0) == (seconds >= 0) == (microseconds >= 0)

The latter are more easily phrased in english as "all nonzero
attributes have the same sign".  I think the current behavior is
rather as if -1.1 were represented as "-2+.9".  The attributes
probably can't be fixed without breaking backwards
compatibility, though.  How about "-timedelta(0, 60)"?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New poll about a macro for safe reference replacing

[Random832]

Nick Coghlan  writes:
> Avoiding those misleading associations is a good argument in favour of
> Py_REPLACE over Py_SETREF - they didn't occur to me before casting my
> votes, and I can definitely see them causing confusion in the future.
>
> So perhaps the combination that makes the most sense is to add
> Py_REPLACE (uses Py_DECREF on destination) & Py_XREPLACE (uses
> Py_XDECREF on destination) to the existing Py_CLEAR?

Is there a strong reason to have an X/plain pair? Py_CLEAR
doesn't seem to have one.  This wasn't a subject of the poll.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] do people use sys._mercurial?

[Random832]

Brett Cannon  writes:
> (and no, we are not going to adopt a generic one; already had that
> debate).

Do you have a link to the relevant discussions?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Mon, Apr 11, 2016, at 13:36, Brett Cannon wrote:
> How about we take something from the "explicit is better than implicit"
> playbook and add a keyword argument to os.fspath() to allow bytes to pass
> through?

Except, we already know how to convert a bytes-path into a str (and vice
versa) with sys.getfilesystemencoding and surrogateescape. So why not
just have the argument specify what return type is desired?

def fspath(path, *, want_bytes=False):
if isinstance(path, (bytes, str)):
ppath = path
else:
try:
ppath = path.__fspath__()
except AttributeError:
raise TypeError
if isinstance(ppath, str):
return ppath.encode(...) if want_bytes else ppath
elif isinstance(ppath, bytes):
return ppath if want_bytes else ppath.decode(...)
else:
raise TypeError

This way the posix os module can call the function and have the bytes
value already prepared for it to pass to the real open() syscall.

You could even add the same thing in other places, e.g. os.path.join
(defaulting to if the first argument is a bytes).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Random832]

On Mon, Apr 11, 2016, at 16:48, Sven R. Kunze wrote:
> On 11.04.2016 22:33, Alexander Walters wrote:
> > If there is headway being made, I do not see it.
> 
> Funny that you brought it up. I was about posting something myself. I 
> cannot agree completely. But starting with a comment from Paul, I 
> realized that pathlib is something different than a string. After doing 
> the research and our issues with pathlib, I found:
> 
> 
> - pathlib just needs to be improved (see my 5 points)
> - os[.path] should not tinkered with

I'm not so sure. Is there any particular reason os.path.join should
require its arguments to be homogenous, rather than allowing
os.path.join('a', b'b', Path('c')) to return 'a/b/c'?

> I know that all of those discussions of a new protocol (path->str, 
> __fspath__ etc. etc.) might be rendered worthless by these two 
> statements. But that's my conclusion.
> 
> "os" and "os.path" are just lower level. "pathlib" is a high-level, 
> convenience library. When using it, I don't want to use "os" or 
> "os.path" anymore. If I still do, "pathlib" needs improving. *Not "os" 
> nor "os.path"*.

The problem isn't you using os. It's you using other modules that use
os. or io, shutil, or builtins.open. Or pathlib, if what *you're* using
is some other path library. Are you content living in a walled garden
where there is only your code and pathlib, and you never might want to
pass a Path to some function someone else (who didn't use pathlib)
wrote?

os is being used as an example because fixing os probably gets you most
other things (that just pass it through to builtins.open which passes it
through to os.open) for free.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Random832]

On Mon, Apr 11, 2016, at 17:04, Sven R. Kunze wrote:
> PS: The only way out that I can imagine is to fix pathlib. I am not in 
> favor of fixing functions of "os" and "os.path" to except "path" 
> objects;

Why not?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Random832]

On Mon, Apr 11, 2016, at 17:15, Ethan Furman wrote:
> So we're trying to make option 2 work before falling back to option 1.
> 
> If you have a way to make pathlib work with the stdlib that doesn't 
> involve "fixing" os and os.path, now is the time to speak up.

Fully general re-dispatch from argument types on any call to a function
that raises TypeError or NotImplemented? [e.g. call
Path.__missing_func__(os.open, path, mode)]

Have pathlib monkey-patch things at import?


On Mon, Apr 11, 2016, at 17:43, Sven R. Kunze wrote:
> So, I might add:
> 
> 3. add more high-level features to pathlib to prevent a downgrade to os 
> or os.path

3. reimplement the entire ecosystem in every walled garden so no-one has
to leave their walled gardens.

What's the point of batteries being included if you can't wire them to
anything?

I don't get what you mean by this whole "different level of abstraction"
thing, anyway. The fact that there is one obvious thing to want to do
with open and a Path strongly suggests that that should be able to be
done by passing the Path to open.

Also, what level of abstraction is builtin open? Maybe we should _just_
leave os alone on the grounds of some holy sacred lowest-level-itude,
but allow io and shutils to accept Path?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Maybe, just maybe, pathlib doesn't belong.

[Random832]

On Tue, Apr 12, 2016, at 10:52, Sven R. Kunze wrote:
> On 12.04.2016 00:56, Random832 wrote:
> > Fully general re-dispatch from argument types on any call to a function
> > that raises TypeError or NotImplemented? [e.g. call
> > Path.__missing_func__(os.open, path, mode)]
> >
> > Have pathlib monkey-patch things at import?
> 
> Implicit conversion. No, thanks.

No more so than __radd__ - I didn't actually mean this as a serious
suggestion, but but python *does* already have multiple dispatch.

> > On Mon, Apr 11, 2016, at 17:43, Sven R. Kunze wrote:
> > I don't get what you mean by this whole "different level of abstraction"
> > thing, anyway.
> 
> Strings are strings. Paths are paths. That's were the difference is.

Yes but why aren't these both "things that you may want to use to open a
file"?

> > The fact that there is one obvious thing to want to do
> > with open and a Path strongly suggests that that should be able to be
> > done by passing the Path to open.
> 
> Path(...).open() is your friend then. I don't see why you need os.open.

Because I'm passing it to modfoo.dosomethingwithafile() which takes a
filename and passes it to shutils, which passes it to builtin open,
which passes it to os.open.

Should Path grow a dosomethingwithmodfoo method?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Tue, Apr 12, 2016, at 12:40, Chris Barker wrote:
> Ah -- there's the fatal flaw -- even Windows needs bytes at the lowest
> level,

Only in the sense that literally everything's bytes at the lowest level.
But the bytes Windows needs are not in an ASCII-compatible encoding so
it's not reasonable to talk about them in the same way as every other
kind of bytes filename.

> but the decision was already made there to use str as the the
> lingua-franca -- i.e. the user NEVER sees a path as a bytestring on
> Windows? I guess that's decided then. str is the exchange format.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Wed, Apr 13, 2016, at 10:21, Nick Coghlan wrote:
> I'd expect the main consumers to be os and os.path, and would honestly
> be surprised if we needed many explicit invocations above that layer,
> other than in pathlib itself.

I made a toy implementation to try this out, and making os.open support
it does not get you builtin open "for free" as I had suspected; builtin
open has its own type checks in _iomodule.c.

Probably anything not implemented in pure python that deals with
filenames is going to have to have its type checking revised.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Wed, Apr 13, 2016, at 11:28, Ethan Furman wrote:
> On 04/13/2016 08:17 AM, Random832 wrote:
> > On Wed, Apr 13, 2016, at 10:21, Nick Coghlan wrote:
> 
> >> I'd expect the main consumers to be os and os.path, and would honestly
> >> be surprised if we needed many explicit invocations above that layer,
> >> other than in pathlib itself.
> >
> > I made a toy implementation to try this out, and making os.open support
> > it does not get you builtin open "for free" as I had suspected; builtin
> > open has its own type checks in _iomodule.c.
> 
> Yup, it will take some effort to make this work.

A corner case just occurred to me...

For functions that will continue to accept str/bytes (and functions that
accept some other type such as Number or file-like objects), what should
be done with an object that is one of these, *and* has an __fspath__
method, *and* this method returns a value other than the object's own
value? Basically, should the protocol check be done unconditionally
(before attempting to use the argument as a string) or only if the
argument is not a string (there's an efficiency argument for this). Or
should it be left "unspecified", with the understanding that such
objects are badly behaved and may not be handled consistently across
different functions / python implementations / cpython versions?

Also, should the os.fspath (or whatever we call it) function itself
accept str/bytes, even if these are not going to implement the protocol?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Wed, Apr 13, 2016, at 15:24, Chris Angelico wrote:
> Is that the intention, or should the exception catching be narrower? I
> know it's clunky to write it in Python, but AIUI it's less so in C:

How is it less so in C? You lose the ability to PyObject_CallMethod.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Wed, Apr 13, 2016, at 16:39, Chris Barker wrote:
> so are we worried that __fspath__ will exist and be callable, but  might
> raise an AttributeError somewhere inside itself? if so isn't it broken
> anyway, so should it be ignored?

Well, if you're going to say "ignore the protocol because it's broken",
where do you stop? What if it raises some other exception? What if it
raises SystemExit? 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Apr 13, 2016 20:06, Chris Barker  wrote:
>
> In this case, I don't know that we need to be tolerant of buggy __fspathname__() implementations -- they should be tested outside these checks, and not be buggy. So a buggy implementation may raise and may be ignored, depending on what Exception the bug triggers -- big deal. The only time it would matter is when the implementer is debugging the implementation.
>
> -CHB
Yes but you can often, and can in this case, restrict the contents of the try block to a single operation - a name access, an attribute, a subscript - and that sharply limits the risk of such a thing happening. Sure, the object's __getattr(ibute)__ could still fail from something deep inside it missing a different attribute, but that's it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Apr 13, 2016 19:06, Brett Cannon  wrote:
> On Wed, 13 Apr 2016 at 15:46 Nikolaus Rath  wrote:
>> When passing an object that is of type str and has a __fspath__
>> attribute, all approaches return the value of __fspath__().
>>
>> However, when passing something of type bytes, the second approach
>> returns the object, while the third returns the value of __fspath__().
>>
>> Is this intentional? I think a __fspath__ attribute should always be
>> preferred.
>
>
> It's very much intentional. If we define __fspath__() to only return strings 
> but still want to minimize boilerplate of allowing bytes to simply pass 
> through without checking a path argument to see if it is bytes then approach 
> #2 is warranted. But if __fspath__() can return bytes then approach #3 allows 
> for it. 

Er, the difference comes in when the object passed to os.fspath is a subclass 
of bytes that, itself, has a __fspath__ method (which may return a str). It's 
unlikely to occur in the wild, but is a semantic difference between this case 
and all other objects with __fspath__ methods.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Wed, Apr 13, 2016, at 23:17, Nick Coghlan wrote:

> - os.fspath -> str (no coercion)
> - os.fsdecode -> str (with coercion from bytes)
> - os.fsencode -> bytes (with coercion from str)
> - os._raw_fspath -> str-or-bytes (no coercion)
> 
> (with "coercion" referring to how the result of __fspath__ and any
> directly passed in str or bytes objects are handled)
> 
> The leading underscore on _raw_fspath would be of the "this is a
> documented and stable API, but you probably don't want to use it
> unless you really know what you're doing" variety, rather than the
> "this is an undocumented and potentially unstable private API"
> variety.

In this scenario could the protocol return bytes?

If the protocol cannot return bytes, then _raw_fspath will only return
bytes if directly passed bytes. This limits its utility for the
functions that consume it (presumably path_convert (os.open and friends)
and builtin open), since they already have to act specially based on the
types of their arguments (builtin open can accept an integer;
path_convert has to behave radically differently on str or bytes input)
and there's no reason they couldn't simply accept bytes directly while
they're doing that.

If the protocol can return bytes, then that means that types (DirEntry?
someone had an alternate path library with a bPath?) which return bytes
via the protocol will proliferate, and cannot be safely passed to
anything that uses os.fspath. Numerous copies of "def myfspath(x):
return os.fsdecode(os._raw_fspath(x))" will proliferate (or they'll just
monkey-patch os.fspath), and no-one actually uses os.fspath except toy
examples.

Why is it so objectionable for os.fspath to do coercion?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Wed, Apr 13, 2016, at 23:27, Nick Coghlan wrote:
> In this kind of case, inheritance tends to trump protocol. For
> example, int subclasses can't override operator.index:
...
> The reasons for that behaviour are more pragmatic than philosophical:
> builtins and their subclasses are extensively special-cased for speed
> reasons, and those shortcuts are encountered before the interpreter
> even considers using the general protocol.
> 
> In cases where the magic method return types are polymorphic (so
> subclasses may want to override them) we'll use more restrictive exact
> type checks for the shortcuts, but that argument doesn't apply for
> typechecked protocols where the result is required to be an instance
> of a particular builtin type (but subclasses are considered
> acceptable).

Then why aren't we doing it for str? Because "try: path =
path.__fspath__()" is more idiomatic than the alternative?

If some sort of reasoned decision has been made to require the protocol
to trump the special case for str subclasses, it's unreasonable not to
apply the same decision to bytes subclasses. The decision should be
"always use the protocol first" or "always use the type match first".

In other words, why not this:

def fspath(path, *, allow_bytes=False):
if isinstance(path, (bytes, str) if allow_bytes else str)
return path
try:
m = path.__fspath__
except AttributeError:
raise TypeError
path = m()
if isinstance(path, (bytes, str) if allow_bytes else str)
return path
raise TypeError
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Thu, Apr 14, 2016, at 02:00, Nick Coghlan wrote:
> > If the protocol can return bytes, then that means that types (DirEntry?
> > someone had an alternate path library with a bPath?) which return bytes
> > via the protocol will proliferate, and cannot be safely passed to
> > anything that uses os.fspath. Numerous copies of "def myfspath(x):
> > return os.fsdecode(os._raw_fspath(x))" will proliferate (or they'll just
> > monkey-patch os.fspath), and no-one actually uses os.fspath except toy
> > examples.
> 
> If folks want coercion, they can just use os.fsdecode(x), as that
> already has a str -> str passthrough from the input to the output
> (unlike codecs.decode) and will presumably be updated to include an
> implicit call to os._raw_fspath() on the passed in object.

This is the first I've heard of any suggestion to have fsdecode accept
non-strings.

> > Why is it so objectionable for os.fspath to do coercion?
> 
> The first problem is that binary paths on Windows basically don't
> work, so it's preferable for them to fail fast regardless of platform,
> rather than to have them implicitly work on *nix, only to fail for
> Windows users using non-ASCII paths later.

Ideally, this warning would be raised from a central place, and even
fspath (and even fsdecode) would go through it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Thu, Apr 14, 2016, at 03:02, Stephen J. Turnbull wrote:
> I have a strong preference for str only, because I still don't see a
> use case for polymorphic __fspath__.

Ultimately we're talking about redundancy and performance here. The "use
case" such as there is one, is if there's a class (be it DirEntry or
whatever else) that natively stores bytes, and __fspath__ has to return
str, then it calls fsdecode and then open immediately turns around and
calls fsencode on the result, accomplishing nothing vs just passing
everything straight through.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Thu, Apr 14, 2016, at 09:40, Nick Coghlan wrote:
> That's not a *new* problem though, it already exists if you pass in a
> mix of bytes and str:
> 
> There's also already a solution (regardless of whether you want bytes
> or str as the result), which is to explicitly coerce all the arguments
> to the same type:

It'd be nice if that went away. Having to do that makes about as much
sense to me as if you had to explicitly coerce an int to a float to add
them together. Sure, explicit is better than implicit, but there are
limits. You're explicitly calling os.path.join; isn't that explicit
enough?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Thu, Apr 14, 2016, at 09:50, Chris Angelico wrote:
> Adding integers and floats is considered "safe" because most people's
> use of floats completely compasses their use of ints. (You'll get
> OverflowError if it can't be represented.) But float and Decimal are
> considered "unsafe":
> 
> >>> 1.5 + decimal.Decimal("1.5")
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: unsupported operand type(s) for +: 'float' and
> 'decimal.Decimal'
> 
> This is more what's happening here. Floats and Decimals can represent
> similar sorts of things, but with enough incompatibilities that you
> can't simply merge them.

And what such incompatibilities exist between bytes and str for the
purpose of representing file paths? At the end of the day, there's
exactly one answer to "what file on disk this represents (or would
represent if it existed)".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Thu, Apr 14, 2016, at 12:05, Stephen J. Turnbull wrote:
> Random832 writes:
> 
>  > And what such incompatibilities exist between bytes and str for the
>  > purpose of representing file paths?
> 
> A plethora of encodings.

Only one encoding, fsencode/fsdecode. All other encodings are not for
filenames.

>  > At the end of the day, there's exactly one answer to "what file on
>  > disk this represents (or would represent if it existed)".
> 
> Nope.  Suppose those bytes were read from a file or a socket?  It's
> dangerous to assume that encoding matches the file system's.

Why can I pass them to os.open, then, or to os.path.join so long as
everything else is also bytes?

On UNIX, the filesystem is in bytes, so saying that bytes can't match
the filesystem is absurd. Converting it to str with fsdecode will
*always, absolutely, 100% of the time* give a str that will address the
same file that the bytes does (even if it's "dangerous" to assume that
was the name the user wanted, that's beyond the scope of what the module
is capable of dealing with).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] MAKE_FUNCTION simplification

[Random832]

On Thu, Apr 14, 2016, at 12:56, Terry Reedy wrote:
> https://docs.python.org/3/library/dis.html#module-dis
> CPython implementation detail: Bytecode is an implementation detail of 
> the CPython interpreter. No guarantees are made that bytecode will not 
> be added, removed, or changed between versions of Python.
> 
> Version = minor release, as opposed to maintenance release.

"between versions" is ambiguous. It could mean that there's no guarantee
that there will be no changes from one version to the next, or it could
mean, even more strongly, that there's no guarantee that there will be
no changes in a maintenance release (which are, after all, released
*between* minor releases)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Random832]

On Thu, Apr 14, 2016, at 13:56, Koos Zevenhoven wrote:
> (1) Code that has access to pathname/filename data and has some level
> of control over what data type comes in. This code may for instance
> choose to deal with either bytes or str
> 
> (2) Code that takes the path or file name that it happens to get and
> does something with it. This type of code can be divided into
> subgroups as follows:
> 
>   (2a) Code that accepts only one type of paths (e.g. str, bytes or
> pathlib) and fails if it gets something else.

Ideally, these should go away.

>   (2b) Code that wants to support different types of paths such as
> str, bytes or pathlib objects. This includes os.path.*, os.scandir,
> and various other standard library code. Presumably there is also
> third-party code that does the same. These functions may want to
> preserve the str-ness or bytes-ness of the paths in case they return
> paths, as the stdlib now does. But new code may even want to return
> pathlib objects when they get such objects as inputs.

Hold on. None of the discussion I've seen has included any way to
specify how to construct a new object representing a different path
other than the ones passed in. Surely you're not suggesting type(a)(b).

Also, how does DirEntry fit in with any of this?

> This is the
> duck-typing or polymorphic code we have been talking about. Code of
> this type (2b) may want to avoid implicit conversions because it makes
> the life of code of the other types more difficult.

As long as the type it returns is still a path/bytes/str (and therefore
can be accepted when the caller passes it somewhere else) what's the
problem?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Random832]

On Fri, Apr 15, 2016, at 16:41, Victor Stinner wrote:
> If the dictionary values are modified during the loop, the dict
> version is increased. But it's allowed to modify values when you
> iterate on *keys*.

Why is iterating over items different from iterating over keys?

in other words, why do I have to write:

for k in dict:
v = dict[k]
...do some stuff...
dict[k] = something

rather than

for k, v in dict.items():
...do some stuff...
dict[k] = something


It's not clear why the latter is something you want to prevent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Random832]

On Fri, Apr 15, 2016, at 23:46, Peter Ludemann via Python-Dev wrote:
> If Python ever adopts the BCPL rule for implicit line continuation if
> the last thing on a line is an operator (or if there's an open
> parentheses), then the break-after-an-operator rule would be more
> persuasive. ;)
>
> [IIRC, the BCPL rule was that there was an implicit continuation if
> the grammar would not allow inserting a semicolon at the end of the
> line, which covered both the open-parens and last-item-is-operator
> cases, and probably a few others.] But I should shut up and leave shut
> discussions to python-ideas.

Sounds like Visual Basic. Meanwhile, Javascript's rule is that there's
an implicit semicolon if and only if the grammar would not allow the
two lines to be considered as a single statement. Insanity comes in
all flavors.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Mon, Apr 18, 2016, at 15:26, Stephen J. Turnbull wrote:
> in
> particular it is not true for DirEntry (which is a "enhanced
> degenerate" path containing only one path segment but also other
> useful information abot the filesystem object addressed)

DirEntry contains multiple path segments - it has the name, and the
directory path that was passed into scandir.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inconsistency of PyModule_AddObject()

[Random832]

On Thu, Apr 28, 2016, at 05:05, Stefan Krah wrote:
> $ valgrind --suppressions=Misc/valgrind-python.supp ./python -c "import
> decimal"
> 
> [...]
> ==16945== LEAK SUMMARY:
> ==16945==definitely lost: 0 bytes in 0 blocks
>  

Well, the obvious flaw with your test case is that a reference is
retained forever in the C static variable basic_context_template.

Now, it is arguable that this may be a reasonably common pattern, and
that this doesn't actually constitute misuse of the API (the reference
count will be wrong, but the object itself is immortal anyway, so it
doesn't matter if it's 2 or 1 since it can't be 0 even with correct
usage)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inconsistency of PyModule_AddObject()

[Random832]

On Thu, Apr 28, 2016, at 10:11, Stefan Krah wrote:
> For actual users of Valgrind this is patently obvious and was
> pretty much the point of my post.

A more relevant point would be that _decimal does *not* use the API in a
way *which would be broken by the proposed change*, regardless of
whether the way in which it uses it is subjectively correct or can cause
leaks.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Convert int() to size_t in Python/C

[Random832]

On Fri, Apr 29, 2016, at 10:45, Marcos Dione wrote:
> One possible solution hat was suggested to me in the #python IRC
> channel was to use that, then test if the resulting value is negative,
> and adjust accordingly, but I wonder if there is a cleaner, more general
> solution (for instance, what if the type was something else, like loff_t,
> although for that one in particular there *is* a convertion
> function/macro).

In principle, you could just use PyLong_AsUnsignedLong (or LongLong),
and raise OverflowError manually if the value happens to be out of
size_t's range. (99% sure that on every linux platform unsigned long is
the same size as size_t.

But it's not like it'd be the first function in OS to call a system call
that takes a size_t. Read just uses Py_ssize_t. Write uses the buffer
protocol, which uses Py_ssize_t. How concerned are you really about the
lost range here? What does the system call return (its return type is
ssize_t) if it writes more than SSIZE_MAX bytes? (This shouldn't be hard
to test, just try copying a >2GB file on a 32-bit system)

I'm more curious about what your calling convention is going to be for
off_in and off_out. I can't think of any other interfaces that have
optional output parameters. Python functions generally deal with output
parameters in the underlying C function (there are a few examples in
math) by returning a tuple.

Maybe return a tuple (returned value, off_in, off_out), where None
corresponds to the input parameter having been NULL (and passing None in
makes it use NULL)?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Convert int() to size_t in Python/C

[Random832]

On Fri, Apr 29, 2016, at 14:11, Marcos Dione wrote:
> These are not output parameters, even if they're pointers. they'r
> using the NULL pointer to signal that the current offsets should not be
> touched, to differentiate from a offset of 0. Something that in Python we
> would use None.

That's not actually true according to the documentation. (And if it
were, they could simply use -1 rather than a null pointer)

If you pass a null pointer in, the file's offset is used and *is*
updated, same as if you used an ordinary read/write call. If you pass a
value in, that value is used *and updated* (which makes it an output
parameter) and the file's offset is left alone.

Documentation below, I've >>>highlighted<<< the part that shows they are
used as output parameters:

   The following semantics apply for off_in, and similar statements
   apply to off_out:

   *  If off_in is NULL, then bytes are read from fd_in starting
   from
  the file offset, and the file offset is adjusted by the number
  of
  bytes copied.

   *  If off_in is not NULL, then off_in must point to a buffer that
  specifies the starting offset where bytes from fd_in will be
  read.
  The file offset of fd_in is not changed, >>>but off_in is
  adjusted
  appropriately.<<<
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path PEP, part 2

[Random832]

On Fri, May 13, 2016, at 07:00, Steven D'Aprano wrote:
> - but os.fspath() will only return str;
> 
> - and os.fspathb() will only return bytes;

And raise an exception if __fspath__ returns the other, I suppose.
What's the use case for these functions? When would I call them rather
than fsdecode and fsencode? (assuming the latter will support the path
protocol - I've got more objections if they're not going to)

Also, what happens if you pass a string to os.fspath? Statements like
"str will not implement __fspath__" have thus far been light on details
of what functions will accept "str or
path-like-object-whose-__fspath__returns-str" and which ones will only
accept the latter (and what, if any, will continue to only accept the
former). If no-one's supposed to directly call dunder methods, and
os.fspath accepts str, then what difference does it make whether this is
implemented by having a special case within os.fspath or by calling
str.__fspath__?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period?

[Random832]

On Sun, May 22, 2016, at 11:15, Peter Ludemann via Python-Dev wrote:
> I just tried a re-flow in emacs (M-q or fill-paragraph) and it didn't
> change my single space to two (it also changed 3 spaces after a period
> to 2; and it preserved my hanging indents).

It won't change a single space to two, but it will add two spaces after
a sentence which was previously at the end of a line... and it *won't*
break a line after a dot followed by a single space (Vim isn't clever
enough for this second bit), preferring to break before the preceding
word instead.

A dot followed by a single space is *permitted* (hence why you observed
it didn't change it to two), it is simply assumed to represent something
other than the end of a sentence, such as an initial in a name.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C99

[Random832]

On Mon, Jun 6, 2016, at 07:23, Sturla Molden wrote:
> Ok, but if third-party developers shall be free to use a C89 compiler for
> their own code, we cannot have C99 in the include files. Otherwise the
> include files will taint the C89 purity of their source code.
> 
> Personally I don't think we need to worry about compilers that don't
> implement C99 features like inline functions in C. How long have the
> Linux
> kernel used inline functions instead of macros? 20 years or more?

Using inline functions instead of macros doesn't have to mean anything
but a performance hit on platforms that don't support them, since the
inline keyword, or some other identifier, could be defined to expand to
an empty token sequence on platforms that do not support it. It's much
lower impact on the source code than some other C99 features.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 572, Assignment Expressions

[Random832]

On Wed, Jul 11, 2018, at 20:10, Guido van Rossum wrote:
> As anticippated, after a final round of feedback I am hereby accepting PEP
> 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/

I know everyone else is probably sick of discussing this (I somehow completely 
missed the discussion until it was almost done) but I have a question...

Why does this not allow assignment to attributes and subscripts? I think this 
is, at least, surprising enough that there should be a rationale section 
explaining it in the PEP. As it is, it's barely even explicitly stated, other 
than the use of 'NAME' in a few places and an offhand mention "and [the 
assignment statement] can assign to attributes and subscripts."
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Random832]

On Tue, May 21, 2019, at 14:17, Christian Heimes wrote:
> thanks for bringing this topic up. Initially I considered http.server, 
> too. But as Guido put it, it's both used and useful for local testing 
> and quick hacks. I'm already facing opposition for modules that are 
> less controversial and useful than http.server, too. 

Could the high-level functionality of http.server (and perhaps socketserver as 
a whole) be rebuilt on top of asyncio, with serve_forever implicitly starting 
an asyncio loop with the server as the only task? Or are the paradigms too 
different?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: update 1

[Random832]

On Thu, May 23, 2019, at 15:27, Steve Holden wrote:
> Besides which, it would be lovely to have a major release that didn't 
> involve any pain at all for the majority of users!
> 
> Our erstwhile BDFL always eschewed two-digit version identifiers- due 
> to the possibility for confusion about collating sequence, I beleive.. 
> We should honour his preferences by going from 3.9 to 4.0.

Is 4.0 still gonna install as "python3", pip3, etc, executable as "py -3" on 
windows? I can see that being a pain point even if nothing else is.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Random832]

On Wed, May 29, 2019, at 01:25, Nick Coghlan wrote:
> Having a single locals() call de-optimize an entire function would be 
> far from ideal.

What if there were a way to explicitly de-optimize a function, rather than 
guessing the user's intent based on looking for locals and exec calls (both of 
which are builtins which could be shadowed or assigned to other variables)?

Also, regardless of anything else, maybe in an optimized function locals should 
return a read-only mapping?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: Comparing dict.values()

[Random832]

On Fri, Jul 26, 2019, at 00:22, Ivan Pozdeev via Python-Dev wrote:
> Since a hash table is an unordered container and keys(), items() and 
> values() are iterators over it, *I would expect the results of any of 
> the comparisons to be undefined.*

keys, items, and values are not iterators. They are view objects, and keys and 
items implement the entire set API (in the same sense that range implements the 
entire sequence API). Values is the odd one out in that it can contain multiple 
instances of each object, so it can't really be considered as a set.

Items also sometimes contains unhashable types, and some methods simply fail in 
that case. I suggest that this precedent provides a way forward - implement the 
entire intuitive "contains the same amount of each value" algorithm [more or 
less Counter(obj1) == Counter(obj2)], and have this fail naturally, throwing 
e.g. an exception "TypeError: unhashable type: 'list'" if any of the values are 
unhashable in the same way that trying to perform certain set operations on an 
items view does.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UCW7SGSZPY2TNXYR2W7AB7NKAVTULZVT/

[Python-Dev] Re: Comparing dict.values()

[Random832]

On Fri, Jul 26, 2019, at 05:45, Steven D'Aprano wrote:
> Nor should we make too much of the fact that Python sets require 
> elements to be hashable. As Java TreeSet demonstrates, we could get an 
> efficient set of unhashable items if we required orderability; and we 
> can get sets of unhashable, unorderable items if we're willing to 
> compromise on efficiency.

And think of what we could do if we were willing to compromise on immutability 
of hashable objects (Java does, in general, make the opposite decision there)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JJDKUFGILK2ESMUZULDB4I2VJFX3VQRC/

[Python-Dev] Re: What to do about invalid escape sequences

[Random832]

On Mon, Aug 5, 2019, at 00:29, raymond.hettin...@gmail.com wrote:
> The warning crops up frequently, often due to 
> third-party packages (such as docutils and bottle) that users can't 
> easily do anything about.  And during live demos and student workshops, 
> it is especially distracting.

Maybe what we really need is a general solution for warning visibility in 
third-party packages. Isn't that the whole reason DeprecationWarning isn't 
visible by default, for that matter?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GUULUXKYZN3UGWPEUPFKKEKNIZOOYJTJ/

[Python-Dev] Re: os.scandir bug in Windows?

[Random832]

On Mon, Oct 19, 2020, at 07:42, Steve Dower wrote:
> On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
> > TLDR: In os.scandir directory entries, atime is always a copy of mtime 
> > rather than the actual access time.
> 
> Correction - os.stat() updates the access time to _now_, while 
> os.scandir() returns the last access time without updating it.

This is surprising - do we know why this happens?

Also, it doesn't seem true on my system with python 3.8.5 [and, yes, I checked 
that last access update is enabled for my test and updates normally when 
reading the file's contents].
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GX3KD4UQKJONCLOZY743WXNGENXL7YG2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 638: Syntactic macros

[Random832]

On Fri, Oct 16, 2020, at 18:59, Dan Stromberg wrote:
> The complexity of a language varies with the square of its feature 
> count,

Says who? I'd assume the orthogonality and regularity of features matters at 
least as much if not more than the number of features, and providing a system 
like this would guarantee some degree of regularity.

Is there some notion of "complexity of a language" [other than by trivially 
*defining* it as the square of the number of features] for which this can be 
shown to be true?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JVJDRCNNSDMDIIFWJ65MBQVTWC375SJW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Random832]

On Tue, Oct 20, 2020, at 07:42, Steve Dower wrote:
> On 20Oct2020 0520, Rob Cliffe wrote:
> > On 19/10/2020 12:42, Steve Dower wrote:
> >> On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
> >>> TLDR: In os.scandir directory entries, atime is always a copy of 
> >>> mtime rather than the actual access time.
> >>
> >> Correction - os.stat() updates the access time to _now_, while 
> >> os.scandir() returns the last access time without updating it.
> >>
> >> Eryk replied with a deeper explanation of the cause, but fundamentally 
> >> this is what you are seeing.
> >>
> >> Feel free to file a bug, but we'll likely only add a vague note to the 
> >> docs about how Windows works here rather than changing anything. If 
> >> anything, we should probably fix os.stat() to avoid updating the 
> >> access time so that both functions behave the same, but that might be 
> >> too complicated.
> >>
> >> Cheers,
> >> Steve
> > Sorry - what you say does not match the behaviour I observe, which is that
> 
> Yes, I posted a correction already (immediately after sending the first 
> email).

ok, see, the correction you posted doesn't address the part of your claim that 
people are taking issue with, which is that *calling os.stat() causes the atime 
to be set to the time of the call to os.stat()*. This is not the same thing as 
[correctly] saying that "calling os.stat() may return a more up-to-date atime, 
the time of the last read, write, or other operation", and the phrasing 
"updates the access time to _now_" certainly *seemed* unambiguous.

And at this point it's not clear to me whether you understand that people are 
reading your claim this way.

What correction, exactly, do you mean? The post I saw with the word 
"Correction" on it is the one that *makes* the claim people are taking issue 
with.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O63FMQYOHASHZ33CWBYQMD3H3XYGT5QC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: os.scandir bug in Windows?

[Random832]

On Fri, Oct 23, 2020, at 02:14, Random832 wrote:
> What correction, exactly, do you mean? The post I saw with the word 
> "Correction" on it is the one that *makes* the claim people are taking 
> issue with.

okay, sorry, I see the other correction post now...

My issue I guess was the same as Eryk Sun, it wasn't clear which parts of the 
previous post you were correcting and which (if any) you stood by, since they 
were about the behavior of different parts of the system, so it didn't register 
as a correction to that part when I originally read it.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U4MZFDDMM4L52DKA6NBB7MKRJJ7QWEOB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules

[Random832]

On Mon, Jan 25, 2021, at 18:44, Chris Jerdonek wrote:
> But to issue a warning when a standard module is being overridden like 
> I was suggesting, wouldn’t you also need to know whether the name of 
> the module being imported is a standard name, which is what 
> says.module_names provides?

I don't think the warning would be only useful for stdlib modules... has any 
thought been given to warning when a module being imported from the current 
directory / script directory is the same as an installed package?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OG7EUYVNNLPZIXRAX23DPK6WIWRKUX5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why aren't we allowing the use of C11?

[Random832]

On Thu, Jan 28, 2021, at 22:57, Emily Bowman wrote:
>  
> On Thu, Jan 28, 2021 at 1:31 PM MRAB  wrote:
> > I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free) 
> > and it supports C11 and C17. 
> 
> While an upgrade for Community is free, for Pro/Enterprise without an 
> annual license it's not.

Something that's hard for me to find a clear answer for in a few minutes of 
searching:

Does the current version of the *Windows SDK* still come with a compiler, and 
can it be used without the Visual Studio IDE [the situation is confusing 
because it seems to install through the visual studio installer now] or at 
least without a license for Visual Studio Pro/Enterprise [i.e. by users who do 
not qualify for community edition] or with earlier versions of the Visual 
Studio IDE?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LUOB5H2S6PTHLZEDJVFUZTXLETS54KNX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Have virtual environments led to neglect of the actual environment?

[Random832]

I was reading a discussion thread 
 about various 
issues with the Debian packaged version of Python, and the following statement 
stood out for me as shocking:

Christian Heimes wrote:
> Core dev and PyPA has spent a lot of effort in promoting venv because we 
> don't want users to break their operating system with sudo pip install.

I don't think sudo pip install should break the operating system. And I think 
if it does, that problem should be solved rather than merely advising users 
against using it. And why is it, anyway, that distributions whose package 
managers can't coexist with pip-installed packages don't ever seem to get the 
same amount of flak for "damaging python's brand" as Debian is getting from 
some of the people in the discussion thread? Why is it that this community is 
resigned to recommending a workaround when distributions decide the 
site-packages directory belongs to their package manager rather than pip, 
instead of bringing the same amount of fiery condemnation of that practice as 
we apparently have for *checks notes* splitting parts of the stdlib into 
optional packages? Why demand that pip be present if we're not going to demand 
that it works properly?

I think that installing packages into the actual python installation, both via 
distribution packaging tools and pip [and using both simultaneously - the 
Debian model of separated dist-packages and site-packages folders seems like a 
reasonable solution to this problem] can and should be a supported paradigm, 
and that virtual environments [or more extreme measures such as shipping an 
entire python installation as part of an application's deployment] should 
ideally be reserved for the rare corner cases where that doesn't work for some 
reason.

How is it that virtual environments have become so indispensable, that no-one 
considers installing libraries centrally to be a viable model anymore? Are 
library maintainers making breaking changes too frequently, reasoning that if 
someone needs the old version they can just venv it? Is there some other cause?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KMRNKSVEPHAXA7BHFT4PWX4EKWYUF4G7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Random832]

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
> Separate directories don't prevent clashes and system breakage. But they
> provide an easy way to *recover* from a broken system.

I think it could be turned into a way to prevent them by A) having 
site-packages always take precedence over dist-packages [i believe this is 
already the case] in normal usage and B) providing an option to the 
interpreter, used by system scripts, to exclude site-packages entirely from the 
path.

Basically, site-packages would effectively be layered on top of "Lib + 
dist-packages" in a similar way to how a venv is layered on top of the main 
python installation - the inverse of the suggestion someone else in the thread 
made for the system python to be a venv. This wouldn't *exactly* be a venv 
because it wouldn't imply the other things that entering a venv does such as 
"python" [and script names such as pip] being an alias for the correct version 
of python, but it would provide the same kind of one-way isolation, whereby the 
"system environment" can influence the "normal environment" and not vice-versa, 
in the same way that packages installed in the main installation affect a venv 
[unless system-site-packages is disabled] but the venv obviously has no effect 
on the main installation.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K3DB2MSBWPDEK3O4UZ5VEQVKW7BFZDRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Python-ideas] Re: Re: Have virtual environments led to neglect of the actual environment?

[Random832]

On Wed, Feb 24, 2021, at 09:08, Paul Moore wrote:
> I don't use Linux much, and I'm definitely not familiar with Linux
> distribution tools, but from what I can gather Linux distributions
> have made the choices:
> 
> 1. Write key operating system utilities in Python.
> 2. Share the Python interpreter and libraries.
> 3. Expose that Python interpreter as the *user's* default Python.

I think 1 *partially* mischaracterizes the problem, because any "system python" 
would logically be used by *every application written in python [or that embeds 
python] distributed by the OS's package management*, not just by "key operating 
system utilities". To suggest otherwise implies that they should not distribute 
any python applications at all.

That also makes asking all of their package maintainers to change their #! line 
to point at a different interpreter [or to pass an option, as I had suggested 
in another post] a more significant ask than the "just change a few core 
utilities" that some people seem to be assuming it would be. It also means that 
making a "system python" does not remove the need to distribute the largish 
subset of python *libraries* that they distribute, because even when these 
libraries aren't used by key OS utilities, they are still used by packaged 
applications.

[this, in turn, means we don't want the user's default python environment to 
stand entirely separate from the system python, or we'll start getting 
complaints along the lines of "I apt-get installed numpy, why can't I import it 
in my python interpreter?" - particularly from users who don't really care if 
it runs a couple versions behind]
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F7YNRI7LWU2VLS6AU6PQZUHHA3M5WFQJ/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Fri, Jun 10, 2016, at 15:54, Theodore Ts'o wrote:
> So even on Python pre-3.5.0, realistically speaking, the "weakness" of
> os.random would only be an issue (a) if it is run within the first few
> seconds of boot, and (b) os.random is used to directly generate a
> long-term cryptographic secret.  If you are fork openssl or ssh-keygen
> to generate a public/private keypair, then you aren't using os.random.

So, I have a question. If this "weakness" in /dev/urandom is so
unimportant to 99% of situations... why isn't there a flag that can be
passed to getrandom() to allow the same behavior?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stop using timeit, use perf.timeit!

[Random832]

On Fri, Jun 10, 2016, at 21:45, Steven D'Aprano wrote:
> If you express your performances as speeds (as "calculations per 
> second") then the harmonic mean is the right way to average them.

That's true in so far as you get the same result as if you were to take
the arithmetic mean of the times and then converted from that to
calculations per second. Is there any other particular basis for
considering it "right"?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Sat, Jun 11, 2016, at 22:37, Theodore Ts'o wrote:
> On Fri, Jun 10, 2016 at 05:14:50PM -0400, Random832 wrote:
> > So, I have a question. If this "weakness" in /dev/urandom is so
> > unimportant to 99% of situations... why isn't there a flag that can be
> > passed to getrandom() to allow the same behavior?
> 
> The intention behind getrandom() is that it is intended *only* for
> cryptographic purposes. 

I'm somewhat confused now because if that's the case it seems to
accomplish multiple unrelated things. Why was this implemented as a
system call rather than a device (or an ioctl on the existing ones)? If
there's a benefit in not going through the non-atomic (and possibly
resource limited) procedure of acquiring a file descriptor, reading from
it, and closing it, why is that benefit not also extended to
non-cryptographic users of urandom via allowing the system call to be
used in that way?

> Anyway, if you don't need cryptographic guarantees, you don't need
> getrandom(2) or getentropy(2); something like this will do just fine:

Then what's /dev/urandom *for*, anyway?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does base64 return bytes?

[Random832]

On Tue, Jun 14, 2016, at 13:05, Joao S. O. Bueno wrote:
> Sorry, it is 2016, and I don't think at this point anyone can consider
> an ASCII string
> as a representative pattern of textual data in any field of application.
> Bytes are not text. Bytes with an associated, meaningful, encoding are
> text.
>   I thought this had been through when Python 3 was out.

Of all the things that anyone has said in this thread, this makes the
*least* contextual sense. The input to base64 encoding, which is what is
under discussion, is not text in any way. It is images, it is zip files,
it is executables, it could be the output of os.urandom (at least,
provided it doesn't block ;) for all anyone cares.

The *output* is only an ascii string in the sense that it is a text
string consisting of characters within (a carefully chosen subset of)
ASCII's repertoire, but the output wasn't what he was claiming should be
bytes in the sentence you replied to. Is your objection to the phrase
"ascii string"?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does base64 return bytes?

[Random832]

On Tue, Jun 14, 2016, at 13:19, Paul Sokolovsky wrote:
> Well, it's easy to remember the conclusion - it was decided to return
> bytes. The reason also wouldn't be hard to imagine - regardless of the
> fact that base64 uses ASCII codes for digits and letters, it's still
> essentially a binary data. 

Only in the sense that all text is binary data. There's nothing in the
definition of base64 specifying ASCII codes. It specifies *characters*
that all happen to be in ASCII's character repertoire.

>And the most natural step for it is to send
> it down the socket (socket.send() accepts bytes), etc.

How is that more natural than to send it to a text buffer that is
ultimately encoded (maybe not even in an ASCII-compatible encoding...
though probably) and sent down a socket or written to a file by a layer
that is outside your control? Yes, everything eventually ends up as
bytes. That doesn't mean that we should obsessively convert things to
bytes as early as possible.

I mean if we were gonna do that why bother even having a unicode string
type at all?

> I'd find it a bit more surprising that binascii.hexlify() returns
> bytes, but I personally got used to it, and consider it a
> consistency thing on binascii module.
> 
> Generally, with Python3 by default using (inefficient) Unicode for
> strings, 

Why is it inefficient?

> any efficient data processing would use bytes, and then one
> appreciates the fact that data encoding/decoding routines also return
> bytes, avoiding implicit expensive conversion to strings.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does base64 return bytes?

[Random832]

On Tue, Jun 14, 2016, at 22:58, Stephen J. Turnbull wrote:
> The RFC is unclear on this point, but I read it as specifying the
> ASCII coded character set, not the ASCII repertoire of (abstract)
> characters.  Therefore, it specifies an invertible mapping from a
> particular set of integers to characters.

There are multiple descriptions of base 64 that specifically mention
using it with EBCDIC and with local character sets of unspecified
nature.

>  > The intention is clearly to represent binary data as *text*.
> 
> It's more subtle than that.  *RFCs do not deal with text.*  Text is
> an internal concept of (some) programming environments.

It's also a human concept. Plenty of RFCs deal with human concept rather
than purely programming topics.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Thu, Jun 16, 2016, at 04:03, Barry Warsaw wrote:
> *If* we can guarantee that os.urandom() will never block or raise an
> exception when only poor entropy is available, then it may be indeed
> indistinguishably backward compatible for most if not all cases.

Why can't we exclude cases when only poor entropy is available from
"most if not all cases"?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Thu, Jun 16, 2016, at 10:04, Barry Warsaw wrote:
> On Jun 16, 2016, at 09:51 AM, Random832 wrote:
> 
> >On Thu, Jun 16, 2016, at 04:03, Barry Warsaw wrote:
> >> *If* we can guarantee that os.urandom() will never block or raise an
> >> exception when only poor entropy is available, then it may be indeed
> >> indistinguishably backward compatible for most if not all cases.  
> >
> >Why can't we exclude cases when only poor entropy is available from
> >"most if not all cases"?
>
> Because if it blocks or raises a new exception on poor entropy it's an
> API break.

Yes, but in only very rare cases. Which as I *just said* makes it
backwards compatible for "most" cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Our responsibilities (was Re: BDFL ruling request: should we block forever waiting for high-quality random bits?)

[Random832]

On Thu, Jun 16, 2016, at 07:34, Donald Stufft wrote:
>   python-dev tends to favor not breaking “working” code over securing
>   existing  APIs, even if “working” is silently doing the wrong thing
>   in a  security  context. This is particularly frustrating when it
>   comes to security  because  security is by it’s nature the act of
>   taking code that would  otherwise  execute and making it error,
>   ideally only in bad situations, but this  “security’s purpose is to
>   make things break” nature clashes with  python-dev’s  default of
>   not breaking “working” code in a way that is personally  draining
>   to me.

I was almost about to reply with "Maybe what we need is a new zen of
python", then I checked. It turns out we already have "Errors should
never pass silently" which fits *perfectly* in this situation. So what's
needed is a change to the attitude that if an error passes silently,
that making it no longer pass silently is a backward compatibility
break.

This isn't Java, where the exceptions not thrown by an API are part of
that API's contract. We're free to throw new exceptions in a new version
of Python.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Why are class dictionaries not accessible?

[Random832]

The documentation states: """Objects such as modules and instances have
an updateable __dict__ attribute; however, other objects may have write
restrictions on their __dict__ attributes (for example, classes use a
dictproxy to prevent direct dictionary updates)."""

However, it's not clear from that *why* direct dictionary updates are
undesirable. This not only prevents you from getting a reference to the
real class dict (which is the apparent goal), but is also the
fundamental reason why you can't use a metaclass to put, say, an
OrderedDict in its place - because the type constructor has to copy the
dict that was used in class preparation into a new dict rather than
using the one that was actually returned by __prepare__.

[Also, the name of the type used for this is mappingproxy, not
dictproxy]
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When to use EOFError?

[Random832]

On Tue, Jun 21, 2016, at 16:48, Serhiy Storchaka wrote:
> There is a design question. If you read file in some format or with some 
> protocol, and the data is ended unexpectedly, when to use general 
> EOFError exception and when to use format/protocol specific exception?
> 
> For example when load truncated pickle data, an unpickler can raise 
> EOFError, UnpicklingError, ValueError or AttributeError. It is possible 
> to avoid ValueError or AttributeError, but what exception should be 
> raised instead, EOFError or UnpicklingError? Maybe convert all EOFError 
> to UnpicklingError?

I think this is the most appropriate. If the calling code needs to know
the original reason it can find it in __cause__.

My instinct, though, (and I'm aware that others may not agree, but I
thought it was worth bringing up) is that loads should actually always
raise a ValueError, i.e. my mental model of loads is like:

def loads(s):
   f = BytesIO(s)
   try:
  return load(f)
   except UnpicklingError as e:
  raise ValueError from e
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com