Re: [Python-Dev] sum(...) limitation

[Chris Barker]

On Sat, Aug 2, 2014 at 1:35 PM, David Wilson  wrote:

> > Repeated list and str concatenation both have quadratic O(N**2)
> > performance, but people frequently build up strings with +
>


> join() isn't preferable in cases where it damages readability while
> simultaneously providing zero or negative performance benefit, such as
> when concatenating a few short strings, e.g. while adding a prefix to a
> filename.
>

Good point -- I was trying to make the point about .join() vs + for strings
in an intro python class last year, and made the mistake of having the
students test the performance.

You need to concatenate a LOT of strings to see any difference at all --  I
know that O() of algorithms is unavoidable, but between efficient python
optimizations and a an apparently good memory allocator, it's really a
practical non-issue.


> Although it's true that join() is automatically the safer option, and
> especially when dealing with user supplied data, the net harm caused by
> teaching rote and ceremony seems far less desirable compared to fixing a
> trivial slowdown in a script, if that slowdown ever became apparent.
>

and it rarely would.

Blocking sum( some_strings) because it _might_ have poor performance seems
awfully pedantic.

As a long-time numpy user, I think sum(a_long_list_of_numbers) has
pathetically bad performance, but I wouldn't block it!

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Chris Barker]

On Mon, Aug 4, 2014 at 11:10 AM, Steven D'Aprano 
wrote:

> On Mon, Aug 04, 2014 at 09:25:12AM -0700, Chris Barker wrote:
>
> > Good point -- I was trying to make the point about .join() vs + for
> strings
> > in an intro python class last year, and made the mistake of having the
> > students test the performance.
> >
> > You need to concatenate a LOT of strings to see any difference at all
>


> If only that were the case, but it isn't. Here's a cautionary tale for
> how using string concatenation can blow up in your face:
>
> Chris Withers asks for help debugging HTTP slowness:
> https://mail.python.org/pipermail/python-dev/2009-August/091125.html



Thanks for that -- interesting story. note that that was not suing sum() in
that case though, which is really the issue at hand.

It shouldn't be hard to demonstrate the difference between repeated
> string concatenation and join, all you need do is defeat sum()'s
> prohibition against strings. Run this bit of code, and you'll see a
> significant difference in performance, even with CPython's optimized
> concatenation:
>

well, that does look compelling, but what it shows is that
sum(a_list_of_strings) is slow compared to ''.join(a_list_of_stings). That
doesn't surprise me a bit -- this is really similar to why:

a_numpy_array.sum()

is going to be a lot faster than:

sum(a_numpy_array)

and why I'll tell everyone that is working with lots of numbers to use
numpy. ndarray.sum know what data type it's deaing with,a nd can do the
loop in C. similarly with ''.join() (though not as optimized.

But I'm not sure we're seeing the big O difference here at all -- but
rather the extra calls though each element in the list's __add__ method.

In the case where you already HAVE a big list of strings, then yes, ''.join
is the clear winner.

But I think the case we're often talking about, and I've tested with
students, is when you are building up a long string on the fly out of
little strings. In that case, you need to profile the full "append to list,
then call join()", not just the join() call:

# continued adding of strings ( O(n^2)? )
In [6]: def add_strings(l):
   ...: s = ''
   ...: for i in l:
   ...: s+=i
   ...: return s

Using append and then join ( O(n)? )
In [14]: def join_strings(list_of_strings):
   : l = []
   : for i in list_of_strings:
   : l.append(i)
   : return ''.join(l)

In [23]: timeit add_strings(strings)
100 loops, best of 3: 831 ns per loop

In [24]: timeit join_strings(strings)
10 loops, best of 3: 1.87 µs per loop

## hmm -- concatenating is faster for a small list of tiny strings

In [31]: strings = list('Hello World')* 1000

strings *= 1000
In [26]: timeit add_strings(strings)
1000 loops, best of 3: 932 µs per loop

In [27]: timeit join_strings(strings)
1000 loops, best of 3: 967 µs per loop

## now about the same.

In [31]: strings = list('Hello World')* 1

In [29]: timeit add_strings(strings)
100 loops, best of 3: 9.44 ms per loop

In [30]: timeit join_strings(strings)
100 loops, best of 3: 10.1 ms per loop

still about he same?

In [31]: strings = list('Hello World')* 100

In [32]: timeit add_strings(strings)
1 loops, best of 3: 1.27 s per loop

In [33]: timeit join_strings(strings)
1 loops, best of 3: 1.05 s per loop

there we go -- slight advantage to joining.

So this is why we've said that the common wisdom about string concatenating
isn't really a practical issue.

But if you already have the strings all in a list, then yes, join() is a
major win over sum()

In fact, I tried the above with sum() -- and it was really, really slow. So
slow I didn't have the patience to wait for it.

Here is a smaller example:

In [22]: strings = list('Hello World')* 1

In [23]: timeit add_strings(strings)
100 loops, best of 3: 9.61 ms per loop

In [24]: timeit sum( strings, Faker() )
1 loops, best of 3: 246 ms per loop

So why is sum() so darn slow with strings compared to a simple loop with +=
?

(and if I try it with a list 10 times as long it takes "forever")

Perhaps the http issue cited was before some nifty optimizations in current
CPython?

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Chris Barker]

On Thu, Aug 7, 2014 at 4:01 PM, Ethan Furman  wrote:

> I don't remember where, but I believe that cPython has an optimization
> built in for repeated string concatenation, which is probably why you
> aren't seeing big differences between the + and the sum().
>

Indeed -- clearly so.

A little testing shows how to defeat that optimization:

  blah = ''
>   for string in ['booyah'] * 10:
>   blah = string + blah
>
> Note the reversed order of the addition.
>

thanks -- cool trick.

Oh, and the join() timings:
> --> timeit.Timer("blah = ''.join(['booya'] * 10)", "blah =
> ''").repeat(3, 1)
> [0.0014629364013671875, 0.0014190673828125, 0.0011930465698242188]
> So, + is three orders of magnitude slower than join.


only one if if you use the optimized form of + and not even that if you
need to build up the list first, which is the common use-case.

So my final question is this:

repeated string concatenation is not the "recommended" way to do this --
but nevertheless, cPython has an optimization that makes it fast and
efficient, to the point that there is no practical performance reason to
prefer appending to a list and calling join()) afterward.

So why not apply a similar optimization to sum() for strings?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Chris Barker]

On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull 
wrote:

> I'm referring to removing the unnecessary information that there's a
>  better way to do it, and simply raising an error (as in Python 3.2,
> say) which is all a RealProgrammer[tm] should ever need!
>

I can't imagine anyone is suggesting that -- disallow it, but don't tell
anyone why?

The only thing that is remotely on the table here is:

1) remove the special case for strings -- buyer beware -- but consistent
and less "ugly"

2) add a special case for strings that is fast and efficient -- may be as
simple as calling "".join() under the hood --no more code than the
exception check.

And I doubt anyone really is pushing for anything but (2)

Steven Turnbull wrote:

>   IMO we'd also want a homogeneous_iterable ABC


Actually, I've thought for years that that would open the door to a lot of
optimizations -- but that's a much broader question that sum(). I even
brought it up probably over ten years ago -- but no one was the least bit
iinterested -- nor are they now -- I now this was a rhetorical suggestion
to make the point about what not to do

  Because obviously we'd want the
> attractive nuisance of "if you have __add__, there's a default
> definition of __sum__"


now I'm confused -- isn't that exactly what we have now?

It's possible that Python could provide some kind of feature that
> would allow an optimized sum function for every type that has __add__,
> but I think this will take a lot of thinking.


does it need to be every type? As it is the common ones work fine already
except for strings -- so if we add an optimized string sum() then we're
done.

 *Somebody* will do it
> (I don't think anybody is +1 on restricting sum() to a subset of types
> with __add__).


uhm, that's exactly what we have now -- you can use sum() with anything
that has an __add__, except strings. Ns by that logic, if we thought there
were other inefficient use cases, we'd restrict those too.

But users can always define their own classes that have a __sum__ and are
really inefficient -- so unless sum() becomes just for a certain subset of
built-in types -- does anyone want that? Then we are back to the current
situation:

sum() can be used for any type that has an __add__ defined.

But naive users are likely to try it with strings, and that's bad, so we
want to prevent that, and have a special case check for strings.

What I fail to see is why it's better to raise an exception and point users
to a better way, than to simply provide an optimization so that it's a mute
issue.

The only justification offered here is that will teach people that summing
strings (and some other objects?) is order(N^2) and a bad idea. But:

a) Python's primary purpose is practical, not pedagogical (not that it
isn't great for that)

b) I doubt any naive users learn anything other than "I can't use sum() for
strings, I should use "".join()". Will they make the leap to "I shouldn't
use string concatenation in a loop, either"? Oh, wait, you can use string
concatenation in a loop -- that's been optimized. So will they learn: "some
types of object shave poor performance with repeated concatenation and
shouldn't be used with sum(). So If I write such a class, and want to sum
them up, I'll need to write an optimized version of that code"?

I submit that no naive user is going to get any closer to a proper
understanding of algorithmic Order behavior from this small hint. Which
leaves no reason to prefer an Exception to an optimization.

One other point: perhaps this will lead a naive user into thinking --
"sum() raises an exception if I try to use it inefficiently, so it must be
OK to use for anything that doesn't raise an exception" -- that would be a
bad lesson to mis-learn

-Chris

PS:
Armin Rigo wrote:

> It also improves a
> lot the precision of sum(list_of_floats) (though not reaching the same
> precision levels of math.fsum()).


while we are at it, having the default sum() for floats be fsum() would be
nice -- I'd rather the default was better accuracy loser performance. Folks
that really care about performance could call math.fastsum(), or really,
use numpy...

This does turn sum() into a function that does type-based dispatch, but
isn't python full of those already? do something special for the types you
know about, call the generic dunder method for the rest.



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Chris Barker]

On Tue, Aug 12, 2014 at 11:21 PM, Stephen J. Turnbull 
wrote:

> Redirecting to python-ideas, so trimming less than I might.


reasonable enough -- you are introducing some more significant ideas for
changes.

I've said all I have to say about this -- I don't seem to see anything
encouraging form core devs, so I guess that's it.

Thanks for the fun bike-shedding...

-Chris





> Chris Barker writes:
>  > On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull <
> step...@xemacs.org>
>  > wrote:
>  >
>  > > I'm referring to removing the unnecessary information that there's a
>  > >  better way to do it, and simply raising an error (as in Python 3.2,
>  > > say) which is all a RealProgrammer[tm] should ever need!
>  > >
>  >
>  > I can't imagine anyone is suggesting that -- disallow it, but don't tell
>  > anyone why?
>
> As I said, it's a regression.  That's exactly the behavior in Python 3.2.
>
>  > The only thing that is remotely on the table here is:
>  >
>  > 1) remove the special case for strings -- buyer beware -- but consistent
>  > and less "ugly"
>
> It's only consistent if you believe that Python has strict rules for
> use of various operators.  It doesn't, except as far as they are
> constrained by precedence.  For example, I have an application where I
> add bytestrings bytewise modulo N <= 256, and concatenate them.  In
> fact I use function call syntax, but the obvious operator syntax is
> '+' for the bytewise addition, and '*' for the concatenation.
>
> It's not in the Zen, but I believe in the maxim "If it's worth doing,
> it's worth doing well."  So for me, 1) is out anyway.
>
>  > 2) add a special case for strings that is fast and efficient -- may be
> as
>  > simple as calling "".join() under the hood --no more code than the
>  > exception check.
>
> Sure, but what about all the other immutable containers with __add__
> methods?  What about mappings with key-wise __add__ methods whose
> values might be immutable but have __add__ methods?  Where do you stop
> with the special-casing?  I consider this far more complex and ugly
> than the simple "sum() is for numbers" rule (and even that is way too
> complex considering accuracy of summing floats).
>
>  > And I doubt anyone really is pushing for anything but (2)
>
> I know that, but I think it's the wrong solution to the problem (which
> is genuine IMO).  The right solution is something generic, possibly a
> __sum__ method.  The question is whether that leads to too much work
> to be worth it (eg, "homogeneous_iterable").
>
>  > > Because obviously we'd want the attractive nuisance of "if you
>  > > have __add__, there's a default definition of __sum__"
>  >
>  > now I'm confused -- isn't that exactly what we have now?
>
> Yes and my feeling (backed up by arguments that I admit may persuade
> nobody but myself) is that what we have now kinda sucks[tm].  It
> seemed like a good idea when I first saw it, but then, my apps don't
> scale to where the pain starts in my own usage.
>
>  > > It's possible that Python could provide some kind of feature that
>  > > would allow an optimized sum function for every type that has
>  > > __add__, but I think this will take a lot of thinking.
>  >
>  > does it need to be every type? As it is the common ones work fine
> already
>  > except for strings -- so if we add an optimized string sum() then we're
>  > done.
>
> I didn't say provide an optimized sum(), I said provide a feature
> enabling people who want to optimize sum() to do so.  So yes, it needs
> to be every type (the optional __sum__ method is a proof of concept,
> modulo it actually being implementable ;-).
>
>  > > *Somebody* will do it (I don't think anybody is +1 on restricting
>  > > sum() to a subset of types with __add__).
>  >
>  > uhm, that's exactly what we have now
>
> Exactly.  Who's arguing that the sum() we have now is a ticket to
> Paradise?  I'm just saying that there's probably somebody out there
> negative enough on the current situation to come up with an answer
> that I think is general enough (and I suspect that python-dev
> consensus is that demanding, too).
>
>  > sum() can be used for any type that has an __add__ defined.
>
> I'd like to see that be mutable types with __iadd__.
>
>  > What I fail to see is why it's better to raise an exception and
>  > point users to a better way, t

Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes & bytearray

[Chris Barker]

On Sun, Aug 17, 2014 at 2:41 PM, Barry Warsaw  wrote:

> I think the biggest API "problem" is that default iteration returns
> integers
> instead of bytes.  That's a real pain.
>

what is really needed for this NOT to be a pain is a byte scalar.

numpy has a scalar type for every type it supports -- this is a GOOD THING
(tm):

In [53]: a = np.array((3,4,5), dtype=np.uint8)

In [54]: a
Out[54]: array([3, 4, 5], dtype=uint8)

In [55]: a[1]
Out[55]: 4

In [56]: type(a[1])
Out[56]: numpy.uint8

In [57]: a[1].shape
Out[57]: ()


The lack of a  character type is a major source of "type errors" in python
(the whole list of strings vs a single string problem -- both return a
sequence when you index into them or iterate over them)

Anyway, the character ship has long since sailed, but maybe a byte scalar
would be a good idea?

And FWIW, I think the proposal does make for a better, cleaner API.

Whether that's worth the deprecation is not clear to me, though as someone
whose been on the verge of making the leap to 3.* for ages, this isn't
going to make any difference.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes & bytearray

[Chris Barker]

On Mon, Aug 18, 2014 at 1:06 PM, Terry Reedy  wrote:

> The byte scalar is an int in range(256). Bytes is an array of such.
>

then why the complaint about iterating over bytes producing ints? Ye,s a
byte owuld be pretty much teh same as an int, but it would have
restrictions - useful ones.

 numpy has a scalar type for every type it supports -- this is a GOOD
>> THING (tm):
>> In [56]: type(a[1])
>> Out[56]: numpy.uint8
>>
>> In [57]: a[1].shape
>> Out[57]: ()
>>
>>
>> The lack of a  character type is a major source of "type errors" in
>> python (the whole list of strings vs a single string problem -- both
>> return a sequence when you index into them or iterate over them)
>>
>
> This is exactly what iterbytes would do  -- yields bytes of size 1.


as I understand it, it would yield a bytes object of length one -- that is
a sequence that _happens_ to only have one item in it -- not the same thing.

Note above. In numpy, when you index out of a 1-d array you get a scalar --
with shape == ()  -- not a 1-d array of length 1. And this is useful, as it
provide s clear termination point when you drill down through multiple
dimensions.

I often wish I could do that with nested lists with strings at the bottom.

[1,2,3] is a sequence of numbers

"this" is a sequence of characters -- oops, not it's not, it's a sequence
of sequences of sequences of ...

I think it would be cleaner if bytes was a sequence of a scalar byte object.

This is a bigger deal for numpy, what with its n-dimensional arrays and
many reducing operations, but the same principles apply.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Chris Barker]

>
>  but disallowing them in higher level
>> > explicitly cross platform abstractions like pathlib.
>>
>
I think the trick here is that posix-using folks claim that filenames are
just bytes, and indeed they can be passed around with a char*, so they seem
to be.

but you can't possible do anything other than pass them around if you
REALLY think they are just bytes.

So really, people treat them as
"bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and
maybe a couple others)-is-ascii-compatible"

If you assume that, then you could write a pathlib that would work. And in
practice, I expect a lot of designed only for posix code works that way.
But of course, this gets ugly if you go to a platform where filenames are
not "bytes-in-some-arbitrary-encoding-where-at-least
the-slash-character-(and maybe a couple others)-is-ascii-compatible", like
windows.

I'm not sure if it's worth having a pathlib, etc. that uses this assumption
-- but it could help us all write code that actually works with this screwy
lack of specification.

 Antoine Pitrou wrote:

> To elaborate specifically about pathlib, it doesn't handle bytes paths
> but allows you to generate them if desired:
> https://docs.python.org/3/library/pathlib.html#operators


but that uses

os.fsencode:  Encode filename to the filesystem encoding

As I understand it, the whole problem with some posix systems is that there
is NO filesystem encoding -- i.e. you can't know for sure what encoding a
filename is in. So you need to be able to pass the bytes through as they
are.

(At least as I read Armin Ronacher's blog)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Chris Barker]

On Wed, Aug 20, 2014 at 9:52 PM, Cameron Simpson  wrote:

> On 20Aug2014 16:04, Chris Barker - NOAA Federal 
> wrote:
>
>>

>  So really, people treat them as
>>>
>> "bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and
>> maybe a couple others)-is-ascii-compatible"
>>
>
> As someone who fought long and hard in the surrogate-escape listdir()
> wars, and was won over once the scheme was thoroughly explained to me, I
> take issue with these assertions: they are bogus or misleading.
>
> Firstly, POSIX filenames _are_ just byte strings. The only forbidden
> character is the NUL byte, which terminates a C string, and the only
> special character is the slash, which separates pathanme components.
>

so they are "just byte strings", oh, except that you can't have a  null,
and the "slash" had better be code 47 (and vice versa). How is that
different than "bytes-in-some-arbitrary-encoding-where-at-least
the-slash-character-is-ascii-compatible"?

(sorry about the "maybe a couple others", I was too lazy to do my research
and be sure).

But my point is that python users want to be able to work with paths, and
paths on posix are not strictly strings with a clearly defined encoding,
but they are also not quite "just arbitrary bytes". So it would be nice if
we could have a pathlib that would work with these odd beasts. I've lost
track a bit as to whether the surrogate-escape solution allows this to all
work now. If it does, then great, sorry for the noise.

Second, a bare low level program cannot do _much_ more than pass them
> around.  It certainly can do things like compute their basename, or other
> path related operations.
>

only if you assume that pesky slash == 47 thing -- it's not much, but it's
not raw bytes either.

The "bytes in some arbitrary encoding where at least the slash character
> (and
> maybe a couple others) is ascii compatible" notion is completely bogus.
> There's only one special byte, the slash (code 47). There's no OS-level
> need that it or anything else be ASCII compatible. I think
> characterizations such as the one quoted are activately misleading.
>

code 47 == "slash" is ascii compatible -- where else did the 47 value come
from?


> I think we'd all agree it is nice to have a system where filenames are all
> Unicode, but since POSIX/UNIX predates it by decades it is a bit late to
> ignore the reality for such systems.


well, the community could have gone to "if you want anything other than
ascii, make it utf-8 -- but always, we're all a bunch of independent
thinkers.

But none of this is relevant -- systems in the wild do what they do --
clearly we all want Python to work with them as best it can.


> There's no _external_ "filesystem encoding" in the sense of something
> recorded in the filesystem that anyone can inspect. But there is the
> expressed locale settings, available at runtime to any program that cares
> to pay attention. It is a workable situation.
>

I haven't run into it, but it seem the folks that have don't think relying
on the locale setting is the least bit workable. If it were, we woldn't be
havin this discussion -- use the locale setting to decide how to decode
filenames -- done.

Oh, and I reject Nick's characterisation of POSIX as "broken". It's
> perfectly internally consistent. It just doesn't match what he wants.
> (Indeed, what I want, and I'm a long time UNIX fanboy.)
>

bug or feature? you decide. Internal consistency is a good start, but it
punts the whole encoding issue to the client software, without giving it
the tools to do it right. I call that "really hard to work with" if not
broken.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Chris Barker]

On Fri, Aug 22, 2014 at 10:09 AM, Glenn Linderman 
wrote:

> What encoding does have a text file (an HTML, to be precise) with
> text in utf-8, ads in cp1251 (ad blocks were included from different
> files) and comments in koi8-r?
>Well, I must admit the HTML was rather an exception, but having a
> text file with some strange characters (binary strings, or paragraphs
> in different encodings) is not that exceptional.
>
>  That's not a text file. That's a binary file containing (hopefully
> delimited, and documented) sections of encoded text in different
> encodings.
>
> Allow me to disagree. For me, this is a text file which I can (and
> do) view with a pager, edit with a text editor, list on a console,
> search with grep and so on. If it is not a text file by strict Python3
> standards then these standards are too strict for me. Either I find a
> simple workaround in Python3 to work with such texts or find a different
> tool. I cannot avoid such files because my reality is much more complex
> than strict text/binary dichotomy in Python3.
>
> First -- we're getting OT here -- this thread was about file and path
names, not the contents of files. But I suppose I brought that in when I
talked about writing file names to files...

The first I'll mention is the one that follows from my description of what
> your file really is: Python3 allows opening files in binary mode, and then
> decoding various sections of it using whatever encoding you like, using the
> bytes.decode() operation on various sections of the file. Determination of
> which sections are in which encodings is beyond the scope of this
> description of the technique, and is application dependent.
>

right -- and you would have wanted to open such file in binary mode with
py2 as well, but in that case, you's have the contents in py2 string
object, which has a few more convenient ways to work with text (at least
ascii-compatible) than the py3 bytes object does.

The third is to specify the UTF-8 with the surrogate escape error handler.
> This allows non-UTF-8 codes to be loaded into memory. You, or algorithms as
> smart as you, could perhaps be developed to detect and manipulate the
> resulting "lone surrogate" codes in meaningful ways, or could simply allow
> them to ride along without interpretation, and be emitted as the original,
> into other files.
>

Just so I'm clear here -- if you write that back out, encoded as utf-8 --
you'll get the exact same binary blob out as came in?

I wonder if this would make it hard to preserve byte boundaries, though.

By the way, IIUC correctly, you can also use the python latin-1 decoder --
anything latin-1 will come through correctly, anything not valid latin-1
will come in as garbage, but if you re-encode with latin-1 the original
bytes will be preserved. I think this will also preserve a 1:1 relationship
between character count and byte count, which could be handy.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Chris Barker]

On Thu, Aug 21, 2014 at 7:42 PM, Oleg Broytman  wrote:

> On Thu, Aug 21, 2014 at 05:30:14PM -0700, Chris Barker - NOAA Federal <
> chris.bar...@noaa.gov> wrote:
> > This brings up the other key problem. If file names are (almost)
> > arbitrary bytes, how do you write one to/read one from a text file
> > with a particular encoding? ( or for that matter display it on a
> > terminal)
>
>There is no such thing as an encoding of text files. So we just
> write those bytes to the file


So I write bytes that are encoded one way into a text file that's encoded
another way, and expect to be abel to read that later? you're kidding,
right? Only if that's  he only thing in the file -- usually not the case
with my text files.

or output them to the terminal. I often do
> that. My filesystems are full of files with names and content in
> at least 3 different encodings - koi8-r, utf-8 and cp1251. So I open a
> terminal with koi8 or utf-8 locale and fonts and some file always look
> weird. But however weird they are it's possible to work with them.
>

Not for me (or many other users) -- terminals are sometimes set with
ascii-only encoding, so non-ascii barfs -- or you get some weird control
characters that mess up your terminal -- dumping arbitrary bytes to a
terminal does not always "just work".


> > And people still want to say posix isn't broken in this regard?
>
>Not at all! And broken or not broken it's what I (for many different
> reasons) prefer to use for my desktops, servers, notebooks, routers and
> smartphones,


Sorry -- that's a Red Herring -- I agree, "broken" or "simple and
consistent" is irrelevant, we all want Python to work as well as it can on
such systems.

The point is that if you are reading a file name from the system, and then
passing it back to the system, then you can treat it as just bytes -- who
cares? And if you add the byte value of 47 thing, then you can even do
basic path manipulations. But once you want to do other things with your
file name, then you need to know the encoding. And it is very, very common
for users to need to do other things with filenames, and they almost always
want them as text that they can read and understand.

Python3 supports this case very well. But it does indeed make it hard to
work with filenames when you don't know the encoding they are in. And
apparently that's pretty common -- or common enough that it would be nice
for Python to support it well. This trick is how -- we'd like the "just
pass it around and do path manipulations" case to work with (almost)
arbitrary bytes, but everything else to work naturally with text (unicode
text).

Which brings us to the "what APIs should accept bytes" question. I think
that's been pretty much answered: All the low-level ones, so that protocol
and library programmers can write code that works on systems with undefined
filename encodings.

But: casual users still need to do the normal things with file names and
paths, and ideally those should work the same way on all systems.

I think the way to do this is to abstract the path concept, like pathlib
does. Back in the day, paths were "just strings", and that worked OK with
py2 strings, because you could put arbitrary bytes in them. But the "py2
strings were perfect" folks seem to not acknowledge that while they are
nice for matching the posix filename model, they were a pain in the neck
when you needed to do somethign else like write them in to a JSON file or
something. From my personal experience, non-ascii filenames are much easier
to deal with if I use unicode for filenames everywhere (py2). Somehow, I
have yet to be bitten by mixed encoding in filenames.

So will using a surrogate-escape error handling with pathlib make all this
just work?

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7

[Chris Barker]

On Sun, Aug 31, 2014 at 3:00 PM, Nick Coghlan  wrote:
>
> However, we still think we should start providing pip by default to Python
> 2.7 users as well, at least as part of the Windows and Mac OS X installers.
>
serious +1 here.

Just last night I was writing up notes for an intro to python class on
getting started. (py2, :-( )

The "go download get-pip.py and run it with python" step(s) is a serious
hiccup. Better than it's been for years, but sure would be nice to just
"install python" then "use pip to install packages"

As for the feature freeze of py2.7 -- this is really an addition to teh
installers, not to python itself (or it can be seen that way, anyway)

Thanks Ned for offering to back port for OS-X !

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4.2rc1 Mac OS X

[Chris Barker]

On Tue, Sep 23, 2014 at 10:27 AM, Ned Deily  wrote:

> Thanks for the feedback, Carol.  Let us know via bugs.python.org of any
> issues you see.  BTW, the new installer format will be coming to Python
> 2.7.9 as well.
>

Are the supported platforms going to be the same? i.e.:

10.3+ -- Intel32+PPC32 (Or is this deprecated already?)

10.6+ -- Intel32_Intel64

I'm actually wondering if it's time for a pure Intel64, probably 10.7+

Building extensions for Universal builds and older OS versions really is a
pain...

-Chris






-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Chris Barker]

On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft  wrote:

> 1) Just always default to —user and add a —system or similar flag, this
> is super easy to change but is a backwards incompatible change and
> would need to go through a deprecation window.
>

Maybe would have been the way to go to begin with, but I think backwards
compatibility should probably trump "better" -- even with a deprecation
window.


> 2) Switch to —user based on if the user has permission to write to the
> site-packages or not.
>

ouch -- no. Why not a clear error message if pip can't write to
site-packages -- something like:

"""
pip doesn't have permissions to write to the system location. If you want
to install this package system-wide, you need to run pip with admin
priviledges (and example here if it's easy), if you want to install for
this user only, pass the "--user" flag to pip install
"""

"In the face of ambiguity, refuse the temptation to guess."

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Chris Barker]

On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft  wrote:

> 2) Switch to —user based on if the user has permission to write to the
>> site-packages or not.
>>
>
> ouch -- no. Why not a clear error message if pip can't write to
> site-packages -- something like:
>
> I fairly strongly believe that the current default is doing a great
> disservice
> to users. I believe that for *most* people --user is the correct option for
> them to be using and the fact that it's not the default and requires opt in
> is a historical artifact more than anything else.
>

OK -- fine -- I think that history may be important -- at least for Py2,
but I agree that --user is a better default.

But what I'm objecting to is is switching the install mode based on the
permissions that the user happens to be running with at the time.

None of us should be routinely running as admin. So what I do (and I image
a LOT of people do) is try to do whatever I need to do, and only if I get
an error to I switch to admin mode (or add sudo, or ???). What I'm
suggesting is that folks that simply expect stuff to get installed into the
usual system python is not thinking ahead of time "oh, I need be admin for
this", but rather, simply do it, and then, if you get a permission error,
make the switch to either admin in mode, or add --user.

If we do switch the default to --user, then this issue goes away.

In short -- too much magic is bad.

All this is also making me think that virtualenv and friends is the real
solution to user installed packages anyway.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[Chris Barker]

OK, this seems weird to me:

For what it’s worth, I almost exclusively write 2/3 compatible code (and
> that’s
> with the “easy” subset of 2.6+ and either 3.2+ or 3.3+)


ouch.


>  However the way it "used" to work
> is that the newest version, with all the new features, would quickly become
> the dominant version within a year or two.


This seems completely contradictory to me: Yes, the 3.* transition can be
difficult, thus the need to support 1.*. But if you are still supporting
2.6, then clearly "the newest version, with all the new features, would
quickly become
the dominant version within a year or two"

But there are those use cases that seem to require sticking with old
version for ages, even if there have not been substantial incomparable
changes.

So we could be on version 2.12 now, and you'd still need to support 2.6,
and still be working in a legacy, least common denominator language. How
does this have anything to do with the 3.* transition?

But plenty of us are kind of stuck on 2.7 at this point -- we can upgrade,
but can't accommodate a major shift (for me it's currently wxPython that's
the blocker -- that may be the only one. Others are either supported or
small enough that we could handle the port ourselves.)

But anyway, if you don't hate 2.6 back in the day, why hate it now?

(yet, I know Donald didn't use the "hate" word).

I guess my pint is that you either would much prefer to be working with the
latest and greatest cool features or not -- but if you do the problem at
this point isn't anything about py3, it's about the fact that many of us
are required to support old versions, period.

-Chris


 However I can't really justify for most situations supporting _only_ those
> things because even today they are not the dominant version (or really
> close
> to it in any number I have access too). This means that if I want to take
> advantage of something newer I'm essentially dropping the largest part of
> the ecosystem.
>

Are you primarily writing packages for others to use? if so, then yes. But
I wonder how many people are in that camp? Don't most of us spend most of
our time writing our own purpose-built code?

That might be a nice thing to see in a survey, actually.


-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Chris Barker]

On Sat, Jan 10, 2015 at 3:28 AM, Nick Coghlan  wrote:

> For the time being, things like PyInstaller, PyRun, Portable Python,
> etc are going to offer a better solution than anything we provide in
> the standard installers.
>

See also Anaconda and Enthought Canopy. I think miniconda, for instance,
may give you just what you need, if you don't want to go the
py2exe/PyInstaller approach (though you probably do want to go that way, as
far as I can tell from your description of your use-case.

I'm inclined to think that this does not belong as part of the standard
installer.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 485 review (isclose())

[Chris Barker]

Thank you Guido.

It'll be nice to see this all come to something.

Thanks to all who contributed to the discussion -- despite this being a
pretty simple function, I learned a lot and far more fully appreciate the
nuance of all of this.

I'll edit the text as you suggest, and then work on a patch -- I'm sure
I'll have questions for Python-dev when I actually do that, but I'll get
started on my  own and see how far I get.

-Chris





On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum  wrote:

> I think it's time to accept PEP 485. I've re-read it once more, and it
> looks like the text is in great shape. (My only recommendation would be to
> update the Abstract to state that we're specifically adding math.isclose().)
>
> A wording question: "This implementation has a flag that lets the user
> select which relative tolerance test to apply -- this PEP does not suggest
> that that be retained, but rather than the weak test be selected." -- I
> think this was meant to say "... rather *that* the weak test be selected",
> right? (It would be nice if the sample implementation defaulted to the
> choice in the PEP.)
>
> However, those are just minor edits, and I hereby approve the PEP. Thanks
> Chris and everyone else for the fruitful discussion (and thanks especially
> to Chris for eventually ending the bikeshedding and writing a PEP that
> explains each of the choices). Congrats!
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 485 review (isclose())

[Chris Barker]

On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker 
wrote:

> I'll edit the text as you suggest,
>

Done.


> and then work on a patch -- I'm sure I'll have questions for Python-dev
> when I actually do that, but I'll get started on my  own and see how far I
> get.
>

OK --  big question 1:

As far as I can tell, the math module is entirely a C extension. So I can:

1) write isclose() in C -- I could probably do that (It's been a while --
I'm a big Cython fan these days...) and I'd probably punt on having it work
with anything other than floats if I did that -- which would fit how most
of the match module works anyway.

2) Write it in Python, and monkey-patch it in to the math module -- I
honestly have no idea how to do that, but as I can add a new name to the
math module after importing it, it should be doable --but I have no idea
where the code would go.

Suggestions??

Thanks,
  -Chris








> -Chris
>
>
>
>
>
> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum 
> wrote:
>
>> I think it's time to accept PEP 485. I've re-read it once more, and it
>> looks like the text is in great shape. (My only recommendation would be to
>> update the Abstract to state that we're specifically adding math.isclose().)
>>
>> A wording question: "This implementation has a flag that lets the user
>> select which relative tolerance test to apply -- this PEP does not suggest
>> that that be retained, but rather than the weak test be selected." -- I
>> think this was meant to say "... rather *that* the weak test be selected",
>> right? (It would be nice if the sample implementation defaulted to the
>> choice in the PEP.)
>>
>> However, those are just minor edits, and I hereby approve the PEP. Thanks
>> Chris and everyone else for the fruitful discussion (and thanks especially
>> to Chris for eventually ending the bikeshedding and writing a PEP that
>> explains each of the choices). Congrats!
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 485 review (isclose())

[Chris Barker]

On Mon, Mar 2, 2015 at 10:39 PM, Guido van Rossum  wrote:

> It should be written in C. I thought you knew. :-)
>

It crossed my mind when we talked about the math module -- but I didn't
really think about it...

I guess it's okay to drop Decimal support (*). Complex support probably
> should be in cmath.
>

fair enough -- and easy enough to do if we do float-only in math, and
complex-only in cmath.


But this is an amendmend to the PEP as accepted. If anyone cares about
> these changes they should speak up, otherwise the PEP should be updated.
>

will do.


> (*) Adding it to the decimal module would require a discussion with
> Raymond Hettinger, but Decimal users can probably copy and paste the
> formula from the PEP.
>
>
yup -- but maybe worth putting in there while we're at it. though as
Decimal is arbitrary precision, maybe it's not needed....

-Chris



> On Monday, March 2, 2015, Chris Barker  wrote:
>
>> On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker 
>> wrote:
>>
>>> I'll edit the text as you suggest,
>>>
>>
>> Done.
>>
>>
>>> and then work on a patch -- I'm sure I'll have questions for Python-dev
>>> when I actually do that, but I'll get started on my  own and see how far I
>>> get.
>>>
>>
>> OK --  big question 1:
>>
>> As far as I can tell, the math module is entirely a C extension. So I can:
>>
>> 1) write isclose() in C -- I could probably do that (It's been a while --
>> I'm a big Cython fan these days...) and I'd probably punt on having it work
>> with anything other than floats if I did that -- which would fit how most
>> of the match module works anyway.
>>
>> 2) Write it in Python, and monkey-patch it in to the math module -- I
>> honestly have no idea how to do that, but as I can add a new name to the
>> math module after importing it, it should be doable --but I have no idea
>> where the code would go.
>>
>> Suggestions??
>>
>> Thanks,
>>   -Chris
>>
>>
>>
>>
>>
>>
>>
>>
>>> -Chris
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum 
>>> wrote:
>>>
>>>> I think it's time to accept PEP 485. I've re-read it once more, and it
>>>> looks like the text is in great shape. (My only recommendation would be to
>>>> update the Abstract to state that we're specifically adding 
>>>> math.isclose().)
>>>>
>>>> A wording question: "This implementation has a flag that lets the user
>>>> select which relative tolerance test to apply -- this PEP does not suggest
>>>> that that be retained, but rather than the weak test be selected." -- I
>>>> think this was meant to say "... rather *that* the weak test be selected",
>>>> right? (It would be nice if the sample implementation defaulted to the
>>>> choice in the PEP.)
>>>>
>>>> However, those are just minor edits, and I hereby approve the PEP.
>>>> Thanks Chris and everyone else for the fruitful discussion (and thanks
>>>> especially to Chris for eventually ending the bikeshedding and writing a
>>>> PEP that explains each of the choices). Congrats!
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Christopher Barker, Ph.D.
>>> Oceanographer
>>>
>>> Emergency Response Division
>>> NOAA/NOS/OR&R(206) 526-6959   voice
>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>>> Seattle, WA  98115   (206) 526-6317   main reception
>>>
>>> chris.bar...@noaa.gov
>>>
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R(206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>
>
> --
> --Guido van Rossum (on iPad)
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 485 review (isclose())

[Chris Barker]

On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman  wrote:

> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>


> > Maybe it's time to rename the math module to _math and create a
> > math.py module, like _decimal/decimal? math.py should end with "from
> > _math import *".
>
> +1
>

What do folks think? If we're going to do this, I'll write isclose() in
python. And I could do the work to split it out, too, I suppose.

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 485 review (isclose())

[Chris Barker]

On Wed, Mar 4, 2015 at 12:33 PM, Guido van Rossum  wrote:

> In this case I disagree. The math module mostly wraps the C math library
> and the bar should remain high for things to be added to it. Let this be
> someone's opportunity to learn C (I guess not Chris's :-).
>

As it happens, the first C code I ever wrote, beyond toy examples, was
python extensions. I dropped that for Cython a good while ago, but  I can
remember enough for this simple function ;-)

I agree that it's not worth going to python for the math module for just
this -- but others were proposing other reasons to do it.

Back to look at K&R ;-)

-Chris




> On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon  wrote:
>
>>
>>
>> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker 
>> wrote:
>>
>>> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman  wrote:
>>>
>>>> On 03/03/2015 01:17 AM, Victor Stinner wrote:
>>>>
>>>
>>>
>>>> > Maybe it's time to rename the math module to _math and create a
>>>> > math.py module, like _decimal/decimal? math.py should end with "from
>>>> > _math import *".
>>>>
>>>> +1
>>>>
>>>
>>> What do folks think? If we're going to do this, I'll write isclose() in
>>> python. And I could do the work to split it out, too, I suppose.
>>>
>>
>> My vote -- as always -- is to do it in Python. If someone is sufficiently
>> motivated to re-implement in C then that's great, but I don't think it
>> should be required to be in C.
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Aware datetime from naive local time Was: Status on PEP-431 Timezones

[Chris Barker]

On Mon, Apr 13, 2015 at 10:45 AM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

>
> Am I wrong, or is this a semantic question as to what "wall" time means?
>>
>
> You are right about what wall() means, but I should have been more
> explicit about knowns and unknowns in the wall(loc, t) = lt equation.
>
> In that equation I considered loc (the geographical place) and lt (the
> time on the clock tower) to be known and t (the universal (UTC) time) to be
> unknown.  A solution to the equation is the value of the unknown (t) given
> the values of the knowns (loc and lt).
>
> The rest of your exposition is correct including "a given UTC time maps to
> one and only one 'wall time' in a given timezone."  However, different UTC
> times may map to the same wall time and some expressible wall times are not
> results of a map of any UTC time.
>

got it. I suggest you perhaps word it something like:

wall_time = f( location, utc_time)

and

utc_time = f( location, utc_time )

These are two different problems, and one is much harder than the other!
(though both are ugly!)

you can, of course shoreten the names to "wall" and "utc" and "loc" if you
like, but I kind of like long, readable names..

-Chris








-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Aware datetime from naive local time Was: Status on PEP-431 Timezones

[Chris Barker]

Sorry to be brain dead here, but I'm a bit lost:

On Fri, Apr 10, 2015 at 4:32 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

> For any given geographical location, loc, and a moment in time t expressed
> as UTC time, one can tell what time was shown on a "local clock-tower."
> This defines a function  wall(loc, t).   This function is a piece-wise
> linear function which may have regular or irregular discontinuities.
>

got it.


>   Because of these discontinuities, an equation wall(loc, t) = lt may have
> 0, 1
> or 2 solutions.
>

This is where I'm confused -- I can see how going from "wall" time ("local"
time, etc) to UTC has 0, 1, or 2 solutions:

One solution most of the time

Zero solutions when we "spring forward" -- i.e. there is no 2:30 am
on March 8, 2015 in the US timezones that use DST

Two solutions when we "fall back", i.e. there are two 2:30 am Nov 1, 2015
in the US timezones that use DST

But I can't see where there are multiple solutions the other way around --
doesn't a given UTC time map to one and only one "wall time" in a given
timezone?

Am I wrong, or is this a semantic question as to what "wall" time means?

Thanks for any clarification,

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Aware datetime from naive local time Was: Status on PEP-431 Timezones

[Chris Barker]

On Mon, Apr 13, 2015 at 12:14 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

> utc_time = f( location, utc_time )
>>
>> These are two different problems, and one is much harder than the other!
>> (though both are ugly!)
>>
>
> You probably meant "utc_time = f( location, wall_time)" in the last
> equation,
>

oops, yes.


> but that would still be wrong.
>


> A somewhat more correct equation would be
>
> utc_time = f^(-1)( location, wall_time)
>

In this case I meant "f" as "a function of", so the two fs were not
intended to be the same. Yes, one is the inverse of another, and in this
case the inverse is not definable (at least not uniquely).

I have no doubt you understand all this (better than I do), I'm just
suggesting that in the discussion we find a way to be as clear as possible
as to which function is being discussed when.

But anyway -- thanks all for hashing this out -- getting something
reasonable into datetime will be very nice.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Chris Barker]

On Tue, Apr 21, 2015 at 2:33 AM, Cory Benfield  wrote:

> It seems like the only place the type annotations will get used is in
> relatively trivial cases where the types are obvious anyway. I don't
> deny that *some* bugs will be caught, but I suspect they'll
> overwhelmingly be crass ones that would have been caught quickly
> anyway.


Well, it'll catch passing in a string instead of a sequence of strings --
one of teh common and semi-insidious type errors I see a lot (at least with
newbies).

Oh wait, maybe it won't -- a string IS a sequence of strings. That's why
this is an insidioous bug in teh first place:

List[string]

will work, yes -- but now I'm locked down the user to an actual, list, yes?
So I try:

Iterable[string]

ah yes -- now users can use tuples of strings, or a custom class, or -- but
wait, darn it, a string IS an iterable of strings.. SIGH.

NOTE: I've thought for years that Python should have a character type for
this reason, but that's a side note.

Oh, and I'm a heavy numpy user. And, in fact, I write a lot of functions
that are essentially statically typed -- i.e. they will only work with a
Nx3 array of float64 for instance. However, in this case, I want run-time
type checking, and I DON'T want the API to be statically typed, so I use
something like:

the_arr = np.asarray( input_object, dypte-np.float64).reshape(-1, 3)

and in my docstring:

:param input_object: the input data to compute something from
:type input_object: (Nx3) numpy array or floats or somethign that can be
turned into one.

Is there any way for type hinting to help here???


- Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Chris Barker]

On Tue, Apr 21, 2015 at 11:58 AM, Paul Sokolovsky  wrote:

> It does, and hope people won't be caught in "static typechecking"
> loop and consider other usages too.
>

I"m confused -- from the bit I've been skimming the discussion, over on
python-ideas, and now here, is that this is all about "static typechecking".

It's not about run-time type checking.
It's not about type-base performance optimization.
It's not about any use of annotations other than types.

What is it about other than static typechecking?

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Chris Barker]

Thank you Jack.

Jack: "I hate code and I want as little of it as possible in our product"

I love that quote -- and I ALWAYS use it when I teach newbies Python. It's
kind of the point of Python -- you can get a lot done by writing very
little code.

I'm still confused about what all this type annotation is about -- yes, I
certainly see lots of advantages to static typing -- but also many
disadvantages. And one of them is a bunch more stuff to read and write.

But key is: If you like typing, use a statically typed language.

If you want the advantages of dynamic typing -- then why do you want all
the extra clutter and "safety"?

And it seems to me that optional/partial typing buys you much of the hassle
and restrictions, and essentially NONE of the safety (type casting pointers
in C, anyone?).

And I've had this debate with proponents of JAVA and the like a bunch --
sure, type checking will catch bugs in your code -- but they are almost
always shallow bugs -- and bugs that if your tests didn't catch them, you
have really lame tests!

On the other had, you just can't get good performance doing low-level
things with a dynamic language. So I use Cython, which, in a way, is Python
with optional static typing.  It gives you down to the metal performance,
where, and only where, you need it, but not really much in the way of type
checking.

However, this is exactly the opposite of what everyone seem to be talking
about using optional typing in Python for -- which is type checking ,but
not any run-time optimizations -- in fact, AFAICT, not even run-time type
checking.

So a whole lot of clutter for very little gain. :-(

NOTE: MyPy is out in the wild -- I'd be really interested to see how it all
is really working out -- even on those "enterprise" code bases -- other
than managers feeling better, are developers finding out that:

"WOW -- this baby caught some nasty, bugs that I would not have found with
testing -- and would have been hard to debug after the fact!"

But in the end, I agree with the OP here -- stub files let pre-run-time
static type checking happen without cluttering up Python for the rest of us
-- so a nice compromise.

So I guess we'll see.

- Chris

On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich  wrote:

> Twelve years ago a wise man said to me "I suggest that you also propose a
> new name for the resulting language"
>
> I talked with many of you at PyCon about the costs of PEP 484. There are
> plenty of people who have done a fine job promoting the benefits.
>
> * It is not optional. Please stop saying that. The people promoting it
> would prefer that everyone use it. If it is approved it will be optional in
> the way that PEP8 is optional. If I'm reading your annotated code it is
> certainly /not/ optional that I understand the annotations.
>
> * Uploading stubs for other people's code is a terrible idea. Who do I
> contact when I update the interface to my library? The random Joe who
> "helped" by uploading annotations three months ago and then quit the
> internet? I don't even want to think about people maliciously adding stubs
> to PyPI.
>
> * The cognitive load is very high. The average function signature will
> double in length. This is not a small cost and telling me it is "optional"
> to pretend that every other word on the line doesn't exist is a farce.
>
> * Every company's style guide is about to get much longer. That in itself
> is an indicator that this is a MAJOR language change and not just some
> "optional" add-on.
>
> * People will screw it up. The same people who can't be trusted to program
> without type annotations are also going to be *writing* those type
> annotations.
>
> * Teaching python is about to get much less attractive. It will not be
> optional for teachers to say "just pretend all this stuff over here doesn't
> exist"
>
> * "No new syntax" is a lie. Or rather a red herring. There are lots of new
> things it will be required to know and just because the compiler doesn't
> have to change doesn't mean the language isn't undergoing a major change.
>
> If this wasn't in a PEP and it wasn't going to ship in the stdlib very few
> people would use it. If you told everyone they had to install a different
> python implementation they wouldn't. This is much worse than that - it is
> Python4 hidden away inside a PEP.
>
> There are many fine languages that have sophisticated type systems. And
> many bondage & discipline languages that make you type things three times
> to make really really sure you meant to type that. If you find those other
> languages appealing I invite you to go use them instead.
>
> -Jack
>
> https://mail.python.org/pipermail/python-dev/2003-February/033291.html
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Chris Barker]

> Oh wait, maybe it won't -- a string IS a sequence of strings. That's why
> this is an insidious bug in the first place.

On Tue, Apr 21, 2015 at 11:32 PM, Terry Reedy  wrote:


> I was just thinking today that for this, typing needs a subtraction
> (difference) operation in addition to an addition (union) operation:
> Difference(Iterable(str), str)
>

Yup -- that might solve, it, but it feels a bit odd -- I can take any
Iterable of string, except a string. -- but what if there are others that
won't work??? But I guess that's the core of putting type hints on a
dynamic language.

Still, I tend to think that this particular issue is really a limitation
with Python's type system -- nothing to do with type hinting.

I can see that a character type seems useless in Python, but there are
lessons from other places: a numpy array is a collection of (usually)
numbers that can be treated as a single entity -- much like a string is a
collection of characters that is treated as a single entity -- in both
cases, it's core to convenience and performance to do that. But with numpy,
when you index an array, you get something back with one less dimension:

index into a 3-d array, you get a 2-d array
index into a 2-d array, you get a 1-d array
index into a 1-d array, you get a scalar -- NOT a length-one 1-d array

Sometimes this is a pain for generic code, but more often than not it's
critical to writing dynamic code -- not because you couldn't do the
operations you want, but because it's important to distinguish between a
scalar and an array that happens to have only one value.

Anyway, the point is that being able to say "all these types, except this
one" would solve this particular problem -- but would it solve any others?
Do we want this to work around a quirk in Pythons string type?

NOTE: I know full well that adding a character type to Python is not worth
it.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Chris Barker]

On Wed, Apr 22, 2015 at 5:45 PM, Guido van Rossum  wrote:

> Given that even if Difference existed, and even if we had a predefined
> type alias for Difference[Iterable[str], str], you' still have to remember
> to mark up all those functions with that annotation. It almost sounds
> simpler to just predefine this function:
>
> def make_string_list(a: Union[str, Iterable[str]]) -> Iterable[str]:
> if isinstance(a, str):
> return [a]
> else:
> return a
>

fair enough -- and I do indeed have that code in various places already.

Somehow, I've always been uncomfortable with checking specifically for the
str type -- guess I want everything to be fully duck-typable.

But then I wouldn't be doing type hints, either, would I?

-Chris
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

[Chris Barker]

On Thu, Sep 7, 2017 at 3:49 PM, Larry Hastings  wrote:

>  But I can cite at least one place in the standard library that would have
> been better if it'd been implemented as a module property:
>
os.stat_float_times().
>
I wish there were a property feature available almost very time I encounter
a "get*" method in the stdlib (or any where):

There are a heck of a lot in the os module:


In [4]: [s for s in dir(os) if s.startswith('get')]
Out[4]:

['get_blocking',
 'get_exec_path',
 'get_inheritable',
 'get_terminal_size',
 'getcwd',
 'getcwdb',
 'getegid',
 'getenv',
 'getenvb',
 'geteuid',
 'getgid',
 'getgrouplist',
 'getgroups',
 'getloadavg',
 'getlogin',
 'getpgid',
 'getpgrp',
 'getpid',
 'getppid',
 'getpriority',
 'getsid',
 'getuid']

Many of those may be good use-cases for getters, but still...

And just yesterday I was getting annoyed by some in sysconfig:

In [6]: [s for s in dir(sysconfig)  if s.startswith('get')]
Out[6]:

['get_config_h_filename',
 'get_config_var',
 'get_config_vars',
 'get_makefile_filename',
 'get_path',
 'get_path_names',
 'get_paths',
 'get_platform',
 'get_python_version',
 'get_scheme_names']


modules serve the very useful function of a global singleton -- if we think
properties are a good idea for classes, then they are a good idea for
modules...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

[Chris Barker]

On Mon, Sep 11, 2017 at 10:20 AM, Victor Stinner 
wrote:

> 2017-09-11 19:00 GMT+02:00 Chris Barker :
> > There are a heck of a lot in the os module:
> > ['get_blocking',
>
> This one is not a good example: it takes a parameter. You cannot
> convert it to a property.
>

I'm sure there are many that really do have a good reason for a getter
function, but not all, by any means.

When designing an API, when I have to choose between property and
> function/method, I prefer function/method over a property when the
> code is slow, especially at the first call.
>
> I prefer to "warn" users than a call (like the first one which fills a
> cache) can be slow.
>
> Well, it's not a strong rule, sometimes I use a property even if the
> first call has to fill a cache :-)
>
> Here the sysconfig has to build an internal cache at the first call
> ... if I recall correctly.
>

If we do get properties on modules, then there is plenty of room to discuss
best API for  a given functionality. And chances are, we won't gratuitously
re-factor the standard library.

But your point is well taken, and makes my point in a way -- if we had
properties, then there would be a getter function only if there was a good
reason for it. As it stands, I have no idea if calling, for instance,
sysconfig.get_config_vars is an expensive operation.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] parallelizing

[Chris Barker]

This really isn't the place to ask this kind of question.

If you want to know how to do something with python, try python-users ,
stack overflow, etc.

If you have an idea about a new feature you think python could have, then
the python-ideas list is the place for that. But if you want anyone to take
it seriously, it should be a better formed idea before you post there.

But:

On Tue, Sep 12, 2017 at 4:43 PM, Matthieu Bec  wrote:

> There are times when you deal with completely independent input/output
> 'pipes' - where parallelizing would really help speed things up.
>
> Can't there be a way to capture that idiom and multi thread it in the
> language itself?
>
> Example:
>
> loop:
>
> read an XML
>
> produce a JSON like
>

Regular old threading works fine for this:

import time
import random
import threading


def process(infile, outfile):
"fake function to simulate a process that takes a random amount of time"
time.sleep(random.random())
print("processing: {} to make {}".format(infile, outfile))


for i in range(10):
threading.Thread(target=process, args=("file%i.xml" % i, "file%i.xml" %
i)).start()


It gets complicated if you need to pass information back and forth, or
worry about race conditions, or manage a queue, or 

But just running a nice self-contained thread safe function in another
thread is pretty straightforward.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] parallelizing

[Chris Barker]

On Wed, Sep 13, 2017 at 12:11 PM, Matthieu Bec  wrote:

> Regarding your example, I think it gives the illusion to work because
> sleep() is GIL aware under the hood.
>
>
It'll work for anything -- it just may not buy you any performance.

I don't know off the top of my head if file I/O captures the GIL -- for
your example of file parsing.

> I don't think it works for process() that mainly runs bytecode, because of
> the GIL.
>
If you are trying to get around the GIL that that is a totally different
question.

But the easy way is to use multiprocessing instead:

import time
import random
import multiprocessing


def process(infile, outfile):
"fake function to simulate a process that takes a random amount of time"
time.sleep(random.random())
print("processing: {} to make {}".format(infile, outfile))

for i in range(10):
multiprocessing.Process(target=process, args=("file%i.xml" % i,
"file%i.xml" % i)).start()

More overhead creating the processes, but no more GIL issues.

Sorry if I wrongly thought that was a language level discussion.
>
>
This list is for discussion of the development of the cPython interpreter.
So this kind of discussion doesn't belong here unless/until it gets to the
point of actually implementing something.

If you have an idea as to how to improve Python, then python-ideas is the
place for that discussion.

But "there should be a way to run threads without the GIL" isn't a
well-enough formed idea to get far there

If you want to discuss further, let's take this offline.

> Can't there be a way to capture that idiom and multi thread it in the
language itself?

>
>> Example:
>>
>> loop:
>>
>> read an XML
>>
>> produce a JSON like
>>
> note about this -- code like this would be using all sorts of shared
modules. The code in those modules is going to be touched by all the
threads. There is no way the python interpreter can know which python
objects are used by what how --- the GIL is there for good (and complex)
reasons, not an easy task to avoid it. It's all using the same interpreter.

Also -- it's not easy to know what code may work OK with the GIL. intensive
computation is bad. But Python is a poor choice for that anyway.

And code that does a lot in C -- numpy, text processing, etc. may not hold
the GIL. And I/O

So for your example of parsing XML and writing JSON -- it may well do a lot
of work without holding the GIL.

No way to know but to profile it.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Chris Barker]

On Thu, Oct 12, 2017 at 3:20 AM, Steve Holden  wrote:


>  The reason I liked "row" as a name is because it resembles  "vector" and
> hence is loosely assocaited with the concept of a tuple as well as being
> familiar to database users. In fact the answer to a relational query was, I
> believe, originally formally defined as a set of tuples.
>
>
Is the intent that these things preserve order? in which case, I like row
is OK (though still don't see what's wrong with record). I still dop'nt
love it though -- it gives the expectation of a row in a data table )or csv
file, or.. which will be a common use case, but really, it doesn't
conceptually have anything to do with tabular data.

in fact, one might want to store a bunch of these in, say, a 2D (or 3D)
array, then row would be pretty weird

I don't much like entity either -- it is either way to generic -- everyting
is an entity! even less specific than "object". Or two specific (and
incorrect) in the lexicon of particular domains.


-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Chris Barker]

I think we've drifted into a new topic, but...


 I was astonished to see them
> implemented using decorators, and I was not the only one.


...


> Python is at a weird point here. At about every new release of Python,
> a new idea shows up that could be easily solved using metaclasses, yet
> every time we hesitate to use them, because of said necessary manual
> intervention for metaclass combination.
>
> So I think we have two options now: We could deprecate metaclasses,
>

I was thinking about this last spring, when I tried to cram all sorts of
python metaprogramming into one 3hr class...

Trying to come up with a an exam[ple for metclasses, I couldn't come up
with anything that couldn't be done more claerly (to me) with a class
decorator.

I also found some commentary on the web (sorry, no links :-( ) indicating
that metacalsses were added before class decorators, and that they really
don't have a compelling use case any more.

Now it seem that not only do they not have a compelling  use case, in some
(many) instances, there are compelling reasons to NOT use them, and rather
use decorators.

So why deprecate them? or at least discourage their use?

The other option would be to simply make metaclasses work properly. We
> would just have to define a way to automatically combine metaclasses.
>

"just"?

Anyway, let's  say that is doable -- would you then be able to do something
with metaclasses that you could not do with decorators? or it in a cleaner,
easier to write or understand way?

There-should-be-one--and-preferably-only-one--obvious-way-to-do-it-ly yours,

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

[Chris Barker]

On Sun, Oct 22, 2017 at 1:42 PM, Wes Turner  wrote:

> Aligning simulation data in context to other events may be enlightening:
> is there a good library for handing high precision time units in Python
> (and/or CFFI)?
>

Well, numpy's datetime64 can be set to use (almost) whatever unit you want:

https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.
datetime.html#datetime-units

Though it uses a single epoch, which I don't think ever made sense with
femtoseconds

And it has other problems, but it was designed that way, just for the
reason.

However, while there has been discussion of improvements, like making the
epoch settable, none of them have happened, which makes me think that no
one is using it for physics experiments, but rather plain old human
calendar time...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

On Mon, Oct 23, 2017 at 5:33 PM, Hasan Diwan  wrote:

> If one simply replaces the 'T' with a space and trims it after the '.',
> IIRC, it parses fine.
>

sure, but really, can anyone argue that it's not a good idea for datetime
ot be able to read the iso format it puts out???

-CHB



> -- H
>
> On Oct 23, 2017 15:16, "Mike Miller"  wrote:
>
>> Hi,
>>
>> Could anyone put this five year-old bug about parsing iso8601 format
>> date-times on the front burner?
>>
>> http://bugs.python.org/issue15873
>>
>> In the comments there's a lot of hand-wringing about different variations
>> that bogged it down, but right now I only need it to handle the output of
>> datetime.isoformat():
>>
>> >>> dt.isoformat()
>> '2017-10-20T08:20:08.986166+00:00'
>>
>> Perhaps if we could get that minimum first step in, it could be iterated
>> on and made more lenient in the future.
>>
>> Thank you,
>> -Mike
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/hasan.diw
>> an%40gmail.com
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

+1 on a classmethod constructor
+0 on a based-on-type default constructor

+inf on SOMETHING!

Let's get passed the bike shedding and make this work!

-CHB


On Wed, Oct 25, 2017 at 2:18 PM, Alex Walters 
wrote:

>
>
> > -Original Message-
> > From: Alexander Belopolsky [mailto:alexander.belopol...@gmail.com]
> > Sent: Wednesday, October 25, 2017 4:33 PM
> > To: Alex Walters 
> > Cc: Elvis Pranskevichus ; Python-Dev  > d...@python.org>; Chris Barker 
> > Subject: Re: [Python-Dev] iso8601 parsing
> >
> > On Wed, Oct 25, 2017 at 3:48 PM, Alex Walters 
> > wrote:
> > >  Why make parsing ISO time special?
> >
> > It's not the ISO format per se that is special, but parsing of str(x).
> > For all numeric types, int, float, complex and even
> > fractions.Fraction, we have a roundtrip invariant T(str(x)) == x.
> > Datetime types are a special kind of numbers, but they don't follow
> > this established pattern.  This is annoying when you deal with time
> > series where it is common to have text files with a mix of dates,
> > timestamps and numbers.  You can write generic code to deal with ints
> > and floats, but have to special-case anything time related.
>
> >>> repr(datetime.datetime.now())
> 'datetime.datetime(2017, 10, 25, 17, 16, 20, 973107)'
>
> You can already roundtrip the repr of datetime objects with eval (if you
> care to do so).  You get iso formatting from a method on dt objects, I
> don’t see why it should be parsed by anything but a classmethod.
>
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

On Wed, Oct 25, 2017 at 4:22 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

> > times. The only difference is that instead of calling the type directly,
> > you call the appropriate classmethod.
> >
> > What am I missing?
>
> Nothing. The only annoyance is that the data processing code typically
> needs to know the type anyway, so the classmethod is one more variable
> to keep track of.


I don't think anyone is arguing that is is hard to do either way, or that
hard to use either way.

I think it comes down to a trade-off between:

Having an API for datetime that is like the datetime for number types:

 int(str(an_int)) == an_int

 so:

 datetime(str(a_datetime)) == a_datetime

Alexander strongly supports that.

Or an API that is perhaps more like the rest of the datetime api, which has
a number of alternate constructors:

 datetime.now()

 datetime.fromordinal()

 datetime.fromtimestamp()

And has:

  datetime.isoformat()

So a

   datetime.fromisoformat()

would make a lot of sense.

I would note that the number types don't have all those alternate
constructors Also, the number types mostly have a single parameter (except
int has an optional base parameter). So I'm not sure the parallel is that
strong.

Would it be horrible if we did both?

After all, right now, datetime has:

In [16]: dt.isoformat()
Out[16]: '2017-10-25T16:30:48.744489'

and
In [18]: dt.__str__()
Out[18]: '2017-10-25 16:30:48.744489'

which do almost the same thing (I understand the "T" is option in iso 8601)

However, maybe they are different when you add a time zone?

ISO 8601 support offsets, but not time zones -- presumably the __str__
supports the full datetime tzinfo somehow. Which may be why .isoformat()
exists.

Though I don't think that means you couldn't have the __init__ parse proper
ISO strings, in addition to the full datetime __str__ results.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

On Wed, Oct 25, 2017 at 7:37 PM, Wes Turner  wrote:

> ISO 8601 support offsets, but not time zones -- presumably the __str__
>> supports the full datetime tzinfo somehow. Which may be why .isoformat()
>> exists.
>>
>
> ISO8601 does support timezones.
> https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators
>

No, it doesn't -- it may call them "timezones", but it only supports
offsets -- that is, and offset of -6 could be US Eastern Standard Time or
US Central Daylight TIme (or I got that backwards :-)  )

The point is that an offset is really easy, and timezones (with Daylight
savings and all that) are a frickin' nightmare, but ARE supported by
datetime

Note that the vocabulary is not precise here, as I see this in the Pyton
docs:

*class *datetime.timezone

A class that implements the tzinfo
 abstract
base class as a fixed offset from the UTC.
So THAT is supported by iso8601, and, indeed maps naturally to it.

Which means we can round trip isp8601 datetimes nicely, but can't round
trip a datetime with a "full featured" tzinfo attached.

I don't think this really has any impact on the proposal, though: it's
clear what to do when parsing a iso Datetime.

I might be wrong, but I think many of the third party libraries listed here
> default to either UTC or timezone-naieve timezones:
> https://github.com/vinta/awesome-python/blob/master/
> README.md#date-and-time
>

This is a key point that I hope is obvious:

If an ISO string has NO offset or timezone indicator, then a naive datetime
should be created.

(I say, I "hope" it's obvious, because the numpy datetime64 implementation
initially (and for years) would apply the machine local timezone to a bare
iso string -- which was a f-ing nightmare!)


> Ctrl-F for 'tzinfo=' in the docs really doesn't explain how to just do it
> with my local time.
>
> Here's an example with a *custom* GMT1 tzinfo subclass:
> https://docs.python.org/3/library/datetime.html#datetime.time.tzname
>

Here it is:

class GMT1(tzinfo):
def utcoffset(self, dt):
return timedelta(hours=1)
def dst(self, dt):
return timedelta(0)
def tzname(self,dt):
return "Europe/Prague"

I hope Prague doesn't do DST, or that would be just wrong ...

What would you call the str argument? Does it accept strptime args or only
> ISO8601?
>

I think Fred answered this, but iso 8601 only. we already have strptime if
you need to parse anything else.

Would all of that string parsing logic be a performance regression from the
> current constructor? Does it accept None or empty string?
>

I suppose you need to do a little type checking first, so a tiny one.

Though maybe just catching an Exception, so really tiny.

The current constructor only takes numbers, so yes the string parsing
version would be slower, but only if you use it...

Deserializing dates from JSON (without #JSONLD and xsd:dateTime (ISO8601))
> types is nasty, regardless (try/except, *custom* schema awareness). And
> pickle is dangerous.
>
> AFAIU, we should not ever eval(repr(dt: datetime)).
>

why not? isn't that what __repr__ is supposed to do?

Or do you mean not that it shouldn't work, but that we shouldn't do it?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

On Mon, Nov 6, 2017 at 11:23 AM, Paul G  wrote:

> (Of course, given that CPython's implementation is order-preserving, a
> bunch of code is probably now being written that implicitly requires on
> this detail, but at least having syntax that makes that clear would give
> people the *option* to make the assumption explicit).


This is a really key point -- a heck of a lot more people use cPython than
read the language spec. And a great deal of code is developed with a
certain level of ignorance -- try something, if it works, and your test
pass (if there are any), then you are done.

So right now, there is more an more code out there that relies on a regular
old dcit being ordered.

I've been struggling with teaching this right now -- my
written-a-couple-years ago materials talk about dicts being arbitrary
order, and then have a little demo of that fact. Now I'm teaching with
Python 3.6, and I had to add in something like:

   cPython does, in fact, preserve order with dicts, but it should be
considered an implementation detail, and not counted on ... (and by the
say, so does PyPy, and )"

I don't know, but I'm going to guess about 0% of my class is going to
remember that...

And if we added o{,,,} syntax it would be even worse, 'cause folks would
forget to use it, as their code wouldn't behave differently (kind of like
the 'b' flag on unix text files, or the u"string" where everything is ascii
in that string...)

in short -- we don't have a choice (unless we add an explicit randomization
as some suggested -- but that just seems perverse...)

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Chris Barker]

On Tue, Nov 7, 2017 at 11:50 AM, Tim Peters  wrote:

> Is it worth guaranteeing that will always "work" (as intended)?  Not
> to me, but I do have code that relies on it now -


This is critically important -- no one looks at the language spec to figure
out how something works -- they try it, and if it works assume it will
continue to work.

if dict order is preserved in cPython , people WILL count on it!

And similarly, having order preserved only until a delete is going to cause
bugs in people's code that are less than careful :-( -- and that's most of
us.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

On Tue, Nov 7, 2017 at 7:21 AM, David Mertz  wrote:

> But like Raymond, I make most of my living TEACHING Python.
>

and I make least of my living TEACHING Python ( :-) ),and:


> I feel like the extra order guarantee would make teaching slightly harder.
>

I can't understand how this possibly makes python (or dicts) harder to
teach -- you can simply say: "dicts insertion order is preserved" or not
mention it at all -- I think most people kind of expect it to be preserved,
which is why I (used to )always bring up the lack-of-ordering of dicts
early on -- but I suspect I simply won't bother mentioning it if it's
decided as a language feature.

I'm sure he feels contrarily. It is true that with 3.6 I can no longer show
> an example where the dict display is oddly changed when printed.
>

Exactly! I have a really hard time deciding how to handle this --
explaining that ordering is not guaranteed, but not being able to
demonstrate it! And frankly, my students are all going to forget what I
"explained" soon enough, and replace it with their experience -- which will
be that dicts retain their order.

But then, unordered sets also wind up sorting small integers on printing,
> even though that's not a guarantee.
>

but it's not hard to make an easy example with order not preserved -- jsut
start with a non order example:

In [6]: s = {3,7,4}

In [7]: s

Out[7]: {3, 4, 7}

or use other types...

And  "set" is a mathematical concept that has no oder, whereas the
"dictionary" metaphor DOES have order...

> Ordering by insertion order (possibly "only until first deletion") is
> simply not obvious to beginners.
>

the "only until first deletion" part is really hard -- I hope we don't go
that route. But I don't think insertion-order is non-obvious --
particularly with literals.


> If we had, hypothetically, a dict that "always alphabetized keys" that
> would be more intuitive to them, for example. Insertion order feels obvious
> to us experts, but it really is an extra cognitive burden to learners
> beyond understanding "key/Val association".
>

again, I don't think so -- I kind of agree if dicts did not preserve order
in practice -- demonstrating that right out of the gate does help make the
"key/Val association" clear -- but if you can't demonstrate it, I think
we're looking at more confusion...

Maybe I'll ask my students this evening -- this is the first class I'm
teaching with py3.6 

We've lived without order for so long that it seems that some of us now
> think data scrambling is a virtue.  But it isn't.  Scrambled data is the
> opposite of human friendly.
>
>
exactly!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

On Wed, Nov 29, 2017 at 10:05 AM, Mike Miller 
wrote:

> This thread isn't about the numerous third-party solutions that are a pip
> command away.  But rather getting a minimal return parser to handle
> datetime.isoformat() into the std library.
>
> It's been needed for years,


indeed what is the holdup? I don't recall anyone saying it was a bad idea
in the last discussion.

Do we just need an implementation?

Is the one in the Bug Report not up to snuff? If not, then what's wrong
with it? This is just not that hard a problem to solve.

-CHB






> and hopefully someone can squeeze it into 3.7 before its too late.  (It
> takes years for a new version to trickle down to Linux dists.)
>
> -Mike
>
>
> On 2017-11-28 13:09, Skip Montanaro wrote:
>
>> It's got its own little parsing language, different than the usual
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.
> barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

On Wed, Nov 29, 2017 at 4:19 PM, Paul G  wrote:

> I can write at least a pure Python implementation in the next few days, if
> not a full C implementation. Shouldn't be too hard since I've got a few
> different Cython implementations sitting around anyway.
>
>
Thanks!

-CHB




> On November 29, 2017 7:06:58 PM EST, Alexander Belopolsky <
> alexander.belopol...@gmail.com> wrote:
>>
>>
>>
>> On Wed, Nov 29, 2017 at 6:42 PM, Chris Barker 
>> wrote:
>>
>>>
>>> indeed what is the holdup? I don't recall anyone saying it was a bad
>>> idea in the last discussion.
>>>
>>> Do we just need an implementation?
>>>
>>> Is the one in the Bug Report not up to snuff? If not, then what's wrong
>>> with it? This is just not that hard a problem to solve.
>>>
>>
>>
>> See my comment from over a year ago: <https://bugs.python.org/
>> issue15873#msg273609>.  The proposed patch did not have a C
>> implementation, but we can use the same approach as with strptime and call
>> Python code from C.  If users will start complaining about performance, we
>> can speed it up in later releases.  Also the new method needs to be
>> documented.  Overall, it does not seem to require more than an hour of work
>> from a motivated developer, but the people who contributed to the issue in
>> the past seem to have lost their interest.
>>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

[Chris Barker]

On Tue, Dec 5, 2017 at 1:18 PM, Guido van Rossum  wrote:
>
>
> I am very worried about this long and rambling PEP,
>

FWIW, I read the PEP on the bus this morning on a phone, and while lng, I
didn't find it too rambling. And this topic has been very often discussed
in very long and rambling mailing list threads, etc. So I think a long (If
not rambling) PEP is in order.

This is a very important topic for Python -- the py2-3 transition got a LOT
of flack, to the point of people claiming that it was easier to learn a
whole new language than convert to py3 -- and THIS particular issue was a
big part of it:

The truth is that any system that does not use a clearly defined encoding
for filenames (and everything else) is broken, plain and simple. But the
other truth is (as talked about in the PEP) they some *nix systems are that
broken because C code that simply passed around char* still works fine. And
no matter how you slice it telling people that they need to fix their
broken system in order for your software to run is not a popular option.

When Python added surrogateescape to its Unicode implementation, the tools
were there to work with broken (OK, I'll be charitable: misconfigured)
systems. Now we just need some easier defaults.

OK, now I'm getting long and rambling

TL;DR -- The proposal in the PEP is an important step forward, and the
issue is fraught with enough history and controversy that a long PEP is
probably a good idea.

So the addition of a better summary of the specification up at the top, and
editing of the rest, and we could have a good PEP.

Too late for this release, but what can you do?


> The "Unicode just works" summary is more a wish than a proper summary of
> the PEP.
>

well, yeah.


> FWIW the relationship with PEP 538 is also pretty unclear. (Or maybe
> that's another case of the forest and the trees.) And that PEP (while
> already accepted) also comes across as rambling and vague, and I have no
> idea what it actually does. And it seems to mention PEP 540 quite a few
> times.
>

I just took another look at 538 -- and yes, the relationship between the
two is really unclear. In particular, with 538, why do we need 540? I
honestly don't know.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] iso8601 parsing

[Chris Barker]

On Wed, Dec 6, 2017 at 3:07 PM, Paul Ganssle  wrote:

> Here is the PR I've submitted:
>
> https://github.com/python/cpython/pull/4699
>
> The contract that I'm supporting (and, I think it can be argued, the only
> reasonable contract in the intial implementation) is the following:
>
> dtstr = dt.isoformat(*args, **kwargs)
> dt_rt = datetime.fromisoformat(dtstr)
> assert dt_rt == dt# The two points represent the
> same absolute time
> assert dt_rt.replace(tzinfo=None) == dt.replace(tzinfo=None)   # And
> the same wall time
>


that looks good.

And I'm sorry, I got a bit lost in the PR, but you are attaching an
"offset" tzinfo, when parsing an iso string that has one, yes?

I see this in the comments in the PR:


"""
This does not support parsing arbitrary ISO 8601 strings - it is only
intended
as the inverse operation of :meth:`datetime.isoformat`
"""

I fully agree that that's the MVP -- but is it that hard to parse arbitrary
ISO8601 strings in once you've gotten this far? It's a bit uglier than I'd
like, but not THAT bad a spec.

what ISO8601 compatible features are not supported?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

Sorry about the email mangling -- I do a lot of my listserve work on the
bus on an iPhone, with the built -in mail client -- and it REALLY sucks for
doing interspersed email replying -- highly encouraging the dreaded top
posting...

But anyway, I think both Steve and I were expressing concerns about "Typing
Creep".  Typing should always be optional in Python, and while this PEP
does keep it optional, Steve's point was that the code in the standard
library serves not only as a library, but as examples of how to write
"robust" python code.

The rest of this note is me -- I'm not pretending ot speak for Steve.

Reading the PEP, this text makes me uneasy:

"A field is defined as any variable identified in__annotations__. That is,
a variable that has a type annotation."

And if I understand the rest of the PEP, while typing itself is optional,
the use of type Annotation is not -- it is exactly what's being used to
generate the fields the user wants.

And the examples are all using typing -- granted, primarily the built in
types, but still:


@dataclass
class C:
a: int   # 'a' has no default value
b: int = 0   # assign a default value for 'b'


This sure LOOKS like typing is required. It also makes me nervous because,
as I understand it, the types aren't actually used in the
implementation (presumable they would be by mypy and the like?). So I think
for folks that aren't using typing and a type checker in their development
process, it would be pretty confusing that this means and what it actually
does. Particularly folks that are coming from a background of a statically
typed language.

Then I see:

"""
Field objects describe each defined field.
...
Its documented attributes are:

name: The name of the field.
type: The type of the field.
...
"""

So again, typing looks to be pretty baked in to the whole concept.

and then:

"""
One place where dataclass actually inspects the type of a field is to
determine if a field is a class variable as defined in PEP 526.
"""

and

"""
The other place where dataclass inspects a type annotation is to determine
if a field is an init-only variable. It does this by seeing if the type of
a field is of type dataclasses.InitVar.
"""

"""
Data Classes will raise a TypeError if it detects a default parameter of
type list, dict, or set.
"""

So: it seems that type hinting, while not required to use Data Classes, is
very much baked into the implementation an examples.

As I said -- this makes me uneasy -- It's a very big step that essentially
promotes the type hinting to a new place in Python -- you will not be able
to use a standard library class without at least a little thought about
types and typing.

I note this:

"""
This discussion started on python-ideas [9] and was moved to a GitHub repo
[10] for further discussion. As part of this discussion, we made the
decision to use PEP 526 syntax to drive the discovery of fields.
"""

I confess I only vaguely followed that discussion -- in fact, mostly I
thought that the concept of Data Classes was a good one, and was glad to
see SOMETHING get implemented, and didn't think I had much to contribute to
the details of how it was done. So these issues may have already been
raised and considered, so carry on.

But:

NOTE: from PEP 526:

"Python will remain a dynamically typed language, and the authors have no
desire to ever make type hints mandatory, even by convention. "

The Data Classes implementation is not making it mandatory by any means,
but it is making it a more "standard" part of the language that can not
simply be ignored anymore. And it seems some features of dataclasses can
only be accessed via actual typing, in addition to the requirement of type
annotations.

If nothing else, the documentation should make it very clear that the
typing aspects of Data Classes is indeed optional, and preferably give some
untyped examples, something like:

@dataclass
class C:
a: None  # 'a' has no default value
b: None = 0   # assign a default value for 'b'


If, in fact, that would be the way to do it.

-Chris




On Fri, Dec 15, 2017 at 3:22 AM, Eric V. Smith  wrote:

> On 12/15/2017 5:56 AM, Steve Holden wrote:
>
>> On Mon, Dec 11, 2017 at 5:10 PM, Chris Barker - NOAA Federal <
>> chris.bar...@noaa.gov <mailto:chris.bar...@noaa.gov>> wrote:
>>
> ...
>
>> However, typing is not currently imported by dataclasses.py.
>>>
>> >
>
>> And there you have an actual reason besides my uneasiness :-)
>>
>> - CHB
>>
>> ​hmm...​
>>
>
> [Agreed with Antoine on the MUA and quoting being confusing.]
>
> The only reason typing isn't i

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

One other note (see my last message).

The PEP should include a summary of the discussion of the decision to use
the type annotation syntax vs other options.

I just looked through all the gitHub issues and found nothing, and started
to look at the python-ideas list archive and got overwhelmed.

So having that justification in the PEP would be good.

-CHB


On Fri, Dec 15, 2017 at 12:07 PM, Chris Barker 
wrote:

> Sorry about the email mangling -- I do a lot of my listserve work on the
> bus on an iPhone, with the built -in mail client -- and it REALLY sucks for
> doing interspersed email replying -- highly encouraging the dreaded top
> posting...
>
> But anyway, I think both Steve and I were expressing concerns about
> "Typing Creep".  Typing should always be optional in Python, and while this
> PEP does keep it optional, Steve's point was that the code in the standard
> library serves not only as a library, but as examples of how to write
> "robust" python code.
>
> The rest of this note is me -- I'm not pretending ot speak for Steve.
>
> Reading the PEP, this text makes me uneasy:
>
> "A field is defined as any variable identified in__annotations__. That
> is, a variable that has a type annotation."
>
> And if I understand the rest of the PEP, while typing itself is optional,
> the use of type Annotation is not -- it is exactly what's being used to
> generate the fields the user wants.
>
> And the examples are all using typing -- granted, primarily the built in
> types, but still:
>
>
> @dataclass
> class C:
> a: int   # 'a' has no default value
> b: int = 0   # assign a default value for 'b'
>
>
> This sure LOOKS like typing is required. It also makes me nervous because,
> as I understand it, the types aren't actually used in the
> implementation (presumable they would be by mypy and the like?). So I think
> for folks that aren't using typing and a type checker in their development
> process, it would be pretty confusing that this means and what it actually
> does. Particularly folks that are coming from a background of a statically
> typed language.
>
> Then I see:
>
> """
> Field objects describe each defined field.
> ...
> Its documented attributes are:
>
> name: The name of the field.
> type: The type of the field.
> ...
> """
>
> So again, typing looks to be pretty baked in to the whole concept.
>
> and then:
>
> """
> One place where dataclass actually inspects the type of a field is to
> determine if a field is a class variable as defined in PEP 526.
> """
>
> and
>
> """
> The other place where dataclass inspects a type annotation is to determine
> if a field is an init-only variable. It does this by seeing if the type of
> a field is of type dataclasses.InitVar.
> """
>
> """
> Data Classes will raise a TypeError if it detects a default parameter of
> type list, dict, or set.
> """
>
> So: it seems that type hinting, while not required to use Data Classes, is
> very much baked into the implementation an examples.
>
> As I said -- this makes me uneasy -- It's a very big step that essentially
> promotes the type hinting to a new place in Python -- you will not be able
> to use a standard library class without at least a little thought about
> types and typing.
>
> I note this:
>
> """
> This discussion started on python-ideas [9] and was moved to a GitHub repo
> [10] for further discussion. As part of this discussion, we made the
> decision to use PEP 526 syntax to drive the discovery of fields.
> """
>
> I confess I only vaguely followed that discussion -- in fact, mostly I
> thought that the concept of Data Classes was a good one, and was glad to
> see SOMETHING get implemented, and didn't think I had much to contribute to
> the details of how it was done. So these issues may have already been
> raised and considered, so carry on.
>
> But:
>
> NOTE: from PEP 526:
>
> "Python will remain a dynamically typed language, and the authors have no
> desire to ever make type hints mandatory, even by convention. "
>
> The Data Classes implementation is not making it mandatory by any means,
> but it is making it a more "standard" part of the language that can not
> simply be ignored anymore. And it seems some features of dataclasses can
> only be accessed via actual typing, in addition to the requirement of type
> annotations.
>
> If nothing else, the documentation should make it very clear that the
> typing as

Re: [Python-Dev] f-strings

[Chris Barker]

On Fri, Dec 15, 2017 at 9:39 AM, Mariatta Wijaya 
wrote:

> I agree it's useful info :)
>
> I went ahead and made a PR [1].
>

Thanks! I added a couple comments to that PR.


> Not sure about updating PEP 498 at this point..
>

A little clarification text would be nice. I made a PR for that:

https://github.com/python/peps/pull/514


-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

Good Bad or Neutral, this discussion makes my point:

Using typing annotation as a necessary part of a standard library module is
injecting typing into "ordinary" python in a new way.

It is no longer going to appear to be completely optional, and only of
concern to those that choose to use it (and mypy or similar).

And I do think it is really bad UI to have something like:

@dataclass
class C:
a: Int = 1
b: float = 1.0


be the recommended (and shown in all the examples, and really be almost the
only way) to define a dataclass, when the type will in fact be completely
ignored by the implementation.

Newbies are going to be confused by this -- they really are.

Anyway, clearly I personally don't think this is a very good idea, but I
see that annotations are a natural and easy way to express the fields
without adding any new syntax.

But most importantly I don't think this should become standard without
consideration of the impact and a deliberate decision to do so.

A note: I don't know who everyone is that was engaged in the gitHub
discussion working out the details, but at least a few core folks are very
engaged in the introduction of type hinting to Python in general -- so I
think a certain perspective may have been over-represented.

Are there other options??

plain old:

@dataclass
class C:
a = 1
b = 1.0

would work, though then there would be no way to express fields without
defaults:

@dataclass
class C:
a = 1
b = None

almost -- but they is there "no default" or is the default None

Would it be impossible to use the annotation syntax, but with the type
optional:

@dataclass
class C:
a : = 1 # filed with default value
b : # field with no default

This is not legal python now, but are there barriers other than not wanting
to make yet more changes to it being legal (i.e. hard/impossible to
unambiguously parse, etc.

Maybe this can all be addresses by more "Untyped" examples  the docs.

-CHB















On Sun, Dec 17, 2017 at 8:54 AM, David Mertz  wrote:

> On Sun, Dec 17, 2017 at 8:22 AM, Guido van Rossum 
> wrote:
>
>> On Sun, Dec 17, 2017 at 2:11 AM, Julien Salort  wrote:
>>
>>> Naive question from a lurker: does it mean that it works also if one
>>> annotates with something that is not a type, e.g. a comment,
>>>
>>> @dataclass
>>> class C:
>>> a: "This represents the amplitude" = 0.0
>>> b: "This is an offset" = 0.0
>>
>>
>> I would personally not use the notation for this, but it is legal code.
>> However static type checkers like mypy won't be happy with this.
>>
>
> Mypy definitely won't like that use of annotation, but documentation
> systems might.  For example, in a hover tooltip in an IDE/editor, it's
> probably more helpful to see the descriptive message than "int" or "float"
> for the attribute.
>
> What about data that isn't built-in scalars? Does this look right to
> people (and will mypy be happy with it)?
>
> @dataclass
> class C:
> a:numpy.ndarray = numpy.random.random((3,3))
> b:MyCustomClass = MyCustomClass("foo", 37.2, 1+2j)
>
> I don't think those look terrible, but I think this looks better:
>
> @dataclass
> class C:
> a:Infer = np.random.random((3,3))
> b:Infer = MyCustomClass("foo", 37.2, 1+2j)
>
> Where the name 'Infer' (or some other spelling) was a name defined in the
> `dataclasses` module.  In this case, I don't want to use `typing.Any` since
> I really do want "the type of thing the default value has."
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

Now that dicts are order-preserving, maybe we should change prettyprint:

In [7]: d = {'one':1, 'two':2, 'three':3}

In [8]: print(d)
{'one': 1, 'two': 2, 'three': 3}

order preserved.

In [9]: pprint.pprint(d)
{'one': 1, 'three': 3, 'two': 2}

order not preserved ( sorted, I presume? )

With arbitrary order, it made sense to sort, so as to always give the same
"pretty" representation. But now that order is "part of" the dict itself,
it seems prettyprint should present the preserved order of the dict.

NOTE: I discovered this making examples for an intro to  Python class -- I
was updating the part where I teach that dicts do not preserve order. I was
using iPython, which, unbeknownst to me, was using pprint under the hood,
so got a different order depending on whether I simply displayed the dict
(which used pprint) or called str() or repr() on it. Pretty confusing.

Will changing pprint be considered a breaking change?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

I'm really surprised no one seems to get my point here.

TL;DR:
My point is that having type annotation syntax required for something in
the stdlib is a significant step toward "normalizing" type hinting in
Python. Whether that's a good idea or not is a judgement call, but it IS a
big step.


@Chris

> People are still allowed not to use dataclasses if they really don't like
> type hints :-)
> Seriously however, annotations are just syntax. In this sense PEP 526 is
> more like PEP 3107,
> and less like PEP 484. People are still free to write:
>
> @dataclass
> class C:
> x: "first coordinate"
> y: "second coordinate"
> plus: "I don't like types"
>

Well, yes, of course, but this is not like PEP 3107, as it introduces a
requirement for annotations (maybe not *type* annotations per se) in the
std lib. Again, that may be the best way to go -- but it should be done
deliberately.

@dataclass

> class C:
> x: ...
> y: ...
>

Ah! I had no idea you could use ellipses to indicate no type. That actually
helps a lot. We really should have that prominent in the docs. And in the
dataclass docs, not just the type hinting docs -- again, people will want
to use these that may not have any interest in nor prior knowledge of type
hints.



> I don't see so big difference between hypothesis (testing lib) using
> annotations for their purposes
> from the situation with dataclasses.
>

The big difference is that hypothesis is not in the standard library. Also,
I didn't know about hypothesis until just now, but their very first example
in the quick start does not use annotation syntax, so it's not as baked in
as it is with dataclasses.


> If you have ideas about how to improve the dataclass docs, this can be
> discussed in the issue https://bugs.python.org/issue32216
>

I'll try to find time to contribute there -- though maybe better to have
the doc draft in gitHub?

> ... the type will in fact be completely ignored by the implementation.
> > Newbies are going to be confused by this -- they really are.
>
> This is not different from
>
> def f(x: int):
> pass
>
> f("What")  # OK
>
> that exists starting from Python 3.0. Although I agree this is confusing,
> the way forward could be just explaining this better in the docs.
>

Again the difference is that EVERY introduction to defining python
functions doesn't use the type hint. And even more to the point, you CAN
define a function without any annotations.

But frankly, I think as type hinting becomes more common, we're going to
see a lot of confusion :-(

If you want my personal opinion about the current situation about type
> hints _in general_, then I can say that
> I have seen many cases where people use type hints where they are not
> needed
> (for example in 10 line scripts or in highly polymorphic functions), so I
> agree that some community
> style guidance (like PEP 8) may be helpful.
>

It's going to get worse before it gets better :-(

@dataclass
> class C:
>  x = field()


that does require that `field` be imported, so not as nice. I kinda like
the ellipses better.

but good to have a way.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

On Mon, Dec 18, 2017 at 7:41 PM, Steven D'Aprano 
wrote:

> > With arbitrary order, it made sense to sort, so as to always give the
> same
> > "pretty" representation. But now that order is "part of" the dict itself,
> > it seems prettyprint should present the preserved order of the dict.
>
> I disagree. Many uses of dicts are still conceptually unordered, even if
> the dict now preserves insertion order. For those use-cases, insertion
> order is of no interest whatsoever, and sorting is still "prettier".
>

and many uses of dicts have "sorted" order as completely irrelevant, and
sorting them arbitrarily is not necessarily pretty (you can't provide a
sort key can you? -- so yes, it's arbitrary)

I'm not necessarily saying we should break things, but I won't agree that
pprint sorting dicts is the "right" interface for what is actually an
order-preserving mapping.

I would think it was only the right choice in the first place in order (get
it?) to get a consistent representation, not because sorting was a good
thing per se.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Chris Barker]

On Tue, Dec 19, 2017 at 8:14 AM, Barry Warsaw  wrote:

> On Dec 18, 2017, at 22:37, Nathaniel Smith  wrote:
>
> > Wait, what? Why would changing pprint (so that it accurately reflects
> > dict's new underlying semantics!) be a breaking change? Are you
> > suggesting it shouldn't be changed in 3.7?
>
> As others have pointed out, exactly because the current behavior is
> documented.  And we all know that if it’s documented (and often even if
> it’s not, but that’s besides the point here) it will be relied upon.
>

Nathaniel Smith has pointed out that  eval(pprint(a_dict)) is supposed to
return the same dict -- so documented behavior may already be broken.

(though I assume order is still ignored when comparing dicts, so:
eval(pprint(a_dict))
== a_dict will still hold.

But practicality beats purity, and a number of folks have already posted
use-cases where they rely on sorted order, so there you go.


> So we can’t change the default behavior.  But I have no problems
> conceptually with giving users options.  The devil is in the details
> though, e.g. should we special case dictionary sorting only?



> Should we use a sort `key` to mirror sorted() and list.sort()?
>

That would be a nice feature! If anything is done, I think we should allow
a key function.

and maybe have key=None as "unsorted"

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Mon, Dec 18, 2017 at 11:49 PM, Eric V. Smith  wrote:

> I also don't think it's surprising that you can put misleading information
> (including non-types) in type annotations. All of the documentation and
> discussions are quite clear that type information is ignored at runtime.
>

Sure -- but that's documentation of type annotations -- someone
uninterested in typing, or completely unaware of it, will not be reading
those docs.


> Data Classes is also not the first use of type annotations in the stdlib:
> https://docs.python.org/3/library/typing.html#typing.NamedTuple


That's in the typing package, yes? collections.namedtuple is unchanged. So
yes, obviously the entire typing package is about typing. This is something
that has nothing to do with typing, but does use the typing syntax. It
really is different.

I haven't started teaching typing to newbies yet -- but I imagine I will
have to some day -- and when I do, it will be in the context of: here is an
optional feature that you can use along with a static type checker. And I
can make it clear that the annotations only apply to the static type
checker, and not run-time behavior.

But using type annotations for something other than providing information
to a static type checker, in an stdlib module, changes that introduction.
And people don't read all the docs -- they read to the first example of how
to use it, and away they go. And if that example is something like:

@dataclass
class C:
a: int
b: float = 0.0

There WILL be confusion.

Paul Moore wrote:

> Also, the fact that no-one raised this issue during the whole time the
> PEP was being discussed (at least as far as I recollect) and that
> Guido (who of all of us should be most aware of what is and isn't
> acceptable use of annotations in the stdlib) approved the PEP,
> suggests to me that this isn't that big a deal.


That suggests to me that the people involved in discussing the PEP may not
be representative of the bulk of Python users. There are a number of us
that are uncomfortable with static typing in general, and the python-dev
community has been criticised for doing too much, moving too fast, and
complicating the language unnecessarily.

The PEP's been accepted, so let's move forward, but please be aware of
these issues with the documentation and examples.

I'll try to contribute to that discussion as well.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] f-strings

[Chris Barker]

On Tue, Dec 19, 2017 at 8:47 AM, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> I don't see any reason not to document tips and tricks with f-strings,
> and this is a nice and useful example.  But it looks like TOOWTDI to
> me.  The syntax is documented (6.1.3.1 in the Library Reference),
> along with a specific relevant example ("Aligning the text and
> specifying a width" in 6.1.3.2).
>
> So -1 on putting the recipe in the reference docs.  I really don't
> think this kind of information belongs in a PEP for sure, and probably
> not even in the Library Reference.


The docs (and I think PEP) have been updated to clearly state that
f-strings use the same formatting specifiers as .format(), and have links
to those docs.

So I think we're good.


> The Tutorial might be a good place
> for it, though.
>

yup.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Wed, Dec 20, 2017 at 5:29 PM, Eric V. Smith  wrote:

> There is definitely a passive bias towards using types with dataclasses in
> that the Eric (the author) doesn't appear to want an example without them
> in the pep/docs.
>
>>
>> I'm not sure what such an example would look like. Do you mean without
>> annotations?
>
>
IIUC, there is not way to make a dataclass without annotations, yes? That
it, using annotations to determine the fields is the one and only way the
decorator works. So it's impossible to give an example without annotations,
yes?


> Or do you mean without specifying the "correct" type, like:
>>
>> @dataclass
>> class C:
>> x: int = 'hello world'
>>
>
It may be a good idea to have an example like that in the docs (but
probably not the PEP) to make it clear that the type is not used in any way
at run time.

But I don't think that anyone is suggesting that would be  a recommended
practice.

I suggest that it be clear in the docs, and ideally in the PEP, that the
dataclass decorator is using the *annotation" syntax, and that the the only
relevant part it uses is that an annotation exists, but the value of the
annotation is essentially (completely?) ignored. So we should have examples
like:

@dataclass
class C:
a: ...  # field with no default
b: ... = 0 # filed with a default value

Then maybe:

@dataclass
class C:
a: "the a parameter" # field with no default
b: "another, different parameter" = 0.0 # field with a default

Then the docs can go to say that if the user wants to specify a type for
use with a static type checking pre-processor, they can do it like so:

@dataclass
class C:
a: int # integer field with no default
b: float = 0.0 # float field with a default

And the types will be recognized by type checkers such as mypy.

And I think the non-typed examples should go first in the docs.

This is completely analogous to how all the other parts of python are
taught. Would anyone suggest that the very first example of a function
definition that a newbie sees would be:

def func(a: int, b:float = 0.0):
body_of_function

Then, _maybe_ way down on the page, you mention that oh, by the way, those
types are completely ignored by Python. And not even give any examples
without types?


>  Re-reading my post you referenced, is it just an example using
typing.Any?

I actually think that is exactly the wrong point -- typing.Any is still
using type hinting -- it's an explicit way to say, "any type will do", but
it's only relevant if you are using a type checker. We really need examples
for folks that don't know or care about type hinting at all.

typing.Any is for use by people that are explicitly adding type hinting,
and should be discussed in type hinting documentation.

>  I'm okay with that in the docs, I just didn't want to focus on it in the
PEP. I want the PEP to only
> have the one reference to typing, for typing.ClassVar. I figure the
people reading the PEP can
> extrapolate to all of the possible uses for annotations that they don't
need to see a typing.Any
> example.

no they don't, but they DO need to see examples without type hints at all.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Thu, Dec 21, 2017 at 11:55 AM, Terry Reedy  wrote:

I think the understanding problem with this feature arises from two
> factors: using annotations to define possibly un-initialized slots is
> non-obvious; a new use of annotations for something other than static
> typing is a bit of a reversal of the recent pronouncement 'annotations
> should only be used for static typing'.


you know, that may be where part of my confusion came from -- all the talk
lately has been about "type hints" and "type annotations" -- the idea of
"arbitrary annotations" has been lost.


> Therefore, getting the permanent doc 'right' is important.
>

yup.


> @dataclass
> class C:
> x
> y = 0
>
> I think the doc should explicitly say that uninitialized fields require
> annotation with something (anything, not necessarily a type) simply to
> avoid NameError during class creation.


would this be possible?

@dataclass
class C:
x:
y: = 0

That is -- the colon indicates an annotation, but in this case, it's a
"nothing" annotation.

It's a syntax error now, but would it be possible to change that? Or would
the parsing be ambiguous? particularly in other contexts.

of course, then we'd need something to store in as a "nothing" annotation
-- empty string? None? (but None might mean something) create yet anther
type for "nothing_annotation"

Hmm, I may have talked myself out of it

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Thu, Dec 21, 2017 at 3:36 PM, Gregory P. Smith  wrote:


>  But we already have ... which does - so I'd suggest that for people who
> are averse to importing anything from typing and using the also quite
> readable Any.  (ie: document this as the expected practice with both having
> the same meaning)
>

I don't think they do, actually - I haven't been following the typing
discussions, but someone in this thread said that ... means "use the type
of teh default" or something like that.


> While I consider the annotation to be a good feature of data classes, it
> seems worth documenting that people not running a type analyzer should
> avoid declaring a type.
>

+1 !


> A worse thing than no-type being specified is a wrong type being
> specified. That appearing in a library will break people who need their
> code to pass the analyzer and pytype, mypy, et. al. could be forced to
> implement a typeshed.pypi of sorts containing blacklists of known bad
> annotations in public libraries and/or actually correct type specification
> overrides for them.
>

and the wrong type could be very common -- folks using "int", when float
would do just fine, or "list" when any iterable would do, the list goes on
and on. Typing is actually pretty complex in Python -- it's hard to get
right, and if you aren't actually running a type checker, you'd never know.

One challenge here is that annotations, per se, aren't only for typing. Bu
tit would be nice if a type checker could see whatever "non-type" is
recommended for dataclasses as "type not specified". Does an ellipses spell
that? or None? or anything that doesn't have to be imported from typing :-)

As for problems with order, if we were to accept
>
> @dataclass
> class Spam:
> beans = True
> ham: bool
>
> style instead, would it be objectionable to require keyword arguments only
> for dataclass __init__ methods?  That'd get rid of the need to care about
> order.
>

wouldn't  that make the "ham: bool" legal -- i.e. no default?

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Fri, Dec 22, 2017 at 8:49 AM, Brett Cannon  wrote:

> I  think it's worth reminding people that if they don't like the fact
>> dataclasses (ab)use type hints for their succinct syntax that you can
>> always use attrs instead to avoid type hints.
>>
>
sure -- but this doesn't really address the issue, the whole reason this is
even a discussion is because dataclasses is going into the standard
library. Third party packages can do whatever they want, of course.

And the concern is that people (in particular newbies) will get confused /
the wrong impression / other-negative-response by the (semi) use of typing
in a standard library module.


> As for those who feel dataclasses will force them to teach type hints and
> they simply don't want to, maybe we could help land protocols
>

Could you please clarify what this is about ???


> But I think the key point I want to make is Guido chose dataclasses to
> support using the type hints syntax specifically over how attrs does
> things, so I don't see this thread trying to work around that going
> anywhere at this point since I haven't seen a solid alternative be proposed
> after all of this debating.
>

And the PEP has been approved.

So the actionable things are:

Writing good docs

Converging on a "recommended" way to do non-typed dataclass fields.

And that should be decided in order to write the docs, (and probably should
be in the PEP).

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Fri, Dec 22, 2017 at 10:10 AM, Stephan Hoyer  wrote:

> On Thu, Dec 21, 2017 at 6:39 AM Ivan Levkivskyi 
> wrote:
>
>>

> * ... (ellipsis): this class may or may not be used with static type
>> checkers, use the inferred type in the latter case
>>
>

> * "field docstring": this class should not be used with static type
>> checkers
>>
>> Assuming this, the second option would be the "real" "don't care". If
>> this makes sense,
>> then we can go the way proposed in https://github.com/python/
>> typing/issues/276 and make ellipsis semantics "official" in PEP 484.
>> (pending Guido's approval)
>>
>
> I am a little nervous about using "..." for inferred types, because it
> could potentially cause confusion with other uses of ellipsis in typing.
>

Isn't that what "make ellipsis semantics "official"" means -- i.e. making
it clear how they are used in typing?

The core problem is that generic annotations are used in dataclasses
without the "type hints" use-case. But:

1) Python is moving to make (PEP 484) type hints be THE recommended usage
for annotations

2) We want the annotations in dataclasses to be "proper" PEP 484 type hints
if they are there.

The challenge is:

- Annotations require a value.
- Any value used might be interpreted by a static type checker.

So we need a way to spell "no type specified" that will not be
mis-interpreted by type checkers, and is in the built in namespace, and
will seem natural to users with no knowledge or interest in static typing.

The ellipses is tempting, because it's a literal that doesn't have any
other obvious meaning in this context. Bu tif it has an incompatible
meaning in PEP 484, then we're stuck.

Is there another Obscure literal that would work?

 - I assume None means "the None type" to type checkers, yes?

 - empty string is one option -- or more to the point, any string -- so
then it could be used as docs as well.

- Is there another Obscure literal that would work? (or not so obscure one
that doesn't have another meaning to type checkers)

Would it be crazy to bring typing.Any into the builtin namespace?

@dataclass:
a: Any
b: Any = 34
c: int = 0

That reads pretty well to me

And having Any available in the built in namespace may help in other cases
where type hints are getting introduced into code that isn't really being
properly type checked.

I don't LOVE it -- to me, Any means "any type will do", or "I don't care
what type this is" and what we really want is "no type specified" -- i.e.
the same thing as plain old Python code without type hints. But practically
speaking, it has the same effect, yes?

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Fri, Dec 22, 2017 at 1:18 PM, MRAB  wrote:

>
>> The function is "any", the type is "Any", and "any" != "Any", although I
> wonder how many people will be caught out by that...


enough that it's a bad idea

oh well.

-CHB




> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.
> barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Chris Barker]

On Sat, Dec 23, 2017 at 5:54 PM, Nick Coghlan  wrote:

>
> I still wonder about the "fields *must* be annotated" constraint though. I
> can understand a constraint that the style be *consistent* (i.e. all fields
> as annotations, or all fields as field instances), since that's needed to
> determine the field order, but I don't see the problem with the "no
> annotations" style otherwise.
>

IIUC, without annotations, there is no way to set a field with no default.

And supporting both approaches violates "only one way to do it" in, I
think, a confusing manner -- particularly if you can't mix and match them.

Also, could does using class attributes without annotations make a mess
when subclassing? -- no I haven't thought that out yet.

-CHB








>
> Cheers,
> Nick.
>
>
>
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Chris Barker]

On Sat, Dec 30, 2017 at 7:27 AM, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

>   Just use the simple rule that a new
> __repr__ is generated unless provided in the dataclass.
>

are we only talking about __repr__ here ???

I interpretted Guido's proposal as being about all methods -- we _may_ want
something special for __repr__, but I hope not.

But +1 for Guido's proposal, not only because it's easy to explain, but
because it more naturally follows the usual inheritance logic:

The decorator's entire point is to auto-generate boilerplate code for you.
Once it's done that it shouldn't, in the end, behave any differently than
if you hand wrote that code. If you hand wrote the methods that the
decorator creates for you, they would override any base class versions. So
that's what it should do.

And the fact that you can optionally tell it not to in some particular case
keeps full flexibility.

-CHB

> I grant that there may be many reasons why one would be deriving

> dataclasses from dataclasses


Will you get the "right" __repr__ now if you derive a datacalss from a
dataclass? That would be a nice feature.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Chris Barker]

On Mon, Jan 1, 2018 at 7:50 PM, Ethan Smith  wrote:

>
> Will you get the "right" __repr__ now if you derive a datacalss from a
>> dataclass? That would be a nice feature.
>>
>
>
> The __repr__ will be generated by the child dataclass unless the user
> overrides it. So I believe this is the "right" __repr__.
>

what I was wondering is if the child will know about all the fields in the
parent -- so it could make a full __repr__.

-CHB




> ~>Ethan Smith
>
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R(206) 526-6959   voice
>> 7600 Sand Point Way NE
>> 
>>   (206) 526-6329   fax
>> Seattle, WA  98115   (206) 526-6317   main reception
>>
>> chris.bar...@noaa.gov
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ethan%
>> 40ethanhs.me
>>
>>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] OS-X builds for 3.7.0

[Chris Barker]

Ned,

It looks like you're still building OS-X the same way as in the past:

Intel 32+64 bit, 10.6 compatibility

Is that right?

Might it be time for an update?

Do we still need to support 32 bit?  From:

https://apple.stackexchange.com/questions/99640/how-old-are-macs-that-cannot-run-64-bit-applications

There has not been a 32 bit-only Mac sold since 2006, and a out-of the box
32 bit OS since 2006 or 2007

I can't find out what the older OS version Apple supports, but I know my IT
dept has been making me upgrade, so I"m going to guess 10.8 or newer...

And maybe we could even get rid of the "Framework" builds..

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

[Chris Barker]

On Wed, Jan 31, 2018 at 3:13 AM, Joni Orponen 
wrote:

> On Wed, Jan 31, 2018 at 12:43 AM, Chris Barker - NOAA Federal <
> chris.bar...@noaa.gov> wrote:
>
>> And maybe we could even get rid of the "Framework" builds..
>>>
>>
>> Please do not. These make life easier for doing things the Apple way for
>> signed sandboxed applications.
>>
>> For the record, are you re-distributing the python.org builds, or
>> re-building yourself?
>>
>
> We are re-building ourselves.
>

Then it makes no difference to you if the pyton.org binaries are Framework
builds... though maybe you want the configure target available.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

[Chris Barker]

On Wed, Jan 31, 2018 at 4:20 AM, INADA Naoki  wrote:

> > Against the official CPython 3.6 (probably .3 or .4) release I see:
> > 1 that is 2.01x faster (python-startup, 24.6ms down to 12.2ms)
> > 5 that are >=1.5x,<1.6x faster.
> > 13 that are >=1.4x,<1.5x faster.
> > 21 that are >=1.3x,<1.4x faster.
> > 14 that are >=1.2x,<1.3x faster.
> > 5 that are >=1.1x,<1.2x faster.
> > 0 that are < 1.1x faster/slower.
> >
> > Pretty good numbers overall I think.
>


> Yay!!  Congrats for all of us!
>

I'm confused -- I _think_ these are performance improvements of the
Anaconda build over the python.org build for OS-X -- so congrats to the
Anaconda team :-)

But a hint that maybe we should do the python.org builds differently!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

[Chris Barker]

>> Ned Deily is in charge of the Mac build (as well as current release
> manager).  Within the last week, he revised the official builds (now two, I
> believe) for 3.7.0b1, due in a day or so.  One will be a future oriented
> 64-bit build.  The PR and What's New have more.
>

What's New doesn't mention it, but Ned's annoucemtn does:

"""
Attention macOS users: with 3.7.0b1, we are providing a choice of
two binary installers.  The new variant provides a 64-bit-only
version for macOS 10.9 and later systems; this variant also now
includes its own built-in version of Tcl/Tk 8.6.  We welcome your
feedback.
"""

So that's a start -- thanks Ned!

He may not be reading this thread, but will read MacOS tracker issues with
> a specific proposal, data and a patch.  Comparisons should be against the
> current master or an installed 3.7.0b1.


I hope the folks on this thread that know what they are doing can test and
make suggestions :-)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Chris Barker]

On Fri, Feb 2, 2018 at 7:38 AM, Elvis Pranskevichus 
wrote:

> On Friday, February 2, 2018 10:08:43 AM EST Eric V. Smith wrote:
> > However, I don't feel very strongly about this. As I've said, I expect
> > the use cases for hash=True to be very, very rare.
>
> Why do you think that the requirement to make a dataclass hashable is a
> "very, very rare" requirement?


I think what's rare is wanting hashability without it being frozen.


> Just put yourself in the shoes of an average Python developer.  You try
> to put a dataclass in a set, you get a TypeError.  Your immediate
> reaction is to add "hash=True".  Things appear to work.


agreed, the easy and obvious way should be to make it frozen -- if it's
hard to make it hashable and not frozen, then that's good.

But it is nice to have the __hash__ generated more you so maybe a flag
for "unfrozen_hashable" -- really klunky, but if that is a rare need, then
there you go.

Or maybe:

If either hash or frozen is specified, it become both frozen and hashable.

If both are specified, then it does what the user is asking for.

-CHB
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Immutability vs. hashability

[Chris Barker]

On Sun, Feb 4, 2018 at 7:54 PM, Nick Coghlan  wrote:

> On 5 February 2018 at 08:31, Guido van Rossum  wrote:
> > On Sun, Feb 4, 2018 at 11:59 AM, Chris Barker - NOAA Federal
> >  wrote:
> >>
> >> I think the folks that are concerned about this issue are quite right
> >> — most Python users equate immutable and hashable—so the dataclass API
> >> should reflect that.
> >
> > Since they are *not* equivalent (consider a tuple containing a list) I'm
> not
> > at all convinced that any API in the core language should "reflect" this
> > misconception, depending on how you meant that.
>
> Lists are themselves mutable, and hence inherently unhashable.
>
> Tuples are themselves immutable, and hence hashable if their contents are.
>
> I interpret Chris's comment as saying that data classes should behave
> the same way that the builtin container types do:
>

pretty much, yes,

But a bit more detail -- I'm commenting on the API, not the capability -
that is, since users often equate hashable and immutability, they will
expect that if they say hash=True, then will get an immutable, and if they
say frozen=True, they will get something hashable (as long as the fields
are hashable, just like a tuple.

That is, even though these concepts are independent, the defaults shouldn't
reflect that.

It's the ability to ask the interpreter to guess what you mean
> "frozen=False, hash=True" that creates the likelihood of confusion.
>

Actually, I think if the user does explicitly specify: "frozen=False,
hash=True", then that's what they should get, and it's a pretty fragile
beast, but apparently there's enough of a use case for folks to want it,
and I don't think it's a confusing API.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Chris Barker]

On Sun, Feb 4, 2018 at 11:57 PM, Gregory P. Smith  wrote:

> +1 using unsafe_hash as a name addresses my concern.
>
mine too -- anyone surprised by using this deserves what they get :-)

-CHB


On Sun, Feb 4, 2018, 9:50 PM Guido van Rossum  wrote:
>
>> Looks like this is turning into a major flamewar regardless of what I
>> say. :-(
>>
>> I really don't want to lose the ability to add a hash function to a
>> mutable dataclass by flipping a flag in the decorator. I'll explain below.
>> But I am fine if this flag has a name that clearly signals it's an unsafe
>> thing to do.
>>
>> I propose to replace the existing (as of 3.7.0b1) hash= keyword for the
>> @dataclass decorator with a simpler flag named unsafe_hash=. This would be
>> a simple bool (not a tri-state flag like the current hash=None|False|True).
>> The default would be False, and the behavior then would be to add a hash
>> function automatically only if it's safe (using the same rules as for
>> hash=None currently). With unsafe_hash=True, a hash function would always
>> be generated that takes all fields into account except those declared using
>> field(hash=False). If there's already a `def __hash__` in the function I
>> don't care what it does, maybe it should raise rather than quietly doing
>> nothing or quietly overwriting it.
>>
>> Here's my use case.
>>
>> A frozen class requires a lot of discipline, since you have to compute
>> the values of all fields before calling the constructor. A mutable class
>> allows other initialization patterns, e.g. manually setting some fields
>> after the instance has been constructed, or having a separate non-dunder
>> init() method. There may be good reasons for using these patterns, e.g. the
>> object may be part of a cycle (e.g. parent/child links in a tree). Or you
>> may just use one of these patterns because you're a pretty casual coder. Or
>> you're modeling something external.
>>
>> My point is that once you have one of those patterns in place, changing
>> your code to avoid them may be difficult. And yet your code may treat the
>> objects as essentially immutable after the initialization phase (e.g. a
>> parse tree). So if you create a dataclass and start coding like that for a
>> while, and much later you need to put one of these into a set or use it as
>> a dict key, switching to frozen=True may not be a quick option. And writing
>> a __hash__ method by hand may feel like a lot of busywork. So this is where
>> [unsafe_]hash=True would come in handy.
>>
>> I think naming the flag unsafe_hash should take away most objections,
>> since it will be clear that this is not a safe thing to do. People who
>> don't understand the danger are likely to copy a worse solution from
>> StackOverflow anyway. The docs can point to frozen=True and explain the
>> danger.
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> greg%40krypto.org
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Immutability vs. hashability

[Chris Barker]

On Mon, Feb 5, 2018 at 3:37 PM, Steven D'Aprano  wrote:

> On Sun, Feb 04, 2018 at 09:18:25PM -0800, Guido van Rossum wrote:
>
> > The way I think of it generally is that immutability is a property of
> > types, while hashability is a property of values.
>
> That's a great way to look at it, thanks.
>

hmm -- maybe we should get a ValueError then when you try to use a
non-hashable value?

In [*9*]: t = ([1,2,3],)


In [*10*]: set(t)

---

TypeError Traceback (most recent call last)

 in ()

> 1 set(t)


TypeError: unhashable type: 'list'


Of course, in this case, the error is triggered by the type of the zeroth
element of the tuple, not by the  value of the tuple per se.

Which means that hashability really is a property of type -- but container
types require a specific way of thinking -- hashability is not determined
by the type of the container (or may not be), but by the types of it's
contents. Is that the value of the container?

So maybe: an object is hashable if it is a hashable type, and if it is a
container, if it's contents are hashable types.


With dataclasses as they stand -- it seems the values of the fields does
not affect hashability:

(this is the version 0.4 from PyPi -- disregard if it's out of date)

Unhashable by default:

In [*14*]: @dataclasses.dataclass()

...: *class* *NoHash*:

...: x = 5

...: l = [1,2,3]

...:


In [*15*]: set((nh,))

---

TypeError Traceback (most recent call last)

 in ()

> 1 set((nh,))


TypeError: unhashable type: 'NoHash'



OK, that's what we expect.



But then if it is hashable:

In [*19*]: @dataclasses.dataclass(hash=*True*)

...: *class* *Hash*:

...: x = 5

...: l = [1,2,3]

...:


In [*20*]: h = Hash()


In [*21*]: set((h,))

Out[*21*]: {Hash()}



All works, regardless of the values of the fields

I haven't looked at the code -- but it appears the hash has nothing to do
with the values of the fields:

In [*23*]: hash(h)

Out[*23*]: 3527539


In [*24*]: h.l.append(6)


In [*25*]: hash(h)

Out[*25*]: 3527539


In [*26*]: h.x = 7


In [*27*]: hash(h)

Out[*27*]: 3527539

and it looks like all instances hash the same:

In [*31*]: h2 = Hash()


In [*32*]: hash(h2)

Out[*32*]: 3527539


In [*33*]: hash(h)

Out[*33*]: 3527539

So I'm wondering how hashablility is useful at all?

But it sure looks like there's a lot of room for confusion and error, even
if it's a frozen dataclass.

This may a case where we need to really make sure the docs are good!

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Apology -- ignore previous message

[Chris Barker]

Sorry -- too big a brain blip this morning.. my experiments were'n't
actually creating dataclasses properly.

Though it does show why the reliance on type hinting in troubling!!

IGNORE that message!

Sorry for the noise.

-CHB


On Mon, Feb 5, 2018 at 3:37 PM, Steven D'Aprano  wrote:

> On Sun, Feb 04, 2018 at 09:18:25PM -0800, Guido van Rossum wrote:
>
> > The way I think of it generally is that immutability is a property of
> > types, while hashability is a property of values.
>
> That's a great way to look at it, thanks.
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Immutability vs. hashability

[Chris Barker]

I don't seem to be getting my own messages back to reply to, but:

yes, of course, dataclasses won't hash if a field is mutable:


In [*58*]: @dataclasses.dataclass(hash=*True*, frozen=*True*)

...: *class* *Hash*:

...: x: int  = 5

...: l: list = dataclasses.field(default_factory=list)

...:

...:

...:

...:


In [*59*]: h = Hash()


In [*60*]: hash(h)

---

TypeError Traceback (most recent call last)

 in ()

> 1 hash(h)


/Users/chris.barker/miniconda2/envs/py3/lib/python3.6/site-packages/dataclasses.py
in __hash__(self)


TypeError: unhashable type: 'list'


And the hash does depend on the values of the fields.So you can REALLY
ignore my previous note.

Again, sorry for the noise.

-CHB



On Mon, Feb 5, 2018 at 3:37 PM, Steven D'Aprano  wrote:

> On Sun, Feb 04, 2018 at 09:18:25PM -0800, Guido van Rossum wrote:
>
> > The way I think of it generally is that immutability is a property of
> > types, while hashability is a property of values.
>
> That's a great way to look at it, thanks.
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Immutability vs. hashability

[Chris Barker]

On Mon, Feb 5, 2018 at 5:17 PM, Steven D'Aprano  wrote:

> I'm not happy about the concept of pandering to the least capable, most
> ignorant programmers by baking a miscomprehension into an important
> standard library API.


I don't think this is "baking a miscomprehension", but rather adhering to
the principle of least surprize.


> (Things would be different if we just outright banned mutable+hashable,
> but I don't think anyone wants that.)
>

yeah, though I'm still not sure what the hashable mutable dataclass use
case is.

Fortunately, I also believe that the number of programmers who would
> fail to draw the right conclusion from the existence of separate
> switches will actually be pretty small in practice. The fact that there
> are two separate switches is a pretty big clue that mutability and
> hashability can be controlled separately.
>

well, yes, but as we all know, more people read code than write it -- so we
should be concerned about what a mid-level programmer will conclude when
s/he sees:

@dataclass(frozen=True)

or

@dataclass(hash=True)

It may not be clear that the other options is available, and what it's
default it.


> I believe that the proposed API is much simpler to understand than your
> revision. We have:
>
> - frozen and hash both default to False;
> - if you explicitly set one, the other uses the default.
>
> This corresponds to a very common, Pythonic pattern that nearly
> everyone is familiar with:
>
> def spam(frozen=False, hash=False):
> ...
>
> which is easy to understand and easy to explain. Versus your proposal:
>
> - if you set neither, then frozen and hash both default to False;
> - but if you explicitly set one, the other uses True, namely the
>   opposite of the standard default.
>
> which corresponds to something harder to describe and much less common:
>
> def spam(frozen=None, hash=None):
> if frozen is hash is None:
> frozen = hash = False
> elif frozen is None:
> frozen = True
> elif hash is None:
> hash = True
> ...
>
> "frozen and hash default to True, unless neither are set, in which case
> they default to False."
>

I agree that it's easier to document, but we all know no one reads
documentation anyway.

My  argument (which may not be correct) is based on the idea that the
hash=True, frozen=False is the rare, specialized use case -- the folks that
need that will figure it out.

(1) I set frozen=True thinking that makes the class hashable so I can
> use it in a set or hash. The first time I actually do so, I get an
> explicit and obvious TypeError. Problem solved.[1]
>
> (2) I set hash=True thinking that makes the class frozen. This scenario
> is more problematic, because there's no explicit and obvious error when
> I get it wrong. Instead, my program could silently do the wrong thing if
> my instances are quietly mutated.
>
> The first error is self-correcting, and so I believe that the second is
> the only one we should worry about.


fair enough.


> - how much should we worry? (how often will this happen?);
>

I probably wouldn't be worried at all, except that the word "frozen" is
used in the built-in frozenset, with this documentation:

"""
The frozenset type is immutable and hashable — its contents cannot be
altered after it is created; it can therefore be used as a dictionary key
or as an element of another set.
"""

and frozenset pops up pretty high in google if you search "frozen python
frozen" (second to the topic of frozen binaries...)


> I don't think this is a failure mode that we need to be concerned with.
> We can't protect everyone from everything.
>

I suppose you're right -- I think were we disagree is how confusing my
proposal is -- I think the common less error prone use cases should be easy
and obvious to do, and the uncommon, more "expert" use-cases should be
possible and clearly documented when the code is read.

But it's not that big a deal -- I'm done -- thanks for specifically
addressing the issues I brought up.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses, frozen and __post_init__

[Chris Barker]

On Tue, Feb 20, 2018 at 8:45 AM, Guido van Rossum  wrote:

> TBH, I don't hate Nick's solution that assigns to __class__ in
> __post_init__. You can probably come up with a class decorator that hides
> construction of the frozen subclass. I do think that it should be used very
> sparingly, as the existence of the subclass is hard to hide: instances of
> MyRecord will show as instances of _LockedMyRecord when printed or
> otherwise inspected, which could be confusing.
>
> There's another very simple low-tech solution:
>
> @dataclass
> class MyRecord:
> a: int
> # etc.
> frozen: bool = False
> def freeze(self):
> self.frozen = True
> def __setattr__(self, name, value):
> if self.frozen: raise AttributeError
> super().__setattr__(name, value)
> def __hash__(self):
> assert self.frozen
> return super().__hash__()
>

This sure seems far more straightforward and les magic than __class__
swapping.


> It can easily be weaponized via an extra decorator, and it's simpler in
> that it doesn't use a magical second class. I'm just not keen on adding the
> extra attribute to all frozen instances created via
> @dataclass(frozen=True), because it's extra space overhead.
>

Is that one attribute that big a deal? I suppose that simple dataclasses
with only two or so attributes would see a significant change, but if
you're really worried about space, wouldn't you use a namedtuple or simply
a tuple anyway?

(In fact you can conclude that I'm not keen on exposing *any* of these
> "solutions" as a flag on the @dataclass() decorator. They all feel pretty
> fragile to me.)
>

so provide a:

@freezable_dataclass

Additional decorator that folks that want the ability to freeze and
unfreeze instances can use?

and if it overrides __set__attr, it could be used an virtually any class to
make a frozen version, yes?

I can't say I have enough need to bother writing that myself, but if others
do -- it seems like a good solution.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467 (Minor API improvements for binary sequences) - any thoughts?

[Chris Barker]

On Wed, Feb 21, 2018 at 11:55 AM, Elias Zamaria  wrote:

> This is about some minor changes to the bytes, bytearray, and memoryview
> classes. Here is the PEP: https://www.python.org/dev/peps/pep-0467/
>


> I am waiting for this to be merged, or approved, or whatever is the next
> step. Someone on the bug tracker mentioned restarting the discussion on the
> mailing list, so that is what I'm trying to do here. Does anyone have any
> thoughts?
>

+1 all around.

One other thought:

: Addition of optimised iterator methods that produce bytes objects

Maybe it would make sense to have a "byte" type that holds a single byte.
It would be an integer that could only hold values from 0-255. Then the
regular iterator could simply return a bunch of single byte objects.

I can't say I've thought it through, but if a byte is a int with restricted
range, then it could act like an int in (almost?) every context, so there
would be no need for a separate iterator.

I also haven't thought through whether there is any real advantage to
having such a type -- but off the top of my head, making a distinction
between a bytes object that happens to be length-one and a single byte
could be handy. I sure do often wish for a character object.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467 (Minor API improvements for binary sequences) - any thoughts?

[Chris Barker]

On Wed, Feb 21, 2018 at 12:21 PM, Ethan Furman  wrote:

> At this point the PEP itself has not been approved, and is undergoing
> changes.  I don't see anything happening with it right now while 3.7 is
> going through it's final stages to release.  Once 3.7.0 is published we can
> come back to this.
>

well, it was originally targeted for 3.5, so it did need kick to get going.

Probably too late for 3.7.0, but no reason not to get it moving if there is
support, and aren't objections.

Anyone know if it paused to controversy or just lack of momentum? From a
quick search, it looks like discussion simply petered out in Sept 2016.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467 (Minor API improvements for binary sequences) - any thoughts?

[Chris Barker]

On Wed, Feb 21, 2018 at 12:39 PM, Steve Holden  wrote:

> I think the chances of a "byte" object are about as good as the chances of
> a character object
>

probably right.


> (though one can always implement such in C extensions, that wouldn't build
> them into the syntax).
>

I think you could simply subclass, too (overriding __new__ and a couple
methods). But that would do exactly no good, unless you used your own
custom string and bytes objects, too. The whole point is that iterating
over a string (Or bytes) always returns an also-iterable object,
ad-infinitum.

This is the cause of the major remaining common "type error" in Python.
(the old integer division used to be the big one)


> The fact that characters are single-byte strings is responsible for
> certain anomalies with (e.g.) the __contains__ operator (list elements
> aren't lists, but string element are strings), but overall the choices made
> lead to sensible, comprehensible code.
>

I'm pretty convinced that the choice not to have a character type has had
basically zero benefits to sensible, comprehensible code, though it's not a
very big deal, either. not a big enough deal for the churn it would cause
to introduce it now, that's for sure.

so +1 for this PEP as is.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The `for y in [x]` idiom in comprehensions

[Chris Barker]

On Fri, Feb 23, 2018 at 9:51 AM, Guido van Rossum  wrote:

> As to the validity or legality of this code, it's both, and working as
> intended.
>
> A list comprehension of the form
>
> [STUFF for VAR1 in SEQ1 for VAR2 in SEQ2 for VAR3 in SEQ3]
>
> should be seen (informally) as
>
> for VAR1 in SEQ1:
> for VAR2 in SEQ2:
> for VAR3 in SEQ3:
> "put STUFF in the result"
>

Thanks -- right after posting, I realized that was the way to unpack it to
understand it. I think my confusion came from two things:

1) I usually don't care in which order the loops are ordered -- i.e., that
could be:

for VAR3 in SEQ3:
for VAR2 in SEQ2:
for VAR1 in SEQ1:
"put STUFF in the result"

As I usually don't care, I have to think about it (and maybe experiment to
be sure). (this is the old Fortran vs C order thing :-)

2) since it's a single expression, I wasn't sure of the evaluation order,
so maybe (in my head) it could have been (optimized) to be:

[STUFF for  VAR1 in Expression_that_evaluates_to_an_iterable1 for VAR2 in
Expression_that_evaluates_to_an_iterable2]

and that could translate to:

IT1 = Expression_that_evaluates_to_an_iterable1
IT2 = Expression_that_evaluates_to_an_iterable2
for VAR1 in IT1:
for VAR2 in IT2:
"put STUFF in the result"

In which case, VAR1 would not be available to
Expression_that_evaluates_to_an_iterable2.

Maybe that was very wrong headed -- but that's where my head went -- and
I'm not a Python newbie (maybe an oddity, though :-) )

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The `for y in [x]` idiom in comprehensions

[Chris Barker]

On Fri, Feb 23, 2018 at 10:45 AM, Guido van Rossum  wrote:

> There are useful things you can only do with comprehensions if the second
> for-loop can use the variable in the first for-loop. E.g.
>
> [(i, j) for i in range(10) for j in range(i)]
>

indeed -- and that is fairly common use-case in nested for loops -- so good
to preserve this.

But I still think the original:

[g(y) for x in range(5) for y in [f(x)]]

Is always going to be confusing to read. Though I do agree that it's not
too bad when you unpack it into for loops:

In [89]: for x in range(5):
...: for y in [f(x)]:
...: l.append(g(y))

BTW, would it be even a tiny bit more efficient to use a tuple in the inner
loop?

[g(y) for x in range(5) for y in (f(x),)]

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ttk.Treeview.insert() does not allow to insert item with iid=0

[Chris Barker]

On Fri, Mar 16, 2018 at 10:54 AM, Игорь Яковченко 
wrote:

> I investigated problem and found that in ttk.py, Treeview.insert(...
> iid=None, ...) in method's body has a check:
> if iid:
> res = self.tk.call(self._w, "insert", parent, index,
> "-id", iid, *opts)
> else:
> res = self.tk.call(self._w, "insert", parent, index, *opts)
> It means that if iid is "True" then use it else autogenerate it.
> Maybe there should be "if iid is not None", not "if iid"? Or there are
> some reasons to do check this way?
>

isn't it considered pythonic to both: use None as a default for "not
specified" AND use:

if something is None

to check if the parameter has been specified?

however, this is a bit of an odd case:

ids are strings, but it allows you to pass in a non-string and stringified
version will be used.

so None should be the only special case -- not "anything false"

(but if the empty string is the root, then it's another special case --
again, good to check for none rather than anything Falsey)

so it probably should do something like:


  if iid is not None:
res = self.tk.call(self._w, "insert", parent, index,
"-id", str(iid), *opts)
else:
res = self.tk.call(self._w, "insert", parent, index, *opts)

note both the check for None and the str() call.

I'm assuming the str() call happens under the hood at the boundary already,
but better to make it explicit in teh Python.

Alternatively: this has been around a LONG time, so the maybe the answer is
"don't do that" -- i.e. don't use anything falsey as an iid. But it would
still be good to make the docs more clear about that.

-CHB

---
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symmetry arguments for API expansion

[Chris Barker]

It seems .as_integer_ratio() has been resolved.

what about the original .is_integer() request? (Or did I miss that somehow?)

Anyway, it seems like __index__() should play a role here somehow... isn't
that how you ask an object for the integer version of itself?

Could float et al. add an __index__ method that would return a ValueError
if the value was not an integer?

Of course, as pointed out earlier in this thread, an "exact" integer is
probably not what you want with a float anyway

-CHB


On Tue, Mar 13, 2018 at 10:29 PM, Tim Peters  wrote:

> [Tim]
> >> An obvious way to extend it is for Fraction() to look for a special
> >> method too, say "_as_integer_ratio()".
>
> [Greg Ewing]
> > Why not __as_integer_ratio__?
>
> Because. at this point, that would be beating a dead horse ;-)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symmetry arguments for API expansion

[Chris Barker]

On Wed, Mar 21, 2018 at 4:42 AM Steven D'Aprano  wrote:

>
>
> > Could float et al. add an __index__ method that would return a ValueError
> > if the value was not an integer?
>
> That would allow us to write things like:
>
> "abcdefgh"[5.0]
>
> which is one of the things __index__ was invented to prevent.


I’m not so sure — it was invented to prevent using e.g. 6.1 as an index,
which int(I) would allow.

More specifically, it was invented to Allow true integers that aren’t a
python int ( like numpy int types).

But, in fact, it is common to use floating point computation to compute an
index — though usually one would make a conscious choice between round()
and floor() and ceil() when doing so.

Passing floor(a_float) as an index is a perfectly reasonable thing to do.

But Guidos point is well taken — Having __index__ fail based on value is
setting people up for bugs down the line.

However, it seems use of is_integer() on a float is setting people up for
exactly the same sorts of bugs.

Another example is that pow() functions sometimes swap to an exact
> algorithm if the power is an int. There's no particular reason why
> x**n and x**n.0 ought to be different, but they are:
>
> py> 123**10
> 792594609605189126649
>
> py> 123**10.0
> 7.925946096051892e+20


I think this is exactly like the __index__ use case. If the exponent is a
literal, use what you mean. If the exponent is a computed float, then you
really don’t want a different result depending on whether the computed
value is exactly an integer or one ULP off.

The user should check/convert to an integer with a method appropriate to
the problem at hand.

If it wasn’t too heavyweight, it might be nice to have some sort of flag on
floats indicating whether they really ARE an integer, rather than happen to
be:

-Created from an integer literal
- created from an integer object
- result of floor(), ceil() or round()

Any others?

But that would be too heavyweight, and not that useful.

In short, is_integer() is an attractive nuisance.

-CHB

PS: for the power example, the “right” solution is to have two operators:
integer power and float power, like we do for float vs floor division. No,
it’s not worth it in this case, but having it be value dependent would be
worse than type dependent.







>
> On the other hand, some might argue that by passing 10.0 as the power, I
> am specifically requesting a float implementation and result. I don't
> wish to argue in favour of either position, but just to point out that
> it is sometimes reasonable to want to know whether a float represents an
> exact integer value or not.
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Chris Barker]

>
>
> Does anybody know examples of the correct use of float.is_integer() in
> real programs? For now it looks just like a bug magnet. I suggest to
> deprecate it in 3.7 or 3.8 and remove in 3.9 or 3.10.


+1

It really doesn’t appear to be the right solution for any problem.

-CHB
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Chris Barker]

On Wed, Mar 21, 2018 at 11:43 PM, Tim Peters  wrote:

> Note:  this is a top-posted essay much more about floating-point
> philosophy than about details.  Details follow from the philosophy,
> and if philosophies don't match the desired details will never match
> either.
>

but of course :-)


> From that fundamental "take floats exactly at face value" view, what
> .is_integer() should do for floats is utterly obvious:


sure -- but I don't think anyone is arguing that -- the question is whether
the function should exist -- and that means not "how should it work?" or
"is it clearly and appropriately defined?" but rather, "is it the "right"
thing to do in most cases, when deployed by folks that haven't thought
deeply about floating point.

> Whether that's _useful_ to you depends on the application you're

> writing at the time.


exactly.

I think pretty much all the real world code that's been shown here for
using .is_integer() is really about type errors (issues). The function at
hand really wants integer inputs -- but wants to allow the user to be
sloppy and provide a float type that happens to be an int. Given Python's
duck-typing nature, maybe that's a good thing? I know I really discourage
dynamic type checking

Also, every example has been for small-ish integers -- exponents,
factorials, etc -- not order 1e300 -- or inf or NaN, etc.

Finally, the use-cases where the value that happens-to-be-an-int is
computed via floating point -- .is_integer() is probably the wrong check --
you probably want  isclose().

The other use-cases: and floor() and ceil() and round() all produce actual
integers -- so no need for that anymore.

All this points to: we don't need .is_integer

All the being said -- the standard for depreciation is much higher bar than
not-adding-it-in-the-first-place.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symmetry arguments for API expansion

[Chris Barker]

On Wed, Mar 21, 2018 at 12:38 PM, Steven D'Aprano 
wrote:

> In fact, Serhiy's suggestion is not correct when x is not a float:
>

This is a key point -- see some of Tim's posts -- like ot or not, you
probably need to know when you are working with a float and when you aren't
-- and what the implications of that are.

I think the key pint is this:

where does the logic of whether a given object represents an integer
belong? At first glance, it clearly belongs with the type -- floats know
how they are represented, as do fractions and decimals -- they can
determine it in an efficient and clearly defined way.

However, my argument is that while an integer-valued float is clearly
defined in the binary representation, what "is this an integer?" means is
actually use-case dependent, and thus the user should be deciding how to do
it (i.e. with isclose() or int(x) == x or 

> If the exponent is a computed float, then you

> > really don’t want a different result depending on whether the computed
> > value is exactly an integer or one ULP off.
>
> I don't think you actually mean to say that. I'm pretty sure that we
> *do* want different results if the exponent differs from an integer by
> one ULP.


yes -- poorly worded -- I mean you want the slightly different result, not
a different type and algorithm - i.e continuity if you had a range of
slightly less than an integer to slightly more than an integer.

> The user should check/convert to an integer with a method appropriate to

> > the problem at hand.
>
> Oh, you mean something like x.is_integer()? I agree!
>
> *wink*
>

That's the point -- is_integer may or may not be appropriate, and whether
it is is a function of use-case, not type.

> If it wasn’t too heavyweight, it might be nice to have some sort of flag
> on
> > floats indicating whether they really ARE an integer, rather than happen
> to
> > be:
> >
> > -Created from an integer literal
> > - created from an integer object
> > - result of floor(), ceil() or round()
>
> I don't understand this.
>

poorly worked again -- I shoud not write these on a phone


> You seem to be saying that none of the following are "really" integer
> valued:
>
> float(10)
> floor(10.1)
> ceil(10.1)
> round(10.1)
>

I meant those are the ones that ARE really integer valued.

turns out that py3 returns an actual int for all of those other than
float() (of course) anyway, so that's pretty much been done -- and no need
for is_integer()

well, it's been fun, but looks like it's sticking around.


-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symmetry arguments for API expansion

[Chris Barker]

On Wed, Mar 21, 2018 at 4:12 PM, Guido van Rossum  wrote:

> Thank you! As you may or may not have noticed in a different thread, we're
> going through a small existential crisis regarding the usefulness of
> is_integer() -- Serhiy believes it is not useful (and even an attractive
> nuisance) and should be deprecated. OTOH the existence of
> dec_mpd_isinteger() seems to validate to me that it actually exposes useful
> functionality (and every Python feature can be abused, so that alone should
> not be a strong argument for deprecation).
>

if not deprecated, then do we add it to all the other numeric types? Which
was the original suggestion, yes?

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symmetry arguments for API expansion

[Chris Barker]

I know this is all done, but for completeness’ sake:

I just noticed math.trunc() and __trunc__().

So wouldn’t the “correct” way to check for an integral value be something
like:

obj.__trunc__() == obj

I don’t think this has any bearing on adding is_integer() methods to
numeric objects, but might if we wanted to add a generic is_integer()
function somewhere.

In any case, I don’t recall it being mentioned in the conversation, so
thought I’d complete the record.

-CHB





On Wed, Mar 21, 2018 at 8:31 PM Guido van Rossum  wrote:

> On Wed, Mar 21, 2018 at 6:48 PM, Chris Barker 
> wrote:
>
>> On Wed, Mar 21, 2018 at 4:12 PM, Guido van Rossum 
>> wrote:
>>
>>> Thank you! As you may or may not have noticed in a different thread,
>>> we're going through a small existential crisis regarding the usefulness of
>>> is_integer() -- Serhiy believes it is not useful (and even an attractive
>>> nuisance) and should be deprecated. OTOH the existence of
>>> dec_mpd_isinteger() seems to validate to me that it actually exposes useful
>>> functionality (and every Python feature can be abused, so that alone should
>>> not
>>>
>> )
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Chris Barker]

On Thu, Apr 19, 2018 at 5:45 AM, Nick Coghlan  wrote:

> * for the while loop use case, I think my discussion with Tim about
> tuple unpacking shows that the loop-and-a-half construct won't be
> going anywhere, so users would still end up needing to learn both
> forms (even for new code)
>

well, yes, but I've always found while awkward in the common case -- and
wanted a better way (maybe an until?).

using:

while True:
   ...

for a loop with a clear testable termination criteria always felt clunky.

Anyone recall the "canonical" way to loop through the lines of a file
before we had the file iterator?

while True:
line = fp.readline()
if not line:
break
do_stuff_with(line)


This is in fact, the only compeling use case for this to me -- maybe we
could make a way to do an assign and test in a while loop enhancement just
for that?

And no, I have no idea what that would look like -- I'm mostly joking.

So -1 on this -- though I agree -- great PEP Chris!

And -1, not -0 because I think this really would add one more complication
to a language that used to be so simple, and still is (and should be) used
a first programming language. I teach a lot of newbies, and I'm realy not
l;ooking forward to one more way to do assignement. It starts simpel enough:

x = something

binds the name, "x" the the object, 'something' is bound to (or a literal)

Then we get into:
 - multiple assignment
 - tuple unpacking (oops, sequence unpacking),
 - augmented assignment
 - oh, that works different depending on whether the object is mutable

We don't need any more.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow

[Chris Barker]

On Tue, Apr 24, 2018 at 2:21 AM, Victor Stinner  wrote:

> Even if the C language allows assignments in if, I avoid them, because
> I regularly have to debug my own code in gdb ;-)
>

I personally haven't written a lot of C, so have no personal experience,
but if this is at all a common approach among experienced C developers, it
tells us a lot.

We shouldn't add a feature that people would make a point of avoiding!

OT note:


> Now the question is which Python are allowed for babies. I recall that
> a colleague was surprised and confused by context managers. Does it
> mean that try/finally should be preferred?


well, no, because try ... finally is even more confusing -- at least that's
the impression I get from spending 20 minutes on it with newbies in my
class last night :-)



>  Or metaclasses?


metaclasses are well known to be advanced juju -- they should not (and
probably don't) show up in everyday code. Even if they are used in everyday
code, the use is usually hidden -- i.e. when a user subclasses from an ORM
model class, they are using metaclasses, but they don't have to know that.

That is -- they fall into the category of stuff that should only be used by
library/framework developers -- and, in fact, the whole point is to make
everyday code easier.

In fact, the number of developers that need to write/debug metaclasses,
context managers, decorators, is far fewer than the number of folks that
USE those things. So the standards are very different.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (name := expression) doesn't fit the narrative of PEP 20

[Chris Barker]

On Thu, Apr 26, 2018 at 1:33 PM, Tim Peters  wrote:

>  And that *is* a thing that you will have to explain to newbies when they
> encounter
> > it for the first time.
>
> Sure.  That doesn't frighten me, though.  It's easy to explain what it
> does - although it may be hard to explain when it's _desirable_ to use
> it.
>

I'm with Raymond here -- though I'm not sure "newbies" is quite right --
I've found that newbies fall into two camps: folks to whom programming
comes naturally, and those that it doesn't (OK, it's a distribution, but a
bimodal one). And folks that are struggling with programming can struggle
even with simple assignment (name binding), particularly when you add even
function local scope. So having one more way to do assignment WILL make it
harder to teach, not because it's that hard, but because it's one more
thing to learn.

But the fact is that as Python has evolved (particularly with the jump to
py3) it has become less and less of a "scripting" language, and more of a
"systems" language. And also harder to learn. Anyone remember CP4E? Python
is not as good choice as a "newbie" language as it once was.

Adding := will move it a little bit more along the complexity path -- not
much, and that's where Python has gone anyway, so as Tim said, no one's
going to suffer either way this decision goes.

Hmm -- I wonder if a "pythonscript" will get forked off one day..

To judge from Stackoverflow volume, the single most misunderstood of
> all Python operators - by far - is "is" -


You now, I think instructors like me are partly responsible. "is" is rarely
useful outside of comparing to singletons. Yet I use it early in
instruction to do checks on name binding and show things with mutablilty,
etc which has the unfortunate side effect of making it seem like a more
common operator than it is.

I've even had students write code like:

if x is 3:

and thanks to interning, it appears to work!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 485 isclose() implementation review requested

[Chris Barker]

Folks,

After a huge delay, I finally found the time to implement the PEP 485
isclose() function, in C. I tihnk it's time for some review. I appologise
for the fact that I have little experience with C, and haven't used the raw
C API for years, but it's a pretty simple function, and there's lots of
code to copy, so I think it's in OK shape. I hav not yet integrated it with
the cPyton source code -- it belongs in mathmodule.c, but I found it easier
to put it in a separate module while figuring it out.

You can find the code in the same gitHub repo as the draft PEP and python
prototyping code:

https://github.com/PythonCHB/close_pep

the code is in:
  is_close_module.c

There is a test module in:
  test_isclose_c.py

and it can be built with:

python3 setup.py build_ext --inplace

Let me know if I should restructure it or put it anywhere else before it
gets reviewed but in the meantime, i welcome any feedback.

Thanks,
  -Chris

A few questions I have off the bat:

C-API (and plain C) questions:
=

* Is there a better way to create a False or True than::

PyBool_FromLong(0) and PyBool_FromLong(1)

* Style question: should I put brackets in an if clause that has only one
line?::

if (some_check) {
just_this_one_expression
}

* I can't find docs for PyDoc_STRVAR: but it looks like it should use it --
how?

* I'm getting a warning in my PyMethodDef clause::

static PyMethodDef IsCloseMethods[] = {
{"isclose", isclose_c, METH_VARARGS | METH_KEYWORDS,
 "determine if two floating point numbers are close"},
{NULL, NULL, 0, NULL}/* Sentinel */
};

is_close_module.c:61:17: warning: incompatible pointer types initializing
  'PyCFunction' (aka 'PyObject *(*)(PyObject *, PyObject *)') with an
  expression of type 'PyObject *(PyObject *, PyObject *, PyObject *)'
  [-Wincompatible-pointer-types]
{"isclose", isclose_c, METH_VARARGS | METH_KEYWORDS,

but it seems to be working -- and I've copied, as well as I can, the
examples.

Functionality Questions:


* What do do with other numeric types?
  - Integers cast fine...
  - Decimal  and Fraction cast fine, too -- but precision is presumably
lost.
  - Complex ? -- add it to cmath?


* It's raising an Exception for negative tolerances: which don't make sense,
  but don't really cause harm either (fabs() is used anyway). I can't say I
recall why I did that
  for the python prototype, but I reproduced in the C version. Should I?

* What about zero tolerance? should equal values still be considered close?
They are now, and tests reflect that.












-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 485 isclose() implementation review requested

[Chris Barker]

Thanks Cristian, that clears up a couple things -- got it compiling without
warning.

But I also discovered that I must have not pushed the latest copy yesterday.

It's on a machine at home -- I'll push it tonight. But the copy on gitHub
now is mostly good -- I think the only changes are handling the docsstrings
better and some more/better tests.

-Chris





On Sun, May 17, 2015 at 4:16 PM, Christian Heimes 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> On 2015-05-18 01:02, Chris Barker wrote:
> > * Is there a better way to create a False or True than::
> >
> > PyBool_FromLong(0) and PyBool_FromLong(1)
>
> You can use the macros Py_RETURN_TRUE and Py_RETURN_FALSE instead of
> return PyBool_FromLong(0).
>
>
> > * Style question: should I put brackets in an if clause that has
> > only one line?::
> >
> > if (some_check) { just_this_one_expression }
>
> I prefer the extra brackets because they make the code more explicit.
> It's really a matter of taste.
>
> > * I can't find docs for PyDoc_STRVAR: but it looks like it should
> > use it -- how?
>
> PyDoc_STRVAR(functionname_doc,
> "isclose(foo) -> bool\n\
> \n\
> long doc string.");
>
> > * I'm getting a warning in my PyMethodDef clause::
> >
> > static PyMethodDef IsCloseMethods[] = { {"isclose", isclose_c,
> > METH_VARARGS | METH_KEYWORDS, "determine if two floating point
> > numbers are close"}, {NULL, NULL, 0, NULL}/* Sentinel */
> > };
>
> You have to type cast the function pointer to a PyCFunction here:
>
>   (PyCFunction)isclose_c
>
> The type cast is required for KEYWORD functions and NOARGS functions.
>
> Christian
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Gcode path

[Chris Barker]

Lisa,

As noted, not the right list.

But seeing this kind of stuff done in High Schools is GREAT!

So one suggestion:

If this is Windows, there are two versions of python for Windows: 32bit and
64bit -- if an installer for a third-party package is looking for one of
those, and the other is installed, you will get the confusing error
something like "python is not installed", when it really means "the correct
version of python is not installed".

So make sure you see which version Replicatorg is expecting and make sure
that's the one you installed.

Good luck,

-Chris



On Mon, May 18, 2015 at 1:18 PM, Guido van Rossum  wrote:

> Hi Lisa,
>
> It's unlikely that anyone on this list can help you with this. It's a
> question you should ask of the ReplicatorG people, not the Python people...
>
> --Guido
>
> On Mon, May 18, 2015 at 12:02 PM, Lisa Colvard <
> lcolv...@horrycountyschools.net> wrote:
>
>>  I am trying to get Replicatorg to use python to gcode but it keeps
>> giving the error “generate Gcode  requires that a Python interpreter be
>> installed.  Would you like to visit the Python download page now?
>>
>>
>>
>> I tell it no because I have it installed.  I have even gone in and
>> selected the path for the gcode.  It still gives me the same error.  How
>> can I fix this?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Lisa Colvard, Ed.D.
>>
>> Conway High School
>>
>> Media/Technology Specialist
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: Python 3.5 beta 1 will be tagged tomorrow

[Chris Barker]

Is it too late to get the isclose() code (PEP 485) into 3.5?

I posted the code here, and got a tiny bit of review, but have not yet
merged it into the source tree -- and don't know the process for getting it
committed to the official source.

So -- too late, or should I try to get that merge done soon -- if so, how?

-Chris



On Fri, May 22, 2015 at 12:53 PM, Larry Hastings  wrote:

>
>
> Howdy howdy.  It's-a me, Larry, your friendly neighborhood Python 3.5
> Release Manager.
>
> Somewhere around 2 or 3pm tomorrow I expect to tag Python 3.5 beta 1.
> We'll actually release beta 1 on Sunday, once the binary installers are all
> built.
>
> Beta 1 is also feature-freeze, meaning no new features may be added to 3.5
> without my permission.  Since it seems useful to have a specific cutoff
> time, please stop adding features at ** 8pm Saturday UTC **.  (That's 1pm
> Pacific Daylight Time.  It's also almost exactly 24 hours from... now.)
>
> I remind you that this time we're trying something new: we're going to
> create the 3.5 branch when we release beta 1, allowing feature development
> (for 3.6) to continue in trunk.  At the point that I check in and push beta
> 1, I'll also merge all checkins from trunk back into the 3.5 branch.  After
> that it'll be responsibility of the person checking in to check their bug
> fixes in to the appropriate place.  So please keep in mind: once the 3.5
> branch becomes generally available on Sunday, the usual rules for a release
> branch will apply: bug fixes for 3.5 should be checked in to the 3.5 branch
> and get merged forward into trunk.
>
> If you have new features you want to ship with Python 3.5, please check
> them in as soon as possible!
>
>
> Thank you for helping to make Python better,
>
>
> */arry*
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: Python 3.5 beta 1 will be tagged tomorrow

[Chris Barker]

On Fri, May 22, 2015 at 2:33 PM, Larry Hastings  wrote:

>  On 05/22/2015 02:29 PM, Chris Barker wrote:
>
> Is it too late to get the isclose() code (PEP 485) into 3.5?
>
> ...

>   Hopefully you can find a core dev familiar enough with the issues
> involved that they can (quickly!) guide it through the process of getting
> it checked in.
>
> Ping!  Anyone willing to sponsor this?

Sorry I waited 'till this late in the game.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com