Re: [Python-Dev] Add __reversed__ methods for dict

[Michael Selik]

Am I correct in saying that the consensus is +1 for inclusion in v3.8?

The last point in the thread was INADA Naoki researching various
implementations and deciding that it's OK to include this feature in 3.8.
As I understand it, Guido was in agreement with INADA's advice to wait for
MicroPython's implementation of v3.7. Since INADA has changed minds, I'm
guessing it's all in favor?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou  wrote:

> Thank you.  Personally, I'd like to see feedback from
> educators/teachers after they take the time to read the PEP and take
> some time to think about its consequences.
>

I've started testing the proposed syntax when I teach. I don't have a large
sample yet, but most students either dislike it or don't appreciate the
benefits. They state a clear preference for shorter, simpler lines at the
consequence of more lines of code. This may partially be caused by the
smaller screen real estate on a projector or large TV than a desktop
monitor.

My intuition is that one strength of Python for beginners is the relative
lack of punctuation and operators compared with most other languages. This
proposal encourages denser lines with more punctuation. Because of the
order of operations, many uses of ``:=`` will also require parentheses.
Even relatively simple uses, like ``if (match := pattern.search(data)) is
not None:`` require doubled parentheses on one side or the other. Beginners
are especially prone to typographical errors with mismatched parentheses
and missing colons and get easily frustrated by the associated syntax
errors.


Given the following options:

A.

if (row := cursor.fetchone()) is None:
raise NotFound
return row


B.

row = cursor.fetchone()
if row is None:
raise NotFound
return row


C.

if (row := cursor.fetchone()) is not None:
return row
raise NotFound


D.

row = cursor.fetchone()
if row is not None:
return row
raise NotFound


The majority of students preferred option B. I also tested some regex match
examples. Results were similar.




> My main concern is we're introducing a second different way of doing
> something which is really fundamental.
>

The few students who like the proposal ask why it requires creating a new
operator instead of repurposing the ``=`` operator.

I'll reserve my personal opinions for a different thread.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Fri, Jun 22, 2018 at 10:02 AM Michael Selik  wrote:

> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou 
> wrote:
>
>> Thank you.  Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
I forgot to add that I don't anticipate changing my lesson plans if this
proposal is accepted. There's already not enough time to teach everything
I'd like. Including a new assignment operator would distract from the
learning objectives.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Fri, Jun 22, 2018 at 10:19 AM Chris Angelico  wrote:

> On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik  wrote:
> > On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou 
> wrote:
> >>
> >> Thank you.  Personally, I'd like to see feedback from
> >> educators/teachers after they take the time to read the PEP and take
> >> some time to think about its consequences.
> >
> >
> > I've started testing the proposed syntax when I teach. I don't have a
> large
> > sample yet, but most students either dislike it or don't appreciate the
> > benefits. They state a clear preference for shorter, simpler lines at the
> > consequence of more lines of code.
>
> This is partly because students, lacking the experience to instantly
> recognize larger constructs, prefer a more concrete approach to
> coding. "Good code" is code where the concrete behaviour is more
> easily understood. As a programmer gains experience, s/he learns to
> grok more complex expressions, and is then better able to make use of
> the more expressive constructs such as list comprehensions.
>

I don't think that's the only dynamic going on here. List comprehensions
are more expressive, but also more declarative and in Python they have nice
parallels with SQL and speech patterns in natural language. The concept of
a comprehension is separate from its particular expression in Python. For
example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Students who are completely new to programming can see the similarity of
list comprehensions to spoken language. They also appreciate the revision
of certain 3-line and 4-line for-loops to comprehensions. I didn't get the
same sense of "Oh! That looks better!" from my students when revising code
with an assignment expression.

Despite my best efforts to cheerlead, some students initially dislike list
comprehensions. However, they come around to the idea that there's a
tradeoff between line density and code block density. Comprehensions have a
3-to-1 or 4-to-1 ratio of code line shrinkage. They're also often used in
sequence, like piping data through a series of transforms. Even if students
dislike a single comprehension, they agree that turning 15 lines into 5
lines improves the readability.

In contrast, an assignment expression only has a 2-to-1 code line
compression ratio. It might save a level of indentation, but I think there
are usually alternatives. Also, the assignment expression is less likely to
be used several times in the same block.

A good pitch for an assignment expression is refactoring a cascade of
regular expressions:


for line in f:
mo = foo_re.search(line)
if mo is not None:
foo(mo.groups())
continue

mo = bar_re.search(line)
if mo is not None:
bar(mo.groups())
continue

mo = baz_re.search(line)
if mo is not None:
baz(mo.groups())
continue


Here the assignment operator makes a clear improvement:

for line in f:
if (mo := foo_re.search(line)) is not None:
foo(mo.groups())
elif (mo := bar_re.search(line)) is not None:
bar(mo.groups())
elif (mo := baz_re.search(line)) is not None:
baz(mo.groups())


However, I think this example is cheating a bit. While I've written similar
code many times, it's almost never just a function call in each if-block.
It's nearly always a handful of lines of logic which I wouldn't want to cut
out into a separate function. The refactor is misleading, because I'd
nearly always make a visual separation with a newline and the code would
still look similar to the initial example.


[0]
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Array_comprehensions
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

This thread started with a request for educator feedback, which I took to
mean observations of student reactions. I've only had the chance to test
the proposal on ~20 students so far, but I'd like the chance to gather more
data for your consideration before the PEP is accepted or rejected.



On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano 
wrote:

> Remember, the driving use-case which started this (ever-so-long)
> discussion was the ability to push data into a comprehension and then
> update it on each iteration, something like this:
>
> x = initial_value()
> results = [x := transform(x, i) for i in sequence]
>

If that is the driving use-case, then the proposal should be rejected. The
``itertools.accumulate`` function has been available for a little while now
and it handles this exact case. The accumulate function may even be more
readable, as it explains the purpose explicitly, not merely the algorithm.
And heck, it's a one-liner.

results = accumulate(sequence, transform)


The benefits for ``any`` and ``all`` seem useful. Itertools has
"first_seen" in the recipes section. While it feels intuitively useful, I
can't recall ever writing something similar myself. For some reason, I
(almost?) always want to find all (counter-)examples and aggregate them in
some way -- min or max, perhaps -- rather than just get the first.

Even so, if it turns out those uses are quite prevalent, wouldn't a new
itertool be better than a new operator? It's good to solve the general
problem, but so far the in-comprehension usage seems to have only a handful
of cases.



On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev <
python-dev@python.org> wrote:

> again, not a huge deal, just a little bit more complexity
>

I worry that Python may experience something of a "death by a thousand
cuts" along the lines of the "Remember the Vasa" warning. Python's greatest
strength is its appeal to beginners. Little bits of added complexity have a
non-linear effect. One day, we may wake up and Python won't be recommended
as a beginner's language.




On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano  wrote:

> On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
>
> Of course they do -- they're less fluent at reading code. They don't
> have the experience to judge good code from bad.
>

On the other hand, an "expert" may be so steeped in a particular subculture
that he no longer can distinguish esoteric from intuitive. Don't be so fast
to reject the wisdom of the inexperienced.



> The question we should be asking is, do we only add features to Python
> if they are easy for beginners? It's not that I especially want to add
> features which *aren't* easy for beginners, but Python isn't Scratch and
> "easy for beginners" should only be a peripheral concern.
>

On the contrary, I believe that "easy for beginners" should be a major
concern.  Ease of use has been and is a, or even the main reason for
Python's success. When some other language becomes a better teaching
language, it will eventually take over in business and science as well.
Right now, Python is Scratch for adults. That's a great thing. Given the
growth of the field, there are far more beginner programmers working today
than there ever have been experts.


Mozilla's array comprehensions are almost identical to Python's, aside
> from a couple of trivial differences:
>

I can't prove it, but I think the phrase ordering difference is not trivial.


> Students who are completely new to programming can see the similarity of
> > [Python] list comprehensions to spoken language.
>
> I've been using comprehensions for something like a decade, and I can't
>

Python: any(line.startswith('#') for line in file)
English: Any line starts with "#" in the file?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Sun, Jun 24, 2018 at 4:57 PM Guido van Rossum  wrote:

> On Sun, Jun 24, 2018 at 2:41 PM Michael Selik  wrote:
>
>> This thread started with a request for educator feedback, which I took to
>> mean observations of student reactions. I've only had the chance to test
>> the proposal on ~20 students so far, but I'd like the chance to gather more
>> data for your consideration before the PEP is accepted or rejected.
>>
>
> Sure. Since the target for the PEP is Python 3.8 I am in no particular
> hurry. It would be important to know how you present it to your students.
>

Absolutely. Since this has come up, I'll make an effort to be more
systematic in data collection.




> On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano 
>> wrote:
>>
>>> Remember, the driving use-case which started this (ever-so-long)
>>> discussion was the ability to push data into a comprehension and then
>>> update it on each iteration, something like this:
>>>
>>> x = initial_value()
>>> results = [x := transform(x, i) for i in sequence]
>>>
>>
>> If that is the driving use-case, then the proposal should be rejected.
>> The ``itertools.accumulate`` function has been available for a little while
>> now and it handles this exact case. The accumulate function may even be
>> more readable, as it explains the purpose explicitly, not merely the
>> algorithm. And heck, it's a one-liner.
>>
>> results = accumulate(sequence, transform)
>>
>
> I think that's a misunderstanding. At the very least the typical use case
> is *not* using an existing transform function which is readily passed to
> accumulate -- instead, it's typically written as a simple expression (e.g.
> `total := total + v` in the PEP) which would require a lambda.
>

Plus, I don't know what kind of students you are teaching, but for me,
> whenever the solution requires a higher-order function (like accumulate),
> this implies a significant speed bump -- both when writing and when reading
> code. (Honestly, whenever I read code that uses itertools, I end up making
> a trip to StackOverflow :-).
>

Mostly mid-career professionals, of highly varying backgrounds. The
higher-order functions do require some cushioning getting into, but I have
some tricks I've learned over the years to make it go over pretty well.


On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano  wrote:
>>
> On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
>>>
>>> Of course they do -- they're less fluent at reading code. They don't
>>> have the experience to judge good code from bad.
>>>
>>
>> On the other hand, an "expert" may be so steeped in a particular
>> subculture that [they] no longer can distinguish esoteric from intuitive.
>> Don't be so fast to reject the wisdom of the inexperienced.
>>
>
> Nor should we cater to them excessively though. While the user is indeed
> king, it's also well known that most users when they are asking for a
> feature don't know what they want (same for kings, actually, that's why
> they have advisors :-).
>
>
>> The question we should be asking is, do we only add features to Python
>>> if they are easy for beginners? It's not that I especially want to add
>>> features which *aren't* easy for beginners, but Python isn't Scratch and
>>> "easy for beginners" should only be a peripheral concern.
>>>
>>
>> On the contrary, I believe that "easy for beginners" should be a major
>> concern.  Ease of use has been and is a, or even the main reason for
>> Python's success. When some other language becomes a better teaching
>> language, it will eventually take over in business and science as well.
>> Right now, Python is Scratch for adults. That's a great thing. Given the
>> growth of the field, there are far more beginner programmers working today
>> than there ever have been experts.
>>
>
> I'm sorry, but this offends me, and I don't believe it's true at all.
> Python is *not* a beginners language, and you are mixing ease of use and
> ease of learning. Python turns beginners into experts at an unprecedented
> rate, and that's the big difference with Scratch.
>

By saying "Scratch for adults" I meant that Python is a language that can
be adopted by beginners and rapidly make them professionals, not that it's
exclusively a beginner's language.

Also, Scratch and similar languages, like NetLogo, have some interesting
features that allow beginners to write some sophisticated parallelism. I
don't mean "beginner's language" in that it's overly simplistic, but that
it enables what would be complex in other languages.

I realize that my phrasing was likely to be misunderstood without knowing
the context that I teach working professionals who are asked to be
immediately productive at high-value tasks.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rationale behind lazy map/filter

[Michael Selik]

> I'm not suggesting restarting at the top (I've elsewhere suggested that
> many such methods would be better as an *iterable* that can be restarted
> at the top by calling iter() multiple times, but that's not the same
> thing). I'm suggesting raising an exception other than StopIteration, so
> that this situation can be detected. If you are writing code that tries
> to resume iterating after the iterator has been exhausted, I have to
> ask: why?

The most obvious case for me would be tailing a file. Loop over the lines
in the file, sleep, then do it again. There are many tasks analogous to
that scenario -- anything querying a shared resource.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.Counter __add__ implementation quirk

[Michael Selik]

Raymond,
I think you made a typographical error in your Counter.update example.

>>> from collections import Counter
>>> c = Counter(a=4, b=2, c=0, d=-2)
>>> d = Counter(a=1, b=-5, c=-2, d=6)
>>> c.update(d)
>>> c
Counter({'a': 5, 'd': 4, 'c': -2, 'b': -3})

Pair programming ;-)


On Tue, Nov 24, 2015 at 1:02 AM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

>
> > On Nov 23, 2015, at 10:43 AM, Vlastimil Brom 
> wrote:
> >
> >> Is there any particular reason counters drop negative values when you
> add
> >> them together?  I definitely expected them to act like ints do when you
> add
> >> negatives, and had to subclass it to get what I think is the obvious
> >> behavior.
> >> ___
> >> Python-Dev mailing list
> > ...
> > Hi,
> > this is probably more appropriate for the general python list rathere
> > then this developers' maillist, however, as I asked a similar question
> > some time ago, I got some detailed explanations for the the current
> > design decissions from the original developer; cf.:
> > https://mail.python.org/pipermail/python-list/2010-March/570618.html
> >
> > (I didn't check possible changes in Counter since that version (3.1 at
> > that time).)
>
> In Python3.2, Counter grew a subtract() method:
>
> >>> c = Counter(a=4, b=2, c=0, d=-2)
> >>> d = Counter(a=1, b=2, c=3, d=4)
> >>> c.subtract(d)
> >>> c
> Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})
>
> The update() method has been around since the beginning:
>
> >>> from collections import Counter
> >>> c = Counter(a=4, b=2, c=0, d=-2)
> >>> d = Counter(a=1, b=-5, c=-2, d=6)
> >>> c.update(d)
> >>> d
> Counter({'d': 6, 'a': 1, 'c': -2, 'b': -5})
>
>
> So, you have two ways of doing counter math:
>
> 1. Normal integer arithmetic using update() and subtract() does straight
> addition and subtraction, either starting with or ending-up with negative
> values.
>
> 2. Saturating arithmetic using the operators: + - & | excludes
> non-positive results.  This supports bag-like behavior (c.f. smalltalk) and
> multiset operations (https://en.wikipedia.org/wiki/Multiset).
>
>
> Raymond
>
>
>
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Sat, Jun 30, 2018 at 9:43 AM Tim Peters  wrote:

> The attractions are instead in the areas of reducing redundancy, improving
> clarity, allowing to remove semantically pointless indentation levels in
> some cases, indeed trading away some horizontal whitespace in otherwise
> nearly empty lines for freeing up a bit of vertical screen space, and in
> the case of comprehensions/genexps adding straightforward ways to
> accomplish some conceptually trivial things that at best require trickery
> now (like emulating a cell object by hand).
>

The examples you provided (some were new in this thread, I think) are
compelling. While my initial reaction to the proposal was mild horror, I'm
not troubled by the scoping questions.

Issues still bothering me:
1. Initial reactions from students was confusion over := vs =
2. This seems inconsistent with the push for type hints

To be fair, I felt a similar gut reaction to f-strings, and now I can't
live without them. Have I become a cranky old man, resistant to change?
Your examples have put me into the "on the fence, slightly worried"
category instead of "clearly a bad idea".

On scoping, beginners seem more confused by UnboundLocalError than by
variables bleeding between what they perceive as separate scopes. The
concept of a scope can be tricky to communicate. Heck, I still make the
mistake of looking up class attributes in instance methods as if they were
globals. Same-scope is natural. Natural language is happy with ambiguity.
Separate-scope is something programmers dreamed up. Only experienced C,
Java, etc. programmers get surprised when they make assumptions about what
syntax in Python creates separate scopes, and I'm not so worried about
those folks. I remind them that the oldest versions of C didn't have block
scopes (1975?) and they quiet down.

The PEP lists many exclusions of where the new := operator is invalid [0].
I unfortunately didn't have a chance to read the initial discussion over
the operator. I'm sure it was thorough :-). What I can observe is that each
syntactical exclusion was caused by a different confusion, probably teased
out by that discussion. Many exclusions means many confusions.

My intuition is that the awkwardness stems from avoiding the replacement of
= with :=. Languages that use := seem to avoid the Yoda-style comparison
recommendation that is common to languages that use = for assignment
expressions. I understand the reluctance for such a major change to the
appearance of Python code, but it would avoid the laundry list of
exclusions. There's some value in parsimony.

Anyway, we've got some time for testing the idea on live subjects.

Have a good weekend, everyone.
-- Michael

PS. Pepe just tied it up for Portugal vs Uruguay. Woo! ... and now Cavani
scored again :-(

[0] https://www.python.org/dev/peps/pep-0572/#exceptional-cases
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Sun, Jul 1, 2018 at 12:39 AM Tim Peters  wrote:

> So, ya, when someone claims [assignment expressions will] make Python
> significantly harder to teach, I'm skeptical of that claim.
>

I don't believe anyone is making that claim. My worry is that assignment
expressions will add about 15 to 20 minutes to my class and a slight
discomfort.

As Mark and Chris said (quoting Mark below), this is just one straw in the
struggle against piling too many things on the haystack. Unlike some
changes to the language, this change of such general use that it won't be
an optional topic. Once widely used, it ain't optional.


On Sun, Jul 1, 2018 at 2:19 AM Mark Dickinson  wrote:

> There's a constant struggle to keep the Python portion of the course large
> enough to be coherent and useful, but small enough to allow time for the
> other topics.
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Sun, Jul 1, 2018 at 5:28 PM Steven D'Aprano  wrote:

> On Sun, Jul 01, 2018 at 08:35:08AM -0700, Michael Selik wrote:
> > On Sun, Jul 1, 2018 at 12:39 AM Tim Peters  wrote:
> >
> > > So, ya, when someone claims [assignment expressions will] make Python
> > > significantly harder to teach, I'm skeptical of that claim.
> > >
> >
> > I don't believe anyone is making that claim. My worry is that assignment
> > expressions will add about 15 to 20 minutes to my class and a slight
> > discomfort.
>
> How do people who teach other languages deal with this?
>

Python may be in a unique situation in the history of programming. It
wouldn't surprise me if more people learned Python last year than any other
programming language.



> Assignment expressions are hardly a new-fangled innovation of Python's.
> They're used in Java, Javascript, Ruby, Julia, R, PHP and of course
> pretty much the entire C family (C, C++, C# at least). What do
> teachers of those languages do?
>

Assignment expressions are not the issue. The real question is: How do
open-source projects balance the addition of new features against the
growth of complexity? It's the same as that "Remember the Vasa" thread.


[...] R [has] *four* different ways of doing assignment.
>

I think that's a good explanation of why I teach Python and not R. The
first time someone asked me to teach a data science course, Python wasn't
the clear winner. In fact, R may have been more popular among
statisticians. I picked Python for the same reason it's more popular in the
industry -- it's the easiest* to use.

* Easiest that gets the job done well.


> As Mark and Chris said (quoting Mark below), this is just one straw in the
> > struggle against piling too many things on the haystack. Unlike some
> > changes to the language, this change of such general use that it won't be
> > an optional topic. Once widely used, it ain't optional.
>
> Without knowing the details of your course, and who they are aimed at,
> we cannot possibly judge this comment.


I disagree. I think the sentiment holds for a great variety of courses and
audiences.



> Decorators are widely used, but surely you don't teach them in a one day
> introductory class aimed at beginners?
>

Most of the time, no. Once, yes, because that's what the team needed. I was
pretty proud of myself for handling that one. Because I had to teach
decorators early, many other important topics were excluded.


Here is the syllabus for a ten week course:
> https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus
>
> Note that decorators and even regular expressions don't get touched
> until week ten. If you can't fit assignment expressions in a ten week
> course, you're doing something wrong. If you can't fit them in a two
> hour beginners course, there is so much more that you aren't covering
> that nobody will notice the lack.
>

It's not about any one particular topic, but the trade-offs between topics.
A 10-week lecture course might be 30 hours of lecture, comparable to a
4-day "bootcamp" style course. I assure you that 4 days doesn't feel long
enough when those last few hours are winding down. There's always more to
say.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Sun, Jul 1, 2018 at 11:36 PM Tim Peters  wrote:

> [Michael Selik]
> > My worry is that assignment expressions will add about 15 to 20
> > minutes to my class and a slight discomfort.
>
> So not intractable - which is my high-order bit ;-)
>
> For those who want more bits of precision (perhaps Guido), while
> quantification is good, it needs context to provide insight.  Like, out of
> how many class hours total?
>

Generally between 20 and 40 hours.


Is 15-20 minutes a little, a lot, par for the course ... compared to other
> topics?
>

I guessed 15-20 minutes, because I'm mentally comparing it to things like
ternary expressions. Odds and ends that make the code better, but not a
major concept that deserves hours.


Will it require you to drop other topics?
>

Yes. It might not seem like much, but every minute counts. I'd probably try
to ignore := unless some pesky student brings it up. It's like someone
saying, "Hey, I heard that Python can't do threads?!" I always say, "Good
question," but internally I'm thinking, "there goes a half hour. What can I
cut today?"



> Would you _save_ twice as much class time if we got rid of "is"? ;-)
>

Ha. You joke, but ``is`` takes about 5 minutes. About 5 or 10 minutes more
if some clever student notices that ``1 is 1`` and I need to explain
Singletons and interpreter optimizations versus language spec.


If it's accepted, do read the PEP
>

I've read it a few times now. I hope I didn't sound like I haven't read it.
That'd be embarrassing.


Meta: About the Vasa, I'm not concerned.
>

Matt Arcidy brought up an interesting point, which I'll quote here: "... I
don't see any importance to the position of educators right now, especially
since these educators in the thread are complaining about an increase in
their personal work, for which it appears they were compensated."

>From my brief observations, it seems that the nattering nabobs of
negativism, such as myself, are mostly educators. I recently started to
wonder if I'd care so much about the language if I didn't teach. I suspect
that if I didn't worry about teaching new features, Python 4 could be
announced tomorrow and I wouldn't really mind.

I suppose it is selfish. But I hope that you [Tim], Guido, and the so many
others who have poured energy into this project will appreciate that it's
not the current users, but the next billion (?!) Pythonistas that will
really keep the language going. Maintaining popularity among educators is a
big part of that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Michael Selik]

On Sun, Jul 1, 2018 at 8:21 PM Matt Arcidy  wrote:

> [...] Can anyone adequately explain why this specific modality of
> learning,  a student-in-a-seat based educator, must outweigh all other
> modalities [...]?


1. It doesn't.
2. It's a proxy for the other modes.

I hope this was an adequate explanation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fuzzing the Python standard library

[Michael Selik]

On Tue, Jul 17, 2018 at 4:57 PM Jussi Judin  wrote:

> Quick answer: undocumented billion laughs/exponential entity expansion
> type of an attack that is accessible through web through any library that
> uses fractions module to parse user input (that are actually available on
> Github).
>

Are you suggesting a warning in the fractions documentation to mention that
large numbers require large amounts of memory?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accessing mailing list archives

[Michael Selik]

Would it be possible to normalize by the number of mailing list members and
also by "active" members? The latter would be tricky to define.

On Mon, Jul 30, 2018 at 3:29 PM Victor Stinner  wrote:

> Hi Bob,
>
> I wrote a basic script to compute the number of emails per PEP. It
> requires to download gzipped mbox files from the web page of archives per
> month, then ungzip them:
>
> https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py
>
> Results:
> https://mail.python.org/pipermail/python-committers/2018-April/005310.html
>
> Victor
>
> Le lundi 30 juillet 2018, Bob Purvy  a écrit :
> > hi all,
> > I've been trying to figure out how to access the archives
> programmatically. I'm sure this is easy once you know, but googling various
> things hasn't worked.  What I want to do is graph the number of messages
> about PEP 572 by time.  (or has someone already done that?)
> > I installed GNU Mailman, and downloaded the gzip'ed archives for a
> number of months and unzipped them, and I suspect that there's some way to
> get them all into a single database, but it hasn't jumped out at me.  If I
> count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines
> in one of those text files, I get slightly different numbers for each.
> > Alternatively, they're maybe already in a database, and I just need API
> access to do the querying?  Can someone help me out?
> > Bob ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Comparisions for collections.Counters

[Michael Selik]

On Wed, Sep 5, 2018 at 3:13 AM Evpok Padding 
wrote:

> According to the [doc][1], `collections.Counter` convenience intersection
> and union functions are meant to help it represent multisets. However, it
> currently lacks comparisons, which would make sense and seems
> straightforward to implement.
>

x = Counter(a=1, b=2)
y = Counter(a=2, b=1)
x > y
?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] switch statement

[Michael Selik]

First, this sounds like it belongs on python-ideas, not python-dev.
Second, when you do send a message to python-ideas, it'll help to
accompany it with a realistic example usage that motivates your
proposal.
On Fri, Sep 21, 2018 at 11:18 AM  wrote:
>
> Hi,
>
> A humble proposal for a switch-like statement syntax for Python:
>
> - - -
> switch blah in (100, 2, 30, 'bumm'):
>   dosomething1()
>   x = 88
> case blah in (44, 55):
>   otherstuff(9)
> case blah in (8):
>   boo()
> else:
>   wawa()
> - - -
>
> So, let's use, and allow only *tuples*.
> As early as possible, build a jump table, based on (foreknown) small integer 
> values. As in other languages.
> Strings may have to be hashed (in "compile time"), to obtain small integer 
> value. Some secondary checking may
> have to be done for exact content equality. (Alternative: do no allow strings 
> at all.)
> For gaps in the integer range: maybe apply some very basic dividing/shifting 
> to "compact" the range. (As
> compilers optimize in other languages, I guess -- but I may be totally 
> wrong.) (For example find "unused bits"
> in the numbers (in 2-base representation). newnum = orignum >> 3 & 6 | 
> orignum & ~6. newnum is smaller (roughly
> 1/8) than orignum.)
> The (achievable) goal is to be faster than hash table lookup. (A hash table 
> with keys 100, 2, 30, 'bumm' etc.)
> And approach the speed of single array-index lookup. (Or even faster in some 
> cases as there will be just jumps
> instead of calls?)
> (I am not an "expert"!)
>
> Let allow fallthrough or not? - To be decided. (Either is compatible with the 
> above.)
>
>
> I know about PEP 3103 and
> https://docs.python.org/3.8/faq/design.html?highlight=switch#why-isn-t-there-a-switch-or-case-statement-in-python
>
> (I do not know how to comment on a PEP, where to discuss a PEP. If this is 
> inappropriate place, please forward it.)
>
> --
>
>
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

On Fri, Sep 28, 2018 at 2:11 PM Sean Harrington  wrote:
> kwarg on Pool.__init__ called `expect_initret`, that defaults to False. When 
> set to True:
> Capture the return value of the initializer kwarg of Pool
> Pass this value to the function being applied, as a kwarg.

The parameter name you chose, "initret" is awkward, because nowhere
else in Python does an initializer return a value. Initializers mutate
an encapsulated scope. For a class __init__, that scope is an
instance's attributes. For a subprocess managed by Pool, that
encapsulated scope is its "globals". I'm using quotes to emphasize
that these "globals" aren't shared.


On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington  wrote:
> On Fri, Sep 28, 2018 at 6:45 PM Antoine Pitrou  wrote:
>> 3. If you don't like globals, you could probably do something like
>> lazily-initialize the resource when a function needing it is executed
>
> if initializing the resource is expensive, we only want to do this ONE time 
> per worker process.

We must have a different concept of "lazily-initialize". I understood
Antoine's suggestion to be a one-time initialize per worker process.


On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington  wrote:
> My simple argument is that the developer should not be constrained to make 
> the objects passed globally available in the process, as this MAY break 
> encapsulation for large projects.

I could imagine someone switching from Pool to ThreadPool and getting
into trouble, but in my mind using threads is caveat emptor. Are you
worried about breaking encapsulation in a different scenario?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

On Sat, Sep 29, 2018 at 5:24 AM Sean Harrington  wrote:
>> On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington  wrote:
>> > My simple argument is that the developer should not be constrained to make 
>> > the objects passed globally available in the process, as this MAY break 
>> > encapsulation for large projects.
>>
>> I could imagine someone switching from Pool to ThreadPool and getting
>> into trouble, but in my mind using threads is caveat emptor. Are you
>> worried about breaking encapsulation in a different scenario?
>
> >> Without a specific example on-hand, you could imagine a tree of function 
> >> calls that occur in the worker process (even newly created objects), that 
> >> should not necessarily have access to objects passed from parent -> 
> >> worker. In every case given the current implementation, they will.

Echoing Antoine: If you want some functions to not have access to a
module's globals, you can put those functions in a different module.
Note that multiprocessing already encapsulates each subprocesses'
globals in essentially a separate namespace.

Without a specific example, this discussion is going to go around in
circles. You have a clear aversion to globals. Antoine and I do not.
No one else seems to have found this conversation interesting enough
to participate, yet.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

You don't like using Pool.starmap and itertools.repeat or a comprehension
that repeats an object?


On Wed, Oct 3, 2018, 6:30 PM Sean Harrington  wrote:

> Hi guys -
>
> The solution to "lazily initialize" an expensive object in the worker
> process (i.e. via @lru_cache) is a great solution (that I must admit I did
> not think of). Additionally, in the second use case of "*passing a large
> object to each worker process*", I also agree with your suggestion to
> "shelter functions in a different module to avoid exposure to globals" as a
> good solution if one is wary of globals.
>
> That said, I still think "*passing a large object from parent process to
> worker processes*" should be easier when using Pool. Would either of you
> be open to something like the following?
>
>def func(x, big_cache=None):
>return big_cache[x]
>
>big_cache =  { str(k): k for k in range(1) }
>
>ls = [ i for i in range(1000) ]
>
> with Pool(func_kwargs={"big_cache": big_cache}) as pool:
>
> pool.map(func, ls)
>
>
> It's a much cleaner interface (which presumably requires a more difficult
> implementation) than my initial proposal. This also reads a lot better than
> the "initializer + global" recipe (clear flow of data), and is less
> constraining than the "define globals in parent" recipe. Most importantly,
> when taking sequential code and parallelizing via Pool.map, this does not
> force the user to re-implement "func" such that it consumes a global
> (rather than a kwarg). It allows "func" to be used elsewhere (i.e. in the
> parent process, from a different module, testing w/o globals, etc...)..
>
> This would essentially be an efficient implementation of Pool.starmap(),
> where kwargs are static, and passed to each application of "func" over our
> iterable.
>
> Thoughts?
>
>
> On Sat, Sep 29, 2018 at 3:00 PM Michael Selik  wrote:
>
>> On Sat, Sep 29, 2018 at 5:24 AM Sean Harrington 
>> wrote:
>> >> On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington 
>> wrote:
>> >> > My simple argument is that the developer should not be constrained
>> to make the objects passed globally available in the process, as this MAY
>> break encapsulation for large projects.
>> >>
>> >> I could imagine someone switching from Pool to ThreadPool and getting
>> >> into trouble, but in my mind using threads is caveat emptor. Are you
>> >> worried about breaking encapsulation in a different scenario?
>> >
>> > >> Without a specific example on-hand, you could imagine a tree of
>> function calls that occur in the worker process (even newly created
>> objects), that should not necessarily have access to objects passed from
>> parent -> worker. In every case given the current implementation, they will.
>>
>> Echoing Antoine: If you want some functions to not have access to a
>> module's globals, you can put those functions in a different module.
>> Note that multiprocessing already encapsulates each subprocesses'
>> globals in essentially a separate namespace.
>>
>> Without a specific example, this discussion is going to go around in
>> circles. You have a clear aversion to globals. Antoine and I do not.
>> No one else seems to have found this conversation interesting enough
>> to participate, yet.
>
>
> >>>
>
>>
>>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

Would this change the other pool method behavior in some way if the user,
for whatever reason, mixed techniques?

imap_unordered will only block when nexting the generator. If the user
mingles nexting that generator with, say, apply_async, could the change
you're proposing have some side-effect?

On Tue, Oct 16, 2018, 5:09 AM Sean Harrington  wrote:

> @Nataniel this is what I am suggesting as well. No cacheing - just storing
> the `fn` on each worker, rather than pickling it for each item in our
> iterable.
>
> As long as we store the `fn` post-fork on the worker process (perhaps as
> global), subsequent calls to Pool.map shouldn't be effected (referencing
> Antoine's & Michael's points that "multiprocessing encapsulates each
> subprocesses globals in a separate namespace").
>
> @Antoine - I'm making an effort to take everything you've said into
> consideration here.  My initial PR and talk
>  was intended to shed light
> on a couple of pitfalls that I often see Python end-users encounter with
> Pool. Moving beyond my naive first attempt, and the onslaught of deserved
> criticism, it seems that we have an opportunity here: No changes to the
> interface, just an optimization to reduce the frequency of pickling.
>
> Raymond Hettinger may also be interested in this optimization, as he
> speaks (with great analogies) about different ways you can misuse
> concurrency in Python . This
> would address one of the pitfalls that he outlines: the "size of the
> serialized/deserialized data".
>
> Is this an optimization that either of you would be willing to review, and
> accept, if I find there is a *reasonable way* to implement it?
>
>
> On Fri, Oct 12, 2018 at 3:40 PM Nathaniel Smith  wrote:
>
>> On Fri, Oct 12, 2018, 06:09 Antoine Pitrou  wrote:
>>
>>> On Fri, 12 Oct 2018 08:33:32 -0400
>>> Sean Harrington  wrote:
>>> > Hi Nathaniel - this if this solution can be made performant, than I
>>> would
>>> > be more than satisfied.
>>> >
>>> > I think this would require removing "func" from the "task tuple", and
>>> > storing the "func" "once per worker" somewhere globally (maybe a class
>>> > attribute set post-fork?).
>>> >
>>> > This also has the beneficial outcome of increasing general performance
>>> of
>>> > Pool.map and friends. I've seen MANY folks across the interwebs doing
>>> > things like passing instance methods to map, resulting in "big" tasks,
>>> and
>>> > slower-than-sequential parallelized code. Parallelizing "instance
>>> methods"
>>> > by passing them to map, w/o needing to wrangle with staticmethods and
>>> > globals, would be a GREAT feature! It'd just be as easy as:
>>> >
>>> > Pool.map(self.func, ls)
>>> >
>>> > What do you think about this idea? This is something I'd be able to
>>> take
>>> > on, assuming I get a few core dev blessings...
>>>
>>> Well, I'm not sure how it would work, so it's difficult to give an
>>> opinion.  How do you plan to avoid passing "self"?  By caching (by
>>> equality? by identity?)?  Something else?  But what happens if "self"
>>> changed value (in the case of a mutable object) in the parent?  Do you
>>> keep using the stale version in the child?  That would break
>>> compatibility...
>>>
>>
>> I was just suggesting that within a single call to Pool.map, it would be
>> reasonable optimization to only send the fn once to each worker. So e.g. if
>> you have 5 workers and 1000 items, you'd only pickle fn 5 times, rather
>> than 1000 times like we do now. I wouldn't want to get any fancier than
>> that with caching data between different map calls or anything.
>>
>> Of course even this may turn out to be too complicated to implement in a
>> reasonable way, since it would require managing some extra state on the
>> workers. But semantically it would be purely an optimization of current
>> semantics.
>>
>> -n
>>
>>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

If imap_unordered is currently re-pickling and sending func each time it's
called on the worker, I have to suspect there was some reason to do that
and not cache it after the first call. Rather than assuming that's an
opportunity for an optimization, I'd want to be certain it won't have edge
case negative effects.


On Tue, Oct 16, 2018 at 2:53 PM Sean Harrington 
wrote:

> Is your concern something like the following?
>
> with Pool(8) as p:
> gen = p.imap_unordered(func, ls)
> first_elem = next(gen)
> p.apply_async(long_func, x)
> remaining_elems = [elem for elem in gen]
>

My concern was passing the same function (or a function with the same
qualname). You're suggesting caching functions and identifying them by
qualname to avoid re-pickling a large stateful object that's shoved into
the function's defaults or closure. Is that a correct summary?

If so, how would the function cache distinguish between two functions with
the same name? Would it need to examine the defaults and closure as well?
If so, that means it's pickling the second one anyway, so there's no
efficiency gain.

In [1]: def foo(a):
   ...: def bar():
   ...: print(a)
   ...: return bar
In [2]: f = foo(1)
In [3]: g = foo(2)
In [4]: f
Out[4]: .bar()>
In [5]: g
Out[5]: .bar()>

If we say pool.apply_async(f) and pool.apply_async(g), would you want the
latter one to avoid serialization, letting the worker make a second call
with the first function object?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington 
wrote:

> The most common use case comes up when passing instance methods (of really
> big objects!) to Pool.map().
>

This reminds me of that old joke: "A patient says to the doctor, 'Doctor,
it hurts when I ...!' The doctor replies, 'Well, don't do that.'"

Further, let me pivot on my idea of __qualname__...we can use the `id` of
> `func` as the cache key to address your concern, and store this `id` on the
> `task` tuple (i.e. an integer in-lieu of the `func` previously stored
> there).
>

Possible. Does the Pool keep a reference to the passed function in the main
process? If not, couldn't the garbage collector free that memory location
and a new function could replace it? Then it could have the same qualname
and id in CPython. Edge case, for sure. Worse, it'd be hard to reproduce as
it'd be dependent on the vagaries of memory allocation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

One idea would be for the Pool method to generate a uuid and slap it on the
function as an attribute. If a function being passed in doesn't have one,
generate one. If it already has one, just pass that instead of pickling.
The child process will keep a cache mapping uuids to functions.

I'm still worried about unintended consequences.


On Thu, Oct 18, 2018 at 9:00 AM Michael Selik 
wrote:

> On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington 
> wrote:
>
>> The most common use case comes up when passing instance methods (of
>> really big objects!) to Pool.map().
>>
>
> This reminds me of that old joke: "A patient says to the doctor, 'Doctor,
> it hurts when I ...!' The doctor replies, 'Well, don't do that.'"
>
> Further, let me pivot on my idea of __qualname__...we can use the `id` of
>> `func` as the cache key to address your concern, and store this `id` on the
>> `task` tuple (i.e. an integer in-lieu of the `func` previously stored
>> there).
>>
>
> Possible. Does the Pool keep a reference to the passed function in the
> main process? If not, couldn't the garbage collector free that memory
> location and a new function could replace it? Then it could have the same
> qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to
> reproduce as it'd be dependent on the vagaries of memory allocation.
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

On Fri, Oct 19, 2018 at 5:01 AM Sean Harrington 
wrote:

> I like the idea to extend the Pool class [to optimize the case when only
> one function is passed to the workers].
>

Why would this keep the same interface as the Pool class? If its workers
are restricted to calling only one function, that should be passed into the
Pool constructor. The map and apply methods would then only receive that
function's args and not the function itself. You're also trying to avoid
the initializer/globals pattern, so you could eliminate that parameter from
the Pool constructor. In fact, it sounds more like you'd want a function
than a class. You can call it "procmap" or similar. That's code I've
written more than once.

results = poolmap(func, iterable, processes=cpu_count())

The nuance is that, since there's no explicit context manager, you'll want
to ensure the pool is shut down after all the tasks are finished, even if
the results generator hasn't been fully consumed.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

This thread seems more appropriate for python-ideas than python-dev.


On Mon, Oct 22, 2018 at 5:28 AM Sean Harrington 
wrote:

> Michael - the initializer/globals pattern still might be necessary if you
> need to create an object AFTER a worker process has been instantiated (i.e.
> a database connection).
>

You said you wanted to avoid the initializer/globals pattern and have such
things as database connections in the defaults or closure of the
task-function, or the bound instance, no? Did I misunderstand?


Further, the user may want to access all of the niceties of Pool, like
> imap, imap_unordered, etc.  The goal (IMO) would be to preserve an
> interface which many Python users have grown accustomed to, and to allow
> them to access this optimization out-of-the-bag.
>

You just said that the dominant use-case was mapping a single
task-function. It sounds like we're talking past each other in some way.
It'll help to have a concrete example of a case that satisfies all the
characteristics you've described: (1) no globals used for communication
between initializer and task-functions; (2) single task-function, mapped
once; (3) an instance-method as task-function, causing a large
serialization burden; and (4) did I miss anything?



> Having talked to folks at the Boston Python meetup, folks on my dev team,
> and perusing stack overflow, this "instance method parallelization" is a
> pretty common pattern that is often times a negative return on investment
> for the developer, due to the implicit implementation detail of pickling
> the function (and object) once per task.
>

I believe you.


> Is anyone open to reviewing a PR concerning this optimization of Pool,
> delivered as a subclass? This feature restricts the number of unique tasks
> being executed by workers at once to 1, while allowing aggressive
> subprocess-level function cacheing to prevent repeated
> serialization/deserialization of large functions/closures. The use case is
> s.t. the user only ever needs 1 call to Pool.map(func, ls) (or friends)
> executing at once, when `func` has a non-trivial memory footprint.
>

You're quite eager to have this PR merged. I understand that. However, it's
reasonable to take some time to discuss the design of what you're
proposing. You don't need it in the stdlib to get your own work done, nor
to share it with others.

>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Michael Selik]

On Fri, Nov 9, 2018 at 4:33 PM Victor Stinner  wrote:

> It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the
> name) to get the new API. The current C API is unchanged.
>

While one can hope that this will be the only time the C API will be
revised, it may be better to number it instead of calling it "NEW". 20
years from now, it won't feel new anymore.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Standard library vs Standard distribution?

[Michael Selik]

On Thu, Nov 29, 2018 at 1:33 PM Nathaniel Smith  wrote:

> > > https://anaconda.com/
> > > https://www.activestate.com/products/activepython/
> > > http://winpython.github.io/
> > > http://python-xy.github.io/
> > > https://www.enthought.com/product/canopy/
> > > https://software.intel.com/en-us/distribution-for-python
> > > http://every-linux-distro-ever.example.com
>
> - Currently, our single-box-of-batteries is doing such a lousy job of
> solving Paul's problem, that people are building whole businesses on
> our failure.
>

Would you say that the Linux community made the same failure, since
companies like Red Hat and Canonical exist? One reason a company might
purchase a distribution is simply to have someone to sue, maybe
indemnification against licensing issues.

Even if there were an official free distribution, companies might still
choose to purchase one.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Standard library vs Standard distribution?

[Michael Selik]

On Thu, Nov 29, 2018 at 5:48 PM Oscar Benjamin 
wrote:

> There have been significant improvements in pip, pypi and the whole
> packaging ecosystem in recent years thanks to the efforts of many
> including Paul. I've been pushing students and others to Anaconda
> simply because I knew that at minimum they would need numpy, scipy and
> matplotlib and that pip would fail to install those. That has changed
> in the last year or two: it is now possible to pip install binaries
> for those packages and many others on the 3 major OSes. I don't see
> any reason to recommend Anaconda to my students now.
>

One advantage of conda over pip is avoidance of typosquatting on PyPI.


It's not a huge problem for my students but I think an important thing
> missing from all of this is a GUI for pip. The situation Paul
> described is that you can instruct someone to install Python using the
> python.org installer but cannot then instruct them to use pip from a
> terminal and I can certainly relate to that. If installing Python gave
> you a GUI from the programs menu that could install the other pieces
> that would be a significant improvement.
>

I haven't checked in about a year, but the Windows installer from python.org
wouldn't set the PATH such that python or pip were not valid executables at
the DOS prompt. Fixing that one (simple?) thing would also be a significant
improvement.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How about updating OrderedDict in csv and configparser to regular dict?

[Michael Selik]

Waiting on review
https://github.com/python/cpython/pull/8014

On Thu, Jan 31, 2019, 12:04 AM INADA Naoki  I'm sorry, configparser is changed already.
> https://bugs.python.org/issue33504
>
> On Thu, Jan 31, 2019 at 4:52 PM INADA Naoki 
> wrote:
> >
> > Hi,
> >
> > csv.DictReader uses OrderedDict by default, from Python 3.6.
> > But it doesn't make sense anymore, like namedtuple._asdict().
> > How about changing default dict type back to regular dict.
> >
> > Python is widely used for handling learge data.  So I think
> > changing default dict type to OrderedDict was performance
> > and memory usage regression in 3.6.
> >
> > Additionally, configparser uses OrderedDict by default from Python 3.6
> too.
> >
> > I am not sure about `parser['section1'] == parser['section2']` is not
> used yet.
> > But we broke it once in 3.6 by changing dict to OrderedDict.  Are there
> any
> > issue report caused by this backward incompatibility?
> >
> > And I think performance and memory efficiency is not so important for
> > configparser, unlike csv.
> >
> > I'm
> >
> > * +1 about changing csv.DictReader's default dict type
> > * +0.5 about changing configparser's default dict type.
> >
> > How do you think?
> >
> > Regards,
> > --
> > INADA Naoki  
>
>
>
> --
> INADA Naoki  
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Michael Selik]

On Tue, Oct 19, 2021 at 9:39 AM Chris Angelico  wrote:

> On Wed, Oct 20, 2021 at 3:25 AM Baptiste Carvello
>  wrote:
> >
> > Le 18/10/2021 à 20:26, Guido van Rossum a écrit :
> > >
> > > y = None  # Default
> > > if config is not None:
> > >   handler = config.get("handler")
> > >   if handler is not None:
> > > parameters = handler.get("parameters")
> > > if parameters is not None:
> > >   y = parameters.get("y")
> > >
> > > […]
> > > Using ?. this can be written as
> > >
> > > y = config?.get("handler")?.get("parameters")?.get("y")
> >
> > Sure, but the EAFP version is not that bad:
> >
> > try:
> >   y = config["handler"]["parameters"]["y"]
> > except KeyError:
> >   y = None
> >
> > which could be further simplified with an exception-catching expression
> > (caveat: keyword usage is pure improvisation, it sounds good, but is
> > probably broken :-) :
> >
> > y = config["handler"]["parameters"]["y"] with KeyError as None
> >
> > The PEP authors would probably reject this as "hiding errors in code",
> > which is true, but none-aware operators also hide errors…
>
> PEP 463 would like to say hi :)
>

In case it saves anyone a couple clicks:
https://www.python.org/dev/peps/pep-0463/

I also prefer more syntactic help with exceptions, rather than more syntax
emphasizing None's uniqueness.

None and its ilk often conflate too many qualities. For example, is it
missing because it doesn't exist, it never existed, or because we never
received a value, despite knowing it must exist? The languages SAS and R
support at least 27 varieties of NA, allowing un-tagged, and tagged with
the letters A-Z to help someone create distinctions between different kinds
of nothingness. IEEE-754 allows about 16 million possible NaNs, which I
believe was intended to allow floating point instructions to pass error
messages along.

If the motivation for this operator is chained lookups, how about adding a
feature to the operator module, first? It seems natural to add a
keyword-only argument to `attrgetter`, and it's a lighter touch than
implementing a new operator. If use becomes widespread, that gives more
weight to PEP 505.

def attrgetter(*attrs, none_aware=False)

https://docs.python.org/3/library/operator.html#operator.attrgetter

Apologies if attrgetter has already been discussed. I didn't see mention of
it in PEP 505.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R5NO5H5BCEIGEUCLPRE7WW5AEG5MRX3Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Michael Selik]

On Wed, Oct 20, 2021 at 1:16 AM Piotr Waszkiewicz 
wrote:

> On Wed, Oct 20, 2021 at 2:33 AM Michael Selik  wrote:
>
>> In case it saves anyone a couple clicks:
>> https://www.python.org/dev/peps/pep-0463/
>> I also prefer more syntactic help with exceptions, rather than more
>> syntax emphasizing None's uniqueness.
>>
>
> Me too, but could you provide me an example where try-except approach is
> more readable when trying to chain attribute lookups (like in the database
> book-publisher-owner example I have provided before).
>

I'd echo the others' examples, taking inspiration from PEP 463.


> If the motivation for this operator is chained lookups, how about adding a
>> feature to the operator module, first? It seems natural to add a
>> keyword-only argument to `attrgetter`, and it's a lighter touch than
>> implementing a new operator. If use becomes widespread, that gives more
>> weight to PEP 505.
>>
>> def attrgetter(*attrs, none_aware=False)
>>
>> https://docs.python.org/3/library/operator.html#operator.attrgetter
>>
>
> I remember using inhouse solution like this at a certain workplace - a
> method accepting list of string arguments and an object, returning the
> value being the result of chained attribute access. And it worked all
> right. The problem I have with such approaches is that the name of the
> attrs are passed as strings.
>

I understand the preference for attributes over strings, but many of the
none-aware examples use keys and indices. If JSON is the main culprit for
deeply nested structures, then you're already using strings and not
attributes. Adding features to `operator` wouldn't preclude accepting PEP
505, so why not get started with a less controversial change that provides
much of the value?

If PEP 505 is accepted, it would need support in the `operator` module.
Might as well design that aspect of the implementation now.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WIKNWUS2W7LOIHAM3FU4FY2SFQRD3K7D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Michael Selik]

On Wed, Oct 20, 2021 at 9:18 AM Piotr Waszkiewicz 
wrote:

> Do you think about something along those lines?
> ```
> phone = book.publisher.owner.phone except AttributeError: None
> ```
>

Yes, that seems reasonable.


> I don't mind this syntax but it would have to be supported by static type
> checkers and IDEs. And currently something like this is not:
> ```
> try:
> phone = book.publisher.owner.phone
> except AttributeError:
> phone = None
> ```
>
> mypy complains:
> ```
> error: Item "None" of "Optional[Publisher]" has no attribute "owner"
> ```
>

That sounds like a feature request for mypy. Would creating a new operator
make it easier to implement analysis of that situation would mypy? My guess
is not. Checking the AST to see if there's a try/except AttributeError
sounds comparable to checking for the use of a none-aware operator. I'm
completely ignorant of how mypy does its analysis, so that's just a wild
guess.

If PEP 505 is accepted, it would need support in the `operator` module.
>> Might as well design that aspect of the implementation now.
>>
>
> I'm sorry but I don't know if I understand that sentence correctly. You
> mean we would have to add an "explicit" function that behaves like a
> maybe-dot operator? Is it actually a requirement when adding new operators?
>

The documentation of the `operator` module says, "The operator module
exports a set of efficient functions corresponding to the intrinsic
operators of Python." It feels like there's an implicit "all" in there. The
table of correspondences looks exhaustive. I haven't noticed any exceptions.

https://docs.python.org/3/library/operator.html#mapping-operators-to-functions
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/46H5MYVVMQMWS4DVJYVFGKWXS3CNYTJC/
Code of Conduct: http://python.org/psf/codeofconduct/