Re: [Python-Dev] CPython on Windows ARM32

[Terry Reedy]

On 2/5/2019 10:10 PM, Zachary Ware wrote:


I'm all for the first two changes (especially the second), and if 10
years of pledged corporate support for a new platform is the price we
have to pay for them, I'm ok with that :).


I would expect that the main question should be the density of 
WinArm32-specific ifdefs in the main code and extensions other than ctypes.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Nick Coghlan]

On Wed, 6 Feb 2019 at 05:17, Neil Schemenauer  wrote:
> My gut reaction is that we shouldn't revert.  However, looking at
> the changes, it seems 'multiprocessing.shared_memory' could be an
> external extension package that lives in PyPI.  It doesn't require
> changes to other interpreter internals.  It doesn't seem to require
> internal Python header files.

The desired dependency in this case goes the other way: we'd like this
in the standard library so that other standard library components can
use it, and it can eventually become part of the "assumed baseline"
that the reference Python interpreter offers to projects building on
top of it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Giampaolo Rodola']

Davin,
I am not familiar with the multiprocessing module, so take the following
with a big grain of salt. I took a look at the PR, then I got an idea of
how multiprocessing module is organized by reading the doc. Here's some
food for thought in terms of API reorganization.

SharedMemoryManager, SharedMemoryServer
---

It appears to me these are the 2 main public classes, and after reading the
doc it seems they really belong to "managers
"
(multiprocessing.managers namespace). Also:
* SharedMemoryManager is a subclass of multiprocessing.managers.SyncManager
* SharedMemoryServer is a subclass of multiprocessing.managers.Server
shared_memory.py could be renamed to _shared_memory.py and managers.py
could import and expose these 2 classes only.

Support APIs


These are objects which seem to be used in support of the 2 classes above,
but apparently are not meant to be public. As such they could simply live
in _shared_memory.py and not be exposed:

- shareable_wrap(): used only in SharedMemoryTracker.wrap()
- SharedMemoryTracker: used only by SharedMemoryServer
- SharedMemory, WindowsNamedSharedMemory, PosixSharedMemory: used by
shareable_wrap() and SharedMemoryTracker
- ShareableList: it appears this is not used, but by reading here

I have a doubt: shouldn't it be register()ed against SharedMemoryManager?

C extension module
--

- ExistentialError, Error - it appears these are not used
- PermissionsException, ExistentialException - I concur with Ronald
Oussoren's review: you could simply use PyErr_SetFromErrno() and let the
original OSError exception bubble up. Same for O_CREAT, O_EXCL, O_CREX,
O_TRUNC which are already exposed in the os module. I have a couple of
other minor nitpicks re. the code but I will comment on the PR.

Compatibility
-

I'm not sure if SyncManager and SharedMemoryManager are fully
interchangeable so I think the doc should clarify this. SyncManager handles
a certain set of types
.
It appears SharedMemoryManager is supposedly able to do the same except for
lists
.
Is my assumption correct? Also, multiprocessing.Manager() by default
returns a SyncManager. If we'll get to a point where SyncManager and
SharedMemoryManager are able to handle the same types it'd be good to
return SharedMemoryManager as the default, but it's probably safer to leave
it for later. Unless they are already there (I don't know) it would be good
to have a full set of unit-tests for all the register()ed types and test
them against SyncManager and SharedMemoryManager. That would give an idea
on the real interchangeability of these 2 classes and would also help
writing a comprehensive doc.

Hope this helps.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/6/19 8:43 AM, Stephane Wirtel wrote:

On 02/05, Barry Warsaw wrote:

On Feb 5, 2019, at 02:24, Stephane Wirtel  wrote:
You’re welcome!  I just pushed an update to add 3.8.0a1 to the set of 
Python’s (including git head).  Do you think there’s a better way to 
publicize these images?


I know that Julien Palard wanted a docker image with all the versions of
Python, see: https://github.com/docker-library/python/issues/373

For my part, I wanted to propose a docker image with the last version of
Python and try to use it for the detection of bugs in the main python
projects (django, numpy, flask, pandas, etc...) with a CI (example:
Gitlab-CI)

First issue: pytest uses the ast module of python and since 3.8.0a1, the
tests do not pass -> new issue for pytest


FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
alphas/betas, so everything's tested when 3.8.0 is released: 
https://fedoraproject.org/wiki/Changes/Python3.8


That should cover the main Python projects, too.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Matthias Klose]

On 06.02.19 13:23, Petr Viktorin wrote:
> FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
> alphas/betas,
> so everything's tested when 3.8.0 is released:
> https://fedoraproject.org/wiki/Changes/Python3.8
> 
> That should cover the main Python projects, too.

well, the real challenge is that all test suites of third party packages still
pass on all architectures.  From past transitions, I know that this costs the
most time and resources.  But yes, targeting 3.8 for Ubuntu 20.04 LTS as well.

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Petr Viktorin]

On 2/6/19 2:26 PM, Matthias Klose wrote:

On 06.02.19 13:23, Petr Viktorin wrote:

FWIW, we're preparing to rebuild all Fedora packages with the 3.8 alphas/betas,
so everything's tested when 3.8.0 is released:
https://fedoraproject.org/wiki/Changes/Python3.8

That should cover the main Python projects, too.


well, the real challenge is that all test suites of third party packages still
pass on all architectures.  From past transitions, I know that this costs the
most time and resources. 


Same experience here.

In Fedora, tests are generally run as part of the build.
(Sorry, that was definitely not obvious from my message!)


But yes, targeting 3.8 for Ubuntu 20.04 LTS as well.


\o/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asking for reversion

[Giampaolo Rodola']

On Wed, Feb 6, 2019 at 12:51 PM Giampaolo Rodola' 
wrote:

>
> Unless they are already there (I don't know) it would be good to have a
> full set of unit-tests for all the register()ed types and test them against
> SyncManager and SharedMemoryManager. That would give an idea on the real
> interchangeability of these 2 classes and would also help writing a
> comprehensive doc.
>

In order to speed up the alpha2 inclusion process I created a PR which
implements what said above:
https://github.com/python/cpython/pull/11772
https://bugs.python.org/issue35917
Apparently SharedMemoryManager works out of the box and presents no
differences with SyncManager, but the list type is not using ShareableList.
When I attempted to register it with "SharedMemoryManager.register('list',
list, ShareableList)" I got the following error:

Traceback (most recent call last):
  File "foo.py", line 137, in test_list
o = self.manager.list()
  File "/home/giampaolo/svn/cpython/Lib/multiprocessing/managers.py", line
702, in temp
proxy = proxytype(
TypeError: __init__() got an unexpected keyword argument 'manager'

I am not sure how to fix that (I'll leave it up to Davin). The tests as-is
are independent from PR-11772 so I suppose they can be reviewed/checked-in
regardless of the changes which will affect shared_memory.py.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] About the future of multi-process Python

[Antoine Pitrou]

Hello,

For the record there are number of initiatives currently to boost the
usefulness and efficiency of multi-process computation in Python.

One of them is PEP 574 (zero-copy pickling with out-of-band buffers),
which I'm working on.

Another is Pierre Glaser's work on allowing pickling of dynamic
functions and classes with the C-accelerated _pickle module (rather than
the slow pure Python implementation):
https://bugs.python.org/issue35900
https://bugs.python.org/issue35911

Another is Davin's work on shared memory managers.

There are also emerging standards like Apache Arrow that provide a
shared, runtime-agnostic, compute-friendly representation for in-memory
tabular data, and third-party frameworks like Dask which are
potentially able to work on top of that and expose nice end-user APIs.

For maximum synergy between these initiatives and the resulting APIs,
it is better if things are done in the open ;-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About the future of multi-process Python

[Steve Dower]

On 06Feb2019 0906, Antoine Pitrou wrote:

For the record there are number of initiatives currently to boost the
usefulness and efficiency of multi-process computation in Python.

One of them is PEP 574 (zero-copy pickling with out-of-band buffers),
which I'm working on.

Another is Pierre Glaser's work on allowing pickling of dynamic
functions and classes with the C-accelerated _pickle module (rather than
the slow pure Python implementation):
https://bugs.python.org/issue35900
https://bugs.python.org/issue35911

Another is Davin's work on shared memory managers.

There are also emerging standards like Apache Arrow that provide a
shared, runtime-agnostic, compute-friendly representation for in-memory
tabular data, and third-party frameworks like Dask which are
potentially able to work on top of that and expose nice end-user APIs.

For maximum synergy between these initiatives and the resulting APIs,
it is better if things are done in the open ;-)


Hopefully our steering council can determine (or delegate the 
determination of) the direction we should go here so we can all be 
pulling in the same direction :)


That said, there are certainly a number of interacting components and 
not a lot of information about how they interact and overlap. A good 
start would be to identify the likely overlap of this work to see where 
they can build upon each other rather than competing, as well as 
estimating the long-term burden of standardising.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Steve Dower]

On 06Feb2019 0054, Terry Reedy wrote:

On 2/5/2019 10:10 PM, Zachary Ware wrote:


I'm all for the first two changes (especially the second), and if 10
years of pledged corporate support for a new platform is the price we
have to pay for them, I'm ok with that :).


I would expect that the main question should be the density of 
WinArm32-specific ifdefs in the main code and extensions other than ctypes.




Agreed. I've asked Paul to post the "final" PR early, even though it 
will take some refactoring as other PRs go in, so that we can see the 
broader picture now.


There's also an option to create an ARM-specific pyconfig.h if 
necessary, but I don't believe it will be.


I created https://bugs.python.org/issue35920 for this work.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Paul Monson via Python-Dev]

The PR is here: https://github.com/python/cpython/pull/11774
Searching _M_ARM I see these #ifdef changes outside of ctypes:
* Include\pyport.h  - adds on to existing MSVC ifdef
* Include\pythonrun.h - adds on to existing MSVC ifdef
* Modules\_decimal\libmpdec\bits.h
* Python\ceval.c - workaround compiler bug, could be replaced with #pragma 
optimize around entire function.

-Original Message-
From: Steve Dower  
Sent: Wednesday, February 6, 2019 11:16 AM
To: Terry Reedy ; python-dev@python.org; Paul Monson 

Subject: Re: [Python-Dev] CPython on Windows ARM32

On 06Feb2019 0054, Terry Reedy wrote:
> On 2/5/2019 10:10 PM, Zachary Ware wrote:
> 
>> I'm all for the first two changes (especially the second), and if 10 
>> years of pledged corporate support for a new platform is the price we 
>> have to pay for them, I'm ok with that :).
> 
> I would expect that the main question should be the density of 
> WinArm32-specific ifdefs in the main code and extensions other than ctypes.
> 

Agreed. I've asked Paul to post the "final" PR early, even though it will take 
some refactoring as other PRs go in, so that we can see the broader picture now.

There's also an option to create an ARM-specific pyconfig.h if necessary, but I 
don't believe it will be.

I created 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.python.org%2Fissue35920&data=02%7C01%7Cpaul.monson%40microsoft.com%7Cf1e74ec935774410f37008d68c678537%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636850773687302395&sdata=OPOHUbWy3%2FFEdjXC5MY8NRMVetZ73Rwo2lsngrLL8rs%3D&reserved=0
 for this work.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Christian Heimes]

On 06/02/2019 02.09, Paul Monson via Python-Dev wrote:
> Updating OpenSSL and libffi are independent of ARM support but need to
> be done as prerequisites.  OpenSSL 1.1.0 doesn't have support for ARM32
> on Windows but OpenSSL 1.1.1 does.
> 
>  
> 
> I have OpenSSL 1.1.1a ready to check in to master with all tests passing
> on x86 and x64 on Windows.  Since work has already been done on this for
> other platforms only very small changes were needed for Windows.

+1 for OpenSSL 1.1.1 from the maintainer of the ssl module. The new
version also introduces TLS 1.3 support. Linux distributions have been
switching to OpenSSL 1.1.1 for a while. If it's good enough for RHEL 8,
then it's good enough for us, too.

Do you want to update Python 3.8 (master) only or also 3.7? I'm not
strictly against updating 3.7. However we have traditionally kept the
OpenSSL version of each branch stable. 1.1.1 comes with new features,
stricter security settings and some ciphers removed.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Steve Dower]

On 06Feb2019 1423, Christian Heimes wrote:

Do you want to update Python 3.8 (master) only or also 3.7? I'm not
strictly against updating 3.7. However we have traditionally kept the
OpenSSL version of each branch stable. 1.1.1 comes with new features,
stricter security settings and some ciphers removed.


I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Ned Deily]

On Feb 6, 2019, at 18:28, Steve Dower  wrote:
> On 06Feb2019 1423, Christian Heimes wrote:
>> Do you want to update Python 3.8 (master) only or also 3.7? I'm not
>> strictly against updating 3.7. However we have traditionally kept the
>> OpenSSL version of each branch stable. 1.1.1 comes with new features,
>> stricter security settings and some ciphers removed.
> I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager.

Me, too.  I am concerned that 1.1.1 support has not had a lot of exposure yet.  
Even the "What's New" document for 3.7 states: "The ssl module has preliminary 
and experimental support for TLS 1.3 and OpenSSL 1.1.1. "

I am OK with fixes for 1.1.1 support but I think it would be premature to 
change the Windows and/or macOS installers from 1.1.0 to 1.1.1.

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Paul Monson via Python-Dev]

Here are the current OpenSSL 1.1.1a changes I have, in a seperate PR
I did some additional testing and have some test failures to investigate 
tomorrows

test_parse_cert_CVE_2019_5010 only fails win32 debug (access violation)
   works for amd64 debug/release and win32 release

test_load_default_certs_env_windows fails on win32 and amd64 retail.  skipped 
on debug

-Original Message-
From: Steve Dower  
Sent: Wednesday, February 6, 2019 3:28 PM
To: Christian Heimes ; Paul Monson 
; python-dev@python.org; Ned Deily 
Subject: Re: [Python-Dev] CPython on Windows ARM32

On 06Feb2019 1423, Christian Heimes wrote:
> Do you want to update Python 3.8 (master) only or also 3.7? I'm not 
> strictly against updating 3.7. However we have traditionally kept the 
> OpenSSL version of each branch stable. 1.1.1 comes with new features, 
> stricter security settings and some ciphers removed.

I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com