[Python-Dev] CPython on Windows ARM32

[Paul Monson via Python-Dev]

Hi Python Developers,



I'm Paul Monson, I've spent about 20 years working with embedded software.  
Since 2010 I've worked for Microsoft as a developer.



Our team is working with CPython on Azure IoT Edge devices that run on 
x64-based devices.



We would like to extend that support to Windows running on ARM 32-bit devices 
and have a working proof-of-concept.  Our team is prepared to provide support 
for CPython for Windows on ARM32 for 10 years, and to provide build bots for 
ARM32.



I like to propose that the initial sequence of PRs could be:
- Update to OpenSSL 1.1.1 (without anything ARM specific) - ready to go
- Migrate to libffi directly (finish 
https://github.com/python/cpython/pull/3806)
- Build file updates for OpenSSL ARM and check into cpython-bin-deps
- Build file updates for CPython ARM
- ctypes updates for ARM
- Test module skips for ARM
- Library updates and related test fixes for ARM



Updating OpenSSL and libffi are independent of ARM support but need to be done 
as prerequisites.  OpenSSL 1.1.0 doesn't have support for ARM32 on Windows but 
OpenSSL 1.1.1 does.



I have OpenSSL 1.1.1a ready to check in to master with all tests passing on x86 
and x64 on Windows.  Since work has already been done on this for other 
platforms only very small changes were needed for Windows.



I have also integrated and tested the current libffi on Windows x64.  Some 
additonal porting of x86 assembler to MSVC tools will need to be done.  I have 
a working port of ARM32 assembler for MSVC but it may need to be brought up to 
date and cleaned up.



The last four all need to go in together, but can be reviewed separately.



We are not planning to support Tk/Tcl on ARM32 because Windows IoT Core, 
Windows containers don't support GDI, which is a depenency of Tk/Tcl.



Since Window IoT Core and Windows container don't support the .msi or .exe 
installers found on python.org my team at Microsoft will build the CPython for 
Windows ARM32 from the official repo and distribute it.



Thanks in advance,

Paul

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Paul Monson via Python-Dev]

The PR is here: https://github.com/python/cpython/pull/11774
Searching _M_ARM I see these #ifdef changes outside of ctypes:
* Include\pyport.h  - adds on to existing MSVC ifdef
* Include\pythonrun.h - adds on to existing MSVC ifdef
* Modules\_decimal\libmpdec\bits.h
* Python\ceval.c - workaround compiler bug, could be replaced with #pragma 
optimize around entire function.

-Original Message-
From: Steve Dower  
Sent: Wednesday, February 6, 2019 11:16 AM
To: Terry Reedy ; python-dev@python.org; Paul Monson 

Subject: Re: [Python-Dev] CPython on Windows ARM32

On 06Feb2019 0054, Terry Reedy wrote:
> On 2/5/2019 10:10 PM, Zachary Ware wrote:
> 
>> I'm all for the first two changes (especially the second), and if 10 
>> years of pledged corporate support for a new platform is the price we 
>> have to pay for them, I'm ok with that :).
> 
> I would expect that the main question should be the density of 
> WinArm32-specific ifdefs in the main code and extensions other than ctypes.
> 

Agreed. I've asked Paul to post the "final" PR early, even though it will take 
some refactoring as other PRs go in, so that we can see the broader picture now.

There's also an option to create an ARM-specific pyconfig.h if necessary, but I 
don't believe it will be.

I created 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.python.org%2Fissue35920&data=02%7C01%7Cpaul.monson%40microsoft.com%7Cf1e74ec935774410f37008d68c678537%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636850773687302395&sdata=OPOHUbWy3%2FFEdjXC5MY8NRMVetZ73Rwo2lsngrLL8rs%3D&reserved=0
 for this work.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] CPython on Windows ARM32

[Paul Monson via Python-Dev]

Here are the current OpenSSL 1.1.1a changes I have, in a seperate PR
I did some additional testing and have some test failures to investigate 
tomorrows

test_parse_cert_CVE_2019_5010 only fails win32 debug (access violation)
   works for amd64 debug/release and win32 release

test_load_default_certs_env_windows fails on win32 and amd64 retail.  skipped 
on debug

-Original Message-
From: Steve Dower  
Sent: Wednesday, February 6, 2019 3:28 PM
To: Christian Heimes ; Paul Monson 
; python-dev@python.org; Ned Deily 
Subject: Re: [Python-Dev] CPython on Windows ARM32

On 06Feb2019 1423, Christian Heimes wrote:
> Do you want to update Python 3.8 (master) only or also 3.7? I'm not 
> strictly against updating 3.7. However we have traditionally kept the 
> OpenSSL version of each branch stable. 1.1.1 comes with new features, 
> stricter security settings and some ciphers removed.

I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is distutils.util.get_platform() the "current" or the "target" platform

[Paul Monson via Python-Dev]

Thanks for the feedback. 
I updated the PR to use get_platform and get_host_platform.
More testing is still needed before it's ready to merge to make sure it still 
does what it was intended to do.

-Original Message-
From: Python-Dev  On 
Behalf Of Xavier de Gaye
Sent: Tuesday, February 19, 2019 12:06 PM
To: Steve Dower ; python-dev@python.org
Subject: Re: [Python-Dev] Is distutils.util.get_platform() the "current" or the 
"target" platform

> [Any reason for dropping python-dev?]

Sorry. just clicked the wrong button.

> And the answer is a resounding "yes, it returns the target platform"? It 
> seems you're saying this, but the wording of your email sounds just enough of 
> a question that I'm not sure whether you are definitively answering it or not.

The answer is yes indeed, it returns the target platform.
This is a listing of the non-obvious steps that lead to this conclusion.

Xavier


On 2/19/19 8:45 PM, Steve Dower wrote:
> [Any reason for dropping python-dev?]
> 
> On 19Feb2019 1139, Xavier de Gaye wrote:
>> Is distutils.util.get_platform() the "current" or the "target" platform ?
> 
> I *think* you're answering this below, yes?
> 
>> * When cross-compiling on posix platforms using autoconf, configure is
>>    run with the '--host=host-type' [1] command line option to specify
>>    the target platform.
>>
>> * The AC_CANONICAL_HOST macro is used by configure.ac to get the
>>    canonical variables `host' and `host_cpu' [2].
>>    Those variables are used to compute _PYTHON_HOST_PLATFORM.
>>
>> * The Makefile generated by configure runs setup.py,
>>    generate-posix-vars, etc... on the build platform using
>>    PYTHON_FOR_BUILD, a native python interpreter that is set
>>    to run with the _PYTHON_HOST_PLATFORM environment variable.
>>
>> * get_platform() in setup.py and in Lib/distutils/util.py returns the
>>    value of _PYTHON_HOST_PLATFORM when cross-compiling.
>>
>> So the process of cross-compilation on posix platforms has
>> get_platform() return the target ('host' in autoconf terminology) platform.
>>
>> [1] 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gnu.org%2Fsavannah-checkouts%2Fgnu%2Fautoconf%2Fmanual%2Fautoconf-2.69%2Fhtml_node%2FSpecifying-Target-Triplets.html&data=02%7C01%7CPaul.Monson%40microsoft.com%7C186de203c3d644c7517208d696a5f266%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636862036817328765&sdata=2PMKn%2Bt5ed82gkSBnew0nT1TA1qzDuU3VZNOFPYPqIM%3D&reserved=0
>> [2] 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gnu.org%2Fsavannah-checkouts%2Fgnu%2Fautoconf%2Fmanual%2Fautoconf-2.69%2Fhtml_node%2FCanonicalizing.html&data=02%7C01%7CPaul.Monson%40microsoft.com%7C186de203c3d644c7517208d696a5f266%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636862036817328765&sdata=zRtcyLjDrkL%2BfcignLCDjtUri29j7mCscdVkGqhNk50%3D&reserved=0
> 
> And the answer is a resounding "yes, it returns the target platform"? It 
> seems you're saying this, but the wording of your email sounds just enough of 
> a question that I'm not sure whether you are definitively answering it or not.
> 
> Cheers,
> Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-dev&data=02%7C01%7CPaul.Monson%40microsoft.com%7C186de203c3d644c7517208d696a5f266%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636862036817328765&sdata=Au0xrggEwNRRAen6dr8GnBuTTvMbxVhSuWOtX67p3Ts%3D&reserved=0
Unsubscribe: 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Foptions%2Fpython-dev%2Fpaulmon%2540microsoft.com&data=02%7C01%7CPaul.Monson%40microsoft.com%7C186de203c3d644c7517208d696a5f266%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636862036817328765&sdata=nVHiEW69so29h6uTbY7XDjPF6%2FwU0pnlNpQEeYSoygY%3D&reserved=0
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Testing cross-compiled python (windows arm32) from build bot

[Paul Monson via Python-Dev]

Hello,

For windows arm32 I would like to be able run tests in the buildbot.
Windows arm32 programs built with MSVC are always cross-compiled.
This means I need to build cpython on x86/x64 and then deploy the build 
artifacts and run the tests on an ARM32 device.

Is there any documentation or examples of how to do this?
If not, are there any recommendations for how to achieve this?

Thanks,
Paul Monson

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com