[Numpy-discussion] Numpy binary wheels and CI for win/arm64 platform

[Niyas Sait]

Hello,

Numpy v1.21.2  added
support for windows/arm64 platforms but we still don't have any systems in
place to produce binary wheels or test win/arm64 packages. I think it will
be good to start looking into this. CPython has an official buildbot worker
running for win/arm64 and official python support for the platform will be
available from the 3.11 release.

It is not yet clear to me how the build and CI system for numpy is deployed
and how to enable support for a new platform like win/arm64.

One of the main issues in supporting win/arm64 build would be due to the
lack of win/arm64 VMs available on the cloud. But I see we have been
producing binary wheels for Apple M1 platforms on pypi and conda repository
for some time which also lacks the cloud VM support. I think we could take
some learnings from Apple M1 support and look at how a similar strategy can
be used for win/arm64.

I would like to hear if anyone has any thoughts on this topic. Also, any
pointers to understand numpy wheel generation and CI flow for similar
platforms would be helpful as well.

Thanks,
Niyas
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Numpy binary wheels and CI for win/arm64 platform

[Niyas Sait]

Hello,

It has been a while since we discussed adding CI/CD support for Windows on
Arm. Let me share a few updates and see if we can find a way forward.

First of all, Thanks for suggesting cibuildwheel. We have couple of patches
in progress to add native [1] and cross-compilation support [2] to
cibuildwheel. They could solve the binary wheel creation and releases.
However, we may still lack support for CI.

I would like to check if there is any interest in using Windows/Arm64 VMs
in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) [4] to
add CI support for WoA platform.

In order to use Azure VMs with GitHub Actions or Azure Pipeline, we must
set up self-hosted runners. I am not sure if that would be an acceptable
workflow for the numpy community. If that could work we could potentially
find ways to fund the required VMs for NumPy project.

Also, we could request access to new Volterra machines for Numpy CI/CD. I
know it is not easy to maintain them as part of CI/CD workflow, but if any
maintainers are interested in using them for CI/CD purposes, we can
progress on that discussion.

Let me know what you think.

Thanks
Niyas

[1] https://github.com/pypa/cibuildwheel/pull/920
[2] https://github.com/pypa/cibuildwheel/pull/1108
[3]
https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines-with-ampere-altra-armbased-processors/
[4]
https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation-experiences-at-scale-with-windows/

On Tue, 2 Nov 2021 at 14:33, Ralf Gommers  wrote:

>
>
> On Tue, Nov 2, 2021 at 3:15 PM Peter Cock 
> wrote:
>
>> 
>>
>> On Tue, Nov 2, 2021 at 1:07 PM Ralf Gommers 
>> wrote:
>> >
>> > Our current wheel build machinery is at
>> https://github.com/MacPython/numpy-wheels/,
>> > but please ignore that. We just merged a PR which uses cibuildwheel
>> into the main repo.
>> > That should be the target. If cibuildwheel has/gains the ability to
>> produce win-arm64 wheels,
>> > that'd be a good first step towards support.
>> >
>>
>> That would be https://github.com/numpy/numpy/pull/20102 I presume?
>>
>
> Yes indeed.
>
> I expect quite a few projects using numpy will follow your lead here.
>> Thanks!
>>
>
> It's not quite our lead - quite a few other projects have been using
> cibuildwheel already for the last year or longer. It's just continued to
> improve, and it is clear now that cibuildwheel is the best supported way to
> build wheels.
>
> Cheers,
> Ralf
>
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: niyas.s...@linaro.org
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Numpy binary wheels and CI for win/arm64 platform

[Niyas Sait]

> When you say "we could request access to new Volterra machines": how do
> you see that playing out? Who would provide them, and who would maintain
> them? Would they be available to the more general scientific python
> community? Who would pay the bill?

Microsoft, Qualcomm, and Arm are very interested in native support for
Numpy and other scientific packages for Windows on Arm platform.
And we have contacts with them to start the relevant discussion if there is
interest to use those dev boxes (local or remote access)
or Azure VMs for CI/CD purposes.

Hope that answers your questions.

Niyas



On Mon, 6 Jun 2022 at 14:51, Matti Picus  wrote:

> When you say "we could request access to new Volterra machines": how do
> you see that playing out? Who would provide them, and who would maintain
> them? Would they be available to the more general scientific python
> community? Who would pay the bill? If the people supporting this niche
> platform stop doing so, can we drop the wheels? CPython has PEP 11 [0],
> maybe this effort should start by writing a similar NEP for the
> scientific python community.
>
> Matti
>
>
> [0] https://peps.python.org/pep-0011
>
>
> On 6/6/22 13:35, Niyas Sait wrote:
> > Hello,
> >
> > It has been a while since we discussed adding CI/CD support for
> > Windows on Arm. Let me share a few updates and see if we can find a
> > way forward.
> >
> > First of all, Thanks for suggesting cibuildwheel. We have couple of
> > patches in progress to add native [1] and cross-compilation support
> > [2] to cibuildwheel. They could solve the binary wheel creation and
> > releases. However, we may still lack support for CI.
> >
> > I would like to check if there is any interest in using Windows/Arm64
> > VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes )
> > [4] to add CI support for WoA platform.
> >
> > In order to use Azure VMs with GitHub Actions or Azure Pipeline, we
> > must set up self-hosted runners. I am not sure if that would be an
> > acceptable workflow for the numpy community. If that could work we
> > could potentially find ways to fund the required VMs for NumPy project.
> >
> > Also, we could request access to new Volterra machines for Numpy
> > CI/CD. I know it is not easy to maintain them as part of CI/CD
> > workflow, but if any maintainers are interested in using them for
> > CI/CD purposes, we can progress on that discussion.
> >
> > Let me know what you think.
> >
> > Thanks
> > Niyas
> >
> > [1] https://github.com/pypa/cibuildwheel/pull/920
> > [2] https://github.com/pypa/cibuildwheel/pull/1108
> > [3]
> >
> https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines-with-ampere-altra-armbased-processors/
> > [4]
> >
> https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation-experiences-at-scale-with-windows/
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: niyas.s...@linaro.org
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Numpy binary wheels and CI for win/arm64 platform

[Niyas Sait]

>
> Hi Niyas, I'd be interested in remote access to a development machine.
> This will help with NumPy and SciPy; and it would allow testing binaries
> resulting from a cross-compilation step.
> I don't have a preference for a Volterra box or an Azure VM - as long as
> it can be used for development purposes the details of getting access are
> not that important I'd say.


Thanks, Ralf for the interest. I will sort it out and will let you know.

I don't think integration in the main NumPy CI is an option right now,
> however I also don't think it is necessary. A standalone daily or weekly
> scheduled job that would notify interested devs should already be very
> helpful.


FYI, Linaro has set up a downstream CI (Nightly) to track NumPy. See log
for one of the previous runs here -
https://gitlab.com/Linaro/windowsonarm/nightly/-/jobs/2622310764.
Hopefully, that will help.

Niyas



On Tue, 21 Jun 2022 at 13:05, Ralf Gommers  wrote:

>
>
> On Mon, Jun 6, 2022 at 5:08 PM Niyas Sait  wrote:
>
>> > When you say "we could request access to new Volterra machines": how do
>> > you see that playing out? Who would provide them, and who would maintain
>> > them? Would they be available to the more general scientific python
>> > community? Who would pay the bill?
>>
>> Microsoft, Qualcomm, and Arm are very interested in native support for
>> Numpy and other scientific packages for Windows on Arm platform.
>> And we have contacts with them to start the relevant discussion if there
>> is interest to use those dev boxes (local or remote access)
>> or Azure VMs for CI/CD purposes.
>>
>
> Hi Niyas, I'd be interested in remote access to a development machine.
> This will help with NumPy and SciPy; and it would allow testing binaries
> resulting from a cross-compilation step.
>
> I don't have a preference for a Volterra box or an Azure VM - as long as
> it can be used for development purposes the details of getting access are
> not that important I'd say.
>
>
>>
>> Hope that answers your questions.
>>
>> Niyas
>>
>>
>>
>> On Mon, 6 Jun 2022 at 14:51, Matti Picus  wrote:
>>
>>> When you say "we could request access to new Volterra machines": how do
>>> you see that playing out? Who would provide them, and who would maintain
>>> them? Would they be available to the more general scientific python
>>> community? Who would pay the bill? If the people supporting this niche
>>> platform stop doing so, can we drop the wheels? CPython has PEP 11 [0],
>>> maybe this effort should start by writing a similar NEP for the
>>> scientific python community.
>>>
>>> Matti
>>>
>>>
>>> [0] https://peps.python.org/pep-0011
>>>
>>>
>>> On 6/6/22 13:35, Niyas Sait wrote:
>>> > Hello,
>>> >
>>> > It has been a while since we discussed adding CI/CD support for
>>> > Windows on Arm. Let me share a few updates and see if we can find a
>>> > way forward.
>>> >
>>> > First of all, Thanks for suggesting cibuildwheel. We have couple of
>>> > patches in progress to add native [1] and cross-compilation support
>>> > [2] to cibuildwheel. They could solve the binary wheel creation and
>>> > releases. However, we may still lack support for CI.
>>> >
>>> > I would like to check if there is any interest in using Windows/Arm64
>>> > VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes )
>>> > [4] to add CI support for WoA platform.
>>> >
>>> > In order to use Azure VMs with GitHub Actions or Azure Pipeline, we
>>> > must set up self-hosted runners. I am not sure if that would be an
>>> > acceptable workflow for the numpy community. If that could work we
>>> > could potentially find ways to fund the required VMs for NumPy project.
>>>
>>
> I don't think integration in the main NumPy CI is an option right now,
> however I also don't think it is necessary. A standalone daily or weekly
> scheduled job that would notify interested devs should already be very
> helpful.
>
> Cheers,
> Ralf
>
>
>
>> >
>>> > Also, we could request access to new Volterra machines for Numpy
>>> > CI/CD. I know it is not easy to maintain them as part of CI/CD
>>> > workflow, but if any maintainers are interested in using them for
>>> > CI/CD purposes, we can progress on that discussion.
>>> >
&