Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Ben Cooksley
Hi all,

For some time now we have had a variety of issues with our Docker/Podman
based CI builds. These have included the lack of GUI test support on
Windows, periodic crashes on FreeBSD, poor IO performance of Windows
builds, issues supporting builds for Flatpak and Snaps and inability to
support either builds or tests where elevated privileges or system session
resources are needed.

In addition to this we've had issues where Linux CI builds have the
capability to trigger OOM events on the CI hosts, which in turn takes out
Windows and (less often) FreeBSD builders. While this does not occur too
often, it does happen from time to time and eventually negatively impacts
the build queue for those platforms.

The need to have dedicated VMs for FreeBSD and Windows on our builders also
makes setting up of a CI build node for KDE software a more complicated and
time intensive task than it otherwise needs to be (and means that the
amount of systems to care for increases by 3 for every CI node we add).

While individually relatively minor, together these issues more than
justify making a significant change to the way we run our CI system - in
this case transitioning from container based builds to VM based builds.

These builds will still take place on dedicated hardware that we control,
however instead of taking place within a container (managed by Podman on
Linux and FreeBSD, or Docker on Windows) they will instead take place
within a VM using a copy-on-write disk image.
VM based builds will unfortunately take a little longer to start (it takes
~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my
personal system) however the benefits we gain should more than outweigh
this small downside.

This has been under development for the past couple of weeks and is now
reaching the point where the only remaining steps are to get it integrated
with the Gitlab CI agent (gitlab-runner) for which prototype code is
already in place, and complete porting of our images over. Once that
happens a complete rebuild of all of our builders will be swiftly
undertaken to transition them completely over to the new VM based
infrastructure.

Specs wise, at this time it is planned for each spawned standard VM to be
provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and
100GB of disk space (although some of that will be occupied by the system
image - approximately 10GB for standard Linux builds and ~30GB or so for
Windows builds). There will be a higher resource tier available for certain
builds however that will be on request only and would need to be justified
(such as Craft needing to build QtWebEngine).

As launching VMs is not the most efficient approach for all workloads,
limited support for running Docker containers will be preserved, however
this support is primarily intended for running linters, sanity checks and
website builds, and is not intended for running general CI/CD builds.

The tooling used by the CI nodes to run VMs is something that should be
fairly trivial for people to run on their own local system should they wish
to run any of those images (say for FreeBSD or Android), although you will
need to setup libvirt yourself (SUSE has very good instructions for this,
Debian less so as their instructions lack installing the packages needed to
provide UEFI and TPM support). The tooling itself was merged this evening
to sysadmin/ci-images (vm-common/ folder) and can be used with the VM
images found at https://storage.kde.org/vm-images/

There is however one downside to this - Qt 5 support.

Over the past few months distributions have been steadily removing packages
and other supporting infrastructure needed to keep Qt 5 builds alive. In
the case of Windows, support for the entire Qt 5 tree has been unmaintained
for some time. For FreeBSD and SUSE a significant number of packages have
been removed - which in the case of SUSE also includes packages needed to
support the building of KJS.  Accordingly, because builds of Frameworks are
a first stepping stone to support building anything else, it will not be
possible for us to produce Qt 5 based VM build images for any of the 3
platforms.

We will therefore have to remove Qt 5 support from the CI system with the
transition to VM based CI.

Please let me know if there are any questions on the above.

Thanks,
Ben


Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Christoph Cullmann
Hi,


On Monday, June 2nd, 2025 at 13:40, Ben Cooksley  wrote:

> Hi all,
> For some time now we have had a variety of issues with our Docker/Podman 
> based CI builds. These have included the lack of GUI test support on Windows, 
> periodic crashes on FreeBSD, poor IO performance of Windows builds, issues 
> supporting builds for Flatpak and Snaps and inability to support either 
> builds or tests where elevated privileges or system session resources are 
> needed.
> 

> In addition to this we've had issues where Linux CI builds have the 
> capability to trigger OOM events on the CI hosts, which in turn takes out 
> Windows and (less often) FreeBSD builders. While this does not occur too 
> often, it does happen from time to time and eventually negatively impacts the 
> build queue for those platforms.
> 

> The need to have dedicated VMs for FreeBSD and Windows on our builders also 
> makes setting up of a CI build node for KDE software a more complicated and 
> time intensive task than it otherwise needs to be (and means that the amount 
> of systems to care for increases by 3 for every CI node we add).
> 

> While individually relatively minor, together these issues more than justify 
> making a significant change to the way we run our CI system - in this case 
> transitioning from container based builds to VM based builds.
> 

> These builds will still take place on dedicated hardware that we control, 
> however instead of taking place within a container (managed by Podman on 
> Linux and FreeBSD, or Docker on Windows) they will instead take place within 
> a VM using a copy-on-write disk image.
> VM based builds will unfortunately take a little longer to start (it takes 
> ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my 
> personal system) however the benefits we gain should more than outweigh this 
> small downside.
> 

> This has been under development for the past couple of weeks and is now 
> reaching the point where the only remaining steps are to get it integrated 
> with the Gitlab CI agent (gitlab-runner) for which prototype code is already 
> in place, and complete porting of our images over. Once that happens a 
> complete rebuild of all of our builders will be swiftly undertaken to 
> transition them completely over to the new VM based infrastructure.
> 

> Specs wise, at this time it is planned for each spawned standard VM to be 
> provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and 100GB 
> of disk space (although some of that will be occupied by the system image - 
> approximately 10GB for standard Linux builds and ~30GB or so for Windows 
> builds). There will be a higher resource tier available for certain builds 
> however that will be on request only and would need to be justified (such as 
> Craft needing to build QtWebEngine).
> 

> As launching VMs is not the most efficient approach for all workloads, 
> limited support for running Docker containers will be preserved, however this 
> support is primarily intended for running linters, sanity checks and website 
> builds, and is not intended for running general CI/CD builds.
> 

> The tooling used by the CI nodes to run VMs is something that should be 
> fairly trivial for people to run on their own local system should they wish 
> to run any of those images (say for FreeBSD or Android), although you will 
> need to setup libvirt yourself (SUSE has very good instructions for this, 
> Debian less so as their instructions lack installing the packages needed to 
> provide UEFI and TPM support). The tooling itself was merged this evening to 
> sysadmin/ci-images (vm-common/ folder) and can be used with the VM images 
> found at https://storage.kde.org/vm-images/
> 

> There is however one downside to this - Qt 5 support.
> 

> Over the past few months distributions have been steadily removing packages 
> and other supporting infrastructure needed to keep Qt 5 builds alive. In the 
> case of Windows, support for the entire Qt 5 tree has been unmaintained for 
> some time. For FreeBSD and SUSE a significant number of packages have been 
> removed - which in the case of SUSE also includes packages needed to support 
> the building of KJS. Accordingly, because builds of Frameworks are a first 
> stepping stone to support building anything else, it will not be possible for 
> us to produce Qt 5 based VM build images for any of the 3 platforms.
> 

> We will therefore have to remove Qt 5 support from the CI system with the 
> transition to VM based CI.
> 

> Please let me know if there are any questions on the above.



Have we some overview how many things on invent.kde.org will loose the the CI 
as they are still Qt 5 only?

Greetings
Christoph

signature.asc
Description: OpenPGP digital signature


Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Benson Muite



On Mon, Jun 2, 2025, at 5:42 PM, Christoph Cullmann wrote:
> Hi,
>
>
> On Monday, June 2nd, 2025 at 13:40, Ben Cooksley  wrote:
>
>> Hi all,
>> For some time now we have had a variety of issues with our Docker/Podman 
>> based CI builds. These have included the lack of GUI test support on 
>> Windows, periodic crashes on FreeBSD, poor IO performance of Windows builds, 
>> issues supporting builds for Flatpak and Snaps and inability to support 
>> either builds or tests where elevated privileges or system session resources 
>> are needed.
>> 
>
>> In addition to this we've had issues where Linux CI builds have the 
>> capability to trigger OOM events on the CI hosts, which in turn takes out 
>> Windows and (less often) FreeBSD builders. While this does not occur too 
>> often, it does happen from time to time and eventually negatively impacts 
>> the build queue for those platforms.
>> 
>
>> The need to have dedicated VMs for FreeBSD and Windows on our builders also 
>> makes setting up of a CI build node for KDE software a more complicated and 
>> time intensive task than it otherwise needs to be (and means that the amount 
>> of systems to care for increases by 3 for every CI node we add).
>> 
>
>> While individually relatively minor, together these issues more than justify 
>> making a significant change to the way we run our CI system - in this case 
>> transitioning from container based builds to VM based builds.
>> 
>
>> These builds will still take place on dedicated hardware that we control, 
>> however instead of taking place within a container (managed by Podman on 
>> Linux and FreeBSD, or Docker on Windows) they will instead take place within 
>> a VM using a copy-on-write disk image.
>> VM based builds will unfortunately take a little longer to start (it takes 
>> ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my 
>> personal system) however the benefits we gain should more than outweigh this 
>> small downside.
>> 
>
>> This has been under development for the past couple of weeks and is now 
>> reaching the point where the only remaining steps are to get it integrated 
>> with the Gitlab CI agent (gitlab-runner) for which prototype code is already 
>> in place, and complete porting of our images over. Once that happens a 
>> complete rebuild of all of our builders will be swiftly undertaken to 
>> transition them completely over to the new VM based infrastructure.
>> 
>
>> Specs wise, at this time it is planned for each spawned standard VM to be 
>> provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and 
>> 100GB of disk space (although some of that will be occupied by the system 
>> image - approximately 10GB for standard Linux builds and ~30GB or so for 
>> Windows builds). There will be a higher resource tier available for certain 
>> builds however that will be on request only and would need to be justified 
>> (such as Craft needing to build QtWebEngine).
>> 
>
>> As launching VMs is not the most efficient approach for all workloads, 
>> limited support for running Docker containers will be preserved, however 
>> this support is primarily intended for running linters, sanity checks and 
>> website builds, and is not intended for running general CI/CD builds.
>> 
>
>> The tooling used by the CI nodes to run VMs is something that should be 
>> fairly trivial for people to run on their own local system should they wish 
>> to run any of those images (say for FreeBSD or Android), although you will 
>> need to setup libvirt yourself (SUSE has very good instructions for this, 
>> Debian less so as their instructions lack installing the packages needed to 
>> provide UEFI and TPM support). The tooling itself was merged this evening to 
>> sysadmin/ci-images (vm-common/ folder) and can be used with the VM images 
>> found at https://storage.kde.org/vm-images/
>> 
>
>> There is however one downside to this - Qt 5 support.
>> 
>
>> Over the past few months distributions have been steadily removing packages 
>> and other supporting infrastructure needed to keep Qt 5 builds alive. In the 
>> case of Windows, support for the entire Qt 5 tree has been unmaintained for 
>> some time. For FreeBSD and SUSE a significant number of packages have been 
>> removed - which in the case of SUSE also includes packages needed to support 
>> the building of KJS. Accordingly, because builds of Frameworks are a first 
>> stepping stone to support building anything else, it will not be possible 
>> for us to produce Qt 5 based VM build images for any of the 3 platforms.
>> 
>
>> We will therefore have to remove Qt 5 support from the CI system with the 
>> transition to VM based CI.
>> 
>
>> Please let me know if there are any questions on the above.
>
>
>
> Have we some overview how many things on invent.kde.org will loose the 
> the CI as they are still Qt 5 only?

Happy to help upgrade some.
Upgrade for marK:
https://invent.kde.org/education/mark/-/merge_requests/10

Examin

invent.k.o/games (Re: Upcoming CI changes - transition to VM based CI)

2025-06-02 Thread Friedrich W. H. Kossebau
Am Montag, 2. Juni 2025, 17:06:15 Mitteleuropäische Sommerzeit schrieb Benson 
Muite:

> Can also help with some of the Games software.

I can speed you (and others) up with looking at the games
(at least those in invent.kde.org/games):

There is one unreleased which needs related work:
Kolorfill, currently Qt5 & Qt6 (remember, changes might need coordination with 
translation managers). Sune, the original author just stated in the chat:
> i will happily approve making it qt6 only

All those released as part of KDE Gear are Qt6-only already since the very 
"Mega 6" release in February last year (first-hand knowledge ;) ).

The other unreleased ones like Atlantik, Arkade, and KMuddy are also already 
Qt6-only. Same with the recently added one, Chessament.

Cheers
Friedrich




Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Thomas Baumgart
On Montag, 2. Juni 2025 17:06:15 CEST Benson Muite wrote:

> 
> On Mon, Jun 2, 2025, at 5:42 PM, Christoph Cullmann wrote:
> > Hi,
> >
> >
> > On Monday, June 2nd, 2025 at 13:40, Ben Cooksley  wrote:
> >
> >> Hi all,
> >> For some time now we have had a variety of issues with our Docker/Podman 
> >> based CI builds. These have included the lack of GUI test support on 
> >> Windows, periodic crashes on FreeBSD, poor IO performance of Windows 
> >> builds, issues supporting builds for Flatpak and Snaps and inability to 
> >> support either builds or tests where elevated privileges or system session 
> >> resources are needed.
> >> 
> >
> >> In addition to this we've had issues where Linux CI builds have the 
> >> capability to trigger OOM events on the CI hosts, which in turn takes out 
> >> Windows and (less often) FreeBSD builders. While this does not occur too 
> >> often, it does happen from time to time and eventually negatively impacts 
> >> the build queue for those platforms.
> >> 
> >
> >> The need to have dedicated VMs for FreeBSD and Windows on our builders 
> >> also makes setting up of a CI build node for KDE software a more 
> >> complicated and time intensive task than it otherwise needs to be (and 
> >> means that the amount of systems to care for increases by 3 for every CI 
> >> node we add).
> >> 
> >
> >> While individually relatively minor, together these issues more than 
> >> justify making a significant change to the way we run our CI system - in 
> >> this case transitioning from container based builds to VM based builds.
> >> 
> >
> >> These builds will still take place on dedicated hardware that we control, 
> >> however instead of taking place within a container (managed by Podman on 
> >> Linux and FreeBSD, or Docker on Windows) they will instead take place 
> >> within a VM using a copy-on-write disk image.
> >> VM based builds will unfortunately take a little longer to start (it takes 
> >> ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my 
> >> personal system) however the benefits we gain should more than outweigh 
> >> this small downside.
> >> 
> >
> >> This has been under development for the past couple of weeks and is now 
> >> reaching the point where the only remaining steps are to get it integrated 
> >> with the Gitlab CI agent (gitlab-runner) for which prototype code is 
> >> already in place, and complete porting of our images over. Once that 
> >> happens a complete rebuild of all of our builders will be swiftly 
> >> undertaken to transition them completely over to the new VM based 
> >> infrastructure.
> >> 
> >
> >> Specs wise, at this time it is planned for each spawned standard VM to be 
> >> provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and 
> >> 100GB of disk space (although some of that will be occupied by the system 
> >> image - approximately 10GB for standard Linux builds and ~30GB or so for 
> >> Windows builds). There will be a higher resource tier available for 
> >> certain builds however that will be on request only and would need to be 
> >> justified (such as Craft needing to build QtWebEngine).
> >> 
> >
> >> As launching VMs is not the most efficient approach for all workloads, 
> >> limited support for running Docker containers will be preserved, however 
> >> this support is primarily intended for running linters, sanity checks and 
> >> website builds, and is not intended for running general CI/CD builds.
> >> 
> >
> >> The tooling used by the CI nodes to run VMs is something that should be 
> >> fairly trivial for people to run on their own local system should they 
> >> wish to run any of those images (say for FreeBSD or Android), although you 
> >> will need to setup libvirt yourself (SUSE has very good instructions for 
> >> this, Debian less so as their instructions lack installing the packages 
> >> needed to provide UEFI and TPM support). The tooling itself was merged 
> >> this evening to sysadmin/ci-images (vm-common/ folder) and can be used 
> >> with the VM images found at https://storage.kde.org/vm-images/
> >> 
> >
> >> There is however one downside to this - Qt 5 support.
> >> 
> >
> >> Over the past few months distributions have been steadily removing 
> >> packages and other supporting infrastructure needed to keep Qt 5 builds 
> >> alive. In the case of Windows, support for the entire Qt 5 tree has been 
> >> unmaintained for some time. For FreeBSD and SUSE a significant number of 
> >> packages have been removed - which in the case of SUSE also includes 
> >> packages needed to support the building of KJS. Accordingly, because 
> >> builds of Frameworks are a first stepping stone to support building 
> >> anything else, it will not be possible for us to produce Qt 5 based VM 
> >> build images for any of the 3 platforms.
> >> 
> >
> >> We will therefore have to remove Qt 5 support from the CI system with the 
> >> transition to VM based CI.
> >> 
> >
> >> Please let me know if the

Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Albert Astals Cid
El dilluns, 2 de juny del 2025, a les 13:39:21 (Hora d’estiu d’Europa 
central), Ben Cooksley va escriure:
> Hi all,
> 
> For some time now we have had a variety of issues with our Docker/Podman
> based CI builds. These have included the lack of GUI test support on
> Windows, periodic crashes on FreeBSD, poor IO performance of Windows
> builds, issues supporting builds for Flatpak and Snaps and inability to
> support either builds or tests where elevated privileges or system session
> resources are needed.
> 
> In addition to this we've had issues where Linux CI builds have the
> capability to trigger OOM events on the CI hosts, which in turn takes out
> Windows and (less often) FreeBSD builders. While this does not occur too
> often, it does happen from time to time and eventually negatively impacts
> the build queue for those platforms.
> 
> The need to have dedicated VMs for FreeBSD and Windows on our builders also
> makes setting up of a CI build node for KDE software a more complicated and
> time intensive task than it otherwise needs to be (and means that the
> amount of systems to care for increases by 3 for every CI node we add).
> 
> While individually relatively minor, together these issues more than
> justify making a significant change to the way we run our CI system - in
> this case transitioning from container based builds to VM based builds.
> 
> These builds will still take place on dedicated hardware that we control,
> however instead of taking place within a container (managed by Podman on
> Linux and FreeBSD, or Docker on Windows) they will instead take place
> within a VM using a copy-on-write disk image.
> VM based builds will unfortunately take a little longer to start (it takes
> ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my
> personal system) however the benefits we gain should more than outweigh
> this small downside.
> 
> This has been under development for the past couple of weeks and is now
> reaching the point where the only remaining steps are to get it integrated
> with the Gitlab CI agent (gitlab-runner) for which prototype code is
> already in place, and complete porting of our images over. Once that
> happens a complete rebuild of all of our builders will be swiftly
> undertaken to transition them completely over to the new VM based
> infrastructure.
> 
> Specs wise, at this time it is planned for each spawned standard VM to be
> provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and
> 100GB of disk space (although some of that will be occupied by the system
> image - approximately 10GB for standard Linux builds and ~30GB or so for
> Windows builds). There will be a higher resource tier available for certain
> builds however that will be on request only and would need to be justified
> (such as Craft needing to build QtWebEngine).
> 
> As launching VMs is not the most efficient approach for all workloads,
> limited support for running Docker containers will be preserved, however
> this support is primarily intended for running linters, sanity checks and
> website builds, and is not intended for running general CI/CD builds.
> 
> The tooling used by the CI nodes to run VMs is something that should be
> fairly trivial for people to run on their own local system should they wish
> to run any of those images (say for FreeBSD or Android), although you will
> need to setup libvirt yourself (SUSE has very good instructions for this,
> Debian less so as their instructions lack installing the packages needed to
> provide UEFI and TPM support). The tooling itself was merged this evening
> to sysadmin/ci-images (vm-common/ folder) and can be used with the VM
> images found at https://storage.kde.org/vm-images/
> 
> There is however one downside to this - Qt 5 support.
> 
> Over the past few months distributions have been steadily removing packages
> and other supporting infrastructure needed to keep Qt 5 builds alive. In
> the case of Windows, support for the entire Qt 5 tree has been unmaintained
> for some time. For FreeBSD and SUSE a significant number of packages have
> been removed - which in the case of SUSE also includes packages needed to
> support the building of KJS.  Accordingly, because builds of Frameworks are
> a first stepping stone to support building anything else, it will not be
> possible for us to produce Qt 5 based VM build images for any of the 3
> platforms.
> 
> We will therefore have to remove Qt 5 support from the CI system with the
> transition to VM based CI.

From previous discussions I had the impression this was only for things that 
wanted to create packages and not for "want to have CI to compile/run tests".

Can you confirm you are proposng a total annihilation of Qt5 support in our 
CI?

Cheers,
  Albert

> 
> Please let me know if there are any questions on the above.
> 
> Thanks,
> Ben






Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Albert Astals Cid
El dilluns, 2 de juny del 2025, a les 16:42:01 (Hora d’estiu d’Europa central), 
Christoph Cullmann va escriure:
> Hi,
> 
> On Monday, June 2nd, 2025 at 13:40, Ben Cooksley  wrote:
> > Hi all,
> > For some time now we have had a variety of issues with our Docker/Podman
> > based CI builds. These have included the lack of GUI test support on
> > Windows, periodic crashes on FreeBSD, poor IO performance of Windows
> > builds, issues supporting builds for Flatpak and Snaps and inability to
> > support either builds or tests where elevated privileges or system
> > session resources are needed.
> > 
> > 
> > In addition to this we've had issues where Linux CI builds have the
> > capability to trigger OOM events on the CI hosts, which in turn takes out
> > Windows and (less often) FreeBSD builders. While this does not occur too
> > often, it does happen from time to time and eventually negatively impacts
> > the build queue for those platforms.
> > 
> > 
> > The need to have dedicated VMs for FreeBSD and Windows on our builders
> > also makes setting up of a CI build node for KDE software a more
> > complicated and time intensive task than it otherwise needs to be (and
> > means that the amount of systems to care for increases by 3 for every CI
> > node we add).
> > 
> > 
> > While individually relatively minor, together these issues more than
> > justify making a significant change to the way we run our CI system - in
> > this case transitioning from container based builds to VM based builds.
> > 
> > 
> > These builds will still take place on dedicated hardware that we control,
> > however instead of taking place within a container (managed by Podman on
> > Linux and FreeBSD, or Docker on Windows) they will instead take place
> > within a VM using a copy-on-write disk image. VM based builds will
> > unfortunately take a little longer to start (it takes ~10 seconds for a
> > VM from any of Linux, FreeBSD or Windows to boot on my personal system)
> > however the benefits we gain should more than outweigh this small
> > downside.
> > 
> > 
> > This has been under development for the past couple of weeks and is now
> > reaching the point where the only remaining steps are to get it
> > integrated with the Gitlab CI agent (gitlab-runner) for which prototype
> > code is already in place, and complete porting of our images over. Once
> > that happens a complete rebuild of all of our builders will be swiftly
> > undertaken to transition them completely over to the new VM based
> > infrastructure.
> > 
> > 
> > Specs wise, at this time it is planned for each spawned standard VM to be
> > provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and
> > 100GB of disk space (although some of that will be occupied by the system
> > image - approximately 10GB for standard Linux builds and ~30GB or so for
> > Windows builds). There will be a higher resource tier available for
> > certain builds however that will be on request only and would need to be
> > justified (such as Craft needing to build QtWebEngine).
> > 
> > 
> > As launching VMs is not the most efficient approach for all workloads,
> > limited support for running Docker containers will be preserved, however
> > this support is primarily intended for running linters, sanity checks and
> > website builds, and is not intended for running general CI/CD builds.
> > 
> > 
> > The tooling used by the CI nodes to run VMs is something that should be
> > fairly trivial for people to run on their own local system should they
> > wish to run any of those images (say for FreeBSD or Android), although
> > you will need to setup libvirt yourself (SUSE has very good instructions
> > for this, Debian less so as their instructions lack installing the
> > packages needed to provide UEFI and TPM support). The tooling itself was
> > merged this evening to sysadmin/ci-images (vm-common/ folder) and can be
> > used with the VM images found at https://storage.kde.org/vm-images/
> > 
> > 
> > There is however one downside to this - Qt 5 support.
> > 
> > 
> > Over the past few months distributions have been steadily removing
> > packages and other supporting infrastructure needed to keep Qt 5 builds
> > alive. In the case of Windows, support for the entire Qt 5 tree has been
> > unmaintained for some time. For FreeBSD and SUSE a significant number of
> > packages have been removed - which in the case of SUSE also includes
> > packages needed to support the building of KJS. Accordingly, because
> > builds of Frameworks are a first stepping stone to support building
> > anything else, it will not be possible for us to produce Qt 5 based VM
> > build images for any of the 3 platforms.
> > 
> > 
> > We will therefore have to remove Qt 5 support from the CI system with the
> > transition to VM based CI.
> > 
> > 
> > Please let me know if there are any questions on the above.
> 
> Have we some overview how many things on invent.kde.org will loose the the
> CI as they are still Qt 5

Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Ben Cooksley
On Tue, Jun 3, 2025 at 3:54 AM Thomas Baumgart  wrote:

> On Montag, 2. Juni 2025 17:06:15 CEST Benson Muite wrote:
>
> >
> > On Mon, Jun 2, 2025, at 5:42 PM, Christoph Cullmann wrote:
> > > Hi,
> > >
> > >
> > > On Monday, June 2nd, 2025 at 13:40, Ben Cooksley 
> wrote:
> > >
> > >> Hi all,
> > >> For some time now we have had a variety of issues with our
> Docker/Podman based CI builds. These have included the lack of GUI test
> support on Windows, periodic crashes on FreeBSD, poor IO performance of
> Windows builds, issues supporting builds for Flatpak and Snaps and
> inability to support either builds or tests where elevated privileges or
> system session resources are needed.
> > >>
> > >
> > >> In addition to this we've had issues where Linux CI builds have the
> capability to trigger OOM events on the CI hosts, which in turn takes out
> Windows and (less often) FreeBSD builders. While this does not occur too
> often, it does happen from time to time and eventually negatively impacts
> the build queue for those platforms.
> > >>
> > >
> > >> The need to have dedicated VMs for FreeBSD and Windows on our
> builders also makes setting up of a CI build node for KDE software a more
> complicated and time intensive task than it otherwise needs to be (and
> means that the amount of systems to care for increases by 3 for every CI
> node we add).
> > >>
> > >
> > >> While individually relatively minor, together these issues more than
> justify making a significant change to the way we run our CI system - in
> this case transitioning from container based builds to VM based builds.
> > >>
> > >
> > >> These builds will still take place on dedicated hardware that we
> control, however instead of taking place within a container (managed by
> Podman on Linux and FreeBSD, or Docker on Windows) they will instead take
> place within a VM using a copy-on-write disk image.
> > >> VM based builds will unfortunately take a little longer to start (it
> takes ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on
> my personal system) however the benefits we gain should more than outweigh
> this small downside.
> > >>
> > >
> > >> This has been under development for the past couple of weeks and is
> now reaching the point where the only remaining steps are to get it
> integrated with the Gitlab CI agent (gitlab-runner) for which prototype
> code is already in place, and complete porting of our images over. Once
> that happens a complete rebuild of all of our builders will be swiftly
> undertaken to transition them completely over to the new VM based
> infrastructure.
> > >>
> > >
> > >> Specs wise, at this time it is planned for each spawned standard VM
> to be provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM
> and 100GB of disk space (although some of that will be occupied by the
> system image - approximately 10GB for standard Linux builds and ~30GB or so
> for Windows builds). There will be a higher resource tier available for
> certain builds however that will be on request only and would need to be
> justified (such as Craft needing to build QtWebEngine).
> > >>
> > >
> > >> As launching VMs is not the most efficient approach for all
> workloads, limited support for running Docker containers will be preserved,
> however this support is primarily intended for running linters, sanity
> checks and website builds, and is not intended for running general CI/CD
> builds.
> > >>
> > >
> > >> The tooling used by the CI nodes to run VMs is something that should
> be fairly trivial for people to run on their own local system should they
> wish to run any of those images (say for FreeBSD or Android), although you
> will need to setup libvirt yourself (SUSE has very good instructions for
> this, Debian less so as their instructions lack installing the packages
> needed to provide UEFI and TPM support). The tooling itself was merged this
> evening to sysadmin/ci-images (vm-common/ folder) and can be used with the
> VM images found at https://storage.kde.org/vm-images/
> > >>
> > >
> > >> There is however one downside to this - Qt 5 support.
> > >>
> > >
> > >> Over the past few months distributions have been steadily removing
> packages and other supporting infrastructure needed to keep Qt 5 builds
> alive. In the case of Windows, support for the entire Qt 5 tree has been
> unmaintained for some time. For FreeBSD and SUSE a significant number of
> packages have been removed - which in the case of SUSE also includes
> packages needed to support the building of KJS. Accordingly, because builds
> of Frameworks are a first stepping stone to support building anything else,
> it will not be possible for us to produce Qt 5 based VM build images for
> any of the 3 platforms.
> > >>
> > >
> > >> We will therefore have to remove Qt 5 support from the CI system with
> the transition to VM based CI.
> > >>
> > >
> > >> Please let me know if there are any questions on the above.
> > >
> > >
> > >
> > > Have we

Re: Upcoming CI changes - transition to VM based CI

2025-06-02 Thread Friedrich W. H. Kossebau
Am Montag, 2. Juni 2025, 16:42:01 Mitteleuropäische Sommerzeit schrieb 
Christoph Cullmann:
> On Monday, June 2nd, 2025 at 13:40, Ben Cooksley  wrote:

> > We will therefore have to remove Qt 5 support from the CI system with the
> > transition to VM based CI.
> > 
> > 
> > Please let me know if there are any questions on the above.
> 
> Have we some overview how many things on invent.kde.org will loose the the
> CI as they are still Qt 5 only?

Okteta will suffer, but hopefully not much:

Okteta sadly still is Qt5-only both in master and current release branch 0.26. 
Otherwise there is only my Qt6 work dump commit branch, still stuck on the one 
remaining but big blocker, replacing the QtScript usage, moving slowly as 
available time permits.

Right now for the Qt5-only release branch 0.26 another bug fix/translations 
update release is planned for Friday this very week, given some more minor 
fixes accumulated again as well as some translation tunings.
No further releases planned (unless a serious bug appears to users before any 
Qt6-based release happens).
So loss of CI (Linux, FreeBSD) not that grave in the release branch (post June 
6th).

Qt5-only master branch also sees no development work for now, as all focus is 
on the QtScript replacement, which is currently done externally in separately 
research/experimental projects/repos, not yet on KDE servers.
So loss of CI (Linux, FreeBSD) also not that grave on master branch.

Besides, using openSUSE TW myself on my development system, that CI job mainly 
doubles my local experience.

The Qt6 work dump commit branch is Qt6-only.
So not affected.

Cheers
Friedrich