Re: Upcoming CI changes - transition to VM based CI
On 2025-06-03, Ben Cooksley wrote: > Not sure why we're wanting to monitor for compilation on an Ubuntu LTS > specifically, would have thought that was something best watched by our > Appimage CD builds. Those are currently performed on a SLES image. The purpose is to test that one can do application development on a not too new released Linux without having to build the entire stack from Qt and up. It builds with distribution provided Frameworks and Qt, rather than what our CI does: master or something close to master. The choice of Ubuntu I think is slightly random and could probably be changed to something else from RedHat, Debian or Suse without upsetting any one, but it should still using packages for everything. I guess the selection was Ubuntu due to whatever whoever did it initially felt most comfortable with. /Sune
Re: Upcoming CI changes - transition to VM based CI
On Tue, Jun 3, 2025 at 11:23 PM Sune Vuorela wrote: > On 2025-06-03, Ben Cooksley wrote: > > Not sure why we're wanting to monitor for compilation on an Ubuntu LTS > > specifically, would have thought that was something best watched by our > > Appimage CD builds. Those are currently performed on a SLES image. > > The purpose is to test that one can do application development on a > not too new released Linux without having to build the entire stack > from Qt and up. > I see. Makes sense to a certain extent, certainly within the application context. > It builds with distribution provided Frameworks and Qt, rather than what > our CI does: master or something close to master. > Frameworks would be the last successful build of master at the time the build started. Qt would depend on the job in question. > > The choice of Ubuntu I think is slightly random and could probably be > changed to something else from RedHat, Debian or Suse without upsetting > any one, but it should still using packages for everything. > I guess the selection was Ubuntu due to whatever whoever did it > initially felt most comfortable with. > /Sune > > Cheers, Ben
Re: Upcoming CI changes - transition to VM based CI
On Tue, Jun 3, 2025 at 9:03 AM Albert Astals Cid wrote: > El dilluns, 2 de juny del 2025, a les 13:39:21 (Hora d’estiu d’Europa > central), Ben Cooksley va escriure: > > Hi all, > > > > For some time now we have had a variety of issues with our Docker/Podman > > based CI builds. These have included the lack of GUI test support on > > Windows, periodic crashes on FreeBSD, poor IO performance of Windows > > builds, issues supporting builds for Flatpak and Snaps and inability to > > support either builds or tests where elevated privileges or system > session > > resources are needed. > > > > In addition to this we've had issues where Linux CI builds have the > > capability to trigger OOM events on the CI hosts, which in turn takes out > > Windows and (less often) FreeBSD builders. While this does not occur too > > often, it does happen from time to time and eventually negatively impacts > > the build queue for those platforms. > > > > The need to have dedicated VMs for FreeBSD and Windows on our builders > also > > makes setting up of a CI build node for KDE software a more complicated > and > > time intensive task than it otherwise needs to be (and means that the > > amount of systems to care for increases by 3 for every CI node we add). > > > > While individually relatively minor, together these issues more than > > justify making a significant change to the way we run our CI system - in > > this case transitioning from container based builds to VM based builds. > > > > These builds will still take place on dedicated hardware that we control, > > however instead of taking place within a container (managed by Podman on > > Linux and FreeBSD, or Docker on Windows) they will instead take place > > within a VM using a copy-on-write disk image. > > VM based builds will unfortunately take a little longer to start (it > takes > > ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my > > personal system) however the benefits we gain should more than outweigh > > this small downside. > > > > This has been under development for the past couple of weeks and is now > > reaching the point where the only remaining steps are to get it > integrated > > with the Gitlab CI agent (gitlab-runner) for which prototype code is > > already in place, and complete porting of our images over. Once that > > happens a complete rebuild of all of our builders will be swiftly > > undertaken to transition them completely over to the new VM based > > infrastructure. > > > > Specs wise, at this time it is planned for each spawned standard VM to be > > provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and > > 100GB of disk space (although some of that will be occupied by the system > > image - approximately 10GB for standard Linux builds and ~30GB or so for > > Windows builds). There will be a higher resource tier available for > certain > > builds however that will be on request only and would need to be > justified > > (such as Craft needing to build QtWebEngine). > > > > As launching VMs is not the most efficient approach for all workloads, > > limited support for running Docker containers will be preserved, however > > this support is primarily intended for running linters, sanity checks and > > website builds, and is not intended for running general CI/CD builds. > > > > The tooling used by the CI nodes to run VMs is something that should be > > fairly trivial for people to run on their own local system should they > wish > > to run any of those images (say for FreeBSD or Android), although you > will > > need to setup libvirt yourself (SUSE has very good instructions for this, > > Debian less so as their instructions lack installing the packages needed > to > > provide UEFI and TPM support). The tooling itself was merged this evening > > to sysadmin/ci-images (vm-common/ folder) and can be used with the VM > > images found at https://storage.kde.org/vm-images/ > > > > There is however one downside to this - Qt 5 support. > > > > Over the past few months distributions have been steadily removing > packages > > and other supporting infrastructure needed to keep Qt 5 builds alive. In > > the case of Windows, support for the entire Qt 5 tree has been > unmaintained > > for some time. For FreeBSD and SUSE a significant number of packages have > > been removed - which in the case of SUSE also includes packages needed to > > support the building of KJS. Accordingly, because builds of Frameworks > are > > a first stepping stone to support building anything else, it will not be > > possible for us to produce Qt 5 based VM build images for any of the 3 > > platforms. > > > > We will therefore have to remove Qt 5 support from the CI system with the > > transition to VM based CI. > > From previous discussions I had the impression this was only for things > that > wanted to create packages and not for "want to have CI to compile/run > tests". > > Can you confirm you are proposng a total annihilation of Qt5 sup
Re: Upcoming CI changes - transition to VM based CI
On Tue, Jun 3, 2025 at 10:32 PM Sune Vuorela wrote: > On 2025-06-02, Ben Cooksley wrote: > > As launching VMs is not the most efficient approach for all workloads, > > limited support for running Docker containers will be preserved, however > > this support is primarily intended for running linters, sanity checks and > > website builds, and is not intended for running general CI/CD builds. > > Okular has compile jobs for testing some minimal baselines of supported > systems. I don't know if Okular is the only one, but I at least think it > is a great idea that we have these kind of jobs also. > > (We want people to be able to build Okular preferably on Ubuntu LTS or > later distributions, but due to the timing of last Ubuntu LTS and the > release of KF6, we currently have it on a 'more modern' ubuntu ) > > Okular also have a custom clazy & clang-tidy jobs that relies on > python3 -u ci-utilities/run-ci-build.py to provision a setup. > I'm unsure which of these will have to be re-done, discarded or anything > else in this setup. > Please note that custom jobs while permitted on our CI are something that the project developers are responsible for supporting. I've taken a quick look and can confirm that build_clazy_clang_tidy won't need any changes - it should automatically carry over without any issues and will run in a VM build without changes being required because it inherits from one of the sysadmin provided jobs. clang_format would be covered as a linter job so that should also be fine in theory. build_ubuntu_24_10 will likely be fine, as long as it is not too resource intensive then it should not have any issues. Not sure why we're wanting to monitor for compilation on an Ubuntu LTS specifically, would have thought that was something best watched by our Appimage CD builds. Those are currently performed on a SLES image. > /Sune > > Thanks, Ben
Re: Upcoming CI changes - transition to VM based CI
On 2025-06-02, Ben Cooksley wrote: > As launching VMs is not the most efficient approach for all workloads, > limited support for running Docker containers will be preserved, however > this support is primarily intended for running linters, sanity checks and > website builds, and is not intended for running general CI/CD builds. Okular has compile jobs for testing some minimal baselines of supported systems. I don't know if Okular is the only one, but I at least think it is a great idea that we have these kind of jobs also. (We want people to be able to build Okular preferably on Ubuntu LTS or later distributions, but due to the timing of last Ubuntu LTS and the release of KF6, we currently have it on a 'more modern' ubuntu ) Okular also have a custom clazy & clang-tidy jobs that relies on python3 -u ci-utilities/run-ci-build.py to provision a setup. I'm unsure which of these will have to be re-done, discarded or anything else in this setup. /Sune
Re: Upcoming CI changes - transition to VM based CI
On Montag, 2. Juni 2025 16:42:01 Mitteleuropäische Sommerzeit Christoph Cullmann wrote: > Have we some overview how many things on invent.kde.org will loose the the > CI as they are still Qt 5 only? Looking at things that still have Qt5 jobs in the CI, it's a pretty mixed bag: * Plugins needed to support KF5 based applications (thumbnailers, KIO workers, etc). How much of this can we retire meanwhile? Non-essential things like thumbnailers might be at least something to start with here. * Libraries only needed to support of the above. * Applications with dual-build configurations. I'd suggest to more aggressively remove those. * Unported applications. Here it would be good to know which ones we actually want/need to keep and which are in a zombie state anyway just short of being archived. I don't mind helping with getting the former ported, but I'd very much like to avoid wasting time on the latter. * Exotic stuff with custom jobs (e.g. kid3, alkimia). * KF5 and other libraries supporting any of the above. Attached is the list of all repos with a Qt5 job in their last master branch pipeline run. Regards, Volker KTouch (education/ktouch): flatpak, freebsd14_qt515, suse_tumbleweed_qt515, windows_qt515 Kig (education/kig): craft_macos_arm64, craft_macos_x86_64, flatpak, freebsd14_qt515, suse_tumbleweed_qt515, windows_qt515 Khipu (education/khipu): cppcheck, flatpak, freebsd14_qt515, suse_tumbleweed_qt515 KStars (education/kstars): build, build-and-test-stable, cppcheck, craft_appimage_qt515_x86_64, craft_macos_x86_64, craft_windows_qt515_x86_64, suse_tumbleweed_qt515, windows_qt515 marK (education/mark): freebsd14_qt515, suse_tumbleweed_qt515, windows_qt515 Rocs (education/rocs): flatpak, freebsd14_qt515, suse_tumbleweed_qt515 KDE Webkit (frameworks/kdewebkit): freebsd_qt515, suse_tumbleweed_qt515 KJSEmbed (frameworks/kjsembed): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 Extra CMake Modules (frameworks/extra-cmake-modules): alpine_qt68, android_qt69, freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt515_static, suse_tumbleweed_qt610, suse_tumbleweed_qt68_static, suse_tumbleweed_qt69, windows_qt515, windows_qt515_static, windows_qt515_static, windows_qt68, xml, yaml KEmoticons (frameworks/kemoticons): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 Frameworks API Documentation Tools (frameworks/kapidox): suse_tumbleweed_qt515, suse_tumbleweed_qt62 KDE4 Support (frameworks/kdelibs4support): freebsd_qt515, suse_tumbleweed_qt515 KDesignerPlugin (frameworks/kdesignerplugin): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 Kross (frameworks/kross): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 kxmlrpcclient (frameworks/kxmlrpcclient): freebsd_qt515, suse_tumbleweed_qt515 KInit (frameworks/kinit): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 KMediaPlayer (frameworks/kmediaplayer): freebsd_qt515, suse_tumbleweed_qt515 KHtml (frameworks/khtml): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 KJS (frameworks/kjs): freebsd_qt515, suse_tumbleweed_qt515, windows_qt515 Kolorfill (games/kolorfill): flatpak, freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt68, windows_qt515, windows_qt68 Glaxnimate (graphics/glaxnimate): android_qt68, craft_android_qt68_arm32, craft_android_qt68_arm64, craft_android_qt68_x86_64, craft_appimage_qt515_x86_64, craft_appimage_qt6_x86_64, craft_macos_qt6_arm64, craft_macos_qt6_x86_64, craft_windows_qt515_x86_64, craft_windows_qt6_x86_64, fdroid_apks_qt68, flatpak, freebsd14_qt515, freebsd14_qt68, json-validation, pypi_deploy_gitlab, pypi_deploy_pypiorg, python_build_wheel, reuse, suse_tumbleweed_qt515, suse_tumbleweed_qt68, windows_qt515, windows_qt68 libkdcraw (graphics/libkdcraw): cppcheck, freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt610, suse_tumbleweed_qt69, windows_qt515, windows_qt68 Mobipocket (graphics/kdegraphics-mobipocket): android_qt69, cppcheck, freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt610, suse_tumbleweed_qt69, windows_qt515, windows_qt68, xml, yaml libkexiv2 (graphics/libkexiv2): cppcheck, freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt610, suse_tumbleweed_qt69, windows_qt515, windows_qt68 Thumbnailers (graphics/kdegraphics-thumbnailers): freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt610, suse_tumbleweed_qt69, xml, yaml Kooka (graphics/kooka): cppcheck, freebsd14_qt515, freebsd14_qt68, json-validation, suse_tumbleweed_qt515, suse_tumbleweed_qt69 KDevelop Mercurial Support (kdevelop/kdev-mercurial): freebsd14_qt515, suse_tumbleweed_qt515 KDevelop Krazy2 Support (kdevelop/kdev-krazy2): freebsd14_qt515, suse_tumbleweed_qt515 KDevelop Embedded Support (kdevelop/kdev-embedded): freebsd14_qt515, suse_tumbleweed_qt515 KDevelop XDebug Support (kdev
Re: Upcoming CI changes - transition to VM based CI
Am 03.06.25 um 11:42 schrieb Ben Cooksley: On Tue, Jun 3, 2025 at 9:03 AM Albert Astals Cid wrote: El dilluns, 2 de juny del 2025, a les 13:39:21 (Hora d’estiu d’Europa central), Ben Cooksley va escriure: > Hi all, > > For some time now we have had a variety of issues with our Docker/Podman > based CI builds. These have included the lack of GUI test support on > Windows, periodic crashes on FreeBSD, poor IO performance of Windows > builds, issues supporting builds for Flatpak and Snaps and inability to > support either builds or tests where elevated privileges or system session > resources are needed. > > In addition to this we've had issues where Linux CI builds have the > capability to trigger OOM events on the CI hosts, which in turn takes out > Windows and (less often) FreeBSD builders. While this does not occur too > often, it does happen from time to time and eventually negatively impacts > the build queue for those platforms. > > The need to have dedicated VMs for FreeBSD and Windows on our builders also > makes setting up of a CI build node for KDE software a more complicated and > time intensive task than it otherwise needs to be (and means that the > amount of systems to care for increases by 3 for every CI node we add). > > While individually relatively minor, together these issues more than > justify making a significant change to the way we run our CI system - in > this case transitioning from container based builds to VM based builds. > > These builds will still take place on dedicated hardware that we control, > however instead of taking place within a container (managed by Podman on > Linux and FreeBSD, or Docker on Windows) they will instead take place > within a VM using a copy-on-write disk image. > VM based builds will unfortunately take a little longer to start (it takes > ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my > personal system) however the benefits we gain should more than outweigh > this small downside. > > This has been under development for the past couple of weeks and is now > reaching the point where the only remaining steps are to get it integrated > with the Gitlab CI agent (gitlab-runner) for which prototype code is > already in place, and complete porting of our images over. Once that > happens a complete rebuild of all of our builders will be swiftly > undertaken to transition them completely over to the new VM based > infrastructure. > > Specs wise, at this time it is planned for each spawned standard VM to be > provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and > 100GB of disk space (although some of that will be occupied by the system > image - approximately 10GB for standard Linux builds and ~30GB or so for > Windows builds). There will be a higher resource tier available for certain > builds however that will be on request only and would need to be justified > (such as Craft needing to build QtWebEngine). > > As launching VMs is not the most efficient approach for all workloads, > limited support for running Docker containers will be preserved, however > this support is primarily intended for running linters, sanity checks and > website builds, and is not intended for running general CI/CD builds. > > The tooling used by the CI nodes to run VMs is something that should be > fairly trivial for people to run on their own local system should they wish > to run any of those images (say for FreeBSD or Android), although you will > need to setup libvirt yourself (SUSE has very good instructions for this, > Debian less so as their instructions lack installing the packages needed to > provide UEFI and TPM support). The tooling itself was merged this evening > to sysadmin/ci-images (vm-common/ folder) and can be used with the VM > images found at https://storage.kde.org/vm-images/ > > There is however one downside to this - Qt 5 support. > > Over the past few months distributions have been steadily removing packages > and other supporting infrastructure needed to keep Qt 5 builds alive. In > the case of Windows, support for the entire Qt 5 tree has been unmaintained > for some time. For FreeBSD and SUSE a significant number of packages have > been removed - which in the case of SUSE also includes packages needed to > support the building of KJS. Accordingly, because builds of Frameworks are > a first stepping stone to support building anything else, it will not be > possible for us to produce Qt 5 based VM build images for any of the 3 > platfo
Re: Upcoming CI changes - transition to VM based CI
El dimarts, 3 de juny del 2025, a les 11:42:08 (Hora d’estiu d’Europa central), Ben Cooksley va escriure: > On Tue, Jun 3, 2025 at 9:03 AM Albert Astals Cid wrote: > > El dilluns, 2 de juny del 2025, a les 13:39:21 (Hora d’estiu d’Europa > > > > central), Ben Cooksley va escriure: > > > Hi all, > > > > > > For some time now we have had a variety of issues with our Docker/Podman > > > based CI builds. These have included the lack of GUI test support on > > > Windows, periodic crashes on FreeBSD, poor IO performance of Windows > > > builds, issues supporting builds for Flatpak and Snaps and inability to > > > support either builds or tests where elevated privileges or system > > > > session > > > > > resources are needed. > > > > > > In addition to this we've had issues where Linux CI builds have the > > > capability to trigger OOM events on the CI hosts, which in turn takes > > > out > > > Windows and (less often) FreeBSD builders. While this does not occur too > > > often, it does happen from time to time and eventually negatively > > > impacts > > > the build queue for those platforms. > > > > > > The need to have dedicated VMs for FreeBSD and Windows on our builders > > > > also > > > > > makes setting up of a CI build node for KDE software a more complicated > > > > and > > > > > time intensive task than it otherwise needs to be (and means that the > > > amount of systems to care for increases by 3 for every CI node we add). > > > > > > While individually relatively minor, together these issues more than > > > justify making a significant change to the way we run our CI system - in > > > this case transitioning from container based builds to VM based builds. > > > > > > These builds will still take place on dedicated hardware that we > > > control, > > > however instead of taking place within a container (managed by Podman on > > > Linux and FreeBSD, or Docker on Windows) they will instead take place > > > within a VM using a copy-on-write disk image. > > > VM based builds will unfortunately take a little longer to start (it > > > > takes > > > > > ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my > > > personal system) however the benefits we gain should more than outweigh > > > this small downside. > > > > > > This has been under development for the past couple of weeks and is now > > > reaching the point where the only remaining steps are to get it > > > > integrated > > > > > with the Gitlab CI agent (gitlab-runner) for which prototype code is > > > already in place, and complete porting of our images over. Once that > > > happens a complete rebuild of all of our builders will be swiftly > > > undertaken to transition them completely over to the new VM based > > > infrastructure. > > > > > > Specs wise, at this time it is planned for each spawned standard VM to > > > be > > > provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and > > > 100GB of disk space (although some of that will be occupied by the > > > system > > > image - approximately 10GB for standard Linux builds and ~30GB or so for > > > Windows builds). There will be a higher resource tier available for > > > > certain > > > > > builds however that will be on request only and would need to be > > > > justified > > > > > (such as Craft needing to build QtWebEngine). > > > > > > As launching VMs is not the most efficient approach for all workloads, > > > limited support for running Docker containers will be preserved, however > > > this support is primarily intended for running linters, sanity checks > > > and > > > website builds, and is not intended for running general CI/CD builds. > > > > > > The tooling used by the CI nodes to run VMs is something that should be > > > fairly trivial for people to run on their own local system should they > > > > wish > > > > > to run any of those images (say for FreeBSD or Android), although you > > > > will > > > > > need to setup libvirt yourself (SUSE has very good instructions for > > > this, > > > Debian less so as their instructions lack installing the packages needed > > > > to > > > > > provide UEFI and TPM support). The tooling itself was merged this > > > evening > > > to sysadmin/ci-images (vm-common/ folder) and can be used with the VM > > > images found at https://storage.kde.org/vm-images/ > > > > > > There is however one downside to this - Qt 5 support. > > > > > > Over the past few months distributions have been steadily removing > > > > packages > > > > > and other supporting infrastructure needed to keep Qt 5 builds alive. In > > > the case of Windows, support for the entire Qt 5 tree has been > > > > unmaintained > > > > > for some time. For FreeBSD and SUSE a significant number of packages > > > have > > > been removed - which in the case of SUSE also includes packages needed > > > to > > > support the building of KJS. Accordingly, because builds of Frameworks > > > > are > > > > > a first stepping ston
Re: Upcoming CI changes - transition to VM based CI
On Tuesday, 3 June 2025 11:42:08 CEST Ben Cooksley wrote: > For FreeBSD the story is much the same as SUSE - packages are being removed > as apps upgrade to Qt 6 and the Qt 5 version of libraries becomes surplus > to requirements. >From the FreeBSD packaging side, we have zero interest in Qt5 for KDE-related things, although KF5 is still packaged for, I guess, some of the lagging applications. Frankly if we could drop Qt5 from the FreeBSD ports tree entirely, we'd be quite happy. [ade] signature.asc Description: This is a digitally signed message part.
