However, we don't quite have that ability... The taskcluster nightly stuff, which is posting to archive.m.o is doing so with a small subset of dedicated machines, which are not overly powerful either in disk space, networking throughput or cpu.
Its also doing so by downloading the artifacts from the taskcluster jobs themselves, and then uploading them to archive.m.o (so is literally copying, which has overhead in terms of ensuring no corruption in the process) The old jobs on archive.m.o/../tinderbox-builds/ were signed, new CI builds are not. Signing is also done via dedicated small subset of machines, so expanding this to every CI job in the taskcluster setup isn't great (for now). The actual definitions/descriptions of the jobs that do the upload to archive.m.o is written in the task graph, depending on signing. To wire this logic up to the CI builds will be at least a full week of 100% dedicated effort, probably more. And that is not accounting for the need to spin up more machines to account for the higher load. This code change would also be relatively risky in terms of taskgraph generation, and need to land on all trees this affects. Storing the artifacts in taskclusters index and archive.m.o (both) is effectively doubling storage costs. We implemented this storage into taskcluster from buildbot over a year ago in order to facilitate the exact "lets give people an overlap of time to switch" -- Things that used archive.m.o/tinderbox-builds that we knew about already switched (e.g. artifact builds) I personally feel the investment to create a stopgap here is not worth it, but feel free to convince chris cooper or chris atlee otherwise. You're also free to copy in any of my points to the bug. ~Justin Wood (Callek) On Tue, Mar 7, 2017 at 1:48 PM, Eric Rahm <er...@mozilla.com> wrote: > Can you add these details to the bug? We should probably take the > conversation on the best way to fix bustage there. > > Given the fallout (read: memory regression tracking is gone) and, as you > noted, we have the ability to continue posting taskcluster builds to > archive.m.o, we should at least continue to post builds to archive.m.o for > at least grace period in order to give AWSY, mozdownload, and others time to > implement a switch to a purely taskcluster-only solution. > > -e > > On Tue, Mar 7, 2017 at 7:43 AM, <jw...@mozilla.com> wrote: >> >> On Monday, March 6, 2017 at 6:08:09 PM UTC-5, Boris Zbarsky wrote: >> > On 3/6/17 5:29 PM, Mike Hommey wrote: >> > > You can get the builds through the taskcluster index. >> > >> > Does that have the same lifetime guarantees as archive.mozilla.org? >> > >> > -Boris >> >> So, to be clear.. >> >> This thread is talking about >> http://archive.mozilla.org/pub/firefox/tinderbox-builds/** >> >> Oldest archive files in there (for autoland) is May 26'th 2016. >> >> On taskcluster's index, there are many ways to get the data (not just by >> buildid) there is date, there is .latest. and there is by revision, but to >> use a date based solution to help answer this, oldest on taskcluster is also >> May 26... >> >> >> https://tools.taskcluster.net/index/artifacts/#gecko.v2.autoland.pushdate.2016.05.26/gecko.v2.autoland.pushdate.2016.05.26 >> >> Releng has traditionally considered the archive.m.o location for >> on-checkin builds to be an implementation detail and not suitable outside of >> our automation. We've also published things to both archive.m.o and >> taskcluster index for a long time now (we even publish osx and windows jobs >> there, so its not like you have to keep both solutions implemented). >> >> Had we realized anyone was using the `tinderbox-builds/` location for >> things before this, I feel we would have notified those projects and the >> community in general more widely when we were to shut them off. >> >> I apologize this broke anyone, but there is a clear path forward. We >> explicitly preserved the archive.m.o uploads for taskcluster nightlies, into >> http://archive.mozilla.org/pub/firefox/nightly/ >> >> ~Justin Wood (Callek) >> _______________________________________________ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
