https://www.phoronix.com/scan.php?page=news_item&px=FreeDesktop-GitLab-2022-Crash
On Mon, Jun 13, 2022 at 9:40 AM Daniel Stone wrote:
> On Mon, 13 Jun 2022 at 05:17, Peter Hutterer
> wrote:
> > On Sun, Jun 12, 2022 at 05:57:05PM -0700, Jeremy Sequoia wrote:
> > > I was going to spend a little bit of time putting out an update to
> XQuartz
> > > to address a few bugs that I've been meaning to squash, but I'm having
> a bit
> > > of an issue pulling down sources.
> > >
> > > Fetching via ssh://g...@gitlab.freedesktop.org is giving me Permission
> denied
> > > (publickey,keyboard-interactive). I'm not sure if the latter is an
> infra
> > > issue or if the ssh key I have stored in my gitlab account are out of
> date
> > > (it's been about a year since I touched this). Unfortunately, I can't
> seem
> > > to access https://gitlab.freedesktop.org to check as it's constantly
> > > presenting me a 504 Gateway timeout.
> > >
> > > Is anyone else able to pull sources via ssh://
> g...@gitlab.freedesktop.org
> > > right now? Is someone looking into the 504 issue?
> >
> > not an fdo admin but judging by the chatter on #freedesktop: no and yes,
> in
> > that order. seems like the infrastructure is in various stages of
> depositing
> > fecal matter on itself and the fixes are involved enough that the admins
> have
> > to be mentally awake, not merely physically.
>
> Yes, that's what's happening. Our (multi-host-replicated etc) Ceph
> storage setup has entered a degraded mode due to the loss of a couple
> of disks - no data has been lost but the cluster is currently unhappy.
> We're walking through fixing this, but have bumped into some other
> issues since, including a newly-flaky network setup, and changes since
> we last provisioned a new storage host.
>
> We're working through them one by one and will have the service back
> up with all our data intact - hopefully in a matter of hours but we
> have no firm ETA right now.
>
> Cheers,
> Daniel
>
