Dear Xen Community,
We regret to inform you that the Xen Project is currently experiencing
unexpected changes due to the sudden shutdown of our colocated (colo) data
center facility by Synoptek. This incident is beyond our control and will
impact the continuity of OSSTest (the gating Xen Project CI loop), which we
heavily rely on as a project. We understand the inconvenience this may
cause to our users and stakeholders, and we are working diligently to
consider different solutions as swiftly as possible.
What Happened?
We received a notification from our colo facility provider Synoptek
indicating the shutdown of their services in Marlborough, where our current
facility is located. As a result, the OSSTest instance hosted at this
facility will likely be disrupted with an anticipated shutdown date of
October 31st, 2024.
Potential Impact
The potential consequence of this shutdown may mean availability of OSSTest
will be impacted, therefore affecting the number of bugs being caught.
Contributors may experience quality issues with code, or face slower
response times as our technical community spends more time resolving
issues. We are aware of the critical nature of the services we provide and
the significant impact this downtime may have.
Our Response
We are in constant communication with the colo facility provider, our IT
contractors, and our Board Members to understand the scope of the problem
and the estimated time for resolution. Additionally, we are exploring
alternative solutions to restore our services as quickly as possible. These
measures include:
Engaging with the Colo Facility Provider (Synoptek): We are working with
the provider to negotiate costs and the likelihood of us moving to another
facility as this was an unforeseen change.
Engaging with third parties: We have initiated discussions with our IT
contractors and third-party providers to find potential alternatives,
including other location facilities and donations of funds to support this
move.
Options we are considering:
-
Move our current hardware to a different co-location.
-
This could be another provider, a member of the Advisory Board, a
University, or a similar organization that offers services for
Open Source
projects.
-
This could include moving all the infrastructure, or just part of it,
depending on costs and space available at the destination.
-
Abandon our hardware, and either re-create OSSTest from scratch at a
different location or switch to a different test system (Gitlab CI).
All options have risks associated.
Further Considerations:
Moving the current hardware is expensive, even to a location provided for
free. It is unknown whether our hardware will survive the move, or if we
will be able to recreate the setup at the new location. Furthermore, our
current IT contractors who help us manage the facility can only serve the
Boston area which means we will need to factor this in when choosing a new
location.
Our technical leadership team has considered abandoning our infrastructure
and creating a new OSSTest instance, however, as it involves acquiring new
hardware there is no guarantee we will be able to set up the new instance
successfully.
An option to mitigate this risk could be to deploy on a metal-as-a-service
provider, but that would require changes to OSSTest.
As a last resort, abandoning OSSTest has been discussed. The project has
no replacement for a test coverage similar to what OSSTest provides. Even
if Gitlab CI expanded test coverage similar to what OSSTest currently
covers, some features are unlikely to be replicated: automatic bisections
on regressions and testing of external projects (Linux kernel, QEMU, OVMF…).
Input from the community
While we work on identifying different options, there are a few steps you
can take to stay informed.
Stay Updated: Keep an eye on our official communication channels for
real-time updates and information about the status of our services.
Get involved: We appreciate you may have feedback and questions, which is
why you are welcome to join the next monthly community call (Thursday 11th
July 2024) to provide your input.
Assistance: We welcome members of our community to provide resources or
suggest ideas that may help our project.
Looking Ahead
Our goal is to identify and implement measures that will minimize the
impact on our project. We will carefully consider the feasibility of
solutions, budget constraints, and the long-term needs of the Xen Project.
We apologize for the inconvenience caused by this unexpected announcement
and appreciate your understanding and patience as we navigate this
challenging situation. Your trust and continued support are invaluable to
us, and we remain committed to developing a powerful and reliable
open-source hypervisor.
Sincerely,
Kelly Choi
Community Manager
Xen Project
