Hi Thomas, On 09/06/24 at 12:17 +0200, Thomas Goirand wrote: > Hi there! > > I'm hereby cc-ing our DPL, to get him involved an eventual storage cluster > purchase for the project. > > I have been mentioning such an object storage driver, so we could use > OpenStack swift for snapshot.d.o for years. I am happy that it finally > brings traction, and that Lucas is implementing this. Thanks Lucas. > > However, it is disappointing to see it moving toward an s3 implementation, > which is a protocol from a closed-source service. I already wrote multiple > times that my company (Infomaniak) was willing to sponsor storage space on > Swift for it. > > FYI, we currently manage more than 110 PB of storage over 7500+ HDD and > growing, so I am not scared at all about storage space. Some clusters we > manage are around 40PB, with billions of files. > > Though I do not envision *any* sponsor to provide the storage space, but > rather, Debian maintaining its own storage cluster. To give you a rough idea > of what this would represent, let me give you some idea of what type of > hardware involved, and it pricing. > > I would currently recommend this type of 2U server: > https://www.aicipc.com/en/productdetail/51224 > > They provide 24 HDD storage, plus 2x SSD for the system. Equipped with a > decent amount of RAM (128 GB) and a CPU, the cost is around 4000 EUR per > server without the HDDs. Currently, 22TB Seagate HDDs are at around 350 EUR > per piece. So such a server fully equipped with HDD would be at around 12000 > EUR per server. If we want 6 of them (which is IMO the bare minimum for > redundancy, as each file is stored 3 times), we're talking of around 75000 > EUR, plus 3 smaller servers to act as auth server (ie: Keytsone), at let's > say 4000 each (which is average price for a decent server with 128 GB of RAM > and 2x SSD system, plus 32 cores CPU), we would end up spending around > 90kEUR for such a storage cluster. This would provide 1 PB of redundant (ie: > copied 3 times) storage space. > > This would need 15U of rack space, plus an eventual switch. > > Though if we want to be safe, we could purchase at least one spare node and > a few HDDs. > > So all together, we're looking at a 100kEUR spending. Note that this type of > swift cluster could also be used for artifact storage for Salsa (gitlab has > a swift backend storage driver).
It looks like you see the work on object-storage backend as a procurement/infrastructure question. I don't think that this is the main issue. Based on what I've done so far (and I still plan to continue working on this, but I have limited time for Debian nowadays), the code also needs deep changes because, if you want an object-storage-based backend to perform adequately, you need to more parallelism for backend-related operations. This is true whether the storage service is AWS S3, OpenStack Swift, Azure Blob Storage, or Ceph Object Storage, or whatever. If you increase the latency between the importer/indexer and the storage service, you need parallelism to hide it and stay with a bandwidth-bound problem. To work on this, you need an object storage backend, but I suspect that once it works with one of them, porting it to another one will be trivial, as the S3-specific bits are really minimal. (and Swift is S3-compatible anyway) Help is welcomed -- my code is at https://salsa.debian.org/lucas/snapshot/-/commits/s3snap/?ref_type=heads Typically a good way to test this is to try to import a small archive (e.g. debian-security with one architecture only) and see if you can get an import time on object storage that is similar to the one on file-based storage. Lucas
