Hey all. I realize there have been a lot of previous conversations on this over the years, but I'm currently trying to understand some behaviors I'm seeing through testing and what makes the most sense for our use case.
Ultimately I'm dealing with roughly a magnitude of over a billion files. These files range in size of 1KB to at most 20MB. I'd say the average file size though is around 128KB. I've started experimenting with dumping over 150 million 64KB test files into a single bucket and sharding said bucket accordingly. List performance as expected drops dramatically. If I do an ls against the root bucket this ls takes 10 or so minutes, presumably because it has to iterate over the index structure which is ultimately multiple GB of data. Here is what I don't understand. Why would doing an LS within one of those folders return fairly quickly even though it contains million+ objects? How come this isn't causing it to iterate over the entire index structure? What's so special about / compared to the subfolders? Does the folder structure some how impact index structure and retrieval or is it simply a difference in behavior of root? I'm also fundamentally trying to understand if I should even care about list performance. I say this because our application itself doesn't need to rely on lists, we purely need it for operational purposes. We need an index to (rclone) objects in and out of buckets occasionally as part of migration efforts, and additionally we would like to implement multi-site replication for DR purposes. Currently I see no indication that rclone to these folders or even out of the folders has any perceivable impact even with 150 million objects, which I also don't completely understand. How is this possible as it seems to depend on listing to know what needs to be sync'd and/or updated? Do you think replication would also perform efficiently or would it cause impact similar to what I'm seeing by running ls against /? The alternative is to obviously shard this more gracefully into 10k+ buckets, but I'm really trying to avoid this. We also have cloud based infrastructure that I would like to utilize s3 for, it would be completely different hierarchy and logic between the different sites since aws limits bucket to a max of 1,000 per account. _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
