Dear Eugen
On 19/06/2025 15.24, Eugen Block wrote:
Just to clarify, you had a warning for 1 large omap,
Not quite, I started out with more than one. Other buckets had large
omaps, too. Just not to the same extent.
then you activated
automatic resharding, which created 167 shards.
Exactly.
Then you ran a deep-
scrub and it now shows lots of warnings?
Yes, the number of warnings went up. From 28 to now 194. Which is 166
more, so the number of new shards...
Can you confirm that the PGs
really had finished deep-scrubbing? If you issue a manual deep-scrub,
that just adds those PGs into the queue, it doesn't mean they will be
scrubbed immediately. And depending on the PG size, the scrubbing can
take some time, too.
I did check the OSD logs at the time, so yes, I can confirm that they
all went through.
Sincerely
Niklaus Hofer
Zitat von Niklaus Hofer <[email protected]>:
Dear Eugen
Thank you for your input. What you say is true, of course. I've read
that some place too. I already ran a deep-scrub on all the involved
PGs and that's when the number of warnings increased drastically. I
assume that before I just had one positively huge omap object for that
pool. Now I have 167 omap objects that are not quite as big, but still
too large.
Sincerely
Niklaus Hofer
On 19/06/2025 14.48, Eugen Block wrote:
Hi,
the warnings about large omap objects are reported when deep-scrubs
happen. So if you resharded the bucket (or Ceph did that for you),
you'll either have to wait for the deep-scrub schedule to scrub the
affected PGs, or you issue a manual deep-scrub on that PG or the
entire pool.
Regards,
Eugen
Zitat von Niklaus Hofer <[email protected]>:
Dear all
We are running Ceph Pacific (16.2.15) with RadosGW and have been
getting "large omap objects" health warnings on the RadosGW index
pool. Indeed we had one bucket in particular that was positively
huge with 8'127'198 objects that had just a single shard. But we
have been seeing the message on some other buckets, too.
Eventually, we activated automatic resharding
(rgw_dynamic_resharding = true) and indeed this bucket was resharded
to now 167 shards. However, I am now getting even more large omap
object warnings. On that same bucket, too. The other buckets have
not been resharded at all. They are not in the queue, either:
| radosgw-admin reshard list
[]
| grep 'Large omap object found' /var/log/ceph/ceph* | grep 'PG: ' |
cut -d: -f 10 | cut -d. -f 4-5 | sort | uniq -c
2 7936686773.215
12 7937604172.149
10 7955243979.1209
9 7955243979.2480
13 7955243979.2481
12 7968198782.110
13 7968913553.67
11 7968913553.68
10 7968913553.69
11 7981210604.1
74 7981624399.1
217 7988881492.1
| radosgw-admin metadata list --metadata-key bucket.instance | grep
-i 7988881492
"<bucket1_name>:<pool_name>.7988881492.1",
| radosgw-admin bucket stats --bucket <bucket1_name>
{
"bucket": "<bucket1_name>",
"num_shards": 167,
[...]
"usage": {
"rgw.main": {
"size": 9669928611955,
"size_actual": 9692804734976,
"size_utilized": 9669928611955,
"size_kb": 9443289661,
"size_kb_actual": 9465629624,
"size_kb_utilized": 9443289661,
"num_objects": 8134437
}
},
Let's check another one, too:
| radosgw-admin metadata list --metadata-key bucket.instance | grep
-i 7968198782.110
"<bucket2_name>:<pool_name>.7968198782.110",
| radosgw-admin bucket stats --bucket <bucket2_name>
[...]
"num_objects": 38690
[...]
According to the documentation in [0], buckets are resharded at a
threshold of 100'000 objects per shard. For both of these, that
applies nicely, so it makes sense that they are not getting
resharded any further.
But why then am I getting these warnings?
Reading the documentation in [1], I can see that warnings are
printed at 200'000 entries per omap object. Can I assume that one
object in an RGW bucket means 1 entry in an omap object? Or is that
a missconception?
Now, here is my working theory. Please let me know if that has any
merit or if I'm completely off:
The affected buckets have versioning activated. Plus object locking
too. They get used by a backup software (Kopia) that uses these
features to provide ransomware protection. So my thinking is that
maybe with versioning active, each object in a bucket could result
in multiple omap entries, maybe one per version or something?
If that is the case, then maybe I should reduce
`rgw_max_objs_per_shard` from 100'000 to something like 10'000 to
have the buckets resharded more aggressively?
But then again, that assumes a lot. For example, that assumes that
the num_objects counter in the bucket stats does not count up on
versioned objects. So my assumption could be completely whack.
What do you think? What can I do to get rid of the large omap
objects? Is more resharding going to help? What else could I check?
Sincerely
Niklaus Hofer
Links:
[0] https://docs.ceph.com/en/latest/radosgw/dynamicresharding/
#confval-rgw_max_objs_per_shard
[1] https://docs.ceph.com/en/latest/rados/operations/health-checks/
#large-omap-objects
--
stepping stone AG
Wasserwerkgasse 7
CH-3011 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
[email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
Managed Kubernetes (Container as a Service) kostenlos testen!
Jetzt bestellen und 500 Franken Startguthaben sichern:
https://www.stoney-cloud.com/caas/
Arbeitszeiten: Dienstag bis Freitag
stepping stone AG
Wasserwerkgasse 7
CH-3011 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
[email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
Managed Kubernetes (Container as a Service) kostenlos testen!
Jetzt bestellen und 500 Franken Startguthaben sichern:
https://www.stoney-cloud.com/caas/
Arbeitszeiten: Dienstag bis Freitag
stepping stone AG
Wasserwerkgasse 7
CH-3011 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
[email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
