Hi,
I’m suffering with our large omap object in the cluster since my colleague
updated from jewel to luminous 12.2.8. This warn is still here more than a year
ago.
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'default.rgw.log'
Search the cluster log for 'Large omap object found' for more details.
Based on the articles that I’ve read it is due to the usage logs. On the
weekend I’ve set the rgw_enable_usage_log = false and restarted all the 3 rgw
services.
I trim the logs as some people on the internet suggested, but it is super slow.
I trim the logs month by month like this:
radosgw-admin usage trim --end-date=2019-09-01 --yes-i-really-mean-it
This command took yesterday more than a day to finish :/
I hope at least it is not increasing anymore since I turned it to false.
The luminous default thresholds are:
Size threshold: 1gb
Key threshold: 2000000
Mine is: 7gb and 41246592
2020-08-16 09:21:11.549128 osd.45 osd.45 10.118.191.247:6824/2262114 2 :
cluster [WRN] Large omap object found. Object: 4:b172cd59:usage::usage.26:head
Key count: 41246592 Size (bytes): 7736069974
Based on this command here is my large omap: for i in `ceph pg ls-by-pool
default.rgw.log | tail -n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i
query |grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ":
1"
4.d: 1
I’ve initiated a deep scrub after it and it is still running after 12 hours and
make stuck pgs:
health: HEALTH_ERR
3 stuck requests are blocked > 4096 sec. Implicated osds 45
It stuck for couple of secs, and after go back to normal, but this is
continously.
At the moment no impact yet, but I think it will have soon.
What should I do? What is the best and easiest way to fix my issue?
We have serious impact with the moving of this pg when an osd crash because I
guess it is big and not healthy.
Here are the omap list:
for i in `rados -p default.rgw.log ls`; do echo -n “$i:“; rados -p
default.rgw.log listomapkeys $i |wc -l; done > /tmp/omapkeys
cat /tmp/omapkeys
“obj_delete_at_hint.0000000078:“0
“meta.history:“0
“obj_delete_at_hint.0000000070:“0
“obj_delete_at_hint.0000000104:“0
“obj_delete_at_hint.0000000026:“0
“obj_delete_at_hint.0000000028:“0
“obj_delete_at_hint.0000000040:“0
“obj_delete_at_hint.0000000015:“0
“obj_delete_at_hint.0000000069:“0
“obj_delete_at_hint.0000000095:“0
“obj_delete_at_hint.0000000003:“0
“obj_delete_at_hint.0000000047:“0
“obj_delete_at_hint.0000000035:“0
“obj_delete_at_hint.0000000037:“0
“obj_delete_at_hint.0000000024:“0
“obj_delete_at_hint.0000000000:“0
“obj_delete_at_hint.0000000031:“0
“obj_delete_at_hint.0000000076:“0
“obj_delete_at_hint.0000000113:“0
“obj_delete_at_hint.0000000005:“0
“obj_delete_at_hint.0000000011:“0
“obj_delete_at_hint.0000000058:“0
“obj_delete_at_hint.0000000052:“0
“obj_delete_at_hint.0000000088:“0
“obj_delete_at_hint.0000000080:“0
“obj_delete_at_hint.0000000090:“0
“obj_delete_at_hint.0000000110:“0
“obj_delete_at_hint.0000000096:“0
“obj_delete_at_hint.0000000087:“0
“obj_delete_at_hint.0000000008:“0
“obj_delete_at_hint.0000000006:“0
“obj_delete_at_hint.0000000029:“0
“obj_delete_at_hint.0000000089:“0
“obj_delete_at_hint.0000000022:“0
“bilog.trim:“0
“obj_delete_at_hint.0000000016:“0
“obj_delete_at_hint.0000000041:“0
“obj_delete_at_hint.0000000018:“0
“obj_delete_at_hint.0000000092:“0
“obj_delete_at_hint.0000000014:“0
“obj_delete_at_hint.0000000112:“0
“obj_delete_at_hint.0000000007:“0
“obj_delete_at_hint.0000000021:“0
“obj_delete_at_hint.0000000064:“0
“obj_delete_at_hint.0000000071:“0
“obj_delete_at_hint.0000000074:“0
“obj_delete_at_hint.0000000081:“0
“obj_delete_at_hint.0000000009:“0
“obj_delete_at_hint.0000000121:“0
“obj_delete_at_hint.0000000125:“0
“obj_delete_at_hint.0000000082:“0
“obj_delete_at_hint.0000000105:“0
“obj_delete_at_hint.0000000059:“0
“obj_delete_at_hint.0000000077:“0
“obj_delete_at_hint.0000000032:“0
“obj_delete_at_hint.0000000053:“0
“obj_delete_at_hint.0000000091:“0
“obj_delete_at_hint.0000000065:“0
“obj_delete_at_hint.0000000083:“0
“obj_delete_at_hint.0000000010:“0
“obj_delete_at_hint.0000000045:“0
“obj_delete_at_hint.0000000002:“0
“obj_delete_at_hint.0000000116:“0
“obj_delete_at_hint.0000000034:“0
“obj_delete_at_hint.0000000101:“0
“obj_delete_at_hint.0000000079:“0
“obj_delete_at_hint.0000000049:“0
“obj_delete_at_hint.0000000117:“0
“obj_delete_at_hint.0000000044:“0
“obj_delete_at_hint.0000000066:“0
“obj_delete_at_hint.0000000068:“0
“obj_delete_at_hint.0000000085:“0
“obj_delete_at_hint.0000000073:“0
“obj_delete_at_hint.0000000038:“0
“obj_delete_at_hint.0000000118:“0
“obj_delete_at_hint.0000000036:“0
“obj_delete_at_hint.0000000103:“0
“obj_delete_at_hint.0000000119:“0
“obj_delete_at_hint.0000000098:“0
“obj_delete_at_hint.0000000027:“0
“obj_delete_at_hint.0000000019:“0
“obj_delete_at_hint.0000000039:“0
“obj_delete_at_hint.0000000100:“0
“obj_delete_at_hint.0000000093:“0
“obj_delete_at_hint.0000000004:“0
“obj_delete_at_hint.0000000063:“0
“obj_delete_at_hint.0000000122:“0
“obj_delete_at_hint.0000000057:“0
“obj_delete_at_hint.0000000054:“0
“obj_delete_at_hint.0000000114:“0
“data_log.0:“0
“obj_delete_at_hint.0000000012:“0
“obj_delete_at_hint.0000000084:“0
“obj_delete_at_hint.0000000043:“0
“obj_delete_at_hint.0000000111:“0
“obj_delete_at_hint.0000000048:“0
“obj_delete_at_hint.0000000020:“0
“obj_delete_at_hint.0000000099:“0
“obj_delete_at_hint.0000000056:“0
“obj_delete_at_hint.0000000072:“0
“obj_delete_at_hint.0000000062:“0
“obj_delete_at_hint.0000000115:“0
“obj_delete_at_hint.0000000033:“0
“obj_delete_at_hint.0000000025:“0
“obj_delete_at_hint.0000000123:“0
“obj_delete_at_hint.0000000108:“0
“obj_delete_at_hint.0000000094:“0
“obj_delete_at_hint.0000000060:“0
“obj_delete_at_hint.0000000109:“0
“obj_delete_at_hint.0000000013:“0
“obj_delete_at_hint.0000000051:“0
“obj_delete_at_hint.0000000106:“0
“obj_delete_at_hint.0000000017:“0
“obj_delete_at_hint.0000000023:“0
“obj_delete_at_hint.0000000107:“0
“obj_delete_at_hint.0000000046:“0
“obj_delete_at_hint.0000000067:“0
“obj_delete_at_hint.0000000086:“0
“obj_delete_at_hint.0000000050:“0
“obj_delete_at_hint.0000000102:“0
“obj_delete_at_hint.0000000124:“0
“obj_delete_at_hint.0000000001:“0
“obj_delete_at_hint.0000000126:“0
“obj_delete_at_hint.0000000055:“0
“obj_delete_at_hint.0000000061:“0
“obj_delete_at_hint.0000000097:“0
“obj_delete_at_hint.0000000042:“0
“obj_delete_at_hint.0000000075:“0
“obj_delete_at_hint.0000000120:“0
“obj_delete_at_hint.0000000030:“0
Thank you
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by copyright or
other legal rules. If you have received it by mistake please let us know by
reply email and delete it from your system. It is prohibited to copy this
message or disclose its content to anyone. Any confidentiality or privilege is
not waived or lost by any mistaken delivery or unauthorized disclosure of the
message. All messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests and to
remove potential malware. Electronic messages may be intercepted, amended, lost
or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
