Thanks for the reply. I’ll give that a try, I wasn’t using the balancer.
> On Jan 2, 2023, at 1:55 AM, Pavin Joseph <[email protected]> wrote: > > Hi Jeff, > > Might be worth checking the balancer [0] status, also you probably want to > use upmap mode [1] if possible. > > [0]: https://docs.ceph.com/en/latest/rados/operations/balancer/#status > [1]: https://docs.ceph.com/en/latest/rados/operations/balancer/#modes > > Kind regards, > Pavin Joseph. > > On 02-Jan-23 12:04 AM, Jeffrey Turmelle wrote: >> Hi Everyone, >> My Nautilus cluster of 6 nodes, 180 OSDs, is having a weird issue I don’t >> know how to troubleshoot. >> I started receiving health warning issues, and the number of PGS not >> deep-scrubbed in time has been increasing. >> # ceph health detail >> HEALTH_WARN 3013 pgs not scrubbed in time >> PG_NOT_SCRUBBED 3013 pgs not scrubbed in time >> pg 1.e99 not scrubbed since 2022-12-21 11:01:10.403525 >> pg 1.e94 not scrubbed since 2022-12-18 06:26:14.086410 >> pg 3.e91 not scrubbed since 2022-12-17 03:00:25.104174 >> pg 1.e90 not scrubbed since 2022-12-18 03:31:44.747218 >> pg 1.e8e not scrubbed since 2022-12-21 12:04:17.111762 >> pg 1.e89 not scrubbed since 2022-12-18 07:20:13.328540 >> ... >> 2963 more pgs… >> Strangely, ceph pg dump gives shows every other PG with 0 objects. An >> attempt to perform a deep scrub (or scrub) on one of these PGs does nothing. >> The cluster appears to be running fine, but obviously there’s an issue. >> What should my next steps be to troubleshoot ? >> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES >> OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP >> VERSION REPORTED UP UP_PRIMARY ACTING >> ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP >> LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN >> 3.e9b 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 22:49:07.629579 0'0 23686:19820 [28,79] 28 >> [28,79] 28 0'0 2022-12-31 22:49:07.629508 >> 0'0 2022-12-31 22:49:07.629508 0 >> 1.e99 60594 0 0 0 0 177433523272 >> 0 0 3046 3046 active+clean 2022-12-21 >> 14:35:08.175858 23686'268137 23686:1732399 [178,115] 178 >> [178,115] 178 23675'267613 2022-12-21 11:01:10.403525 >> 23675'267613 2022-12-21 11:01:10.403525 0 >> 3.e9a 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 09:16:48.644619 0'0 23686:22855 [51,140] 51 >> [51,140] 51 0'0 2022-12-31 09:16:48.644568 >> 0'0 2022-12-30 02:35:23.367344 0 >> 1.e98 59962 0 0 0 0 177218669411 >> 0 0 3035 3035 active+clean 2022-12-28 >> 14:14:49.908560 23686'265576 23686:1357499 [92,86] 92 >> [92,86] 92 23686'265445 2022-12-28 14:14:49.908522 >> 23686'265445 2022-12-28 14:14:49.908522 0 >> 3.e95 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 06:09:39.442932 0'0 23686:22757 [48,83] 48 >> [48,83] 48 0'0 2022-12-31 06:09:39.442879 >> 0'0 2022-12-18 09:33:47.892142 0 >> 1.e97 60062 0 0 0 0 176721095235 >> 0 0 3050 3050 active+clean 2022-12-31 >> 21:19:33.758473 23686'267934 23686:1514273 [137,123] 137 >> [137,123] 137 23686'267916 2022-12-31 21:19:33.758417 >> 23686'267713 2022-12-27 19:16:27.025326 0 >> 3.e94 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 10:00:38.864773 0'0 23686:18478 [101,1] 101 >> [101,1] 101 0'0 2022-12-31 10:00:38.864730 >> 0'0 2022-12-28 22:28:13.790168 0 >> 1.e96 59753 0 0 0 0 175411602155 >> 0 0 3083 3083 active+clean 2022-12-28 >> 14:13:32.186265 23686'264255 23686:1676359 [54,170] 54 >> [54,170] 54 23686'264120 2022-12-28 14:13:32.186220 >> 23686'264120 2022-12-28 14:13:32.186220 0 >> 3.e97 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2023-01-01 >> 11:11:13.202546 0'0 23686:7175 [57,112] 57 >> [57,112] 57 0'0 2023-01-01 11:11:13.202473 >> 0'0 2023-01-01 11:11:13.202473 0 >> 1.e95 60091 0 0 0 0 176012659050 >> 0 0 3033 3033 active+clean 2022-12-31 >> 08:17:27.280405 23686'267681 23686:1441943 [135,133] 135 >> [135,133] 135 23686'267648 2022-12-31 08:17:27.280354 >> 23686'267555 2022-12-28 15:45:47.863185 0 >> 3.e96 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2023-01-01 >> 10:57:01.640503 0'0 23686:18395 [116,8] 116 >> [116,8] 116 0'0 2023-01-01 10:57:01.637962 >> 0'0 2023-01-01 10:57:01.637962 0 >> 1.e94 59926 0 0 0 0 176764011646 >> 0 0 3099 3099 active+clean+scrubbing+deep 2023-01-01 >> 11:11:14.074902 23686'265734 23686:1438653 [40,56] 40 >> [40,56] 40 23662'264884 2022-12-18 06:26:14.086410 >> 23662'264714 2022-12-13 08:04:24.265324 0 >> 3.e91 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-17 >> 03:00:25.104235 0'0 23685:1849 [77,31] 77 >> [77,31] 77 0'0 2022-12-17 03:00:25.104174 >> 0'0 2022-12-15 21:38:28.004594 0 >> 1.e93 60193 0 0 0 0 177054652617 >> 0 0 3054 3054 active+clean 2023-01-01 >> 02:21:23.871342 23686'268286 23686:1470308 [72,169] 72 >> [72,169] 72 23686'268274 2023-01-01 02:21:23.871272 >> 23686'268169 2022-12-29 17:21:15.256027 0 >> 3.e90 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 19:35:50.398721 0'0 23686:11647 [125,45] 125 >> [125,45] 125 0'0 2022-12-31 19:35:50.398635 >> 0'0 2022-12-31 19:35:50.398635 0 >> 1.e92 60342 0 0 0 0 177585802610 >> 0 0 3098 3098 active+clean 2022-12-31 >> 11:20:33.735990 23686'265365 23686:1493691 [47,91] 47 >> [47,91] 47 23686'265321 2022-12-31 11:20:33.735945 >> 23686'265321 2022-12-31 11:20:33.735945 0 >> 3.e93 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 18:34:30.921757 0'0 23686:19914 [28,101] 28 >> [28,101] 28 0'0 2022-12-31 18:34:30.921695 >> 0'0 2022-12-20 01:15:26.329060 0 >> 1.e91 59922 0 0 0 0 176178294337 >> 0 0 3078 3078 active+clean 2022-12-31 >> 16:23:13.129135 23686'266109 23686:1498886 [72,157] 72 >> [72,157] 72 23686'266084 2022-12-31 16:23:13.129091 >> 23686'266084 2022-12-31 16:23:13.129091 0 >> 3.e92 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2023-01-01 >> 03:21:41.819236 0'0 23686:1548 [73,68] 73 >> [73,68] 73 0'0 2023-01-01 03:21:41.816785 >> 0'0 2022-12-27 03:21:17.768390 0 >> 1.e90 59885 0 0 0 0 176295573088 >> 0 0 3059 3059 active+clean+scrubbing+deep 2023-01-01 >> 11:11:15.428300 23686'264880 23686:1309008 [39,58] 39 >> [39,58] 39 23662'264095 2022-12-18 03:31:44.747218 >> 23662'264007 2022-12-16 01:42:56.447456 0 >> 3.e8d 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-28 >> 13:45:08.250912 0'0 23686:13114 [10,58] 10 >> [10,58] 10 0'0 2022-12-28 13:45:08.250851 >> 0'0 2022-12-28 13:45:08.250851 0 >> 1.e8f 59858 0 0 0 0 175774668550 >> 0 0 3013 3013 active+clean 2022-12-28 >> 14:14:45.310609 23686'268346 23686:1246625 [5,9] 5 >> [5,9] 5 23686'268182 2022-12-28 14:14:45.310564 >> 23686'268182 2022-12-28 14:14:45.310564 0 >> 3.e8c 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2023-01-01 >> 10:49:45.386653 0'0 23686:11814 [150,100] 150 >> [150,100] 150 0'0 2023-01-01 10:49:45.386577 >> 0'0 2022-12-27 09:43:45.177707 0 >> 1.e8e 59897 0 0 0 0 175613850948 >> 0 0 3053 3053 active+clean+scrubbing+deep 2023-01-01 >> 11:11:21.286209 23686'267818 23686:1778964 [106,120] 106 >> [106,120] 106 23664'267253 2022-12-21 12:04:17.111762 >> 23662'266968 2022-12-15 00:56:58.941970 0 >> 3.e8f 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-28 >> 13:45:06.708181 0'0 23686:22818 [66,64] 66 >> [66,64] 66 0'0 2022-12-28 13:45:06.708120 >> 0'0 2022-12-28 13:45:06.708120 0 >> 1.e8d 60464 0 0 0 0 177786454089 >> 0 0 3088 3088 active+clean 2022-12-31 >> 19:37:07.860682 23686'266321 23686:1723741 [82,136] 82 >> [82,136] 82 23686'266298 2022-12-31 19:37:07.860638 >> 23686'266242 2022-12-30 15:30:59.925650 0 >> 3.e8e 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 10:17:48.873972 0'0 23686:11722 [145,88] 145 >> [145,88] 145 0'0 2022-12-31 10:17:48.873922 >> 0'0 2022-12-22 19:29:59.342170 0 >> 1.e8c 59747 0 0 0 0 175420690634 >> 0 0 3011 3011 active+clean 2022-12-28 >> 14:14:10.375574 23686'264942 23686:1426381 [167,79] 167 >> [167,79] 167 23686'264765 2022-12-28 14:14:10.375538 >> 23686'264765 2022-12-28 14:14:10.375538 0 >> 3.e89 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 11:10:25.652472 0'0 23686:12379 [176,76] 176 >> [176,76] 176 0'0 2022-12-31 11:10:25.652431 >> 0'0 2022-12-31 11:10:25.652431 0 >> 1.e8b 60177 0 0 0 0 176212803132 >> 0 0 3084 3084 active+clean 2022-12-31 >> 14:34:27.266317 23686'266021 23686:1390892 [61,54] 61 >> [61,54] 61 23686'265981 2022-12-31 14:34:27.266247 >> 23662'265380 2022-12-18 03:30:40.569564 0 >> 3.e88 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 22:12:22.542301 0'0 23686:22513 [84,109] 84 >> [84,109] 84 0'0 2022-12-31 22:12:22.542239 >> 0'0 2022-12-21 19:52:56.474546 0 >> 1.e8a 59937 0 0 0 0 175967277330 >> 0 0 3092 3092 active+clean 2023-01-01 >> 06:30:33.023442 23686'266318 23686:1569176 [73,117] 73 >> [73,117] 73 23686'266314 2023-01-01 06:30:33.023360 >> 23686'266182 2022-12-28 10:24:38.743599 0 >> 3.e8b 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2023-01-01 >> 09:42:09.452649 0'0 23686:11736 [123,135] 123 >> [123,135] 123 0'0 2023-01-01 09:42:09.452584 >> 0'0 2022-12-22 21:10:31.693983 0 >> 1.e89 60399 0 0 0 0 177399669362 >> 0 0 3080 3080 active+clean 2022-12-18 >> 07:20:13.328618 23686'267906 23686:1593362 [59,168] 59 >> [59,168] 59 23662'267319 2022-12-18 07:20:13.328540 >> 23662'267152 2022-12-13 19:52:53.012182 0 >> 3.e8a 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-28 >> 13:45:09.179473 0'0 23686:18789 [108,64] 108 >> [108,64] 108 0'0 2022-12-28 13:45:09.179409 >> 0'0 2022-12-28 13:45:09.179409 0 >> 1.e88 60153 0 0 0 0 177311741723 >> 0 0 3075 3075 active+clean 2023-01-01 >> 04:21:47.767674 23686'266189 23686:1668748 [137,81] 137 >> [137,81] 137 23686'266172 2023-01-01 04:21:47.767620 >> 23686'266128 2022-12-30 22:45:30.912755 0 >> 3.e85 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-28 >> 13:45:12.962084 0'0 23686:13019 [112,45] 112 >> [112,45] 112 0'0 2022-12-28 13:45:12.962034 >> 0'0 2022-12-28 13:45:12.962034 0 >> 1.e87 59929 0 0 0 0 176533295427 >> 0 0 3049 3049 active+clean 2022-12-31 >> 18:16:00.647528 23686'264595 23686:1527102 [117,170] 117 >> [117,170] 117 23686'264560 2022-12-31 18:16:00.647476 >> 23686'264494 2022-12-30 06:21:02.489588 0 >> 3.e84 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2023-01-01 >> 06:36:20.342735 0'0 23686:7209 [150,62] 150 >> [150,62] 150 0'0 2023-01-01 06:36:20.342691 >> 0'0 2022-12-20 01:13:12.470701 0 >> 1.e86 60155 0 0 0 0 177238192213 >> 0 0 3076 3076 active+clean 2022-12-31 >> 23:29:33.487171 23686'267316 23686:1251745 [64,42] 64 >> [64,42] 64 23686'267298 2022-12-31 23:29:33.487126 >> 23686'267298 2022-12-31 23:29:33.487126 0 >> 3.e87 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 18:38:55.362599 0'0 23686:22345 [21,171] 21 >> [21,171] 21 0'0 2022-12-31 18:38:55.362538 >> 0'0 2022-12-20 02:27:20.505038 0 >> 1.e85 59960 0 0 0 0 176462625858 >> 0 0 3039 3039 active+clean 2022-12-28 >> 14:12:58.614834 23686'267960 23686:1703075 [178,3] 178 >> [178,3] 178 23686'267812 2022-12-28 14:12:58.614794 >> 23686'267812 2022-12-28 14:12:58.614794 0 >> 3.e86 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 21:23:52.856914 0'0 23686:25444 [64,144] 64 >> [64,144] 64 0'0 2022-12-31 21:23:52.856861 >> 0'0 2022-12-23 05:07:09.853120 0 >> 1.e84 60382 0 0 0 0 177545253581 >> 0 0 3052 3052 active+clean 2022-12-31 >> 10:34:48.971952 23686'265991 23686:1470790 [118,33] 118 >> [118,33] 118 23686'265947 2022-12-31 10:34:48.971907 >> 23686'265806 2022-12-28 11:10:20.014077 0 >> 3.e81 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-28 >> 13:45:13.079440 0'0 23686:11568 [141,72] 141 >> [141,72] 141 0'0 2022-12-28 13:45:13.079391 >> 0'0 2022-12-28 13:45:13.079391 0 >> 1.e83 59549 0 0 0 0 174492785801 >> 0 0 3055 3055 active+clean 2023-01-01 >> 05:30:29.513513 23686'264403 23686:1357982 [47,39] 47 >> [47,39] 47 23686'264396 2023-01-01 05:30:29.513468 >> 23686'264350 2022-12-31 00:37:11.780263 0 >> 3.e80 0 0 0 0 0 0 >> 0 0 0 0 active+clean 2022-12-31 >> 12:12:35.898591 0'0 23686:18485 [93,125] 93 >> [93,125] 93 0'0 2022-12-31 12:12:35.898517 >> 0'0 2022-12-25 03:23:24.868798 0 >> 1.e82 59984 0 0 0 0 176200333351 >> 0 0 3003 3003 active+clean+scrubbing+deep 2023-01-01 >> 11:11:24.225656 23686'266139 23686:1668141 [102,131] 102 >> [102,131] 102 23662'265369 2022-12-18 07:17:59.598616 >> 23662'265369 2022-12-18 07:17:59.598616 0 >> Thanks for any help, >> jeff >> -- >> Jeff Turmelle >> International Research Institute for Climate & Society >> Columbia University Climate School >> _______________________________________________ >> ceph-users mailing list -- [email protected] >> To unsubscribe send an email to [email protected]
_______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
