Jakov Varenina created GEODE-10401:
--------------------------------------
Summary: Oplog recovery takes too long due to fault in fastutil
library
Key: GEODE-10401
URL: https://issues.apache.org/jira/browse/GEODE-10401
Project: Geode
Issue Type: Bug
Reporter: Jakov Varenina
{color:#0e101a}As we already know, the .drf file delete operations only contain
OplogEntryID. During recovery, the server reads (byte by byte) each
OplogEntryID and stores it in a HashSet to use later when recovering .crf
files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet. The
OplogEntryID of type {color}_{color:#0e101a}integer{color}_{color:#0e101a} will
be stored in IntOpenHashSet, and {color}_{color:#0e101a}long
integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
optimization and performance factors. OplogEntryID starts with a zero and
increments throughout time. Recovery speed could differ depending on which
HashSet is used, so please consider that when estimating .drf recovery
time.{color}
{color:#0e101a}We have observed in logs that between exception (There is a
large number of deleted entries) and the previous log have passed more than 4
minutes (sometimes even more).{color}
{color:#0e101a}{"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
store dataDiskStore.","metadata":{color}
{color:#0e101a}{"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
is a large number of deleted entries within the disk-store, please execute an
offline{color}
{color:#0e101a}compaction.","metadata":{color}
{color:#0e101a} {color}
{color:#0e101a}When the above exception occurs, that means that the limit of
{color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
IntOpenHashSet has been reached. In that case, the server rolls to the new
IntOpenHashSet, where an exception and the delay could happen again.{color}
{color:#0e101a}The problem is that due to the fault in FastUtil dependency
(IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
multiple times before the max size is reached. The{color}
_{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368 onwards
for each new entry until the max size. This rehashing adds several minutes to
.drf Oplog recovery.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
