[
https://issues.apache.org/jira/browse/HDFS-17864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068843#comment-18068843
]
ASF GitHub Bot commented on HDFS-17864:
---------------------------------------
github-actions[bot] commented on PR #8130:
URL: https://github.com/apache/hadoop/pull/8130#issuecomment-4139223736
We're closing this stale PR because it has been open for 100 days with no
activity. This isn't a judgement on the merit of the PR in any way. It's just a
way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working
on it, please feel free to re-open it and ask for a committer to remove the
stale tag and review again.
Thanks all for your contribution.
> Improve fsimage load time by making LightWeightGSet and NameCache thread-safe
> -----------------------------------------------------------------------------
>
> Key: HDFS-17864
> URL: https://issues.apache.org/jira/browse/HDFS-17864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: khazhen
> Priority: Major
> Labels: pull-request-available
>
> HDFS-14617 allows the inode and inode directory sections of the fsimage to be
> loaded in parallel.
> However, increasing the configured number of sections and threads has
> diminishing returns as there are some synchronized points in the loading code
> to protect some in memory structures.
> Currently, there are mainly 3 data structures that need to be protected by
> synchronized blocks:
> # INodeMap (internally based on LightWeightGSet, but it is not thread-safe)
> # BlocksMap (internally based on LightWeightGSet, but it is not thread-safe)
> # NameCache (it is not thread-safe by itself)
> To further improve FSImage loading speed, this PR attempts to make the above
> 3 data structures thread-safe, and then use multiple threads to initialize
> them when NameNode starts.
> Additionally, some optimizations have been made to reduce GC overheads during
> FSImage parsing.
> In our tests, the FSImage loading time (165M inodes & 258M blocks) was
> reduced from 183s to 73s.
> *1. Making LightWeightGSet thread-safe*
> LightWeightGSet is a HashMap-like data structure that uses a fixed-length
> array as hash buckets, with each array element storing the head node of an
> independent linked list.
> Since each linked list is independent, we can allocate a lock for each
> bucket to protect the corresponding linked list.
> To trade off between memory consumption and concurrency, we can let
> multiple buckets share a lock and use a hash-based mapping.
> To minimize changes, we don't plan to implement a completely thread-safe
> GSet to replace LightWeightGSet, as this would require significant changes
> and is unnecessary since all operations on LightWeightGSet are synchronized
> once NameNode finishes starting up.
> We introduced an external synchronization tool GSetConcurrencyController
> to ensure the thread safety of LightWeightGSet during NameNode startup.
> Another issue that needs to be addressed is the GSet's size. Currently,
> the size in LightWeightGSet is not an atomic variable, and even if we use
> segmented locks to protect hash buckets, the size is still inaccurate.
> Fortunately, in the FSImage loading scenario, we can clearly know the
> expected size of INodeMap and BlocksMap after loading, so we can correct its
> size after loading is complete.
> *2. Making NameCache thread-safe*
> This is simpler compared to LightWeightGSet. We only need to combine
> ConcurrentHashMap and AtomicInteger to implement a thread-safe version of
> NameCache.
> *3. Reducing GC pressure during FSImage loading*
> After completing steps 1 and 2, we found that GC gradually became a new
> bottleneck. After analysis, we discovered that the parseDelimitedFrom method
> in ProtoBuffer creates a 4096-byte array as cache when parsing each INode
> object.
> To optimize this issue, we introduced the DelimitedProtoBufParseHelper
> utility class to reuse the cache array.
> Appendix: Test environment and configuration information
> *Hadoop version*: current trunk, including previous fsimage loading
> optimizations: HDFS-13694, HDFS-14617, HDFS-15493, HDFS-13694
> *FSImage information*:
> Size: 20G (165M inodes & 258M blocks)
> *Config:*
> dfs.image.parallel.threads=16
> dfs.image.parallel.target.sections=128
> dfs.image.parallel.load=true
> *new config in this patch:*
> dfs.image.concurrent.init.inode.map.enable=true
> dfs.image.name.cache.init.thread.num=16
> dfs.image.block.map.init.thread.num=16
> *java version*: jdk17 (as GC became a new bottleneck, jdk17 performs better
> than jdk8 in our test )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
