HDFS-17864. Improve fsimage load time by making LightWeightGSet and NameCache thread-safe #8130
+937
−54
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refer to HDFS-17864.
HDFS-14617 allows the inode and inode directory sections of the fsimage to be loaded in parallel.
However, increasing the configured number of sections and threads has diminishing returns as there are some synchronized points in the loading code to protect some in memory structures.
Currently, there are mainly 3 data structures that need to be protected by synchronized blocks:
To further improve FSImage loading speed, this PR attempts to make the above 3 data structures thread-safe, and then use multiple threads to initialize them when NameNode starts.
Additionally, some optimizations have been made to reduce GC overheads during FSImage parsing.
In our tests, the FSImage loading time (165M inodes & 258M blocks) was reduced from 180s to 73s.
1. Making LightWeightGSet thread-safe
LightWeightGSet is a HashMap-like data structure that uses a fixed-length array as hash buckets, with each array element storing the head node of an independent linked list.
Since each linked list is independent, we can allocate a lock for each bucket to protect the corresponding linked list.
To trade off between memory consumption and concurrency, we can let multiple buckets share a lock and use a hash-based mapping.
To minimize changes, we don't plan to implement a completely thread-safe GSet to replace LightWeightGSet, as this would require significant changes and is unnecessary since all operations on LightWeightGSet are synchronized once NameNode finishes starting up.
We introduced an external synchronization tool GSetConcurrencyController to ensure the thread safety of LightWeightGSet during NameNode startup.
Another issue that needs to be addressed is the GSet's size. Currently, the size in LightWeightGSet is not an atomic variable, and even if we use segmented locks to protect hash buckets, the size is still inaccurate.
Fortunately, in the FSImage loading scenario, we can clearly know the expected size of INodeMap and BlocksMap after loading, so we can correct its size after loading is complete.
2. Making NameCache thread-safe
This is simpler compared to LightWeightGSet. We only need to combine ConcurrentHashMap and AtomicInteger to implement a thread-safe version of NameCache.
3. Reducing GC pressure during FSImage loading
After completing steps 1 and 2, we found that GC gradually became a new bottleneck. After analysis, we discovered that the parseDelimitedFrom method in ProtoBuffer creates a 4096-byte array as cache when parsing each INode object.
To optimize this issue, we introduced the DelimitedProtoBufParseHelper utility class to reuse the cache array.
Appendix: Test environment and configuration information
Hadoop version: current master, including previous fsimage loading optimizations: HDFS-13694, HDFS-14617, HDFS-15493
FSImage information:
Size: 20G (165M inodes & 258M blocks)
Config:
dfs.image.parallel.threads=16
dfs.image.parallel.target.sections=128
dfs.image.parallel.load=true
new config in this patch:
dfs.image.concurrent.init.inode.map.enable=true
dfs.image.name.cache.init.thread.num=16
dfs.image.block.map.init.thread.num=16