sodonnel opened a new pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index URL: https://github.com/apache/hadoop/pull/1028 Initial code I used to perform my benchmarks for the above change. There are two parts to the change: 1) The code used to write the sub-sections to the image, which is fairly simple. 2) The code used to process the sub-sections in parallel. As much as possible this reuses the original code in multiple threads, but the inodeDirectory code needed to be moved around a bit to avoid synchronization issues. The only sections with parallel loading so far are inode and inodeDirectory. So far, there are no new tests for this feature. That is something I need to look into further if we agree this is a good change to move forward with.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
