xinglin opened a new pull request #3201:
URL: https://github.com/apache/hadoop/pull/3201
Test:
We create 1 million dirs and then stop and re-start the namenode. Namenode
restarts successfully.
```
xinglin@xinglin-mn1 ~/p/h/h/t/hadoop-3.4.0-SNAPSHOT (fgl-saveloadfs)>
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
hdfs://localhost:9000 -op mkdirs -threads 50 -dirs 1000000 -dirsPerDir 100
-keepResults
2021-07-13 14:59:39,703 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2021-07-13 14:59:40,264 INFO namenode.NNThroughputBenchmark: Starting
benchmark: mkdirs
2021-07-13 14:59:40,412 INFO namenode.NNThroughputBenchmark: Generate
1000000 inputs for mkdirs
2021-07-13 14:59:40,669 INFO namenode.NNThroughputBenchmark: Log level =
ERROR
2021-07-13 14:59:40,894 INFO namenode.NNThroughputBenchmark: Starting
1000000 mkdirs(s).
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark:
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: --- mkdirs
inputs ---
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: nrDirs = 1000000
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: nrThreads = 50
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: nrDirsPerDir =
100
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: --- mkdirs
stats ---
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: # operations:
1000000
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: Elapsed Time:
33637
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: Ops per sec:
29729.16728602432
2021-07-13 15:00:15,524 INFO namenode.NNThroughputBenchmark: Average Time: 1
xinglin@xinglin-mn1 ~/p/h/h/t/hadoop-3.4.0-SNAPSHOT (fgl-saveloadfs)>
./bin/hdfs dfs -ls -R / | wc
2021-07-13 15:01:23,813 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
1010002 8080016 156814967
xinglin@xinglin-mn1 ~/p/h/h/t/hadoop-3.4.0-SNAPSHOT (fgl-saveloadfs)>
./sbin/stop-dfs.sh
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [localhost]
2021-07-13 15:09:47,518 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
xinglin@xinglin-mn1 ~/p/h/h/t/hadoop-3.4.0-SNAPSHOT (fgl-saveloadfs)>
./sbin/start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [localhost]
2021-07-13 15:10:05,055 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
xinglin@xinglin-mn1 ~/p/h/h/t/hadoop-3.4.0-SNAPSHOT (fgl-saveloadfs)> jps
61745 DataNode
61635 NameNode
61878 SecondaryNameNode
95382 CmdLineAppRunner
65082
62030 Jps
xinglin@xinglin-mn1 ~/p/h/h/t/hadoop-3.4.0-SNAPSHOT (fgl-saveloadfs)>
./bin/hdfs dfs -ls -R / | wc
2021-07-13 15:10:20,998 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
1010002 8080016 156814967
```
Log output from the namenode.
```
xinglin@xinglin-mn1 ~/p/h/h/t/h/logs (fgl-saveloadfs)> grep LOADING
hadoop-xinglin-namenode-xinglin-mn1.linkedin.biz.log
INFO org.apache.hadoop.hdfs.server.namenode.FSImage: LOADING_FSIMAGE: loaded
1 inodes into inodeMap
INFO org.apache.hadoop.hdfs.server.namenode.FSImage: LOADING_EDITS: loaded 1
inodes into inodeMap
INFO org.apache.hadoop.hdfs.server.namenode.FSImage: LOADING_FSIMAGE: loaded
1010003 inodes into inodeMap
INFO org.apache.hadoop.hdfs.server.namenode.FSImage: LOADING_EDITS: loaded
1010003 inodes into inodeMap
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
