Hi Nishant,
The debug message shows there are not enough racks configured to satisfy the
rack awareness.
http://hadoop.apache.org/docs/r3.0.0-alpha4/hadoop-project-dist/hadoop-common/RackAwareness.html
If you don't need to place replicas in different racks, you can simply ignore
the debug message.
Regards,
Akira
On 2017/08/01 11:53, Nishant Verma wrote:
Hello
On my namenode logs, I am seeing these error lines quite frequently. What is
the reason behind this and how to overcome this? I have three data nodes and
hdfs dfsadmin -report says 3 Live Datanodes.
Kafka Connect is responsible of writing records from kafka topic to hdfs here.
The data produce speed on kafka is high. 17 million records per hour. We have
given a 30 minute rotate time interval to kafka connect to write the records on
hdfs.
2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: *BLOCK*
NameNode.addBlock: file
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
fileId=16394 for DFSClient_NONMAPREDUCE_-1139929652_127
2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK*
getAdditionalBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
inodeId 16394 for DFSClient_NONMAPREDUCE_-1139929652_127
2017-08-01 09:12:55,217 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
choose remote rack (location = ~/default-rack), fallback to local rack
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:702)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:582)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:350)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:216)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:113)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:128)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1567)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: DIR*
FSDirectory.addBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
with blk_1073741825_1001 block is added to the in-memory file system
2017-08-01 09:12:55,220 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW],
ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]}
for
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange:
persistNewBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
with new block blk_1073741825_1001{UCState=UNDER_CONSTRUCTION,
truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW],
ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]},
current total block count is 1
Thanks
Nishant Verma
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
