Hi Nishant, Looks like you didn’t configured HDFS HA properties in hdfs-site.xml files in your client configuration. Please configure these properties and try.
dfs.nameservices=ha-cluster-dev1 dfs.ha.namenodes.ha-cluster-dev1=nn1,nn2 dfs.namenode.rpc-address.ha-cluster-dev1.nn1=hostname1.hadoop.com:8020 dfs.namenode.rpc-address.ha-cluster-dev1.nn2=hostname2.hadoop.com:8020 dfs.client.failover.proxy.provider.ha-cluster-dev1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider Please refer https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html Regards, Surendra ________________________________ From: Nishant Verma [[email protected]] Sent: Tuesday, July 04, 2017 12:50 PM To: [email protected] Subject: Error while using namespace ID from external system like kafka connect to access hdfs cluster Not sure if this is the exact forum for this query. HDFS is 2.7.3 Kafka Connect is 3.2.0 and Kafka is 0.10.2.0. We are using Kafka Connect HDFS Connector to pull records from Kafka topic to HDFS. Till yesyerday, we had just one namenode in our cluster and we were using "hdfs.url" as "hdfs://abc.xyz.com:9000<http://abc.xyz.com:9000/>" in the REST POST request of Kafka Connect HDFS Connector for the connector to start. And it used to work. The same URL was defined in core-site.xml at fs.defaultFS property. Now, we have updated our cluster to a active-standby namenode kind of cluster and we are using namespace now instead of hdfs node ip and port. We have updated our core-site.xml to fs.defaultFS values as hdfs://ha-cluster-dev1. We are able to do the active and standby switch of namenodes. We are also able to list hdfs directories using hadoop fs -ls hdfs://ha-cluster-dev1/. We have updated our REST request to "hdfs.url":"hdfs://cfms-ha-cluster-dev1/" and rest other properties in the payload are same. Now, as soon as we send this REST POST request, all the tasks die immediately and hdfs connector does not start. The log says below error: [2017-07-04 16:57:12,951] INFO Couldn't start HdfsSinkConnector: (io.confluent.connect.hdfs.HdfsSinkTask:84) org.apache.kafka.connect.errors.ConnectException: java.lang.reflect.InvocationTargetException at io.confluent.connect.hdfs.storage.StorageFactory.createStorage(StorageFactory.java:31) at io.confluent.connect.hdfs.DataWriter.<init>(DataWriter.java:168) at io.confluent.connect.hdfs.HdfsSinkTask.start(HdfsSinkTask.java:76) at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:231) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:145) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:408) at io.confluent.connect.hdfs.storage.StorageFactory.createStorage(StorageFactory.java:29) ... 11 more Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: ha-cluster-dev1 at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2691) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:420) at io.confluent.connect.hdfs.storage.HdfsStorage.<init>(HdfsStorage.java:39) ... 15 more Caused by: java.net.UnknownHostException: ha-cluster-dev1 ... 27 more [2017-07-04 16:57:12,952] INFO Shutting down HdfsSinkConnector. (io.confluent.connect.hdfs.HdfsSinkTask:85) And all the tasks die with error trace as: java.lang.NullPointerException at io.confluent.connect.hdfs.HdfsSinkTask.close(HdfsSinkTask.java:121) at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:317) at org.apache.kafka.connect.runtime.WorkerSinkTask.closePartitions(WorkerSinkTask.java:480) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:152) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) Clearly, the request is looking for hdfs.url to be a host:port combination but my namespace is not a host per se and hence the error. I am having hdfs cluster namenodes on different machines and kafka connect nodes on different machines. How am I supposed to run kafka connect (or generically any other external system) now using nameservice ID and fix this error? Thanks
