Hi,
Iam new to solr. I have integrated solr4.9.0 and Hadoop 2.3.0.
I have changed the solrconfig.xml file so that it can index and store the data
on hdfs.
Solrconfig.xml
<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://xxx.xx.xx.xx:50070/user/solr/data</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.blockcache.write.enabled">true</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>
Commands
java -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs
-Dsolr.data.dir=hdfs:// xxx.xx.xx.xx:50070/user/solr/data/collection1 -
Dsolr.updatelog=hdfs:// xxx.xx.xx.xx:50070/user/solr/data/collection1 -jar
start.jar
Error
48454 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.core.CoreContainer â
Unable to create core: collection1
org.apache.solr.common.SolrException: Problem creating directory:
hdfs://172.29.17.40:50070/user/solr/data/collection1
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:868)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:261)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:253)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: Problem creating directory:
hdfs://172.29.17.40:50070/user/solr/data/collection1
at
org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:87)
at
org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:148)
at
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:267)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:479)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:774)
... 12 more
Caused by: java.io.IOException: Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group
tag did not match expected tag.; Host Details : local host is:
"cldx-1310-1182/172.29.17.40"; destination host is: "cldx-1310-1182":50070;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:62)
... 17 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag.
at
com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at
com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
at
com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
48485 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.core.CoreContainer â
null:org.apache.solr.common.SolrException: Unable to create core: collection1
at
org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:261)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:253)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.solr.common.SolrException: Problem creating directory:
hdfs://172.29.17.40:50070/user/solr/data/collection1
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:868)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
... 10 more
Caused by: java.lang.RuntimeException: Problem creating directory:
hdfs://172.29.17.40:50070/user/solr/data/collection1
at
org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:87)
at
org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:148)
at
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:267)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:479)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:774)
... 12 more
Caused by: java.io.IOException: Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group
tag did not match expected tag.; Host Details : local host is:
"cldx-1310-1182/172.29.17.40"; destination host is: "cldx-1310-1182":50070;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:62)
... 17 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag.
at
com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at
com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
at
com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
48535 [main] INFO org.apache.solr.servlet.SolrDispatchFilter â
user.dir=/home/huser1/solr-4.9.0/example
48536 [main] INFO org.apache.solr.servlet.SolrDispatchFilter â
SolrDispatchFilter.init() done
49200 [main] INFO org.eclipse.jetty.server.AbstractConnector â Started
SocketConnector@0.0.0.0:8983<mailto:SocketConnector@0.0.0.0:8983>
Can anyone look into this and help me with the solution.
Regards,
Jeniba
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or
privileged information for the intended recipient(s). Unintended recipients are
prohibited from taking action on the basis of information in this e-mail and
using or disseminating the information, and must notify the sender and delete
it from their system. L&T Infotech will not accept responsibility or liability
for the accuracy or completeness of, or the presence of any virus or disabling
code in this e-mail"
