Cloudera has plans here. I'll be working on further hdfs / Solrcloud options in
the near future.
- Mark
> On Dec 26, 2013, at 11:33 AM, Greg Walters <greg.walt...@answers.com> wrote:
>
> YouPeng,
>
> While I'm unable to help you with the issue that you're seeing I did want to
> comment here and say that I have previously brought up the same goal that
> you're trying to accomplish on this mailing list but received no feedback or
> input. I think it makes sense that Solr should not try to make its index
> directories distinct and redundant per shard/core while running on HDFS as
> data redundancy and locality is handled at a different layer in the software
> stack.
>
> +1 to this topic because I'd love to see Solr handle replication/redundancy
> more smartly on HDFS
>
> Thanks,
> Greg
>
>
>> On Dec 24, 2013, at 1:57 AM, YouPeng Yang <yypvsxf19870...@gmail.com> wrote:
>>
>> Hi users
>>
>> Solr supports for writing and reading its index and transaction log files
>> to the HDFS distributed filesystem.
>> **I am curious about that there are any other futher improvement about
>> the integration with HDFS.*
>> **For the solr native replication will make multiple copies of the
>> master node's index. Because of the native replication of HDFS,there is no
>> need to do that.It just to need that multiple cores in solrcloud share the
>> same index directory in HDFS?*
>>
>>
>> The above supposition is what I want to achive when we are integrating
>> SolrCloud with HDFS (Solr 4.6).
>> To make sure of our application high available,we still have to take
>> the solr replication with some tricks.
>>
>> Firstly ,noting that solr's index directory is made up of
>> *collectionName/coreNodeName/data/index *
>>
>> *collectionName/coreNodeName/data/tlog*
>> So to achive this,we want to create multi cores that use the same hdfs
>> index directory .
>>
>> I have tested this within solr 4.4 by expilcitly indicating the same
>> coreNodeName.
>>
>> For example:
>> Step1, a core was created with the name=core1 and shard=core_shard1 and
>> collection=clollection1 and coreNodeName=*core1*
>> Step2. create another core with the name=core2 and shard=core_shard1 and
>> collection=clollection1 and coreNodeName=
>> *core1*
>> * T*he two core share the same shard ,collection and coreNodeName.As a
>> result,the two core will get the same index data which is stored in the
>> hdfs directory :
>> hdfs://myhdfs/*clollection1*/*core1*/data/index
>> hdfs://myhdfs/*clollection1*/*core1*/data/tlog
>>
>> Unfortunately*, *as the solr 4.6 was released,we upgraded . the above
>> goal failed. We could not create a core with both expilcit shard and
>> coreNodeName.
>> Exceptions are as [1].
>> * Can some give some help?*
>>
>>
>> Regards
>> [1]------------------------------------------------------------------------------------------------------------------
>> 64893635 [http-bio-8080-exec-1] INFO org.apache.solr.cloud.ZkController
>> ?.publishing core=hdfstest3 state=down
>> 64893635 [http-bio-8080-exec-1] INFO org.apache.solr.cloud.ZkController
>> ?.numShards not found on descriptor - reading it from system property
>> 64893698 [http-bio-8080-exec-1] INFO org.apache.solr.cloud.ZkController
>> ?.look for our core node name
>>
>>
>>
>> 64951227 [http-bio-8080-exec-17] INFO org.apache.solr.core.SolrCore
>> ?.[reportCore_201208] webapp=/solr path=/replication
>> params={slave=false&command=details&wt=javabin&qt=/replication&version=2}
>> status=0 QTime=107
>>
>>
>> 65213770 [http-bio-8080-exec-1] INFO org.apache.solr.cloud.ZkController
>> ?.waiting to find shard id in clusterstate for hdfstest3
>> 65533894 [http-bio-8080-exec-1] ERROR org.apache.solr.core.SolrCore
>> ?.org.apache.solr.common.SolrException: Error CREATEing SolrCore
>> 'hdfstest3': Could not get shard id for core: hdfstest3
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:535)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.solr.common.SolrException: Could not get shard id for
>> core: hdfstest3
>> at
>> org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1302)
>> at
>> org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1248)
>> at
>> org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1399)
>> at
>> org.apache.solr.core.CoreContainer.preRegisterInZk(CoreContainer.java:942)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:492)
>> ... 20 more
>
