We have a few ideas we may end up working on. I'm going to start the
implementation of the first idea pretty soon - I'll create a JIRA issue
which will be the best way to keep tabs on the progress. I don't expect
an initial implementation to take long.
That's for something akin to the HBase region server model. A shared
index among leader and replicas is a longer term item we might explore.
A lot of trade offs depending on what we try and do, so nothing set in
stone yet, but I'll start on phase 1 impl any day now.
- Mark
On 12/29/2013 09:08 AM, YouPeng Yang wrote:
> Hi Mark
>
> Are there have roadmap or plan about your futher work on Solr and with.
> I do anticipate your great work.
>
>
> Regards
>
>
> 2013/12/27 Mark Miller <markrmil...@gmail.com>
>
>> 1. The exception and change in experience on the move to 4.6 seems like it
>> could be a bug we want to investigate.
>>
>> 2. Solr storing data on hdfs in other ways seems like a different issue /
>> improvement.
>>
>> 3. You shouldn't try and force more than one core to use the same index on
>> hdfs. This would be bad.
>>
>> 4. You really want to use the solr.hdfs.home setting described in the
>> documentation IMO.
>>
>> - Mark
>>
>>> On Dec 26, 2013, at 1:56 PM, Greg Walters <greg.walt...@answers.com>
>> wrote:
>>>
>>> Mark,
>>>
>>> I'd be happy to but some clarification first; should this issue be about
>> creating cores with overlapping names and the stack trace that YouPeng
>> initially described, Solr's behavior when storing data on HDFS or YouPeng's
>> other thread (Maybe a bug for solr 4.6 when create a new core) that looks
>> like it might be a near duplicate of this one?
>>>
>>> Thanks,
>>> Greg
>>>
>>>> On Dec 26, 2013, at 12:40 PM, Mark Miller <markrmil...@gmail.com>
>> wrote:
>>>>
>>>> Can you file a JIRA issue?
>>>>
>>>> - Mark
>>>>
>>>>> On Dec 24, 2013, at 2:57 AM, YouPeng Yang <yypvsxf19870...@gmail.com>
>> wrote:
>>>>>
>>>>> Hi users
>>>>>
>>>>> Solr supports for writing and reading its index and transaction log
>> files
>>>>> to the HDFS distributed filesystem.
>>>>> **I am curious about that there are any other futher improvement about
>>>>> the integration with HDFS.*
>>>>> **For the solr native replication will make multiple copies of the
>>>>> master node's index. Because of the native replication of HDFS,there
>> is no
>>>>> need to do that.It just to need that multiple cores in solrcloud share
>> the
>>>>> same index directory in HDFS?*
>>>>>
>>>>>
>>>>> The above supposition is what I want to achive when we are integrating
>>>>> SolrCloud with HDFS (Solr 4.6).
>>>>> To make sure of our application high available,we still have to take
>>>>> the solr replication with some tricks.
>>>>>
>>>>> Firstly ,noting that solr's index directory is made up of
>>>>> *collectionName/coreNodeName/data/index *
>>>>>
>>>>> *collectionName/coreNodeName/data/tlog*
>>>>> So to achive this,we want to create multi cores that use the same hdfs
>>>>> index directory .
>>>>>
>>>>> I have tested this within solr 4.4 by expilcitly indicating the same
>>>>> coreNodeName.
>>>>>
>>>>> For example:
>>>>> Step1, a core was created with the name=core1 and shard=core_shard1 and
>>>>> collection=clollection1 and coreNodeName=*core1*
>>>>> Step2. create another core with the name=core2 and shard=core_shard1
>> and
>>>>> collection=clollection1 and coreNodeName=
>>>>> *core1*
>>>>> * T*he two core share the same shard ,collection and coreNodeName.As a
>>>>> result,the two core will get the same index data which is stored in the
>>>>> hdfs directory :
>>>>> hdfs://myhdfs/*clollection1*/*core1*/data/index
>>>>> hdfs://myhdfs/*clollection1*/*core1*/data/tlog
>>>>>
>>>>> Unfortunately*, *as the solr 4.6 was released,we upgraded . the above
>>>>> goal failed. We could not create a core with both expilcit shard and
>>>>> coreNodeName.
>>>>> Exceptions are as [1].
>>>>> * Can some give some help?*
>>>>>
>>>>>
>>>>> Regards
>>>>>
>> [1]------------------------------------------------------------------------------------------------------------------
>>>>> 64893635 [http-bio-8080-exec-1] INFO
>> org.apache.solr.cloud.ZkController
>>>>> ?.publishing core=hdfstest3 state=down
>>>>> 64893635 [http-bio-8080-exec-1] INFO
>> org.apache.solr.cloud.ZkController
>>>>> ?.numShards not found on descriptor - reading it from system property
>>>>> 64893698 [http-bio-8080-exec-1] INFO
>> org.apache.solr.cloud.ZkController
>>>>> ?.look for our core node name
>>>>>
>>>>>
>>>>>
>>>>> 64951227 [http-bio-8080-exec-17] INFO org.apache.solr.core.SolrCore
>>>>> ?.[reportCore_201208] webapp=/solr path=/replication
>>>>>
>> params={slave=false&command=details&wt=javabin&qt=/replication&version=2}
>>>>> status=0 QTime=107
>>>>>
>>>>>
>>>>> 65213770 [http-bio-8080-exec-1] INFO
>> org.apache.solr.cloud.ZkController
>>>>> ?.waiting to find shard id in clusterstate for hdfstest3
>>>>> 65533894 [http-bio-8080-exec-1] ERROR org.apache.solr.core.SolrCore
>>>>> ?.org.apache.solr.common.SolrException: Error CREATEing SolrCore
>>>>> 'hdfstest3': Could not get shard id for core: hdfstest3
>>>>> at
>>>>>
>> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:535)
>>>>> at
>>>>>
>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
>>>>> at
>>>>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>>>> at
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
>>>>> at
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
>>>>> at
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>>>>> at
>>>>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>>> at
>>>>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>> at
>>>>>
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>>> at
>>>>>
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>> at
>>>>>
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>> at
>>>>>
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>> at
>>>>>
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
>>>>> at
>>>>>
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>> at
>>>>>
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>>>> at
>>>>>
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
>>>>> at
>>>>>
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>> at
>>>>>
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>>>> at
>>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> at
>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>> at java.lang.Thread.run(Thread.java:722)
>>>>> Caused by: org.apache.solr.common.SolrException: Could not get shard
>> id for
>>>>> core: hdfstest3
>>>>> at
>>>>>
>> org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1302)
>>>>> at
>>>>>
>> org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1248)
>>>>> at
>>>>> org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1399)
>>>>> at
>>>>>
>> org.apache.solr.core.CoreContainer.preRegisterInZk(CoreContainer.java:942)
>>>>> at
>>>>>
>> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:492)
>>>>> ... 20 more
>>>
>>
>
