Shashank,
I had a quick look at:
https://lucene.apache.org/solr/guide/6_6/running-solr-on-hdfs.html
Did you enable the Block Cache and the solr.hdfs.nrtcachingdirectory?
cheers -- Rick
On 2017-10-03 09:22 PM, Shashank Pedamallu wrote:
Hi,
I’m trying an experiment in which, I’m loading a core of 1.27GB with 5621600
documents on 2 Solr setups. On first setup, dataDir is pointed as a
NRTCachingDirectory as a standard path in my local. On second setup, it is
pointed as HdfsDirectory. As part of loading the core, I see following log:
2017-10-04 01:07:50.102 UTC INFO
(searcherExecutor-12-thread-1-processing-x:staging_1gb-core-1)
[core='x:staging_1gb-core-1'] org.apache.solr.core.SolrCore@2247
[staging_1gb-core-1] Registered new searcher
Searcher@10fe9415[staging_1gb-core-1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_12bk(6.4.2):C2886542)
Uninverting(_1eu5(6.4.2):C743800) Uninverting(_18kl(6.4.2):c331485)
Uninverting(_1lt0(6.4.2):c284012) Uninverting(_1xx5(6.4.2):C654477)
Uninverting(_1qsg(6.4.2):C658237) Uninverting(_1qf4(6.4.2):c24903)
Uninverting(_1xwv(6.4.2):c16734) Uninverting(_1xwb(6.4.2):c17777)
Uninverting(_1xww(6.4.2):C174) Uninverting(_1xy9(6.4.2):c878)
Uninverting(_1xxf(6.4.2):c354) Uninverting(_1xxp(6.4.2):c508)
Uninverting(_1xx6(6.4.2):C150) Uninverting(_1xxz(6.4.2):c545)
Uninverting(_1xxg(6.4.2):C190) Uninverting(_1xyj(6.4.2):c690)
Uninverting(_1xyd(6.4.2):C144)))}
This step takes about 132 milli-secods in setup 1 (i.e., with
NRTCachingDirectoryFactory).
The same step takes about 21 minutes on second setup (i.e., with
HdfsDirectoryFactory).
Does the load time of a Solr Core drops so badly on a HdfsFileSystem? Is this
expected?
Thanks,
Shashank
.
