Hi, SolrCloud is designed with an assumption that you should be able to upload your whole disk-based conf folder into ZK, and that you should be able to add an empty Solr node to a cluster and it would download all config from ZK. So immediately a splitting strategy automatically handled by ZkSolresourceLoader for large files could be one way forward, i.e. store synonyms.txt as e.g. __001_synonyms.txt __002_synonyms.txt....
Feel free to open a JIRA issue for this so we can get a proper resolution. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com 7. mai 2013 kl. 09:55 skrev Roman Chyla <roman.ch...@gmail.com>: > We have synonym files bigger than 5MB so even with compression that would > be probably failing (not using solr cloud yet) > Roman > On 6 May 2013 23:09, "David Parks" <davidpark...@yahoo.com> wrote: > >> Wouldn't it make more sense to only store a pointer to a synonyms file in >> zookeeper? Maybe just make the synonyms file accessible via http so other >> boxes can copy it if needed? Zookeeper was never meant for storing >> significant amounts of data. >> >> >> -----Original Message----- >> From: Jan Høydahl [mailto:jan....@cominvent.com] >> Sent: Tuesday, May 07, 2013 4:35 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr Cloud with large synonyms.txt >> >> See discussion here >> http://lucene.472066.n3.nabble.com/gt-1MB-file-to-Zookeeper-td3958614.html >> >> One idea was compression. Perhaps if we add gzip support to SynonymFilter >> it >> can read synonyms.txt.gz which would then fit larger raw dicts? >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> >> 6. mai 2013 kl. 18:32 skrev Son Nguyen <s...@trancorp.com>: >> >>> Hello, >>> >>> I'm building a Solr Cloud (version 4.1.0) with 2 shards and a Zookeeper >> (the Zookeeer is on different machine, version 3.4.5). >>> I've tried to start with a 1.7MB synonyms.txt, but got a >> "ConnectionLossException": >>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /configs/solr1/synonyms.txt >>> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266) >>> at >> org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:270) >>> at >> org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:267) >>> at >> >> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java >> :65) >>> at >> org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:267) >>> at >> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:436) >>> at >> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:315) >>> at >> org.apache.solr.cloud.ZkController.uploadToZK(ZkController.java:1135) >>> at >> org.apache.solr.cloud.ZkController.uploadConfigDir(ZkController.java:955) >>> at >> org.apache.solr.core.CoreContainer.initZooKeeper(CoreContainer.java:285) >>> ... 43 more >>> >>> I did some researches on internet and found out that because Zookeeper >> znode size limit is 1MB. I tried to increase the system property >> "jute.maxbuffer" but it won't work. >>> Does anyone have experience of dealing with it? >>> >>> Thanks, >>> Son >> >>
