Re: SolrCloud and 2MB Synonym file

2013-11-28 Thread Puneet Pawaia
Hi Yago, Since the value you have mentioned is less than what I have used and it is working, I won't rock the boat. Thanks for the env.sh direction. I'll use it next time I need to set things up. Regards Puneet On Thu, Nov 28, 2013 at 4:01 PM, Yago Riveiro wrote: > In my case I have: > > - Zo

Re: SolrCloud and 2MB Synonym file

2013-11-28 Thread Yago Riveiro
In my case I have: - Zookeeper: a file into conf folder named zookeeper-env.sh with: #!/usr/bin/env bash ZOO_ENV="-Djute.maxbuffer=10485761" The zookeeper-env.sh is loaded automatically by the zkEnv.sh if the file exists. - Solr: the "-Djute.maxbuffer=10485761" param in start command --

Re: SolrCloud and 2MB Synonym file

2013-11-28 Thread Puneet Pawaia
Hi Managed to get it working by adding -Djute.maxbuffer=5242880 to zkServer.sh, zkCli.sh and solr startup command. Please let me know if this value for jute.maxbuffer is within acceptable limits Regards Puneet On Thu, Nov 28, 2013 at 6:40 AM, Puneet Pawaia wrote: > I am running an ensemble. >

Re: SolrCloud and 2MB Synonym file

2013-11-27 Thread Puneet Pawaia
I am running an ensemble. Can I get examples of how to use the option? I think there are not many examples available of the exact usage. Regards Puneet On 27 Nov 2013 23:23, "Yago Riveiro" wrote: > How are you launching Solr? > > Do you have an ensemble or you're running zookeeper embedded? > >

Re: SolrCloud and 2MB Synonym file

2013-11-27 Thread Timothy Potter
I'm curious how much compression you get with your synonym file using something basic like gzip? If significant, would it make sense to store the compressed syn file in ZooKeeper (or any other metadata you need to distribute around the cluster)? This would require the code that reads the syn file f

Re: SolrCloud and 2MB Synonym file

2013-11-27 Thread Mark Miller
They are just trying to keep users from using ZK in a bad way. Storing and accessing a ton of huge files is not what ZooKeeper was designed for. A 1MB limit is a fairly arbitrary limiter to make sure you don’t shoot yourself in the foot and store lots of large files. With modern networks and har

Re: SolrCloud and 2MB Synonym file

2013-11-27 Thread Yago Riveiro
How are you launching Solr? Do you have an ensemble or you're running zookeeper embedded? Yes, doc says that jute.maxbuffer is dangerous, but without it you can stored nothing with more than 1M in zookeeper … and in some point you can have a clusterstate.json with a size greater than 1M -

Re: SolrCloud and 2MB Synonym file

2013-11-27 Thread Puneet Pawaia
Yago, not sure if this is a good idea. Docs say this is dangerous stuff. Anyway, not being a linux or java expert, I would appreciate if you could point me to an implementation of this. Regards Puneet Pawaia On 27 Nov 2013 22:54, "Yago Riveiro" wrote: > You can use the jute.maxbuffer > 1M as

Re: SolrCloud and 2MB Synonym file

2013-11-27 Thread Yago Riveiro
You can use the jute.maxbuffer > 1M as a workaround. You must set -Djute.maxbuffer in zookeeper and solr to work properly -- Yago Riveiro Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote: > Hi > > I am trying to setup a t

SolrCloud and 2MB Synonym file

2013-11-27 Thread Puneet Pawaia
Hi I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym file is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on Ubuntu 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any experience with using such synonym files? Can I store them in some other lo