I'm curious how much compression you get with your synonym file using something basic like gzip? If significant, would it make sense to store the compressed syn file in ZooKeeper (or any other metadata you need to distribute around the cluster)? This would require the code that reads the syn file from ZooKeeper to be able to de-compress it. Seems like this would be a nice-to-have in SolrCloud in general - the ability to read / write files to ZooKeeper in compressed format.
Tim On Wed, Nov 27, 2013 at 11:29 AM, Mark Miller <markrmil...@gmail.com> wrote: > They are just trying to keep users from using ZK in a bad way. Storing and > accessing a ton of huge files is not what ZooKeeper was designed for. A 1MB > limit is a fairly arbitrary limiter to make sure you don’t shoot yourself in > the foot and store lots of large files. With modern networks and hardware, > setting it to 3MB and uploading your 2MB syn file is not going to be a > problem. Solr doesn’t read and write those files often, nor use ZooKeeper > much at all in a stable state. Upping that limit and putting in a few config > files that are a few MB is not going to break anything. > > - Mark > > On Nov 27, 2013, at 12:31 PM, Puneet Pawaia <puneet.paw...@gmail.com> wrote: > >> Yago, not sure if this is a good idea. Docs say this is dangerous stuff. >> >> Anyway, not being a linux or java expert, I would appreciate if you could >> point me to an implementation of this. >> >> Regards >> Puneet Pawaia >> On 27 Nov 2013 22:54, "Yago Riveiro" <yago.rive...@gmail.com> wrote: >> >>> You can use the jute.maxbuffer > 1M as a workaround. >>> >>> You must set -Djute.maxbuffer in zookeeper and solr to work properly >>> >>> -- >>> Yago Riveiro >>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig) >>> >>> >>> On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote: >>> >>>> Hi >>>> >>>> I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym >>> file >>>> is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on >>> Ubuntu >>>> 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any >>>> experience with using such synonym files? Can I store them in some other >>>> location other than the Config Folder since the config folder is loaded >>>> into ZooKeeper. >>>> >>>> TIA >>>> >>>> Puneet Pawaia >>> >>> >
