I think we'd all love to see those improvements land in Solr. I was involved in the work at AOL WebMail where the LotsOfCores idea originated. We had many of the problems that you've had to solve yourself. I remember that we switched to compound file format to reduce file descriptors. Also we had to switch back to the Log Merge Policy from TieredMergePolicy because TieredMergePolicy increased the overall random disk i/o and we had latency issues because of it.
On Mon, Oct 7, 2013 at 1:23 PM, Soyez Olivier <olivier.so...@worldline.com>wrote: > Hello, > > In my company, we use Solr in production to offer full text search on > mailboxes. > We host dozens million of mailboxes, but only webmail users have such > feature (few millions). > We have the following use case : > - non static indexes with more update (indexing and deleting), than > select requests (ratio 7:1) > - homogeneous configuration for all indexes > - not so much user at the same time > > We started to index mailboxes with Solr 1.4 in 2010, on a subset of > 400,000 users. > - we had a cluster of 50 servers, 4 Solr per server, 2000 users per Solr > instance > - we grow to 6000 users per Solr instance, 8 Solr per server, 60Go per > index (~2 million users) > - we upgraded to Solr 3.5 in 2012 > As indexes grew, IOPS and the response times have increased more and more. > > The index size was mainly due to stored fields (large .fdt files) > Retrieving these fields from the index was costly, because of many seek > in large files, and no limit usage possible. > There is also an overhead on queries : too many results are filtered to > find only results concerning user. > For these reason and others, like not pooled users, hardware savings, > better scoring, some requests that do not support filtering, we have > decided to use the LotsOfCores feature. > > Our goal was to change the current I/O usage : from lots of random I/O > access on huge segments to mostly sequential I/O access on small segments. > For our use case, it's not a big deal, that the first query to one not > yet loaded core will be slow. > And, we don’t need to fit all the cores into memory at once. > > We started from the SOLR-1293 issue and the LotsOfCores wiki page to > finally use a patched Solr 4.2.1 LotsOfCores in production (1 user = 1 > core). > We don't need anymore to run so many Solr per node. We are now able to > have around 50000 cores per Solr and we plan to grow to 100,000 cores > per instance. > In a first time, we used the solr.xml persistence. All cores have > loadOnStartup="false" and transient="true" attributes, so a cold start > is very quick. The response times were better than ever, in comparaison > with poor response times, we had before using LotsOfCores. > > We added 2 Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > > Then, to improve performance and avoid synchronization in the solr.xml > persistence : we disabled it. > The drawback is we cannot see anymore all the availables cores list with > the admin core status command, only those warmed up. > Finally, we can achieve very good performances with Solr LotsOfCores : > - Index 5 emails (avg) + commit + search : x4.9 faster response time > (Mean), x5.4 faster (95th per) > - Delete 5 documents (avg) : x8.4 faster response time (Mean) x7.4 > faster (95th per) > - Search : x3.7 faster response time (Mean) 4x faster (95th per) > > In fact, the better performance is mainly due to the little size of each > index, but also thanks to the isolation between cores (updates and > queries on many mailboxes don’t have side effects to each other). > One important thing with the LotsOfCores feature is to take care of : > - the number of file descriptors, it used a lot (need to increase global > max and per process fd) > - the value of the transientCacheSize depending of the RAM size and the > PermGen allocated size > - the leak of ClassLoader that increase minor GC times, when CMS GC is > enabled (use -XX:+CMSClassUnloadingEnabled) > - the overhead to parse solrconfig.xml and load dependencies to open > each core > - lotsOfCores doesn’t work with SolrCloud, then we store indexes > location outside of Solr. We have Solr proxies to route requests to the > right instance. > > Not in production, we try the core discovery feature in Solr 4.4 with a > lots of cores. > When you start, it spend a lot of times to discover cores due to a big > number of cores, meanwhile all requests fail (SolrDispatchFilter.init() > not done yet). It will be great to have for example an option for a core > discovery in background, or just to be able to disable it, like we do in > our use case. > > If someone is interested in these new options for LotsOfCores feature, > just tell me > > > Ce message et les pièces jointes sont confidentiels et réservés à l'usage > exclusif de ses destinataires. Il peut également être protégé par le secret > professionnel. Si vous recevez ce message par erreur, merci d'en avertir > immédiatement l'expéditeur et de le détruire. L'intégrité du message ne > pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra > être recherchée quant au contenu de ce message. Bien que les meilleurs > efforts soient faits pour maintenir cette transmission exempte de tout > virus, l'expéditeur ne donne aucune garantie à cet égard et sa > responsabilité ne saurait être recherchée pour tout dommage résultant d'un > virus transmis. > > This e-mail and the documents attached are confidential and intended > solely for the addressee; it may also be privileged. If you receive this > e-mail in error, please notify the sender immediately and destroy it. As > its integrity cannot be secured on the Internet, the Worldline liability > cannot be triggered for the message content. Although the sender endeavours > to maintain a computer virus-free network, the sender does not warrant that > this transmission is virus-free and will not be liable for any damages > resulting from any virus transmitted. > -- Regards, Shalin Shekhar Mangar.
