Hello,
- embedded server is not the best way, usually - lucene perfectly indexes in multiple thread concurrently. Single writer per directory is called concurrently. - with solrj you can use ConcurrentUpdateSolr server, or call StreamingUpdateSolrServer in multiple threads, or just updates docs in parallel through plain SolrServer - Also, there is SOLR-3585 it adds server-side concurrency for handling long single thread requests (it's intended to work with StreamingUpdateSolrServer). - if you want to distribute your indexes it's what SolrCloud is done for, then you can search these indices in parallel. - kind of esoteric to me, after you build indexes distributed you can try to merge them in the single solid one http://wiki.apache.org/solr/MergingSolrIndexes - NFS almost never provides enough consistency, ie. they are hardly useful for indexing. - one million is a fairly small amount, in average it should be indexed in few mins. I doubt that you really need to distribute indexing. On Wed, Aug 22, 2012 at 8:53 AM, ksu wildcats <ksu.wildc...@gmail.com>wrote: > We have a webapp that has embedded solr integrated in it. > It essentially handles creating separate index (core) per client and it is > currently setup such that there can only be one index write operation per > core. > Say if we have 1 Million documents that needs be to Indexed, our app reads > each document and writes it to index (using embedded solr library). > > I am looking into ways to speed up indexing time and I was wondering if it > would be possible to have our app run on multiple servers and each server > process indexing docs concurrently. I was thinking of having Index storage > on NFS that can be accessed by all servers. > > I am not entirely sure but reading through documentation my understanding > is > that we cannot have multiple index writers (even if they are running on > different servers) write to same index directory simultaneously. is that > correct? > > If there is a limitation on concurrent writes to same index directory then > do i need to have each server build a separate index (more like a cores > within core) and merge all the sub indexes into main index to speed up the > indexing time? > > Please let me know if am heading in correct path or if there are better > alternatives to speed up indexing time? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-Index-Concurrency-Is-it-possible-to-have-multiple-threads-write-to-same-index-tp4002544.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Sincerely yours Mikhail Khludnev Tech Lead Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
