Hi Erick, Thanks for your mail, please let me go through with my use case. I am having around 20-40 Billion Records to index with each record is having around 200-400 fields, the data is sensor data so it can be easily stored in Integer or Float. Now to index this huge amount of data I am going with the indexing through EmbeddedSolrServer which was working fine but I was looking out for a way to move these generated indexes to different shards possibly without copying pasting it to each machines but some other approach as to submit this indexes to some shard and let the shard take care of it distributing it over leader and replica. I want to mention one more thing, as I started indexing with EmbeddedSolrServer it went fine for some million of starting documents but there after the indexing speed is pathetically slow, it indexed around 20GB in a day and just have indexed 9 GB in another 2 days. Any indexing optimization approach also requested.
Hope this makes things much clearer. Looking forward to soon hear from you. Thanks and Regards! On Fri, May 30, 2014 at 9:09 PM, Erick Erickson <erickerick...@gmail.com> wrote: > You can copy to the shards and use the mergindexes command, the > MapReduceIndexerTool follows that approach. > > But really, what is the higher-level use-case you're trying to support? > This feels a little like an XY problem. You could do things like > 1> index to a different collection then use collection aliasing to switch > 2> just re-index to the current collection. > 3> use the MapReduceIndexerTool (admittedly it needs Hadoop). > > All in all, it feels like you're doing work you don't need to do. But > that's a guess since you haven't told us what the use-case is. > > Best, > Erick > > > On Thu, May 29, 2014 at 7:22 AM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi, > > > > On Wed, May 28, 2014 at 4:25 AM, Vineet Mishra <clearmido...@gmail.com > > >wrote: > > > > > Hi All, > > > > > > Has anyone tried with building Offline indexes with EmbeddedSolrServer > > and > > > posting it to Shards. > > > > > > > What do you mean by "posting it to shards"? How is that different than > > copying them manually to the right location in FS? Could you please > > elaborate? > > > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > FYI, I am done building the indexes but looking out for a way to post > > these > > > index files on shards. > > > Copying the indexes manually to each shard's replica is possible and is > > > working fine but I don't want to go with that approach. > > > > > > Thanks! > > > > > >
