My very first question is whether you're _absolutely sure_ that the
indexes you're merging have the same hash range. If not you're in for
a world of hurt.
You might try merging on local disks rather than in HDFS...
Best,
Erick
On Wed, Sep 20, 2017 at 11:05 AM, Avinash Patil
<avinash.pa...@securonix.com> wrote:
>
>
> I have a lot of data coming in SolrCloud and we create multiple collections
> dynamically after a collection threshold is reached.Currently to maintain
> fast search response speeds after 100M docs a new collection is triggered(
> 300G in HDFS ) . After SolrCloud ( CDH solr 4.10.3) reaches 150 - 200
> collections . So i am trying merge 10 multiple collections to a single new
> collection shard using merge api in solrj. But the merge index api is running
> very slow i.e almost 5 mins to merge a collection a single shard.
>
> CoreAdminResponse mergeIndexes = CoreAdminRequest.mergeIndexes(destShard,
> arr, new String[0], secClient);
> LOGGER.debug(" merge response - {} ", mergeIndexes);
> Thread.sleep(1000l);
> secClient.commit(destCollection);
>
> I need information on if there is a way to make the merge faster . I tried
> optimizing and committing a collection before we run the merge.
>
> Also any information on how merge runs in the background (does it copy the
> entire index folder ? ) will also be useful
>
>
>
> Regards
> Avinash Patil
> Software Engineer
> Securonix
> Security Analytics. Delivered. <http://www.securonix.com/>
> Mobile:
> Email:
> Winner of 12 2016 Information Security Product Guide Global Excellence Awards
> <http://www.securonix.com/securonix-wins-12-info-security-products-guide-global-excellence-awards/>
> Winner of Seven Golden Bridge Awards
> <http://www.securonix.com/security-analytics-pioneer-securonix-wins-seven-golden-bridge-awards-including-grand-trophy-winner/>
> Named 2015 Innovator in Cyber Threat Analysis and Intelligence by SC Magazine
>
> <http://www.scmagazine.com/cyberthreat-analysis-and-intelligence-innovators-2015/article/458248/5/>
>
>
>
>
>
> --
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law. If
> you are not the intended recipient, you should delete this message and any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, by you is strictly prohibited.
