Re: Performance of Bulk Importing TSV File in Solr 8

2020-01-10 Thread Paras Lehana
Hi Joseph, Although your indexing rate is fast at around 2800 docs/sec, you can play with values of autoCommit, mergePolicy and ramBufferSize. You can post existing values of these to make us comment about those. As Mikhail suggested, batches can increase performance by committing in between. O

Re: Performance of Bulk Importing TSV File in Solr 8

2020-01-02 Thread Mikhail Khludnev
Hello, Joseph. This rate looks good to me, although if the node is idling and has a plenty of free RAM, you can dissect this file by unix tools and submit these partitions for import in parallel. Hanging connection seems like a bug. On Thu, Jan 2, 2020 at 10:09 PM Joseph Lorenzini wrote: > Hi

Performance of Bulk Importing TSV File in Solr 8

2020-01-02 Thread Joseph Lorenzini
Hi all, I have TSV file that contains 1.2 million rows. I want to bulk import this file into solr where each row becomes a solr document. The TSV has 24 columns. I am using the streaming API like so: curl -v ' http://localhost:8983/solr/example/update?stream.file=/opt/solr/results.tsv&separator=%