Re: SOLR takes more than 9 hours to index 300000 rows

trix DB as backend and I am trying to find out >> the >> parameter for setting the JDBC fetch size.. >> >> Do you think that this problem will be mostly due to fetch size? >> >> Thanks, >> Barani >> > > > -- View this message in context: http://old.nabble.com/SOLR-takes-more-than-9-hours-to-index-30-rows-tp27805403p27825172.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR takes more than 9 hours to index 300000 rows

gt;> Do you think that this problem will be mostly due to fetch size? >> >> Thanks, >> Barani >> > > > -- View this message in context: http://old.nabble.com/SOLR-takes-more-than-9-hours-to-index-30-rows-tp27805403p27806552.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR takes more than 9 hours to index 300000 rows

Do keep looking into the batchSize, but I think I might have found the issue. If I understand things correctly, you will need to add processor="CachedSqlEntityProcessor" to your first entity. It's only specified on the other two. Assuming you have enough RAM and heap space available in your

Re: SOLR takes more than 9 hours to index 300000 rows

gt;> as rship, b.RepositoryName as rrepname from >> MetaModel.POC.BinaryRelationShip a, MetaModel.POC.Repository b where >> a.Object2RepositoryId=b.repositoryId" --> 3 seconds >> >> As I am using CachedSqlEntityProcessor I assume that SOLR first issues >> these >> select statements (mentioned above first) and then it match based on >> cacheKey (from caching), so SOLR should ideally take (addition of time >> taken to execute the above 3 queries + some time for doing filtering >> based >> on cacheKey ). But in my case its taking hours and hours for indexing. >> >> Can someone please let me know if I am doing anything wrong which might >> cause this issue? >> > > > -- View this message in context: http://old.nabble.com/SOLR-takes-more-than-9-hours-to-index-30-rows-tp27805403p27806375.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR takes more than 9 hours to index 300000 rows

I have now learned more about the CachedSqlEntityProcessor, so in theory it should be done with the server connections, but if it were me, I'd verify that. How big are the resultsets from those queries? SOLR has to put all the table data into RAM, which is very likely going to take considerab

Re: SOLR takes more than 9 hours to index 300000 rows

At the 9+ hour mark, is your database server showing active connections that are sending data, or is all the activity local to SOLR? We have a 40 million row database in MySQL, with each row comprising more than 80 fields. I'm including the config from one of our shards. There are about 6.6

SOLR takes more than 9 hours to index 300000 rows

ght cause this issue? Thanks, Barani -- View this message in context: http://old.nabble.com/SOLR-takes-more-than-9-hours-to-index-30-rows-tp27805403p27805403.html Sent from the Solr - User mailing list archive at Nabble.com.