1) You may have a small primary table but for each ID in it, you seem to be calling another 6 tables with nested SQL queries. Perhaps you need to cache those calls: http://wiki.apache.org/solr/DataImportHandler#CachedSqlEntityProcessor
2) You seem to be double-dipping into the main table tab_story2 in the nested entity, perhaps there is a way to avoid that 3) You are sorting the main table in the outside query. Why? You are going to process every record anyway. 4) Auto-commit is probably way too expensive here. Try setting it to every 2 minutes without changing anything else and see how many more entities you process in the same X minutes. In Solr 4+, there are better options for commit. Regards, Alex On Wed, Apr 24, 2013 at 3:25 PM, srinalluri <nallurisr...@yahoo.com> wrote: > Hi, > > Environment is Solr 3.6.1. The database is having enough indexes. The box is > having enough memory. The DB is performance is good. Auto commit is enabled > for every 1 minute. > Please see the following entity. The full-import of this entity is taking > over 48 hours to complete on production environment. The number records in > the main table is around 700,000 only. I tried materialized view, but that > view is having duplicate records. So I can't go with materialized view for > all these queries. > Can someone please suggest how to improve the performance for full-import? > > <entity name="oracle-article" dataSource="oracle" pk="VCMID" > preImportDeleteQuery="content_type:article AND repository:oracleqa" Personal blog: http://blog.outerthoughts.com/ LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch - Time is the quality of nature that keeps events from happening all at once. Lately, it doesn't seem to be working. (Anonymous - via GTD book)
