There is no support for throttling built into DIH. You can probably write a Transformer which sleeps a while after every N requests to simulate throttling. On 26 Nov 2013 14:21, "Dileepa Jayakody" <dileepajayak...@gmail.com> wrote:
> Hi All, > > I have a requirement to import a large amount of data from a mysql database > and index documents (about 1000 documents). > During indexing process I need to do a special processing of a field by > sending a enhancement requests to an external Apache Stanbol server. > I have configured my dataimport-handler in solrconfig.xml to use the > StanbolContentProcessor in the update chain, as below; > > *<updateRequestProcessorChain name="stanbolInterceptor">* > * <processor > class="com.solr.stanbol.processor.StanbolContentProcessorFactory"/>* > * <processor class="solr.RunUpdateProcessorFactory" />* > * </updateRequestProcessorChain>* > > * <requestHandler name="/dataimport" class="solr.DataImportHandler"> * > * <lst name="defaults"> * > * <str name="config">data-config.xml</str>* > * <str name="update.chain">stanbolInterceptor</str>* > * </lst> * > * </requestHandler>* > > My sample data-config.xml is as below; > > *<dataConfig>* > *<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" > url="jdbc:mysql://localhost:3306/solrTest" user="test" password="test123" > batchSize="1" />* > * <document name="stanboldata">* > * <entity name="stanbolrequest" query="SELECT * FROM documents">* > * <field column="id" name="id" />* > * <field column="content" name="content" />* > * <field column="title" name="title" />* > * </entity>* > * </document>* > *</dataConfig>* > > When running a large import with about 1000 documents, my stanbol server > goes down, I suspect due to heavy load from the above Solr > Stanbolnterceptor. > I would like to throttle the dataimport in batches, so that Stanbol can > process a manageable number of requests concurrently. > Is this achievable using batchSize parameter in dataSource element in the > data-config? > Can someone please give some ideas to throttle the dataimport load in Solr? > > Thanks, > Dileepa >
