Re: How to use batchSize in DataImportHandler to throttle updates in a batch-mode

2013-12-01 Thread Dileepa Jayakody
I actually tweaked the Stanbol server to handle more results and successfully ran 10K imports within 30 minutes with no server issue. I'm looking for further improving the results with regard to the efficiency and NLP accuracy. Thanks, Dileepa On Sun, Dec 1, 2013 at 8:17 PM, Dileepa Jayakody wro

Re: How to use batchSize in DataImportHandler to throttle updates in a batch-mode

2013-12-01 Thread Dileepa Jayakody
Thanks all, for your valuable ideas into this matter. I will try them. :) Regards, Dileepa On Sun, Dec 1, 2013 at 6:05 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > There is no support for throttling built into DIH. You can probably write a > Transformer which sleeps a while afte

Re: How to use batchSize in DataImportHandler to throttle updates in a batch-mode

2013-12-01 Thread Shalin Shekhar Mangar
There is no support for throttling built into DIH. You can probably write a Transformer which sleeps a while after every N requests to simulate throttling. On 26 Nov 2013 14:21, "Dileepa Jayakody" wrote: > Hi All, > > I have a requirement to import a large amount of data from a mysql database > a

Re: How to use batchSize in DataImportHandler to throttle updates in a batch-mode

2013-12-01 Thread William Bell
Well I think your issue is batchSize. batchSize="1" should be batchSize="-1" I also recommend you use *readOnly="true"* On Tue, Nov 26, 2013 at 1:50 AM, Dileepa Jayakody wrote: > Hi All, > > I have a requirement to import a large amount of data from a mysql database > and index documents (about

How to use batchSize in DataImportHandler to throttle updates in a batch-mode

2013-11-26 Thread Dileepa Jayakody
Hi All, I have a requirement to import a large amount of data from a mysql database and index documents (about 1000 documents). During indexing process I need to do a special processing of a field by sending a enhancement requests to an external Apache Stanbol server. I have configured my dataimpo