Well, personally I like to use SolrJ rather than DIH for both
debugging ease and the reasons outlined here:
https://lucidworks.com/2012/02/14/indexing-with-solrj/
FWIW
Erick
On Thu, Jun 1, 2017 at 7:59 AM, Josh Lincoln <josh.linc...@gmail.com> wrote:
> I had the same issue as Vrinda and found a hacky way to limit the number of
> times deltaImportQuery was executed.
>
> As designed, solr executes *deltaQuery* to get a list of ids that need to
> be indexed. For each of those it executes *deltaImportQuery*, which is
> typically very similar to the full *query*.
>
> I constructed a deltaQuery to purposely only return 1 row. E.g.
>
> deltaQuery = "SELECT id FROM table WHERE rownum=1" // written for
> oracle, likely requires a different syntax for other dbs. Also, it occurred
> to you could probably include the date>= '${dataimporter.last_index_time}'
> filter here so this returns 0 rows if no data has changed
>
> Since *deltaImportQuery now *only gets called once I needed to add the
> filter logic to *deltaImportQuery *to only select the changed rows (that
> logic is normally in *deltaQuery*). E.g.
>
> deltaImportQuery = [normal import query] WHERE date >=
> '${dataimporter.last_index_time}'
>
>
> This significantly reduced the number of database queries for delta
> imports, and sped up the processing.
>
> On Thu, Jun 1, 2017 at 6:07 AM Amrit Sarkar <sarkaramr...@gmail.com> wrote:
>
>> Erick,
>>
>> Thanks for the pointer. Getting astray from what Vrinda is looking for
>> (sorry about that), what if there are no sub-entities? and no
>> deltaImportQuery passed too. I looked into the code and determine it
>> calculates the deltaImportQuery itself,
>> SQLEntityProcessor:getDeltaImportQuery(..)::126.
>>
>> Ideally then, a full-import or the delta-import should take similar time to
>> build the docs (fetch next row). I may very well be going entirely wrong
>> here.
>>
>> Amrit Sarkar
>> Search Engineer
>> Lucidworks, Inc.
>> 415-589-9269 <(415)%20589-9269>
>> www.lucidworks.com
>> Twitter http://twitter.com/lucidworks
>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>>
>> On Thu, Jun 1, 2017 at 1:50 PM, vrindavda <vrinda...@gmail.com> wrote:
>>
>> > Thanks Erick,
>> >
>> > But how do I solve this? I tried creating Stored proc instead of plain
>> > query, but no change in performance.
>> >
>> > For delta import it in processing more documents than the total
>> documents.
>> > In this case delta import is not helping at all, I cannot switch to full
>> > import each time. This was working fine with less data.
>> >
>> > Thank you,
>> > Vrinda Davda
>> >
>> >
>> >
>> > --
>> > View this message in context: http://lucene.472066.n3.
>> > nabble.com/Number-of-requests-spike-up-when-i-do-the-delta-
>> > Import-tp4338162p4338444.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>>
