Delta-Import not loading child entities?
I'm running into an issue with the SqlEntityProcessor where full imports are running fine, but delta imports don't load the child entities. More specifically, if the parent deltaQuery returns some records, then its deltaImportQuery runs, but the child entities' queries don't run like they would for a full import. My question is, should the child entities be loaded during delta imports, or do I need to do the join in my parent object's deltaImportQuery? I'm running Solr 4.8.1 and I'm using the SortedMapBackedCache for the child entities. Thanks, Drew
Re: Delta-Import not loading child entities?
A little more info... removing the cache fixes everything. Are delta queries incompatible with the cache? There isn't a lot of current documentation on this, far as I can tell. Thanks, Drew On Fri, Jun 13, 2014 at 2:46 PM, Drew Mazurek wrote: > I'm running into an issue with the SqlEntityProcessor where full imports > are running fine, but delta imports don't load the child entities. More > specifically, if the parent deltaQuery returns some records, then its > deltaImportQuery runs, but the child entities' queries don't run like they > would for a full import. > > My question is, should the child entities be loaded during delta imports, > or do I need to do the join in my parent object's deltaImportQuery? > > I'm running Solr 4.8.1 and I'm using the SortedMapBackedCache for the > child entities. > > Thanks, > Drew > >
Re: Delta-Import not loading child entities?
Sure... here's the document stanza, simplified: In the log, I'm seeing the "profile" deltaQuery run (which in testing is correctly returning 0 records), and then the "object" deltaQuery runs, which returns 1 record. Then it logs that the delta import is complete, but it never calls any SQL related to the profile entity. One thing to note, if you're wondering... the CASTs are required because it seems the SortedMapBackedCache wants Strings for keys. In the database, my keys are MySQL BIGINTs. Without the casts, Java ClassCastExceptions are thrown on import. Thanks, Drew On Fri, Jun 13, 2014 at 3:37 PM, Gora Mohanty wrote: > On 14 June 2014 00:36, Drew Mazurek wrote: > > > > A little more info... removing the cache fixes everything. Are delta > > queries incompatible with the cache? There isn't a lot of current > > documentation on this, far as I can tell. > > Not quite au courant myself with 4.8.1 (need to install new Jave > somewhere), > but it would help to post your DIH configuration file. > > Regards, > Gora >
Re: Delta-Import not loading child entities?
I thought about this, and is it possible that for delta imports Solr is
expecting a persistent cache, like the old BerkleyBackedCache? I can't
imagine any other reason why it wouldn't run the subentity queries on delta
imports.
- Drew
On Fri, Jun 13, 2014 at 4:37 PM, Drew Mazurek wrote:
> Sure... here's the document stanza, simplified:
>
>
>
> deltaQuery="select object_id from objects o where
> o.object_type in (2,3)
> AND o.date_modified > '${dataimporter.last_index_time}'"
>
> deltaImportQuery="SELECT o.object_id,o.object_name,
> CAST(p.profile_id AS CHAR) AS owner_profile_id
> FROM `objects` o
> JOIN containers c on c.container_id=o.container_id
> LEFT JOIN profiles p on o.created_by=p.user_id AND
> c.org_id=p.org_id
> WHERE o.object_id=${dataimporter.delta.object_id}">
>
> cacheImpl="SortedMapBackedCache"
> where="profile_id=object.owner_profile_id"
>
> query="select CAST(profile_id AS CHAR) AS profile_id,
> user_id as owner_id,first_name as owner_first_name,
> last_name as owner_last_name,preferred_name as
> owner_preferred_name
> from profiles"
>
> deltaQuery="select profile_id,user_id from profiles where
> date_modified > '${dataimporter.last_index_time}'"
>
> parentDeltaQuery="select object_id from objects where
> created_by='${profile.owner_id}'"/>
>
>
>
>
> In the log, I'm seeing the "profile" deltaQuery run (which in testing is
> correctly returning 0 records), and then the "object" deltaQuery runs,
> which returns 1 record. Then it logs that the delta import is complete,
> but it never calls any SQL related to the profile entity.
>
> One thing to note, if you're wondering... the CASTs are required because
> it seems the SortedMapBackedCache wants Strings for keys. In the database,
> my keys are MySQL BIGINTs. Without the casts, Java ClassCastExceptions are
> thrown on import.
>
> Thanks,
> Drew
>
>
>
>
>
> On Fri, Jun 13, 2014 at 3:37 PM, Gora Mohanty wrote:
>
>> On 14 June 2014 00:36, Drew Mazurek wrote:
>> >
>> > A little more info... removing the cache fixes everything. Are delta
>> > queries incompatible with the cache? There isn't a lot of current
>> > documentation on this, far as I can tell.
>>
>> Not quite au courant myself with 4.8.1 (need to install new Jave
>> somewhere),
>> but it would help to post your DIH configuration file.
>>
>> Regards,
>> Gora
>>
>
>
