On 8/31/2016 1:54 PM, Stahle, Patrick wrote:
> I am having problems getting the delta import working. Full import
> works fine. I am using current version of solr (6.1). I have been
> looking at this pretty much all day and can't find what I am not doing
> correctly... I did try the Using query attribute for both full and
> delta import and that worked, but as soon I ran it for a full import
> via clean=true my queries performance went very bad (oracle execution
> plain must of went bonkers). Anyways, I would appreciate any help.
One possibility for performance issues with programs that use JDBC: The
JDBC driver may be buffering the entire result set in memory before
releasing it to the dataimport handler. Oracle may have a JDBC option
that causes it to stream results as they are requested, rather than
buffer them. Upgrading the driver jar may be required. I found a
document saying that version 12c of the oracle driver does a much better
job than earlier versions when it comes to memory management.
> That exact same query executed in dbeaver SQL client:
<snip>
> SELECT COUNT(bo.LXOID) FROM lxbo_current bo WHERE bo.LXMODDATE >
> TO_DATE('2016-08-28 19:28:07', 'yyyy-mm-dd HH24:MI:SS')
> 250
Before I discovered that dbeaver is a Java program, these were two ideas
that I had about the difference in rowcount between Solr and dbeaver:
1) The user in the JDBC connection details is somehow blocked from
seeing the matching records in the database. 2) There's a bug in the
JDBC driver you're using that causes the query to return zero rows.
Since dbeaver is Java, just like Solr, those possibilities seem less
likely, but if the following doesn't help, you should explore them.
I did notice that the value for pk that you have chosen (ID) doesn't
show up in deltaQuery. The field there seems to be "id" ... which will
be a different field than "ID". I wonder if maybe Solr is skipping
those rows because they don't have the defined pkfield? Try changing
the field name in deltaQuery to uppercase, and make the back-reference
in deltaImportQuery match it. I don't know if that's going to help, but
it's an idea.
Thanks,
Shawn
