On 3/27/2012 11:14 AM, Mark Miller wrote:
On Mar 27, 2012, at 10:51 AM, Shawn Heisey wrote:
On 3/26/2012 6:43 PM, Mark Miller wrote:
It doesn't get thrown because that logic needs to continue - you don't
necessarily want one bad document to stop all the following documents from
being added. So the exception is sent to that method with the idea that you can
override and do what you would like. I've written sample code around stopping
and throwing an exception, but I guess its not totally trivial. Other ideas for
reporting errors have been thrown around in the past, but no work on it has
gotten any traction.
It looks like StreamingUpdateSolrServer is not meant for situations where strict error
checking is required. I think the documentation should reflect that. Would you be
opposed to a javadoc update at the class level (plus a wiki addition) like the following?
"Because document inserts are handled as background tasks, exceptions and errors
that occur during those operations will not be available to the calling program, but they
will be logged. For example, if the Solr server is down, your program must determine
this on its own. If you need strict error handling, use CommonsHttpSolrServer." If
my wording is bad, feel free to make suggestions.
It might make sense to accumulate the errors in a fixed-size queue and
report them either when the queue fills up or when the client commits
(assuming the commit will wait for all outstanding inserts to complete
or fail). This is what we do client-side when performing multi-threaded
inserts. Sounds great in theory, I think, but then I haven't delved in
to SUSS at all ... just a suggestion, take it or leave it. Actually I
wonder whether SUSS is necessary of you do the threading client-side?
You might get a similar perf gain; I know we see a substantial speedup
that way. because then your updates spawn multiple threads in the
server anyway, don't they?
- Mike
