You can override the method that logs the error - then parse the msg
for the doc ids?
On Tue, Sep 4, 2012 at 6:03 AM, Kissue Kissue wrote:
> Hi Lance,
>
> As far as i can see, one document failing does not fail the entire update.
> From my logs i can see the error logged in the logs but indexing
Hi Lance,
As far as i can see, one document failing does not fail the entire update.
>From my logs i can see the error logged in the logs but indexing just
continues to the next document. This happens with the
StreamingUpdateSolrServer which is multithreaded.
Thanks.
On Tue, Jun 19, 2012 at 9:58
When one document fails, the entire update fails, right? Is there now
a mode where successful documents are added and failed docs are
dropped?
If you want to know if a document is in the index, search for it!
There is no other guaranteed way.
On Sun, Jun 17, 2012 at 3:14 PM, Jack Krupansky wrote
You should also call the glue code ;-):
Protocol.registerProtocol("http", http);
regards
Torsten
smime.p7s
Description: S/MIME cryptographic signature
AddOn: You can even set a custom http factory for commons-http (which is
used by SolrStreamingUpdateServer) at all to influence socket options,
example is:
final Protocol http = new Protocol("http",
MycustomHttpSocketFactory.getSocketFactory(), 80);
and MycustomHttpSocketFactory.getSocketFactory
Am Freitag, den 15.06.2012, 18:22 +0100 schrieb Kissue Kissue:
> Hi,
>
> Does anybody know what the default connection timeout setting is for
> StreamingUpdateSolrServer? Can i explicitly set one and how?
>
> Thanks.
Use a custom HttpClient to set one (only snippets, should be clear, if
not tel
You could instantiate an anonymous instance of StreamingUpdateSolrServer
that has a "handleError" method that then parses the exception message to
get the request URI. If there isn't enough information there, you could add
a dummy request option to your original request that was a document
iden
Thanks Sami. Has anybody had any need to explicitly set the connection
timeout? Just trying to understand how folks use it.
Thanks.
On Fri, Jun 15, 2012 at 7:01 PM, Sami Siren wrote:
> The api doc for version 3.6.0 is available here:
>
> http://lucene.apache.org/solr/api/org/apache/solr/client/
The api doc for version 3.6.0 is available here:
http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html
I think the default is coming from your OS if you are not setting it explicitly.
--
Sami Siren
On Fri, Jun 15, 2012 at 8:22 PM, Kissue Kissue wrot
On 3/27/2012 11:14 AM, Mark Miller wrote:
On Mar 27, 2012, at 10:51 AM, Shawn Heisey wrote:
On 3/26/2012 6:43 PM, Mark Miller wrote:
It doesn't get thrown because that logic needs to continue - you don't
necessarily want one bad document to stop all the following documents from
being added.
https://issues.apache.org/jira/browse/SOLR-445
This JIRA reflects the slightly different case of wanting better
reporting of *which* document failed in a multi-document packet, it
doesn't specifically address SUSS. But it might serve to give you some
ideas if you tackle this.
On Tue, Mar 27, 2012
On Mar 27, 2012, at 10:51 AM, Shawn Heisey wrote:
> On 3/26/2012 6:43 PM, Mark Miller wrote:
>> It doesn't get thrown because that logic needs to continue - you don't
>> necessarily want one bad document to stop all the following documents from
>> being added. So the exception is sent to that m
On 3/26/2012 6:43 PM, Mark Miller wrote:
It doesn't get thrown because that logic needs to continue - you don't
necessarily want one bad document to stop all the following documents from
being added. So the exception is sent to that method with the idea that you can
override and do what you wo
Like I said, you have to extend the class and override the error method.
Sent from my iPhone
On Mar 27, 2012, at 2:29 AM, Shawn Heisey wrote:
> On 3/26/2012 10:25 PM, Shawn Heisey wrote:
>> The problem is that I currently have no way (that I know of so far) to
>> detect that a problem happene
On 3/26/2012 10:25 PM, Shawn Heisey wrote:
The problem is that I currently have no way (that I know of so far) to
detect that a problem happened. As far as my code is concerned,
everything worked, so it updates my position tracking and those
documents will never be inserted. I have not yet de
On 3/26/2012 6:43 PM, Mark Miller wrote:
It doesn't get thrown because that logic needs to continue - you don't
necessarily want one bad document to stop all the following documents from
being added. So the exception is sent to that method with the idea that you can
override and do what you wo
It doesn't get thrown because that logic needs to continue - you don't
necessarily want one bad document to stop all the following documents from
being added. So the exception is sent to that method with the idea that you can
override and do what you would like. I've written sample code around s
On 3/20/2012 8:11 PM, Chris Hostetter wrote:
:> Is there any way to get get the threads within SUSS objects to immediately
:> exit without creating other issues? Alternatively, if immediate isn't
:> possible, the exit could take 1-2 seconds. I could not find any kind of
:> method in the API
: > Is there any way to get get the threads within SUSS objects to immediately
: > exit without creating other issues? Alternatively, if immediate isn't
: > possible, the exit could take 1-2 seconds. I could not find any kind of
: > method in the API that closes down the object.
you should take
On 3/15/2012 5:53 PM, Shawn Heisey wrote:
Is there any way to get get the threads within SUSS objects to
immediately exit without creating other issues? Alternatively, if
immediate isn't possible, the exit could take 1-2 seconds. I could
not find any kind of method in the API that closes down
Hi all,
I’m having basically the exact same problem someone described in this email to
the list from just over a year ago (see below). The only suggested solution
given on the thread at the time was to ping the server before sending an add,
which I’m not particularly keen on; least of all becaus
On Tue, Sep 6, 2011 at 6:56 PM, simon wrote:
> If you're batching the documents when you send them to Solr with the #add
> method, you may be out of luck - Solr doesn't do a very good job of
> reporting which document in a batch caused the failure.
>
> If you reverted to CommonsHTTPServer and adde
If you're batching the documents when you send them to Solr with the #add
method, you may be out of luck - Solr doesn't do a very good job of
reporting which document in a batch caused the failure.
If you reverted to CommonsHTTPServer and added a doc at a time there
wouldn't be any ambiguity, but
Hi Mark,
The implementation is logging anyway, we have subclassed
StreamingUpdateSolrServer and used handleError to log, but inspecting the
stack trace in in the handleError method
does not give any clue about the document(s) that failed. We have a solution
that uses Solr as backend for indexing
The default impl logs with slf4j - just setup logging properly and you will see
the results?
Alternatively, you can subclass and impl that method however you'd like.
On Sep 5, 2011, at 6:36 PM, Leonardo Souza wrote:
> Hi,
>
> Inspecting StreamingUpdateSolrServer#handleError i can't see how to
: Invalid Date Math String:'2011-04-22T05:35:37Z '
...
: I notice the trailing spaces but in my document they are not present:
...
:
...
: I am adding the field to the document with:
...
: document.addField(element, sb); //sb is a string
There is at least one
Yes. Each thread uses its own connection, and each becomes a new
thread in the servlet container.
On Mon, Mar 7, 2011 at 2:54 AM, Isan Fulia wrote:
> Hi all,
> I am using StreamingUpdateSolrServer with queuesize = 5 and threadcount=4
> The no. of connections created are same as threadcount.
> Is
My current solution is to use the ping() function -- which doesn't run in a
thread -- to test the connection before trying to send the data to the Solr
index. It isn't elegant, but it works.
If anyone has a better idea, I'd like to hear it.
-- Chris
On Thu, Dec 30, 2010 at 11:10 AM, Christophe
On Thu, Apr 29, 2010 at 7:51 PM, Yonik Seeley
wrote:
> I'm trying to reproduce now... single thread adding documents to a
> multithreaded client, StreamingUpdateSolrServer(addr,32,4)
>
> I'm currently at the 2.5 hour mark and 100M documents - no issues so far.
I let it go to 500M docs... everythi
I'm trying to reproduce now... single thread adding documents to a
multithreaded client, StreamingUpdateSolrServer(addr,32,4)
I'm currently at the 2.5 hour mark and 100M documents - no issues so far.
-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague
On Thu, Apr 29, 2010 at 5:12 PM, Yon
What is the garbage collection status when this happens?
What are the open sockets in the OS when this happens? Run 'netstat
-an | fgrep 8983' where 8983 is the Solr incoming port number.
A side note on sockets:
SUSS uses the MultiThreadedHttpConnectionManager but never calls
MultiThreadedHttpCo
On Thu, Apr 29, 2010 at 6:04 PM, Lance Norskog wrote:
> In solrconfig.xml, there is a parameter controlling remote streaming:
>
>
> multipartUploadLimitInKB="2048000" />
>
> 1) Is this relevant with the SUSS?
No, this relates to solr pulling data from another source (via stream.url
In solrconfig.xml, there is a parameter controlling remote streaming:
1) Is this relevant with the SUSS?
2) It seems to be 'true' in the example default, which may not be a good idea.
On Thu, Apr 29, 2010 at 2:12 PM, Yonik Seeley
wrote:
> On Fri, Apr 16, 2010 at 1:34 PM, Sascha
On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott wrote:
> In my case the whole application hangs and never recovers (CPU utilization
> goes down to near 0%). Interestingly, the problem reproducibly occurs only
> if SUSS is created with *more than 2* threads.
Is your application also using multiple t
I experienced the hang described with the Solr 1.4.0 build.
Yonik - I also thought the streaming updater was blocking on commits but
updates never resumed.
To be honest I was in a bit of a rush to meet a deadline so after spending a
day or so tinkering I bailed out and just wrote a component by h
Hi Yonik,
thanks for your fast reply.
Yonik Seeley wrote:
Thanks for the report Sascha.
So after the hang, it never recovers? Some amount of hanging could be
visible if there was a commit on the Solr server or something else to
cause the solr requests to block for a while... but it should retu
Thanks for the report Sascha.
So after the hang, it never recovers? Some amount of hanging could be
visible if there was a commit on the Solr server or something else to
cause the solr requests to block for a while... but it should return
to normal on it's own...
Looking at the stack trace, it lo
Hi Yonik,
Yonik Seeley wrote:
Stephen, were you running stock Solr 1.4, or did you apply any of the
SolrJ patches?
I'm trying to figure out if anyone still has any problems, or if this
was fixed with SOLR-1711:
I'm using the latest trunk version (rev. 934846) and constantly running
into the sam
Stephen, were you running stock Solr 1.4, or did you apply any of the
SolrJ patches?
I'm trying to figure out if anyone still has any problems, or if this
was fixed with SOLR-1711:
* SOLR-1711: SolrJ - StreamingUpdateSolrServer had a race condition that
could halt the streaming of documents. (At
On Mon, Jan 25, 2010 at 7:27 PM, Jake Brownell wrote:
> I swapped our indexing process over to the streaming update server, but now
> I'm seeing places where our indexing code adds several documents, but
> eventually hangs. It hangs just before the completion message, which comes
> directly aft
I'll have to defer that one for now.
2010/1/26 Tim Terlegård
> 2010/1/26 Erick Erickson :
> > > My indexing script has been running all
> > > night and has accomplished nothing. I see lots of disk activity
> > > though, which is weird.
> >
> >
> > One explanation would be that you're memory-
2010/1/26 Erick Erickson :
> > My indexing script has been running all
> > night and has accomplished nothing. I see lots of disk activity
> > though, which is weird.
>
>
> One explanation would be that you're memory-starved and
> the disk activity you see is thrashing. How much memory
> do you all
<<< My indexing script has been running all
night and has accomplished nothing. I see lots of disk activity
though, which is weird.>>>
One explanation would be that you're memory-starved and
the disk activity you see is thrashing. How much memory
do you allocate to your JVM? A further indication
2010/1/26 Jake Brownell :
> I swapped our indexing process over to the streaming update server, but now
> I'm seeing places where our indexing code adds several documents, but
> eventually hangs. It hangs just before the completion message, which comes
> directly after sending to solr. I found
Unless I slept through it, you still need to explicitly commit, even with SUSS.
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
- Original Message
> From: "erikea...@yahoo.com"
> To: "solr-user@lucene.
gt;> System.out.println(this.getName()+"
>> optimized");
>> } catch (Exception e) {
>> e.printStackTrace();
>> System.err.println(&qu
server.optimize();
>> System.out.println(this.getName()+"
>> optimized");
>> } catch (Exception e) {
>> e.printStackTrace();
>>
e.printStackTrace();
>System.err.println("optimizer
> "+e.getLocalizedMessage());
>System.exit(0);
> }
>}
>
System.err.println("optimizer
"+e.getLocalizedMessage());
System.exit(0);
}
}
}
}.start();
} catch (Exception e) {
e.printStackTrace();
}
}
-Original Message-
Which Java release is this? There are known thread-blocking problems
in Java 1.5.
Also, what sockets are used during this time? Try 'netstat -s | fgrep
8983' (or your Solr URL port #) and watch the active, TIME_WAIT,
CLOSE_WAIT sockets build up. This may give a hint.
On Tue, Oct 13, 2009 at 8:47
On Thu, Apr 16, 2009 at 3:45 AM, Marc Sturlese wrote:
>
> Hey there,
> I have been reading about StreamingUpdateSolrServer but can't catch exactly
> how it works:
>
> More efficient index construction over http with solrj. If your doing it,
> this is a fantastic performance improvement.
StreamingU
51 matches
Mail list logo
