Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: ...The rationale with the solrconfig stuff is that a broken config should behave as best it can. This is great if you are running a real site with people actively using it - it is a pain in the ass if you are getting started and don't notice errors I think it's a PITA in any case, I like my systems to fail loudly when something's wrong in the configs (with details about what's happening, of course). -Bertrand
Re: merely a suggestion: schema.xml validator or better schema validation logging
Bertrand Delacretaz wrote: On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: ...The rationale with the solrconfig stuff is that a broken config should behave as best it can. This is great if you are running a real site with people actively using it - it is a pain in the ass if you are getting started and don't notice errors I think it's a PITA in any case, I like my systems to fail loudly when something's wrong in the configs (with details about what's happening, of course). -Bertrand I think it's interesting seeing the difference. The system at CNET obviously needed to fail gracefully before it needed to fail fast. I have the luxury of a dev environment and fail-fast is exactly the kinda thing I want so I know about as many limitations and problems as soon as possible. Having this behavior toggled would be idea. Version the solrconfig.xml between a fail-graceful for your production branch and a fail-fast for your dev branch. Jed
JVM random crashes
Hi everybody. I've been using Solr for more than two months and I was really happy with it since yesterday. I moved my application to a faster machine and everything went wrong. I have a PHP script, which uses libcurl to post records to Solr. It's sending 100 records at a time. After all 2500 records have been posted, the script sends a and . I'm running the script several times and the JVM crashes randomly. Unforunately, I have been unable to reproduce this bug on another machine. Right now I do not have access to that computer, and I'm also bothered that I can't reboot it from a PuTTY terminal. But tomorrow, I'll at least figure out what's going on with the reboot. I tried using JDK 1.5.0_05, 1.5.0_10, 1.5.0_11, and 1.6.0. I used the Java -server mode, tried increasing the heap size, but the JVM still crashes. I tried Solr 1.1 and two of the last nightly builds but the problem remains. What's amazing is that there is no such problem on two other machines that have the same configuration - FC6, Tomcat 5.5.20, etc. By the way, I edited Solr's web.xml to password protect solr/update and solr/admin, but I don't think this causes the problem. I have absolutely no clue what is going on. Could it be a problem in Solr's schema or configuration? Or maybe a bug in JVM, or Solr, or Tomcat ? Any help would be greatly appreciated. Here are two of the error logs : # # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x061a8681, pid=26358, tid=3051350928 # # Java VM: Java HotSpot(TM) Client VM (1.6.0-b105 mixed mode, sharing) # Problematic frame: # V [libjvm.so+0x1a8681] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x095a3c00): VMThread [id=26360] siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x2068 Registers: EAX=0x2010, EBX=0x0400, ECX=0x, EDX=0x0001 ESP=0xb5dfda64, EBP=0xb5dfda68, ESI=0x8e9b0090, EDI=0x8e96e688 EIP=0x061a8681, CR2=0x2068, EFLAGS=0x00010212 Top of Stack: (sp=0xb5dfda64) 0xb5dfda64: 0400 b5dfda98 062e4d39 0400 0xb5dfda74: 8e9b0090 0003 0003 95090da8 0xb5dfda84: 8e1eb710 0001 0003 8e96e684 0xb5dfda94: 908be858 b5dfdac8 061e5330 8e96e684 0xb5dfdaa4: 8e127efc b5dfdac8 062fab3d 8e127ed4 0xb5dfdab4: b566e405 908be860 90220370 8ea23d74 0xb5dfdac4: 064366b4 b5dfdae8 062e4e4f 908be6e0 0xb5dfdad4: 8e96e678 b5dfdb08 0003 06421560 Instructions: (pc=0x061a8681) 0x061a8671: 74 2c 8b 45 0c ba 01 00 00 00 8b 40 04 83 c0 08 0x061a8681: 8b 40 58 83 e0 07 83 f8 05 74 13 31 d2 49 75 09 Stack: [0xb5d7e000,0xb5dff000), sp=0xb5dfda64, free space=510k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1a8681] V [libjvm.so+0x2e4d39] V [libjvm.so+0x1e5330] V [libjvm.so+0x2e4e4f] V [libjvm.so+0x328db7] V [libjvm.so+0x328fef] V [libjvm.so+0x328a70] V [libjvm.so+0x1d22e7] V [libjvm.so+0x1d1a92] V [libjvm.so+0x1da23a] V [libjvm.so+0x1cf744] V [libjvm.so+0x183d45] V [libjvm.so+0x1cfacc] V [libjvm.so+0x3aed8e] V [libjvm.so+0x3c0f17] V [libjvm.so+0x3c059c] V [libjvm.so+0x3c076a] V [libjvm.so+0x3c034f] V [libjvm.so+0x3066b3] C [libpthread.so.0+0x53db] VM_Operation (0xb4e66260): generation collection for allocation, mode: safepoint, requested by thread 0xb566d400 --- P R O C E S S --- Java Threads: ( => current thread ) 0xb569d400 JavaThread "TP-Monitor" daemon [_thread_blocked, id=26401] 0xb568c400 JavaThread "TP-Processor4" daemon [_thread_in_native, id=26400] 0xb56a0800 JavaThread "TP-Processor3" daemon [_thread_blocked, id=26399] 0xb5672400 JavaThread "TP-Processor2" daemon [_thread_blocked, id=26398] 0xb5671c00 JavaThread "TP-Processor1" daemon [_thread_blocked, id=26397] 0xb5670800 JavaThread "http-81-Monitor" [_thread_blocked, id=26396] 0xb566f800 JavaThread "http-81-Processor25" daemon [_thread_blocked, id=26395] 0xb566e400 JavaThread "http-81-Processor24" daemon [_thread_in_native, id=26394] 0xb566d400 JavaThread "http-81-Processor23" daemon [_thread_blocked, id=26393] 0xb566c000 JavaThread "http-81-Processor22" daemon [_thread_blocked, id=26392] 0xb566b000 JavaThread "http-81-Processor21" daemon [_thread_blocked, id=26391] 0xb5669c00 JavaThread "http-81-Processor20" daemon [_thread_blocked, id=26390] 0xb5668c00 JavaThread "http-81-Processor19" daemon [_thread_blocked, id=26389] 0xb565e000 JavaThread "http-81-Processor18" daemon [_thread_blocked, id=26388] 0xb565cc00 JavaThread "http-81-Processor17" daemon [_thread_blocked, id=26387] 0xb565bc00 JavaThread "http-81-Processor16" daemon [_thread_blocked, id=26386] 0xb565a800 JavaThread "http-81-Processor15" daemon [_thread_blocked, id=26385] 0xb5659800 JavaThread "http-81-Processor14" daemon [_thread_blocked, id=26384] 0xb5658400 JavaThread "http-81-Processor13" daemon [_thread_blocked, id=26383] 0
Re: JVM random crashes
On 3/3/07, Dimitar Ouzounov <[EMAIL PROTECTED]> wrote: ...I tried using JDK 1.5.0_05, 1.5.0_10, 1.5.0_11, and 1.6.0. I used the Java -server mode, tried increasing the heap size, but the JVM still crashes... Makes me wonder if it's not your new machine that's at fault (hardware or OS problem maybe?). -Bertrand
Re: JVM random crashes
But what hardware problem could it be? Tomorrow I'll make sure that the memory is fine, but nothing else comes to my mind. It may be OS-related - probably a buggy version of some library. But which library? On 3/3/07, Bertrand Delacretaz <[EMAIL PROTECTED]> wrote: On 3/3/07, Dimitar Ouzounov <[EMAIL PROTECTED]> wrote: > ...I tried using JDK 1.5.0_05, 1.5.0_10, 1.5.0_11, and 1.6.0. I used the Java > -server mode, tried increasing the heap size, but the JVM > still crashes... Makes me wonder if it's not your new machine that's at fault (hardware or OS problem maybe?). -Bertrand
Re: JVM random crashes
On 3/3/07, Dimitar Ouzounov <[EMAIL PROTECTED]> wrote: But what hardware problem could it be? Tomorrow I'll make sure that the memory is fine, but nothing else comes to my mind. Memory, motherboard, etc. Try http://www.memtest86.com/ to test this. It may be OS-related - probably a buggy version of some library. But which library? Yep, we've seen that in the past. I'd recommend going with OS versions that vendors test with. The commercial RHEL or the free clone of it http://www.centos.org/, would be my recommendation. -Yonik
logging off
I'm trying to disable all logging from Solr, or at least re-route it to a file. I was finally able to disable Jetty logging through a custom org.mortbay.log.Logger class, but I am still seeing the Solr logs, which seem to come from java.util.logging.Logger. Is there a thing I can do in solrconfig to do this? -Brian
Re: merely a suggestion: schema.xml validator or better schema validation logging
I was bit by this, tool. It made getting started a lot harder. I think I had something outside of an instead of inside. More recently, I got a query time exception from a mis-formatted field. Right now, Solr accesses the DOM as needed (at runtime) to fetch information. There isn't much up-front checking beyond the XML parser. wunder On 3/3/07 12:50 AM, "Bertrand Delacretaz" <[EMAIL PROTECTED]> wrote: > On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: > >> ...The rationale with the solrconfig stuff is that a broken config should >> behave as best it can. This is great if you are running a real site >> with people actively using it - it is a pain in the ass if you are >> getting started and don't notice errors > > I think it's a PITA in any case, I like my systems to fail loudly when > something's wrong in the configs (with details about what's happening, > of course). > > -Bertrand
Re: logging off
On Mar 3, 2007, at 12:56 PM, Brian Whitman wrote:
I'm trying to disable all logging from Solr, or at least re-route
it to a file.
Hi Brian, all you have to do is create a logging.properties file and
call this before starting up solr:
System.setProperty("java.util.logging.config.file", home+"/conf/
logging.properties");
Your logging.properties file can look like this:
---
handlers= java.util.logging.FileHandler
# Default global logging level.
.level= INFO
# default file output is in user's home directory.
java.util.logging.FileHandler.pattern = %h/Library/Application\
Support/Your Company/solr%u.log
java.util.logging.FileHandler.limit = 5
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
---
And that will disable console logging. For jetty logging, you need to
create a custom Log class that looks like this
package com.yourcompany.solr;
import java.io.IOException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.solr.core.Config;
import org.apache.solr.core.SolrCore;
import org.apache.solr.schema.IndexSchema;
import org.apache.solr.servlet.SolrDispatchFilter;
import org.apache.solr.servlet.SolrServlet;
import org.apache.solr.servlet.SolrUpdateServlet;
import org.mortbay.jetty.Handler;
import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.log.Logger;
import org.mortbay.jetty.servlet.FilterHolder;
import org.mortbay.util.DateCache;
public class NoLog implements Logger
{
private static boolean debug = System.getProperty("DEBUG",null)!
=null;
private String name;
static
{
}
public NoLog()
{
this(null);
}
public NoLog(String name)
{
this.name=name==null?"":name;
}
public boolean isDebugEnabled()
{
return debug;
}
public void setDebugEnabled(boolean enabled)
{
debug=enabled;
}
public void info(String msg,Object arg0, Object arg1)
{
}
public void debug(String msg,Throwable th)
{
}
public void debug(String msg,Object arg0, Object arg1)
{
}
public void warn(String msg,Object arg0, Object arg1)
{
}
public void warn(String msg, Throwable th)
{
}
private String format(String msg, Object arg0, Object arg1)
{
return "";
}
public Logger getLogger(String name)
{
if ((name==null && this.name==null) ||
(name!=null && name.equals(this.name)))
return this;
return new NoLog(name);
}
public String toString()
{
return "NOLOG"+name;
}
}
---
And then this before starting jetty
System.setProperty("org.mortbay.log.class",
"com.yourcompany.solr.NoLog");
NoLog noLogger = new NoLog();
org.mortbay.log.Log.setLog(noLogger);
Now, stop talking to yourself... :)
-Brian
Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/2/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: How do you all feel about returning an error when you add a document with unknown fields? +1 dynamicField definitions can be used if desired (including "*" to match every undefined field). -Yonik
Re: logging off
sweet. the logging is java logging... (not one i really know how to deal with) Can you try setting system property like this: http://www.exampledepot.com/egs/java.util.logging/Props.html Brian Whitman wrote: I'm trying to disable all logging from Solr, or at least re-route it to a file. I was finally able to disable Jetty logging through a custom org.mortbay.log.Logger class, but I am still seeing the Solr logs, which seem to come from java.util.logging.Logger. Is there a thing I can do in solrconfig to do this? -Brian
Re: JVM random crashes
Yonik Seeley wrote: On 3/3/07, Dimitar Ouzounov <[EMAIL PROTECTED]> wrote: But what hardware problem could it be? Tomorrow I'll make sure that the memory is fine, but nothing else comes to my mind. Memory, motherboard, etc. Try http://www.memtest86.com/ to test this. It may be OS-related - probably a buggy version of some library. But which library? Yep, we've seen that in the past. I'd recommend going with OS versions that vendors test with. The commercial RHEL or the free clone of it http://www.centos.org/, would be my recommendation. I'm running a lot of CentOS 4.4 myself, on i686 and x86_64 processors. I'm testing out Solr on an i686 with JDK 1.5 and I'm running a production copy of Nutch on x86_64 JDK 1.5, Tomcat 1.5. It's been rock solid. From trying to install Java in the past on FC5, I read a lot about how you had to be rather careful to make absolutely certain that you had no conflicting gjc libs in your path. If this is a production box, I'd got with a longer-supported OS than FC6. If the server is only for searching and apache, I don't think FC6 will give you any noticeable performance boost over CentOS 4.4. FC6's performance enhancements with glibc-hash-binding won't affect a JVM. Jed
Re: merely a suggestion: schema.xml validator or better schema validation logging
Yonik Seeley wrote: On 3/2/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: How do you all feel about returning an error when you add a document with unknown fields? +1 dynamicField definitions can be used if desired (including "*" to match every undefined field). If dynamicField definitions are removed from the schema.xml file (and your fields are not referencing them), does this have the same effect of disabling unknown-field generation? Jed
Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/3/07, Jed Reynolds <[EMAIL PROTECTED]> wrote: If dynamicField definitions are removed from the schema.xml file (and your fields are not referencing them), does this have the same effect of disabling unknown-field generation? Yes. You should get an error if you add a document with a field that doesn't match a defined field or a dynamic field. There still may be a bug that Ryan mentioned about unknown fields simply being ignored, but that should be fixed if true. -Yonik
Re: merely a suggestion: schema.xml validator or better schema validation logging
: I almost didn't notice the exception fly by because there's s much : log output, and I can see why I might not have noticed. Yay for : scrollback! (Hrm, I might not have wanted to watch logging for 4 : instances of solr all at once. Might explain why so much logging.) FYI: Solr logs a lot of stuff at the INFO and DEBUG levels, but "errors" will always be at the SEVERE level (unless they aren't actualy SEVERE and are just exceptions encountered during trivial unimportant things in which case they are loged at the WARNING level) it's up to your servlet container how verbose to be (ie: what level to log) you should be able to configure it to put WARNING and SEVERE messages in a seperate log file even. : > Anyway, I agree that some config errors could be handled in a more : > user-friendly manner, and it would be nice if config failures could : > make it to the front-page admin screen or something. : : That would groovy! i've been thinking a Servlet that didn't depend on any special Solr code (so it will work even if SolrCore isn't initialized) but registeres a log handler and records the last N messages from Solr above a certain level would be handy to refer people to when they are having issues and aren't overly comfortable with log files. -Hoss
Re: merely a suggestion: schema.xml validator or better schema validation logging
: > I spent a long time tracking down an error with a document set with an : > uppercase field name to something configured with a lowercase field. : Isn't this the kind of error that XML validation is supposed to address? it could be ... except that: 1) we can't using standard DTD/XSD style validation because we don't know all the field names (not to mention dynamic fields) 2) XML is just one of hte transoports for sending updates ... we expect to support a lot more customizable formats in the near future. -Hoss
Re: merely a suggestion: schema.xml validator or better schema validation logging
: Right now, Solr accesses the DOM as needed (at runtime) to fetch : information. There isn't much up-front checking beyond the XML : parser. bingo, and adding more upfront checking is hard for at least two reasons i can think of... 1) keeping a DTD up to date is a pain sa new features are added 2) the way some options are passed to plugable classes makes it impossible to validate (ie: tokenizers, caches, etc...) -Hoss
Re: merely a suggestion: schema.xml validator or better schema validation logging
There still may be a bug that Ryan mentioned about unknown fields simply being ignored, but that should be fixed if true. I just looked into this - /trunk code is fine. I wasn't noticing the errors because the response code is always 200 with an error included in the xml. My code was only checking errors on non-200 response codes Is there enough general interest in having error response codes to change the standard web.xml config to let the SolrDispatchFilter handle /select? handle-select true
Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: Is there enough general interest in having error response codes to change the standard web.xml config to let the SolrDispatchFilter handle /select? /select should already use HTTP error codes, right? -Yonik
Re: Casting Exception with Similarity
: I figured out my problem. My own jar must be in the examples/solr/lib : directory (which does not exist in the download). I found a hint to : this on the mailing list. The docs don't indicate this anywhere : promenant. Perhaps the lib directory should exist in the default : download in the future? it's mentioned in both the plugin wiki i listed, as well as the README for solr.home (example/solr/README.txt) ... do you have any suggestions about where else we should document it? we don't include the lib directory in the example solr home because adding "plugins" is considered a little above and beyond the basic usage .. we try to keep the example as simple as possible. -Hoss
Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/3/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: > Is there enough general interest in having error response codes to > change the standard web.xml config to let the SolrDispatchFilter > handle /select? /select should already use HTTP error codes, right? i see whats happening... I ran into this while writing the SolrDispatchFilter - had me stumped for a while. The SolrServlet passes along the status code from a SolrException. This works great if you throw a SolrException with a 'valid' HTTP status code (400, etc). But MANY of the SolrExceptions use a status code '1'. Then it depends on the servlet container what is actually sent to the client. I know resin and jetty do different things. In the SolrDispatchFilter, I send a HTTP status code 500 if the SolrException status is less then 100.
Re: merely a suggestion: schema.xml validator or better schema validation logging
/update does send 200 even if there was an error. after SOLR-173 we may want to change the default solrconfig to map /update so that everything has a consistent error format. On 3/3/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: > Is there enough general interest in having error response codes to > change the standard web.xml config to let the SolrDispatchFilter > handle /select? /select should already use HTTP error codes, right? -Yonik
Re: merely a suggestion: schema.xml validator or better schema validation logging
For anyone not on the dev list, I just posted: http://issues.apache.org/jira/browse/SOLR-179 so it is not lost, I also posted Otis' bug report: http://issues.apache.org/jira/browse/SOLR-180
Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: But MANY of the SolrExceptions use a status code '1'. Hmmm, I did an audit of the exceptions before we entered the incubator, and I thought I caught all the ones that generated anything out of the 400 and 500 range and could be thrown during a query (most of the "1" return codes had to do with schema or config parsing I think). Any I missed should be fixed. Then it depends on the servlet container what is actually sent to the client. I know resin and jetty do different things. In the SolrDispatchFilter, I send a HTTP status code 500 if the SolrException status is less then 100. That sounds fine. I didn't realize it could even vary by container. -Yonik
Re: merely a suggestion: schema.xml validator or better schema validation logging
On 3/3/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 3/3/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: > But MANY of the SolrExceptions use a status > code '1'. Hmmm, I did an audit of the exceptions before we entered the incubator, and I thought I caught all the ones that generated anything out of the 400 and 500 range and could be thrown during a query (most of the "1" return codes had to do with schema or config parsing I think). Any I missed should be fixed. I clearly overstated the case with "MANY" - and you are right, none are reachable from /select, so i must be off base about the /select response code stuff. quick search shows IndexSchema.java - 3, "1" status codes DirectUpdateHandler.java - 2, "2" status codes UpdateHandler.java - 2, "1" status codes everthing else has 500,400,503
Re: merely a suggestion: schema.xml validator or better schema validation logging
Chris Hostetter wrote: : I almost didn't notice the exception fly by because there's s much : log output, and I can see why I might not have noticed. Yay for you should be able to configure it to put WARNING and SEVERE messages in a seperate log file even. Certainly! I learned to reconfigure tomcat's logging when I was doing my Nutch deployment. I'm very likely going to reconfigure my logging. i've been thinking a Servlet that didn't depend on any special Solr code (so it will work even if SolrCore isn't initialized) but registeres a log handler and records the last N messages from Solr above a certain level would be handy to refer people to when they are having issues and aren't overly comfortable with log files. Yeah, like a ring buffer for last x number warning|severe messages. I'm pretty used to looking at apache log files. Some errors pointing out configuration or operational failure (like running out of file descriptors) on the admin and status pages would be helpful because I think that some people are probably going to check those pages first, possibly because they're deving and not necessarily watching logs. I'd still use Solr even if it didn't have a logging servlet, tho ;-) Jed
Re: Boosting one copyField contributor
: Lucene boosts are per-field (not per-field-value) for a particular : document, so you can't have a multi-valued field with some values : boosted higher than others. this little bit of subtlety in Lucene terminology has always bugged me ... the way i like to think of it is that field boosts are per field *name* not per instnace of "Field" (or in solr's case: "...not per instance of ") -Hoss
Re: logging off
: Hi Brian, all you have to do is create a logging.properties file and
: call this before starting up solr:
:
: System.setProperty("java.util.logging.config.file", home+"/conf/
: logging.properties");
it's not neccessary to execute any javacode to configurate
java.util.logging ... that property can be set on the commandline before
executing java.
: And that will disable console logging. For jetty logging, you need to
: create a custom Log class that looks like this
i'm no jetty expert, but this should also be controllable via jetty
configs, without writting any custom code .. greping the xml files in the
example/etc for "log" should turn up a lot of pointers ... much of what is
there is commented out by default -- and i believe that's why it goes to
STDOUT instead of specific files.
-Hoss
