Multiple search components in one handler - ie spellchecker

[Andrew Nagy]

Hello - I am attempting to add the spellCheck component in my "search" 
requesthandler so when a users does a search, they get the results and spelling 
corrections all in one query just like the way the facets work.

I am having some trouble accomplishing this - can anyone point me to 
documentation (other than http://wiki.apache.org/solr/SpellCheckComponent) on 
how to do this or an example solrconfig that would do this correctly?

My guess is there is something preventing this from happening since the 
requesthandler tag is loading the search component and not the spellchecker 
component?

Thanks and sorry if my message is a bit vague

Andrew

RE: Multiple search components in one handler - ie spellchecker

[Andrew Nagy]

Thanks for getting back to me Geoff.  Although, that is pretty much what I 
have.  Maybe if I show my solrconfig someone might be able to point out what I 
have incorrect?  The problem is that nothing related to the spelling options 
are show in the results, just the normal expected search results.  Here is my 
query:
http://localhost:8080/solr/select?q=*:*&rows=1&spellcheck=true&spellcheck.q=scandanava&spellcheck.build=true


  
allfields
0.7 
./spellchecker
  
  text



  
explicit
  
  
spellcheck

  



From: Geoffrey Young [EMAIL PROTECTED]
Sent: Friday, July 25, 2008 2:13 PM
To: solr-user@lucene.apache.org
Subject: Re: Multiple search components in one handler - ie spellchecker

Andrew Nagy wrote:
> Hello - I am attempting to add the spellCheck component in my
> "search" requesthandler so when a users does a search, they get the
> results and spelling corrections all in one query just like the way
> the facets work.
>
> I am having some trouble accomplishing this - can anyone point me to
> documentation (other than
> http://wiki.apache.org/solr/SpellCheckComponent) on how to do this or
> an example solrconfig that would do this correctly?

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200806.mbox/[EMAIL 
PROTECTED]

in general, just add the

   
 spellcheck
   

bit to your existing handler after following setup in the twiki docs.

you can ignore the part about the exceptions, as that has been fixed in
trunk.

HTH

--Geoff

RE: Multiple search components in one handler - ie spellchecker

[Andrew Nagy]

Exactly - however the spellcheck component is not working for my setup.  The 
spelling suggestions never show in the response.  I think I have the solrconfig 
setup incorrectly.  Also my solr/data/spell index that is created is empty.  
Something is not configured correctly, any ideas?

Andrew

From: Geoffrey Young [EMAIL PROTECTED]
Sent: Friday, July 25, 2008 3:04 PM
To: solr-user@lucene.apache.org
Subject: Re: Multiple search components in one handler - ie spellchecker

Andrew Nagy wrote:
> Thanks for getting back to me Geoff.  Although, that is pretty much
> what I have.  Maybe if I show my solrconfig someone might be able to
> point out what I have incorrect?  The problem is that nothing related
> to the spelling options are show in the results, just the normal
> expected search results.

right.  the spellcheck component does not issue a separate query *after*
running the spellcheck, it merely offers suggestions in parallel with
your existing query.

the results are more like

   "below are the results for $query.  did you mean $suggestions?"

HTH

--Geoff

RE: SpellCheckComponent problems (was: Multiple search components in one handler - ie spellchecker)

[Andrew Nagy]

Shalin - yes the allfields field exists in my schema.xml file.  It is a field 
that has all of the text from all of the fields concatenated together into one 
field.

My spellCheckIndexDir is created and has 2 segment files, but I think the index 
is empty.  When I initiate the 1st spellcheck.build=true ... the results load 
immediately ... I would imagine some time delay as it builds the index.

Any other ideas?

Andrew

> -Original Message-
> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> Sent: Friday, July 25, 2008 3:35 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Multiple search components in one handler - ie
> spellchecker
>
> On Sat, Jul 26, 2008 at 12:37 AM, Andrew Nagy
> <[EMAIL PROTECTED]>
> wrote:
>
> > Exactly - however the spellcheck component is not working for my
> setup.
> >  The spelling suggestions never show in the response.  I think I have
> the
> > solrconfig setup incorrectly.  Also my solr/data/spell index that is
> created
> > is empty.  Something is not configured correctly, any ideas?
> >
> > Andrew
> > 
> > From: Geoffrey Young [EMAIL PROTECTED]
> > Sent: Friday, July 25, 2008 3:04 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Multiple search components in one handler - ie
> spellchecker
> >
> > Andrew Nagy wrote:
> > > Thanks for getting back to me Geoff.  Although, that is pretty much
> > > what I have.  Maybe if I show my solrconfig someone might be able
> to
> > > point out what I have incorrect?  The problem is that nothing
> related
> > > to the spelling options are show in the results, just the normal
> > > expected search results.
> >
> > right.  the spellcheck component does not issue a separate query
> *after*
> > running the spellcheck, it merely offers suggestions in parallel with
> > your existing query.
> >
> > the results are more like
> >
> >   "below are the results for $query.  did you mean $suggestions?"
> >
> > HTH
> >
> > --Geoff
> >
> >
> >
> Is the allfields in your spell checker configuration in your
> schema.xml? Can
> you see the spellcheckIndexDir created inside the Solr's data
> directory?
>
> --
> Regards,
> Shalin Shekhar Mangar.

RE: SpellCheckComponent problems (was: Multiple search components in one handler - ie spellchecker)

[Andrew Nagy]

> -Original Message-
> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> Sent: Monday, July 28, 2008 10:09 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SpellCheckComponent problems (was: Multiple search
> components in one handler - ie spellchecker)
>
> Can you show us the query you are issuing? Make sure you add
> spellcheck=true
> to the query as a parameter to turn on spell checking.

http://localhost:8080/solr/select?q=*:*&spellcheck=true&spellcheck.q=scandanava&spellcheck.build=true

Shows this:


0
73


...



Andrew

RE: SpellCheckComponent problems (was: Multiple search components in one handler - ie spellchecker)

[Andrew Nagy]

I was just reviewing the solr logs and I noticed the following:

Jul 28, 2008 11:52:01 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error loading class 
'org.apache.solr.handler.component.SpellCheckComponent'

It looks like the SpellCheckComponent is not getting loaded.  What could cause 
this?  Im running the july25 nightly build.

Here is a list of the libs from my /tmp/jetty/webapp/WEB-INF/lib dir:
-rw-r--r--  1 root root  84199 Jul 25 08:14 apache-solr-common-nightly.jar
-rw-r--r--  1 root root 889903 Jul 25 08:14 apache-solr-nightly.jar
-rw-r--r--  1 root root  46725 May 10  2007 commons-codec-1.3.jar
-rw-r--r--  1 root root  22017 Jan  6  2008 commons-csv-1.0-SNAPSHOT-r609327.jar
-rw-r--r--  1 root root  53082 Mar  1  2007 commons-fileupload-1.2.jar
-rw-r--r--  1 root root 305001 Sep 11  2007 commons-httpclient-3.1.jar
-rw-r--r--  1 root root  83613 Jun 15  2007 commons-io-1.3.1.jar
-rw-r--r--  1 root root  38015 Jun 14  2007 commons-logging-1.0.4.jar
-rw-r--r--  1 root root 249154 Sep 21  2007 junit-4.3.jar
-rw-r--r--  1 root root 115101 Jun 19 13:46 lucene-analyzers-2.4-dev.jar
-rw-r--r--  1 root root 730352 Jun 19 13:46 lucene-core-2.4-dev.jar
-rw-r--r--  1 root root  87390 Jun 19 13:46 lucene-highlighter-2.4-dev.jar
-rw-r--r--  1 root root  32693 Jun 19 13:46 lucene-queries-2.4-dev.jar
-rw-r--r--  1 root root  91029 Jun 19 13:46 lucene-snowball-2.4-dev.jar
-rw-r--r--  1 root root  18422 Jun 19 13:46 lucene-spellchecker-2.4-dev.jar
-rw-r--r--  1 root root 179348 Jun 14  2007 stax-1.2.0-dev.jar
-rw-r--r--  1 root root  25863 Jun 14  2007 stax-api-1.0.jar
-rw-r--r--  1 root root 128475 Jun 14  2007 stax-utils.jar

could I be missing a jar?

Thanks
Andrew

> -Original Message-
> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> Sent: Monday, July 28, 2008 11:24 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SpellCheckComponent problems (was: Multiple search
> components in one handler - ie spellchecker)
>
> Hi Andrew,
>
> Your configuration which you specified in the earlier thread looks
> fine.
> Your query is also ok. The complete lack of spell check results in the
> response you pasted suggests that the SpellCheckComponent is not added
> to
> the SearchHandler's list of components.
>
> Can you check your solrconfig.xml again? I'm sorry but it doesn't seem
> like
> a problem with the spell checker itself. Also check if there are any
> exceptions in the Solr log/console.
>
> On Mon, Jul 28, 2008 at 8:32 PM, Andrew Nagy
> <[EMAIL PROTECTED]>wrote:
>
> > > -Original Message-
> > > From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> > > Sent: Monday, July 28, 2008 10:09 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: SpellCheckComponent problems (was: Multiple search
> > > components in one handler - ie spellchecker)
> > >
> > > Can you show us the query you are issuing? Make sure you add
> > > spellcheck=true
> > > to the query as a parameter to turn on spell checking.
> >
> >
> >
> http://localhost:8080/solr/select?q=*:*&spellcheck=true&spellcheck.q=sc
> andanava&spellcheck.build=true
> >
> > Shows this:
> > 
> > 
> > 0
> > 73
> > 
> > 
> > ...
> > 
> > 
> >
> > Andrew
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.

RE: SpellCheckComponent problems (was: Multiple search components in one handler - ie spellchecker)

[Andrew Nagy]

Hmm ... sorry, that was the output of a java program that uses solr that I ran 
and noticed the error.  That error doesn't happen when I start solr.  Sorry for 
the confusion.

I just changed my schema to have a dedicated field for spelling called 
"spelling" and I created a new field type for the spellcheck component called 
"textSpell".
Here is the segment of my solrconfig.xml:


  
spelling
0.7 
./spellchecker
  
  textSpell



  
explicit
  
  
spellcheck

  


I will need to reindex my documents again - I will check to see if that has any 
effect on my problem.

Andrew


> -Original Message-
> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> Sent: Monday, July 28, 2008 12:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SpellCheckComponent problems (was: Multiple search
> components in one handler - ie spellchecker)
>
> No, SpellCheckComponent was in the nightly long before July 25. There
> must
> be a stack trace after that error message. Can you post that?
>
> On Mon, Jul 28, 2008 at 9:26 PM, Andrew Nagy
> <[EMAIL PROTECTED]>wrote:
>
> > I was just reviewing the solr logs and I noticed the following:
> >
> > Jul 28, 2008 11:52:01 AM org.apache.solr.common.SolrException log
> > SEVERE: org.apache.solr.common.SolrException: Error loading class
> > 'org.apache.solr.handler.component.SpellCheckComponent'
> >
> > It looks like the SpellCheckComponent is not getting loaded.  What
> could
> > cause this?  Im running the july25 nightly build.
> >
> > Here is a list of the libs from my /tmp/jetty/webapp/WEB-INF/lib dir:
> > -rw-r--r--  1 root root  84199 Jul 25 08:14 apache-solr-common-
> nightly.jar
> > -rw-r--r--  1 root root 889903 Jul 25 08:14 apache-solr-nightly.jar
> > -rw-r--r--  1 root root  46725 May 10  2007 commons-codec-1.3.jar
> > -rw-r--r--  1 root root  22017 Jan  6  2008
> > commons-csv-1.0-SNAPSHOT-r609327.jar
> > -rw-r--r--  1 root root  53082 Mar  1  2007 commons-fileupload-
> 1.2.jar
> > -rw-r--r--  1 root root 305001 Sep 11  2007 commons-httpclient-
> 3.1.jar
> > -rw-r--r--  1 root root  83613 Jun 15  2007 commons-io-1.3.1.jar
> > -rw-r--r--  1 root root  38015 Jun 14  2007 commons-logging-1.0.4.jar
> > -rw-r--r--  1 root root 249154 Sep 21  2007 junit-4.3.jar
> > -rw-r--r--  1 root root 115101 Jun 19 13:46 lucene-analyzers-2.4-
> dev.jar
> > -rw-r--r--  1 root root 730352 Jun 19 13:46 lucene-core-2.4-dev.jar
> > -rw-r--r--  1 root root  87390 Jun 19 13:46 lucene-highlighter-2.4-
> dev.jar
> > -rw-r--r--  1 root root  32693 Jun 19 13:46 lucene-queries-2.4-
> dev.jar
> > -rw-r--r--  1 root root  91029 Jun 19 13:46 lucene-snowball-2.4-
> dev.jar
> > -rw-r--r--  1 root root  18422 Jun 19 13:46 lucene-spellchecker-2.4-
> dev.jar
> > -rw-r--r--  1 root root 179348 Jun 14  2007 stax-1.2.0-dev.jar
> > -rw-r--r--  1 root root  25863 Jun 14  2007 stax-api-1.0.jar
> > -rw-r--r--  1 root root 128475 Jun 14  2007 stax-utils.jar
> >
> > could I be missing a jar?
> >
> > Thanks
> > Andrew
> >
> > > -Original Message-
> > > From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> > > Sent: Monday, July 28, 2008 11:24 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: SpellCheckComponent problems (was: Multiple search
> > > components in one handler - ie spellchecker)
> > >
> > > Hi Andrew,
> > >
> > > Your configuration which you specified in the earlier thread looks
> > > fine.
> > > Your query is also ok. The complete lack of spell check results in
> the
> > > response you pasted suggests that the SpellCheckComponent is not
> added
> > > to
> > > the SearchHandler's list of components.
> > >
> > > Can you check your solrconfig.xml again? I'm sorry but it doesn't
> seem
> > > like
> > > a problem with the spell checker itself. Also check if there are
> any
> > > exceptions in the Solr log/console.
> > >
> > > On Mon, Jul 28, 2008 at 8:32 PM, Andrew Nagy
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > > -Original Message-
> > > > > From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> > > > > Sent: Monday, July 28, 2008 10:09 AM
> > > > > To: solr-user@lucene.apache.org
> > > > > Subject: Re: SpellCheckComponent problems (was: Multiple search
> > > > > components in one handler - ie spellchecker)
> > > > >
> > > > > Can you show us the query you are issuing? Make sure you add
> > > > > spellcheck=true
> > > > > to the query as a parameter to turn on spell checking.
> > > >
> > > >
> > > >
> > >
> http://localhost:8080/solr/select?q=*:*&spellcheck=true&spellcheck.q=sc
> > > andanava&spellcheck.build=true
> > > >
> > > > Shows this:
> > > > 
> > > > 
> > > > 0
> > > > 73
> > > > 
> > > > 
> > > > ...
> > > > 
> > > > 
> > > >
> > > > Andrew
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shalin Shekhar Mangar.
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.

RE: SpellCheckComponent problems (was: Multiple search components in one handler - ie spellchecker)

[Andrew Nagy]

Well I will include the stack trace for the aforementioned error:

Jul 28, 2008 12:20:17 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error loading class 
'org.apache.solr.handler.component.SpellCheckComponent'
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:227)
at 
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:232)
at 
org.apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLoader.java:83)
at 
org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:140)
at org.apache.solr.core.SolrCore.loadSearchComponents(SolrCore.java:565)
at org.apache.solr.core.SolrCore.(SolrCore.java:371)
at org.solrmarc.marc.MarcImporter.(MarcImporter.java:95)
at org.solrmarc.marc.MarcImporter.main(MarcImporter.java:559)
Caused by: java.lang.ClassNotFoundException: 
org.apache.solr.handler.component.SpellCheckComponent
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:580)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:242)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:211)
... 7 more

The line 95 of MarcImporter.java (the solr import program I am using) is the 
instantiation of SolrCore.  So maybe somehow the spellCheckComponent is not 
getting loaded?

This is the error output I get thrown by instantiating SolrCore:
org.apache.solr.common.SolrException: Unknown Search Component: spellcheck
at org.apache.solr.core.SolrCore.getSearchComponent(SolrCore.java:597)
at 
org.apache.solr.handler.component.SearchHandler.inform(SearchHandler.java:107)
at 
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:264)
at org.apache.solr.core.SolrCore.(SolrCore.java:398)
at org.solrmarc.marc.MarcImporter.(MarcImporter.java:95)
at org.solrmarc.marc.MarcImporter.main(MarcImporter.java:559)

Andrew

> -Original Message-
> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> Sent: Monday, July 28, 2008 12:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SpellCheckComponent problems (was: Multiple search
> components in one handler - ie spellchecker)
>
> No, SpellCheckComponent was in the nightly long before July 25. There
> must
> be a stack trace after that error message. Can you post that?
>
> On Mon, Jul 28, 2008 at 9:26 PM, Andrew Nagy
> <[EMAIL PROTECTED]>wrote:
>
> > I was just reviewing the solr logs and I noticed the following:
> >
> > Jul 28, 2008 11:52:01 AM org.apache.solr.common.SolrException log
> > SEVERE: org.apache.solr.common.SolrException: Error loading class
> > 'org.apache.solr.handler.component.SpellCheckComponent'
> >
> > It looks like the SpellCheckComponent is not getting loaded.  What
> could
> > cause this?  Im running the july25 nightly build.
> >
> > Here is a list of the libs from my /tmp/jetty/webapp/WEB-INF/lib dir:
> > -rw-r--r--  1 root root  84199 Jul 25 08:14 apache-solr-common-
> nightly.jar
> > -rw-r--r--  1 root root 889903 Jul 25 08:14 apache-solr-nightly.jar
> > -rw-r--r--  1 root root  46725 May 10  2007 commons-codec-1.3.jar
> > -rw-r--r--  1 root root  22017 Jan  6  2008
> > commons-csv-1.0-SNAPSHOT-r609327.jar
> > -rw-r--r--  1 root root  53082 Mar  1  2007 commons-fileupload-
> 1.2.jar
> > -rw-r--r--  1 root root 305001 Sep 11  2007 commons-httpclient-
> 3.1.jar
> > -rw-r--r--  1 root root  83613 Jun 15  2007 commons-io-1.3.1.jar
> > -rw-r--r--  1 root root  38015 Jun 14  2007 commons-logging-1.0.4.jar
> > -rw-r--r--  1 root root 249154 Sep 21  2007 junit-4.3.jar
> > -rw-r--r--  1 root root 115101 Jun 19 13:46 lucene-analyzers-2.4-
> dev.jar
> > -rw-r--r--  1 root root 730352 Jun 19 13:46 lucene-core-2.4-dev.jar
> > -rw-r--r--  1 root root  87390 Jun 19 13:46 lucene-highlighter-2.4-
> dev.jar
> > -rw-r--r--  1 root root  32693 Jun 19 13:46 lucene-queries-2.4-
> dev.jar
> > -rw-r--r--  1 root root  91029 Jun 19 13:46 lucene-snowball-2.4-
> dev.jar
> > -rw-r--r--  1 root root  18422 Jun 19 13:46 lucene-spellchecker-2.4-
> dev.jar
> > -rw-r--r--  1 root root 179348 Jun 14  2007 stax-1.2.0-dev.jar
> > -rw-r--r--  1 root root  25863 Jun 14  2007 s

RE: SpellCheckComponent problems (was: Multiple search components in one handler - ie spellchecker)

[Andrew Nagy]

> -Original Message-
> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
> Sent: Monday, July 28, 2008 12:38 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SpellCheckComponent problems (was: Multiple search
> components in one handler - ie spellchecker)
>
> Well that means the nightly solr jar you are using is older than you
> think
> it is. Try running solr normally without the program and see if you can
> get
> it working.

Well my import program has an older copy of the solr libs ...  so we can ignore 
that problem.

However my problem still stands when I run solr normally from my July25 
snapshot.  There are no errors - and no output to the solr logs when I post a 
query.

Have you or anyone been able to successfully add the spellcheckcomponent to the 
default select searchhandler?


Thanks
Andrew

TimeExceededException

[Andrew Nagy]

Hello - I am a part of a larger group working on an import tool called 
SolrMarc.  I am running into an error that I'm not sure what is causing it and 
looking for any leads.

I am getting the following exception on the SolrCore constructor:
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/lucene/search/TimeLimitedCollector$TimeExceededException
at org.apache.solr.core.SolrConfig.(SolrConfig.java:128)
at org.apache.solr.core.SolrConfig.(SolrConfig.java:97)
...

Any ideas what might cause this?  I am working from the July 25 nightly 
snapshot.  Could I be missing a jar or something?

Thanks!
Andrew

Solr1.3 Freeze

[Andrew Nagy]

I read on the Solr 1.3 wiki page that there is a code freeze as of today, is 
this still accurate?  Moreover - does this mean that Solr1.3 will most likely 
ship with Lucene 2.4-dev or is there any plan to wait for lucene 2.4 to be 
released?

I know scheduling questions are annoying, but I am curious as to how to better 
manage a project that uses solr and how releases should be scheduled around 
that.

Thanks!
Andrew

RE: NOTICE: multicore.xml changed to solr.xml, format changes as well

[Andrew Nagy]

Chris - thanks for the alert.  Can you please clarify the usage of the default 
attribute that is documented to be used in the "core" node.  Solr-545 has a 
note about this being removed and it is not shown in the new example solr.xml 
file.

Thanks
Andrew

> -Original Message-
> From: Chris Hostetter [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, August 12, 2008 2:01 PM
> To: solr-user@lucene.apache.org
> Subject: NOTICE: multicore.xml changed to solr.xml, format changes as
> well
>
>
> If you've bene using the trunk (and/or nightly builds) and you take
> advantage of the MultiCore features in Solr pelase be aware...
>
> As of r685244 (committed a few moments ago) Solr no longer looks for a
> "multicore.xml" file.  It instead looks for a "solr.xml" file.
>
> solr.xml supports all of the options that multicore.xml supported,
> however
> they have been "tweaked" slightly (in some cases renamed, in other
> attributes have been moved from one XML tag to another).
>
> A detailed example can be seen in example/multicore/solr.xml...
>
> http://svn.apache.org/viewvc/lucene/solr/trunk/example/multicore/solr.x
> ml?view=markup
>
> For more information, please see SOLR-689...
> https://issues.apache.org/jira/browse/SOLR-689
>
> Volunteers to help update the wiki documentation would be appreciated.
>
>
> -Hoss

RE: NOTICE: multicore.xml changed to solr.xml, format changes as well

[Andrew Nagy]

Okay - I found the removal of the default attribute in 
https://svn.apache.org/viewvc/lucene/solr/trunk/src/java/org/apache/solr/core/MultiCore.java?
 r1=606335&r2=602003

I will update the documentation on the multicore changes.

Andrew

> -Original Message-
> From: Chris Hostetter [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, August 13, 2008 2:27 AM
> To: solr-user@lucene.apache.org
> Subject: RE: NOTICE: multicore.xml changed to solr.xml, format changes
> as well
>
>
> : Chris - thanks for the alert.  Can you please clarify the usage of
> the
> : default attribute that is documented to be used in the "core" node.
> : Solr-545 has a note about this being removed and it is not shown in
> the
> : new example solr.xml file.
>
> Any attribute that was in the old example multicore.xml has a
> corrisponding attribute in the example solr.xml ...
>
> https://svn.apache.org/viewvc/lucene/solr/trunk/example/multicore/solr.
> xml?r1=650331&r2=685244
>
> ...no functionality was changed at all in this commit, it was just
> renamed.
>
> i don't know anything about a "default" attribute, other then the fact
> that the previous commit to that file (r650331) had the message
> "default
> is no longer a multicore concept"
>
> https://svn.apache.org/viewvc/lucene/solr/trunk/example/multicore/solr.
> xml
>
> -Hoss

RE: Solr1.3 Freeze

[Andrew Nagy]

Thank Grant for the update.  We have found that the lucene-2.4-dev libs are a 
bit out dated.  My colleague is going to open a bug about this.  Has any 
thought been made as to what snapshot of the lucene-2.4-dev libs will be used 
for solr 1.3?  I also like the idea of renaming them to lucene-2.4-solr.  I 
think having libs with "-dev" can be a bit scary to people who are putting solr 
1.3 in production.

Thanks!
Andrew

> -Original Message-
> From: Grant Ingersoll [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, August 13, 2008 9:44 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr1.3 Freeze
>
> We are getting there, though, and it is a high priority.  :-)  It will
> use 2.4-dev, as far as I can tell, unless the Lucene community all of
> a sudden decides to ship.  Not to worry, however, as Hoss has
> explained a number of times, it is the Lucene PMC that makes releases,
> not Lucene Java or Solr communities.  So, by us making a Solr release
> w/ that version, we are effectively releasing that copy of Lucene.
> Maybe we should call it lucene-2.4-solr.jar so as to drop the word dev
> from it.
>
> One of the best things the community can do to help, is to download
> and try out trunk, as well as the release candidates that we will be
> posting during the freeze, and give us feedback.  Because 1.3 has a
> lot of changes (we hope for the better), this will be especially
> important.
>
> -Grant
>
> On Aug 12, 2008, at 10:45 PM, Chris Hostetter wrote:
>
> >
> > : I read on the Solr 1.3 wiki page that there is a code freeze as of
> > : today, is this still accurate?  Moreover - does this mean that
> > Solr1.3
> > : will most likely ship with Lucene 2.4-dev or is there any plan to
> > wait
> > : for lucene 2.4 to be released?
> >
> > People who are interested in following/discussing the release process
> > should keep tabs on solr-dev ... Grant volunteered to act as the
> > Release
> > Manager for 1.3, and (to paraphrase his comments from a few hours
> > ago) he
> > does not feel we are quite ready for a feature freeze.
> >
> >
> > -Hoss
> >
>

multicore can't find the data directories

[Andrew Nagy]

I am trying to setup a multicore system implementation.  I just upgraded to 
today's snapshot and have converted my multicore.xml to solr.war and I also 
changed the xml to match the new schema.  However, now that I have done that, 
Solr is not finding my data directory.  With the use of multicore - does each 
core have its own data directory?

I have the following setup:

solr/core0/conf
solr/core1/conf
solr/solr.xml

When I look at the admin (http://localhost:8080/solr/core0/admin/stats.jsp) it 
shows the following:
readerDir : 
org.apache.lucene.store.FSDirectory@/usr/local/solr/data/index

and numDocs is reporting 0.

However I have my index with plenty of records located in solr/core0/data/index

Is Solr looking in the wrong directory, or do I have my index in the wrong 
place?

Thanks
Andrew

RE: multicore can't find the data directories

[Andrew Nagy]

Nevermind - sorry.  The data directory in my solrconfig.xml was not changed to 
the correct path.  Now it's alive!

Andrew

> -Original Message-
> From: Andrew Nagy [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, August 13, 2008 12:13 PM
> To: solr-user@lucene.apache.org
> Subject: multicore can't find the data directories
>
> I am trying to setup a multicore system implementation.  I just
> upgraded to today's snapshot and have converted my multicore.xml to
> solr.war and I also changed the xml to match the new schema.  However,
> now that I have done that, Solr is not finding my data directory.  With
> the use of multicore - does each core have its own data directory?
>
> I have the following setup:
>
> solr/core0/conf
> solr/core1/conf
> solr/solr.xml
>
> When I look at the admin
> (http://localhost:8080/solr/core0/admin/stats.jsp) it shows the
> following:
> readerDir :
> org.apache.lucene.store.FSDirectory@/usr/local/solr/data/index rg.apache.lucene.store.FSDirectory@/usr/local/solr/data/index>
>
> and numDocs is reporting 0.
>
> However I have my index with plenty of records located in
> solr/core0/data/index
>
> Is Solr looking in the wrong directory, or do I have my index in the
> wrong place?
>
> Thanks
> Andrew
>
>

RE: more multicore fun

[Andrew Nagy]

Doug - I had this same problem today.  If you look at my post from earlier 
today you will see the problem.  You will need to adjust the solr.data.dir 
value in the solrconfig.xml.

Maybe this also needs to be changed in the example solrconfig.xml document?

Andrew

> -Original Message-
> From: Doug Steigerwald [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, August 13, 2008 2:53 PM
> To: solr-user@lucene.apache.org
> Subject: more multicore fun
>
> OK.  Last question for a while (hopefully), but something else with
> multicore seems to be wrong.
>
> 
>
>  
>  
>
> 
>
> $ java -jar start.jar
> ...
> INFO: [core0] Opening new SolrCore at solr/core0/, dataDir=./solr/data/
> ...
> INFO: [core1] Opening new SolrCore at solr/core1/, dataDir=./solr/data/
> ...
>
> The instanceDir seems to be fine, but the dataDir isn't being set
> correctly.  The dataDir is actually example/solr/data instead of
> example/solr/core{0|1}/data.
>
> http://localhost:8983/solr/admin/multicore shows the exact same path
> to the index for both cores.  Am I missing something that the example
> multicore config doesn't use?
>
> Thanks.
> Doug

RE: more multicore fun

[Andrew Nagy]

Thanks for clarifing that Ryan - I was a bit confused too...

> Before 1.3 is released, you will either be able to:
> 1. set the dataDir from your solr.xml config
>
>

I have been perusing the multicore code and found that the "default" attribute 
was removed.  It also appears that the "dataDir" attribute was removed as well, 
is this true?

Thanks!
Andrew

odd 500 error

[Andrew Nagy]

Hello - I stumbled across an odd error which my intuition is telling me is a 
bug.

Here is my installation:
Solr Specification Version: 1.2.2008.08.13.13.05.16
Lucene Implementation Version: 2.4-dev 685576 - 2008-08-13 10:55:25

I did the following query today:
author:(r*a* AND fisher)

And get the following 500 error:

maxClauseCount is set to 1024

org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 
1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:165)
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:156)
at 
org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:63)
at org.apache.lucene.search.WildcardQuery.rewrite(WildcardQuery.java:54)
at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:385)
at 
org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:163)
at org.apache.lucene.search.Query.weight(Query.java:94)
at org.apache.lucene.search.Searcher.createWeight(Searcher.java:175)
at org.apache.lucene.search.Searcher.search(Searcher.java:126)
at org.apache.lucene.search.Searcher.search(Searcher.java:105)
at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:167)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1156)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:341)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)


Thanks
Andrew

Retrieving a non-indexed but stored field

[Andrew Nagy]

Hello - I have the following field:



However, when I do a search, the url field does not display.  Does the field 
also need to be indexed in order to retrieve the data?

Thanks
Andrew

RE: Retrieving a non-indexed but stored field

[Andrew Nagy]

Sorry for the late follow-up.  I am doing this, but get nothing back.

Can anyone replicate this problem?

Andrew

From: Erik Hatcher [EMAIL PROTECTED]
Sent: Tuesday, October 14, 2008 12:36 PM
To: solr-user@lucene.apache.org
Subject: Re: Retrieving a non-indexed but stored field

On Oct 14, 2008, at 12:16 PM, Andrew Nagy wrote:

> Hello - I have the following field:
>
>  multiValued="true"/>
>
> However, when I do a search, the url field does not display.  Does
> the field also need to be indexed in order to retrieve the data?

No... but it does need to be requested, either as fl=url, or fl=*

Erik

RE: Retrieving a non-indexed but stored field

[Andrew Nagy]

Sorry for the late follow-up.  I am doing this, but get nothing back.

Can anyone replicate this problem?

Andrew

From: Erik Hatcher [EMAIL PROTECTED]
Sent: Tuesday, October 14, 2008 12:36 PM
To: solr-user@lucene.apache.org
Subject: Re: Retrieving a non-indexed but stored field

On Oct 14, 2008, at 12:16 PM, Andrew Nagy wrote:

> Hello - I have the following field:
>
>  multiValued="true"/>
>
> However, when I do a search, the url field does not display.  Does
> the field also need to be indexed in order to retrieve the data?

No... but it does need to be requested, either as fl=url, or fl=*

Erik

RE: Retrieving a non-indexed but stored field

[Andrew Nagy]

Nope - I made the schema change and then indexed all of my content.

I can confirm that the URL string is included, cause when I change my schema 
back to have both stored and indexed, it shows the URL data in the search 
results.  When I change it to stored and not indexed, no data is returned.

Andrew

From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Yonik Seeley [EMAIL 
PROTECTED]
Sent: Wednesday, November 05, 2008 1:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Retrieving a non-indexed but stored field

On Wed, Nov 5, 2008 at 11:47 AM, Andrew Nagy <[EMAIL PROTECTED]> wrote:
> Sorry for the late follow-up.  I am doing this, but get nothing back.

Did you change the field to "stored" in the schema after you added the document?
I've never seen anyone having this problem, so perhaps verify that you
are actually sending the url field when indexing.

-Yonik

Facet on certain characters

[Andrew Nagy]

Hello, I was wondering if there is a way to facet on certain characters of a 
field.  For example, I would like to get a facet count on how many of my titles 
start with the letter A, B, C, etc.

Is this possible with SOLR?

Thanks
Andrew

Apache web server logs in solr

[Andrew Nagy]

Hello, I was thinking that solr - with its built in faceting - would make for a 
great apache log file storage system.  I was wondering if anyone knows of any 
module or library for apache to write log files directly to solr or to a lucene 
index?

Thanks
Andrew

performance questions

[Andrew Nagy]

Here are a few SOLR performance questions:

1. I have noticed with 500,000+ records that my facets run quite fast regarding 
my dataset when there is a large number of matches, but on a small result set 
(say 10 - 50) the facet queries become very slow.  Any suggestions as to how to 
improve this?

2. Someone asked me if SOLR utilizes anything like a "stored procedure" to make 
queries faster.  Does SOLR support anything such as this?

Thanks
Andrew

RE: Re:

[Andrew Nagy]

Sorry for not explaining my self clearly: I have header=true as you can see 
from the curl command and there is a header line in the csv file.

And sorry for the missing subject line.

Andrew

From: Ryan McKinley [EMAIL PROTECTED]
Sent: Sunday, December 02, 2007 5:15 PM
To: solr-user@lucene.apache.org
Subject: Re:

Andrew Nagy wrote:
> Hello - I am trying out the CSV importer and am curious with an error that I 
> am consistently running into.  What am I doing incorrectly here?  I am 
> importing a pipe delimited CSV file with quotes encapsulation.
>
> Thanks
> Andrew
>
> curl 
> http://localhost:8080/solr/update/csv?header=true%26seperator=%7C%26encapsulator=%22%26commit=true%26stream.file=import/homes.csv
>
> 
> 
> 
> Error 400 
> 
> HTTP ERROR: 400CSVLoader: must specify 
> fieldnames=<fields>* or header=true
> RequestURI=/solr/update/csv href="http://jetty.mortbay.org/";>Powered by Jetty://
>

take a look at the error:

"CSVLoader: must specify fieldnames=<fields>* or header=true"

either list the fieldnames or use the 1st line as the field names
(header=true)

ryan

[no subject]

[Andrew Nagy]

Hello - I am trying out the CSV importer and am curious with an error that I am 
consistently running into.  What am I doing incorrectly here?  I am importing a 
pipe delimited CSV file with quotes encapsulation.

Thanks
Andrew

curl 
http://localhost:8080/solr/update/csv?header=true%26seperator=%7C%26encapsulator=%22%26commit=true%26stream.file=import/homes.csv




Error 400 

HTTP ERROR: 400CSVLoader: must specify 
fieldnames=* or header=true
RequestURI=/solr/update/csvhttp://jetty.mortbay.org/";>Powered by Jetty://

RE: Re:

[Andrew Nagy]

> On Dec 2, 2007, at 5:43 PM, Ryan McKinley wrote:
>>
>>
>> try \& rather then %26
>
>
> or just put quotes around the whole url. I think curl does the right thing 
> here.

I tried all the methods: converting & to %26, converting & to \& and 
encapsulating the url with quotes.  All give the same error.

curl 
http://localhost:8080/solr/update/csv?header=true\&seperator=%7C\&encapsulator=%22\&commit=true\&stream.file=import/homes.csv



Error 400 

HTTP ERROR: 400CSVLoader: must specify 
fieldnames=* or header=true
RequestURI=/solr/update/csvhttp://jetty.mortbay.org/";>Powered by Jetty://

Any other ideas?  How do other do this?

Thanks
Andrew

RE: RE: Re:

[Andrew Nagy]

Ugh ... I shouldn't be coding on a sunday night - especially after the eagles 
lost again!

I spelled separator correctly this time :) - But still no luck.

curl 
'http://localhost:8080/solr/update/csv?header=true&separator=%7C&encapsulator=%22&commit=true&stream.file=import/homes.csv'
 -H 'Content-type:text/plain; charset=utf-8'



Error 400 

HTTP ERROR: 400CSVLoader: must specify 
fieldnames=* or header=true
RequestURI=/solr/update/csvhttp://jetty.mortbay.org/";>Powered by Jetty://

RE: Re:

[Andrew Nagy]

Ryan, i didn't know there was a debugger - this could come in handy for other 
things.  Thanks!

I tried it out and it looks like everything is being parsed correctly when 
passing the url in quotes:

curl 
"http://localhost:8080/solr/debug/dump?header=true&separator=%7C&encapsulator=%22&commit=true";
 --data-binary @homes.csv -H 'Content-type:text/plain; charset=utf-8'


00org.apache.solr.handler.DumpRequestHandlertrue|true"trueexplicittrue|true"/debug/dump


I tried again with update/csv and same error occurs:
curl 
"http://localhost:8080/solr/update/csv?header=true&separator=%7C&encapsulator=%22&commit=true";
 --data-binary @homes.csv -H 'Content-type:text/plain; charset=utf-8'



Error 400 

HTTP ERROR: 400CSVLoader: must specify 
fieldnames=* or header=true
RequestURI=/solr/update/csvhttp://jetty.mortbay.org/";>Powered by Jetty://

RE: Re:

[Andrew Nagy]

Perfect!  That did it.

Thanks for debugging this with me :)

Andrew

From: Ryan McKinley [EMAIL PROTECTED]
Sent: Sunday, December 02, 2007 7:10 PM
To: solr-user@lucene.apache.org
Subject: Re:

hymmm - give it a try without specifying "header=true"

Looks like if you don't specify header=true, it defaults to true - but
if you do, it throws an error.

I think there may be a bug... Yonik, should line 243 be:

   } else if (!hasHeader) {
      ^!!!
ryan


Andrew Nagy wrote:
> Ryan, i didn't know there was a debugger - this could come in handy for other 
> things.  Thanks!
>
> I tried it out and it looks like everything is being parsed correctly when 
> passing the url in quotes:
>
> curl 
> "http://localhost:8080/solr/debug/dump?header=true&separator=%7C&encapsulator=%22&commit=true";
>  --data-binary @homes.csv -H 'Content-type:text/plain; charset=utf-8'
> 
> 
> 0 name="QTime">0 name="handler">org.apache.solr.handler.DumpRequestHandler name="params">true| name="commit">true" name="params">true name="echoParams">explicittrue name="separator">|true name="encapsulator">" name="path">/debug/dump
> 
>
> I tried again with update/csv and same error occurs:
> curl 
> "http://localhost:8080/solr/update/csv?header=true&separator=%7C&encapsulator=%22&commit=true";
>  --data-binary @homes.csv -H 'Content-type:text/plain; charset=utf-8'
> 
> 
> 
> Error 400 
> 
> HTTP ERROR: 400CSVLoader: must specify 
> fieldnames=<fields>* or header=true
> RequestURI=/solr/update/csv href="http://jetty.mortbay.org/";>Powered by Jetty://
>

Limit on # of fields to query on

[Andrew Nagy]

I am testing around with a new feature in my system that uses Solr and I am 
testing a query that has a search on the same field OR'd together over 150 
times.  I know this sounds pretty ridiculous and as I said I am just playing 
around.  However Solr just returns a blank page and doesn't process the query.  
Is there a limit to the number of fields that you can have in your query?

My example is to do some inventory checking with something like:
id:1 or id:5 or id:7 or id:8 ... and so on for roughly 176 times (which I think 
would be the worst case scenario in my catalog)

Thanks!
Andrew

schema version bug

[Andrew Nagy]

Hello - I stumbled upon a odd bug, or what appears to be a bug, today.  I have 
been using my own custom version numbers for my schema and tried to change the 
version number from 0.8 to 0.8.1 which rendered solr useless yielding a schema 
parsing error.  I then tried to change it to 0.8-1 with the same results.

Is this a bug or a "feature"?

Thanks
Andrew

POSTing repeated fields to Solr

[Andrew Nagy]

Hello - I was wondering if there is a work around with POSTing repeated fields 
to Solr.  I am using Jetty as my container with Solr 1.2.

I tried something like:
http://localhost:8080/solr/select/?q=author:(smith)&rows=0&start=0&facet=true&facet.mincount=1&facet.limit=10&facet.field=authorlast&facet.field=authorfirst

I am only getting back facets from the last facet.field.  With a GET request I 
get back all of the facet fields.  I am assuming this is a limitation with 
Jetty?  How are others doing this?

Thanks
Andrew

RE: POSTing repeated fields to Solr

[Andrew Nagy]

> On 4-Jun-08, at 2:22 PM, Andrew Nagy wrote:
>
> > Hello - I was wondering if there is a work around with POSTing
> > repeated fields to Solr.  I am using Jetty as my container with Solr
> > 1.2.
> >
> > I tried something like:
> >
> http://localhost:8080/solr/select/?q=author:(smith)&rows=0&start=0&face
> t=true&facet.mincount=1&facet.limit=10&facet.field=authorlast&facet.fie
> ld=authorfirst
> >
> > I am only getting back facets from the last facet.field.  With a GET
> > request I get back all of the facet fields.  I am assuming this is a
> > limitation with Jetty?  How are others doing this?
>
> It could be, but I would be surprised.  How are you constructing the
> POST request  (you listed a url, which only makes sense as a GET)?

I was just listing the GET representation of the URL.  I am using the 
PEAR::HTTP_Request library for PHP.  If you don't think this is a Jetty 
limitation, then it is probably something with the library that I am using.  I 
will investigate there 1st.

Thanks!
Andrew

Initial import problems

[Andrew Nagy]

Hello, I am new to SOLR but very excited for it's possibilities.

I am having some difficulties with my data import which I hope can be 
solved very easily.
First I wrote an xslt to transform my xml into the solr schema and 
modified the schema.xml to match the fields that I created.  I then ran 
the post.sh on my 492,000 records that I have.  Near the end of the 
process the records stopped being added due to a memory heap error.  I 
obviously maxed the allotted memory for the import.  Next time I will 
import less at a time!


I then posted a commit statement.  I went to my solr admin site and 
looked at the statistics.  It said 372,000 records (roughly) were stored 
and 1 commit.  I tried to do a search but no matter what I search for I 
get 0 results.  I even tried title:"the" (assuming it is not blocking 
the stop word, it should return something!).


It appears to me that the search is not searching any records.  Any idea 
as to what I might need to do, or should I start over from scratch and 
re-import my records in smaller chunks?


Thanks!
Andrew

Changing the schema.xml

[Andrew Nagy]

What is necessary for the effects of changing the schema.xml to take 
effect for all of my records?  I restarted tomcat, but it does not seem 
that my changes have taken effect.


I wanted to change a full-text field from type:string to type:text to 
allow for better searching, but do no see any differences when I make my 
changes to the schema.xml.


Thanks!
Andrew

Facet Performance

[Andrew Nagy]

In September there was a thread [1] on this list about heterogeneous 
facets and their performance.  I am having a similar issue and am 
unclear as the resolution of this thread.


I performed a search against my dataset (492,000 records) and got the 
results I am looking for in .3 seconds.  I then set facet to true and 
got results in 16 seconds and the facets include data that is not in my 
result set, it is from the entire set.  How do I limit the faceting to 
my results set and speed up the results?


Thanks!
Andrew

[1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg00955.html

Re: Facet Performance

[Andrew Nagy]

Yonik Seeley wrote:


1) facet on single-valued strings if you can
2) if you can't do (1) then enlarge the fieldcache so that the number
of filters (one per possible term in the field you are filtering on)
can fit.


I wll try this out.


3) facet counts are limited to the results of the query, filtered by
any filters.   Is there a reason you think they are not?


No, you are right.  I was thrown off at 1st.

On complaint about the faceting though:  Why is the element that is 
returned called "1st".  This seems like a poor choice for an element 
name.  Why not just name the element what is in the "name" attribute?  
It would make parsing much easier!


Thanks!
Andrew

Re: Facet Performance

[Andrew Nagy]

Yonik Seeley wrote:


1) facet on single-valued strings if you can
2) if you can't do (1) then enlarge the fieldcache so that the number
of filters (one per possible term in the field you are filtering on)
can fit.


I changed the filterCache to the following:
   

However a search that normally takes .04s is taking 74 seconds once I 
use the facets since I am faceting on 4 fields.


Can you suggest a better configuration that would solve this performance 
issue, or should I not use faceting?
I figure I could run the query twice, once limited to 20 records and 
then again with the limit set to the total number of records and develop 
my own facets.  I have infact done this before with a different back-end 
and my code is processed in under .01 seconds.


Why is faceting so slow?

Andrew

Re: Facet Performance

[Andrew Nagy]

Yonik Seeley wrote:


On 12/8/06, Andrew Nagy <[EMAIL PROTECTED]> wrote:


I changed the filterCache to the following:


However a search that normally takes .04s is taking 74 seconds once I
use the facets since I am faceting on 4 fields.



The first time or subsequent times?
Is your filterCache big enough yet?  What do you see for evictions and
hit ratio?


Here are the stats, Im still a newbie to SOLR, so Im not totally sure 
what this all means:

lookups : 1530036
hits : 2
hitratio : 0.00
inserts : 1530035
evictions : 1504435
size : 25600
cumulative_lookups : 1530036
cumulative_hits : 2
cumulative_hitratio : 0.00
cumulative_inserts : 1530035
cumulative_evictions : 1504435

Could you suggest a better configuration based on this?




Can you suggest a better configuration that would solve this performance
issue, or should I not use faceting?



Faceting isn't something that will always be fast... one often needs
to design things in a way that it can be fast.

Can you give some examples of your faceted queries?
Can you show the field and fieldtype definitions for the fields you
are faceting on?
For each field that you are faceting on, how many different terms are 
in it?


My data is 492,000 records of book data.  I am faceting on 4 fields: 
author, subject, language, format.
Format and language are fairly simple as their are only a few unique 
terms.  Author and subject however are much different in that there are 
thousands of unique terms.


Thanks for your help!
Andrew

Re: Facet Performance

[Andrew Nagy]

Chris Hostetter wrote:


: Could you suggest a better configuration based on this?

If that's what your stats look like after a single request, then i would
guess you would need to make your cache size at least 1.6 million in order
for it to be of any use in improving your facet speed.
 

Would this have any strong impacts on my system?  Should I just set it 
to an even 2 million to allow for growth?



: My data is 492,000 records of book data.  I am faceting on 4 fields:
: author, subject, language, format.
: Format and language are fairly simple as their are only a few unique
: terms.  Author and subject however are much different in that there are
: thousands of unique terms.

by the looks of it, you have a lot more then a few thousand unique terms
in those two fields ... are you tokenizing on these fields?  that's
probably not what you want for ields you're going to facet on.
 

All of these fields are set as "string" in my schema, so if I understand 
the fields correctly, they are not being tokenized.  I also have an 
author field that is set as "text" for searching.


Thanks
Andrew

Re: Facet Performance

[Andrew Nagy]

Yonik Seeley wrote:


Are they multivalued, and do they need to be.
Anything that is of type "string" and not multivalued will use the
lucene FieldCache rather than the filterCache.


The author field is multivalued.  Will this be a strong performance issue?

I could make multiple author fields as to not have the multivalued field 
and then only facet on the first author.


Thanks
Andrew

Re: Facet Performance

[Andrew Nagy]

J.J. Larrea wrote:


Unfortunately which strategy will be chosen is currently undocumented and 
control is a bit oblique:  If the field is tokenized or multivalued or Boolean, 
the FilterQuery method will be used; otherwise the FieldCache method.  I expect 
I or others will improve that shortly.
 

Good to hear, cause I can't really get away with not having a 
multi-valued field for author.


Im really excited by solr and really impressed so far.

Thanks!
Andrew

Re: Facet Performance

[Andrew Nagy]

Erik Hatcher wrote:


On Dec 8, 2006, at 2:15 PM, Andrew Nagy wrote:

My data is 492,000 records of book data.  I am faceting on 4  fields: 
author, subject, language, format.
Format and language are fairly simple as their are only a few  unique 
terms.  Author and subject however are much different in  that there 
are thousands of unique terms.



When encountering difficult issues, I like to think in terms of the  
user interface.  Surely you're not presenting 400k+ authors to the  
users in one shot.  In Collex, we have put an AJAX drop-down that  
shows the author facet (we call it name on the UI, with various roles  
like author, painter, etc).  You can see this in action here:


In our data, we don't have unique authors for each records ... so let's 
say out of the 500,000 records ... we have 200,000 authors.  What I am 
trying to display is the top 10 authors from the results of a search.  
So I do a search for title:"Gone with the wind" and I would like to see 
the top 10 matching authors from these results.


But no worries, I have written my own facet handler and I am now back to 
under a second with faceting!


Thanks for everyone's help and keep up the good work!

Andrew

Result: numFound inaccuracies

[Andrew Nagy]

Hello, me again.

I have been running some extensive tests of my search engine and have 
been seeing inaccuracies with the "numFound" attribute.  It tends to 
return 1 more than what is actually show in the XML.


Is this a bug, or could I be doing something wrong?

I have a specific example in front of me at the moment where my query 
found 2 records, yet I get: "


Any ideas?

Andrew

Re: Result: numFound inaccuracies

[Andrew Nagy]

- Original Message -
From: Yonik Seeley <[EMAIL PROTECTED]>
Date: Friday, December 8, 2006 6:01 pm
Subject: Re: Result: numFound inaccuracies
To: solr-user@lucene.apache.org

> 
> start is 0 based :-)
> 

Man do I feel dumb!

Andrew

New SOLR installation problems

[Andrew Nagy]

I installed the 12-8 snapshot of solr on my 64bit RH AS server and 
whenever I go to the admin page I get the following error:


SEVERE: Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.solr.core.SolrCore


Any ideas as to what is causing this?

Thanks
Andrew

Re: New SOLR installation problems

[Andrew Nagy]

Thanks Yonik for the reply. 
I am using tomcat, and there is nothing in the catalina.out file.  The 
access log just reports the same error I see in the browser which is 
reported below.
I am starting tomcat from my solr directory which has the data, bin and 
conf directories as subdirectories.  So the CWD should be correct.  I 
have this same setup on another server that I have been working on with 
no problem.  Im kinda lost with this one.

Is their a setting in the solrconfig.xml file that I should be looking at?

Andrew

Yonik Seeley wrote:


On 12/11/06, Andrew Nagy <[EMAIL PROTECTED]> wrote:


I installed the 12-8 snapshot of solr on my 64bit RH AS server and
whenever I go to the admin page I get the following error:

SEVERE: Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.solr.core.SolrCore

Any ideas as to what is causing this?



Look through the logs of whatever servlet container you are using for
the first exception thrown.  It most likely has something to do with
not being able to find the solr config files.

-Yonik

Re: New SOLR installation problems

[Andrew Nagy]

Nevermind, I got it working now.  Had the paths setup incorrectly. 
Dumb++


Andrew

Andrew Nagy wrote:

Thanks Yonik for the reply. I am using tomcat, and there is nothing in 
the catalina.out file.  The access log just reports the same error I 
see in the browser which is reported below.
I am starting tomcat from my solr directory which has the data, bin 
and conf directories as subdirectories.  So the CWD should be 
correct.  I have this same setup on another server that I have been 
working on with no problem.  Im kinda lost with this one.
Is their a setting in the solrconfig.xml file that I should be looking 
at?


Andrew

Yonik Seeley wrote:


On 12/11/06, Andrew Nagy <[EMAIL PROTECTED]> wrote:


I installed the 12-8 snapshot of solr on my 64bit RH AS server and
whenever I go to the admin page I get the following error:

SEVERE: Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.solr.core.SolrCore

Any ideas as to what is causing this?




Look through the logs of whatever servlet container you are using for
the first exception thrown.  It most likely has something to do with
not being able to find the solr config files.

-Yonik

multiple collections

[Andrew Nagy]

I was wondering how I might create multiple collections that have 
different field sets under solr.  Would I have to have multiple 
implementations of solr running, or can I have more than one schema.xml 
file per "collection" ?


Thanks
Andrew

relevance ranking and scoring

[Andrew Nagy]

I have 2 questions about the SOLR relevancy system.

1. Why is it when I search for an exact phrase of a title of a record I 
have it generally does not come up as the 1st record in the results?


ex: title:(gone with the wind), the record comes up 3rd.  A record with 
the term "wind" as the first word in the title comes up 1st.

ex: title:"gone with the wind", the record comes up 1st.

Is this because the word "wind" is the only noun?

2. The "score" that is associated with each value is quite odd, what 
does it represent.  I generally get results with the top record being 
somewhere around 3.0 or 2.0 and most records are below 1.



Thanks!
Andrew

Re: relevance ranking and scoring

[Andrew Nagy]

Yonik Seeley wrote:

Things you can try:
- post the debugging output (including score explain) for the query

I have attached the output.

- try disabling length normalization for the title field, then remove
the entire index and re-idnex.
- try the dismax handler, which can generate sloppy phrase queries to
boost results containing all terms.
- try a different similarity implementation
(org.apache.lucene.misc.SweetSpotSimilarity from lucene)
Can you explain what these 3 options mean?  I would like to get a better 
understanding of the guts of SOLR/Lucene but I am too busy working on my 
application that uses it to spend time with the internals.


Thanks
Andrew




 0
 102


 
  2.7548285
  Farnol, Jeffery,
  PR6011.A75.W56 1939
  Book
  97525
  eng
  1939, c1934.
  Triangle Books,
  Winds of chance /
  Winds of change [sic]
 
 
  2.5437002
  Simpson, John E.,
  QC939.L37S56 1994
  Book
  433518
  0521452112
  eng
  ill., maps ;
  1994.
  Cambridge University Press,
  Sea breeze.
  
  Sea breeze and local winds /
  Sea breeze and local wind.
 
 
  2.438136
  Hobbs, William Herbert,
  G743.H6 1968
  Book
  192408
  eng
  illus., maps, ports. ;
  [1968, c1930]
  Greenwood Press,
  Meteorology
  
  Arctic regions.
  Exploring about the North Pole of the winds.
  North Pole of the winds.
 
 
  2.4319565
  Mitchell, Margaret,
  PS3525.I972G6 1996
  Book
  426657
  0684826259 (alk. paper)
  eng
  ill. ;
  c1996.
  Scribner,
  Women
  History
  Georgia
  Gone with the wind /
 
 
  2.4319565
  Gable, Clark,Leigh, Vivien,Howard, Leslie,
	De Havilland, Olivia.Mitchell, Thomas,McDaniel, Hattie,McQueen, Butterfly.
	Fleming, Victor,Mitchell, Margaret,
  VT3188 VHS
  Video
  529954
  eng
  sd., col. ;
  c1999.
  Time Warner Co.,
  War films.Feature films.
  
  United States
  Gone with the wind
 
 
  2.4319565
  Mitchell, Margaret,
  PS3525.I972G6 1993
  Book
  534773
  0446365386
  eng
  [1993], c1936.
  Warner Books,
  United StatesGeorgia
  Gone with the wind /
 
 
  1.7023697
  Pyron, Darden Asbury.
  PS3525.I972G687 1983
  Book
  27783
  081300747X (pbk. : alk. paper)
  eng
  c1983.
  University Presses of Florida,
  Mitchell, Margaret,
  Gone with the wind (Motion picture)
  Southern States
  Recasting :"Gone with the wind" in American culture /
 
 
  1.6493776
  Stuttgarter Bläserquintett.Haydn, Joseph,Reicha, Anton,
	Danzi, Franz,Lickl, Johann Georg,
  CD257
  Divertimento, Nr. 1, B-Dur : Chorale St. Antoni / Joseph Haydn (10:24) -- Bläserquintett Es-Dur, op. 88, 2 / Anton Reicha (14:06) -- Bläserquintett B-Dur, op. 56, 1 / Franz Danzi (13:31) -- Quintetto concertante, F-Dur / Johann Georg Lickl (20:41).
  Audio
  555810
 
  digital, stereo. ;
  1989.
  Pilz,
  Vienna master seriesVienna master series
  Wind quintets (Bassoon, clarinet, flute, horn, oboe)Suites (Bassoon, clarinet, flute, horn, oboe)
  
  Bläserquintett San Antoni und andere BläserquintetteWind-player quintet San Antoni and other wind-player quintets /
  Wind-player quintet San Antoni and other wind-player quintets
 
 
  1.4591739
  Taylor, Helen,
  PS3525.I972G688 1989
  Book
  312906
  0813514800 :
  eng
  1989.
  Rutgers University Press,
  Mitchell, Margaret,Mitchell, Margaret,Mitchell, Margaret,
  Gone with the wind (Motion picture)
  WomenHistorical fiction, AmericanMotion picture audiences
  Books and readingFilm and video adaptations.
  Scarlett's women :Gone with the wind and its female fans /
 
 
  1.4591739
  Vertrees, Alan David,
  PN1997.G59V47 1997
  Book
  508240
  0292787294 (pbk. : alk. paper)
  eng
  ill. ;
  c1997.
  University of Texas Press,
  Texas film studies seriesTexas film studies series
  Selznick, David O.,
  Gone with the wind (Motion picture)
  Selznick's vision :Gone with the wind and Hollywood filmmaking /
 


 title:(gone with the wind) OR title2:(gone with the wind)
 title:(gone with the wind) OR title2:(gone with the wind)
 (title:gone title:wind) (title2:gone title2:wind)
 (title:gone title:wind) (title2:gone title2:wind)
 
  
2.7548285 = (MATCH) sum of:
  1.0556406 = (MATCH) product of:
2.1112812 = (MATCH) sum of:
  2.1112812 = (MATCH) weight(title:wind in 490046), product of:
0.394033 = queryWeight(title:wind), product of:
  8.573011 = idf(docFreq=252)
  0.045962032 = queryNorm
5.3581324 = (MATCH) fieldWeight(title:wind in 490046), product of:
  1.0 = tf(termFreq(title:wind)=1)
  8.573011 = idf(docFreq=252)
  0.625 = fieldNorm(field=title, doc=490046)
0.5 = coord(1/2)
  1.6991879 = (MATCH) product of:
3.3983757 = (MATCH) sum of:
  3.3983757 = (MATCH) weight(title2:wind in 490046), product of:
0.55892086 = queryWeight(title2:wind), product of:
  12.16049 = idf(docFreq=6)
  0.045962032 = queryNorm
6.080245 = (MATCH) fieldWeight(title2:wind in 490046), product of:
  1.0 = tf(termFreq(title2:wind)=1)
  12.16049 = idf(docFreq=6)
  0.5 = fieldNorm(field=title2

Re: relevance ranking and scoring

[Andrew Nagy]

Yonik Seeley wrote:

On 1/23/07, Andrew Nagy <[EMAIL PROTECTED]> wrote:

Yonik Seeley wrote:
> Things you can try:
> - post the debugging output (including score explain) for the query
I have attached the output.
> - try disabling length normalization for the title field, then remove
> the entire index and re-idnex.
> - try the dismax handler, which can generate sloppy phrase queries to
> boost results containing all terms.
> - try a different similarity implementation
> (org.apache.lucene.misc.SweetSpotSimilarity from lucene)
Can you explain what these 3 options mean?  I would like to get a better
understanding of the guts of SOLR/Lucene but I am too busy working on my
application that uses it to spend time with the internals.


Let's start with the first... add a debugQuery=on
parameter to your request and post the full result here.
You can get the same effect through the
query form on the solr admin pages by checking the "Debug: explain" 
checkbox.

I attached the results to my last email, are you not able to see them?

Andrew

Re: relevance ranking and scoring

[Andrew Nagy]

Yonik Seeley wrote:

Ok, here is your query:
title:(gone with the wind) OR title2:(gone
with the wind)
And here it is parsed:
(title:gone title:wind) (title2:gone 
title2:wind)


First, notice how stopwords were removed, so "with" and "the" will not
count in the results.

You are querying across two different fields.
Notice how the first two documents both have "wind" in both title and 
title2,

while the third document "gone with the wind" has no title2 field (and
hence can't match on it).

In the first documents, the scores for the matches on title and title2
both contribute to the score.  For the third document, it's penalized
by not matching in both the title and title2 field.

You could look at the dismax handler... it helps constructs queries, a
component of which are DisjunctionMaxQueries (they don't add together
scores from different fields, but just take the highest score from any
matching field for a term).

You could also see how changing or removing the stopword list affects 
relevance.

Wow, thanks for the verbose response.  This gives me a lot to go on!

What about term ranking, could I rank the phrases searched in title 
higher than title2?


Thanks!
Andrew

Re: relevance ranking and scoring

[Andrew Nagy]

Yonik Seeley wrote:


What about term ranking, could I rank the phrases searched in title
higher than title2?


Absolutely... standard lucene syntax for boosting will give you that
in the standard query handler.

title:(gone with the wind)^3.0 OR title2:(gone with the wind)

That did it!  Thanks for the Help!
What value do the numbers carry in the ranking?  I arbitrarily choose 
the number 5 cause it's an easy number :)


I am a bit nervous about the dismax query system as I have quite a bit 
of other content that could skew the results.
Whats the difference between the dismax query handler and listing all of 
the fields in my search and separating them with an OR?


Thanks!
Andrew

Re: MoreLikeThis similarity-type queries in Solr

[Andrew Nagy]

Bertrand Delacretaz wrote:

On 1/31/07, Brian Whitman <[EMAIL PROTECTED]> wrote:

Does Solr have support for the Lucene query-contrib "MoreLikeThis"
query type or anything like it? ...


Yes, there's a patch in http://issues.apache.org/jira/browse/SOLR-69 -
if you try it, please add your comments on that page.


Anyword on something like this being incorporated into the official SOLR 
release?


Andrew

Re: facet optimizing

[Andrew Nagy]

Gunther, Andrew wrote:

Yes most all terms are multi-valued which I can't avoid.
Since the data is coming from a library catalogue I am translating a
subject field to make a subject facet.  That facet alone is the biggest,
hovering near 39k.  If I remove this facet.field things return faster.
So am I to assume that this particular field bogging down operations and
there are no other optimization options besides cutting down this field?
  


Andrew, I haven't yet found a successful way to implement the SOLR 
faceting for library catalog data.  I developed my own system, so for 
every query, I first query 20 records.  Let's say it find 1000 records 
and returns the 20 records.  Then I make a second query returning all 
1000 records and build my own facets based on the 1000 records.  It's a 
bit faster than using SOLRs faceting system, but as you said.  For large 
records it still takes a bit of time.  I implemented it using AJAX so it 
doesn't slow down the loading of the page.


I'd be curious if anyone has been able to find a better way using SOLRs 
faceting system


Andrew

Re: Hierarchical Facets

[Andrew Nagy]

Hmm ... I had a brain storm.

Could I do something like this:
Dir1/Subdir1/SubSubDir1

Then query collection:"Dir1/Subdir1" and get the facets on collection at 
that point to see all of the subsubdirectories?


Is their any better method?

Andrew

Andrew Nagy wrote:
I am running into a stumbling block and can only find a way to solve 
the problem with some sort of hierarchical faceting system.  I am in 
the process of moving my records from eXist (http://exist.sf.net) to 
Solr, but the problem is with the lack of a "directory structure" that 
exist has.  I figure I could easily solve this problem by setting a 
field called "collection" and then faceting on that field to get my 
list of "directories" however this will not allow me to do 
subdirectories.


Any ideas on how to implement such a thing in solr?

Thanks!
Andrew

Hierarchical Facets

[Andrew Nagy]

I am running into a stumbling block and can only find a way to solve the 
problem with some sort of hierarchical faceting system.  I am in the 
process of moving my records from eXist (http://exist.sf.net) to Solr, 
but the problem is with the lack of a "directory structure" that exist 
has.  I figure I could easily solve this problem by setting a field 
called "collection" and then faceting on that field to get my list of 
"directories" however this will not allow me to do subdirectories.


Any ideas on how to implement such a thing in solr?

Thanks!
Andrew

Installation woes

[Andrew Nagy]

Hello, I am trying to install another copy of solr on a server.  I have 
done this many times before, but am running into errors now and I am not 
sure what is causing them.


I unzipped a copy of 1.1.0 and placed the .war file into tomcat.  Then I 
created the solr directory with my bin, data, conf directories.
When I restart tomcat (from the solr directory) and go to the Solr admin 
website everything looks fine.  But, when I pull up the statistics page, 
I only have information in the Query Handlers box, the rest are empty.  
When I click on ping, I see a 500 error from tomcat.


What am I forgetting to do?

Thanks
Andrew

Re: Installation woes

[Andrew Nagy]

Argh!  Thanks Yonik for pointing out the log files, duh!  I had a 
malformed line in my schema.xml.  Nice feature to add down the line, 
although I know there is a lot of work going into the admin interface so 
who knows if it is already thought of.  Schema Debugger?  Maybe one day 
I will dig into the code guts and try to contribute.


Andrew


Yonik Seeley wrote:

On 3/7/07, Andrew Nagy <[EMAIL PROTECTED]> wrote:

Hello, I am trying to install another copy of solr on a server.  I have
done this many times before, but am running into errors now and I am not
sure what is causing them.

I unzipped a copy of 1.1.0 and placed the .war file into tomcat.  Then I
created the solr directory with my bin, data, conf directories.
When I restart tomcat (from the solr directory) and go to the Solr admin
website everything looks fine.  But, when I pull up the statistics page,
I only have information in the Query Handlers box, the rest are empty.
When I click on ping, I see a 500 error from tomcat.

What am I forgetting to do?


Did you point solr to the solr_home (containing your bin, data, and
conf directories)?
Or do you rely on the default of $CWD/solr?

If that's not it, check all the tomcat log files for the first
exception you see.

-Yonik

fetch only facets

[Andrew Nagy]

Is their a way to not return any docs and only facets?  I tried setting 
the fl equal to blank, but then i get everything back. 


Thanks
Andrew

cache sizes

[Andrew Nagy]

Is their a science to choosing a cache sizes?  I have about 500,000 
records and am seeing a lot of evictions, about 50% of lookups.  What 
factors can i look at to determine what my cache sizes should be?


Here are my cache statistics:

filterCache 
class: org.apache.solr.search.LRUCache 
version: 1.0 
description: LRU Cache(maxSize=10, initialSize=5, 
autowarmCount=5, 
[EMAIL PROTECTED]) 
stats: lookups : 3195074

hits : 1541249
hitratio : 0.48
inserts : 1653824
evictions : 1553824
size : 10
cumulative_lookups : 3195074
cumulative_hits : 1541249
cumulative_hitratio : 0.48
cumulative_inserts : 1653825
cumulative_evictions : 1553825
  
name: queryResultCache 
class: org.apache.solr.search.LRUCache 
version: 1.0 
description: LRU Cache(maxSize=10, initialSize=5000, 
autowarmCount=5000, 
[EMAIL PROTECTED]) 
stats: lookups : 1976

hits : 872
hitratio : 0.44
inserts : 1199
evictions : 0
size : 1104
cumulative_lookups : 1976
cumulative_hits : 872
cumulative_hitratio : 0.44
cumulative_inserts : 1199
cumulative_evictions : 0
  
name: documentCache 
class: org.apache.solr.search.LRUCache 
version: 1.0 
description: LRU Cache(maxSize=512, initialSize=512) 
stats: lookups : 177530

hits : 13255
hitratio : 0.07
inserts : 164275
evictions : 163763
size : 512
cumulative_lookups : 177530
cumulative_hits : 13255
cumulative_hitratio : 0.07
cumulative_inserts : 164275
cumulative_evictions : 163763

Facet fields question

[Andrew Nagy]

Hello

I have 2 fields that I am faceting on, both of which are of type 
"string."  The first field is a copyfield from a "text" field copied to 
a "string" field for faceting.  The other is purely a "string" field.  
The faceted results of the copyfield are accurate; however the facet 
results of the 2nd string field all have the ends of the words cut off.  
How can I get the 2nd string field to display in it's original form?  Do 
I need to make a copyField for all fields that are being faceted on?


Also, both fields are multivalued.

Thanks!
Andrew

SEVERE: Error filterStart

[Andrew Nagy]

Hello, I downloaded the latest nightly snapshot of Solr and replaced my 
existing war with the new one.  Once I restarted tomcat, I get this error:


SEVERE: Error filterStart
Apr 5, 2007 10:11:28 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [/solr] startup failed due to previous errors

Any ideas as to what is causing this?  I deleted my index to start with 
a clean slate but I did not change any of my config files, do I need to 
update these or are the backwards compatible?


Thanks!
Andrew

Schema validator/debugger

[Andrew Nagy]

Does anyone have a good method of debugging a schema?
I have been struggling to get my new schema to run for the past couple 
of days and just do not see anything wrong with the XML document.


Thanks
Andrew

Re: Schema validator/debugger

[Andrew Nagy]

Ryan McKinley wrote:
What errors are you getting?  Are there exceptions in the log when it 
starts up?

Just a null pointer exception.

I added a few fields to my schema, and then replaced my solr war file 
with the latest build (see my message from a week ago).  It wouldn't 
work, so I assumed something wrong with the latest build.  I then 
reverted back to my 1.1.0 copy and am getting these errors.  The XML is 
valid, but that is all I can test.


Here is a snapshot of my logs, everything starts up fine - but then I 
click on "ping" and get a server error:


INFO: Deploying web application archive solr.war
Apr 10, 2007 11:35:23 AM org.apache.solr.servlet.SolrServlet init
INFO: SolrServlet.init()
Apr 10, 2007 11:35:23 AM org.apache.solr.servlet.SolrServlet init
INFO: No /solr/home in JNDI
Apr 10, 2007 11:35:23 AM org.apache.solr.servlet.SolrServlet init
INFO: user.dir=/office/local
Apr 10, 2007 11:35:23 AM org.apache.solr.core.Config getInstanceDir
INFO: Solr home defaulted to 'solr/'
Apr 10, 2007 11:35:23 AM org.apache.solr.core.SolrConfig initConfig
INFO: Loaded SolrConfig: solrconfig.xml
Apr 10, 2007 11:35:23 AM org.apache.solr.core.RequestHandlers 
INFO: adding requestHandler standard=solr.StandardRequestHandler
Apr 10, 2007 11:35:23 AM org.apache.solr.core.RequestHandlers 
INFO: adding requestHandler dismax=solr.DisMaxRequestHandler
Apr 10, 2007 11:35:23 AM org.apache.solr.core.RequestHandlers 
INFO: adding requestHandler partitioned=solr.DisMaxRequestHandler
Apr 10, 2007 11:35:23 AM org.apache.solr.core.RequestHandlers 
INFO: adding requestHandler instock=solr.DisMaxRequestHandler
Apr 10, 2007 11:35:23 AM org.apache.solr.core.SolrCore 
INFO: Opening new SolrCore at solr/, dataDir=solr/data
Apr 10, 2007 11:35:23 AM org.apache.solr.schema.IndexSchema readConfig
INFO: Reading Solr Schema
Apr 10, 2007 11:35:23 AM org.apache.solr.schema.IndexSchema readConfig
INFO: Schema name=example
Apr 10, 2007 11:35:23 AM org.apache.solr.schema.IndexSchema readConfig
INFO: default search field is title
Apr 10, 2007 11:35:23 AM org.apache.solr.schema.IndexSchema readConfig
INFO: query parser default operator is OR
Apr 10, 2007 11:35:23 AM org.apache.solr.schema.IndexSchema readConfig
INFO: unique key field: id
Apr 10, 2007 11:35:23 AM org.apache.solr.servlet.SolrUpdateServlet init
INFO: SolrUpdateServlet.init() done
Apr 10, 2007 11:35:24 AM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8081
Apr 10, 2007 11:35:24 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Apr 10, 2007 11:35:24 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/25  config=null
Apr 10, 2007 11:35:24 AM org.apache.catalina.storeconfig.StoreLoader load
INFO: Find registry server-registry.xml at classpath resource
Apr 10, 2007 11:35:24 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 2659 ms
Apr 10, 2007 11:35:40 AM org.apache.solr.core.SolrException log
SEVERE: java.lang.RuntimeException: java.lang.NullPointerException
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:363)
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:259)
   at 
org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:152)
   at 
org.apache.solr.request.DisMaxRequestHandler.handleRequest(DisMaxRequestHandler.java:231)

   at org.apache.solr.core.SolrCore.execute(SolrCore.java:595)
   at org.apache.jsp.admin.ping_jsp._jspService(ping_jsp.java:75)
   at 
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
   at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)
   at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
   at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
   at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
   at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
   at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
   at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
   at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
   at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
   at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
   at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
   at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
   at 
org.apache.tomcat.ut

Re: Schema validator/debugger

[Andrew Nagy]

Andrew Nagy wrote:

Ryan McKinley wrote:
What errors are you getting?  Are there exceptions in the log when it 
starts up?

Just a null pointer exception.

I added a few fields to my schema, and then replaced my solr war file 
with the latest build (see my message from a week ago).  It wouldn't 
work, so I assumed something wrong with the latest build.  I then 
reverted back to my 1.1.0 copy and am getting these errors.  The XML 
is valid, but that is all I can test.


I just installed SOLR-75 patch and the "schema browser" is able to view 
the schema perfectly.  When I used the default schema with SOLR, 
everything is fine, but when I replace my schema ... it's throws this 
NullPointerException 


Any ideas?

Andrew

Re: Schema validator/debugger

[Andrew Nagy]

Ryan McKinley wrote:

Are you using the example solrconfig.xml?  The stack trace looks like
an error finding the solr.home index directory, that is configured in
solrconfig.xml, not schema.xml
Yeah, I noticed that too ... but I don't understand why it can't find 
the home.  I have the data home set to ./solr/data and I always start 
tomcat from the parent directory.


If the example schema.xml works, can you try adding a little bit of
your schema at a time?

Yeah, that is my last resort.  I guess I have no choice!

Thanks
Andrew

Re: Schema validator/debugger

[Andrew Nagy]

Greg Ludington wrote:

I just installed SOLR-75 patch and the "schema browser" is able to view
the schema perfectly.  When I used the default schema with SOLR,
everything is fine, but when I replace my schema ... it's throws this
NullPointerException 


One thing to note is that the "schema browser" as currently in SOLR-75
just transforms the xml file -- it does not represent any of the
actual parsing of the configuration by Solr upon startup, and so would
not be able to catch any errors Solr might encounter in the schema.
Sure, but it's an easier way for my non-textual brain to grasp the 
schema file :)


Nice job, BTW.

Off topic a bit, Has anyone set forth to build a new admin interface for 
SOLR?  I build a lot of admin interfaces for my day job and would love 
to give the admin module a bit of a tune-up (I won't use the term overhaul).

Re: Schema validator/debugger

[Andrew Nagy]

Andrew Nagy wrote:

Ryan McKinley wrote:

If the example schema.xml works, can you try adding a little bit of
your schema at a time?

Yeah, that is my last resort.  I guess I have no choice!
So I did this, and I ended up removing all of my fields and copyfields 
with still no luck.  I took a fresh copy of the example schema, replaced 
my fields and copyfields and it still the error persists.


One thing I noticed is that in the admin it goes from "Solr Admin 
(example)" to "Solr Admin (unknown)" which makes me think it is having a 
hard time parsing the schema file?


Would it help if I posted my schema file?

Thanks!
Andrew

Re: Schema validator/debugger

[Andrew Nagy]

Yonik Seeley wrote:

On 4/11/07, Andrew Nagy <[EMAIL PROTECTED]> wrote:

> If the example schema.xml works, can you try adding a little bit of
> your schema at a time?
Yeah, that is my last resort.  I guess I have no choice!


That certainly is strange... Sounds like you definitely hit some sort
of Solr bug, but I don't see how an error in schema parsing can be
lost w/o it going into a log somewhere.  Did you check the other
Tomcat log files?

The only tomcat log file that has any mention of an error is catalina.out


Perhaps try putting your schema.xml into the example distributed with
Solr (jetty based) and see what happens.
I can try this.  Will jetty create a log file somewhere? 


Thanks
Andrew

Re: Schema validator/debugger

[Andrew Nagy]

Ryan McKinley wrote:


Off topic a bit, Has anyone set forth to build a new admin interface for
SOLR?  I build a lot of admin interfaces for my day job and would love
to give the admin module a bit of a tune-up (I won't use the term 
overhaul).




i think we definitely need an updated admin interface, yes!

Ideally, we could get rid of all the JSP and do everything with
requestHandlers + XSLT.  Then everything could work with only a JRE -
it would not require a JDK and all the jsp jars.  AND this would let
other clients have access to the same info with JSON/xml etc

check:
http://issues.apache.org/jira/browse/SOLR-162

This puts a lot of the functionality as handlers, but still needs an
XSLT front end
Yes, Erik Hatcher showed this off a bit at a conference I was at in 
February.  I was thinking about building off of these xml files.  I do 
quite a bit with XSLT in my day job, so I would love to play around with 
it.  I think this is something that is desperately needed in SOLR, I 
know it would make my life a lot easier :)


I will take a stab at some of this soon ...

Andrew

Re: Schema validator/debugger

[Andrew Nagy]

So maybe I am doing something really dumb, but for testing purposes I 
took a copy of the example schema.xml file and removed all of the fields 
except for 2 to start with something really basic and it is throwing the 
nullpointerexception again.


I attached the file if it is of any help

I am using a fresh copy of the nightly build.

Ryan McKinley wrote:


Would it help if I posted my schema file?



yes






  

  


































  

  




  







  
  







  





  







  


 


 
   

   
   

 

 
 id

 
 title

 
 

  

 
 

Re: Schema validator/debugger

[Andrew Nagy]

Yonik Seeley wrote:

I dropped your schema.xml directly into the Solr example (using
Jetty), fired it up, and everything works fine!?

Okay, I switched over to Jetty and now I get a different error:
SEVERE: org.apache.solr.core.SolrException: undefined field text


Are you sure you are using the stock solrconfig.xml, etc?

Yep.


I apologize ahead of time if I am being blatantly stupid about this. :)

Re: Schema validator/debugger

[Andrew Nagy]

Ryan McKinley wrote:

With a clean checkout, you can run:
$ ant example
$ cd example
$ java -jar start.jar

and things work OK.

But, when you delete all but the two fields, you get an exception 
somewhere?
Well, I was working from my own directory, not the example directory.  I 
can give that a try ... but all I did was copy everything from the 
example dir to a different dir.


But, basically ... yes.

I will delete all instances of solr on my server and start from scratch 
... i must have something screwy somewhere.

Re: Schema validator/debugger

[Andrew Nagy]

Yonik Seeley wrote:

Oh wait... Andrew, were you always testing via "ping"?

Check out what the ping query is configured as in solrconfig.xml:

   
qt=dismax&q=solr&start=3&fq=id:[* TO *]&fq=cat:[* 
TO *]

   

Perhaps we should change it to something simple by default???  "q=solr"?

That solves the Jetty failure mystery... so it looks like you either
have a tomcat setup problem, or a Solr bug that only shows under
tomcat.


Yes, this is the problem!  Good catch :)  I have been testing via ping.

However this still does not solve my original problem ... I will dig a 
bit more and see what I can find.


Thanks
Andrew

Ant build question

[Andrew Nagy]

Hello, I would like to play with patch SOLR-69 and am trying to rebuild 
solr using ant with some difficulties.  When I try to run ant, I get an 
error saying that it can't find junit, but junit is in the ant lib 
directory.  I had a file called ant-junit.jar in the lib directory and 
copied it to junit.jar, but it still gave me the error.  Any ideas as to 
what is causing this?


Thanks
Andrew

# ant example
Buildfile: build.xml

checkJunitPresence:

BUILD FAILED
/office/src/apache-solr-nightly/build.xml:240: 
##

 JUnit not found.
 Please make sure junit.jar is in ANT_HOME/lib, or made available
 to Ant using other mechanisms like -lib or CLASSPATH.
 ##

Total time: 0 seconds

# echo $ANT_HOME
/usr/local/ant

# ls /usr/local/ant/lib/
ant-antlr.jar ant-apache-oro.jar   ant-commons-net.jar  
ant-javamail.jar  ant-junit.jar ant-starteam.jar   ant-vaj.jar   
junit.jar
ant-apache-bcel.jar   ant-apache-regexp.jarant-icontract.jar
ant-jdepend.jar   ant-launcher.jar  ant-stylebook.jar  ant-weblogic.jar  
README
ant-apache-bsf.jarant-apache-resolver.jar  ant-jai.jar  
ant-jmf.jar   ant-netrexx.jar   ant-swing.jar  ant-xalan1.jar
xercesImpl.jar
ant-apache-log4j.jar  ant-commons-logging.jar  ant.jar  
ant-jsch.jar  ant-nodeps.jarant-trax.jar   ant-xslp.jar  
xml-apis.jar

Re: Ant build question

[Andrew Nagy]

Erik Hatcher wrote:

ant-junit.jar != junit.jar

rename it back, and grab junit.jar from junit.org

Easy enough.

Thanks!
Andrew

compile error with SOLR 69 MoreLikeThis patch

[Andrew Nagy]

I downloaded and patched my solr source with the latest solr69 patch and 
whenever I run ant I get an error:


[javac] 
/office/src/apache-solr-nightly/src/java/org/apache/solr/handler/MoreLikeThisHandler.java:145: 
cannot find symbol

[javac] symbol  : variable DEFAULT_MIN_DOC_FREQ
[javac] location: class org.apache.lucene.search.similar.MoreLikeThis
[javac]   mlt.setMinDocFreq(
params.getInt(MoreLikeThisParams.MIN_DOC_FREQ,  
MoreLikeThis.DEFAULT_MIN_DOC_FREQ));
[javac] 
^



Did I miss something, or maybe I have a non-compatible version of SOLR?

Thanks
Andrew

RE: compile error with SOLR 69 MoreLikeThis patch

[Andrew Nagy]

That did the trick.  However, now I am trying to apply the patch to a fresh 
copy of solr on another server.  Whenever I do so, I get the following error 
from the patch command:

patching file src/java/org/apache/solr/request/StandardRequestHandler.java
Hunk #1 succeeded at 23 with fuzz 2 (offset -1 lines).
Hunk #2 FAILED at 31.
Hunk #3 succeeded at 146 with fuzz 2 (offset 13 lines).
1 out of 3 hunks FAILED -- saving rejects to file 
src/java/org/apache/solr/request/StandardRequestHandler.java.rej
(Stripping trailing CRs from patch.)
patching file src/java/org/apache/solr/handler/MoreLikeThisHandler.java
(Stripping trailing CRs from patch.)
patching file src/java/org/apache/solr/util/MoreLikeThisParams.java
(Stripping trailing CRs from patch.)
patching file example/solr/conf/solrconfig.xml
Hunk #1 FAILED at 385.
1 out of 1 hunk FAILED -- saving rejects to file 
example/solr/conf/solrconfig.xml.rej


Any ideas on how to get passed this?

Andrew

From: Brian Whitman [EMAIL PROTECTED]
Sent: Wednesday, May 16, 2007 11:14 AM
To: solr-user
Subject: Re: compile error with SOLR 69 MoreLikeThis patch

Change it to DEFALT or change the spelling error in the Lucene version.

On May 16, 2007, at 12:13 PM, Andrew Nagy wrote:

> I downloaded and patched my solr source with the latest solr69
> patch and whenever I run ant I get an error:
>
> [javac] /office/src/apache-solr-nightly/src/java/org/apache/solr/
> handler/MoreLikeThisHandler.java:145: cannot find symbol
> [javac] symbol  : variable DEFAULT_MIN_DOC_FREQ
> [javac] location: class org.apache.lucene.search.similar.MoreLikeThis
> [javac]   mlt.setMinDocFreq(params.getInt
> (MoreLikeThisParams.MIN_DOC_FREQ,
> MoreLikeThis.DEFAULT_MIN_DOC_FREQ));
> [javac]
>   ^
>
>
> Did I miss something, or maybe I have a non-compatible version of
> SOLR?
>
> Thanks
> Andrew

--
http://variogr.am/
[EMAIL PROTECTED]

RE: compile error with SOLR 69 MoreLikeThis patch

[Andrew Nagy]

That's the problem, as I suspected.  I was using a fresh copy of the 1.1.0.  I 
want to use the MLT feature in my production environment, but I guess I will 
have to roll out the trunk version to my production server.

Thanks!

From: Brian Whitman [EMAIL PROTECTED]
Sent: Thursday, May 24, 2007 10:36 AM
To: solr-user@lucene.apache.org
Subject: Re: compile error with SOLR 69 MoreLikeThis patch

On May 24, 2007, at 10:29 AM, Andrew Nagy wrote:

> That did the trick.  However, now I am trying to apply the patch to
> a fresh copy of solr on another server.  Whenever I do so, I get
> the following error from the patch command:
>

I just patched trunk two days ago with the latest (posted three days
ago) SOLR-69 patch, cleanly. Are you sure you are trying to patch the
trunk version?

RE: compile error with SOLR 69 MoreLikeThis patch

[Andrew Nagy]

Chris, thanks for the tip.  I think I am okay with pushing the trunk to my 
production server.  As we say around here, if you want to be on the bleeding 
edge, you have to be okay with bleeding every once in a while :)

Thanks again
Andrew

From: Chris Hostetter [EMAIL PROTECTED]
Sent: Thursday, May 24, 2007 2:18 PM
To: solr-user@lucene.apache.org
Subject: RE: compile error with SOLR 69 MoreLikeThis patch

: That's the problem, as I suspected.  I was using a fresh copy of the
: 1.1.0.  I want to use the MLT feature in my production environment, but
: I guess I will have to roll out the trunk version to my production
: server.

if your goal is to use the MoreLikeThisHanler then you don't need the
entire trun .. it should work fine with Solr 1.1.0 ... the
mlt functionality added to StandardRequestHandler might be a bit more
challenging, but it should still be possible to retrofit the newer
StandardRequestHandler into the 1.1.0 release if you wnated to.

(Not that i dont' think the trunk is stable, i'm just saying there are
ways to get it working without completley upgrading)



-Hoss

RE: compile error with SOLR 69 MoreLikeThis patch

[Andrew Nagy]

While I am on this topic, I think it might be nice to have a nightly build for 
downloading or is their something like that in place, now?

From: Chris Hostetter [EMAIL PROTECTED]
Sent: Thursday, May 24, 2007 2:18 PM
To: solr-user@lucene.apache.org
Subject: RE: compile error with SOLR 69 MoreLikeThis patch

: That's the problem, as I suspected.  I was using a fresh copy of the
: 1.1.0.  I want to use the MLT feature in my production environment, but
: I guess I will have to roll out the trunk version to my production
: server.

if your goal is to use the MoreLikeThisHanler then you don't need the
entire trun .. it should work fine with Solr 1.1.0 ... the
mlt functionality added to StandardRequestHandler might be a bit more
challenging, but it should still be possible to retrofit the newer
StandardRequestHandler into the 1.1.0 release if you wnated to.

(Not that i dont' think the trunk is stable, i'm just saying there are
ways to get it working without completley upgrading)



-Hoss

RE: compile error with SOLR 69 MoreLikeThis patch

[Andrew Nagy]

duh++

Also, Where should I put the jar file for the patch?

From: Erik Hatcher [EMAIL PROTECTED]
Sent: Thursday, May 24, 2007 2:37 PM
To: solr-user@lucene.apache.org
Subject: Re: compile error with SOLR 69 MoreLikeThis patch

Andrew,

Nightlies are available here: <http://people.apache.org/builds/lucene/
solr/nightly/> (a link exists on the wiki main page, for future
reference).

Erik


On May 24, 2007, at 2:28 PM, Andrew Nagy wrote:

> While I am on this topic, I think it might be nice to have a
> nightly build for downloading or is their something like that in
> place, now?
> 
> From: Chris Hostetter [EMAIL PROTECTED]
> Sent: Thursday, May 24, 2007 2:18 PM
> To: solr-user@lucene.apache.org
> Subject: RE: compile error with SOLR 69 MoreLikeThis patch
>
> : That's the problem, as I suspected.  I was using a fresh copy of the
> : 1.1.0.  I want to use the MLT feature in my production
> environment, but
> : I guess I will have to roll out the trunk version to my production
> : server.
>
> if your goal is to use the MoreLikeThisHanler then you don't need the
> entire trun .. it should work fine with Solr 1.1.0 ... the
> mlt functionality added to StandardRequestHandler might be a bit more
> challenging, but it should still be possible to retrofit the newer
> StandardRequestHandler into the 1.1.0 release if you wnated to.
>
> (Not that i dont' think the trunk is stable, i'm just saying there are
> ways to get it working without completley upgrading)
>
>
>
> -Hoss

MoreLikeThis woes

[Andrew Nagy]

Hello,  I have been playing off and on with the more like this patch and I 
really want to get it working well.  I have the patch installed and I have 
about 500K bibliographic records in my solr index.

My MLT query uses a fieldlist of about 5 or 6 fields.  There are a mix of 
string and text fields.  They are all in a TermVector.  I have played around 
with the mindf values.  With about 90% of my mlt queries solr it returns no 
matches and the remaining 10% get completely irrelevant results.

Are there any "best practices" for the MLT search or am I possibly doing 
something wrong?

Thanks
Andrew

Spell Check Handler

[Andrew Nagy]

Hello, is there any documentation on how to use the new spell check module?

Thanks
Andrew

Facet Field Limits

[Andrew Nagy]

Hello, I would like to generate a list of facets, let's say on 5 fields.  I 
have the facet limit set to 5 so that for each of the 5 fields there will only 
by up to 5 values.

My question is: Is there a way to change the limit per field?  Let's say on 
facet 2 I would like to display 10 values instead of 5 like the other facets.

Thanks!
Andrew