schema configuration with different kind of score report

[Sushan Rungta]

Kindly guide me that how shall I configure solr lucene with the below 
kind of requirements:


The query is "abc"

Documents are:

a) abc
b) abcd
c) xyz ab c mno
d) ab

I require the score for each of the above mentioned documents with 
the above mentioned query to be displayed as:


For document (a) 100%  (Computation: abc/abc)
For document (b) 75%(Computation: abc/abcd)
For document (c) 33%(Computation: abc/xyz ab c mno)
For document (d) 75%	(Computation: abc/ab, where document is found 
with complete match)


regards,

Sushan Rungta

Re: schema configuration with different kind of score report

[Avlesh Singh]

Why not stick to lucene score for each document then building your own? The
easiest way of getting the relevance score for each document is to add the
"debugQuery=true" parameter to your request handler.

Cheers
Avlesh

On Mon, Aug 17, 2009 at 12:32 PM, Sushan Rungta  wrote:

> Kindly guide me that how shall I configure solr lucene with the below kind
> of requirements:
>
> The query is "abc"
>
> Documents are:
>
> a) abc
> b) abcd
> c) xyz ab c mno
> d) ab
>
> I require the score for each of the above mentioned documents with the
> above mentioned query to be displayed as:
>
> For document (a) 100%  (Computation: abc/abc)
> For document (b) 75%(Computation: abc/abcd)
> For document (c) 33%(Computation: abc/xyz ab c mno)
> For document (d) 75%(Computation: abc/ab, where document is found with
> complete match)
>
> regards,
>
> Sushan Rungta
>
>

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Constantijn Visinescu]

near the bottom of my web.xml (just above ) i got


   solr/home
   path/to/solr
   java.lang.String


While you're at it you might want to make sure the following line in your
solrconfig.xml is commented out
  

next you should copy the sorl directory (the one with the conf, data and bin
subdiretories) to the path you specified in the web.xml

Repackage the war and redeploy, that should do it :)


On Sat, Aug 15, 2009 at 11:59 AM, Aaron Aberg  wrote:

> Ok, I'm still having a problem. I believe that I have to set the
> solr.solr.home variable somewhere. In Windows, that was easy because
> you could just open up the system tray app for tomcat and set it
> there. How can I do that in linux? After googling, it suggested that I
> set an environment variable in the web.xml file inside the war file. I
> even saw a commented out section for setting the home variable. I set
> it in there but it still is giving me this error:
>
> INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader 
> INFO: Solr home set to 'solr/'
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-6.1.3.jar' to
> Solr classloader
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar'
> to Solr classloader
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar'
> to Solr classloader
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jsp-2.1/' to Solr
> classloader
> Aug 14, 2009 6:16:42 PM org.apache.solr.servlet.SolrDispatchFilter init
> SEVERE: Could not start SOLR. Check solr/home property
> java.lang.ExceptionInInitializerError
>
> Any ideas?
>
> --Aaron
>
> On Sat, Aug 15, 2009 at 12:40 AM, Shalin Shekhar
> Mangar wrote:
> > On Sat, Aug 15, 2009 at 6:52 AM, Aaron Aberg 
> wrote:
> >
> >> My client is using a dedicated server with plesk for the control
> >> panel. The hosting provider says that anything done using the control
> >> panel is supported by their tech support, so if i try anything using
> >> SSH, it voids that warranty. Its easy to install a servlet through
> >> plesk anyway, I upload the war file using the the java servlet
> >> installer. A sample servlet has been installed so I know this part
> >> works.
> >>
> >> However, when I install solr, i get what looks like a warning icon and
> >> if I hover over it the tool tip text says this:
> >>
> >> "Actual status of the application does not correspond to the status
> >> retrieved from the database."
> >>
> >> The host providers support team says that there is something wrong
> >> with the war file (big help). Since I kind of stuck using tomcat 5.5,
> >> is there an older version of solr that I should be using? How can I
> >> fix this so that I can use solr?
> >>
> >
> > Solr works fine with Tomcat 5.5
> >
> > Solr is a little different from other web applications in that it uses a
> > specified directory as "solr home" which contains configuration (schema,
> > solrconfig etc). The solrconfig also tells solr where to store the index.
> >
> > The solr home can be specified in many ways (environment variable, JNDI
> or
> > even in web.xml). I guess you should specify the absolute path to the
> solr
> > home in one of these ways (specifying it in web.xml and re-packaging the
> war
> > may be easier for you?). You'd need to upload the contents of the solr
> home
> > at the specified location on your server (look at the example directory
> > which comes with the solr download). Specify the absolute path to data
> > directory in solrconfig.xml (make sure it is writable).
> >
> > http://wiki.apache.org/solr/SolrTomcat
> > http://wiki.apache.org/solr/HackingSolr
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
>

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Aaron Aberg]

Thanks for the help. I commented out that line in solrconfig.xml like
you said. my web.xml file has this entry in it:


solr/home
/usr/share/tomcat5/solr
java.lang.String
  

And here is my file structure for solr home:

/usr/share/tomcat5/solr/
/usr/share/tomcat5/solr/bin
/usr/share/tomcat5/solr/bin/* (assuming everything under bin is correct)
/usr/share/tomcat5/solr/data (created and chown'ed this to tomcat:tomcat)
/usr/share/tomcat5/solr/logs
/usr/share/tomcat5/solr/conf
/usr/share/tomcat5/solr/conf/* (assuming everything under conf is correct)
/usr/share/tomcat5/solr/start.jar
/usr/share/tomcat5/solr/etc
/usr/share/tomcat5/solr/etc/webdefault.xml
/usr/share/tomcat5/solr/etc/jetty.xml
/usr/share/tomcat5/solr/testing.mywebk9.com
/usr/share/tomcat5/solr/testing.mywebk9.com/conf
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/schema.xml
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/solrconfig.xml
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/synonyms.txt
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/example_atom.xsl
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/luke.xsl
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/example_rss.xsl
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/example.xsl
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/elevate.xml
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/scripts.conf
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/protwords.txt
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/spellings.txt
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/admin-extra.html
/usr/share/tomcat5/solr/testing.mywebk9.com/conf/stopwords.txt
/usr/share/tomcat5/solr/lib
/usr/share/tomcat5/solr/lib/jetty-6.1.3.jar
/usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar
/usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar
/usr/share/tomcat5/solr/lib/jsp-2.1
/usr/share/tomcat5/solr/lib/jsp-2.1/jsp-api-2.1.jar
/usr/share/tomcat5/solr/lib/jsp-2.1/jsp-2.1.jar
/usr/share/tomcat5/solr/lib/jsp-2.1/ant-1.6.5.jar
/usr/share/tomcat5/solr/lib/jsp-2.1/core-3.1.1.jar
/usr/share/tomcat5/solr/solr.xml (<-- see! here is my solr.xml setup
for multicore)
/usr/share/tomcat5/solr/mywebk9.com
/usr/share/tomcat5/solr/mywebk9.com/conf
/usr/share/tomcat5/solr/mywebk9.com/conf/schema.xml
/usr/share/tomcat5/solr/mywebk9.com/conf/solrconfig.xml
/usr/share/tomcat5/solr/mywebk9.com/conf/synonyms.txt
/usr/share/tomcat5/solr/mywebk9.com/conf/xslt
/usr/share/tomcat5/solr/mywebk9.com/conf/xslt/example_atom.xsl
/usr/share/tomcat5/solr/mywebk9.com/conf/xslt/luke.xsl
/usr/share/tomcat5/solr/mywebk9.com/conf/xslt/example_rss.xsl
/usr/share/tomcat5/solr/mywebk9.com/conf/xslt/example.xsl
/usr/share/tomcat5/solr/mywebk9.com/conf/elevate.xml
/usr/share/tomcat5/solr/mywebk9.com/conf/scripts.conf
/usr/share/tomcat5/solr/mywebk9.com/conf/protwords.txt
/usr/share/tomcat5/solr/mywebk9.com/conf/spellings.txt
/usr/share/tomcat5/solr/mywebk9.com/conf/admin-extra.html
/usr/share/tomcat5/solr/mywebk9.com/conf/stopwords.txt
/usr/share/tomcat5/solr/webapps
/usr/share/tomcat5/solr/webapps/solr.war

I'm still getting this error:

Aug 16, 2009 6:14:44 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
locateInstanceDir
INFO: Using JNDI solr.home: /usr/share/tomcat5/solr
Aug 16, 2009 6:14:44 PM org.apache.solr.core.CoreContainer$Initializer
initialize
INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader 
INFO: Solr home set to '/usr/share/tomcat5/solr/'
Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-6.1.3.jar' to
Solr classloader
Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Adding 'file:/usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar'
to Solr classloader
Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar'
to Solr classloader
Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jsp-2.1/' to Solr classloader
Aug 16, 2009 6:14:44 PM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start SOLR. Check solr/home property

I don't understand... it says solr home set to
'/usr/share/tomcat5/solr/' but the error says that there is something
wrong with the solr/home directory. Is my file structure incorrect and
its been masked by this error?

--Aaron

On Mon, Aug 17, 2009 at 12:57 AM, Constantijn
Visinescu wrote:
> near the bottom of my web.xml (just above ) i got
>
>    
>       solr/home
>       path/to/solr
>       java.lang.String
>    
>
> While you're at it you might want to make sure the following line in your
> solrconfig.xml is commented out
>  
>
> next you should c

Re: schema configuration with different kind of score report

[Sushan Rungta]

This doesnot solve my purpose, as my requirement is different. Kindly 
check the document "d",
which I have mentioned & the computation of score for that kind of 
document will be different.


Hence, some sort of different query will be applied, which I am 
unable to ascertain.


Regards,

Sushan Rungta


At 12:44 PM 8/17/2009, Avlesh Singh wrote:

Why not stick to lucene score for each document then building your own? The
easiest way of getting the relevance score for each document is to add the
"debugQuery=true" parameter to your request handler.

Cheers
Avlesh

On Mon, Aug 17, 2009 at 12:32 PM, Sushan Rungta  wrote:

> Kindly guide me that how shall I configure solr lucene with the below kind
> of requirements:
>
> The query is "abc"
>
> Documents are:
>
> a) abc
> b) abcd
> c) xyz ab c mno
> d) ab
>
> I require the score for each of the above mentioned documents with the
> above mentioned query to be displayed as:
>
> For document (a) 100%  (Computation: abc/abc)
> For document (b) 75%(Computation: abc/abcd)
> For document (c) 33%(Computation: abc/xyz ab c mno)
> For document (d) 75%(Computation: abc/ab, where document is found with
> complete match)
>
> regards,
>
> Sushan Rungta
>
>

Re: schema configuration with different kind of score report

[Avlesh Singh]

I am definitely missing something here.
Do you want to fetch a document if one of its field contains "ab" given a
search term "abc"? If you can design a field and query your index so that
you can fetch such a document, Lucene (and hence Solr) would automagically
give you the relevance score.

Cheers
Avlesh

On Mon, Aug 17, 2009 at 2:16 PM, Sushan Rungta  wrote:

> This doesnot solve my purpose, as my requirement is different. Kindly check
> the document "d",
> which I have mentioned & the computation of score for that kind of document
> will be different.
>
> Hence, some sort of different query will be applied, which I am unable to
> ascertain.
>
> Regards,
>
> Sushan Rungta
>
>
> At 12:44 PM 8/17/2009, Avlesh Singh wrote:
>
>> Why not stick to lucene score for each document then building your own?
>> The
>> easiest way of getting the relevance score for each document is to add the
>> "debugQuery=true" parameter to your request handler.
>>
>> Cheers
>> Avlesh
>>
>> On Mon, Aug 17, 2009 at 12:32 PM, Sushan Rungta 
>> wrote:
>>
>> > Kindly guide me that how shall I configure solr lucene with the below
>> kind
>> > of requirements:
>> >
>> > The query is "abc"
>> >
>> > Documents are:
>> >
>> > a) abc
>> > b) abcd
>> > c) xyz ab c mno
>> > d) ab
>> >
>> > I require the score for each of the above mentioned documents with the
>> > above mentioned query to be displayed as:
>> >
>> > For document (a) 100%  (Computation: abc/abc)
>> > For document (b) 75%(Computation: abc/abcd)
>> > For document (c) 33%(Computation: abc/xyz ab c mno)
>> > For document (d) 75%(Computation: abc/ab, where document is found
>> with
>> > complete match)
>> >
>> > regards,
>> >
>> > Sushan Rungta
>> >
>> >
>>
>
>
>

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Constantijn Visinescu]

Not sure what's going on but i see jetty stuff scrolling by, that can't be
right :)
Jetty and Tomcat are 2 seperate webservers for serving java applications.
the 2 of them mixing doesn't sound like a good idea.
Jetty is included in the examples for .. well .. example purposes ... but
it's not a part of solr itself and you don't want it mucking up tomcat :)

Try throwing everything out except the conf (and maybe the bin) dir and see
if it works ?

The data dir should automatically be recreated with a blank index if solr is
starting up properly.


On Mon, Aug 17, 2009 at 10:30 AM, Aaron Aberg  wrote:

> Thanks for the help. I commented out that line in solrconfig.xml like
> you said. my web.xml file has this entry in it:
>
> 
>solr/home
> /usr/share/tomcat5/solr
> java.lang.String
>  
>
> And here is my file structure for solr home:
>
> /usr/share/tomcat5/solr/
> /usr/share/tomcat5/solr/bin
> /usr/share/tomcat5/solr/bin/* (assuming everything under bin is correct)
> /usr/share/tomcat5/solr/data (created and chown'ed this to tomcat:tomcat)
> /usr/share/tomcat5/solr/logs
> /usr/share/tomcat5/solr/conf
> /usr/share/tomcat5/solr/conf/* (assuming everything under conf is correct)
> /usr/share/tomcat5/solr/start.jar
> /usr/share/tomcat5/solr/etc
> /usr/share/tomcat5/solr/etc/webdefault.xml
> /usr/share/tomcat5/solr/etc/jetty.xml
> /usr/share/tomcat5/solr/testing.mywebk9.com
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/schema.xml
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/solrconfig.xml
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/synonyms.txt
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/example_atom.xsl
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/luke.xsl
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/example_rss.xsl
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/xslt/example.xsl
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/elevate.xml
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/scripts.conf
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/protwords.txt
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/spellings.txt
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/admin-extra.html
> /usr/share/tomcat5/solr/testing.mywebk9.com/conf/stopwords.txt
> /usr/share/tomcat5/solr/lib
> /usr/share/tomcat5/solr/lib/jetty-6.1.3.jar
> /usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar
> /usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar
> /usr/share/tomcat5/solr/lib/jsp-2.1
> /usr/share/tomcat5/solr/lib/jsp-2.1/jsp-api-2.1.jar
> /usr/share/tomcat5/solr/lib/jsp-2.1/jsp-2.1.jar
> /usr/share/tomcat5/solr/lib/jsp-2.1/ant-1.6.5.jar
> /usr/share/tomcat5/solr/lib/jsp-2.1/core-3.1.1.jar
> /usr/share/tomcat5/solr/solr.xml (<-- see! here is my solr.xml setup
> for multicore)
> /usr/share/tomcat5/solr/mywebk9.com
> /usr/share/tomcat5/solr/mywebk9.com/conf
> /usr/share/tomcat5/solr/mywebk9.com/conf/schema.xml
> /usr/share/tomcat5/solr/mywebk9.com/conf/solrconfig.xml
> /usr/share/tomcat5/solr/mywebk9.com/conf/synonyms.txt
> /usr/share/tomcat5/solr/mywebk9.com/conf/xslt
> /usr/share/tomcat5/solr/mywebk9.com/conf/xslt/example_atom.xsl
> /usr/share/tomcat5/solr/mywebk9.com/conf/xslt/luke.xsl
> /usr/share/tomcat5/solr/mywebk9.com/conf/xslt/example_rss.xsl
> /usr/share/tomcat5/solr/mywebk9.com/conf/xslt/example.xsl
> /usr/share/tomcat5/solr/mywebk9.com/conf/elevate.xml
> /usr/share/tomcat5/solr/mywebk9.com/conf/scripts.conf
> /usr/share/tomcat5/solr/mywebk9.com/conf/protwords.txt
> /usr/share/tomcat5/solr/mywebk9.com/conf/spellings.txt
> /usr/share/tomcat5/solr/mywebk9.com/conf/admin-extra.html
> /usr/share/tomcat5/solr/mywebk9.com/conf/stopwords.txt
> /usr/share/tomcat5/solr/webapps
> /usr/share/tomcat5/solr/webapps/solr.war
>
> I'm still getting this error:
>
> Aug 16, 2009 6:14:44 PM

'Connection reset' in DataImportHandler Development Console

[Andrew Clegg]

Hi folks,

I'm trying to use the Debug Now button in the development console to test
the effects of some changes in my data import config (see attached).

However, each time I click it, the right-hand frame fails to load -- it just
gets replaced with the standard 'connection reset' message from Firefox, as
if the server's dropped the HTTP connection.

Everything else seems okay -- I can run queries in Solr Admin without any
problems, and all the other buttons in the dev console work -- status,
document count, reload config etc.

There's nothing suspicious in Tomcat's catalina.out either. If I hit Reload
Config, then Status, then Debug Now, I get this:


17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImportHandler
processConfiguration
INFO: Processing configuration from solrconfig.xml: {config=dataconfig.xml}
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
INFO: Data Configuration loaded successfully
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: id is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: title is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: doc_type is a required field in SolrSchema . But not found in
DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: id is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: title is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: doc_type is a required field in SolrSchema . But not found in
DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: id is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: title is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: doc_type is a required field in SolrSchema . But not found in
DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: id is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: title is a required field in SolrSchema . But not found in DataConfig
17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
verifyWithSchema
INFO: doc_type is a required field in SolrSchema . But not found in
DataConfig
17-Aug-2009 13:12:12 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={clean=false&command=reload-config&commit=true&qt=/dataimport}
status=0 QTime=5 
17-Aug-2009 13:12:21 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={clean=false&command=status&commit=true&qt=/dataimport} status=0
QTime=0 


(The warnings are because the doc_type field comes out of the JDBC result
set automatically by column name -- this isn't a problem.)

Also, there's no entry in the Tomcat access log for the debug request
either, just the first two:


[17/Aug/2009:13:12:12 +0100] HTTP/1.1 cookie:- request:-  GET /solr/select
200 ?clean=false&commit=true&qt=%2Fdataimport&command=reload-config GET
/solr/select?clean=false&commit=t
rue&qt=%2Fdataimport&command=reload-config HTTP/1.1
[17/Aug/2009:13:12:21 +0100] HTTP/1.1 cookie:- request:-  GET /solr/select
200 ?clean=false&commit=true&qt=%2Fdataimport&command=status GET
/solr/select?clean=false&commit=true&qt=
%2Fdataimport&command=status HTTP/1.1


PS... Nightly build, 30th of July.

Thanks,

Andrew.

http://www.nabble.com/file/p25005850/dataconfig.xml dataconfig.xml 
-- 
View this message in context: 
http://www.nabble.com/%27Connection-reset%27-in-DataImportHandler-Development-Console-tp25005850p25005850.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: 'Connection reset' in DataImportHandler Development Console

[Noble Paul നോബിള്‍ नोब्ळ्]

apparently I do not see any command full-import, delta-import being
fired. Is that true?

On Mon, Aug 17, 2009 at 5:55 PM, Andrew Clegg wrote:
>
> Hi folks,
>
> I'm trying to use the Debug Now button in the development console to test
> the effects of some changes in my data import config (see attached).
>
> However, each time I click it, the right-hand frame fails to load -- it just
> gets replaced with the standard 'connection reset' message from Firefox, as
> if the server's dropped the HTTP connection.
>
> Everything else seems okay -- I can run queries in Solr Admin without any
> problems, and all the other buttons in the dev console work -- status,
> document count, reload config etc.
>
> There's nothing suspicious in Tomcat's catalina.out either. If I hit Reload
> Config, then Status, then Debug Now, I get this:
>
>
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImportHandler
> processConfiguration
> INFO: Processing configuration from solrconfig.xml: {config=dataconfig.xml}
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> loadDataConfig
> INFO: Data Configuration loaded successfully
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: id is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: title is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: doc_type is a required field in SolrSchema . But not found in
> DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: id is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: title is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: doc_type is a required field in SolrSchema . But not found in
> DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: id is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: title is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: doc_type is a required field in SolrSchema . But not found in
> DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: id is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: title is a required field in SolrSchema . But not found in DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.handler.dataimport.DataImporter
> verifyWithSchema
> INFO: doc_type is a required field in SolrSchema . But not found in
> DataConfig
> 17-Aug-2009 13:12:12 org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> params={clean=false&command=reload-config&commit=true&qt=/dataimport}
> status=0 QTime=5
> 17-Aug-2009 13:12:21 org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> params={clean=false&command=status&commit=true&qt=/dataimport} status=0
> QTime=0
>
>
> (The warnings are because the doc_type field comes out of the JDBC result
> set automatically by column name -- this isn't a problem.)
>
> Also, there's no entry in the Tomcat access log for the debug request
> either, just the first two:
>
>
> [17/Aug/2009:13:12:12 +0100] HTTP/1.1 cookie:- request:-  GET /solr/select
> 200 ?clean=false&commit=true&qt=%2Fdataimport&command=reload-config GET
> /solr/select?clean=false&commit=t
> rue&qt=%2Fdataimport&command=reload-config HTTP/1.1
> [17/Aug/2009:13:12:21 +0100] HTTP/1.1 cookie:- request:-  GET /solr/select
> 200 ?clean=false&commit=true&qt=%2Fdataimport&command=status GET
> /solr/select?clean=false&commit=true&qt=
> %2Fdataimport&command=status HTTP/1.1
>
>
> PS... Nightly build, 30th of July.
>
> Thanks,
>
> Andrew.
>
> http://www.nabble.com/file/p25005850/dataconfig.xml dataconfig.xml
> --
> View this message in context: 
> http://www.nabble.com/%27Connection-reset%27-in-DataImportHandler-Development-Console-tp25005850p25005850.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: 'Connection reset' in DataImportHandler Development Console

[Andrew Clegg]

Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
> 
> apparently I do not see any command full-import, delta-import being
> fired. Is that true?
> 

It seems that way -- they're not appearing in the logs. I've tried Debug Now
with both full and delta selected from the dropdown, no difference either
way.

If I click the Full Import button it starts an import okay. I don't have to
Full Import manually every time I want to debug a config change do I? That's
not what the docs say. (A full import takes about 6 or 7 hours...)

Thanks,

Andrew.
-- 
View this message in context: 
http://www.nabble.com/%27Connection-reset%27-in-DataImportHandler-Development-Console-tp25005850p25006284.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: HTTP ERROR: 500 No default field name specified

[Kevin Miller]

I am no longer getting this error.  I downloaded the latest nightly
build this morning the document I wanted worked without any problems. 


Kevin Miller
Web Services

-Original Message-
From: Kevin Miller [mailto:kevin.mil...@oktax.state.ok.us] 
Sent: Thursday, August 13, 2009 3:35 PM
To: solr-user@lucene.apache.org
Subject: HTTP ERROR: 500 No default field name specified

I have a different error once I direct the curl to look in the correct
folder for the file.  I am getting an HTTP ERROR: 500 No default field
name specified.

I am using a test word document in the exampledocs folder.  I am issuing
the curl command from the exampledocs folder.  Following is the command
I am using:

c:\curl\bin\curl
"http://echo12:8983/solr/update/extract?literal.id=doc1uprefix=attr_&map
.content=attr_content&commit=true" -F "myfi...@badnews.doc"

curl is installed on my machine at c:\curl and the .exe file is located
at c:\curl\bin

Can someone please direct me to where I can look to find out how to the
a default field name?


Kevin Miller
Oklahoma Tax Commission
Web Services

Re: Boosting relevance as terms get nearer to each other

[Michael]

Anybody have any suggestions or hints?  I'd love to score my queries in a
way that pays attention to how close together terms appear.
Michael

On Thu, Aug 13, 2009 at 12:01 PM, Michael  wrote:

> Hello,
> I'd like to score documents higher that have the user's search terms nearer
> each other.  For example, if a user searches for
>
>   a AND b AND c
>
> the standard query handler should return all documents with [a] [b] and [c]
> in them, but documents matching the phrase "a b c" should get a boost over
> those with "a x b c" over those with "b x y c z a", etc.
>
> To accomplish this, I thought I might replace the user's query with
>
>   "a b c"~10
>
> hoping that the slop term gets a higher and higher score the closer
> together [a] [b] and [c] appear.  This doesn't seem to be the case in my
> experiments; when I debug the query, there's no component of the score based
> on how close together [a] [b] and [c] are.  And I'm suspicious that this
> would make my queries a whole lot slower -- in reality my users' queries get
> expanded quite a bit already, and I'd thus need to add many slop terms.
>
> Perhaps instead I could modify the Standard query handler to examine the
> distance between all ANDed tokens, and boost proportionally to the inverse
> of their average distance apart.  I've never modified a query handler before
> so I have no idea if this is possible.
>
> Any suggestions on what approach I should take?  The less I have to modify
> Solr, the better -- I'd prefer a query-side solution over writing a plugin
> over forking the standard query handler.
>
> Thanks in advance!
> Michael
>

Re: Boosting relevance as terms get nearer to each other

[Mark Miller]

Dismax QueryParser with pf and ps params?

http://wiki.apache.org/solr/DisMaxRequestHandler

--
- Mark

http://www.lucidimagination.com



Michael wrote:

Anybody have any suggestions or hints?  I'd love to score my queries in a
way that pays attention to how close together terms appear.
Michael

On Thu, Aug 13, 2009 at 12:01 PM, Michael  wrote:

  

Hello,
I'd like to score documents higher that have the user's search terms nearer
each other.  For example, if a user searches for

  a AND b AND c

the standard query handler should return all documents with [a] [b] and [c]
in them, but documents matching the phrase "a b c" should get a boost over
those with "a x b c" over those with "b x y c z a", etc.

To accomplish this, I thought I might replace the user's query with

  "a b c"~10

hoping that the slop term gets a higher and higher score the closer
together [a] [b] and [c] appear.  This doesn't seem to be the case in my
experiments; when I debug the query, there's no component of the score based
on how close together [a] [b] and [c] are.  And I'm suspicious that this
would make my queries a whole lot slower -- in reality my users' queries get
expanded quite a bit already, and I'd thus need to add many slop terms.

Perhaps instead I could modify the Standard query handler to examine the
distance between all ANDed tokens, and boost proportionally to the inverse
of their average distance apart.  I've never modified a query handler before
so I have no idea if this is possible.

Any suggestions on what approach I should take?  The less I have to modify
Solr, the better -- I'd prefer a query-side solution over writing a plugin
over forking the standard query handler.

Thanks in advance!
Michael




  

Re: A Buzzword Problem!!!

[Grant Ingersoll]

Sounds like you just need a buzzword field (indexed, stored) that is  
analyzed containing each of the terms associated with that buzzword.   
Then, just do the search against that field and return that field.


On Aug 15, 2009, at 11:03 PM, Ninad Raut wrote:


I want searchable buzzword word and basewords associated with them.
So when I query say for a word like *personality* I also want its  
associated

baseword *person*.
Also if I search for baseword *person* I want to get all the words
associated with the baseword like *personality,persona *etc.

On Fri, Aug 14, 2009 at 11:48 PM, Grant Ingersoll  
wrote:


Do you need to know, when you match which type of word it was, or  
do you

just need to know if there was a match?

On Aug 14, 2009, at 5:17 AM, Ninad Raut wrote:

Hi,
I have a scenario in which I need to store Buzz words and their  
frequency

in
a particular document.
Also along with the buzzwords I have possible basewords, portar  
words

associated with the buzzwords.
Buzzword,Baseword,Portar word all need to be searchable.
How can I use dynamic fields and my Solr schema?
Regards,
Ninad



--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using

Solr/Lucene:
http://www.lucidimagination.com/search




--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

DIH opening searchers for every doc.

[Lucas F. A. Teixeira]

Hello all,

I'm trying Data Import Handler for the first time to generate my index based
on my db.
Looking the server's logs, I can see the index process is opening a new
searcher for every doc. Is this what we should expect? why? If not, how can
I avoid it? I think if this wasn't being done, could increase indexing
speed.

Thanks,

INFO: Opening searc...@e07ac8 main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@b37a79 main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@f7e7e6 main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@37bd6a main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@ec9b6e main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@6892fd main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@967af7 main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@4e8de main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@aa8a6c main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening searc...@c27c36 main
Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 




Lucas Frare Teixeira .·.
- lucas...@gmail.com
- blog.lucastex.com
- twitter.com/lucastex

Re: Boosting relevance as terms get nearer to each other

[Mark Miller]

PhraseQuery's do score higher if the terms are found closer together.


does that imply that during the computation of the score for "a b c"~100, 
sloppyFreq() will be called?


Yes. PhraseQuery uses PhraseWeight, which creates a SloppyPhraseScorer, which 
takes into account Similiarity.sloppyFreq(matchLength).


Michael wrote:

Thanks for the suggestion.  Unfortunately, my implementation requires the
Standard query parser -- I sanitize and expand user queries into deeply
nested queries with custom boosts and other bells and whistles that make
Dismax unappealing.
I see from the docs that Similarity.sloppyFreq() is a method for returning a
higher score for small edit distances, but it's not clear when that is used.
 If I make a (Standard) query like
  a AND b AND c AND "a b c"~100
does that imply that during the computation of the score for "a b
c"~100, sloppyFreq() will be called?  That's great for my needs,
assuming the 100 slop doesn't increase query time horribly.

Michael

On Mon, Aug 17, 2009 at 10:15 AM, Mark Miller  wrote:

  

Dismax QueryParser with pf and ps params?

http://wiki.apache.org/solr/DisMaxRequestHandler

--
- Mark

http://www.lucidimagination.com




Michael wrote:



Anybody have any suggestions or hints?  I'd love to score my queries in a
way that pays attention to how close together terms appear.
Michael

On Thu, Aug 13, 2009 at 12:01 PM, Michael  wrote:



  

Hello,
I'd like to score documents higher that have the user's search terms
nearer
each other.  For example, if a user searches for

 a AND b AND c

the standard query handler should return all documents with [a] [b] and
[c]
in them, but documents matching the phrase "a b c" should get a boost
over
those with "a x b c" over those with "b x y c z a", etc.

To accomplish this, I thought I might replace the user's query with

 "a b c"~10

hoping that the slop term gets a higher and higher score the closer
together [a] [b] and [c] appear.  This doesn't seem to be the case in my
experiments; when I debug the query, there's no component of the score
based
on how close together [a] [b] and [c] are.  And I'm suspicious that this
would make my queries a whole lot slower -- in reality my users' queries
get
expanded quite a bit already, and I'd thus need to add many slop terms.

Perhaps instead I could modify the Standard query handler to examine the
distance between all ANDed tokens, and boost proportionally to the
inverse
of their average distance apart.  I've never modified a query handler
before
so I have no idea if this is possible.

Any suggestions on what approach I should take?  The less I have to
modify
Solr, the better -- I'd prefer a query-side solution over writing a
plugin
over forking the standard query handler.

Thanks in advance!
Michael






  







  



--
- Mark

http://www.lucidimagination.com

Re: spellcheck component in 1.4 distributed

[Ian Connor]

Hi,

Just a quick update to the list. Mike and I were able to apply it to 1.4 and
it works. We have it loaded on a few production servers and there is an odd
"StringIndexOutOfBoundsException" error but most of the time it seems to
work just fine.

On Fri, Aug 7, 2009 at 7:30 PM, mike anderson wrote:

> I am e-mailing to inquire about the status of the spellchecking component
> in
> 1.4 (distributed). I saw SOLR-785, but it is unreleased and for 1.5. Any
> help would be much appreciated.
> Thanks in advance,
> Mike
>



-- 
Regards,

Ian Connor
pubget.com

Re: Boosting relevance as terms get nearer to each other

[Michael]

Thanks for the suggestion.  Unfortunately, my implementation requires the
Standard query parser -- I sanitize and expand user queries into deeply
nested queries with custom boosts and other bells and whistles that make
Dismax unappealing.
I see from the docs that Similarity.sloppyFreq() is a method for returning a
higher score for small edit distances, but it's not clear when that is used.
 If I make a (Standard) query like
  a AND b AND c AND "a b c"~100
does that imply that during the computation of the score for "a b
c"~100, sloppyFreq() will be called?  That's great for my needs,
assuming the 100 slop doesn't increase query time horribly.

Michael

On Mon, Aug 17, 2009 at 10:15 AM, Mark Miller  wrote:

> Dismax QueryParser with pf and ps params?
>
> http://wiki.apache.org/solr/DisMaxRequestHandler
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> Michael wrote:
>
>> Anybody have any suggestions or hints?  I'd love to score my queries in a
>> way that pays attention to how close together terms appear.
>> Michael
>>
>> On Thu, Aug 13, 2009 at 12:01 PM, Michael  wrote:
>>
>>
>>
>>> Hello,
>>> I'd like to score documents higher that have the user's search terms
>>> nearer
>>> each other.  For example, if a user searches for
>>>
>>>  a AND b AND c
>>>
>>> the standard query handler should return all documents with [a] [b] and
>>> [c]
>>> in them, but documents matching the phrase "a b c" should get a boost
>>> over
>>> those with "a x b c" over those with "b x y c z a", etc.
>>>
>>> To accomplish this, I thought I might replace the user's query with
>>>
>>>  "a b c"~10
>>>
>>> hoping that the slop term gets a higher and higher score the closer
>>> together [a] [b] and [c] appear.  This doesn't seem to be the case in my
>>> experiments; when I debug the query, there's no component of the score
>>> based
>>> on how close together [a] [b] and [c] are.  And I'm suspicious that this
>>> would make my queries a whole lot slower -- in reality my users' queries
>>> get
>>> expanded quite a bit already, and I'd thus need to add many slop terms.
>>>
>>> Perhaps instead I could modify the Standard query handler to examine the
>>> distance between all ANDed tokens, and boost proportionally to the
>>> inverse
>>> of their average distance apart.  I've never modified a query handler
>>> before
>>> so I have no idea if this is possible.
>>>
>>> Any suggestions on what approach I should take?  The less I have to
>>> modify
>>> Solr, the better -- I'd prefer a query-side solution over writing a
>>> plugin
>>> over forking the standard query handler.
>>>
>>> Thanks in advance!
>>> Michael
>>>
>>>
>>>
>>
>>
>>
>
>
>
>
>

Re: DIH opening searchers for every doc.

[Noble Paul നോബിള്‍ नोब्ळ्]

DIH does not open searchers for each doc. Do you have any autocommit enabled?

On Mon, Aug 17, 2009 at 8:17 PM, Lucas F. A. Teixeira wrote:
> Hello all,
>
> I'm trying Data Import Handler for the first time to generate my index based
> on my db.
> Looking the server's logs, I can see the index process is opening a new
> searcher for every doc. Is this what we should expect? why? If not, how can
> I avoid it? I think if this wasn't being done, could increase indexing
> speed.
>
> Thanks,
>
> INFO: Opening searc...@e07ac8 main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@b37a79 main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@f7e7e6 main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@37bd6a main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@ec9b6e main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@6892fd main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@967af7 main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@4e8de main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@aa8a6c main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening searc...@c27c36 main
> Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
>
>
>
>
> Lucas Frare Teixeira .·.
> - lucas...@gmail.com
> - blog.lucastex.com
> - twitter.com/lucastex
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

RE: Which server parameters to tweak in Solr if I expect heavy writes and light reads?

[Fuad Efendi]

In my personal experience: ramBufferSizeMB=8192 helps to keep many things in
RAM and to delay Index Merge forever (I have single segment 10G with almost
100 mlns docs after 24 hours)

Heavy I/O was a problem before, and I solved it


-Original Message-
From: Archon810 [mailto:archon...@gmail.com] 
Sent: August-14-09 6:21 PM
To: solr-user@lucene.apache.org
Subject: Which server parameters to tweak in Solr if I expect heavy writes
and light reads?


I am facing scalability issues designing a new Solr cluster and I need to
master to be able to handle a relatively high rate of updates with almost no
reads - they can be done via slaves.

My existing Solr instance is occupying a huge amount of RAM, in fact it
started swapping at only 4.5mil docs. I am interested in making the
footprint as little as possible in RAM, even if it affects search
performance.

So, which Solr config values can I tweak in order to accomplish this?

Thank you.

P.S. Cross posted to
http://stackoverflow.com/questions/1280447/which-server-parameters-to-tweak-
in-solr-if-i-expect-heavy-writes-and-light-reads
for additional help.
-- 
View this message in context:
http://www.nabble.com/Which-server-parameters-to-tweak-in-Solr-if-I-expect-h
eavy-writes-and-light-reads--tp24979526p24979526.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Boosting relevance as terms get nearer to each other

[Michael]

Great, thank you Mark!
Michael

On Mon, Aug 17, 2009 at 10:48 AM, Mark Miller  wrote:

> PhraseQuery's do score higher if the terms are found closer together.
>
>  does that imply that during the computation of the score for "a b
>>> c"~100, sloppyFreq() will be called?
>>>
>>
> Yes. PhraseQuery uses PhraseWeight, which creates a SloppyPhraseScorer,
> which takes into account Similiarity.sloppyFreq(matchLength).
>
>
>
> Michael wrote:
>
>> Thanks for the suggestion.  Unfortunately, my implementation requires the
>> Standard query parser -- I sanitize and expand user queries into deeply
>> nested queries with custom boosts and other bells and whistles that make
>> Dismax unappealing.
>> I see from the docs that Similarity.sloppyFreq() is a method for returning
>> a
>> higher score for small edit distances, but it's not clear when that is
>> used.
>>  If I make a (Standard) query like
>>  a AND b AND c AND "a b c"~100
>> does that imply that during the computation of the score for "a b
>> c"~100, sloppyFreq() will be called?  That's great for my needs,
>> assuming the 100 slop doesn't increase query time horribly.
>>
>> Michael
>>
>> On Mon, Aug 17, 2009 at 10:15 AM, Mark Miller 
>> wrote:
>>
>>
>>
>>> Dismax QueryParser with pf and ps params?
>>>
>>> http://wiki.apache.org/solr/DisMaxRequestHandler
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>> Michael wrote:
>>>
>>>
>>>
 Anybody have any suggestions or hints?  I'd love to score my queries in
 a
 way that pays attention to how close together terms appear.
 Michael

 On Thu, Aug 13, 2009 at 12:01 PM, Michael  wrote:





> Hello,
> I'd like to score documents higher that have the user's search terms
> nearer
> each other.  For example, if a user searches for
>
>  a AND b AND c
>
> the standard query handler should return all documents with [a] [b] and
> [c]
> in them, but documents matching the phrase "a b c" should get a boost
> over
> those with "a x b c" over those with "b x y c z a", etc.
>
> To accomplish this, I thought I might replace the user's query with
>
>  "a b c"~10
>
> hoping that the slop term gets a higher and higher score the closer
> together [a] [b] and [c] appear.  This doesn't seem to be the case in
> my
> experiments; when I debug the query, there's no component of the score
> based
> on how close together [a] [b] and [c] are.  And I'm suspicious that
> this
> would make my queries a whole lot slower -- in reality my users'
> queries
> get
> expanded quite a bit already, and I'd thus need to add many slop terms.
>
> Perhaps instead I could modify the Standard query handler to examine
> the
> distance between all ANDed tokens, and boost proportionally to the
> inverse
> of their average distance apart.  I've never modified a query handler
> before
> so I have no idea if this is possible.
>
> Any suggestions on what approach I should take?  The less I have to
> modify
> Solr, the better -- I'd prefer a query-side solution over writing a
> plugin
> over forking the standard query handler.
>
> Thanks in advance!
> Michael
>
>
>
>
>



>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>

Re: spellcheck component in 1.4 distributed

[Mark Miller]

Ian Connor wrote:

Hi,

Just a quick update to the list. Mike and I were able to apply it to 1.4 and
it works. We have it loaded on a few production servers and there is an odd
"StringIndexOutOfBoundsException" error but most of the time it seems to
work just fine.
  

Do you happen to have the stack trace?

--
- Mark

http://www.lucidimagination.com

Questions about MLT

[Avlesh Singh]

I have an index of documents which contain these two fields:



Using the MLT handler with similarity field as city_id works fine and as
expected, however with categories it does not work at all. I tried looking
at "interestingTerms" in the latter case and but the handler does not return
anything. Something to do with multiValued fields?
I am using Solr 1.3.

Any help would be appreciated.

Cheers
Avlesh

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Aaron Aberg]

Ok. Did that. Still got that error. Here is the log (it's not adding
jetty stuff anymore) Here is the log. I included the exception this
time. It looks like its blowing up on something related to XPath. Do
you think its having an issue with one of my xml files?

Aug 17, 2009 2:37:35 AM org.apache.catalina.core.ApplicationContext log
INFO: Manager: start: Starting web application at '/solr'
Aug 17, 2009 2:37:36 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader
locateInstanceDir
INFO: Using JNDI solr.home: /usr/share/tomcat5/solr
Aug 17, 2009 2:37:36 AM org.apache.solr.core.CoreContainer$Initializer
initialize
INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader 
INFO: Solr home set to '/usr/share/tomcat5/solr/'
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Reusing parent classloader
Aug 17, 2009 2:37:36 AM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start SOLR. Check solr/home property
java.lang.ExceptionInInitializerError
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:169)
at 
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:104)
at 
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at 
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221)
at 
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302)
at 
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78)
at 
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635)
at 
org.apache.catalina.core.StandardContext.start(StandardContext.java:4222)
at 
org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1176)
at 
org.apache.catalina.manager.ManagerServlet.doGet(ManagerServlet.java:369)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: XPathFactory#newInstance()
failed to create an XPathFactory for the default object model:
http://java.sun.com/jaxp/xpath/dom with the
XPathFactoryConfigurationException:
javax.xml.xpath.XPathFactoryConfigurationException: No XPathFctory
implementation found for the object model:
http://java.sun.com/jaxp/xpath/dom
at javax.xml.xpath.XPathFactory.newInstance(Unknown Source)
at org.apache.solr.core.Config.(Config.java:41)
... 27 more
Aug 17, 2009 2:37:36 AM org.apache.catalina.core.StandardContext filterStart
SEVERE: Exception starting filter SolrRequestFilter
java.lang.NoClassDefFoundError: org.apache.solr.core.Config
at 
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:76)
at 
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221)
at 
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302)
at 
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78)
at 
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635)
at 
org.apache.catalina.core.StandardContext.start(StandardContext.java:4222)
at 
org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:11

Re: DIH opening searchers for every doc.

[Lucas F. A. Teixeira]

No I don't. It's commented.

This is giving me 40 docs/sec indexing, witch is very a poor rate.
(I know this rate depends in a lot of things, including that my database is
not in the same network and other stuff, but I think I can get more than
this).

Any clues on what is probably happening to open this searchers?

Thanks!

[]s,


Lucas Frare Teixeira .·.
- lucas...@gmail.com
- blog.lucastex.com
- twitter.com/lucastex


2009/8/17 Noble Paul നോബിള്‍ नोब्ळ् 

> DIH does not open searchers for each doc. Do you have any autocommit
> enabled?
>
> On Mon, Aug 17, 2009 at 8:17 PM, Lucas F. A. Teixeira
> wrote:
> > Hello all,
> >
> > I'm trying Data Import Handler for the first time to generate my index
> based
> > on my db.
> > Looking the server's logs, I can see the index process is opening a new
> > searcher for every doc. Is this what we should expect? why? If not, how
> can
> > I avoid it? I think if this wasn't being done, could increase indexing
> > speed.
> >
> > Thanks,
> >
> > INFO: Opening searc...@e07ac8 main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@b37a79 main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@f7e7e6 main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@37bd6a main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@ec9b6e main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@6892fd main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@967af7 main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@4e8de main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@aa8a6c main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> > INFO: Opening searc...@c27c36 main
> > Aug 17, 2009 11:46:03 AM org.apache.solr.search.SolrIndexSearcher 
> >
> >
> >
> >
> > Lucas Frare Teixeira .·.
> > - lucas...@gmail.com
> > - blog.lucastex.com
> > - twitter.com/lucastex
> >
>
>
>
> --
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

RE: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Fuad Efendi]

Not sure SOLR can work in such environment without asking Hosting Support
for making a lot of secific changes... such as giving specific permissions
to specific folders, setting ulimit -n, dealing with exact versions and
vendors of Java, memory parameters, and even libraries which may overwrite
SOLR-dependencies (many such companies have everything in their main
classpath already, including even ANT and Lucene)

SSH is better option...


-Original Message-
From: Aaron Aberg [mailto:aaronab...@gmail.com] 
Sent: August-14-09 9:23 PM
To: solr-user@lucene.apache.org
Subject: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

My client is using a dedicated server with plesk for the control
panel. The hosting provider says that anything done using the control
panel is supported by their tech support, so if i try anything using
SSH, it voids that warranty. Its easy to install a servlet through
plesk anyway, I upload the war file using the the java servlet
installer. A sample servlet has been installed so I know this part
works.

However, when I install solr, i get what looks like a warning icon and
if I hover over it the tool tip text says this:

"Actual status of the application does not correspond to the status
retrieved from the database."

The host providers support team says that there is something wrong
with the war file (big help). Since I kind of stuck using tomcat 5.5,
is there an older version of solr that I should be using? How can I
fix this so that I can use solr?

The only thing that I can find regarding this issue is this link:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200904.mbox/<69de1
8140904150913n66a8c78cgf064b53cd2440...@mail.gmail.com>

And the replier to the problem mentioned removing solr.xml. I thought
that if i tried that and re-wared the app that might fix it but not
such file exists in the war file I have.

Does anyone have any ideas?

--Aaron

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Aaron Aberg]

Sorry Fuad, that isn't very helpful. I also mentioned that this was a
dedicated server so none of those things are an issue. I am using SSH
right now to setup solr home etc. though.

--Aaron

On Mon, Aug 17, 2009 at 10:00 AM, Fuad Efendi wrote:
> Not sure SOLR can work in such environment without asking Hosting Support
> for making a lot of secific changes... such as giving specific permissions
> to specific folders, setting ulimit -n, dealing with exact versions and
> vendors of Java, memory parameters, and even libraries which may overwrite
> SOLR-dependencies (many such companies have everything in their main
> classpath already, including even ANT and Lucene)
>
> SSH is better option...
>
>
> -Original Message-
> From: Aaron Aberg [mailto:aaronab...@gmail.com]
> Sent: August-14-09 9:23 PM
> To: solr-user@lucene.apache.org
> Subject: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS
>
> My client is using a dedicated server with plesk for the control
> panel. The hosting provider says that anything done using the control
> panel is supported by their tech support, so if i try anything using
> SSH, it voids that warranty. Its easy to install a servlet through
> plesk anyway, I upload the war file using the the java servlet
> installer. A sample servlet has been installed so I know this part
> works.
>
> However, when I install solr, i get what looks like a warning icon and
> if I hover over it the tool tip text says this:
>
> "Actual status of the application does not correspond to the status
> retrieved from the database."
>
> The host providers support team says that there is something wrong
> with the war file (big help). Since I kind of stuck using tomcat 5.5,
> is there an older version of solr that I should be using? How can I
> fix this so that I can use solr?
>
> The only thing that I can find regarding this issue is this link:
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200904.mbox/<69de1
> 8140904150913n66a8c78cgf064b53cd2440...@mail.gmail.com>
>
> And the replier to the problem mentioned removing solr.xml. I thought
> that if i tried that and re-wared the app that might fix it but not
> such file exists in the war file I have.
>
> Does anyone have any ideas?
>
> --Aaron
>
>
>

RE: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Fuad Efendi]

What is solr.xml for?

INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml Aug 17, 2009
2:37:36 AM org.apache.solr.core.SolrResourceLoader 


java.lang.NoClassDefFoundError: org.apache.solr.core.Config

- can't find configuration... XPath needs to load XML to configure Config.


solr.xml???


-Original Message-
From: Aaron Aberg [mailto:aaronab...@gmail.com] 
Sent: August-17-09 12:48 PM
To: solr-user@lucene.apache.org
Subject: Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on
CentOS

Ok. Did that. Still got that error. Here is the log (it's not adding
jetty stuff anymore) Here is the log. I included the exception this
time. It looks like its blowing up on something related to XPath. Do
you think its having an issue with one of my xml files?

Aug 17, 2009 2:37:35 AM org.apache.catalina.core.ApplicationContext log
INFO: Manager: start: Starting web application at '/solr'
Aug 17, 2009 2:37:36 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader
locateInstanceDir
INFO: Using JNDI solr.home: /usr/share/tomcat5/solr
Aug 17, 2009 2:37:36 AM org.apache.solr.core.CoreContainer$Initializer
initialize
INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader 
INFO: Solr home set to '/usr/share/tomcat5/solr/'
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Reusing parent classloader
Aug 17, 2009 2:37:36 AM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start SOLR. Check solr/home property
java.lang.ExceptionInInitializerError
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:169)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java
:104)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilter
Config.java:221)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFil
terConfig.java:302)
at
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterCon
fig.java:78)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:36
35)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4222)
at
org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1176)
at
org.apache.catalina.manager.ManagerServlet.doGet(ManagerServlet.java:369)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:172)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase
.java:525)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processC
onnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.jav
a:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWo
rkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:685)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: XPathFactory#newInstance()
failed to create an XPathFactory for the default object model:
http://java.sun.com/jaxp/xpath/dom with the
XPathFactoryConfigurationException:
javax.xml.xpath.XPathFactoryConfigurationException: No XPathFctory
implementation found for the object model:
http://java.sun.com/jaxp/xpath/dom
at javax.xml.xpath.XPathFactory.newInstance(Unknown Source)
at org.apache.solr.core.Config.(Config.java:41)
... 27 more
Aug 17, 2009 2:37:36 AM org.apache.catalina.core.StandardContext filterStart
SEVERE: Exception starting filter SolrRequestFilter
java.lang.NoClassDefFoundError: org.apache.solr.core.Config
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterConfi

Re: Performance Tuning: segment_merge:index_update=5:1 (timing)

[Jason Rutherglen]

Fuad,

I'd recommend indexing in Hadoop, then copying the new indexes to Solr
slaves.  This removes the need for Solr master servers.  Of course
you'd need a Hadoop cluster larger than the number of master servers
you have now.  The merge indexes command (which can be taxing on the
servers because it performs a copy) could be used.

It would be good to improve Solr's integration with Hadoop as
otherwise reindexing (such as for a schema change) becomes an onerous
task

-J

On Tue, Aug 11, 2009 at 2:31 PM, Fuad Efendi wrote:
> Forgot to add: committing only once a day
>
> I tried mergeFactor=1000 and performance of index write was extremely good
> (more than 50,000,000 updates during part of a day)
> However, "commit" was taking 2 days or more and I simply killed process
> (suspecting that it can break my harddrive); I had about 8000 files in index
> that day... 3 minutes waiting until new small *.del file appear, and after
> several thousands of such files I killed process.
>
> Most probably "delete" in Lucene... it needs rewrite inverted index (in
> fact, to optimize)...? not sure
>
>
>
> -Original Message-
>
> Never tried profiling;
> 3000-5000 docs per second if SOLR is not busy with segment merge;
>
> During segment merge 99% CPU, no disk swap; I can't suspect I/O...
>
> During document updates (small batches 100-1000 docs) only 5-15% CPU
>
> -server 2048Gb option of JVM (which is JRockit) + 256M for RAM Buffer
>
> I can't suspect garbage collection... I'll try to do the same with much
> better hardware tomorrow (2 quad-core instead of single double-core, SCSI
> RAID0 instead of single SAS, 16Gb for Tomcat instead of current 2Gb) but
> constant rate 5:1 is very suspicious...
>
>
>
> -Original Message-
> From: Grant Ingersoll
> Sent: August-11-09 5:01 PM
>
> Have you tried profiling?  How often are you committing?  Have you
> looked at Garbage Collection or any of the usual suspects like that?
>
>
> On Aug 11, 2009, at 4:49 PM, Fuad Efendi wrote:
>
>> In a heavily loaded Write-only Master SOLR, I have 5 minutes of RAM
>> Buffer
>> Flash / Segment Merge per 1 minute of (heavy) batch document updates.
>
> Define heavy.  How many docs per second?
>
>
>
>
>
>
>

RE: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Fuad Efendi]

Aaron,

Do you have solr.war in your %TOMCAT%/webapps folder? Is your solr/home in
another than /webapps location? Try to install sample Tomcat with SOLR on
local dev-box and check it's working...


-Original Message-
From: Fuad Efendi [mailto:f...@efendi.ca] 
Sent: August-17-09 1:33 PM
To: solr-user@lucene.apache.org
Subject: RE: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on
CentOS

What is solr.xml for?

INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml Aug 17, 2009
2:37:36 AM org.apache.solr.core.SolrResourceLoader 


java.lang.NoClassDefFoundError: org.apache.solr.core.Config

- can't find configuration... XPath needs to load XML to configure Config.


solr.xml???

RE: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Fuad Efendi]

Looks like you are using SOLR multicore, with solr.xml... I never tried
it...
The rest looks fine, except suspicious solr.xml


-Original Message-
From: Fuad Efendi [mailto:f...@efendi.ca] 
Sent: August-17-09 1:33 PM
To: solr-user@lucene.apache.org
Subject: RE: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on
CentOS

What is solr.xml for?

INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml Aug 17, 2009
2:37:36 AM org.apache.solr.core.SolrResourceLoader 


java.lang.NoClassDefFoundError: org.apache.solr.core.Config

- can't find configuration... XPath needs to load XML to configure Config.


solr.xml???


-Original Message-
From: Aaron Aberg [mailto:aaronab...@gmail.com] 
Sent: August-17-09 12:48 PM
To: solr-user@lucene.apache.org
Subject: Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on
CentOS

Ok. Did that. Still got that error. Here is the log (it's not adding
jetty stuff anymore) Here is the log. I included the exception this
time. It looks like its blowing up on something related to XPath. Do
you think its having an issue with one of my xml files?

Aug 17, 2009 2:37:35 AM org.apache.catalina.core.ApplicationContext log
INFO: Manager: start: Starting web application at '/solr'
Aug 17, 2009 2:37:36 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader
locateInstanceDir
INFO: Using JNDI solr.home: /usr/share/tomcat5/solr
Aug 17, 2009 2:37:36 AM org.apache.solr.core.CoreContainer$Initializer
initialize
INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader 
INFO: Solr home set to '/usr/share/tomcat5/solr/'
Aug 17, 2009 2:37:36 AM org.apache.solr.core.SolrResourceLoader
createClassLoader
INFO: Reusing parent classloader
Aug 17, 2009 2:37:36 AM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start SOLR. Check solr/home property
java.lang.ExceptionInInitializerError
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:169)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java
:104)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilter
Config.java:221)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFil
terConfig.java:302)
at
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterCon
fig.java:78)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:36
35)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4222)
at
org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1176)
at
org.apache.catalina.manager.ManagerServlet.doGet(ManagerServlet.java:369)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:172)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase
.java:525)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processC
onnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.jav
a:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWo
rkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:685)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: XPathFactory#newInstance()
failed to create an XPathFactory for the default object model:
http://java.sun.com/jaxp/xpath/dom with the
XPathFactoryConfigurationException:
javax.xml.xpath.XPathFactoryConfigurationException: No XPathFctory
implementation found for the object model:
http://java.sun.com/jaxp/xpath/dom
at javax.xml.xpath.XPathFactory.newInstance(Unknown Source)
at org.apache.solr.core.Config.(Config.java:41)
... 27 mor

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Aaron Aberg]

On Mon, Aug 17, 2009 at 10:58 AM, Fuad Efendi wrote:
> Looks like you are using SOLR multicore, with solr.xml... I never tried
> it...
> The rest looks fine, except suspicious solr.xml

whats suspicious about it? is it in the wrong place? Is it not suppose
to be there?

technically my war file is not in my webapps directory. I'm using
plesk and it installed my war file here:
tomcat5/psa-webapps/mywebk9.com/solr.war

I have installed a sample war file and its in the same location. It works fine.

Re: delta-import using a full-import command is not working

[djain101]

Any help?
-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25011540.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: delta-import using a full-import command is not working

[Avlesh Singh]

Solr and your database are different machines? If yes, are their dates
synchronized?
If you have access to your database server logs, looking at the queries that
DIH generated might help.

Cheers
Avlesh

On Mon, Aug 17, 2009 at 11:40 PM, djain101  wrote:

>
> Any help?
> --
> View this message in context:
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25011540.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: delta-import using a full-import command is not working

[djain101]

Yes, database and Solr are different machines and their dates are not
synchronized. Could that be the issue? Why the date difference between Solr
and DB machine fails to put the timestamp from dataimport.properties file? 

Thanks,
Dharmveer



Avlesh Singh wrote:
> 
> Solr and your database are different machines? If yes, are their dates
> synchronized?
> If you have access to your database server logs, looking at the queries
> that
> DIH generated might help.
> 
> Cheers
> Avlesh
> 
> On Mon, Aug 17, 2009 at 11:40 PM, djain101 
> wrote:
> 
>>
>> Any help?
>> --
>> View this message in context:
>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25011540.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25012532.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Performance Tuning: segment_merge:index_update=5:1 (timing)

[Fuad Efendi]

Hi Jason,

After moving to more RAM and CPUs and setting ramBufferSizeMB=8192 problem
disappeared; I had 100 mlns documents added in 24 hours almost without any
index merge (mergeFactor=10). Lucene flushes to disk the segment when RAM
buffer is full; then MergePolicy orchestrates...

However, 500Gb Seagate SATA got quickly broken on SuSE Linux 10 & Tyan
Thunder motherboard :((( - when SOLR tried to merge 2 segments, about
10Gb... I reinstalled SLES and started again; I ordered SAS RAID Adaptec &
Seagate Cheetah 15K.5 SAS

I am wondering how one can run Nutch on SATA (if Nutch is fast enough)... I
had constant problems with Oracle block corruption on Seagate Barracuda SATA
several years ago, then moved to Cheetah...

Good SCSI controller (with dedicated CPU and cache!!!) + Cheetah 15K.5 (with
16Mb cache!!!) - and we don't need to flush 8Kb if we changed few hundred
bytes only... it's not easy to assemble good "commodity" hardware from
parts...

I am going to use Hadoop for pre-data-mining before indexing with SOLR; I
use currently mix of MySQL & HBase...

Thanks for the input!



-Original Message-
From: Jason Rutherglen [mailto:jason.rutherg...@gmail.com] 
Sent: August-17-09 1:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Performance Tuning: segment_merge:index_update=5:1 (timing)

Fuad,

I'd recommend indexing in Hadoop, then copying the new indexes to Solr
slaves.  This removes the need for Solr master servers.  Of course
you'd need a Hadoop cluster larger than the number of master servers
you have now.  The merge indexes command (which can be taxing on the
servers because it performs a copy) could be used.

It would be good to improve Solr's integration with Hadoop as
otherwise reindexing (such as for a schema change) becomes an onerous
task

-J

On Tue, Aug 11, 2009 at 2:31 PM, Fuad Efendi wrote:
> Forgot to add: committing only once a day
>
> I tried mergeFactor=1000 and performance of index write was extremely good
> (more than 50,000,000 updates during part of a day)
> However, "commit" was taking 2 days or more and I simply killed process
> (suspecting that it can break my harddrive); I had about 8000 files in
index
> that day... 3 minutes waiting until new small *.del file appear, and after
> several thousands of such files I killed process.
>
> Most probably "delete" in Lucene... it needs rewrite inverted index (in
> fact, to optimize)...? not sure
>
>
>
> -Original Message-
>
> Never tried profiling;
> 3000-5000 docs per second if SOLR is not busy with segment merge;
>
> During segment merge 99% CPU, no disk swap; I can't suspect I/O...
>
> During document updates (small batches 100-1000 docs) only 5-15% CPU
>
> -server 2048Gb option of JVM (which is JRockit) + 256M for RAM Buffer
>
> I can't suspect garbage collection... I'll try to do the same with much
> better hardware tomorrow (2 quad-core instead of single double-core, SCSI
> RAID0 instead of single SAS, 16Gb for Tomcat instead of current 2Gb) but
> constant rate 5:1 is very suspicious...
>
>
>
> -Original Message-
> From: Grant Ingersoll
> Sent: August-11-09 5:01 PM
>
> Have you tried profiling?  How often are you committing?  Have you
> looked at Garbage Collection or any of the usual suspects like that?
>
>
> On Aug 11, 2009, at 4:49 PM, Fuad Efendi wrote:
>
>> In a heavily loaded Write-only Master SOLR, I have 5 minutes of RAM
>> Buffer
>> Flash / Segment Merge per 1 minute of (heavy) batch document updates.
>
> Define heavy.  How many docs per second?
>
>
>
>
>
>
>

Query not working as expected

[Matt Schraeder]

I'm attempting to write a query as follows:
 
($query^10) OR (NOT ($query)) which effectively would return everything, but if 
it matches the first query it will get a higher score and thus be sorted first 
in the result set.  Unfortunately the results are not coming back as expected. 
 
($query) works by itself and gets X rows
(NOT ($query)) works by itself and gets Y rows
 
You would expect ($query) OR (NOT ($query)) to return X+Y rows but it is only 
returning X rows.
 
What am I doing wrong?

SolrJ question

[Paul Tomblin]

If I put an object into a SolrInputDocument and store it, how do I
query for it back?  For instance, I stored a java.net.URI in a field
called "url", and I want to query for all the documents that match a
particular URI.  The query syntax only seems to allow Strings, and if
I just try query.setQuery("url:" + uri.toString()) I get an error
because of the colon after "http" in the URI.

I'm really new to Solr, so please let me know if I'm missing something
basic here.

-- 
http://www.linkedin.com/in/paultomblin

RE: SolrJ question

[Harsch, Timothy J. (ARC-SC)[PEROT SYSTEMS]]

Assuming you have written the SolrInputDocument to the server, you would next 
query.  See ClientUtils.escapeQueryChars.  Also you need to be cognizant of 
URLEncoding at times.

-Original Message-
From: ptomb...@gmail.com [mailto:ptomb...@gmail.com] On Behalf Of Paul Tomblin
Sent: Monday, August 17, 2009 2:12 PM
To: solr-user@lucene.apache.org
Subject: SolrJ question

If I put an object into a SolrInputDocument and store it, how do I
query for it back?  For instance, I stored a java.net.URI in a field
called "url", and I want to query for all the documents that match a
particular URI.  The query syntax only seems to allow Strings, and if
I just try query.setQuery("url:" + uri.toString()) I get an error
because of the colon after "http" in the URI.

I'm really new to Solr, so please let me know if I'm missing something
basic here.

-- 
http://www.linkedin.com/in/paultomblin

RE: SolrJ question

[Ensdorf Ken]

You can escape the string with

org.apache.lucene.queryParser.QueryParser.escape(String query)

http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/queryParser/QueryParser.html#escape%28java.lang.String%29



> -Original Message-
> From: ptomb...@gmail.com [mailto:ptomb...@gmail.com] On Behalf Of Paul
> Tomblin
> Sent: Monday, August 17, 2009 5:12 PM
> To: solr-user@lucene.apache.org
> Subject: SolrJ question
>
> If I put an object into a SolrInputDocument and store it, how do I
> query for it back?  For instance, I stored a java.net.URI in a field
> called "url", and I want to query for all the documents that match a
> particular URI.  The query syntax only seems to allow Strings, and if
> I just try query.setQuery("url:" + uri.toString()) I get an error
> because of the colon after "http" in the URI.
>
> I'm really new to Solr, so please let me know if I'm missing something
> basic here.
>
> --
> http://www.linkedin.com/in/paultomblin

RE: Query not working as expected

[Harsch, Timothy J. (ARC-SC)[PEROT SYSTEMS]]

The rows parameter would prevent you from getting all docs back.  It is set by 
default to 10 I believe.

-Original Message-
From: Matt Schraeder [mailto:mschrae...@btsb.com] 
Sent: Monday, August 17, 2009 2:04 PM
To: solr-user@lucene.apache.org
Subject: Query not working as expected

I'm attempting to write a query as follows:
 
($query^10) OR (NOT ($query)) which effectively would return everything, but if 
it matches the first query it will get a higher score and thus be sorted first 
in the result set.  Unfortunately the results are not coming back as expected. 
 
($query) works by itself and gets X rows
(NOT ($query)) works by itself and gets Y rows
 
You would expect ($query) OR (NOT ($query)) to return X+Y rows but it is only 
returning X rows.
 
What am I doing wrong?

Re: SolrJ question

[Paul Tomblin]

On Mon, Aug 17, 2009 at 5:28 PM, Harsch, Timothy J. (ARC-SC)[PEROT
SYSTEMS] wrote:
> Assuming you have written the SolrInputDocument to the server, you would next 
> query.

I'm sorry, I don't understand what you mean by "you would next query."
 There appear to be some words missing from that sentence.



-- 
http://www.linkedin.com/in/paultomblin

Re: SolrJ question

[Paul Tomblin]

On Mon, Aug 17, 2009 at 5:30 PM, Ensdorf Ken wrote:
> You can escape the string with
>
> org.apache.lucene.queryParser.QueryParser.escape(String query)
>
> http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/queryParser/QueryParser.html#escape%28java.lang.String%29
>

Does this mean I should have converted my objects to string before
writing them to the server?

-- 
http://www.linkedin.com/in/paultomblin

RE: SolrJ question

[Ensdorf Ken]

> Does this mean I should have converted my objects to string before
> writing them to the server?
>

I believe SolrJ takes care of that for you by calling toString(), but you would 
need to convert explicitly when you query (and then escape).

RE: Query not working as expected

[Matt Schraeder]

That isn't the problem, as I am looking at "numFound" and not actual rows 
returned.  In all searches the rows returned is less than the number found.

>>> timothy.j.har...@nasa.gov 8/17/2009 4:30:38 PM >>>
The rows parameter would prevent you from getting all docs back.  It is set by 
default to 10 I believe.

-Original Message-
From: Matt Schraeder [mailto:mschrae...@btsb.com] 
Sent: Monday, August 17, 2009 2:04 PM
To: solr-user@lucene.apache.org
Subject: Query not working as expected

I'm attempting to write a query as follows:

($query^10) OR (NOT ($query)) which effectively would return everything, but if 
it matches the first query it will get a higher score and thus be sorted first 
in the result set.  Unfortunately the results are not coming back as expected. 

($query) works by itself and gets X rows
(NOT ($query)) works by itself and gets Y rows

You would expect ($query) OR (NOT ($query)) to return X+Y rows but it is only 
returning X rows.

What am I doing wrong?

Re: SolrJ question

[Paul Tomblin]

On Mon, Aug 17, 2009 at 5:36 PM, Ensdorf Ken wrote:
>> Does this mean I should have converted my objects to string before
>> writing them to the server?
>>
>
> I believe SolrJ takes care of that for you by calling toString(), but you 
> would need to convert explicitly when you query (and then escape).
>

Hmmm.  It's not working right.  I've added a 5 documents, 3 with the
URL set to "http://xcski.com/pharma/"; and 2 with the URL set to
"http://xcski.com/nano/";.  Doing other sorts of queries seems to be
pulling back the right data:

 [DEBUG] 34:20 (Solr.java:getForConcept:116)
 [java] search term = fribbet, concept = pharma
 [java]
 [java] Aug 17, 2009 5:34:20 PM org.apache.solr.core.SolrCore execute
 [java] INFO: [] webapp=null path=/select
params={q=fribbet&fq=concept%3Apharma} hits=1 status=0 QTime=9
 [java] [DEBUG] 34:20 (Solr.java:getForConcept:130)
 [java] got doc SolrDocument[{id=2:http://xcski.com/pharma/,
concept=pharma, text=this is a third big long chunk of text containing
the word fribbet, title=this is the third title, keywords=pills,drugs,
origDoctype=html, chunkNum=2, url=http://xcski.com/pharma/}]

 But if I want to restrict it to a specific URL, I use

   SolrQuery query = new SolrQuery();
query.setQuery("url:" + ClientUtils.escapeQueryChars(url));

and it's not returning anything.  Log4j output looks like:

 [java] [DEBUG] 34:20 (Solr.java:getAllForURL:89)
 [java] getting for URL: http://xcski.com/nano/
 [java]
 [java] Aug 17, 2009 5:34:20 PM org.apache.solr.core.SolrCore execute
 [java] INFO: [] webapp=null path=/select
params={q=url%3Ahttp%5C%3A%5C%2F%5C%2Fxcski%5C.com%5C%2Fnano%5C%2F}
hits=0 status=0 QTime=16
 [java] [DEBUG] 34:20 (Solr.java:main:229)
 [java] found: 0

Actually, looking at that, it looks like it's escaped the URL twice,
converting ":" into "%3A", then converting that to "%5C%3A".  Could
that be?



-- 
http://www.linkedin.com/in/paultomblin

Re: Query not working as expected

[Mark Miller]

Matt Schraeder wrote:

I'm attempting to write a query as follows:
 
($query^10) OR (NOT ($query)) which effectively would return everything, but if it matches the first query it will get a higher score and thus be sorted first in the result set.  Unfortunately the results are not coming back as expected. 
 
($query) works by itself and gets X rows

(NOT ($query)) works by itself and gets Y rows
 
You would expect ($query) OR (NOT ($query)) to return X+Y rows but it is only returning X rows.
 
What am I doing wrong?



  
I believe that Solr will only allow a put 'not' in the top level of a 
Boolean query - also not sure if it supports NOT or just !.


In any case, to use it deeper than the top level you must use the 
MatchAllDocsQuery syntax (obviously doesn't apply to DisMax):


($query^10) OR (*:* NOT ($query))



--
- Mark

http://www.lucidimagination.com

Maximum number of values in a multi-valued field.

[Arv]

All,
We are considering some new changes to our Solr schema to better support
some new functionality for our application. To that extent, we want to add
an additional field that is multi-valued, but will contain a large number of
values per document. Potentially up to 2000 values on this field per
document.

Questions:
- Is this wise?
- Though we will not be faceting on this field, are there any implications
for performance?
- I understand that the XML in/out will be large, and we may need to stop
this field being sent back on every query, as this field is essentially used
as a filter only.

The reason I am asking is that our instance of Solr currently works
wonderfully and is very fast, and I am wary of doing anything that will
affect this.  So, any pointer on design here will help.

-- Aravind

-- 
View this message in context: 
http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015685.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Maximum number of values in a multi-valued field.

[Jason Rutherglen]

Your term dictionary will grow somewhat, which means the term
index could consume more memory. Because the term dictionary has
grown there could be less performance in looking up terms but
that is unlikely to affect your application. How many unique
terms will there be?

On Mon, Aug 17, 2009 at 3:50 PM, Arv wrote:
>
> All,
> We are considering some new changes to our Solr schema to better support
> some new functionality for our application. To that extent, we want to add
> an additional field that is multi-valued, but will contain a large number of
> values per document. Potentially up to 2000 values on this field per
> document.
>
> Questions:
> - Is this wise?
> - Though we will not be faceting on this field, are there any implications
> for performance?
> - I understand that the XML in/out will be large, and we may need to stop
> this field being sent back on every query, as this field is essentially used
> as a filter only.
>
> The reason I am asking is that our instance of Solr currently works
> wonderfully and is very fast, and I am wary of doing anything that will
> affect this.  So, any pointer on design here will help.
>
> -- Aravind
>
> --
> View this message in context: 
> http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015685.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Maximum number of values in a multi-valued field.

[Aravind Naidu]

Hi,
The possibility is that all items in this field could be unique. Let me
clarify.
The main Solr index is a for a list of products. Some products belong to
catalogues.  So, the consideration is to add a multi-valued field to put the
id of the catalogue in each product as a multi-valued field to be used as a
filter.

-- Aravind


Jason Rutherglen-2 wrote:
> 
> Your term dictionary will grow somewhat, which means the term
> index could consume more memory. Because the term dictionary has
> grown there could be less performance in looking up terms but
> that is unlikely to affect your application. How many unique
> terms will there be?
> 
> On Mon, Aug 17, 2009 at 3:50 PM, Arv wrote:
>>
>> All,
>> We are considering some new changes to our Solr schema to better support
>> some new functionality for our application. To that extent, we want to
>> add
>> an additional field that is multi-valued, but will contain a large number
>> of
>> values per document. Potentially up to 2000 values on this field per
>> document.
>>
>> Questions:
>> - Is this wise?
>> - Though we will not be faceting on this field, are there any
>> implications
>> for performance?
>> - I understand that the XML in/out will be large, and we may need to stop
>> this field being sent back on every query, as this field is essentially
>> used
>> as a filter only.
>>
>> The reason I am asking is that our instance of Solr currently works
>> wonderfully and is very fast, and I am wary of doing anything that will
>> affect this.  So, any pointer on design here will help.
>>
>> -- Aravind
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015685.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015945.html
Sent from the Solr - User mailing list archive at Nabble.com.

DataImportHandler - very slow delta import

[Matthew Painter]

Hi,
 
We are using Solr's DataImportHandler to populate the Solr index from a
SQL Server database of nearly 4,000,000 rows. Whereas the population
itself is very fast (around 1000 rows per second), the delta import is
only processing around one row a second. 
 
Is this a known performance issue? We are using Solr 1.3.
 
For reference, the abridged entity configuration (cuts indicated by
'...') is below:
 
  

...

 
Thanks,
Matt

This e-mail message and any attachments are CONFIDENTIAL to the addressee(s) 
and may also be LEGALLY PRIVILEGED.  If you are not the intended addressee, 
please do not use, disclose, copy or distribute the message or the information 
it contains.  Instead, please notify me as soon as possible and delete the 
e-mail, including any attachments.  Thank you.

Re: delta-import using a full-import command is not working

[djain101]

After debugging dataimporter code, i found that it is a bug in the
dataimporter  code itself. doFullImport() in DataImporter class is not
loading last index time where as doDeltaImport() is. The code snippet from
doFullImport() is:

if (requestParams.commit)
  setIndexStartTime(new Date());


where as in doDeltaImport() is:

if (requestParams.commit) {
Date lastModified = writer.loadIndexStartTime();
setIndexStartTime(new Date());
setLastIndexTime(lastModified);
  }

Looks like it is fixed in latest version of DataImporter.java. I will test
with that.

-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25016765.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrJ question

[Paul Tomblin]

On Mon, Aug 17, 2009 at 5:47 PM, Paul Tomblin wrote:

> Hmmm.  It's not working right.  I've added a 5 documents, 3 with the
> URL set to "http://xcski.com/pharma/"; and 2 with the URL set to
> "http://xcski.com/nano/";.  Doing other sorts of queries seems to be
> pulling back the right data:


Of course, It doesn't help that my url field was set to
indexed="false" in the schema.  Changing it to true fixed it.

-- 
http://www.linkedin.com/in/paultomblin

Re: delta-import using a full-import command is not working

[djain101]

Looks like this issue has been fixed on Sept 20, 2008 against issue SOLR-768.
Can someone please let me know which one is a stable jar after Sept 20,
2008. 



djain101 wrote:
> 
> After debugging dataimporter code, i found that it is a bug in the
> dataimporter 1.3  code itself. doFullImport() in DataImporter class is not
> loading last index time where as doDeltaImport() is. The code snippet from
> doFullImport() is:
> 
> if (requestParams.commit)
>   setIndexStartTime(new Date());
> 
> 
> where as in doDeltaImport() is:
> 
> if (requestParams.commit) {
> Date lastModified = writer.loadIndexStartTime();
> setIndexStartTime(new Date());
> setLastIndexTime(lastModified);
>   }
> 
> Looks like it is fixed in latest version of DataImporter.java. I will test
> with that.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
Sent from the Solr - User mailing list archive at Nabble.com.

SOLR - extremely strange behavior! Documents disappeared...

[Funtick]

After running an application which heavily uses MD5 HEX-representation as
 for SOLR v.1.4-dev-trunk:

1. After 30 hours: 
101,000,000 documents added

2. Commit: 
numDocs = 783,714 
maxDoc = 3,975,393

3. Upload new docs to SOLR during 1 hour(!!!), then commit, then
optimize:
numDocs=1,281,851
maxDocs=1,281,851

It looks _extremely_ strange that within an hour I have such a huge increase
with same 'average' document set...

I am suspecting something goes wrong with Lucene buffer flush / index merge
OR SOLR - Unique ID handling...

According to my own estimates, I should have about 10,000,000 new documents
now... I had 0.5 millions within an hour, and 0.8 mlns within a day; same
'random' documents.

This morning index size was about 4Gb, then suddenly dropped below 0.5 Gb.
Why? I haven't issued any "commit"...

I am using ramBufferMB=8192






-- 
View this message in context: 
http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25017728.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: JVM Heap utilization & Memory leaks with Solr

[Funtick]

Can you tell me please how many non-tokenized single-valued fields your
schema uses, and how many documents?
Thanks,
Fuad


Rahul R wrote:
> 
> My primary issue is not Out of Memory error at run time. It is memory
> leaks:
> heap space not being released after doing a force GC also. So after
> sometime
> as progressively more heap gets utilized, I start running out of
> memory
> The verdict however seems unanimous that there are no known memory leak
> issues within Solr. I am still looking at my application to analyse the
> problem. Thank you.
> 
> On Thu, Aug 13, 2009 at 10:58 PM, Fuad Efendi  wrote:
> 
>> Most OutOfMemoryException (if not 100%) happening with SOLR are because
>> of
>>
>> http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/FieldCache.
>> html
>> - it is used internally in Lucene to cache Field value and document ID.
>>
>> My very long-term observations: SOLR can run without any problems few
>> days/months and unpredictable OOM happens just because someone tried
>> sorted
>> search which will populate array with IDs of ALL documents in the index.
>>
>> The only solution: calculate exactly amount of RAM needed for
>> FieldCache...
>> For instance, for 100,000,000 documents single instance of FieldCache may
>> require 8*100,000,000 bytes (8 bytes per document ID?) which is almost
>> 1Gb
>> (at least!)
>>
>>
>> I didn't notice any memory leaks after I started to use 16Gb RAM for SOLR
>> instance (almost a year without any restart!)
>>
>>
>>
>>
>> -Original Message-
>> From: Rahul R [mailto:rahul.s...@gmail.com]
>> Sent: August-13-09 1:25 AM
>> To: solr-user@lucene.apache.org
>>  Subject: Re: JVM Heap utilization & Memory leaks with Solr
>>
>> *You should try to generate heap dumps and analyze the heap using a tool
>> like the Eclipse Memory Analyzer. Maybe it helps spotting a group of
>> objects holding a large amount of memory*
>>
>> The tool that I used also allows to capture heap snap shots. Eclipse had
>> a
>> lot of pre-requisites. You need to apply some three or five patches
>> before
>> you can start using it My observations with this tool were that
>> some
>> Hashmaps were taking up a lot of space. Although I could not pin it down
>> to
>> the exact HashMap. These would either be weblogic's or Solr's I will
>> anyway give eclipse's a try and see how it goes. Thanks for your input.
>>
>> Rahul
>>
>> On Wed, Aug 12, 2009 at 2:15 PM, Gunnar Wagenknecht
>> wrote:
>>
>> > Rahul R schrieb:
>> > > I tried using a profiling tool - Yourkit. The trial version was free
>> for
>> > 15
>> > > days. But I couldn't find anything of significance.
>> >
>> > You should try to generate heap dumps and analyze the heap using a tool
>> > like the Eclipse Memory Analyzer. Maybe it helps spotting a group of
>> > objects holding a large amount of memory.
>> >
>> > -Gunnar
>> >
>> > --
>> > Gunnar Wagenknecht
>> > gun...@wagenknecht.org
>> > http://wagenknecht.org/
>> >
>> >
>>
>>
>>
> 
> 

-- 
View this message in context: 
http://www.nabble.com/JVM-Heap-utilization---Memory-leaks-with-Solr-tp24802380p25017767.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR - extremely strange behavior! Documents disappeared...

[Mark Miller]

I'd say you have a lot of documents that have the same id.
When you add a doc with the same id, first the old one is deleted, then the
new one is added (atomically though).

The deleted docs are not removed from the index immediately though - the doc
id is just marked as deleted.

Over time though, as segments are merged due to hitting triggers while
adding new documents, deletes are removed (which deletes depends on which
segments have been merged).

So if you add a tone of documents over time, many with the same ids, you
would likely see this type of maxDoc, numDoc churn. maxDoc will include
deleted docs while numDoc will not.


-- 
- Mark

http://www.lucidimagination.com

On Mon, Aug 17, 2009 at 11:09 PM, Funtick  wrote:

>
> After running an application which heavily uses MD5 HEX-representation as
>  for SOLR v.1.4-dev-trunk:
>
> 1. After 30 hours:
> 101,000,000 documents added
>
> 2. Commit:
> numDocs = 783,714
> maxDoc = 3,975,393
>
> 3. Upload new docs to SOLR during 1 hour(!!!), then commit, then
> optimize:
> numDocs=1,281,851
> maxDocs=1,281,851
>
> It looks _extremely_ strange that within an hour I have such a huge
> increase
> with same 'average' document set...
>
> I am suspecting something goes wrong with Lucene buffer flush / index merge
> OR SOLR - Unique ID handling...
>
> According to my own estimates, I should have about 10,000,000 new documents
> now... I had 0.5 millions within an hour, and 0.8 mlns within a day; same
> 'random' documents.
>
> This morning index size was about 4Gb, then suddenly dropped below 0.5 Gb.
> Why? I haven't issued any "commit"...
>
> I am using ramBufferMB=8192
>
>
>
>
>
>
> --
> View this message in context:
> http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25017728.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: SOLR - extremely strange behavior! Documents disappeared...

[Funtick]

But how to explain that within an hour (after commit) I have had about
500,000 new documents, and within 30 hours (after commit) only 1,300,000?

Same _random_enough_ documents... 

BTW, SOLR Console was showing only few hundreds "deletesById" although I
don't use any deleteById explicitly; only "update" with "allowOverwrite" and
"uniqueId".




markrmiller wrote:
> 
> I'd say you have a lot of documents that have the same id.
> When you add a doc with the same id, first the old one is deleted, then
> the
> new one is added (atomically though).
> 
> The deleted docs are not removed from the index immediately though - the
> doc
> id is just marked as deleted.
> 
> Over time though, as segments are merged due to hitting triggers while
> adding new documents, deletes are removed (which deletes depends on which
> segments have been merged).
> 
> So if you add a tone of documents over time, many with the same ids, you
> would likely see this type of maxDoc, numDoc churn. maxDoc will include
> deleted docs while numDoc will not.
> 
> 
> -- 
> - Mark
> 
> http://www.lucidimagination.com
> 
> On Mon, Aug 17, 2009 at 11:09 PM, Funtick  wrote:
> 
>>
>> After running an application which heavily uses MD5 HEX-representation as
>>  for SOLR v.1.4-dev-trunk:
>>
>> 1. After 30 hours:
>> 101,000,000 documents added
>>
>> 2. Commit:
>> numDocs = 783,714
>> maxDoc = 3,975,393
>>
>> 3. Upload new docs to SOLR during 1 hour(!!!), then commit, then
>> optimize:
>> numDocs=1,281,851
>> maxDocs=1,281,851
>>
>> It looks _extremely_ strange that within an hour I have such a huge
>> increase
>> with same 'average' document set...
>>
>> I am suspecting something goes wrong with Lucene buffer flush / index
>> merge
>> OR SOLR - Unique ID handling...
>>
>> According to my own estimates, I should have about 10,000,000 new
>> documents
>> now... I had 0.5 millions within an hour, and 0.8 mlns within a day; same
>> 'random' documents.
>>
>> This morning index size was about 4Gb, then suddenly dropped below 0.5
>> Gb.
>> Why? I haven't issued any "commit"...
>>
>> I am using ramBufferMB=8192
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25017728.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: 
http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25017826.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Funtick]

It is NOT sample war, it is SOLR application: solr.war - it should be!!! I
usually build from source and use dist/apache-solr-1.3.war instead, so I am
not sure about solr.war

solr.xml contains configuration for multicore; most probably something is
wrong with it.
Would be better if you try to install Tomcat on localbox and play with it
before going to production...



Aaron Aberg wrote:
> 
> On Mon, Aug 17, 2009 at 10:58 AM, Fuad Efendi wrote:
>> Looks like you are using SOLR multicore, with solr.xml... I never tried
>> it...
>> The rest looks fine, except suspicious solr.xml
> 
> whats suspicious about it? is it in the wrong place? Is it not suppose
> to be there?
> 
> technically my war file is not in my webapps directory. I'm using
> plesk and it installed my war file here:
> tomcat5/psa-webapps/mywebk9.com/solr.war
> 
> I have installed a sample war file and its in the same location. It works
> fine.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Cannot-get-solr-1.3.0-to-run-properly-with-plesk-9.2.1-on-CentOS-tp24980824p25017895.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR - extremely strange behavior! Documents disappeared...

[Funtick]

One more hour, and I have +0.5 mlns more (after commit/optimize)

Something strange happening with SOLR buffer flush (if we have single
segment???)... explicit commit prevents it...

30 hours, with index flush, commit: 783,714
+ 1 hour, commit, optimize: 1,281,851
+ 1 hour, commit, optimize: 1,786,552

Same random docs retrieved from web...



Funtick wrote:
> 
> 
> But how to explain that within an hour (after commit) I have had about
> 500,000 new documents, and within 30 hours (after commit) only 783,714?
> 
> Same _random_enough_ documents... 
> 
> BTW, SOLR Console was showing only few hundreds "deletesById" although I
> don't use any deleteById explicitly; only "update" with "allowOverwrite"
> and "uniqueId".
> 
> 
> 
> 
> markrmiller wrote:
>> 
>> I'd say you have a lot of documents that have the same id.
>> When you add a doc with the same id, first the old one is deleted, then
>> the
>> new one is added (atomically though).
>> 
>> The deleted docs are not removed from the index immediately though - the
>> doc
>> id is just marked as deleted.
>> 
>> Over time though, as segments are merged due to hitting triggers while
>> adding new documents, deletes are removed (which deletes depends on which
>> segments have been merged).
>> 
>> So if you add a tone of documents over time, many with the same ids, you
>> would likely see this type of maxDoc, numDoc churn. maxDoc will include
>> deleted docs while numDoc will not.
>> 
>> 
>> -- 
>> - Mark
>> 
>> http://www.lucidimagination.com
>> 
>> On Mon, Aug 17, 2009 at 11:09 PM, Funtick  wrote:
>> 
>>>
>>> After running an application which heavily uses MD5 HEX-representation
>>> as
>>>  for SOLR v.1.4-dev-trunk:
>>>
>>> 1. After 30 hours:
>>> 101,000,000 documents added
>>>
>>> 2. Commit:
>>> numDocs = 783,714
>>> maxDoc = 3,975,393
>>>
>>> 3. Upload new docs to SOLR during 1 hour(!!!), then commit, then
>>> optimize:
>>> numDocs=1,281,851
>>> maxDocs=1,281,851
>>>
>>> It looks _extremely_ strange that within an hour I have such a huge
>>> increase
>>> with same 'average' document set...
>>>
>>> I am suspecting something goes wrong with Lucene buffer flush / index
>>> merge
>>> OR SOLR - Unique ID handling...
>>>
>>> According to my own estimates, I should have about 10,000,000 new
>>> documents
>>> now... I had 0.5 millions within an hour, and 0.8 mlns within a day;
>>> same
>>> 'random' documents.
>>>
>>> This morning index size was about 4Gb, then suddenly dropped below 0.5
>>> Gb.
>>> Why? I haven't issued any "commit"...
>>>
>>> I am using ramBufferMB=8192
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25017728.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25017967.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: JVM Heap utilization & Memory leaks with Solr

[Funtick]

BTW, you should really prefer JRockit which really rocks!!!

"Mission Control" has necessary toolongs; and JRockit produces _nice_
exception stacktrace (explaining almost everything) in case of even OOM
which SUN JVN still fails to produce.


SolrServlet still catches "Throwable":

} catch (Throwable e) {
  SolrException.log(log,e);
  sendErr(500, SolrException.toStr(e), request, response);
} finally {





Rahul R wrote:
> 
> Otis,
> Thank you for your response. I know there are a few variables here but the
> difference in memory utilization with and without shards somehow leads me
> to
> believe that the leak could be within Solr.
> 
> I tried using a profiling tool - Yourkit. The trial version was free for
> 15
> days. But I couldn't find anything of significance.
> 
> Regards
> Rahul
> 
> 
> On Tue, Aug 4, 2009 at 7:35 PM, Otis Gospodnetic
> > wrote:
> 
>> Hi Rahul,
>>
>> A) There are no known (to me) memory leaks.
>> I think there are too many variables for a person to tell you what
>> exactly
>> is happening, plus you are dealing with the JVM here. :)
>>
>> Try jmap -histo:live PID-HERE | less and see what's using your memory.
>>
>> Otis
>> --
>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>
>>
>>
>> - Original Message 
>> > From: Rahul R 
>> > To: solr-user@lucene.apache.org
>> > Sent: Tuesday, August 4, 2009 1:09:06 AM
>> > Subject: JVM Heap utilization & Memory leaks with Solr
>> >
>> > I am trying to track memory utilization with my Application that uses
>> Solr.
>> > Details of the setup :
>> > -3rd party Software : Solaris 10, Weblogic 10, jdk_150_14, Solr 1.3.0
>> > - Hardware : 12 CPU, 24 GB RAM
>> >
>> > For testing during PSR I am using a smaller subset of the actual data
>> that I
>> > want to work with. Details of this smaller sub-set :
>> > - 5 million records, 4.5 GB index size
>> >
>> > Observations during PSR:
>> > A) I have allocated 3.2 GB for the JVM(s) that I used. After all users
>> > logout and doing a force GC, only 60 % of the heap is reclaimed. As
>> part
>> of
>> > the logout process I am invalidating the HttpSession and doing a
>> close()
>> on
>> > CoreContainer. From my application's side, I don't believe I am holding
>> on
>> > to any resource. I wanted to know if there are known issues surrounding
>> > memory leaks with Solr ?
>> > B) To further test this, I tried deploying with shards. 3.2 GB was
>> allocated
>> > to each JVM. All JVMs had 96 % free heap space after start up. I got
>> varying
>> > results with this.
>> > Case 1 : Used 6 weblogic domains. My application was deployed one 1
>> domain.
>> > I split the 5 million index into 5 parts of 1 million each and used
>> them
>> as
>> > shards. After multiple users used the system and doing a force GC,
>> around
>> 94
>> > - 96 % of heap was reclaimed in all the JVMs.
>> > Case 2: Used 2 weblogic domains. My application was deployed on 1
>> domain.
>> On
>> > the other, I deployed the entire 5 million part index as one shard.
>> After
>> > multiple users used the system and doing a gorce GC, around 76 % of the
>> heap
>> > was reclaimed in the shard JVM. And 96 % was reclaimed in the JVM where
>> my
>> > application was running. This result further convinces me that my
>> > application can be absolved of holding on to memory resources.
>> >
>> > I am not sure how to interpret these results ? For searching, I am
>> using
>> > Without Shards : EmbeddedSolrServer
>> > With Shards :CommonsHttpSolrServer
>> > In terms of Solr objects this is what differs in my code between normal
>> > search and shards search (distributed search)
>> >
>> > After looking at Case 1, I thought that the CommonsHttpSolrServer was
>> more
>> > memory efficient but Case 2 proved me wrong. Or could there still be
>> memory
>> > leaks in my application ? Any thoughts, suggestions would be welcome.
>> >
>> > Regards
>> > Rahul
>>
>>
> 
> 

-- 
View this message in context: 
http://www.nabble.com/JVM-Heap-utilization---Memory-leaks-with-Solr-tp24802380p25018165.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR - extremely strange behavior! Documents disappeared...

[Funtick]

UPDATE:

After few more minutes (after previous commit):
docsPending: about 7,000,000

After commit:
numDocs: 2,297,231

Increase = 2,297,231 - 1,281,851 = 1,000,000 (average)

So that I have 7 docs with same ID in average.

Having 100,000,000 and then dropping below 1,000,000 is strange; it is a bug
somewhere... need to investigate ramBufferSize and MergePolicy, including
SOLR uniqueId implementation...



Funtick wrote:
> 
> After running an application which heavily uses MD5 HEX-representation as
>  for SOLR v.1.4-dev-trunk:
> 
> 1. After 30 hours: 
> 101,000,000 documents added
> 
> 2. Commit: 
> numDocs = 783,714 
> maxDoc = 3,975,393
> 
> 3. Upload new docs to SOLR during 1 hour(!!!), then commit, then
> optimize:
> numDocs=1,281,851
> maxDocs=1,281,851
> 
> It looks _extremely_ strange that within an hour I have such a huge
> increase with same 'average' document set...
> 
> I am suspecting something goes wrong with Lucene buffer flush / index
> merge OR SOLR - Unique ID handling...
> 
> According to my own estimates, I should have about 10,000,000 new
> documents now... I had 0.5 millions within an hour, and 0.8 mlns within a
> day; same 'random' documents.
> 
> This morning index size was about 4Gb, then suddenly dropped below 0.5 Gb.
> Why? I haven't issued any "commit"...
> 
> I am using ramBufferMB=8192
> 
> 
> 
> 
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25018221.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR - extremely strange behavior! Documents disappeared...

[Funtick]

sorry for typo in prev msg,

Increase = 2,297,231 - 1,786,552  = 500,000 (average)

RATE (non-unique-id:unique-id) = 7,000,000 : 500,000 = 14:1

but 125:1 (initial 30 hours) was very strange...



Funtick wrote:
> 
> UPDATE:
> 
> After few more minutes (after previous commit):
> docsPending: about 7,000,000
> 
> After commit:
> numDocs: 2,297,231
> 
> Increase = 2,297,231 - 1,281,851 = 1,000,000 (average)
> 
> So that I have 7 docs with same ID in average.
> 
> Having 100,000,000 and then dropping below 1,000,000 is strange; it is a
> bug somewhere... need to investigate ramBufferSize and MergePolicy,
> including SOLR uniqueId implementation...
> 
> 
> 
> Funtick wrote:
>> 
>> After running an application which heavily uses MD5 HEX-representation as
>>  for SOLR v.1.4-dev-trunk:
>> 
>> 1. After 30 hours: 
>> 101,000,000 documents added
>> 
>> 2. Commit: 
>> numDocs = 783,714 
>> maxDoc = 3,975,393
>> 
>> 3. Upload new docs to SOLR during 1 hour(!!!), then commit, then
>> optimize:
>> numDocs=1,281,851
>> maxDocs=1,281,851
>> 
>> It looks _extremely_ strange that within an hour I have such a huge
>> increase with same 'average' document set...
>> 
>> I am suspecting something goes wrong with Lucene buffer flush / index
>> merge OR SOLR - Unique ID handling...
>> 
>> According to my own estimates, I should have about 10,000,000 new
>> documents now... I had 0.5 millions within an hour, and 0.8 mlns within a
>> day; same 'random' documents.
>> 
>> This morning index size was about 4Gb, then suddenly dropped below 0.5
>> Gb. Why? I haven't issued any "commit"...
>> 
>> I am using ramBufferMB=8192
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-Documents-disappeared...-tp25017728p25018263.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: delta-import using a full-import command is not working

[Noble Paul നോബിള്‍ नोब्ळ्]

you can take a nightly of DIH jar alone. It is quite stable

On Tue, Aug 18, 2009 at 8:21 AM, djain101 wrote:
>
> Looks like this issue has been fixed on Sept 20, 2008 against issue SOLR-768.
> Can someone please let me know which one is a stable jar after Sept 20,
> 2008.
>
>
>
> djain101 wrote:
>>
>> After debugging dataimporter code, i found that it is a bug in the
>> dataimporter 1.3  code itself. doFullImport() in DataImporter class is not
>> loading last index time where as doDeltaImport() is. The code snippet from
>> doFullImport() is:
>>
>> if (requestParams.commit)
>>       setIndexStartTime(new Date());
>>
>>
>> where as in doDeltaImport() is:
>>
>> if (requestParams.commit) {
>>         Date lastModified = writer.loadIndexStartTime();
>>         setIndexStartTime(new Date());
>>         setLastIndexTime(lastModified);
>>       }
>>
>> Looks like it is fixed in latest version of DataImporter.java. I will test
>> with that.
>>
>>
>
> --
> View this message in context: 
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: DataImportHandler - very slow delta import

[Noble Paul നോബിള്‍ नोब्ळ्]

delta imports are likely to be far slower that the full imports
because it makes one db call per changed row. if you can write the
"query" in such a way that it gives only the changed rows, then write
a separate entity (directly under ) and just run a
full-import with that entity only.

On Tue, Aug 18, 2009 at 6:32 AM, Matthew
Painter wrote:
> Hi,
>
> We are using Solr's DataImportHandler to populate the Solr index from a
> SQL Server database of nearly 4,000,000 rows. Whereas the population
> itself is very fast (around 1000 rows per second), the delta import is
> only processing around one row a second.
>
> Is this a known performance issue? We are using Solr 1.3.
>
> For reference, the abridged entity configuration (cuts indicated by
> '...') is below:
>
>              query="select archwaypublic.getSolrIdentifier(oid, 'agency')
> as oid, oid as realoid, archwaypublic.getSolrIdentifier(oid, 'agency')
> as id, code, name, ..."
>   deltaQuery="select oid from publicagency with (nolock) where
> modifiedtime > '${dataimporter.last_index_time}'"
>   deletedPkQuery="select archwaypublic.getSolrIdentifier(entityoid,
> 'agency') as oid from pendingsolrdeletions with (nolock) where
> entitytype='agency'">
>
> ...
> 
>
> Thanks,
> Matt
>
> This e-mail message and any attachments are CONFIDENTIAL to the addressee(s) 
> and may also be LEGALLY PRIVILEGED.  If you are not the intended addressee, 
> please do not use, disclose, copy or distribute the message or the 
> information it contains.  Instead, please notify me as soon as possible and 
> delete the e-mail, including any attachments.  Thank you.
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: delta-import using a full-import command is not working

[djain101]

Can you please point me to the url for downloading latest DIH? Thanks for
your help.


Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
> 
> you can take a nightly of DIH jar alone. It is quite stable
> 
> On Tue, Aug 18, 2009 at 8:21 AM, djain101 wrote:
>>
>> Looks like this issue has been fixed on Sept 20, 2008 against issue
>> SOLR-768.
>> Can someone please let me know which one is a stable jar after Sept 20,
>> 2008.
>>
>>
>>
>> djain101 wrote:
>>>
>>> After debugging dataimporter code, i found that it is a bug in the
>>> dataimporter 1.3  code itself. doFullImport() in DataImporter class is
>>> not
>>> loading last index time where as doDeltaImport() is. The code snippet
>>> from
>>> doFullImport() is:
>>>
>>> if (requestParams.commit)
>>>       setIndexStartTime(new Date());
>>>
>>>
>>> where as in doDeltaImport() is:
>>>
>>> if (requestParams.commit) {
>>>         Date lastModified = writer.loadIndexStartTime();
>>>         setIndexStartTime(new Date());
>>>         setLastIndexTime(lastModified);
>>>       }
>>>
>>> Looks like it is fixed in latest version of DataImporter.java. I will
>>> test
>>> with that.
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
> 
> 

-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25018880.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: delta-import using a full-import command is not working

[Noble Paul നോബിള്‍ नोब्ळ्]

http://people.apache.org/builds/lucene/solr/nightly/

you can just replace the dataimporthandler jar in your current
installation and it should be fine

On Tue, Aug 18, 2009 at 11:18 AM, djain101 wrote:
>
> Can you please point me to the url for downloading latest DIH? Thanks for
> your help.
>
>
> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>>
>> you can take a nightly of DIH jar alone. It is quite stable
>>
>> On Tue, Aug 18, 2009 at 8:21 AM, djain101 wrote:
>>>
>>> Looks like this issue has been fixed on Sept 20, 2008 against issue
>>> SOLR-768.
>>> Can someone please let me know which one is a stable jar after Sept 20,
>>> 2008.
>>>
>>>
>>>
>>> djain101 wrote:

 After debugging dataimporter code, i found that it is a bug in the
 dataimporter 1.3  code itself. doFullImport() in DataImporter class is
 not
 loading last index time where as doDeltaImport() is. The code snippet
 from
 doFullImport() is:

 if (requestParams.commit)
       setIndexStartTime(new Date());


 where as in doDeltaImport() is:

 if (requestParams.commit) {
         Date lastModified = writer.loadIndexStartTime();
         setIndexStartTime(new Date());
         setLastIndexTime(lastModified);
       }

 Looks like it is fixed in latest version of DataImporter.java. I will
 test
 with that.


>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> -
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>
>>
>
> --
> View this message in context: 
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25018880.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: delta-import using a full-import command is not working

[djain101]

I replaced the dataimporthandler.jar from 8/7/2009 build in WEB-INF/lib of
solr.war but on restarting of JBOSS, it threw me following exception but if
i revert back to 1.3 jar then it loads the class fine. Is there any
compatibilty issue between latest dataimporthandler.jar and solr1.3.war?

INFO: xsltCacheLifetimeSeconds=5
23:04:00,442 ERROR [STDERR] Aug 17, 2009 11:04:00 PM
org.apache.solr.common.SolrException log
SEVERE: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.solr.handler.dataimport.DataImportHandler
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:257)
at
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:278)


Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
> 
> http://people.apache.org/builds/lucene/solr/nightly/
> 
> you can just replace the dataimporthandler jar in your current
> installation and it should be fine
> 
> On Tue, Aug 18, 2009 at 11:18 AM, djain101
> wrote:
>>
>> Can you please point me to the url for downloading latest DIH? Thanks for
>> your help.
>>
>>
>> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>>>
>>> you can take a nightly of DIH jar alone. It is quite stable
>>>
>>> On Tue, Aug 18, 2009 at 8:21 AM, djain101
>>> wrote:

 Looks like this issue has been fixed on Sept 20, 2008 against issue
 SOLR-768.
 Can someone please let me know which one is a stable jar after Sept 20,
 2008.



 djain101 wrote:
>
> After debugging dataimporter code, i found that it is a bug in the
> dataimporter 1.3  code itself. doFullImport() in DataImporter class is
> not
> loading last index time where as doDeltaImport() is. The code snippet
> from
> doFullImport() is:
>
> if (requestParams.commit)
>       setIndexStartTime(new Date());
>
>
> where as in doDeltaImport() is:
>
> if (requestParams.commit) {
>         Date lastModified = writer.loadIndexStartTime();
>         setIndexStartTime(new Date());
>         setLastIndexTime(lastModified);
>       }
>
> Looks like it is fixed in latest version of DataImporter.java. I will
> test
> with that.
>
>

 --
 View this message in context:
 http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
 Sent from the Solr - User mailing list archive at Nabble.com.


>>>
>>>
>>>
>>> --
>>> -
>>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25018880.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
> 
> 

-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25019032.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: delta-import using a full-import command is not working

[Noble Paul നോബിള്‍ नोब्ळ्]

OK, I thought you were using an older version of 1.4. the new DIH is
not compatible with 1.3

On Tue, Aug 18, 2009 at 11:37 AM, djain101 wrote:
>
> I replaced the dataimporthandler.jar from 8/7/2009 build in WEB-INF/lib of
> solr.war but on restarting of JBOSS, it threw me following exception but if
> i revert back to 1.3 jar then it loads the class fine. Is there any
> compatibilty issue between latest dataimporthandler.jar and solr1.3.war?
>
> INFO: xsltCacheLifetimeSeconds=5
> 23:04:00,442 ERROR [STDERR] Aug 17, 2009 11:04:00 PM
> org.apache.solr.common.SolrException log
> SEVERE: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.solr.handler.dataimport.DataImportHandler
>        at java.lang.Class.forName0(Native Method)
>        at java.lang.Class.forName(Class.java:247)
>        at
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:257)
>        at
> org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:278)
>
>
> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>>
>> http://people.apache.org/builds/lucene/solr/nightly/
>>
>> you can just replace the dataimporthandler jar in your current
>> installation and it should be fine
>>
>> On Tue, Aug 18, 2009 at 11:18 AM, djain101
>> wrote:
>>>
>>> Can you please point me to the url for downloading latest DIH? Thanks for
>>> your help.
>>>
>>>
>>> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:

 you can take a nightly of DIH jar alone. It is quite stable

 On Tue, Aug 18, 2009 at 8:21 AM, djain101
 wrote:
>
> Looks like this issue has been fixed on Sept 20, 2008 against issue
> SOLR-768.
> Can someone please let me know which one is a stable jar after Sept 20,
> 2008.
>
>
>
> djain101 wrote:
>>
>> After debugging dataimporter code, i found that it is a bug in the
>> dataimporter 1.3  code itself. doFullImport() in DataImporter class is
>> not
>> loading last index time where as doDeltaImport() is. The code snippet
>> from
>> doFullImport() is:
>>
>> if (requestParams.commit)
>>       setIndexStartTime(new Date());
>>
>>
>> where as in doDeltaImport() is:
>>
>> if (requestParams.commit) {
>>         Date lastModified = writer.loadIndexStartTime();
>>         setIndexStartTime(new Date());
>>         setLastIndexTime(lastModified);
>>       }
>>
>> Looks like it is fixed in latest version of DataImporter.java. I will
>> test
>> with that.
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com


>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25018880.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> -
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>
>>
>
> --
> View this message in context: 
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25019032.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: delta-import using a full-import command is not working

[djain101]

How can i get the version of DIH which fixes this issue and is compatible
with 1.3?


Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
> 
> OK, I thought you were using an older version of 1.4. the new DIH is
> not compatible with 1.3
> 
> On Tue, Aug 18, 2009 at 11:37 AM, djain101
> wrote:
>>
>> I replaced the dataimporthandler.jar from 8/7/2009 build in WEB-INF/lib
>> of
>> solr.war but on restarting of JBOSS, it threw me following exception but
>> if
>> i revert back to 1.3 jar then it loads the class fine. Is there any
>> compatibilty issue between latest dataimporthandler.jar and solr1.3.war?
>>
>> INFO: xsltCacheLifetimeSeconds=5
>> 23:04:00,442 ERROR [STDERR] Aug 17, 2009 11:04:00 PM
>> org.apache.solr.common.SolrException log
>> SEVERE: java.lang.NoClassDefFoundError: Could not initialize class
>> org.apache.solr.handler.dataimport.DataImportHandler
>>        at java.lang.Class.forName0(Native Method)
>>        at java.lang.Class.forName(Class.java:247)
>>        at
>> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:257)
>>        at
>> org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:278)
>>
>>
>> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>>>
>>> http://people.apache.org/builds/lucene/solr/nightly/
>>>
>>> you can just replace the dataimporthandler jar in your current
>>> installation and it should be fine
>>>
>>> On Tue, Aug 18, 2009 at 11:18 AM, djain101
>>> wrote:

 Can you please point me to the url for downloading latest DIH? Thanks
 for
 your help.


 Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>
> you can take a nightly of DIH jar alone. It is quite stable
>
> On Tue, Aug 18, 2009 at 8:21 AM, djain101
> wrote:
>>
>> Looks like this issue has been fixed on Sept 20, 2008 against issue
>> SOLR-768.
>> Can someone please let me know which one is a stable jar after Sept
>> 20,
>> 2008.
>>
>>
>>
>> djain101 wrote:
>>>
>>> After debugging dataimporter code, i found that it is a bug in the
>>> dataimporter 1.3  code itself. doFullImport() in DataImporter class
>>> is
>>> not
>>> loading last index time where as doDeltaImport() is. The code
>>> snippet
>>> from
>>> doFullImport() is:
>>>
>>> if (requestParams.commit)
>>>       setIndexStartTime(new Date());
>>>
>>>
>>> where as in doDeltaImport() is:
>>>
>>> if (requestParams.commit) {
>>>         Date lastModified = writer.loadIndexStartTime();
>>>         setIndexStartTime(new Date());
>>>         setLastIndexTime(lastModified);
>>>       }
>>>
>>> Looks like it is fixed in latest version of DataImporter.java. I
>>> will
>>> test
>>> with that.
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
>

 --
 View this message in context:
 http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25018880.html
 Sent from the Solr - User mailing list archive at Nabble.com.


>>>
>>>
>>>
>>> --
>>> -
>>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25019032.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
> 
> 

-- 
View this message in context: 
http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25019071.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: delta-import using a full-import command is not working

[Noble Paul നോബിള്‍ नोब्ळ्]

The only way is to backport the patch to 1.3 . If you are confortable
doing that just modify the relevant code and do an "ant dist" to get
the jar

On Tue, Aug 18, 2009 at 11:42 AM, djain101 wrote:
>
> How can i get the version of DIH which fixes this issue and is compatible
> with 1.3?
>
>
> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>>
>> OK, I thought you were using an older version of 1.4. the new DIH is
>> not compatible with 1.3
>>
>> On Tue, Aug 18, 2009 at 11:37 AM, djain101
>> wrote:
>>>
>>> I replaced the dataimporthandler.jar from 8/7/2009 build in WEB-INF/lib
>>> of
>>> solr.war but on restarting of JBOSS, it threw me following exception but
>>> if
>>> i revert back to 1.3 jar then it loads the class fine. Is there any
>>> compatibilty issue between latest dataimporthandler.jar and solr1.3.war?
>>>
>>> INFO: xsltCacheLifetimeSeconds=5
>>> 23:04:00,442 ERROR [STDERR] Aug 17, 2009 11:04:00 PM
>>> org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.NoClassDefFoundError: Could not initialize class
>>> org.apache.solr.handler.dataimport.DataImportHandler
>>>        at java.lang.Class.forName0(Native Method)
>>>        at java.lang.Class.forName(Class.java:247)
>>>        at
>>> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:257)
>>>        at
>>> org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:278)
>>>
>>>
>>> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:

 http://people.apache.org/builds/lucene/solr/nightly/

 you can just replace the dataimporthandler jar in your current
 installation and it should be fine

 On Tue, Aug 18, 2009 at 11:18 AM, djain101
 wrote:
>
> Can you please point me to the url for downloading latest DIH? Thanks
> for
> your help.
>
>
> Noble Paul നോബിള്‍  नोब्ळ्-2 wrote:
>>
>> you can take a nightly of DIH jar alone. It is quite stable
>>
>> On Tue, Aug 18, 2009 at 8:21 AM, djain101
>> wrote:
>>>
>>> Looks like this issue has been fixed on Sept 20, 2008 against issue
>>> SOLR-768.
>>> Can someone please let me know which one is a stable jar after Sept
>>> 20,
>>> 2008.
>>>
>>>
>>>
>>> djain101 wrote:

 After debugging dataimporter code, i found that it is a bug in the
 dataimporter 1.3  code itself. doFullImport() in DataImporter class
 is
 not
 loading last index time where as doDeltaImport() is. The code
 snippet
 from
 doFullImport() is:

 if (requestParams.commit)
       setIndexStartTime(new Date());


 where as in doDeltaImport() is:

 if (requestParams.commit) {
         Date lastModified = writer.loadIndexStartTime();
         setIndexStartTime(new Date());
         setLastIndexTime(lastModified);
       }

 Looks like it is fixed in latest version of DataImporter.java. I
 will
 test
 with that.


>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25017612.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> -
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25018880.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com


>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25019032.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> -
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>
>>
>
> --
> View this message in context: 
> http://www.nabble.com/delta-import-using-a-full-import-command-is-not-working-tp24989144p25019071.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com