Date Range query in Solr

[Bhawani Sharma]

Hi all,
what will be the syntex of this sql query 
SELECT * FROM table WHERE date <= CURRENT_DATE

in solr format ?


Thanks in advance.

Regards:
Bhawani Sharma
-- 
View this message in context: 
http://www.nabble.com/Date-Range-query-in-Solr-tp21214216p21214216.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Delta DataImport is not picking the modified value in DB

[Manupriya]

Thanks Shalin and Noble for all the help.

I looked into the source code of 1.3 for the Null Pointer that I was
getting. And indeed, there was a problem. In release 1.3, SqlEntityProcessor
doesnt handle 'deltaImportQuery' for delta imports. But this problem is
fixed in the nightly build. 

So now I have one question, we were planning to use Delta Import feature of
Solr in our production environment. And the latest stable release (1.3)
doesnt support all the features for Delta Import. So is it advisable to use
one of the nightly builds in production environement? I am new to Solr, so I
need expert opinion in decision making.

--
Apart from this, I realized, Solr is 'case sensitive', while resolving
variable values. 

For example, the following entity definition gives SQl syntax exception
because the varaible is 'id' in deltaQuery and 'ID' in deltaImportQuery.



And the following definition works fine.



It took me sometime to figure this out and also some looking into the code.
I wrongly assumed it to behave like normal SQL query. 

Is this case-sensitivity intentional? Is it documented somewhere that I can
refer for constructing queries for Delta/Full Import?

Thanks,
Manu


Noble Paul നോബിള്‍ नोब्ळ् wrote:
> 
> On Mon, Dec 29, 2008 at 5:11 PM, Manupriya 
> wrote:
>>
>> Hi Shalin,
>>
>> I am using apache-solr-1.3.0 in my project. I thought it to be the latest
>> release. Does it not support delta import?
> It should, 1.3 of solr was the first realease of DIH. A lot of
> features got added to DIH since then. The nightly build of DIH is
> compatible with Solr 1.3. BTW 1.3 supports delta import.
> 
>>
>> Thanks,
>> Manu
>>
>>
>> Shalin Shekhar Mangar wrote:
>>>
>>> Which version of Solr are you using? The deltaImportQuery feature was
>>> added
>>> recently, you'd need a nightly build for it to work.
>>>
>>> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
>>> wrote:
>>>

 Hi Noble,

 I tried with the deltaImportQuery as well. But still I am getting the
 same
 exception on the server console -

 Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport

 SEVERE: Delta Import Failed
 java.lang.NullPointerException
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at

 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at

 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at

 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at

 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

 Another thing, I do have proper data in the DB.

 Thanks,
 Manu


 Noble Paul നോബിള്‍ नोब्ळ् wrote:
 >
 > The same example with deltaImportQuery would look as follows
 > 
 >>>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
 />
 >
 >>>> >deltaQuery="select id from item where last_modified >
 > '${dataimporter.last_index_time}'"
 > deltaImportQuery="select * from item where
 ID=${dataimporter.delta.ID}"
 >>
 >
 >
 >
 >
 >>>> >query="select DESCRIPTION from FEATURE where
 > ITEM_ID='${item.ID}'"
 >deltaQuery="select ITEM_ID from FEATURE where
 > last_modified > '${dataimporter.last_index_time}'"
 >parentDeltaQuery="select ID from item where
 > ID=${feature.ITEM_ID}">
 >
 >
 >
 >>>> >query="select CATEGORY_ID from item_category where
 > ITEM_ID='${item.ID}'"
 >deltaQuery="select ITEM_ID, CATEGORY_ID from
 > item_category where last_modified >
 '${dataimporter.last_index_time}'"
 >parentDeltaQuery="select ID from item where
 > ID=${item_category.ITEM_ID}">
 >>>> >query="select DESCRIPTION from category where
 ID
 =
 > '${item_category.CATEGORY_ID}'"
 >deltaQuery="select ID from category wher

Re: Date Range query in Solr

[Shalin Shekhar Mangar]

There is no table in Solr -- all fields are in the same schema.

For the date you can try using the following as the 'q' parameter:
date:[* TO NOW]

On Tue, Dec 30, 2008 at 1:43 PM, Bhawani Sharma wrote:

>
> Hi all,
> what will be the syntex of this sql query
> SELECT * FROM table WHERE date <= CURRENT_DATE
>
> in solr format ?
>
>
> Thanks in advance.
>
> Regards:
> Bhawani Sharma
> --
> View this message in context:
> http://www.nabble.com/Date-Range-query-in-Solr-tp21214216p21214216.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: Delta DataImport is not picking the modified value in DB

[Shalin Shekhar Mangar]

On Tue, Dec 30, 2008 at 3:33 PM, Manupriya wrote:

>
> Thanks Shalin and Noble for all the help.
>
> I looked into the source code of 1.3 for the Null Pointer that I was
> getting. And indeed, there was a problem. In release 1.3,
> SqlEntityProcessor
> doesnt handle 'deltaImportQuery' for delta imports. But this problem is
> fixed in the nightly build.


DataImportHandler in the Solr 1.3 release supports delta imports. In that
version we used to generate the query ourselves after detecting the changed
primary key. This was not very flexible and worked only for very simple
queries. Due to this the deltaImportQuery attribute was introduced (after
release 1.3). You can use deltaImportQuery to exactly specify what query
should be executed once the primary key of a changed row is identified. The
example given in the wiki should work with Solr 1.3 release.

So now I have one question, we were planning to use Delta Import feature of
> Solr in our production environment. And the latest stable release (1.3)
> doesnt support all the features for Delta Import. So is it advisable to use
> one of the nightly builds in production environement? I am new to Solr, so
> I
> need expert opinion in decision making.


If the problem was related only to case sensitive names and your import
works without deltaImportQuery then you should use 1.3 otherwise you don't
have a lot of choice :)

All the normal caveats of trunk code apply to 1.4, however I believe it is
fairly stable.


>
> --
> Apart from this, I realized, Solr is 'case sensitive', while resolving
> variable values.
>
> For example, the following entity definition gives SQl syntax exception
> because the varaible is 'id' in deltaQuery and 'ID' in deltaImportQuery.
>
>deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'"
>   deltaImportQuery="select * from item where
> ID=${dataimporter.delta.ID}">
>
> 
> And the following definition works fine.
>
> deltaQuery="select ID from item where last_modified >
> '${dataimporter.last_index_time}'"
>   deltaImportQuery="select * from item where
> ID=${dataimporter.delta.ID}">
>
> It took me sometime to figure this out and also some looking into the code.
> I wrongly assumed it to behave like normal SQL query.
>
> Is this case-sensitivity intentional? Is it documented somewhere that I can
> refer for constructing queries for Delta/Full Import?


There was a bug related to case-sensitivity in 1.3 which has been fixed in
trunk. Now, unless you specify the 'name' attribute for a field DIH does
case-insensitive matches. If you are providing a 'name' it must be in the
same case as the column name returned by SQL.

-- 
Regards,
Shalin Shekhar Mangar.

Re: Delta DataImport is not picking the modified value in DB

[Noble Paul നോബിള്‍ नोब्ळ्]

You do not have to take up complete trunk. Solr can continue to be 1.3
and DIH can be a nightly . DIH is shipped as a separate jar in the
same .war file

This can minimize your risks



On Tue, Dec 30, 2008 at 4:54 PM, Shalin Shekhar Mangar
 wrote:
> On Tue, Dec 30, 2008 at 3:33 PM, Manupriya wrote:
>
>>
>> Thanks Shalin and Noble for all the help.
>>
>> I looked into the source code of 1.3 for the Null Pointer that I was
>> getting. And indeed, there was a problem. In release 1.3,
>> SqlEntityProcessor
>> doesnt handle 'deltaImportQuery' for delta imports. But this problem is
>> fixed in the nightly build.
>
>
> DataImportHandler in the Solr 1.3 release supports delta imports. In that
> version we used to generate the query ourselves after detecting the changed
> primary key. This was not very flexible and worked only for very simple
> queries. Due to this the deltaImportQuery attribute was introduced (after
> release 1.3). You can use deltaImportQuery to exactly specify what query
> should be executed once the primary key of a changed row is identified. The
> example given in the wiki should work with Solr 1.3 release.
>
> So now I have one question, we were planning to use Delta Import feature of
>> Solr in our production environment. And the latest stable release (1.3)
>> doesnt support all the features for Delta Import. So is it advisable to use
>> one of the nightly builds in production environement? I am new to Solr, so
>> I
>> need expert opinion in decision making.
>
>
> If the problem was related only to case sensitive names and your import
> works without deltaImportQuery then you should use 1.3 otherwise you don't
> have a lot of choice :)
>
> All the normal caveats of trunk code apply to 1.4, however I believe it is
> fairly stable.
>
>
>>
>> --
>> Apart from this, I realized, Solr is 'case sensitive', while resolving
>> variable values.
>>
>> For example, the following entity definition gives SQl syntax exception
>> because the varaible is 'id' in deltaQuery and 'ID' in deltaImportQuery.
>>
>> >   deltaQuery="select id from item where last_modified >
>> '${dataimporter.last_index_time}'"
>>   deltaImportQuery="select * from item where
>> ID=${dataimporter.delta.ID}">
>>
>> 
>> And the following definition works fine.
>>
>> >deltaQuery="select ID from item where last_modified >
>> '${dataimporter.last_index_time}'"
>>   deltaImportQuery="select * from item where
>> ID=${dataimporter.delta.ID}">
>>
>> It took me sometime to figure this out and also some looking into the code.
>> I wrongly assumed it to behave like normal SQL query.
>>
>> Is this case-sensitivity intentional? Is it documented somewhere that I can
>> refer for constructing queries for Delta/Full Import?
>
>
> There was a bug related to case-sensitivity in 1.3 which has been fixed in
> trunk. Now, unless you specify the 'name' attribute for a field DIH does
> case-insensitive matches. If you are providing a 'name' it must be in the
> same case as the column name returned by SQL.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
--Noble Paul

Re: High response times after snapshot install

[Victor Hogemann]

Hello Otis,

Thank you for the reply.

But, if the high response time is caused by IO and CPU usage... it  
shouldn't come back to regular response time after a while? Also,  
we're not seeing that much CPU usage.


What if some requests got stuck, and the old searchers are not being  
destroyed? Would it account for the high response times?


PS.: Sorry my poor english, I'm not a native speaker :-)

Victor Hogemann - Ideais Tecnologia

Tel (RJ) +55 +21 3553-1301 (R217)
Tel (SP) +55 +11 4063-0443 (R217)
Fax +55 +21 3553-1302

Rua da Assembléia, nº 98, 9º andar
Centro | Rio de Janeiro | Brasil
CEP 20011-000

On Dec 29, 2008, at 4:26 PM, Otis Gospodnetic wrote:


Hello Victor,

The high response times could be caused by long and intense warming  
of the new searcher(s), which could be causing high IO (it sounds  
like all instances share the same index on the same file system) and  
high CPU usage (lots of cache items being copied from the old  
searcher).



Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 

From: Victor Hogemann 
To: Solr-User 
Sent: Monday, December 29, 2008 7:53:44 AM
Subject: High response times after snapshot install

Greetings,

Before anything I'd like to thank the Solr team for such great  
piece of

software! Thank you.

Now... back to my problem. I'm experiencing slowdowns on a Solr 1.3  
multicore
installation after a snapshot install, response times go up from  
5ms to above

5sec, and don't go down unless we restart the servers.

Our index has more than 20 fields, and the slave servers got some  
50hits/sec.
We're running it on Weblogic, with 6 Weblogic Nodes/Solr instances  
per machine,

sharing the same index on the filesystem.

Any insight on what is happening?

Thanks.

Victor Hogemann - Ideais Tecnologia

Tel (RJ) +55 +21 3553-1301 (R217)
Tel (SP) +55 +11 4063-0443 (R217)
Fax +55 +21 3553-1302

Rua da Assembléia, nº 98, 9º andar
Centro | Rio de Janeiro | Brasil
CEP 20011-000

synonyms.txt file updated frequently

[Alexander Ramos Jardim]

Hello guys,

As the title suggests I must update my synonyms.txt file frequently. What is
the best approach? Should I send a commit after the file is updated? Does
Solr need to be restarted after the file changes?

-- 
Alexander Ramos Jardim

Re: synonyms.txt file updated frequently

[Smiley, David W.]

You need to reload the core (if you're using SolrCore).  If you're not using 
SolrCore then I'm afraid you'll have to bring down them up the web-app.

~ David


On 12/30/08 8:12 AM, "Alexander Ramos Jardim" 
 wrote:

Hello guys,

As the title suggests I must update my synonyms.txt file frequently. What is
the best approach? Should I send a commit after the file is updated? Does
Solr need to be restarted after the file changes?

--
Alexander Ramos Jardim

Re: synonyms.txt file updated frequently

[Grant Ingersoll]

I'd probably write a new TokenFilter that was aware of the reload  
policy (in a generic way) such that I didn't have to go through a  
whole core reload every time.  Are you just using them during query  
time or also during indexing?


-Grant

On Dec 30, 2008, at 8:12 AM, Alexander Ramos Jardim wrote:


Hello guys,

As the title suggests I must update my synonyms.txt file frequently.  
What is
the best approach? Should I send a commit after the file is updated?  
Does

Solr need to be restarted after the file changes?

--
Alexander Ramos Jardim


--
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Re: synonyms.txt file updated frequently

[Alexander Ramos Jardim]

Hey Grant,

Thanks for the info!

2008/12/30 Grant Ingersoll 

> I'd probably write a new TokenFilter that was aware of the reload policy
> (in a generic way) such that I didn't have to go through a whole core reload
> every time.  Are you just using them during query time or also during
> indexing?
>

I am using it at indexing time.


>
> -Grant
>
>
> On Dec 30, 2008, at 8:12 AM, Alexander Ramos Jardim wrote:
>
>  Hello guys,
>>
>> As the title suggests I must update my synonyms.txt file frequently. What
>> is
>> the best approach? Should I send a commit after the file is updated? Does
>> Solr need to be restarted after the file changes?
>>
>> --
>> Alexander Ramos Jardim
>>
>
> --
> Grant Ingersoll
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
>
>
>


-- 
Alexander Ramos Jardim

Re: Local Solr - Error creating document with DIH

[MapYours]

Thanks,
That worked perfectly.


Noble Paul നോബിള്‍ नोब्ळ् wrote:
> 
> apparently the local solr expects everything as string. I guess we
> must raise an issue with them.
> meanwhile you can convert those double fields to String using
> TemplateTransformer
> 
> 
>   
> 
> 
> 
> On Tue, Dec 30, 2008 at 5:49 AM, MapYours  wrote:
>>
>> I have my data import handler set up correctly it connects and streams
>> results returned from the database (sqlserver 2005) using JTDS driver.
>>
>> However, no documents are indexed:
>> Dec 29, 2008 2:03:07 PM org.apache.solr.handler.dataimport.SolrWriter
>> upload
>> WARNING: Error creating document :
>> SolrInputDocumnt[{incident_num=incident_num(1.0)={08518194 },
>> incident_date=incident_date(1.0)={2008-12-29 00:00:00.0},
>> location=location(1.0)={4300BLK SW 173RD AVE
>> }, lat=lat(1.0)={45.4882}, lng=lng(1.0)={-122.85455}}]
>> java.lang.ClassCastException: java.lang.Double cannot be cast to
>> java.lang.String
>>at
>> com.pjaol.search.solr.update.LocalUpdaterProcessor.processAdd(LocalUpdateProcessorFactory.java:136)
>>at
>> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
>>at
>> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:288)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
>>at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
>>at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>
>> Here is my schema:
>> 
>>   
>>   
>>   
>>
>> 
>>   
>>   
>>
>>> stored="true"/>
>>  
>>
>>  
>>  incident_num
>>
>>  
>>  location
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Local-Solr---Error-creating-document-with-DIH-tp21211133p21211133.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> --Noble Paul
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Local-Solr---Error-creating-document-with-DIH-tp21211133p21220673.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Understanding Filters

[k_richard]

Jim -

I'm interested in doing something similar and was wondering if you ever got
a response.  I've looked at the solr source code and have some ideas about
where to apply additional filtering but they're not that elegant.

>From what I can see, if you add an additional search component after the
initial query you would need to update all of the saved doc sets/lists with
your filtered data to enable downstream search components to work
effectively.  However, there's warnings strewn about the code that warn
against modifying these.

Any thoughts?

Thanks.

- Kevin


Jim Murphy wrote:
> 
> I'm still trying to filter my search results with external data.  I have
> ~100 million documents in the index.  I want to use the power of lucene to
> knock that index down to 10-100 with keyword searching and a few other
> regular query terms.  With that smaller subset I'd like to apply a filter
> based on making calls to an external system to further reduce that set to
> 5-20.
> 
> Looking at filter queries and lucense search filters it seems that they
> iterate over the entire index to create a bitset of documents to be
> included in the query.  This seems the inverse of my needs.  I can't make
> ~100 milion external calls to filter - I want lucene to handle that heavy
> lifting.
> 
> I'm trying to figure out the right place to hook to let paging and caching
> in Solr work as normal but drop out result documents based on that
> expensive external call.
> 
> Thanks, and sorry for the repeat requests. 
> 
> Jim
> 

-- 
View this message in context: 
http://www.nabble.com/Understanding-Filters-tp18742220p21221020.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: High response times after snapshot install

[Otis Gospodnetic]

Hi Victor,

The IO and CPU usage should go down with time, unless you are reopening your 
searcher too frequently and your server is always busy doing what I described 
in the previous email.  You should provide us with relevant info/numbers from 
your Solr admin stats page, output of top, vmstat, and such.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Victor Hogemann 
> To: solr-user@lucene.apache.org
> Sent: Tuesday, December 30, 2008 7:56:13 AM
> Subject: Re: High response times after snapshot install
> 
> Hello Otis,
> 
> Thank you for the reply.
> 
> But, if the high response time is caused by IO and CPU usage... it shouldn't 
> come back to regular response time after a while? Also, we're not seeing that 
> much CPU usage.
> 
> What if some requests got stuck, and the old searchers are not being 
> destroyed? 
> Would it account for the high response times?
> 
> PS.: Sorry my poor english, I'm not a native speaker :-)
> 
> Victor Hogemann - Ideais Tecnologia
> 
> Tel (RJ) +55 +21 3553-1301 (R217)
> Tel (SP) +55 +11 4063-0443 (R217)
> Fax +55 +21 3553-1302
> 
> Rua da Assembléia, nº 98, 9º andar
> Centro | Rio de Janeiro | Brasil
> CEP 20011-000
> 
> On Dec 29, 2008, at 4:26 PM, Otis Gospodnetic wrote:
> 
> > Hello Victor,
> > 
> > The high response times could be caused by long and intense warming of the 
> > new 
> searcher(s), which could be causing high IO (it sounds like all instances 
> share 
> the same index on the same file system) and high CPU usage (lots of cache 
> items 
> being copied from the old searcher).
> > 
> > 
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > 
> > 
> > 
> > - Original Message 
> >> From: Victor Hogemann 
> >> To: Solr-User 
> >> Sent: Monday, December 29, 2008 7:53:44 AM
> >> Subject: High response times after snapshot install
> >> 
> >> Greetings,
> >> 
> >> Before anything I'd like to thank the Solr team for such great piece of
> >> software! Thank you.
> >> 
> >> Now... back to my problem. I'm experiencing slowdowns on a Solr 1.3 
> >> multicore
> >> installation after a snapshot install, response times go up from 5ms to 
> >> above
> >> 5sec, and don't go down unless we restart the servers.
> >> 
> >> Our index has more than 20 fields, and the slave servers got some 
> >> 50hits/sec.
> >> We're running it on Weblogic, with 6 Weblogic Nodes/Solr instances per 
> machine,
> >> sharing the same index on the filesystem.
> >> 
> >> Any insight on what is happening?
> >> 
> >> Thanks.
> >> 
> >> Victor Hogemann - Ideais Tecnologia
> >> 
> >> Tel (RJ) +55 +21 3553-1301 (R217)
> >> Tel (SP) +55 +11 4063-0443 (R217)
> >> Fax +55 +21 3553-1302
> >> 
> >> Rua da Assembléia, nº 98, 9º andar
> >> Centro | Rio de Janeiro | Brasil
> >> CEP 20011-000
> > 

Re: synonyms.txt file updated frequently

[Grant Ingersoll]

On Dec 30, 2008, at 11:05 AM, Alexander Ramos Jardim wrote:


Hey Grant,

Thanks for the info!

2008/12/30 Grant Ingersoll 

I'd probably write a new TokenFilter that was aware of the reload  
policy
(in a generic way) such that I didn't have to go through a whole  
core reload

every time.  Are you just using them during query time or also during
indexing?



I am using it at indexing time.


I think that is a bit more problematic.  How do you deal with new  
documents having the new synonyms while old docs don't?


Any particular reason you use syns at indexing and not search?  Not  
saying there aren't reasons to do it, just query side usually works  
better for this very reason.

Re: synonyms.txt file updated frequently

[Smiley, David W.]

Grant, the Solr wiki recommends doing expansion at index time and gives reasons:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#head-2c461ac74b4ddd82e453dc68fcfc92da77358d46

Query-time doesn't work for multi-word expansion.  For everyone's convenience, 
I'll quote the remainder of the problems:


Even when you aren't worried about multi-word synonyms, idf differences still 
make index time synonyms a good idea. Consider the following scenario:

*  An index with a "text" field, which at query time uses the SynonymFilter 
with the synonym TV, Televesion and expand="true"
*  Many thousands of documents containing the term "text:TV"
*  A few hundred documents containing the term "text:Television"

A query for text:TV will expand into (text:TV text:Television) and the lower 
docFreq for text:Television will give the documents that match "Television" a 
much higher score then docs that match "TV" comparably -- which may be somewhat 
counter intuitive to the client. Index time expansion (or reduction) will 
result in the same idf for all documents regardless of which term the original 
text contained.

~ David Smiley

On 12/30/08 4:33 PM, "Grant Ingersoll"  wrote:



On Dec 30, 2008, at 11:05 AM, Alexander Ramos Jardim wrote:

> Hey Grant,
>
> Thanks for the info!
>
> 2008/12/30 Grant Ingersoll 
>
>> I'd probably write a new TokenFilter that was aware of the reload
>> policy
>> (in a generic way) such that I didn't have to go through a whole
>> core reload
>> every time.  Are you just using them during query time or also during
>> indexing?
>>
>
> I am using it at indexing time.

I think that is a bit more problematic.  How do you deal with new
documents having the new synonyms while old docs don't?

Any particular reason you use syns at indexing and not search?  Not
saying there aren't reasons to do it, just query side usually works
better for this very reason.

Associating scores with fields?

[mcox]

Hi,

I'd like to get scores for all fields within my returned docs. My current
solution is to parse the debug information but because I often have multiple
fields with the same name this doesn't always work too well.

Is there a way to get something to the effect of: 


Thanks,

Mike
-- 
View this message in context: 
http://www.nabble.com/Associating-scores-with-fields--tp21225634p21225634.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: http internal error if i enable debugQuery=on

[vinay kumar kaku]

you mean query term? if so i do have ?q=xyz etc... if not what is that 
parameter you are talking about?i will send to only one mailing list here 
on.thanks,vinay> Date: Mon, 29 Dec 2008 18:38:33 -0800> From: 
otis_gospodne...@yahoo.com> Subject: Re: http internal error if i enable 
debugQuery=on> To: solr-user@lucene.apache.org> > Hi,> > Do you have q=XXX 
parameter in your URL as well?  Try specifying it.> > > Otis --> Sematext -- 
http://sematext.com/ -- Lucene - Solr - Nutch> > > > - Original Message 
>> From: vinay kumar kaku >> To: 
solr-user@lucene.apache.org>> Sent: Monday, December 29, 2008 6:09:24 PM>> 
Subject: http internal error if i enable debugQuery=on>> >> >> Hi,  i got solr 
on 23rd dec and using for my application. i have a filter >> 
&fq=-filtername[1+TO+*] all works fine until i say &debugQuery=on i get below 
>> error from my tomcat catalina.out logfile. if i change that filter to say >> 
&fq=-filtername[1+TO+10]&debugQuery=on it works fine.HTTP Status 500 - >> 
java.lang.NullPointerException: value cannot be null 
java.lang.RuntimeException: >> java.lang.NullPointerException: value cannot be 
null at >> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:469) 
at >> 
org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:75)
 >> at >> 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:179)
 >> at >> 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
 >> at >> 
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:233)
 >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1325) at >> 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) 
>> at >> 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
 >> at >> 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 >> at >> 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 >> at >> 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 >> at >> 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 >> at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 
>> at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) 
at >> 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 >> at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) >> 
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 
at >> 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
 >> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) 
at >> java.lang.Thread.run(Thread.java:619) Caused by: 
java.lang.NullPointerException: >> value cannot be null at 
org.apache.lucene.document.Field.(Field.java:275) >> at 
org.apache.lucene.document.Field.(Field.java:251) at >> 
org.apache.solr.search.QueryParsing.writeFieldVal(QueryParsing.java:306) at >> 
org.apache.solr.search.QueryParsing.toString(QueryParsing.java:338) at >> 
org.apache.solr.search.QueryParsing.toString(QueryParsing.java:401) at >> 
org.apache.solr.search.QueryParsing.toString(QueryParsing.java:466) ... 20 more 
>> thanks,vinay>> 
_>> Send e-mail 
anywhere. No map, no compass.>> 
http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008>
 
_
Send e-mail faster without improving your typing skills.
http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008

Re: http internal error if i enable debugQuery=on

[Otis Gospodnetic]

Vinay,

You may want to check your email setting - your emails are very hard to read 
because of the wrapping text.
"q" is the parameter I was talking about.  I can't tell from your emails what 
you might be doing wrong or where the bug might be.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: vinay kumar kaku 
> To: solr-user@lucene.apache.org
> Sent: Tuesday, December 30, 2008 8:56:34 PM
> Subject: RE: http internal error if i enable debugQuery=on
> 
> 
> you mean query term? if so i do have ?q=xyz etc... if not what is that 
> parameter 
> you are talking about?i will send to only one mailing list here 
> on.thanks,vinay> 
> Date: Mon, 29 Dec 2008 18:38:33 -0800> From: otis_gospodne...@yahoo.com> 
> Subject: Re: http internal error if i enable debugQuery=on> To: 
> solr-user@lucene.apache.org> > Hi,> > Do you have q=XXX parameter in your URL 
> as 
> well?  Try specifying it.> > > Otis --> Sematext -- http://sematext.com/ -- 
> Lucene - Solr - Nutch> > > > - Original Message >> From: vinay kumar 
> kaku >> To: solr-user@lucene.apache.org>> Sent: Monday, 
> December 29, 2008 6:09:24 PM>> Subject: http internal error if i enable 
> debugQuery=on>> >> >> Hi,  i got solr on 23rd dec and using for my 
> application. 
> i have a filter >> &fq=-filtername[1+TO+*] all works fine until i say 
> &debugQuery=on i get below >> error from my tomcat catalina.out logfile. if i 
> change that filter to say >> &fq=-filtername[1+TO+10]&debugQuery=on it works 
> fine.HTTP Status 500 - >> java.lang.NullPointerException: value cannot be 
> null 
> java.lang.RuntimeException: >> java.lang.NullPointerException: value cannot 
> be 
> null at >> 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:469) 
> at >> 
> org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:75)
>  
> >> at >> 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:179)
>  
> >> at >> 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>  
> >> at >> 
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:233)
>  
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1325) at >> 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>  
> >> at >> 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>  
> >> at >> 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>  
> >> at >> 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>  
> >> at >> 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>  
> >> at >> 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
>  
> >> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 
> >> 
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
> >> at 
> >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) 
> at >> 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>  
> >> at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) 
> >> 
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 
> at 
> >> 
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>  
> >> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) 
> >> at 
> >> java.lang.Thread.run(Thread.java:619) Caused by: 
> java.lang.NullPointerException: >> value cannot be null at 
> org.apache.lucene.document.Field.(Field.java:275) >> at 
> org.apache.lucene.document.Field.(Field.java:251) at >> 
> org.apache.solr.search.QueryParsing.writeFieldVal(QueryParsing.java:306) at 
> >> 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:338) at >> 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:401) at >> 
> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:466) ... 20 
> more 
> >> thanks,vinay>> 
> _>> Send 
> e-mail 
> anywhere. No map, no compass.>> 
> http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008>
>  
> 
> _
> Send e-mail faster without improving your typing skills.
> http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008

Re: Associating scores with fields?

[Otis Gospodnetic]

Mike,

There is nothing that exposes per-field scores in Solr and not even in Lucene, 
but if you want per-field scores I'd recommend looking at Lucene Scorer & 
friends first.  See also http://lucene.apache.org/java/2_4_0/scoring.html

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: mcox 
> To: solr-user@lucene.apache.org
> Sent: Tuesday, December 30, 2008 7:48:28 PM
> Subject: Associating scores with fields?
> 
> 
> Hi,
> 
> I'd like to get scores for all fields within my returned docs. My current
> solution is to parse the debug information but because I often have multiple
> fields with the same name this doesn't always work too well.
> 
> Is there a way to get something to the effect of: 
> 
> 
> Thanks,
> 
> Mike
> -- 
> View this message in context: 
> http://www.nabble.com/Associating-scores-with-fields--tp21225634p21225634.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Problem with query response writer when upgrading from Solr1.2 to solr1.3

[Pooja Verlani]

Hi,



I just upgraded my system from Solr 1.2 to Solr 1.3. I am using the same
plugin for the queryResponseWriter that I used in Solr1.2. Problem here is
that when I am using *wt* parameter as the plugin name with full package
then I don't get the response which I used to get in 1.2 and when I don't
give WT parameter, I get the perfect response from the default
XMLResponseWriter as expected. Also the above problem occurs only when we
use Shards and also, response is perfect when wt is given with hl=true. This
occurs only when I am using distributed query on multiple shards, on
individual shards it working fine i.e. when we use /select clause on
individual shards.
(http://localhost:8081/solr/select?q=%22indian%20railways%22&qt=modified&fl=*,score&wt=custom&hl=true).




On individual shards, the custom responsewriters are working absolutely fine
but not with combining shards or using /distrib/


http://localhost:8081/solr/distrib?q=%22indian%20railways%22&qt=modified&fl=*,score&wt=custom&hl=true




Please help.





This is part of solrconfig.xml



   

 

   x,y,z

 

   



**

**

* *

*
*

Regards,**

*Pooja
*

Re: Problem with query response writer when upgrading from Solr1.2 to solr1.3

[Shalin Shekhar Mangar]

The wt parameter accepts the 'name' you gave in solrconfig while specifying
your custom response writer. I don't think there has been any change in
handling of wt which would result in the behavior you are seeing.

Are there any exceptions in the Solr log? Is the jar with the custom
response writer present in all shards?

On Wed, Dec 31, 2008 at 9:44 AM, Pooja Verlani wrote:

> Hi,
>
>
>
> I just upgraded my system from Solr 1.2 to Solr 1.3. I am using the same
> plugin for the queryResponseWriter that I used in Solr1.2. Problem here is
> that when I am using *wt* parameter as the plugin name with full package
> then I don't get the response which I used to get in 1.2 and when I don't
> give WT parameter, I get the perfect response from the default
> XMLResponseWriter as expected. Also the above problem occurs only when we
> use Shards and also, response is perfect when wt is given with hl=true.
> This
> occurs only when I am using distributed query on multiple shards, on
> individual shards it working fine i.e. when we use /select clause on
> individual shards.
> (
> http://localhost:8081/solr/select?q=%22indian%20railways%22&qt=modified&fl=*,score&wt=custom&hl=true
> ).
>
> <
> http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/select?q=%22indian%20railways%22&qt=blogs&fl=*,score&wt=blogs&hl=true%29.Please
> >
>
>
> On individual shards, the custom responsewriters are working absolutely
> fine
> but not with combining shards or using /distrib/
>
> <
> http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/select?q=%22indian%20railways%22&qt=blogs&fl=*,score&wt=blogs&hl=true%29.Please
> >
>
> http://localhost:8081/solr/distrib?q=%22indian%20railways%22&qt=modified&fl=*,score&wt=custom&hl=true
> <
> http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/distrib?q=%22indian%20railways%22&qt=blogs&fl=*,score&wt=blogspdrm&hl=true
> >
>
>
> <
> http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/distrib?q=%22indian%20railways%22&qt=blogs&fl=*,score&wt=blogspdrm&hl=true
> >
>
> Please help.
>
>
>
>
>
> This is part of solrconfig.xml
>
>
>
>   
>
> 
>
>   x,y,z
>
> 
>
>   
>
>
>
> * class="FirstResponseWriter"/>*
>
> **
>
> * *
>
> *
> *
>
> Regards,**
>
> *Pooja
> *
>



-- 
Regards,
Shalin Shekhar Mangar.