Re: Looking for Developers

[Ravi Gidwani]

May I suggest a new mailing list like solr-jobs (if it does not exist) or
something for such emails ? I think it is also important for the solr
developers to get emails about job opportunities ? No ?

~Ravi.

On Tue, Oct 26, 2010 at 11:42 PM, Pradeep Singh  wrote:

> This is the second time he has sent this shit. Kill his subscription. Is it
> possible?
>
> On Tue, Oct 26, 2010 at 10:38 PM, Yuchen Wang  wrote:
>
> > UNSUBSCRIBE
> >
> > On Tue, Oct 26, 2010 at 10:15 PM, Igor Chudov  wrote:
> >
> > > UNSUBSCRIBE
> > >
> > > On Wed, Oct 27, 2010 at 12:14 AM, ST ST  wrote:
> > > > Looking for Developers Experienced in Solr/Lucene And/OR FAST Search
> > > Engines
> > > > from India (Pune)
> > > >
> > > > We are looking for off-shore India Based Developers who are
> proficient
> > in
> > > > Solr/Lucene and/or FAST search engine .
> > > > Developers in the cities of Pune/Bombay in India are preferred.
> > > Development
> > > > is for projects based in US for a reputed firm.
> > > >
> > > > If you are proficient in Solr/Lucene/FAST and have 5 years minimum
> > > industry
> > > > experience with atleast 3 years in Search Development,
> > > > please send me your resume.
> > > >
> > > > Thanks
> > > >
> > >
> >
>

upper limit to boost weight/value ?

[Ravi Gidwani]

Hello:

Is there any upper limit to the boost weight/value ? For example in the
following query :

&qf=exact_title^2000+exact_category^1900+exact_tags^1700

are these boost values acceptable and work as expected ?

Thanks,
~Ravi Gidwani

Re: upper limit to boost weight/value ?

[Ravi Gidwani]

Thanks Lance.

On Sun, May 8, 2011 at 7:34 PM, Lance Norskog  wrote:

> There is no upper limit. These are floats. But they can be small too.
> Boost < 1 are 'under normal'.
>
> One radix sorting trick is to boost one field 1000 and another field
> 5. If the first field is a string facet, this gives each group of
> results in one long query. Lucene sorting does radix also, of course,
> but sometimes it is not the best tool.
>
>
>
>
> On 5/8/11, Ravi Gidwani  wrote:
> > Hello:
> >
> > Is there any upper limit to the boost weight/value ? For example in the
> > following query :
> >
> > &qf=exact_title^2000+exact_category^1900+exact_tags^1700
> >
> > are these boost values acceptable and work as expected ?
> >
> > Thanks,
> > ~Ravi Gidwani
> >
>
>
> --
> Lance Norskog
> goks...@gmail.com
>

Re: Remove the deleted docs from the Solr Index

[Ravi Gidwani]

Hi Shalin:

>   I get your point about not knowing what has been deleted from the 
> database. So this is what even I am looking for:
>
> 0) A document (id=100) is currently part of solr index.(
> 1) Lets say the application deleted a record with id=100 from database.
>
> 2) Now I need to execute some DIH command to say remove document where 
> id=100. I dont expect the DIH to automatically detect what has been deleted,
> but I am looking for a DIH command/special-command to request deletion from 
> index.
>
> Is that possible ? also as an alternate solution, is it possible to build 
> index using DIH, and use the solr.XmlUpdateRequestHandler request handler to 
> delete/update these one off documents ?
> Is this something you will recommend ?
>
> Thanks,
> ~Ravi Gidwani.
>
> On Tue, Dec 29, 2009 at 3:03 AM, Mohamed Parvez  wrote:
>
> > I have looked in the that thread earlier. But there is no option there for
>
> > a
> > solution from Solr side.
> >
> > I mean the two more options there are
> > 1] Use database triggers instead of DIH to manage updating the index :-
> > This out of question as we cant run 1000 odd triggers every hour to delete.
>
> >
> > 2] Some sort of ORM use its interception:-
> > This is also out of question as the deletes happens form external system or
> > directly on the database, not through our application.
> >
> >
>
> > To Say in Short, Solr Should have something thing to keep the index synced
> > with the database. As of now its one way street, updates rows, on DB will
> > go
> > to the index. Deleted rows in the DB, will not be deleted from the Index
>
> >
> >
> How can Solr figure out what has been deleted? Should it go through each row
> and comparing against each doc? Even then some things are not possible
> (think indexed fields). It would be far efficient to just do a full-import
>
> each time instead.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>

Boost but not filter

[Ravi Gidwani]

Hello:
 I have a basic question:

I am using dismax, Solr 1.4. Let's say I have query where q=sometext  
and it returns me 50 results.but let's say now I want to rank all  
those (say 10) documents higher where field:abc.
Note I just want rank them higher based on the field value and not  
limit the results to these 10 documents.


1) I have tried using bq (something like q=sometext&bq=field:abc ^  
100 ) but this filters/limits the results giving me only those 10  
documents.


2) using qf does not help as I can't boost fields based on field value  
I.e abc


Appreciate any help, comments.

TIA
~Ravi

Sent from my iPhone

Re: Remove the deleted docs from the Solr Index

[Ravi Gidwani]

Lance:
  At times we dont have the freedom make these Database changes.
Currently I am in this situation. Hence the requirement on the DIH.

~Ravi.


On Sat, Jan 2, 2010 at 3:44 PM, Lance Norskog  wrote:

> The other option is to have a 'deleted' column in your table, and have
> the application 'delete' operation set that field. In the DIH you
> query this column with 'deletedPkQuery'.
>
> Or, you can use triggers to maintain a new table with the IDs of
> deleted rows. This will allow you to have a batch job that deletes all
> IDs from this list.
>
> On Tue, Dec 29, 2009 at 10:40 AM, Mohamed Parvez  wrote:
> > Ditto. There should have been an DIH command to re-sync the Index with
> the
> > DB.
> > Right now it looks like one way street form DB to Index.
> >
> >
> > On Tue, Dec 29, 2009 at 3:07 AM, Ravi Gidwani  >wrote:
> >
> >> Hi Shalin:
> >>
> >> >   I get your point about not knowing what has been deleted
> from
> >> the database. So this is what even I am looking for:
> >> >
> >> > 0) A document (id=100) is currently part of solr index.(
> >> > 1) Lets say the application deleted a record with id=100 from
> database.
> >> >
> >> > 2) Now I need to execute some DIH command to say remove document where
> >> id=100. I dont expect the DIH to automatically detect what has been
> deleted,
> >> > but I am looking for a DIH command/special-command to request deletion
> >> from index.
> >> >
> >> > Is that possible ? also as an alternate solution, is it possible to
> build
> >> index using DIH, and use the solr.XmlUpdateRequestHandler request
> handler to
> >> delete/update these one off documents ?
> >> > Is this something you will recommend ?
> >> >
> >> > Thanks,
> >> > ~Ravi Gidwani.
> >> >
> >> > On Tue, Dec 29, 2009 at 3:03 AM, Mohamed Parvez 
> >> wrote:
> >> >
> >> > > I have looked in the that thread earlier. But there is no option
> there
> >> for
> >> >
> >> > > a
> >> > > solution from Solr side.
> >> > >
> >> > > I mean the two more options there are
> >> > > 1] Use database triggers instead of DIH to manage updating the index
> :-
> >> > > This out of question as we cant run 1000 odd triggers every hour to
> >> delete.
> >> >
> >> > >
> >> > > 2] Some sort of ORM use its interception:-
> >> > > This is also out of question as the deletes happens form external
> >> system or
> >> > > directly on the database, not through our application.
> >> > >
> >> > >
> >> >
> >> > > To Say in Short, Solr Should have something thing to keep the index
> >> synced
> >> > > with the database. As of now its one way street, updates rows, on DB
> >> will
> >> > > go
> >> > > to the index. Deleted rows in the DB, will not be deleted from the
> >> Index
> >> >
> >> > >
> >> > >
> >> > How can Solr figure out what has been deleted? Should it go through
> each
> >> row
> >> > and comparing against each doc? Even then some things are not possible
> >> > (think indexed fields). It would be far efficient to just do a
> >> full-import
> >> >
> >> > each time instead.
> >> >
> >> > --
> >> > Regards,
> >> > Shalin Shekhar Mangar.
> >> >
> >> >
> >>
> >
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>

Re: Rules engine and Solr

[Ravi Gidwani]

Avlesh:
   I am currently working on some of kind rules in front
(application side) of our solr instance. These rules are more application
specific and are not general. Like deciding which fields to facet, which
fields to return in response, which fields to highlight, boost value for
each field (both at query time and at index time).
  The approach I have taken is to define a database table which
holds these fields parameters. Which are then interpreted by my application
to decide the query to be sent to Solr. This allow tweaking the Solr fields
on the fly and hence influence the search results.

I will be interested to hear from you about the "Kind" of rules you talk
about and your approach towards it. Are these "Rules" like a regular
expression that when matched with the "user query", execute a specific "solr
query" ?

~Ravi

On Tue, Jan 5, 2010 at 8:25 PM, Avlesh Singh  wrote:

> >
> > Your question appears to be an "XY Problem" ... that is: you are dealing
> > with "X", you are assuming "Y" will help you, and you are asking about
> "Y"
> > without giving more details about the "X" so that we can understand the
> full
> > issue.  Perhaps the best solution doesn't involve "Y" at all? See Also:
> > http://www.perlmonks.org/index.pl?node_id=542341
> >
> Hahaha, thats classic Hoss!
> Thanks for introducing me to the XY problem. Had I known the two
> completely,
> I wouldn't have posted it on the mailing list. And I wasn't looking for a
> "solution" either. Anyways, as I replied back earlier, I'll get back with
> questions once I get more clarity.
>
> Cheers
> Avlesh
>
> On Wed, Jan 6, 2010 at 2:02 AM, Chris Hostetter  >wrote:
>
> >
> > : I am planning to build a rules engine on top search. The rules are
> > database
> > : driven and can't be stored inside solr indexes. These rules would
> > ultimately
> > : two do things -
> > :
> > :1. Change the order of Lucene hits.
> > :2. Add/remove some results to/from the Lucene hits.
> > :
> > : What should be my starting point? Custom search handler?
> >
> > This smells like an XY problem ... can you elaborate on the types of
> > rules/conditions/situations when you want #1 and #2 listed above to
> > happen?
> >
> > http://people.apache.org/~hossman/#xyproblem
> 
> > XY Problem
> >
> > Your question appears to be an "XY Problem" ... that is: you are dealing
> > with "X", you are assuming "Y" will help you, and you are asking about
> "Y"
> > without giving more details about the "X" so that we can understand the
> > full issue.  Perhaps the best solution doesn't involve "Y" at all?
> > See Also: http://www.perlmonks.org/index.pl?node_id=542341
> >
> >
> >
> >
> >
> > -Hoss
> >
> >
>

Search query log using solr

[Ravi Gidwani]

Hi All:
 I am currently using solr 1.4 as the search engine for my
application. I am planning to add a search query log that will capture all
the search queries (and more information like IP,user info,date time,etc).
I understand I can easily do this on the application side capturing all the
search request, logging them in a DB/File before sending them to solr for
execution.
 But I wanted to check with the forum if there was any better
approach OR best practices OR anything that has been added to Solr for such
requirement.

The idea is then to use this search log for statistical as well as improving
the search results.

Please share your experience/ideas.

TIA
~Ravi.

Synonyms from Database

[Ravi Gidwani]

Hi :
 Is there any work done in providing synonyms from a database instead of
synonyms.txt file ? Idea is to have a dictionary in DB that can be enhanced
on the fly in the application. This can then be used at query time to check
for synonyms.

I know I am not putting thoughts to the performance implications of this
approach, but will love to hear about others thoughts.

~Ravi.

Re: Synonyms from Database

[Ravi Gidwani]

Thanks all for your replies.

I guess what I meant by Query time, and as I understand solr  (and I may be
wrong here) I can add synonyms.txt in the query analyser as follows:

  


   
 

By this my understanding is , even if the document (at index time) has a
word "mathematics" and my synonyms.txt file has:

mathematics=>math,maths,

a query for "math" will match "mathematics". Since we have the synonyms.txt
in the query analyzer. So I was curious about the database approach on
similar lines.

I get the point of the performance, and I think that is a big NO NO for this
approach. But the idea was to allow changing the synonyms on the fly (more
like adaptive synonyms) and improve the hits.

I guess the only way (as Otis suggested) is to rewrite the file and reload
configuration (as Peter suggested). This might be a performance hit (rewrite
the file) and reload, but I guess still much better than the reading from DB
?

Thanks again for your comments.

~Ravi.


2010/1/10 Noble Paul നോബിള്‍ नोब्ळ् 

> On Sun, Jan 10, 2010 at 1:04 PM, Otis Gospodnetic
>  wrote:
> > Ravi,
> >
> > I think if your synonyms were in a DB, it would be trivial to
> periodically dump them into a text file Solr expects.  You wouldn't want to
> hit the DB to look up synonyms at query time...
> Why query time. Can it not be done at startup time ?
> >
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
> >
> >
> >
> > - Original Message 
> >> From: Ravi Gidwani 
> >> To: solr-user@lucene.apache.org
> >> Sent: Sat, January 9, 2010 10:20:18 PM
> >> Subject: Synonyms from Database
> >>
> >> Hi :
> >>  Is there any work done in providing synonyms from a database
> instead of
> >> synonyms.txt file ? Idea is to have a dictionary in DB that can be
> enhanced
> >> on the fly in the application. This can then be used at query time to
> check
> >> for synonyms.
> >>
> >> I know I am not putting thoughts to the performance implications of this
> >> approach, but will love to hear about others thoughts.
> >>
> >> ~Ravi.
> >
> >
>
>
>
> --
> -
> Noble Paul | Systems Architect| AOL | http://aol.com
>

Re: Adaptive search?

[Ravi Gidwani]

Shalin:
   Can you point me to pages/resources that talk about this approach
in details ? OR can you provide more details on the schema and the
function(?) used for ranking the documents.

Thanks,
~Ravi.

On Mon, Jan 11, 2010 at 1:00 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Fri, Jan 8, 2010 at 3:41 AM, Otis Gospodnetic <
> otis_gospodne...@yahoo.com
> > wrote:
>
> >
> > - Original Message 
> >
> > > From: Shalin Shekhar Mangar 
> > > To: solr-user@lucene.apache.org
> > > Sent: Wed, December 23, 2009 2:45:21 AM
> > > Subject: Re: Adaptive search?
> > >
> > > On Wed, Dec 23, 2009 at 4:09 AM, Lance Norskog wrote:
> > >
> > > > Nice!
> > > >
> > > > Siddhant: Another problem to watch out for is the feedback problem:
> > > > someone clicks on a link and it automatically becomes more
> > > > interesting, so someone else clicks, and it gets even more
> > > > interesting... So you need some kind of suppression. For example, as
> > > > individual clicks get older, you can push them down. Or you can put a
> > > > cap on the number of clicks used to rank the query.
> > > >
> > > >
> > > We use clicks/views instead of just clicks to avoid this problem.
> >
> > Doesn't a click imply a view?  You click to view.  I must be missing
> > something...
> >
> >
> I was talking about boosting documents using past popularity. So a user
> searches for X and gets 10 results. This view is recorded for each of the
> 10
> documents and added to the index later. If a user clicks on result #2, the
> click is recorded for doc #2 and added to index. We boost using
> clicks/view.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

scenario with FQ parameter

[Ravi Gidwani]

HI all:
  I am trying to figure out a way to do the following:

   &qf=field1^10 field2^20 field^100&fq=*:9+OR+(field1:"xyz")

*Expected Results:
The above should return me documents where 9 appears in any of the fields
(field1,field2 or field3) OR field1 matches "xyz".
*
I know I can use copy field (say 'text') to copy all the fields and then
use:

&qf=field1^10 field2^20 field^100&fq=*text*:9+OR+(field1:"xyz" ^100.0)

but doing so , the boost weights specified in the 'qf' field have no effect
on the score.

I am using solr 1.4 and the searchHandler is dismax.

Is there any way I can achieve the above expected results but still affect
the score with qf parameter ?

Thanks,
~Ravi Gidwani.

Re: Wildcard Search and Filter in Solr

[Ravi Gidwani]

Ashok:
  May be this will help:
http://gravi2.blogspot.com/2009/05/solr-wildcards-and-omitnorms.html

~Ravi

On Tue, Jan 26, 2010 at 9:56 PM, ashokcz  wrote:

>
> Hi just looked at the analysis.jsp and found out what it does during index
> /
> query
>
> Index Analyzer
> Intel
> intel
> intel
> intel
> intel
> intel
>
> Query Analyzer
> Inte*
> Inte*
> inte*
> inte
> inte
> inte
> int
>
> I think somewhere my configuration or my definition of the type "text" is
> wrong.
> This is my configuration .
>
> 
>  
>  
>  
> class="solr.WordDelimiterFilterFactory" generateNumberParts="1"
> generateWordParts="1"/>
>
>  
>  
>  
>   
>
>
> 
>  
>   ignoreCase="true"
> synonyms="synonyms.txt"/>
>  
>   class="solr.WordDelimiterFilterFactory" generateNumberParts="1"
> generateWordParts="1"/>
>  
>  
>  
>  
>
>
>
> I think i am missing some basic configuration for doing wildcard searches .
> but could not figure it out .
> can someone help please
>
>
> Ahmet Arslan wrote:
> >
> >
> >> Hi ,
> >> I m trying to use wildcard keywords in my search term and
> >> filter term . but
> >> i didnt get any results.
> >> Searched a lot but could not find any lead .
> >> Can someone help me in this.
> >> i m using solr 1.2.0 and have few records indexed with
> >> vendorName value as
> >> Intel
> >>
> >> In solr admin interface i m trying to do the search like
> >> this
> >>
> >>
> http://localhost:8983/solr/select?indent=on&version=2.2&q=intel&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=
> >>
> >> and i m getting the result properly
> >>
> >> but when i use q=inte* no records are returned.
> >>
> >> the same is the case for Filter Query on using
> >> &fq=VendorName:"Intel" i get
> >> my results.
> >>
> >> but on using &fq=VendorName:"Inte*" no results are
> >> returned.
> >>
> >> I can guess i doing mistake in few obvious things , but
> >> could not figure it
> >> out ..
> >> Can someone pls help me out :) :)
> >
> > If &q=intel returns documents while q=inte* does not, it means that
> > fieldType of your defaultSearchField is reducing the token intel into
> > something.
> >
> > Can you find out it by using /admin/anaysis.jsp what happens to "Intel
> > intel" at index and query time?
> >
> > What is your defaultSearchField? Is it VendorName?
> >
> > It is expected that &fq=VendorName:Intel returns results while
> > &fq=VendorName:Inte* does not. Because prefix queries are not analyzed.
> >
> >
> > But it is strange that q=inte* does not return anything. Maybe your index
> > analyzer is reducing Intel into int or ıntel?
> >
> > I am not 100% sure but solr 1.2.0  may use default locale in lowercase
> > operation. What is your default locale?
> >
> > It is better to see what happens word Intel using analysis.jsp page.
> >
> >
> >
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Wildcard-Search-and-Filter-in-Solr-tp27306734p27334486.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>