Re: Autocompletion with Solritas

Hi, here is my solution. It has been some time since I last looked at it, but it works fine. :-) $(function() { $("#qterm").autocomplete('/solr/epg/suggest', { extraParams: { 'terms.prefix': function() { return $("#qterm").val(); }

SolrQuery and escaping special characters

Hi, I am using Solr v1.4 and SolrJ on the client side. I am not sure how SolrJ behaves regarding "escaping" special characters [1] in a query string. SolrJ does URL encoding of the query string it sends to Solr. Do I need to escape special characters [1] when I construct a SolrQuery object or n

Re: SolrQuery and escaping special characters

> My understanding is that SolrJ users are supposed to escape > special characters, therefore (b) is the correct way. > If this is the case, what's the best way to escape a query > string which > might contain field names and URIs in their field values? Easiest thing is to use RawQParserPlugin or

custom scoring phrase queries

Hi, I want to know if its posiible to get a higher score in a phrase query when the matching is on the left side of the field. For example: doc1=name:stores peter john doc2=name:peter john stores doc3=name:peter john something if you do a search with name="peter john" the resultset i want to ge

Re: Autocompletion with Solritas

Yup, that's basically what I've done too, here's the script part: I didn't touch the example solrconfig, though putting the params in the request handler is the better way, as you have.

How to LOCK the index-dir for changes from the IndexWriter

Hi, im writing a RequestHandler that manages backups of the index-directory. Yea, i know theres already the Replication-RequestHandler that is also capable of creating backups, but i want the backupprocess to do some more action and not to depend on index-commit-points as the ReplicationHandler

Nested table support ability

I want to use Solr for free text and parameter based search. I need help to validate if Solr can help me in achieving following requirement. Say I have two database tables having one to many relationships. 1. Customer - Customer Id, Customer Name, Profile 2. Role - Role Type, Start date, End Date

finding out why a document is in the result

Hi, We want to tell the user why a document is in the result set. The first solution that came to mind here is using highlighting. However we do not really need to present the highlighted text, we just want to present the user a list of fields where we had matches (for example "address, hobbies

RE: finding out why a document is in the result

Hi Lukas, Have you tried setting the debug mode (debugQuery=on)? It provides very detailed info about the scoring, it might even be too much for a regular user but for us it was very helpful at times. Regards, Tom -Original Message- From: Lukas Kahwe Smith [mailto:m...@pooteeweet.org] S

Re: finding out why a document is in the result

On 18.06.2010, at 12:00, Fornoville, Tom wrote: > Hi Lukas, > > Have you tried setting the debug mode (debugQuery=on)? > It provides very detailed info about the scoring, it might even be too > much for a regular user but for us it was very helpful at times. yeah .. that was the second thing i

Nested table support ability

I want to use Solr for free text and parameter based search. I need help to validate if Solr can help me in achieving following requirement. Say I have two database tables having one to many relationships. 1. Customer - Customer Id, Customer Name, Profile 2. Role - Role Type, Start date, End Date

Re: Autocompletion with Solritas

Looks like a typo below, Chantal, and another comment below too... On Jun 18, 2010, at 3:32 AM, Chantal Ackermann wrote: $(function() { $("#qterm").autocomplete('/solr/epg/suggest', { extraParams: { 'terms.prefix': function() { return $("#qterm").v

Re: federated / meta search

Hi Joe & Markus, sounds good! Maybe I should better add a note on the Wiki page on federated search [1]. Thanks, Sascha [1] http://wiki.apache.org/solr/FederatedSearch Joe Calderon wrote: yes, you can use distributed search across shards with different schemas as long as the query only refe

Re: Field Collapsing SOLR-236

Hi Moazzam, Where did u get the src code from?? I am downloading it from https://svn.apache.org/repos/asf/lucene/solr/branches/branch-1.4 and the latest revision in this location is 955469. so applying the latest patch(dated 17th june 2010) on it still generates errors. Any P

Re: Field Collapsing SOLR-236

Hi Rakhi, The patch is not compatible with 1.4. If you want to work with the trunk. I'll need to get the src from https://svn.apache.org/repos/asf/lucene/dev/trunk/ Martijn On 18 June 2010 13:46, Rakhi Khatwani wrote: > Hi Moazzam, > >                  Where did u get the src code from?? > > I

Data Import Handler Rich Format Documents

I have a database containing Metadata from a content management system. Part of that data includes a URL pointing to the actual published document which can be an HTML file or a PDF, MS Word/Excel/Powerpoint, etc. I'm already indexing the Metadata and that provides a lot of value. The custom

Re: How to LOCK the index-dir for changes from the IndexWriter

Alex, For something like that you may just want to directly use one of the Lucene lock classes to create a lock: http://search-lucene.com/?q=lock&fc_project=Lucene&fc_type=source+code e.g. http://search-lucene.com/c/Lucene:/src/java/org/apache/lucene/store/SingleInstanceLockFactory.java||makeLo

Re: custom scoring phrase queries

Marco, I don't think there is anything in Solr to do that (is there?), but you could do it with some coding if you combined the "regular query" with SpanFirstQuery with bigger boost: http://search-lucene.com/jd/lucene/org/apache/lucene/search/spans/SpanFirstQuery.html Oh, here are some exampl

How to open/update/delete remote index ?

Hi, I am working with solr in production which is configured on remote server . I need to delete some documents from solr index. I know this can be done by curl by calling solr "update" request handler. But i'm looking for GUI tool. I tried luke but luke doesn't open remote index. Do we have a

Re: custom scoring phrase queries

Hi Otis, Finally i construct my own function query that gives more score if the value is at the start of the field. But, its possible to tell solr to use spanFirstQuery without coding. I think i have read that its no possible. Thanks, Marco Martínez Bautista http://www.paradigmatecnologico.com

Re: How to open/update/delete remote index ?

Hi, I don't think there is a GUI for this, other than the Web browser. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: abhay kumar > To: solr-user@lucene.apache.org > Sent: Fri, June

Re: Data Import Handler Rich Format Documents

Tod, You didn't mention Tika, which makes me think you are not aware of it... You could implement a custom Transformer that uses Tika to perform rich doc text extraction, just like ExtractingRequestHandler does it (see http://wiki.apache.org/solr/ExtractingRequestHandler ). Maybe you could even

Re: Nested table support ability

Hello, The short answer is that you need to flatten everything. Your index then has some column-db-like redundancy, but queries become simple and flat. But: See http://blog.sematext.com/2010/06/02/lucene-digest-may-2010-3/ and https://issues.apache.org/jira/browse/LUCENE-2454 in particular

Re: How to open/update/delete remote index ?

What kind of GUI are you looking for here? It'd be easy to hack a "delete this hit" link into the /browse view that now resides on trunk Solr, for example. But I hesitate to add that in at the risk of someone deleting things inadvertently, but perhaps an "admin" mode would be the way to bu

Comma delemitered words shawn in terms like one word.

Hello. In indexing text I have such string John,Mark,Sam. Then I looks at it in TermVectorComponent it looks like this johnmarksam. I am using this type for storing data What filter I need to use to get John Mark Sam as di

MappingCharFilterFactory equivalent for use after tokenizer?

Hi, Is there a token filter which do the same job as MappingCharFilterFactory but after tokenizer, reading the same config file? -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com Training in Europe - www.solrtraining.com

Re: Data Import Handler Rich Format Documents

On 6/18/2010 9:12 AM, Otis Gospodnetic wrote: Tod, You didn't mention Tika, which makes me think you are not aware of it... You could implement a custom Transformer that uses Tika to perform rich doc text extraction, just like ExtractingRequestHandler does it (see http://wiki.apache.org/solr/E

Re: OOM on sorting on dynamic fields

Hello, we are experiencing OOM exceptions in our single core solr instance (on a (huge) amazon EC2 machine). We investigated a lot in the mailing list and through jmap/jhat dump analyzing and the problem resides in the lucene FieldCache that fills the heap and blows up the server. Our index is qui

Re: Peformance tuning

Otis Gospodnetic-2 wrote: > > Smaller merge factor will make things worse - > - Whoops... Ill guess Ill change it from 5 to the default 10 -- View this message in context: http://lucene.472066.n3.nabble.com/Peformance-tuning-tp904540p905726.html Sent from the Solr - User mailing list archiv

Re: Peformance tuning

Otis Gospodnetic-2 wrote: > > You may want to try the RPM tool, it will show you what inside of that > QueryComponent is really slow. > We are already using it :) Where should I be concentrating on? Transaction trace? -- View this message in context: http://lucene.472066.n3.nabble.com/Pefo

customize the search algorithm of solr

Are there any means by which we can customize the search of solr, by plugins etc ?? i have been working on a research based project to implement a new search algorithm for search engines.I wanna know if i can make solr use this algorithm to decide the resultant documents, and still allow me to

Re: Autocompletion with Solritas

Hi Erik, thanks so much for your feedback! > > hightlight? highlight :) ups... Seems that this parameter is false by default, though. At least it never complained. *g* > This is one of the beauties of the VelocityResponseWriter, freeing the > client from having to deal with a Solr data str

Re: Data Import Handler Rich Format Documents

Tod, I don't think DIH can do that, but who knows, let's see what others say. Yes, Nutch uses TIKA, too. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: Tod > To: solr-user@lucene.apa

Re: MappingCharFilterFactory equivalent for use after tokenizer?

> Is there a token filter which do the same job as > MappingCharFilterFactory but after tokenizer, reading the > same config file? No, closest thing can be PatternReplaceFilterFactory. http://lucene.apache.org/solr/api/org/apache/solr/analysis/PatternReplaceFilterFactory.html

Re: How to LOCK the index-dir for changes from the IndexWriter

Thx for the reply. I tried those lock-methods already but still cant it get to work. Heres what i did in my RequestHandler at least. Type "Directory" and all Lock stuff comes from org.apache.lucene.store.* and the copyFiles method is the same as for the replication-RequestHandler. I still nev

Re: Comma delemitered words shawn in terms like one word.

set generateWordParts=1 on wordDelimiter or use PatternTokenizerFactory to split on commas http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.PatternTokenizerFactory you can use the analysis page to see what your filter chains are going to do before you index /admin/analysis.jsp

performance sorting multivalued field

hey there! can someone explain me how impacts to have multivalued fields when sorting? I have read in other threads how does it affect when faceting but couldn't find any info of the impact when sorting Thanks in advance -- View this message in context: http://lucene.472066.n3.nabble.com/perfor

Re: dismax and AND as the default operator

Standard DisMax does not fully support explicit AND/OR. You can prove that by trying to say q=fuel+OR+cell and see that the score stays the same (given mm=100%) It appears that DisMax does SOME intelligent handling of AND/OR/NOT, because it adds the "+" on the AND and a "-" on the NOT. But adding

Re: ranking question

Consider upgrading to the 3.1 branch which gives you true sort by function http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com Training in Europe - www.solrtraining.com On 18. juni 2010, at 01.23, Chris Hostetter wr

Re: finding out why a document is in the result

Are you wanting to do thin on every single user query, and present to the end user which words matched where? In that case debugQuery may be too much, and I would look into creating a custom debugComponent optimized to only outputting the core parts of the "explain" section that you need. If th

Re: performance sorting multivalued field

do you mean sorting facets? or sorting search results? you can't sort search results by a multivalued field - which value would it use? Erik On Jun 18, 2010, at 12:45 PM, Marc Sturlese wrote: hey there! can someone explain me how impacts to have multivalued fields when sorting?

solr indexing takes a long time and is not reponsive to abort command

Hi, I have multi-core solr setup. All cores finished indexing in reasonable time but one. I look at the dataimport info for the one that's hanging. The process is still in busy state but no requests made or rows fetched. The database side just showed the process is waiting for future command an

RE: performance sorting multivalued field

Hi, I have sort on multivalued field with field collapse plugin. Solr always use the first value it gets from the search result when sorting multivalued fileds. I might be wrong but I vaguely remember it's the smallest value. Wen -Original Message- From: Erik Hatcher [mailto:erik.hatc.

Re: how to index the words of a lecture transcript, and the timecodes for each word?

After some more research, it seems that I might be able to use payloads to store the timecodes with the words, though this would appear to require some custom java code. I found this post useful: http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ (thanks, Grant!) O

Re: performance sorting multivalued field

I mean sorting the query results, not facets. I am asking because I have added a multivalued field that has as much 10 values. But 70% of the docs has just 1 or 2 fields of this multiValued field. I am not doing faceting. Since I have added the multiValued field, "java old gen" seems to get full m

Re: Data Import Handler Rich Format Documents

On 6/18/2010 11:24 AM, Otis Gospodnetic wrote: Tod, I don't think DIH can do that, but who knows, let's see what others say. Yes, Nutch uses TIKA, too. Otis Looks like the ExtractingRequestHandler uses Tika as well. I might just use this but I'm wondering if there will be a large performan

Re: Data Import Handler Rich Format Documents

: > I don't think DIH can do that, but who knows, let's see what others say. : Looks like the ExtractingRequestHandler uses Tika as well. I might just use : this but I'm wondering if there will be a large performance difference between : using it to batch content in over rolling my own Transform

Re: Autocompletion with Solritas

On Jun 17, 2010, at 8:34pm, Erik Hatcher wrote: Your wish is my command. Check out trunk, fire up Solr (ant run- example), index example data, hit http://localhost:8983/solr/browse - type in search box. That works - excellent! Now I'm trying to build a distribution from trunk that I can u

Re: SolrQuery and escaping special characters

: I am not sure how SolrJ behaves regarding "escaping" special characters : [1] in a query string. SolrJ encodes your strings for "transport" -- ie: it handles URL escaping if it's sending hte query in a GET URL -- but id doesn't do "query parser escaping" ... mainly because it has no way of kn

Re: Autocompletion with Solritas

On Jun 18, 2010, at 2:56 PM, Ken Krugler wrote: Your wish is my command. Check out trunk, fire up Solr (ant run- example), index example data, hit http://localhost:8983/solr/browse - type in search box. That works - excellent! Now I'm trying to build a distribution from trunk that I can us

Re: MappingCharFilterFactory equivalent for use after tokenizer?

It would be nice to have, because sometimes you want to normalize accents and other characters but want to wait until other filters have run. Especially if those filters are dictionary based and therefore need the original word form. Do you have a clue of how different a CharFilter is from a nor

Re: Data Import Handler Rich Format Documents

On Fri, Jun 18, 2010 at 2:42 PM, Chris Hostetter wrote: > I'm confused ... You're using DIH, and some of your fields are URLs to > documents that you want to parse with Tika? > > Why would you need a custom Transformer? Yeah, I can definitely vouch that DIH can handle this without additional codi

Bizarre TFV output

Hi, I am using a recent nightly build of Solr with no significant schema mods. I index a couple documents and view the TFV's in this query.

Re: Bizarre TFV output

darn evolution... Anyway, I am using a recent nightly build of Solr with no significant schema mods. I index a couple documents and view the TFV's in this query. http://localhost:8080/solr4/select/?q=search&start=0&rows=10&indent=on&qt=tvrh&tv.tf=true&tv=true&fl=text_t&tv.docids=ALL30002 It

Re: Data Import Handler Rich Format Documents

On 6/18/2010 2:42 PM, Chris Hostetter wrote: : > I don't think DIH can do that, but who knows, let's see what others say. : Looks like the ExtractingRequestHandler uses Tika as well. I might just use : this but I'm wondering if there will be a large performance difference between : using it to

Re: MailEntityProcessor class cast exception

: org.apache.solr.handler.dataimport.MailEntityProcessor cannot be cast to : org.apache.solr.handler.dataimport.EntityProcessor ... : I did try to rebuild the solr nightly, but I still receive the same error. : I have all of the required jar's (AFAIK) in my application's lib folder. : :

Re: Bizarre TFV output

: It shows some unwanted and possibly erroneous terms. they may be unwanted, but if it's returning them then they are in your index ... you know the docId and field in question (it's in your URL) so you can look at the source text, paste it into anslysis.jsp and see exactly why those terms are

Re: Bizarre TFV output

Thanks for the explanation Chris. I'll try it but the term " " strikes me as not very legitimate and the source text is just space bounded words so even if its doing what it is supposed to, I'm not sure this term is helpful in the index. I'm kinda new to TFV's though, so much to learn. On Fri,

Re: How to open/update/delete remote index ?

I tried luke via ssh -X ... with success ;-) > Hi, > > I don't think there is a GUI for this, other than the Web browser. > > > Otis > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Lucene ecosystem search :: http://search-lucene.com/ > > > > - Original Message >

Re: Bizarre TFV output

: Thanks for the explanation Chris. I'll try it but the term : " " : : strikes me as not very legitimate and the source text is just space : bounded words so even if its doing what it is supposed to, I'm not sure : this term is helpful in the index. i didnt' say it was helpful -- i just said th

Re: solr indexing takes a long time and is not reponsive to abort command

Did you kill the process or does a reload help afterwards? Did you look into the logs? Are there errors saying sth. of a write-lock? Peter. > Hi, > > I have multi-core solr setup. All cores finished indexing in reasonable time > but one. I look at the dataimport info for the one that's hanging.

Re: Bizarre TFV output

Well stated. You are correct. Here is the field It uses the text field type as its defined in Solr schema. I didn't change it. The input text is a 6 page UTF-8 text document, the relevant line the term seems to be related to. Just a sentence with no specific boundaries. "...perform more quer

solr indexing takes a long time and is not reponsive to abort command

I don’t see my last email showed in the mailing list so I’m sending again. Below is the original email. Hi, I have multi-core solr setup. All cores finished indexing in reasonable time but one. I look at the dataimport info for the one that’s hanging. The process is still in busy state but no

RE: solr indexing takes a long time and is not reponsive to abort command

Sorry if you received duplicate email from me. I checked the log, there is no error in the log and no write-lock message in the log. Where else can I check for more information? Can I see if any query is running? I finally killed the process and run it again. This situation happened couple t

Re: Autocompletion with Solritas

On Jun 18, 2010, at 2:56 PM, Ken Krugler wrote: 3. I tried "ant create-package" from trunk/solr, and got this error near the end: /Users/kenkrugler/svn/lucene/lucene-trunk/solr/common-build.xml: 252: /Users/kenkrugler/svn/lucene/lucene-trunk/solr/contrib/ velocity/src not found. I don't s

Re: solr indexing takes a long time and is not reponsive to abort command

DIH has a UI in Solr Admin that will show you the status of the indexing process. Not sure if you can see that in your Solr or not. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: Y

Re: Data Import Handler Rich Format Documents

I think you can use existing ExtractingRequestHandler to do the job, i.e. add child entity to your DIH metadata http://localhost:8983/solr/update/extract?extractOnly=true&wt=xml&indent=on&stream.url=${metadata.url}"; dataSource="solr"> That's not working example, just basic

Re: Data Import Handler Rich Format Documents

: I think you can use existing ExtractingRequestHandler to do the job, : i.e. add child entity to your DIH metadata why would you do this instead of using the TikaEntityProcessor as i already suggested in my earlier mail? -Hoss

Re: MappingCharFilterFactory equivalent for use after tokenizer?

Indeed. Also, it should be possible to output multiple synonyms based on the mapping: word_with_umlaut should be become word_with_u and word_with_ue as synonyms. (Ok, maybe this example is wrong, but it illustrates the idea.) On Fri, Jun 18, 2010 at 12:17 PM, Jan Høydahl / Cominvent wrote: > It w

Re: Autocompletion with Solritas

Hi Erik, On Jun 17, 2010, at 8:34pm, Erik Hatcher wrote: Your wish is my command. Check out trunk, fire up Solr (ant run- example), index example data, hit http://localhost:8983/solr/browse - type in search box. Just used jQuery's autocomplete plugin and the terms component for now, on t

Re: MappingCharFilterFactory equivalent for use after tokenizer?

On Fri, Jun 18, 2010 at 7:11 PM, Lance Norskog wrote: > Indeed. Also, it should be possible to output multiple synonyms based > on the mapping: word_with_umlaut should be become word_with_u and > word_with_ue as synonyms. (Ok, maybe this example is wrong, but it > illustrates the idea.) > > I don

Re: federated / meta search

Yes, you can do this. You need to have a common system for creating unique ids for the documents. Also, there's an odd problem around relevance. Relevance scoring is based on all of the terms in a field in the whole index, and there is a "statistical fingerprint" of this for an index. With two ind

Re: OOM on sorting on dynamic fields

The Lucene implementation of sorting creates an array of four-byte ints for every document in the index, and another array of the unique values in the field. If the timestamps are 'date' or 'tdate' in the schema, they do not need the second array. You can also sort by a field's with a function que

Re: customize the search algorithm of solr

Solr uses Lucene's algorithms, so lucene-user is the right place for this topic. There is a project to add BM25 (or something like that) to Lucene as an alternate scorer. This may show you how to drop in your own scorer. On Fri, Jun 18, 2010 at 8:14 AM, sarfaraz masood wrote: > > Are there any m

Re: solr indexing takes a long time and is not reponsive to abort command

Does this happen over and over? Does it happen every time? On Fri, Jun 18, 2010 at 1:19 PM, Ya-Wen Hsu wrote: > I don’t see my last email showed in the mailing list so I’m sending again. > Below is the original email. > > Hi, > > I have multi-core solr setup. All cores finished indexing in reaso

Re: SolrQuery and escaping special characters

Thank you for the example, Ahmet! Paolo- what you did in choice 'b' does what you want - it escapes the colon in the URI. But Ahmet's example is a better way because it does not have the 'double-escaping' problem that us old Unix types are so familiar with. On 6/18/10, Chris Hostetter wrote: > >

Re: Autocompletion with Solritas

Have a look at suggest.vm - the "name" field is used in there too. Just those two places, layout.vm and suggest.vm. And I had already added a ## TODO in my local suggest.vm: ## TODO: make this more generic, maybe look at the request terms.fl? or just take the first terms field in the re

Re: federated / meta search

Lance, which project in Solr are you referring to? Thanks, Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: Lance Norskog > To: solr-user@lucene.apache.org > Sent: Fri, June 18, 2010 8:

Re: Can query boosting be used with a custom request handlers?

Hi Chris: Can you please elaborate on how to use the QParser framework? Thanks! -John On Fri, Jun 11, 2010 at 10:56 AM, Chris Hostetter wrote: > > : So it's possible to use both dismax and custom request handler in the > same query? > > it *really* depends on the request handler ... if it

Re: How to open/update/delete remote index ?

HI, @Erik I am looking for a LUKE like command line GUI tool without browser which can open/delete remote index. Regards, Abhay On Sat, Jun 19, 2010 at 1:20 AM, Peter Karich wrote: > I tried luke via > > ssh -X ... > > with success ;-) > > > Hi, > > > > I don't think there is a GUI for this, o