Re: DIH delta import - last modified date

While invoking the delta-import you may, pass the value as a request parameter. That value can be used in the query as ${dih.request.xyz} where as xyz is the request parameter name On Wed, Jan 20, 2010 at 1:15 AM, Yao Ge wrote: > > I am struggling with the concept of delta import in DIH. Accordi

Re: Data storage, and textual analysis

On Tue, 19 Jan 2010 12:02:27 -0800 (PST) Otis Gospodnetic wrote: > Gora, > > What you are seeing are the *stored* values, which are the > original, unchanged field values. Analysis is applied to text for > *indexing* purposes. [...] Ah, of course. Seems obvious now, and I had misread this messa

Re: complex query

Thanks for your attention. I think I was just a little over-worried about search performance,. Fortunately, solr works pretty nice until present, it's fast enough for me. 2010/1/20 Chris Hostetter > > : I have considered building lucene index like: > : Document: { title, content, author, frien

Re: DIH delta import - last modified date

I run DIH in the same machine with database, hence avoided the problem. 2010/1/20 Yao Ge > > I am struggling with the concept of delta import in DIH. According the to > documentation, the delta import will automatically record the last index > time stamp and make it available to use for the delt

SOLR Performance Tuning: Fuzzy Searches, Distance, BK-Tree

Hi, I am wondering: will SOLR or Lucene use caches for fuzzy searches? I mean per-term caching or something, internal to Lucene, or may be SOLR (SOLR may use own query parser)... Anyway, I implemented BK-Tree and playing with it right now, I altered FuzzyTermEnum class of Lucene... http://en.wik

Re: TermsComponent, multiple fields, total count

On Jan 19, 2010, at 3:55 PM, Otis Gospodnetic wrote: a search "quality" meter. As in indicate the total number of matches (doesnt need to be accurate, just a ballpark figure especially if there are a lot of matches) As in, you want each suggestion include the number of documents it would

Re: solr blocking on commit

A couple of follow up questions: - What type of garbage collector is in use? - How often are you optimizing the index? - In solrconfig.xml what is the setting for ? - Right before and after you see this pause, check the output of http://:/solr/admin/system, specifically the output of and send thi

Re: solr blocking on commit

I'll play with the GC settings and watch memory usage (I've done a little bit of this already), but I have a sense that this isn't the problem. I should also note that in order to create the really long pauses I need to post xml files full of documents that haven't been added in a long time / ever

Re: Design Question - Dynamic Field Names (*)

First Thanks for the response. Yes, mostly likely we want to optimize to one index file. I think it is possible, coming from the RDBMS world we might be over complicating the solution. *Requirement:* - We are indexing CSV files and generating field names dynamically from the "header" line. User

Re: Extracting URLs while indexing

Do you mean you want the URLs to be extracted on the client? If so, no. Filters/analyzers reside on the server, not the client. You'll have to do it with custom code Erick On Tue, Jan 19, 2010 at 5:48 PM, Bogdan Vatkov wrote: > Hi, > > I want to extract URLs (http://..., as well as file://..

Re: Problem indexing files

I'm not sure if i fully understand what you mean, but is it possible that "contid" is your uniqueKey field? ... if so you are probably adding two documents, both with the same ID, so the second overwrites the first. If i'm mistaken, then could you elaborate a bit more -- provide your configs/s

Re: Unstemming after solr.PorterStemFilterFactory

This is completely unnecessary. Fields can be both indexed and stored, and the operations are orthogonal. That is, when you specify that a field is indexed, it is run through an analyzer and the *tokens* are indexed, after any stemming, casing, etc. Stored means that the original value, before an

Re: termsComponent and filter queries

You may be able to use faceting for this. Use facet.method=enum - it will be more efficient for this specific use. The main problem is that you can't specify a start term for faceting though (you can only use numeric offset / limit into the list). To do more will require either adding some terms

Re: solr blocking on commit

On Tue, Jan 19, 2010 at 5:57 PM, Steve Conover wrote: > I'm using latest solr 1.4 with java 1.6 on linux.  I have a 3M > document index that's 10+GB.  We currently give solr 12GB of ram to > play in and our machine has 32GB total. > > We're seeing a problem where solr blocks during commit - it won

solr blocking on commit

I'm using latest solr 1.4 with java 1.6 on linux. I have a 3M document index that's 10+GB. We currently give solr 12GB of ram to play in and our machine has 32GB total. We're seeing a problem where solr blocks during commit - it won't server /select requests - in some cases for more than 15-30 s

Extracting URLs while indexing

Hi, I want to extract URLs (http://..., as well as file://... or even //.) while pushing documents into Solr. Is it possible with the Filters/Analyzers available nowadays? I looked into the doc but could not find anything related to it. Best regards, Bogdan

Re: Unstemming after solr.PorterStemFilterFactory

I am using fields like: which contain multi-line text, not just single strings, what does "stored values" mean? I am relatively new to Solr I solved my issue by copy/pasting and enhancing the SnowballPorterFilterFactory class by creating SnowballPorterWithUnstemLowerCaseFilterFactory I added lo

Re: Unstemming after solr.PorterStemFilterFactory

Bogdan, You can get them from stored values of your fields, if you are storing them. Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch - Original Message > From: Bogdan Vatkov > To: solr-user@lucene.apache.org > Sent: Tue, January 19, 2010 5:28:51 PM > Subject: Unstem

Re: Number of values limitation in multivalued field

As far as I know, there's no underlying difference between adding all 42K tokens one at a time (mutlivalued) or all at once (singlevalued), with one rather technical difference: If you've changed the positionIncrementGap to something other than "1" in your schema, then the token offsets delta betwe

Unstemming after solr.PorterStemFilterFactory

Hi, I am indexing with the solr.PorterStemFilterFactory included but then I need to access the unstemmed versions of the terms, what would be the easiest way to get the unstemmed version? Thanks in advance. Best regards, Bogdan -- Best regards, Bogdan

Re: Restricting Facet to FilterQuery in combination with mincount

: Now, I was wondering whether it is possible to find that out. It would allow : to show 0 counts of values that are produced by the query (q), and at the same : time exclude all facet values that are already excluded by the filter query. : : Applying facetting to a subset (subselect / filterset)

Large Query Strings and performance

I am using Solr 1.4 with large query strings with 20+ terms and faceting on a single multi-valued field in a 1 million record system. I am using Solr to categorize text, that why the query strings are big. The performance get's worse the more search terms are used. Is there any way I can improv

Re: complex query

: I have considered building lucene index like: : Document: { title, content, author, friends } That seems like the right appraoch. : Thus, author and friends are two seperate fields. so I can boost them : seperately. : The problem is, if a document's author is the logged-in user, it's uncessar

Re: TermsComponent, multiple fields, total count

On 19.01.2010, at 21:55, Otis Gospodnetic wrote: > Hi Lukas, > > > - Original Message > >> From: Lukas Kahwe Smith > >> I want to use TermsComponent for both auto complete suggestions but also >> showing > > Is TermsComponent really that good for AutoComplete? > Have a look at ht

Re: Google Commerce Search

>From the Solr Wiki about Query-time synonym expansion "...synonyms containing multiple words..The recommended approach for dealing with synonyms like this, is to expand the synonym when indexing. This is because there are two potential issues that can arise at query time" http://wiki.apache.org/so

Please help: Failing tests

I'm tring to run the unit tests from Eclipse. Almost half the tests are failing, and I don't know what I'm doing wrong. This is what I've done: 1. Checked out the code outside Eclipse's workspace 2. File > New > Project > Java project. 3. Create project from existing source" 4. Five compiler er

Re: TermsComponent, multiple fields, total count

Hi Lukas, - Original Message > From: Lukas Kahwe Smith > I want to use TermsComponent for both auto complete suggestions but also > showing Is TermsComponent really that good for AutoComplete? Have a look at http://www.sematext.com/demo/ac/index.html - doesn't use TC. > a search

Re: Google Commerce Search

While Solr is functionally platform independent, I have seen much better performance on Linux than Windows under high load (related to SOLR-465). MitchK wrote: > > As you know, Solr is fully written in Java and Java is still > plattform-independent. ;) > Learn more about Solr on http://www.luc

RE: Rounding dates on sort and filter

Good point. So it doesn't sound like there's a way to do this without adding a new field or reindexing. Thanks anyway. - Charlie -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Tuesday, January 19, 2010 2:04 PM To: solr-user@lucene.apache.org Subject

Re: Contributors - Solr in Action Case Studies

Hi Gora, Thanks, this sounds interesting, as I don't think we explicitly cover phonetic searches and talking explicitly about languages other than English will be useful to some readers. Let's take further discussion off-line. Thanks, Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene

Re: Google Commerce Search

And what I recommended to my Fortune 1 client;) ...actually, just one correction: > Secondly you should know that, you can not update or push Synonyms at run > time. You can, if you are okay with query-time synonym expansion. The new replication can be used to replicate not only indices,

Re: Number of values limitation in multivalued field

Thanks Erik, I was not aware of the maxFieldLength. * Query performance compared to storing data by zipcode. Schema to accommodate this would have 42K * 60 documents approx. Also to consider duplicate document data with varying zipcode in the index. Hope this makes sense. We however wanted to un

termsComponent and filter queries

I have a field that has millions of values, and I need to get "the next X values" in alpha order. The terms component works fabulously for this. Here is a cooked up example of the terms a b f q r rr rrr y z zzz So if I ask for the 3 terms after "r", I get "rr", "rrr" and "y". But now I'd

Re: Rounding dates on sort and filter

Charlie, Query-time terms/tokens need to match what's in your index, and my guess is that if you just altered query-time date field analysis, you'd get a mismatch. Easy enough to check through Solr Admin Analysis page. Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch -

Re: Data storage, and textual analysis

Gora, What you are seeing are the *stored* values, which are the original, unchanged field values. Analysis is applied to text for *indexing* purposes. Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch - Original Message > From: Gora Mohanty > To: solr-user@lucene.a

Re: Number of values limitation in multivalued field

You should be able to do this no problem. Do be aware of the maxfieldlength though, it defaults to 10,000 tokens but you can change it in your schema.xml. Beware, there are TWO instances of this in the schema file. See: http://search.lucidimagination.com/search/document/30616a061f8c4bf6/solr_ignori

Re: Tokenization and wild card search

Listen to Ahmet, ignore me. I missed "catenatewords=1", which should produce the tokens exactly as Ahmet said. So standard wildcarding should work it seems to me Sorry 'bout that Erick On Tue, Jan 19, 2010 at 12:01 PM, Ahmet Arslan wrote: > > > I want the following searches to work: >

DIH delta import - last modified date

I am struggling with the concept of delta import in DIH. According the to documentation, the delta import will automatically record the last index time stamp and make it available to use for the delta query. However in many case when the last_modified date time stamp in the database lag behind the

Re: Design Question - Dynamic Field Names (*)

On Sat, Jan 16, 2010 at 3:33 AM, Kumaravel Kandasami < kumaravel.kandas...@gmail.com> wrote: > Need to your suggestion in best designing the following requirement. > > - We have two indexes. > Index 1: "name_index", > Fields: > "id" - indexed, not stored > "field_name" - indexed, stored. > > Ind

Re: build path

If you are going to run the unit tests in Eclipse, then for the given JUnit run configuration, add the -Dsolr.solr.home= to the VM arguments section of the run configuration for the given test. On Tue, Jan 19, 2010 at 12:34 AM, Wangsheng Mei wrote: > I this it is. > > solr has a default servlet

Re: Replication Condition (Swapping indexers)

On Thu, Jan 14, 2010 at 6:30 AM, wojtekpia wrote: > > I have a deployment with 2 indexers (2 cores in a single servlet > container), > and a farm of searchers that replicate from one of the indexers. Once in a > while I need to re-index all my data, so I do that on my second indexer > (while my o

Data storage, and textual analysis

Hi, Another simple query. I have set up a field to hold phonetic equivalents, with the relevant part of schema.xml looking like: Here, com.srijan.search.solr.analysis.AspellFilterFactory is a custom filter that provides a phonetic soundslike equivalent for Indian languages transliterated

Rounding dates on sort and filter

I've got a legacy date field that I'd like to round for sorting and filtering. Right now, the index is large enough that sorting or filtering on a date field takes 10-20 seconds (unless it's cached). I know this is because the date field's precision is down to the millisecond, and I don't really ne

Comparison of Solr with Sharepoint Search

Has anyone done a functionality comparison of Solr with Sharepoint/Fast Search? If yes, kindly share a few details here. Thanks for your help in advance! Regards, Abhishek.

Number of values limitation in multivalued field

* Can we define a field in our schema as multiValued (with stored=false, indexed=true) that will hold upto 42K zipcode values associated to each document? * Is there any query time performance impact with this. * Is there any impact on index time. The number of documents we are talking here is not

Re: Updating a single field in a Solr document

Here's the corresponding issue: https://issues.apache.org/jira/browse/SOLR-139 On Tue, Jan 19, 2010 at 1:36 PM, Raghuveer Kancherla < raghuveer.kanche...@aplopio.com> wrote: > Is this feature planned in any of the future releases. I ask because it > will > help me plan my system architecture acco

RE: Tokenization and wild card search

> I want the following searches to work: > >   MyField:SDD_Expedition_PCB > > This should match the word "SDD_Expedition_PCB" only, and > not matching individual words such as "SDD" or "Expedition", > or "PCB". > > And the following search: > >   MyField:SDD_Expedition* > > Should match a

Re: Tokenization and wild card search

What I suspect would work is phrase queries with no slop. Unfortunately, to get this to work right you need wildcards inside phrases, which is NOT supported out of the box. However, see SOLR 1604 for patches that address this... http://issues.apache.org/jira/browse/SOLR-1604 HTH Erick P.S. Are

Re: Updating a single field in a Solr document

Shalin Shekhar Mangar wrote: On Mon, Jan 18, 2010 at 5:11 PM, Raghuveer Kancherla < raghuveer.kanche...@aplopio.com> wrote: Hi, I have 2 fields one with captures the category of the documents and an other which is a pre processed text of the document. Text of the document is fairly large. Th

Re: Updating a single field in a Solr document

Is this feature planned in any of the future releases. I ask because it will help me plan my system architecture accordingly. Thanks, Raghu On Tue, Jan 19, 2010 at 7:28 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > On Mon, Jan 18, 2010 at 5:11 PM, Raghuveer Kancherla < > raghuve

Re: Field collapsing patch error

this has come up before, my suggestions would be to use the 12/24 patch with trunk revision 892336 http://www.lucidimagination.com/search/document/797549d29e1810d9/solr_1_4_field_collapsing_what_are_the_steps_for_applying_the_solr_236_patch 2010/1/19 Licinio Fernández Maurelo : > Hi folks, > > i'

Re: Tokenization and wild card search

You are correct, the way I'm using tokenization is my issue. It's too late to re-index now, this is why I'm looking for a search syntax that will to make the search work. I have tried various search syntax with no luck. Is there no search syntax to make this work without re-indexing?! --

Re: Urgent: SOLR Indexing missing tokens

Hi I was doing the same mistake mentioned in this URL. http://search.lucidimagination.com/search/document/30616a061f8c4bf6/solr_ignoring_maxfieldlength maxFieldLength is there at 2 places. earlier changed at the indexDefaults now changed at mainIndex section also. it worked. Thanks Mark & Erick.

Re: Urgent: SOLR Indexing missing tokens

Hi Erik, Yes, i deleted the index and re-indexed after increasing the value (i have restarted tomcat as well) but still no luck. but i was just wondering the field that i am trying to index has the complete document text in it as i am storing that. but not getting the complete terms/tokens into t

Re: Urgent: SOLR Indexing missing tokens

Can anyone suggest/guide me on this. Best Regards, Kranti K K Parisa 2010/1/19 Kranti™ K K Parisa > Hi Mark, > > I changed the value to 1,000,000,000 to just test my luck. > > But unfortunately I am still not getting the index for all Token. > > Please suggest. > > Best Regards, > Kranti K K

Field collapsing patch error

Hi folks, i've downloaded solr release 1.4 and tried to apply latest field collapsing patchi've found . Found errors : d...@backend05:~/workspace/solr-release-1.4.0$ patch -p0 -i SOLR-236.patch patching file src/test/test

Re: Urgent: SOLR Indexing missing tokens

Did you reindex the documents you examined? That limit is applied when you index. Try searching the user list for maxfieldlength, this topic has been discussed many times and you should find a solution. HTH Erick 2010/1/19 Kranti™ K K Parisa > Can anyone suggest/guide me on this. > > Best Rega

Re: Tokenization and wild card search

I'm pretty sure you're going to be disappointed about the re-indexing part. I'm pretty sure that WordDelimiterFilterFactory is tokenizing your input in ways you don't expect, making your use-case hard to accomplish. It's basically splitting your input on all non-alpha characters, so you're indexi

Re: Restricting Facet to FilterQuery in combination with mincount

Hi Shalin, thanks for taking your time (reading it twice!). Rephrasing the question: (suppose mincount=0 and facet.limit > all possible facet values) Currently, the facet results include ALL values for that facet field. Say I have a field color and when I look at the statistics (LUKE), I can s

Re: Urgent: SOLR Indexing missing tokens

Hi Mark, I changed the value to 1,000,000,000 to just test my luck. But unfortunately I am still not getting the index for all Token. Please suggest. Best Regards, Kranti K K Parisa 2010/1/19 Kranti™ K K Parisa > Hi Mark, > > As you see my config file contains the value as 10,000 > 1 >

Re: Urgent: SOLR Indexing missing tokens

Hi Mark, As you see my config file contains the value as 10,000 1 But when I check thru Lukeall jar file I can see the Term count around 3,000. Please suggest. Best Regards, Kranti K K Parisa 2010/1/19 Mark Miller > It limits the number of tokens that will be indexed. > > Kranti™ K K P

Re: Urgent: SOLR Indexing missing tokens

It limits the number of tokens that will be indexed. Kranti™ K K Parisa wrote: > Hi Mark, > > I really appreciate the quick reply. > > here is what I have in the config xml > > 32 > 2147483647 > * 1* > 1000 > 1 > > Does this matter with Tokens?? Because the field I am using

Re: Urgent: SOLR Indexing missing tokens

Hi Mark, I really appreciate the quick reply. here is what I have in the config xml 32 2147483647 * 1* 1000 1 Does this matter with Tokens?? Because the field I am using is having the full content of the file ( I checked that using Lukeall jar file), how ever Tokens are n

Re: Urgent: SOLR Indexing missing tokens

Kranti™ K K Parisa wrote: > Hi All, > > I have a problem using SOLR indexing. I am trying to index 96 pages PDF file > (using PDFBox for extracting the file contents into String). But > surprisingly SOLR Indexing is not done for the full document. Means I can't > get all the token how ever the fiel

Urgent: SOLR Indexing missing tokens

Hi All, I have a problem using SOLR indexing. I am trying to index 96 pages PDF file (using PDFBox for extracting the file contents into String). But surprisingly SOLR Indexing is not done for the full document. Means I can't get all the token how ever the field contains the full text of the PDF a

RE: Tokenization and wild card search

I want the following searches to work: MyField:SDD_Expedition_PCB This should match the word "SDD_Expedition_PCB" only, and not matching individual words such as "SDD" or "Expedition", or "PCB". And the following search: MyField:SDD_Expedition* Should match any word starting with "S

Re: Best wasy to solve Parent-Child relationship without Denormalizing?

Hi, SIREn [1] could help you to solve this task (look at the different indexing examples). But actually, only a Lucene extension is available. If you want to use it into Solr, you will have to implement your own Solr plugin (which should require only a limited amount of work). [1] http://sir

RE: Interesting OutOfMemoryError on a 170M index

Thanks. Turns out the problem was related to throughput - I wasn't getting enough docs indexed per second and an internal queue in a vendor library was growing without bound. Using the StreamingUpdateSolrServer fixed that. -Nick -Original Message- From: Shalin Shekhar Mangar [mailto:shal

Re: Updating a single field in a Solr document

On Mon, Jan 18, 2010 at 5:11 PM, Raghuveer Kancherla < raghuveer.kanche...@aplopio.com> wrote: > Hi, > I have 2 fields one with captures the category of the documents and an > other > which is a pre processed text of the document. Text of the document is > fairly large. > The category of the docum

Best wasy to solve Parent-Child relationship without Denormalizing?

Hi, Here is an extract of my data schema in which my user should be able to issue the following search: company_description:pharmaceutical AND product_description:cosmetic [Company profile] Company name Company url Company description Company user rating [Product profile] ..

Re: How to backup / dump solr database

yes, it is possible. and copying the index is exactly how to go about it. what didn't work exactly? be sure that the index directory goes under data/ and looks just like your production environment. Erik On Jan 19, 2010, at 8:08 AM, jmf wrote: Hi, I'm using solr with the Plone

How to backup / dump solr database

Hi, I'm using solr with the Plone CMS. I have just following some tutorials, and I would like to 'dump' the solr database on production server and make it run on my developement environement. Both are linux. So first the question is : is it possible ? Next how could I do this. I have try to si

Re: Tokenization and wild card search

> I have an issue and I'm not sure how to address it, so I > hope someone can help me. > > I have the following text in one of my fields: > "ABC_Expedition_ERROR".   When I search on it > like: "MyField:SDD_Expedition_PCB" (without quotes) it will > fail to find me only this word “ABC_Expedition_

Re: filter query parsing problem

Ahmet, Thanks so much for the help. I will give it a shot. John On Mon, Jan 18, 2010 at 4:40 PM, Ahmet Arslan wrote: >> I am submitting a query and it seems >> to be parsing incorrectly.  Here >> is the query with the debug output.  Any ideas what >> the problem is: >> >> >>   >>     ((VLog:

Re: Interesting OutOfMemoryError on a 170M index

On Thu, Jan 14, 2010 at 4:04 AM, Minutello, Nick < nick.minute...@credit-suisse.com> wrote: > Agreed, commit every second. > > Assuming I understand what you're saying correctly: > There shouldn't be any index readers - as at this point, just writing to > the index. > Did I understand correctly wh

Re: Fastest way to use solrj

2010/1/19 Noble Paul നോബിള്‍ नोब्ळ् : > 2010/1/19 Tim Terlegård : >>    server = new CommonsHttpSolrServer("http://localhost:8983/solr";) >>    server.setRequestWriter(new BinaryRequestWriter()) >>    request = new UpdateRequest() >>    request.setAction(UpdateRequest.ACTION.COMMIT, true, true); >

Re: Does specifying a smaller number of rows in search improve efficiency?

On Jan 18, 2010, at 11:17 AM, Yonik Seeley wrote: > On Mon, Jan 18, 2010 at 8:57 AM, Erick Erickson > wrote: >> Nope. The problem is that SOLR needs to create a ranked >> list. It has to search the entire corpus every time. There's >> always the possibility that the very last document examined

Re: Restricting Facet to FilterQuery in combination with mincount

On Wed, Jan 13, 2010 at 4:55 PM, Chantal Ackermann < chantal.ackerm...@btelligent.de> wrote: > Hi all, > > is it possible to restrict the returned facets to only those that apply to > the filter query but still use mincount=0? Keeping those that have a count > of 0 but apply to the filter, and at

Re: build path

I this it is. solr has a default servlet container "jetty" with the downloaded package under folder "example" . but I use tomcat a lot, so I deployed solr on tomcat using solr.war. I don't know why solr will use jetty as default. 2010/1/19 Siv Anette Fjellkårstad > I apologize for the newbie q

Re: Fastest way to use solrj

2010/1/19 Tim Terlegård : > There are a few ways to use solrj. I just learned that I can use the > javabin format to get some performance gain. But when I try the binary > format nothing is added to the index. This is how I try to use this: > >    server = new CommonsHttpSolrServer("http://localhos