Integration Solr with Interwoven TeamSite

Hi all, I am using solr for searching and working fine with independent tomcat server. I want to integrate solr with Interwoven TeamSite. But No Idea of Integration solr with teamsite.. please help.

Re: DIH field options

The wiki page has most of the info you need *http://wiki*.apache.org/*solr*/DataImportHandler To use multi-value fields, your schema.xml must define it with multiValued="true" On 3/11/10 10:58 PM, blargy wrote: How can you simply add a static value like? How does one add a static multi-val

DIH field options

How can you simply add a static value like?  How does one add a static multi-value field? Is there any documentation on all the options for the field tag in data-config.xml? Thanks for the help -- View this message in context: http://old.nabble.com/DIH-field-options-tp27873996p27873996.html

Best Practices for Runtime Index Updates

Hi, What are the Best Practices for Runtime Index Updates? Means we have index and user may add some data like tags, notes..etc to each solr document. during this scenario how quick we could update the index, and how quick we could show the updates to the end user on UI? Best Regards, Kranti K K

local solr geo_distance

Hi I'm getting geo_distance as str eventhough I'm define the field as tdouble. my search looks like /solr/select?&qt=geo&lat=xx.xx&long=yy.yy&q=*&radius=10 Is there anyway i can get is as double instead of str -- View this message in context: http://old.nabble.com/local-solr-geo_distance-tp27873

How to get Facet results only on a range of search results documents

Hi, I would like to return Facet results only on the range of search results (say 1-100) not on the whole set of search results. Any idea how can I do it? Here is the reason I want to do it: My document set is quite huge: About 100 Million documents. When a query is run, the returned results are

Re: Architectural help

What is DIH? I feel like I'm saying, "Duh . . .", sorry. Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Thu, 3/11/10, Constantijn Visinescu wrote: > From: Constant

Re: embedded server / servlet container

How would that work in a PHP environment. I've already come to my own conclusion that using the JSON output would be safer (definitely) and faster (probably) than using PHP output and eval(); So what to do when it gets to the PHP process is no problem. But it's setting up an embedded server on

Re: Cleaning up dirty OCR

> > I don't deal with a lot of multi-lingual stuff, but my understanding is > that this sort of thing gets a lot easier if you can partition your docs > by language -- and even if you can't, doing some langauge detection on the > (dirty) OCRed text to get a language guess (and then partition by lan

Re: Cleaning up dirty OCR

: Interesting. I wonder though if we have 4 million English documents and 250 : in Urdu, if the Urdu words would score badly when compared to ngram : statistics for the entire corpus. Well it doesn't have to be a strict ratio cutoff .. you could look at the average frequency of all character

Re: How to edit / compile the SOLR source code

Erik, That was a wonderful explanation, I hope many folks in this forum will be benefited from the explanation you have given here. Actually I Googled and found the solution when you had earlier mentioned that I can do a leading wildcard without hacking the code. I found out the patch that ha

Re: How to edit / compile the SOLR source code

Leaving aside some historical reasons, the root of the issue is that any search has to identify all the terms in a field that satisfy it. Let's take a normal non-leading wildcard case first. Finding all the terms like 'some*' will have to deal with many fewer terms than 's*'. Just dealing with tha

Re: Solr Performance Issues

I dont mean to turn this into a sales pitch, but there is a tool for Java app performance management that you may find helpful. Its called New Relic (www.newrelic.com) and the tool can be installed in 2 minutes. It can give you very deep visibility inside Solr and other Java apps. (Full disclosure

Re: field length normalization

The fieldNorm is computed like this: fieldNorm = lengthNorm * documentBoost * documentFieldBoosts and the lengthNorm is: lengthNorm = 1/(numTermsInField)**.5 [note that the value is encoded as a single byte, so there is some precision loss] So the values are not pre-set for the lengthNorm, but

Re: How to edit / compile the SOLR source code

Eric, Thanks a lot for your reply. I was able to successfully hack the query parser and enabled the leading wild card search. As of today I hacked the code for this reason only, I am not sure how to make the leading wild card search to work without hacking the code and this type of search is th

Re: Cleaning up dirty OCR

We've been thinking about running some kind of a classifier against each book to select books with a high percentage of dirty OCR for some kind of special processing. Haven't quite figured out a multilingual feature set yet other than the punctuation/alphanumeric and character block ideas mention

Re: Cleaning up dirty OCR

Interesting. I wonder though if we have 4 million English documents and 250 in Urdu, if the Urdu words would score badly when compared to ngram statistics for the entire corpus. hossman wrote: > > > > Since you are dealing with multiple langugaes, and multiple varient usages > of langauge

RE: Scaling indexes with high document count

Hi, Thanks for your reply (an apologies for the orig msg being ent multiple times to the list - googlemail problems). I actually meant to put 'maxBufferredDocs'. I admit I'm not that familar with this parameter, but as I understand it, it is the number of documents that are held in ram

Re: Cleaning up dirty OCR

On Mar 11, 2010, at 1:34 PM, Chris Hostetter wrote: > I wonder if one way to try and generalize > the idea of "unlikely" letter combinations into a math problem (instead of > grammer/spelling problem) would be to score all the hapax legomenon > words in your index Hmm, how about a classifier?

Re: Cleaning up dirty OCR

: We can probably implement your suggestion about runs of punctuation and : unlikely mixes of alpha/numeric/punctuation. I'm also thinking about : looking for unlikely mixes of unicode character blocks. For example some of : the CJK material ends up with Cyrillic characters. (except we would hav

Re: Cleaning up dirty OCR

On Thu, Mar 11, 2010 at 4:14 PM, Tom Burton-West wrote: > > Thanks Simon, > > We can probably implement your suggestion about runs of punctuation and > unlikely mixes of alpha/numeric/punctuation.  I'm also thinking about > looking for unlikely mixes of unicode character blocks.  For example some

Re: Cleaning up dirty OCR

Thanks Simon, We can probably implement your suggestion about runs of punctuation and unlikely mixes of alpha/numeric/punctuation. I'm also thinking about looking for unlikely mixes of unicode character blocks. For example some of the CJK material ends up with Cyrillic characters. (except we wo

Re: Cleaning up dirty OCR

On Thu, Mar 11, 2010 at 3:37 PM, Burton-West, Tom wrote: > Thanks Robert, > > I've been thinking about this since you suggested it on another thread.  One > problem is that it would also remove real words. Apparently 40-60% of the > words in large corpora occur only once > (http://en.wikipedia.

Re: dismax and WordDelimiterFilterFactory with PreserveOriginal = 1

Kind of a shot in the dark here, but your parameters for index and query on WordDelimiterFilterFactory are different, especially suspicious is catenateWords. You could test this by looking in your index with the SOLR admin page and/or Luke to see what your actual terms are. And don't forget y

RE: Cleaning up dirty OCR

Thanks Robert, I've been thinking about this since you suggested it on another thread. One problem is that it would also remove real words. Apparently 40-60% of the words in large corpora occur only once (http://en.wikipedia.org/wiki/Hapax_legomenon.) There are a couple of use cases where r

Re: How to edit / compile the SOLR source code

See Trey's comment, but before you go there. What about SOLR's wildcard searching capabilities aren't working for you now? There are a couple of tricks for making leading wildcard searches work quickly, but this is a solved problem. Although whether the existing solutions work in your situatio

RE: dismax and WordDelimiterFilterFactory with PreserveOriginal = 1

Yonik, thank you for your reply. When I don't use PreserveOriginal = 1 for WordDelimiterFilterFactory, the query "ain't" is parsed as "ain t" and no match is found in this case too. If I remove ' from the query, then I can get results. I used the analysis tool and see the term ain't is processed

Re: Solr Performance Issues

How much of your memory are you allocating to the JVM and how much are you leaving free? If you don't leave enough free memory for the OS, the OS won't have a large enough disk cache, and you will be hitting the disk for lots of queries. You might want to monitor your Disk I/O using iostat an

How to sort facet values lexicographically in descending order?

Hi folks, is there a way to sort facet values lexicographically in descending order? If it's not possible right now, are there any feasible workarounds to accomplish this? Note: I've seen issue SOLR-1672, but it does not solve my problem since it deals with facet counts only. Best, Sascha

Re: dismax and WordDelimiterFilterFactory with PreserveOriginal = 1

On Thu, Mar 11, 2010 at 1:07 PM, Ya-Wen Hsu wrote: > Hi all, > > I'm facing the same issue as previous post here: > http://www.mail-archive.com/solr-user@lucene.apache.org/msg19511.html. Since > no one answers this post, I thought I'll ask again. In my case, I use below > setting for index > g

Re: issue with delete index

On Thu, Mar 11, 2010 at 12:22 PM, muneeb wrote: > I have made some changes to my schema, including setting of omitNorms to > false for a few fields. I am using Solr1.4 with SolrJ client. I deleted my > index using the client: > > solrserver.deleteByQuery("*:*"); > solrserver.optimize(); Solr impl

Multi valued fields

Hi All, I'd like to know if it is possible to do the following on a multi-value field: Given the following data: document A: field1 = [ A B C D] document B: field 1 = [A B] document C: field 1 = [A] Can I build a query such as : -field: A which will return all documents that d

Re: Profiling Solr

On Thu, Mar 11, 2010 at 1:11 PM, Jean-Sebastien Vachon wrote: > Hi, > > I'm trying to identify the bottleneck to get acceptable performance of a > single shard containing 4.7 millions of documents using my own machine (Mac > Pro - Quad Core with 8Gb of RAM with 4Gb allocated to the JVM). > > I t

Profiling Solr

Hi, I'm trying to identify the bottleneck to get acceptable performance of a single shard containing 4.7 millions of documents using my own machine (Mac Pro - Quad Core with 8Gb of RAM with 4Gb allocated to the JVM). I tried using YourKit but I don't get anything about Solr classes. I'm new to

dismax and WordDelimiterFilterFactory with PreserveOriginal = 1

Hi all, I'm facing the same issue as previous post here: http://www.mail-archive.com/solr-user@lucene.apache.org/msg19511.html. Since no one answers this post, I thought I'll ask again. In my case, I use below setting for index and for query. When I use query with word "ain't", no result is

Re: field length normalization

: : Did you reindex after setting omitNorms to false? I'm not sure whether or : not it is needed, but it makes sense. Yes i deleted the old index and reindexed it. Just to add another fact, that the titlles length is less than 10. I am not sure if solr has pre-set values for length normalizatio

Re: field length normalization

Did you reindex after setting omitNorms to false? I'm not sure whether or not it is needed, but it makes sense. On Thu, Mar 11, 2010 at 5:34 PM, muneeb wrote: > > Hi, > > In my schema, the document title field has "omitNorms=false", which, if I > am > not wrong, causes length of titles to be cou

Re: How to edit / compile the SOLR source code

Yep, as you've discovered, the import from ant build file doesn't work for the solr build.xml in eclipse. There is an excellent how-to for getting Solr up and running in Eclipse for debugging purposes here: http://www.lucidimagination.com/developers/artiicles/setting-up-apache-solr-in-eclipse Onc

Highlighting Results

Hi All Im not sure where i'm going wrong but highlighting does not seem to work for me. I have indexed around 5000 PDF documents which went well. Running normal queries against the attr_content works well. When adding any hl code it does not seem to make a bit of difference. Here is an example

Re: embedded server / servlet container

: I am trying to provide an embedded server to a web application deployed in a : servlet container (like tomcat). If you are trying to use Solr inside another webapp, my suggestion would just be to incorporate the existing Solr servlets, jsps, dispatch filter, and web.xml specifics from solr in

issue with delete index

Hi, I have made some changes to my schema, including setting of omitNorms to false for a few fields. I am using Solr1.4 with SolrJ client. I deleted my index using the client: solrserver.deleteByQuery("*:*"); solrserver.optimize(); But after reindexing and running the queries i don't see any di

Re: Scaling indexes with high document count

: I wonder if anyone might have some insight/advice on index scaling for high : document count vs size deployments... Your general approach sounds reasonable, although specifics of how you'll need to tune the caches and how much hardware you'll need will largely depend on the specifics of the d

Re: Architectural help

: We have about 5M records ranging in size all coming from a DB source (only 2 : tables). What will be the most efficient way of indexing all of these : documents? I am looking at DIH but before I go down that road I wanted to The main question to ask yourself is what your indexing freshness req

How to edit / compile the SOLR source code

Hi, Sorry for asking this very simple question but I am very new to SOLR and I want to play with its source code. As a initial step I have a requirement to enable wildcard search (*text) in SOLR. I am trying to figure out a way to import the complete SOLR build to Eclipse and edit QueryParsing.j

Re: HTMLStripTransformer not working with data importer

Hi- I can't seem to make any of the transfomers work, I am using the DataImporter to pull in data from a wordpress instance (see below). Neither REGEX or HTMLStrip seems to do anything to my content. Do I have to include a separate jar with the transformers? Are the transformers in 1.4 (particu

Re: Snapshot / Distribution Process

: Subject: Snapshot / Distribution Process : In-Reply-To: <27854256.p...@talk.nabble.com> http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a fresh email.

Re: What does means ~2, ~3, ~4 in DisjunctionMaxQuery?

On Mar 11, 2010, at 11:42 AM, Marc Sturlese wrote: I am debuggin a 2 words query build using dismax. So it's build from DisjunctionMaxQueries being the minShouldMatch 100% and tie breaker multiplier = 0.3 +((DisjunctionMaxQuery((content:john | title:john~0.3) DisjunctionMaxQuery((content:malone

What does means ~2, ~3, ~4 in DisjunctionMaxQuery?

I am debuggin a 2 words query build using dismax. So it's build from DisjunctionMaxQueries being the minShouldMatch 100% and tie breaker multiplier = 0.3 +((DisjunctionMaxQuery((content:john | title:john~0.3) DisjunctionMaxQuery((content:malone | title:malone)~0.3))~2) And a 3 words one (with sa

Re: Snapshot / Distribution Process

Have you started rsyncd on the master? Make sure that it is enabled before you start: http://wiki.apache.org/solr/SolrCollectionDistributionOperationsOutline You can also tried running snappuller with the -V option to et more debugging info. Bill On Wed, Mar 10, 2010 at 4:09 PM, Lars R. Noldan

Re: distinct on my result

Hi, Try replacing KeywordTokenizerFactory with a WhitespaceTokenizerFactory so it'll create separate terms per word. After a reindex it should work. Regards, gwk On 3/11/2010 4:33 PM, stocki wrote: hey, okay i show your my settings ;) i use an extra core with the standard requesthandler.

Re: Solr Performance Issues

Hi Erick, The way the load test works is that it picks up 5000 queries, splits them according to the number of threads (so if we have 10 threads, it schedules 10 threads - each one sending 500 queries). So it might be possible that the number of queries at a point later in time is greater than the

Re: distinct on my result

hey, okay i show your my settings ;) i use an extra core with the standard requesthandler. SCHEMA.XML so i copy my names to the field suggest and use the EdgeNGramFilter and some others

Re: Call for presentations - Berlin Buzzwords - Summer 2010

On 11.03.2010 Isabel Drost wrote: > Call for Presentations Berlin Buzzwords It should have been http://berlinbuzzwords.de of course... Isabel signature.asc Description: This is a digitally signed message part.

release schedule?

Hello I'm new to this list, so please excuse me if I'm asking in the wrong place. I have been tasked with planning the next release of our software. Today, we are using Solr 1.4.0, and we plan to release a new version of our software later this year. I would like to know, if possible:

Re: Content Highlighting

Please see: http://wiki.apache.org/solr/UsingMailingLists and repost with additional information. Best Erick On Thu, Mar 11, 2010 at 10:10 AM, Lee Smith wrote: > With the highlighting options will Solr highlight the found text something > like goo

Content Highlighting

With the highlighting options will Solr highlight the found text something like google search does ? I cant seem to get this working ? Hope someone can advise.

Re: Solr Performance Issues

How many outstanding queries do you have at a time? Is it possible that when you start, you have only a few queries executing concurrently but as your test runs you have hundreds? This really is a question of how your load test is structured. You might get a better sense of how it works if your te

Re: mincount doesn't work with FacetQuery

: I'm faceting with a query range (with addFacetQuery) and setting mincount to : 10 (with setFacetMinCount(10)), but Solr is not respecting this mincount; : it's still giving me all responses, even those having less than 10 retrieved : documents. if by "all responses" you mean "all facet queries"

Solr Performance Issues

Hi everyone, I have an index corresponding to ~2.5 million documents. The index size is 43GB. The configuration of the machine which is running Solr is - Dual Processor Quad Core Xeon 5430 - 2.66GHz (Harpertown) - 2 x 12MB cache, 8GB RAM, and 250 GB HDD. I'm observing a strange trend in the queri

Call for presentations - Berlin Buzzwords - Summer 2010

Call for Presentations Berlin Buzzwords http://buzzwordsberlin.de Berlin Buzzwords 2010 - Search, Store, Scale 7/8 June 2010 This is to announce the Berlin Buzzwords 2010. The first conference on scalable and open search, data process

Re: distinct on my result

Hi, I'm no expert on the full-text search features of Solr but I guess that has something to do with your fieldtype, or query. Are you using the standard request handler or dismax for your queries? And what analysers are you using on your product name field? Regards, gwk On 3/11/2010 3:24

Index size on disk

Hello, I needed an easy way to see the index size (the actual size on disk, not just the number of documents indexed) and as i didn't found anything for doing that on the documentation or on the list, I coded a fast solution. I added the Index size as a statistic of the searcher, that way the va

Re: distinct on my result

okay. we have a lot of products and i just importet the name of each product to a core. make an edgengram to this and my autoCOMPLETION runs. but i want an auto-suggestion: example. autoCompletion--> I: "harry" O: "harry potter..." but when the input ist --> I. "potter" -- O: / so wh

Aggregate functions on faceted result

Hi. We would like to be able to create trend graphs which have date in the X-axle and sum(pagerank) on the Y-Axle. We have the field pageRank stored as an external field (since it is updated all the time). I have started to build a SearchComponent which will be named something like FacetFunctionC

Re: mincount doesn't work with FacetQuery

Steve - I'm a bit confused... each facet.query (using HTTP parameter nomenclature) only adds a single value to the response, the number of docs within the current constraints that match that query. facet.mincount is specifically for facet.field, which adds a name/ value pair for each valu

Re: Distributed search fault tolerance

I guess I must be including too much information in my questions, running into "tl;dr" with them. Later today when I have more time I'll try to make it more bite-size. On 3/9/2010 2:28 PM, Shawn Heisey wrote: I attended the Webinar on March 4th. Many thanks to Yonik for putting that on. Tha

Apache Solr module with drupal - where to change key word in context?

I am using the apache solr module with our Drupal site. Out data is not clean enough to use the key word in context blurb under the title in the result set. I would like to change it to the first N characters in the body of the node. Can anyone direct me to the file and line(s) where I would do th

Re: Advance Search

Have you looked at dismax? Erick On Thu, Mar 11, 2010 at 4:40 AM, Suram wrote: > > > How can i achieve the advance search in solr . > > i need search books like (eg title = "The Book of Three",author= "Lloyd > Alexander", price = "99.00") > > How can i querying this > -- > View this message in

SolrJ - separte different results from the same facet query?

I'm faceting with a two different query ranges while using addFacetQuery. I wonder wether it's possible using SolrJ to extract the result of each query range separately. Here's is my example: addFacetQuery("price:[* TO 150]"); addFacetQuery("price:[151 TO 300]"); etc. addFacetQuery("date:[* TO NOW

Re: Architectural help

Assuming you create the view in such a way that it returns 1 row for each solrdocument you want indexed: yes On Wed, Mar 10, 2010 at 7:54 PM, blargy wrote: > > So I can just create a view (or temporary table) and then just have a > simple > "select * from (view or table)" in my DIH config? > >

mincount doesn't work with FacetQuery

I'm faceting with a query range (with addFacetQuery) and setting mincount to 10 (with setFacetMinCount(10)), but Solr is not respecting this mincount; it's still giving me all responses, even those having less than 10 retrieved documents. I'm wondering wether there's another way to define the minc

field length normalization

Hi, In my schema, the document title field has "omitNorms=false", which, if I am not wrong, causes length of titles to be counted in the scoring. But when I query with: "word1 word2 word3" I dont know why still the top two documents title have these words and other words, where as the document

Re: embedded server / servlet container

Hi all. Anyone involved in a configuration like this? Or is it a wrong approach or the wrong place to ask for? Thanks in advance, Dino. -- 2010/3/10 Dino Di Cola > Dear all, > I am trying to provide an embedded server to a web application deployed in > a servlet container (like tomcat). > I woul

Re: index merge

Hi All, Thank you for the very valuable suggestions. I am planning to try using the Master - Slave configuration. Best Rgds, Mark. On Mon, Mar 8, 2010 at 11:17 AM, Mark Miller wrote: > On 03/08/2010 10:53 AM, Mark Fletcher wrote: > >> Hi Shalin, >> >> Thank you for the mail. >> My main purpose

Multiple SOLR queries on same index

Hi, Is it possible to execute multiple SOLR queries (basically same structure/fields but due to the headersize limitations for long query URLs, thinking of having multiple SOLR queries) on single index like a batch or so? Best Regards, Kranti K K Parisa

Cant commit on 125 GB index

Hi, I'm having timeouts commiting on a 125 GB index with about 2200 docs. I'm inserting new docs every 5m and commiting after that. I would like to try the autocommit option and see if I can get better results. I need the docs indexed available for searching in about 10 minutes after

Advance Search

How can i achieve the advance search in solr . i need search books like (eg title = "The Book of Three",author= "Lloyd Alexander", price = "99.00") How can i querying this -- View this message in context: http://old.nabble.com/Advance-Search-tp27861279p27861279.html Sent from the Solr - User

Facet pagination

Is there a way to get *total count of facets* per field? Meaning, if my facets are - 305807 264748 181084 130546 98544 82741 77157 Then, is the underneath possible?

Re: Updating FAQ for International Characters?

So I am using Sunspot to post over, which means an extra layer of indirection between mean and my XML! I will look tomorrow. On Mar 10, 2010, at 7:21 PM, Chris Hostetter wrote: : Any time a character like that was index Solr through a unknown entity error. : But if converted to À or À t