Solr4 distributed IDF

Hi there, Does there exist any issue ticket about the distributed IDF feature in solr4? Or maybe there already have some patches that I can use? Thank you very much. -- Ke Wu, Best Regards

lucene multi term query

Hi there, Here is what I want to do: I have some documents that contain two fields, say "title" and "body", and they are both NOT stored. And now I want the user to be able to search inside both "title" and "body" with some keyword, say "lucene", using MultiTermQuery (I don't want to use two s

Re: RessourceLoader in Solr 4 beta

Hi, I reply to myslef. In fact in Solr 4 beta, org.apache.lucene.analysis.util.ResourceLoader is a SolrResourceLoader. However getConfigDir do not return the absolute path to the config directory. I mimiced the openResource method from ResourceLoader in order to write my own getConfigDir me

Re: Solr Shard Replicas sharing files

I was playing with the thought because want to find a way to easily rebalance hardware resources. And it takes a lot of time to move replicas around because the shards we are serving are many (25 and counting) and large (50GB). If all replicas could share a single fileset, it would be so easy to

Re: lucene multi term query

Hi Eric, This is a common question :) Here are some previous answers: http://search-lucene.com/?q=which+field+hit+match&fc_project=Solr Otis  Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm  - Original Message - > From: Eric Wu > To: solr-user@

Re: Solr4 distributed IDF

Hi Eric, This will show you some previous discussions, as well as the JIRA issue with oldish patches: http://search-lucene.com/?q=distributed+IDF&fc_project=Solr  Otis  Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm  - Original Message - > Fr

Re: optimum solr core size

Jame, That really depends on your hardware, query load and profile, search latency and other performance requirements. That said, a 123 GB core is a pretty big core :) Otis Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm  - Original Message - >

Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

Hi, Is it possible to have a Solr documents with deeply nested data structures? e.g. (in JSON) { "name": "Fred", "measurements": { "chest": "15", "legs": "32", ... } } ? Thanks -- Regards, Asfand Yar Qazi Team 87 - High Throughput Gene Targetin

Re: Single Core or Multiple Core?

Hi, I am posting my query as it is very much related to this one. I have an application which creates different dataset on each invocation. With dataset, I mean different columns/schema/metadata and different data ofcourse. The only fields common to the data generated are the primary key. I hav

Re: Null Pointer Exception on DIH with MySQL

I really, really, really _hate_ it when I have to blame mysterious corruptions and the problem magically goes away, it leaves me worried that the problem is just lurking out there somewhere waiting...waitingwaiting with my name on it ... Anyway, thanks for bringing closure to this. Erick On

Re: Patch 2429 for solr1.3?

I very much doubt it, all I can say is "give it a try and see". But those code lines are so divergent that I would be extremely surprised if you could even apply the patch, much less get it to work. If you _do_ try it, there'll be no support, you're on your own. But I suspect that if you simply t

Re: Multiple Versions getting formed while replicating in solr 1.4.1

Wait, when you say "the data can be pushed to the slave...on need basis" do you mean you're _indexing_ data to the slave? This is a definite no-no, Solr will get all confused about what the proper version is and all bets are off. If you need to control replication to the slave, I suggest you index

Re: Patch 2429 for solr1.3?

Thanks Erick ,Local params are there from solr 1.4 Regards Sujatha On Thu, Aug 30, 2012 at 5:26 PM, Erick Erickson wrote: > I very much doubt it, all I can say is "give it a try and see". But those > code lines are so divergent that I would be extremely surprised if > you could even apply the p

Re: Solr and query abortion

The first thing I'd do is run your query with &debguQuery=on and look at the "timings" section. That'll tell you what component is taking all the time and should help you figure out where the problem is But worst-case you could implement a custom component to stop processing after some set num

Re: Maximum index size on single instance of Solr

Here's a blog outlining why this is so hard to answer: http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ Just one example from your post, you mention index size as a metric. It's often useless. Stored data ('stored="true" ') is placed in file

Re: Configure logging with Solr 4 on Tomcat 7

Thank you guys. I've managed to configure log4j with Solr. I put log4j.properties into WEB-INF/classes, and only keep *slf4j-log4j12-1.6.6.jar, jcl-over-slf4j-1.6.4.jar, log4j-1.2.17.jar *in the WEB-INF/lib, eventually, that works. On Wed, Aug 29, 2012 at 9:08 PM, Erick Erickson wrote: > Have yo

Extract footer/header text out of Word docs

Hi. Is it possible to specifically extract footer/header and body text out of a Word document using Solr? In other words, we'd like to index/store those items in different Solr fields. Also, is it possible to search on specific styles within a Word document? Can these attributes be indexed? Tha

solr indexing slows down after few minutes

Hi I am indexing our website using solr. After crawling the website I am adding the contents to the solr server and committing. My slor server is on 3.6 version. I observed that initially the indexing is very fast. It added around 33 docs/sec; making 2000 / minute. But then after the 30 mins or s

Re: Extract footer/header text out of Word docs

Hi Alex, I think you may get better help on the Tika mailing list - Solr uses Tika to parse rich text docs and extract text from them.  I don't know if Tika can figure out what's from a header and a footer... Otis  Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.

RE: Extract footer/header text out of Word docs

Tika can do this but Solr doesn't use the BoilerpipeContentHandler. Perhaps it should made configurable which content handler Solr uses and in case of the BoilerpipeContentHandler which extractor implementation to use. -Original message- > From:Otis Gospodnetic > Sent: Thu 30-Aug-2012

Re: Hierarchical faceting and filter query exclusions

Thanks, Erick. I have looked at the documentation at length and done a number of tests. I don't see a way to do exclusions with hierarchical facets. Thanks, Nick On Aug 29, 2012, at 9:50 PM, "Erick Erickson" wrote: > See "Tagging and excluding filters" here: > http://lucidworks.lucidimagina

Re: Sorting on mutivalued fields still impossible?

You can also use a "Field Mutating Update Processor" to do a "smart" copy of a multi-valued field to a sortable single-valued field. See: http://wiki.apache.org/solr/UpdateRequestProcessor#Field_Mutating_Update_Processors Such as using the maximum value via MaxFieldValueUpdateProcessorFactory.

Re: lucene multi term query

If you add the &debugQuery=true request option, the "explain" section of the response will have all of that info, and a lot more. That may be a lot to sift through, but at least it is there. You can identify each document and then check which field names appear in the scoring for that document.

Re: Hierarchical faceting and filter query exclusions

You are correct, it doesn't work : Queries like : http://localhost:8983/solr/collection1/select?q=*:*&facet=on&facet.pivot={!ex=a_tag}field1,field2&facet.limit=5&rows=0&fq={!tag=a_tag}field3:"filter"; result in the following response : 400 1 on *:* 5 {!ex=a_tag}field1,field

Re: Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

The general rule is that you need to flatten your data. So, you would have "chest_measurement" and "leg_measurement" fields. -- Jack Krupansky -Original Message- From: Asfand Qazi Sent: Thursday, August 30, 2012 6:03 AM To: solr-user@lucene.apache.org Subject: Possible to have Solr do

Re: Hierarchical faceting and filter query exclusions

Tanguy, thanks for the confirmation! We found the same issue with facet.missing parameter as well. Sent from my iPad On Aug 30, 2012, at 10:18 AM, "Tanguy Moal" wrote: > You are correct, it doesn't work : > > Queries like : > http://localhost:8983/solr/collection1/select?q=*:*&facet=on&face

Re: Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

On 30/08/12 15:19, Jack Krupansky wrote: The general rule is that you need to flatten your data. So, you would have "chest_measurement" and "leg_measurement" fields. -- Jack Krupansky Ah. What if I cannot flatten it because I have an array of hashes? Thanks Asfand Yar Qazi -Original

Re: Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

You can always flatten. Remember, you are using SOLR to find the records. So, just make sure you flatten so each result represents one unit of information for your purposes. So, you push all leg_measurements into one big leg_measurement multiValue field (and return all ?? that have at least one le

Re: Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

There are multi-valued fields as well. You just have to be creative in the flattening process. And there is a "join" capability as well: http://wiki.apache.org/solr/Join In any case, try to take the simplest approaches first before getting overly complex. Maybe start by focusing on what you

RE: Dataimport Handler in solr 3.6.1

There were 2 major changes to DIH Cache functionality in Solr 3.6, only 1 of which was carried to Solr 4.0: - Solr 3.6 had 2 MAJOR changes: 1. We support pluggable caches so that you can write your own cache implemetations and cache however you want. The goal here is to allow you to cache to

Re: Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

On 30/08/12 15:51, Alexandre Rafalovitch wrote: Don't treat SOLR as your primary database with complex structure, it is not built for that. On 30/08/12 16:01, Jack Krupansky wrote: Maybe start by focusing on what you expect that a user query will look like in Solr. Yeah, thanks guys - I'll

Re: Possible to have Solr documents with deeply nested data structures (i.e. 'hashes within hashes')?

On Aug 30, 2012, at 8:07 AM, Asfand Qazi wrote: > The consensus around here seems to be to start using multiple cores to hold > the different bits of a deeply nested record anyway, but I'll remember the > flattening suggestions made. It is hard for me to understand how that could possibly work.

use of filter queries in Lucene/Solr Alpha40 and Beta4.0

Hi all, My query against an index is (I leaved out some of the facet fields) f.navBranchlib.facet.limit=1000& facet=on&facet.mincount=1& facet.limit=100& bq=navBranchlib:A100^1000& bq=navBranchlib:UFSW^1000& start=0&q=+(+%2Bmitbestimmung++)+& facet.field=navNetwork& qt=sb-bbfull-01 -> qt refers to

Re: use of filter queries in Lucene/Solr Alpha40 and Beta4.0

And what happens if you use a straight Lucene query for the filter query? Change: fq={!term+f%3DnavNetwork}nebis to fq=navNetwork:nebis What field type is navNetwork? String or text? -- Jack Krupansky -Original Message- From: Günter Hipler Sent: Thursday, August 30, 2012 11:

Field collapsing / Grouping - How to make SOLR return resultset of 2 field groups (intersection)?

Hi, I have an question on SOLR grouping functionaliy. I have 2 fields for grouping, these 2 field can have different keywords stored in them Ex: Field1: CD, book, e-book Field2: repo1, repo2, repo3, repo4 Now I want to group the combination of CD/repo1 , book/repo2, e-book/repo3,e-book/repo4

Re: Maximum index size on single instance of Solr

We have a 48GB index size on a single shard. 20+ million documents. Recently migrated to SOLR 3.5 But we have a cluster of SOLR servers for hosting searches. But i do see to migrate to SOLR sharding going forward. Thanx Pravesh -- View this message in context: http://lucene.472066.n3.nabble.

Re: solr indexing slows down after few minutes

Did you checked wiki: http://wiki.apache.org/lucene-java/ImproveIndexingSpeed http://wiki.apache.org/lucene-java/ImproveIndexingSpeed Do you commit often? Do you index with multiple threads? Also try experimenting with various available MergePolicies introduced from SOLR 3.4 onwards Thanx Prave

Re: optimum solr core size

How many documents are there in the index? How many stored/indexed fields? There is no magic number as yet for defining the size of a single core(whether no. of docs OR the size of index), but 123GB seems to be on a higher side, so, you could definitely go for sharding of indexes. BTW, how are you

Re: use of filter queries in Lucene/Solr Alpha40 and Beta4.0

The "q" and "bq" params have changed slightly between your first query and the query where you add the "fq" param ... because of how "bq" is additively added to the main query, it's possible this difference may account for the behavior your are seeing -- double check the debugQuery output for

Re: Correctly importing and producing null in search results

Erick: Thanks for your reply. Simply omitting the null fields is an intriguing idea, and I will test this out. Why not use the JSON response writer? Two reasons: our clients dictate a particular JSON schema that changes on a query-by-query basis. The schemas can be quite complex. Also, we rol

Re: Load Testing in Solr

Hi Dhaivat, JMeter is a nice tool. But it all depends what sort of load are you expecting, how complex queries are you expecting(sorting/filtering/textual searches). You need to consider all these to benchmark. Thanx Pravedsh -- View this message in context: http://lucene.472066.n3.nabble.com

Re: AW: AW: auto completion search with solr using NGrams in SOLR

You wouldnt need any tokenizer, I'm not sure if Solr will allow that. You also need to move the Edge n gram filter to index from query (I think Ahmet suggested this). It will look something like this: note that pattern

Re: AW: AW: auto completion search with solr using NGrams in SOLR

> I tried with below said changes, but getting same result as > earlier. > > suggest/?q="michael ja" > --- > positionIncrementGap="100" > omitNorms="true"> >     >       class="solr.KeywordTokenizerFactory" /> >       class="solr.LowerCaseFilterFactory" /> >       class="s

Re: Query Time problem on Big Index Solr 3.5

How often the documents are added in the index? Try lessen down the optimize frequency. BTW do you optimize only on master (which should be the desired way). Also specifically for dates ranges, try to use the filter queries, this way it would be cached and would thus be faster. This would also a

Re: Load Testing in Solr

Thanks Pravedsh for your reply. i ll use the JMeter tool . On Thu, Aug 30, 2012 at 11:10 PM, pravesh wrote: > Hi Dhaivat, > JMeter is a nice tool. But it all depends what sort of load are you > expecting, how complex queries are you expecting(sorting/filtering/textual > searches). You need to c

Re: Query Time problem on Big Index Solr 3.5

Have you tried sharding ? The best practices recommend sharding when your index grows too large and query time is slow. On Tue, Aug 28, 2012 at 2:02 AM, mpcmarcos wrote: > Hello, > > I have a problem, I'm working with Solr 3.5, with a index that has > 8.000.000 > of documents (13Gb), each docum

Static template column in DIH

Hi everyone, I have defined a template column like this: I dont have a field called Category in my query, but defined in schema.xml. I'm expecting it to populate the Category field with a fixed value "Books". It doesnt work. What am I missing ? Any help is much appreciated. Thanks

RE: Static template column in DIH

You might just be missing . See http://wiki.apache.org/solr/DataImportHandler#TemplateTransformer James Dyer E-Commerce Systems Ingram Content Group (615) 213-4311 -Original Message- From: Kiran Jayakumar [mailto:kiranjuni...@gmail.com] Sent: Thursday, August 30, 2012 1:54 PM To: solr

Re: Static template column in DIH

Thank you sir, works like a charm ! On Thu, Aug 30, 2012 at 12:06 PM, Dyer, James wrote: > You might just be missing > . > > See http://wiki.apache.org/solr/DataImportHandler#TemplateTransformer > > James Dyer > E-Commerce Systems > Ingram Content Group > (615) 213-4311 > > > -Original Messa

Solr 1.4 very slow - 60.000 documents

Hi, I've a Solr 1.4 with about 60.000 documents but is getting very slow in the searches. The website have usually very very few users. The machine have 512MB Ram and I'm serving with Nginx one Python based web app that is very low weight. How can I search for the cause of the Solr slowdown? Any

about phrase synonyms

Hi all, I am new to Solr. Now I have got a problem so I hope you guys can help me out. In my synonyms.txt file I have got a lot of phrases like following: ecu,electronic control unit,environmental control unit, when I query ecu, the debugQuery shows this: "parsedquery":"+DisjunctionMaxQuery

RE: Solr4 distributed IDF

Hi Ke, Have you seen ? Steve -Original Message- From: Eric Wu [mailto:eirik...@gmail.com] Sent: Thursday, August 30, 2012 3:05 AM To: solr-user@lucene.apache.org Subject: Solr4 distributed IDF Hi there, Does there exist any issue ticket

Re: Solr 1.4 very slow - 60.000 documents

First, add &debugQuery=true to your query request and then look at the "timing" section of the response to verify whether the "QueryComponent" is taking up the bulk of the time. Sometimes, faceting or highlighting or other components can be the real performance hogs. -- Jack Krupansky -Or

Send plain text file to solr for indexing

Hello, I used to use solrcell, which has built-in tika support to handle both extraction and indexing of raw documents. Now I got another text extraction provider to convert raw document to a plain text txt file so I want to let solr bypass that extraction phase. Is there a way I can send the plai

Re: Send plain text file to solr for indexing

SolrCell can be used to index raw text files as well. -- Jack Krupansky -Original Message- From: Bing Hua Sent: Thursday, August 30, 2012 6:28 PM To: solr-user@lucene.apache.org Subject: Send plain text file to solr for indexing Hello, I used to use solrcell, which has built-in tika

Re: Facet not working in Solr

: Query parser = dismax : range query : : q=field:[a TO b] : : DOESN'T WORK (Less important question. WHY?) dismax does not support range query syntax, it never has... https://wiki.apache.org/solr/DisMaxQParserPlugin#Query_Syntax : Facet: ... : DOESN'T WORK. Define "DOESN'T WORK" ...

Re: Frequently Updated Index and Caching

: filtercache. Whether you're getting decent hit ratios is what tells you whether : disabling caching is a good idea or not. Right ... further down on the page you mentioned is a note about disabling caching. I've beefed it up with some additional info... https://wiki.apache.org/solr/SolrCachi

Re: Solr4 distributed IDF

The math for "confidence values" in probability theory shows that distributed DF does not matter after not very many documents. If you have 10s of thousands of documents in each shard, don't worry. On Thu, Aug 30, 2012 at 1:19 PM, Steven A Rowe wrote: > Hi Ke, > > Have you seen

Re: Solr4 distributed IDF

Hi, Lance We may have unbalanced shards, does it matter? And do you know any post that has the detailed math about this? Thank you very much. On Fri, Aug 31, 2012 at 8:25 AM, Lance Norskog wrote: > The math for "confidence values" in probability theory shows that > distributed DF does not m

Re: Solr4 distributed IDF

Hi Steven and Otis, Thank you! That's very helpful information :) On Fri, Aug 31, 2012 at 4:19 AM, Steven A Rowe wrote: > Hi Ke, > > Have you seen ? > > Steve > > -Original Message- > From: Eric Wu [mailto:eirik...@gmail.com] > Sent:

Re: Solr4 distributed IDF

That is true if you randomly distribute the documents. If they are distributed according to topic, there can be some big anomalies. Also, the DFs for rare terms will have bigger errors. There is some statistical theorem about this, but I can't remember it right now. Thanks to Zipf, most of your

Re: Solr4 distributed IDF

Hi Walter, Thank you for your help. I think you are right, the most important issue here is "the most selective terms are rare". So I probably still need to implement distributed IDF to get better results. On Fri, Aug 31, 2012 at 8:36 AM, Walter Underwood wrote: > That is true if you randoml

need basic information

I am trying to use solr for processing logs and searching them. But i dont see a clear sample anywhere for this type of scenario. i need to index plain text files , example content - -mm-dd : Account 123 created -mm-dd : Account 123 updated Account 123 being spread across multipl

solr hanging regularly, thread dump attached, PDFOperator

If we have more than half a dozen processes posting updates to our Solr instance (each about 1/sec), we get regular and fairly reproducible solr hangs, requiring a kill -9 of the solr process. We're running Solr 3.4, a fairly out-of-the-box solr/jetty setup. I've got a thread dump and some GC logg

Re: Facet not working in Solr

Okay my bad for the lack of info : - When I say doesn't work means numFound = 0 - Whereas range queries on the same fields give documents - Even adding a value in q gives numFound = 0. using all 3 types of parsers. Even if dismax doesn't support range queries. edismax and lucene sh

Re: AW: AW: auto completion search with solr using NGrams in SOLR

Hi, thanks, I checked with given changes, getting below error saying that SOLR is not allowing without tokenizer. org.apache.solr.common.SolrException: analyzer without class or tokenizer & filter list at org.apache.solr.schema.IndexSchema.readAnalyzer(IndexSchema.java:914) at org.apache.solr.sc

Re: Facet not working in Solr

Again to elaborate, the documents all have an indexed field called u_height which is between 160.0 to 172.0 Range queries work on them with the lucene and edismax. But faceting using the following URL gives numFound = 0, and all 0's in the facet counts q=&fq=&start=0&rows=100&fl=u_height,id,scor

Re: Facet not working in Solr

: But faceting using the following URL gives numFound = 0, and all 0's in the ... : q=&fq=&start=0&rows=100&fl=u_height,id,score&wt=&debugQuery=on&explainOther=&facet=on&facet.range.other=all& ... : Adding a value in q with qf set like so : : *q=165.0&qf=u_height* : &fq=&start=

Re: solr hanging regularly, thread dump attached, PDFOperator

I've heard of a couple of situations where the parsing of multiple PDF files in parallel was problematic with lots of blocked threads. There may or may not be bugs in PDFBox, but one workaround is simply to only send one PDF at a time to Solr, or at least a smaller number in parallel. I don't

Query during a query

Hi list, I want to get distinct data from a single solr field when ever a search query is started by an user. How can I do this? Regards, Johannes

Re: Facet not working in Solr

Well yes, Thank you so much. I had lost that point of * in the main query. With that I get what I expect. This was awesome. Thanks so much On Fri, Aug 31, 2012 at 9:49 AM, Chris Hostetter wrote: > > : But faceting using the following URL gives numFound = 0, and all 0's in > the > ... >

Re: Query Time problem on Big Index Solr 3.5

13 GB index isn't too big, but going forward index sharding is the solutions for large single core indexes. This way you can scale out. This links will give you some info: http://lucidworks.lucidimagination.com/display/solr/Distributed+Search+with+Index+Sharding http://lucidworks.lucidimagination

Re: Query during a query

Did you checked SOLR Field Collapsing/Grouping. http://wiki.apache.org/solr/FieldCollapsing http://wiki.apache.org/solr/FieldCollapsing If this is what you are looking for. Thanx Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/Query-during-a-query-tp4004624p400463

Antwort: Re: Query during a query

Thanks for the answer, but I want to know how I can do a seperate query before the main query. And I only want this data in my programm. The user won't see it. I need the values from one field to get some information from an external source while the main query is executed. pravesh schrieb am

Re: need basic information

One basic and trivial solution could be to have schema like; Date (of type date/string) --> this would store the '-mm-dd' format date Tag (of type string) --> the text/tag 'Account' goes into this account-id (of type sint/int) --> account id like '123' goes into this action (of type sting) -->