Re: How to start troubleshooting a content extraction issue

You can test the standalone content extraction with the tika-app.jar - Command to output in text format - java -jar tika-app-0.8.jar --text file_path For more options java -jar tika-app-0.8.jar --help Use the correct tika-app version jar matching the Solr build. Regards, Jayendra On Wed, Aug 1

Re: Solr 3.3 crashes after ~18 hours?

Hi, googling "hotspot server 19.1-b02" shows that you are not alone with hanging threads and crashes. And not only with solr. Maybe try another JAVA? Bernd Am 10.08.2011 17:00, schrieb alexander sulz: Okay, with this command it hangs. Also: I managed to get a Thread Dump (attached). regards

Re: frange not working in query

The default sort is on relevance. I want to give an option to users to sort the results by date (latest on top). This works fine for queries which have few results (upto 100). However, it brings inaccurate results as soon as the figure reaches 1000s. I am trying to limit the sorting to top few r

Re: Indexing tweet and searching "@keyword" OR "#keyword"

Do you really want a search on "ipad" to *fail* to match input of "#ipad"? Or vice-versa? My requirement is : I want to search both '#ipad' and 'ipad' for q='ipad' BUT for q='#ipad' I want to search ONLY '#ipad' excluding 'ipad'. On 10 August 2011 19:49, Erick Erickson wrote: > Please look mo

LockObtainFailedException

Hi, We are doing streaming update to solr for multiple user, We are getting Aug 10, 2011 11:56:55 AM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/lib/solr/data/index/write.lock at org.apache

Hudson build issues

Whenever I try to build this on our hudson server it says it can't find org.apache.lucene:lucene-xercesImpl:jar:4.0-SNAPSHOT. Is the Apache repo lacking this artifact? -- View this message in context: http://lucene.472066.n3.nabble.com/Hudson-build-issues-tp3244563p3244563.html Sent from the Sol

Re: Can't mix Synonyms with Shingles?

On Wed, Aug 10, 2011 at 7:10 PM, Jeff Wartes wrote: > > After some further playing around, I think I understand what's going on. > Because the SynonymFilterFactory pays attention to term position when it > inserts a multi-word synonym, I had assumed it scanned for matches in a way > that respec

Re: Increasing the highlight snippet size

Well, only after I posted this question in a public forum, I found the cause of my problem. I was using "hl.fragSize", instead of "hl.fragsize". After correcting the case, it worked as expected. Thanks. On Wed, Aug 10, 2011 at 3:19 PM, Sang Yum wrote: > I was just trying to set it a ridiculousl

RE: Can't mix Synonyms with Shingles?

After some further playing around, I think I understand what's going on. Because the SynonymFilterFactory pays attention to term position when it inserts a multi-word synonym, I had assumed it scanned for matches in a way that respected term position as well. (ie, for a two-word synonym, I assu

Re: Cache replication

Thanks for the advice paul, but post processing is a must for me given the nature of my application. I haven't had problems yet though. -- View this message in context: http://lucene.472066.n3.nabble.com/Cache-replication-tp3240708p3244202.html Sent from the Solr - User mailing list archive at

Re: Increasing the highlight snippet size

I was just trying to set it a ridiculously large number to make it work. What I am seeing is that hl.fragsize doesn't seem to make any difference in term of highlight snippet size... I just tried the query with hl.fragsize set to 1000. Same result as 10. On Wed, Aug 10, 2011 at 2:20 PM, simon wro

Re: Increasing the highlight snippet size

an hl.fragsize of 1000 is problematical, as Solr parses that parameter as a 32 bit int... that's several bits more. -Simon On Wed, Aug 10, 2011 at 4:59 PM, Sang Yum wrote: > Hi, > > I have been trying to increase the size of the highlight snippets using > "hl.fragSize" parameter, without

Re: Example Solr Config on EC2

Yes you can promote a slave to be master refer http://wiki.apache.org/solr/SolrReplication#enable.2BAC8-disable_master.2BAC8-slave_in_a_node In AWS one can use an elastic IP(http://aws.amazon.com/articles/1346) to refer to the master and this can be assigned to slaves as they assume the role of ma

Increasing the highlight snippet size

Hi, I have been trying to increase the size of the highlight snippets using "hl.fragSize" parameter, without much success. It seems that hl.fragSize is not making any difference at all in terms of snippet size. For example, compare the following two set of query/results: http://10.1.1.51:8983/so

Solr 3.3: DIH configuration for Oracle

Hello, all! I want to create a good DIH configuration for my Oracle database with deltas support. Unfortunately I am not able to do it well as DIH has the strange restrictions. I want to explain a problem on a simple example. In a reality my database has very difficult structure. Initial c

RE: Can't mix Synonyms with Shingles?

Hi Steven, The token separator was certainly a deliberate choice, are you saying that after applying shingles, synonyms can only match shingled terms? The term analysis suggests the original tokens still exist. You've made me realize that only certain synonyms seem to have problems though, so

Re: Problem with xinclude in solrconfig.xml

Sorry for the spam. I just figured it out. Thanks. On Wed, Aug 10, 2011 at 2:17 PM, Way Cool wrote: > Hi, Guys, > > Based on the document below, I should be able to include a file under the > same directory by specifying relative path via xinclude in solrconfig.xml: > http://wiki.apache.org/solr

Problem with xinclude in solrconfig.xml

Hi, Guys, Based on the document below, I should be able to include a file under the same directory by specifying relative path via xinclude in solrconfig.xml: http://wiki.apache.org/solr/SolrConfigXml However I am getting the following error when I use relative path (absolute path works fine thou

Re: Example Solr Config on EC2

If I were to build a master with multiple slaves, is it possible to promote a slave to be the new master if the original master fails? Will all the slaves pickup right where they left off, or any time the master fails will we need to completely regenerate all the data? If this is possible, are th

RE: Building a facet query in SolrJ

: query.addFacetQuery(MyField + ":" + "\"" + uri + "\""); ... : But when I examine queryResponse.getFacetFields, it's an empty list, if "facet.query" constraints+counts do not come back in the "facet.field" section of hte response. they come back in the "facet.query" section of the re

RE: Can't mix Synonyms with Shingles?

Hi Jeff, Hi Jeff, You have configured ShingleFilterFactory with a token separator of "", so e.g. "International Corporation" will output the shingle "InternationalCorporation". If this is the form you want to use for synonym matching, it must exist in your synonym file. Does it? Steve > --

Re: Error loading a custom request handler in Solr 4.0

: custom request handler, so I wrote the minimal class (attached), compiled : and jar'd it, and placed it in example/lib. I added this to solrconfig.xml: that's the crux of hte issue. example/lib is where the jetty libraries live -- not solr plugins. you should either put your custom jar's in t

Can't mix Synonyms with Shingles?

I would like to combine the ShingleFilterFactory with a SynonymFilterFactory in a field type. I've looked at something like this using the analysis.jsp tool: ... ... However, when a ShingleFilterFacto

RE: query time problem

Thanks Simon for these tracks. Here's my answers : Can you tell if GC is happening more frequently than usual/expected ? GC is OK. Is the index optimized - if not, how many segments ? According to the statistics page from the admin : One shard (master/slave) has 10 segments The other shard (m

RE: Building a facet query in SolrJ

Hi -- I do get facets for all the values of MyField when I specify the facet field, but that's not what I want. I just want facets for a subset of the values of MyField. That's why I'm trying to use the facet queries, to just get facets for those values. -Rich -Original Message- From:

Re: Building a facet query in SolrJ

Try making your queries, manually, to see this closer in action... q=MyField: and see what you get. In this case, because your URI contains characters that make the default query parser unhappy, do this sort of query instead: {!term f=MyField} That way the query is "parsed" properly into

Re: Error loading a custom request handler in Solr 4.0

Thanks Simon. I'll try again tomorrow. Tom On 10 August 2011 18:46, simon wrote: > This is in trunk (up to date). Compiler is 1.6.0_26 > > classpath was > > dist/apache-solr-solrj-4.0-SNAPSHOT.jar:dist/apache-solr-core-4.0-SNAPSHOT.jar > built from trunk just prior by 'ant dist' > > I'd try a

How to start troubleshooting a content extraction issue

Hello So, I'm a newbie to Solr and Tika and whatnot, so please use simple words for me :P I am running Solr on Tomcat 7 on Windows Server 2008 r2, running as the search engine for a Drupal web site. Up until recently, everything has been fine - searching works, faceting works, etc. Recently a u

Re: query time problem

Off the top of my head ... Can you tell if GC is happening more frequently than usual/expected ? Is the index optimized - if not, how many segments ? It's possible that one of the shards is behind a flaky network connection. Is the 10s performance just for the Solr query or wallclock time at t

Re: Error loading a custom request handler in Solr 4.0

This is in trunk (up to date). Compiler is 1.6.0_26 classpath was dist/apache-solr-solrj-4.0-SNAPSHOT.jar:dist/apache-solr-core-4.0-SNAPSHOT.jar built from trunk just prior by 'ant dist' I'd try again with a clean trunk . -Simon On Wed, Aug 10, 2011 at 1:20 PM, Tom Mortimer wrote: > Interest

RE: Building a facet query in SolrJ

I take it back. I didn't find it. I corrected my values and the facet queries still don't find what I want. The values I'm looking for are URIs, so they look like: http://place.org/abc/def I add the facet query like so: query.addFacetQuery(MyField + ":" + "\"" + uri + "\""); I print the query

Re: Error loading a custom request handler in Solr 4.0

Interesting.. is this in trunk (4.0)? Maybe I've broken mine somehow! What classpath did you use for compiling? And did you copy anything other than the new jar into lib/ ? thanks, Tom On 10 August 2011 18:07, simon wrote: > It's working for me. Compiled, inserted in solr/lib, added the confi

query time problem

Hi, I've noticed poor performance for my solr queries in the past few days. Queries of that type : http://server:5000/solr/select?q=story_search_field_en:(water boston) OR story_search_field_fr:(water boston)&rows=350&start=0&sort=r_modify_date desc&shards=shard1:5001/solr,shard2:5002

Re: Error loading a custom request handler in Solr 4.0

It's working for me. Compiled, inserted in solr/lib, added the config line to solrconfig. when I send a /flaxtest request i get 0 16 Hello! I was doing this within a core defined in solr.xml -Simon On Wed, Aug 10, 2011 at 11:46 AM, Tom Mortimer wrote: > Sure - > > import org.apache.solr

Re: [Help Wanted] Graphics and other help for new Lucene/Solr website

Looks nice! Font seems too light to read with comfort though. > Hi, > > We are in the process of putting up a new > Lucene/Solr/PyLucene/OpenRelevance website. You can see a preview at > http://lucene.staging.apache.org/lucene/. It is more or less a look and > feel copy of Mahout and Open For B

RE: [Help Wanted] Graphics and other help for new Lucene/Solr website

The site looks great. And thank you for including the ManifoldCF link. ;-) Karl -Original Message- From: ext Grant Ingersoll [mailto:gsing...@apache.org] Sent: Wednesday, August 10, 2011 10:09 AM To: solr-user@lucene.apache.org; java-u...@lucene.apache.org Subject: [Help Wanted] Graphic

Re: Is optimize needed on slaves if it replicates from optimized master?

Very well explained. Thanks. Yes, we do optimize Index before replication. I am not particularly worried about disk space usage. I was more curious of that behavior. *Pranav Prakash* "temet nosce" Twitter | Blog | Google

Re: how to ignore case in solr search field?

You can use solr.LowerCaseFilterFactory in an analyser chain for both indexing and queries. The schema.xml supplied with example has several field types using this (including "text_general"). Tom On 10 August 2011 16:42, nagarjuna wrote: > Hi please help me .. >how to ignore case w

Re: Error loading a custom request handler in Solr 4.0

Sure - import org.apache.solr.request.SolrQueryRequest; import org.apache.solr.response.SolrQueryResponse; import org.apache.solr.handler.RequestHandlerBase; public class FlaxTestHandler extends RequestHandlerBase { public FlaxTestHandler() { } public void handleRequestBody(SolrQueryReq

Re: Error loading a custom request handler in Solr 4.0

Th attachment isn't showing up (in gmail, at least). Can you inline the relevant bits of code ? On Wed, Aug 10, 2011 at 11:05 AM, Tom Mortimer wrote: > Hi, > Apologies if this is really basic. I'm trying to learn how to create a > custom request handler, so I wrote the minimal class (attached), c

RE: Dates off by 1 day?

Ah, great! I knew the problem was between the keyboard and the chair. Thanks! -Original Message- From: Sethi, Parampreet [mailto:parampreet.se...@teamaol.com] Sent: Wednesday, August 10, 2011 10:25 AM To: solr-user@lucene.apache.org Subject: Re: Dates off by 1 day? The Date difference is

Re: Dates off by 1 day?

The Date difference is coming because of different time zones. In Solr the date is stored as Zulu time zone and Solrj is returning date in CDT timezone (jvm is picking system time zone.) > 2002-05-13T00:00:00Z > I get: > > --> Sun May 12 19:00:00 CDT 2002 You can convert Date

Dates off by 1 day?

Hi all- I apologize in advance if this turns out to be a problem between the keyboard and the chair, but I'm confused about why my date field is correct in the index, but wrong in SolrJ. I have a field defined as a date in the index: And if I use the admin site to query the data, I g

Re: Cache replication

Consider putting a cache (memcached, redis, etc) *in front* of your solr slaves. Just make sure to update it when replication occurs. didier On Tue, Aug 9, 2011 at 6:07 PM, arian487 wrote: > I'm wondering if the caches on all the slaves are replicated across (such as > queryResultCache).  That i

Re: Solr 3.3 crashes after ~18 hours?

On Wed, Aug 10, 2011 at 11:00 AM, alexander sulz wrote: > Okay, with this command it hangs. It doesn't look like a hang from this thread dump. It doesn't look like any solr requests are executing at the time the dump was taken. Did you do this from the command line? curl "http://localhost:8983/

RE: Building a facet query in SolrJ

Oops. I think I found it. My desiredFieldValues list has the wrong info. Knew there was something simple wrong. From: Simon, Richard T Sent: Wednesday, August 10, 2011 10:55 AM To: solr-user@lucene.apache.org Cc: Simon, Richard T Subject: Building a facet query in SolrJ Hi - I'm trying to do a (

Error loading a custom request handler in Solr 4.0

Hi, Apologies if this is really basic. I'm trying to learn how to create a custom request handler, so I wrote the minimal class (attached), compiled and jar'd it, and placed it in example/lib. I added this to solrconfig.xml: When I started Solr with java -jar start.jar, I got this: ...

Re: Solr 3.3 crashes after ~18 hours?

Okay, with this command it hangs. Also: I managed to get a Thread Dump (attached). regards Am 05.08.2011 15:08, schrieb Yonik Seeley: On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz wrote: Usually you get a XML-Response when doing commits or optimize, in this case I get nothing in return, but

Re: paging size in SOLR

when you say queryResultCache, does it only cache n number of result for the last one query or more than one queries? On 10 August 2011 20:14, simon wrote: > Worth remembering there are some performance penalties with deep > paging, if you use the page-by-page approach. may not be too much of a

Building a facet query in SolrJ

Hi - I'm trying to do a (I think) simple facet query, but I'm not getting the results I expect. I have a field, MyField, and I want to get facets for specific values of that field. That is, I want a FacetField if MyField is "ABC", "DEF", etc. (a specific list of values), but not if MyField is an

RE: paging size in SOLR

I would imagine the performance penalties with deep paging will ALSO be there if you just ask for 1 rows all at once though, instead of in, say, 100 row paged batches. Yes? No? -Original Message- From: simon [mailto:mtnes...@gmail.com] Sent: Wednesday, August 10, 2011 10:44 AM To: s

Re: paging size in SOLR

Worth remembering there are some performance penalties with deep paging, if you use the page-by-page approach. may not be too much of a problem if you really are only looking to retrieve 10K docs. -Simon On Wed, Aug 10, 2011 at 10:32 AM, Erick Erickson wrote: > Well, if you really want to you ca

Re: paging size in SOLR

Well, if you really want to you can specify start=0 and rows=1 and get them all back at once. You can do page-by-page by incrementing the "start" parameter as you indicated. You can keep from re-executing the search by setting your queryResultCache appropriately, but this affects all searches

Re: Is optimize needed on slaves if it replicates from optimized master?

This is expected behavior. You might be optimizing your index on the master after every set of changes, in which case the entire index is copied. During this period, the space on disk will at least double, there's no way around that. If you do NOT optimize, then the slave will only copy changed se

Re: frange not working in query

I meant the frange query, of course On Wed, Aug 10, 2011 at 10:21 AM, simon wrote: > Could you tell us what you're trying to achieve with the range query ? > It's not clear. > > -Simon > > On Wed, Aug 10, 2011 at 5:57 AM, Amit Sawhney wrote: >> Hi All, >> >> I am trying to sort the results on a

Re: frange not working in query

Could you tell us what you're trying to achieve with the range query ? It's not clear. -Simon On Wed, Aug 10, 2011 at 5:57 AM, Amit Sawhney wrote: > Hi All, > > I am trying to sort the results on a unix timestamp using this query. > > http://url.com:8983/solr/db/select/?indent=on&version=2.1&q={

Re: Indexing tweet and searching "@keyword" OR "#keyword"

Please look more carefully at the documentation for WDDF, specifically: split on intra-word delimiters (all non alpha-numeric characters). WordDelimiterFilterFactory will always throw away non alpha-numeric characters, you can't tell it do to otherwise. Try some of the other tokenizers/analyzers

AW: Problem with DIH: How to map key value pair stored in 1-N relation from a JDBC Source?

Thanks, for this quick and enlightening answer! I didn't consider that a Transformer can create new columns. In combination with dynamic fields it is exactly what I was looking for. Thanks James ^^ -Ursprüngliche Nachricht- Von: Dyer, James [mailto:james.d...@ingrambook.com] Gesendet

Re: document indexing

With the first option you can be page specific in your search results and searches. Field collapsing/grouping will help with your normalisation issue. (what you have listed is different to what I listed you don't have a unique key) Option 2 means you loose any ability to reference page, but as you

Re: unique terms and multi-valued fields

Well, it depends (tm). If you're talking about *indexed* terms, then the value is stored only once in both the cases you mentioned below. There's really very little difference between a non-multi-valued field and a multi-valued field in terms of how it's stored in the searchable portion of the ind

[Help Wanted] Graphics and other help for new Lucene/Solr website

Hi, We are in the process of putting up a new Lucene/Solr/PyLucene/OpenRelevance website. You can see a preview at http://lucene.staging.apache.org/lucene/. It is more or less a look and feel copy of Mahout and Open For Biz websites. This new site, IMO, both looks better than the old one and

RE: How come this query string starts with wildcard?

I think this is because ")" is treated as a token delimiter. So "(foo)bar" is treated the same as "(foo) bar" (that is, bar is treated as a separate word). So "(foo)*" is really parsed as "(foo) *" and thus the * is treated as the start of a new word. -Michael

Re: Date faceting per last hour, three days and last week

I would use facet queries: facet.query=date:[NOW-1DAY TO NOW] facet.query=date:[NOW-3DAY TO NOW] facet.query=date:[NOW-7DAY TO NOW] -- View this message in context: http://lucene.472066.n3.nabble.com/Date-faceting-per-last-hour-three-days-and-last-week-tp3242364p3242574.html Sent from the Solr -

How come this query string starts with wildcard?

While going through my error logs of Solr, i found that a user had fired a query - jawapan ujian bulanan thn 4 (bahasa melayu). This was converted to following for autosuggest purposes - jawapan?ujian?bulanan?thn?4?(bahasa?melayu)* by the javascript code. Solr threw the exception Cannot parse 'jaw

paging size in SOLR

hi, i want to retrieve all the data from solr (say 10,000 ids ) and my page size is 1000 . how do i get back the data (pages) one after other ?do i have to increment the "start" value each time by the page size from 0 and do the iteration ? In this case am i querying the index 10 time instead of on

Date faceting per last hour, three days and last week

Hi, I'm trying date faceting per last 24 hours, three days and last week, but I don't know how to do it. I have a DateField and I want to set different ranges, it is posible? I understand the example from solr wiki

Re: document indexing

Could you please tell me schema.xml "fields" tag content for such case? Currently index data is something like this: PageID BookID Text 1 1"some text" 2 1"some text" 3 1"some text" 4 1"some text" 5 2"some text" 6

Re: Possible bug in FastVectorHighlighter

Worked fine. Thanks a lot! Massimo On 09/08/2011 11:58, Jayendra Patil wrote: Try using - Regards, Jayendra On Tue, Aug 9, 2011 at 4:46 AM, Massimo Schiavon wrote: In my Solr (3.3) configuration I specified these two params: when I do a simple search I obtain correctly h

RE: Trying to index pdf docs - lazy loading error - ClassNotFoundException: solr.extraction.ExtractingRequestHandler

I have had a mistake with the configs files. From the example directory all works correctly. Thanks to all. --- Rode González Libnova, SL Paseo de la Castellana, 153-Madrid [t]91 449 08 94 [f]91 141 21 21 www.libnova.es > -Mensaje original- > De: Rode González [mailto:r...@libnova.es] >

frange not working in query

Hi All, I am trying to sort the results on a unix timestamp using this query. http://url.com:8983/solr/db/select/?indent=on&version=2.1&q={!frange%20l=0.25}query($qq)&qq=nokia&sort=unix-timestamp%20desc&start=0&rows=10&qt=dismax&wt=dismax&fl=*,score&hl=on&hl.snippets=1 When I run this query, it

Re: document indexing

It really does depend upon what you want to do in your app but from the info given I'd go for denormalizing by repeating the least number of values. So in your case that would be book PageID+BookID(uniqueKey), pageID, PageVal1, PageValn, BookID, BookName On 10 August 2011 09:46, directorscott

Re: Is optimize needed on slaves if it replicates from optimized master?

That is not true. Replication is roughly a copy of the diff between the >> master and the slave's index. > > In my case, during replication entire index is copied from master to slave, during which the size of index goes a little over double. Then it shrinks to its original size. Am I doing somethi

document indexing

Hello, First of all, I am a beginner and i am trying to develop a sample application using SolrNet. I am struggling about schema definition i need to use to correspond my needs. In database, i have Books(bookId, name) and Pages(pageId, bookId, text) tables. They have master-detail relationship. I

Re: Is optimize needed on slaves if it replicates from optimized master?

Sure there is actually no optimizing on the slave needed, but after calling optimize on the slave the write.lock will be removed. So why is the replication process not doing this? Regards Bernd Am 10.08.2011 10:57, schrieb Shalin Shekhar Mangar: On Wed, Aug 10, 2011 at 1:11 PM, Bernd Fehling<

Re: Is optimize needed on slaves if it replicates from optimized master?

On Wed, Aug 10, 2011 at 1:11 PM, Bernd Fehling < bernd.fehl...@uni-bielefeld.de> wrote: > > From what I see on my slaves, yes. > After replication has finished and new index is in place and new reader has > started > I have always a write.lock file in my index directory on slaves, even > though th

Re: Is optimize needed on slaves if it replicates from optimized master?

From what I see on my slaves, yes. After replication has finished and new index is in place and new reader has started I have always a write.lock file in my index directory on slaves, even though the index on master is optimized. Regards Bernd Am 10.08.2011 09:12, schrieb Pranav Prakash: Do

Re: Indexing tweet and searching "@keyword" OR "#keyword"

I tried tweaking "WordDelimiterFactory" but I won't accept # OR @ symbols and it ignored totally. I need solution plz suggest. On 4 August 2011 21:08, Jonathan Rochkind wrote: > It's the WordDelimiterFactory in your filter chain that's removing the > punctuation entirely from your index, I think