SolrCloud performance in VM environment

2013-10-21 Thread Tom Mortimer
Hi everyone, I've been working on an installation recently which uses SolrCloud to index 45M documents into 8 shards on 2 VMs running 64-bit Ubuntu (with another 2 identical VMs set up for replicas). The reason we're using so many shards for a relatively small index is that there are complex filte

Re: SolrCloud performance in VM environment

2013-10-22 Thread Tom Mortimer
Boogie, Shawn, Thanks for the replies. I'm going to try out some of your suggestions today. Although, without more RAM I'm not that optimistic.. Tom On 21 October 2013 18:40, Shawn Heisey wrote: > On 10/21/2013 9:48 AM, Tom Mortimer wrote: > >> Hi everyone, >>

Re: SolrCloud performance in VM environment

2013-10-22 Thread Tom Mortimer
Just tried it with no other changes than upping the RAM to 128GB total, and it's flying. I think that proves that RAM is good. =) Will implement suggested changes later, though. cheers, Tom On 22 October 2013 09:04, Tom Mortimer wrote: > Boogie, Shawn, > > Thanks for the repl

eDisMax and Boolean operator case-sensitivity

2013-11-06 Thread Tom Mortimer
Hi, I'm using eDisMax query parser, and need to support Boolean operators AND and OR. It seems from testing that these are *not* case sensitive, e.g. setting mm to 0, "oscar AND wilde" returns the same results as "oscar and wilde" (15 hits) while "oscar foo wilde" returns the same results as "osca

Re: eDisMax and Boolean operator case-sensitivity

2013-11-06 Thread Tom Mortimer
Oh, good grief - I was just reading that page, how did I miss that? *derp* Thanks Shawn!!! Tom On 6 November 2013 18:59, Shawn Heisey wrote: > On 11/6/2013 11:46 AM, Tom Mortimer wrote: > >> I'm using eDisMax query parser, and need to support Boolean operators AND >>

eDisMax, multiple language support and stopwords

2013-11-07 Thread Tom Mortimer
Hi all, Thanks for the help and advice I've got here so far! Another question - I want to support stopwords at search time, so that e.g. the query "oscar and wilde" is equivalent to "oscar wilde" (this is with lowercaseOperators=false). Fair enough, I have stopword "and" in the query analyser cha

Re: eDisMax, multiple language support and stopwords

2013-11-07 Thread Tom Mortimer
Ah, thanks Markus. I think I'll just add the Boolean operators to the stopwords list in that case. Tom On 7 November 2013 12:01, Markus Jelsma wrote: > This is an ancient problem. The issue here is your mm-parameter, it gets > confused because for separate fields different amount of tokens ar

Re: newbie getting started with solr

2013-11-07 Thread Tom Mortimer
Hi Eric, Solr configuration can certainly be confusing at first. And for some time after. :P If you're running start.jar from the example folder (which is fine for testing, and I've known some people to use it for production systems) then the default solr home is example/solr. This contains solr

Restricting search results by field value

2012-12-05 Thread Tom Mortimer
Hi everyone, I've got a problem where I have docs with a source_id field, and there can be many docs from each source. Searches will typically return docs from many sources. I want to restrict the number of docs from each source in results, so there will be no more than (say) 3 docs from source

Re: Restricting search results by field value

2012-12-06 Thread Tom Mortimer
t; > > > On 12/05/2012 12:04 PM, Tom Mortimer wrote: >> Hi everyone, >> >> I've got a problem where I have docs with a source_id field, and there can >> be many docs from each source. Searches will typically return docs from many >> sources. I want to

Re: Restricting search results by field value

2012-12-06 Thread Tom Mortimer
p.main=true > > On Thu, Dec 6, 2012 at 2:35 AM, Tom Mortimer wrote: > >> Sounds like it's worth a try! Thanks Andre. >> Tom >> >> On 5 Dec 2012, at 17:49, Andre Bois-Crettez wrote: >> >>> If you do grouping on source_id, it should be enoug

copyField at search time / multi-language support

2011-03-28 Thread Tom Mortimer
Hi, Here's my problem: I'm indexing a corpus with text in a variety of languages. I'm planning to detect these at index time and send the text to one of a suitably-configured field (e.g. "mytext_de" for German, "mytext_cjk" for Chinese/Japanese/Korean etc.) At search time I want to search all of

Highlighting not working

2011-04-07 Thread Tom Mortimer
Hi, I'm having trouble getting highlighting to work for a large text field. This field can be in several languages, so I'm sending it to one of several fields configured appropriately (e.g. "cv_text_en") and then copying it to a common field for storage and display ("cv_text"). The relevant fragme

Re: Highlighting not working

2011-04-07 Thread Tom Mortimer
I guess what I'm asking is - can Solr highlight non-indexed fields? Tom On 7 April 2011 11:33, Tom Mortimer wrote: > Hi, > > I'm having trouble getting highlighting to work for a large text > field. This field can be in several languages, so I'm sending it to > on

Re: Highlighting not working

2011-04-07 Thread Tom Mortimer
Problem solved. *bangs head on desk* T On 7 April 2011 11:33, Tom Mortimer wrote: > Hi, > > I'm having trouble getting highlighting to work for a large text > field. This field can be in several languages, so I'm sending it to > one of several fields configured approp

Error loading a custom request handler in Solr 4.0

2011-08-10 Thread Tom Mortimer
Hi, Apologies if this is really basic. I'm trying to learn how to create a custom request handler, so I wrote the minimal class (attached), compiled and jar'd it, and placed it in example/lib. I added this to solrconfig.xml: When I started Solr with java -jar start.jar, I got this: ...

Re: Error loading a custom request handler in Solr 4.0

2011-08-10 Thread Tom Mortimer
n "Flax"; } public String getVersion() { return "Flax"; } } On 10 August 2011 16:43, simon wrote: > Th attachment isn't showing up (in gmail, at least). Can you inline > the relevant bits of code ? > > On Wed, Aug 10, 2011 at 11:05 AM, Tom Mortimer wrote: >

Re: how to ignore case in solr search field?

2011-08-10 Thread Tom Mortimer
You can use solr.LowerCaseFilterFactory in an analyser chain for both indexing and queries. The schema.xml supplied with example has several field types using this (including "text_general"). Tom On 10 August 2011 16:42, nagarjuna wrote: > Hi please help me .. >how to ignore case w

Re: Error loading a custom request handler in Solr 4.0

2011-08-10 Thread Tom Mortimer
ib, added the config > line to solrconfig. > > when I send a /flaxtest request i get > > > > 0 > 16 > > Hello! > > > I was doing this within a core defined in solr.xml > > -Simon > > On Wed, Aug 10, 2011 at 11:46 AM, Tom Mortimer wrote: &g

Re: Error loading a custom request handler in Solr 4.0

2011-08-10 Thread Tom Mortimer
'ant dist' > > I'd try again with a clean trunk . > > -Simon > > On Wed, Aug 10, 2011 at 1:20 PM, Tom Mortimer wrote: > > Interesting.. is this in trunk (4.0)? Maybe I've broken mine somehow! > > > > What classpath did you use for compiling?

Re: Solr4 how to make it do this?

2012-09-18 Thread Tom Mortimer
Hi George, I don't think this will work. The synonyms will be added after the query is parsed, so you'll have terms like "bed:3" rather than matching "3" against the bed field. If I was implementing this I'd try doing some pattern matching before passing the query to Solr, e.g.: "3 bed

Re: Personalized Boosting

2012-09-18 Thread Tom Mortimer
Hi, Would this do the job? http://wiki.apache.org/solr/QueryElevationComponent Tom On 18 Sep 2012, at 01:36, deniz wrote: > Hello All, > > I have a requirement or a pre=requirement for our search application. > Basically the engine will be on a website with plenty of users and more than > 2

Re: Personalized Boosting

2012-09-19 Thread Tom Mortimer
I'm still not sure I understand what it is you're trying to do. Index-time or query-time boosts would probably be neater and more predictable than multiple field instances, though. http://wiki.apache.org/solr/UpdateXmlMessages#Optional_attributes_for_.22field.22 http://wiki.apache.org/solr/SolrR

Re: ID reference field - Needed but not searchable or retrievable

2012-09-20 Thread Tom Mortimer
Hi James, If you don't want this field to be included in user searches, just omit it from the search configuration (e.g. if using eDisMax parser, don't put it in the qf list). To keep it out of search results, exclude it from the fl list. See http://wiki.apache.org/solr/CommonQueryParam

Solr 4.0 - disappointing results sharding on 1 machine

2012-09-20 Thread Tom Mortimer
Hi all, After reading http://carsabi.com/car-news/2012/03/23/optimizing-solr-7x-your-search-speed/ , I thought I'd do my own experiments. I used 2M docs from wikipedia, indexed in Solr 4.0 Beta on a standard EC2 large instance. I compared an unsharded and 2-shard configuration (the latter set

Re: Solr 4.0 - disappointing results sharding on 1 machine

2012-09-20 Thread Tom Mortimer
Before anyone asks, these results were obtained warm. On 20 Sep 2012, at 14:39, Tom Mortimer wrote: > Hi all, > > After reading > http://carsabi.com/car-news/2012/03/23/optimizing-solr-7x-your-search-speed/ > , I thought I'd do my own experiments. I used 2M docs from wiki

Re: AutoIndexing

2012-09-25 Thread Tom Mortimer
Hi Darshan, Can you give us some more details, e.g. what do you mean by "database"? A RDBMS? Which software? How are you indexing it (or intending to index it) to Solr? etc... cheers, Tom On 25 Sep 2012, at 09:55, darshan wrote: > Hi All, > >Is there any way where I can aut

Re: How can I create about 100000 independent indexes in Solr?

2012-09-25 Thread Tom Mortimer
Hi, Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries? Tom On 25 Sep 2012, at 03:21, 韦震宇 wrote: > Dear all, >The company I'm working in have a website to server more than 10 > customers, and every customer

Re: AutoIndexing

2012-09-25 Thread Tom Mortimer
t; -----Original Message- > From: Tom Mortimer [mailto:tom.m.f...@gmail.com] > Sent: Tuesday, September 25, 2012 3:31 PM > To: solr-user@lucene.apache.org > Subject: Re: AutoIndexing > > Hi Darshan, > > Can you give us some more details, e.g. what do you mean by "