document id in nutch/solr

A quite standard configuration of nutch seems to autoamtically map "url" to "id". Two questions: - Where is such mapping defined? I can't find it anywhere in nutch-site.xml or schema.xml. The latter does define the "id" field as well as its uniqueness, but not the mapping. - Given that nutch nutc

Re: solr spatial search with distance to search results

You've specified the field _type_ not the field _name_ in your fl list. try fl=geolocation blah blah And try raw goedist() rather than _dist_:geodist() maybe? Best Erick On Wed, Jun 19, 2013 at 1:23 PM, PeterKerk wrote: > I was reading this: http://wiki.apache.org/solr/SpatialSearch > > I have

Re: Wildcards and Phrase queries

Wouldn't imagine they're production ready, they haven't been touched in months. So I'd say you're on your own here in terms of whether you wanted to use these for production. I confess I don't know what state they were left in or why they were never committed. FWIW, Erick On Wed, Jun 19, 2013 a

Re: what does a zero score mean?

So, the reason is taht I'm getting zero values on FieldNorm. documentation tells me that there are 3 factors in play here: - LengthNorm --> can this be zero? - index-time boost --> is this the boost value we get from nutch? - field-boost --> none specified. Can somebody help here? On Tue, Jun 1

Re: Sharding and Replication

Erick, Thanks for your reply. You are right about 10 updates being batch up - It was hard to figure out due to large number of updates/logging that happens in our system. We are batching 1000 updates every time. Here is my observation from leader and replica - 1. Leader logs are clearly indica

Re: Adding documents in Solr plugin

: This could be a very useful feature. To do it properly, you'd want some : new update syntax, extending that of the atomic updates. That is, a new : custom request handler could do it, but might now be the best way. the biggest complexity to implementing this in a general way would be dealing w

Re: Can we solr serve data to any reporting engine? like Crystal, Japser, Birt etc

Wonderful. Thanks a lot everyone.. Solr rocks --Pradeep On 21-Jun-2013 11:54 PM, "Gora Mohanty" wrote: > On 21 June 2013 23:14, pradeep kumar wrote: > > Actually i tried same. I asked the same questions in jasper report forum > > first and i got similar answer.. > > I saw unanswered or not compl

Re: shardkey

On Fri, Jun 21, 2013 at 6:08 PM, Joshi, Shital wrote: > But now Solr stores composite id in the document id Correct, it's the document id itself that contains everything needed for tje compositeId router to determine the hash. > It would only use it to calculate hash key but while storing compo

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Here's a top-level TOC (I couldn't fit this into Lulu's 1024 char description limit!): Part I. Introduction to Solr Ch 1. Introduction Ch 2. Basic Concepts and Terms Ch 3. Solr Tutorial Ch 4. Solr as a WebApp Ch 5. Request/Response Model Ch 6. Aggregate Functions Part II. Data Definition

Re: another transaction log + commit question

I wouldn't expect the transaction log to go to 0 because the transaction log is kept around for "peer synch" so it can catch up another node that is a little behind. But I _would_ expect it to roll over. So my question is whether the files within your log directories roll over. What you should see

Re: How to dynamically add geo fields to a query using a request handler

What version of Solr are you using? I don't see any SolrQuery.set(). But with 4.x I think you want solrQuery.addValueToParam(); On Wed, Jun 19, 2013 at 8:06 AM, ade-b wrote: > Hi > > We have a request handler defined in solrconfig.xml that specifies a list of > fields to return for the request us

Re: Solr Cloud Hangs consistently .

Thanks for letting us know! Erick On Wed, Jun 19, 2013 at 7:18 AM, Rishi Easwaran wrote: > Update!! > > Got SOLR cloud working, was able to do 90k document inserts with > replicationFactor=2, with my jmeter script, previously was getting stuck with > 3k inserts or less. > After some investigat

Re: Sharding and Replication

Update are batched, but it's on a per-request basis. So, if you're sending one document at a time you'll won't get any batching. If you send 10 docs at a time and they happen to go to 10 different shards, you'll get 10 different update requests. If you're sending 1,000 docs per update you' should

RE: shardkey

Thanks for answering my questions on shardkey. We experimented with implicit router and it works like you said. We're experimenting with composite id router. This document http://searchhub.org/2013/06/13/solr-cloud-document-routing/ says: "A shard key can be pre-pended to the unique document id

Re: SPLITSHARD throws error

On 6/21/2013 3:06 PM, Joshi, Shital wrote: This is what we have in solr.xml. We use one solr.xml and pass dataDir, shard as java system property. (e.g.: -Dsolr.data.dir=$HOME/solr_data/solr6 -Dsolr.ulog.dir=$HOME/solr_data/solr6_tranlog -DnumShards=5 -Dshard=shard1) There'

RE: SPLITSHARD throws error

This is what we have in solr.xml. We use one solr.xml and pass dataDir, shard as java system property. (e.g.: -Dsolr.data.dir=$HOME/solr_data/solr6 -Dsolr.ulog.dir=$HOME/solr_data/solr6_tranlog -DnumShards=5 -Dshard=shard1) -Original Message- From: Shawn Heisey [mail

Re: SPLITSHARD throws error

On 6/21/2013 2:26 PM, Joshi, Shital wrote: Hi, We have 5 shards with replication factor 2 (total 10 jvm instances). Our shards are named (shardid) shard1,shard2,shard3,shar4 and shar5 and collection name is collection1. When we execute this command: curl --proxy '' "http://$HOST_NAME:8983/so

SPLITSHARD throws error

Hi, We have 5 shards with replication factor 2 (total 10 jvm instances). Our shards are named (shardid) shard1,shard2,shard3,shar4 and shar5 and collection name is collection1. When we execute this command: curl --proxy '' "http://$HOST_NAME:8983/solr/admin/collections?action=SPLITSHARD&collec

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Great suggestion. I was going to do AutoCompete way back in February, but SpellCheck took way too long and I just needed to move on (to solrconfig.) I'll try to plan it for Stage 2 (mid July), if not sooner. Note: There is an autocomplete example in the Tutorial section of the book (Page 49, I

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

As one of the early reviewers of the manuscript, I always had high hopes for this work. I now have the pdf from lulu; do not have time now to dive deeply, but will comment that it seems, to me at least, well worth owning. Jack On Fri, Jun 21, 2013 at 11:41 AM, Jack Krupansky wrote: > Okay, it's

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Hi Jack, Just bought the book. One thing I'd love to see in the next edition, based on your list of candidates above is: - Autocomplete deep dive I've been working implementing this recently and it's much more complex than just dropping in the Suggester as per the wiki as I'm sure you know. A di

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

I'll work on that and see if I can put it on Lulu in the description. -- Jack Krupansky -Original Message- From: Alexandre Rafalovitch Sent: Friday, June 21, 2013 3:08 PM To: solr-user@lucene.apache.org Subject: Re: The book: Solr 4.x Deep Dive - Early Access Release #1 On Fri, Jun 21

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

On Fri, Jun 21, 2013 at 2:41 PM, Jack Krupansky wrote: > Here are the topics that are NOT in the current early-access edition: Congratulations. Is there a full (top-level) table-of-content somewhere? Lulu's preview cuts off too early because the TOC is deeply nested. Regards, Alex. Personal

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Pricing model is still up for grabs. For now, it's just $10 a shot. No fancy plan or anything. I don't expect most people to want an update every month, maybe a couple of times a year. Or, basically, when some major change occurs, like I add coverage of an area that wasn't covered before, like

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

On 22 June 2013 00:11, Jack Krupansky wrote: > Okay, it's DONE. Here's the Lulu link, ready to go: > > http://www.lulu.com/shop/jack-krupansky/solr-4x-deep-dive-early-access-release-1/ebook/product-21079719.html > > (Or, go to Lulu.com and just search for "Solr" - It's the only hit so far.) > > Pr

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Okay, it's DONE. Here's the Lulu link, ready to go: http://www.lulu.com/shop/jack-krupansky/solr-4x-deep-dive-early-access-release-1/ebook/product-21079719.html (Or, go to Lulu.com and just search for "Solr" - It's the only hit so far.) Price is $9.99 for now (I get $8.10 of that, BTW, in case

RE: Restarting SOLR will remove all cache?

Restarting Solr won't clear the disk cache. When I'm doing perf testing, I'll sometimes run this on the server before each test to clear out the disk cache: echo 1 > /proc/sys/vm/drop_caches -Michael -Original Message- From: Learner [mailto:bbar...@gmail.com] Sent: Friday, June 21, 201

Re: Can we solr serve data to any reporting engine? like Crystal, Japser, Birt etc

On 21 June 2013 23:14, pradeep kumar wrote: > Actually i tried same. I asked the same questions in jasper report forum > first and i got similar answer.. > I saw unanswered or not completed thread in other forums related to solr. Um, the very first link when searching Google for "Solr Jasper repo

Re: Can we solr serve data to any reporting engine? like Crystal, Japser, Birt etc

You have a catch-22 and lack of cross-cutting experts present on the list (at least for now). I think you need to drop down to the level of features. For example, Solr can output flat CSV. Can Jasper consume CSV from a custom URL? If so, you can get at least some data from one to another. Can Jas

Re: Solr4 cluster setup for high performance reads

Thanks for the update guys, I am working on the suggestions shared by you. One last question about the solrcloud setup. What is the recommended cluster size for solrcloud? I have 3 nodes of solr and 3 nodes of ZK (running on the same machine, but a different JVM). And after 2-3 days I notice that

Re: Restarting SOLR will remove all cache?

There are no disk caches as such. There is no point in testing without caches. Also, Lucene has field caches required for sorting which cannot be turned off. On Fri, Jun 21, 2013 at 11:22 PM, Learner wrote: > I have a very simple question. Does restarting SOLR removes all caches > (including disk

Restarting SOLR will remove all cache?

I have a very simple question. Does restarting SOLR removes all caches (including disk caches if any?). I have disabled all caches in solrconfig.xml but even then I see that there is some caching happening all the time. I am currently doing some performance testing and I dont want cache to play a

Re: Can we solr serve data to any reporting engine? like Crystal, Japser, Birt etc

Actually i tried same. I asked the same questions in jasper report forum first and i got similar answer.. I saw unanswered or not completed thread in other forums related to solr. Anyone has any idea about using solr with any search engines? On Fri, Jun 21, 2013 at 8:44 PM, Gora Mohanty wrote:

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Thanks for the feedback. Ah... those magic words: "When it's done"! In all honesty my book will never be "done" - in the sense that Solr will never be "done". This is the problem with a print book - it's "finished", literally and figuratively. We have 4.3.1, with 4.4 coming soon, then... I w

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

I'll definitely take a look at LeanPub as a potential path forwards beyond my initial publication on Lulu. My criteria for "Stage 0" and "Stage 1" is simply: Get it out there ASAP! Then, based on feedback, decide what makes most sense to pursue. You know, agile - incremental progress. -- Jack

Re: update solr.xml dynamically to add new cores

Gr8! thanks a lot! -- View this message in context: http://lucene.472066.n3.nabble.com/update-solr-xml-dynamically-to-add-new-cores-tp4071800p4072190.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

On 6/21/2013 7:04 AM, Jack Krupansky wrote: None of this is cast in stone. My current thought is to publish this initial release at $4.99 or $7.99, and then set a revised price for the second or third release. Recently, I bought this book: http://www.manning.com/grainger/ I did the MEAP plu

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Thanks for the feedback. I'm not sure if I can update the PDF for an existing SKU, or if I would have to put out each release as new SKU. I'm expecting the latter. I'll find out soon enough. I still intend to eventually do the multi-volume edition, including print, but first things first - and

Re: Varnish

I presume you mean https://www.varnish-cache.org/ That's the first I'd heard of it. Thanks Jack On Thu, Jun 20, 2013 at 10:48 PM, William Bell wrote: > Who is using varnish in front of SOLR? > > Anyone have any configs that work with the cache control headers of SOLR? > > -- > Bill Bell > billnb

Re: Issues in the Fuzzy Query !

Hi Mark, What you said is correct . But still if I do an AND operation the fussy query doesn't work.Surprisingly OR is working perfectly but that does not give me what I want. {!lucene q.op=OR}(TRIGRAM_FIELDNAME1:(FRANK INDIANO))^0.82 OR pkey:"" OR ((FIELDNAME1:FRANK~2)* AND *(FIELDNAME1:INDIANO

Re: Informal poll on running Solr 4 on Java 7 with G1GC

On 6/20/2013 10:22 PM, William Bell wrote: > It would be good to see some CMS configs too... Can you send your java > params? Here is what we use in production. We run multiple collections with small documents. One is 3M docs, one is 9M, one is 2M, and the other three are small. We use Amazon m

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

I'm sure people will pay 10$ ;-) Am 21.06.2013 um 15:12 schrieb Ralf Heyde: > I think people will pay 10$ :) > > > On 06/21/2013 03:04 PM, Jack Krupansky wrote: >> I’m expecting to self-publish the first Early Access Release for my book, >> Solr 4.x Deep Dive, on lulu.com sometime today. It is

Re: update solr.xml dynamically to add new cores

Yes, that makes sense. It's what I do, anyway! :) Michael Della Bitta Applications Developer o: +1 646 532 3062 | c: +1 917 477 7906 appinions inc. “The Science of Influence Marketing” 18 East 41st Street New York, NY 10017 t: @appinions | g+: plus.google.co

Re: Can we solr serve data to any reporting engine? like Crystal, Japser, Birt etc

On 21 June 2013 20:34, pradeep kumar wrote: > Hello Solr users, > > I need some information about using solr for Report & analytics. > > Most importantly, my is question whether solr can serve data to any > reporting engine? like Crystal, JapserReport, Birt, DynamicReports etc. [...] You should p

Re: SolrCloud: no "timing" when no result in distributed mode

On 6/21/2013 8:53 AM, Elodie Sannier wrote: > Unfortunately I cannot use the 4.3.0 version because I'm using solr.xml > sharedLib which does not work in 4.3.0 (cf. > https://issues.apache.org/jira/browse/SOLR-4791). You have two options: Download the 4.3.0 source code, patch it with the patch att

Can we solr serve data to any reporting engine? like Crystal, Japser, Birt etc

Hello Solr users, I need some information about using solr for Report & analytics. Most importantly, my is question whether solr can serve data to any reporting engine? like Crystal, JapserReport, Birt, DynamicReports etc. If yes how can we use? Any examples/articles/blogs? Could some one explai

Re: Issues in the Fuzzy Query !

In Solr 4.x, FuzzyQuery has switch to using an integer editing distance, with a maximum of 2. Your 0.8 may have gotten mapped to a distance of 0, or maybe 1. Switch to using 1 or 2. This new implementation is much faster (FST), even if it is somewhat less flexible. See: http://lucene.apache.or

Re: SolrCloud: no "timing" when no result in distributed mode

Unfortunately I cannot use the 4.3.0 version because I'm using solr.xml sharedLib which does not work in 4.3.0 (cf. https://issues.apache.org/jira/browse/SOLR-4791). Elodie On 06/21/2013 03:30 PM, James Thomas wrote: Seems to work fine for me on 4.3.0, maybe you can try a newer version. 4.3.1 i

SolrCloud: 500 error with combination of debug and group in distributed search

Hello, I am using SolrCloud 4.2.1 with two shards, when I'm grouping on a field and using the "debug" parameter in distributed mode, I have a 500 error. http://localhost:8983/solr/select?q=*:*&group=true&group.field=popularity&debug=true (idem with debug=timing, query or results) Serv

RE: Issues in the Fuzzy Query !

Hi , Below is what I am passing to solr .Only the fuzzy part is creating the problem for me- {!lucene q.op=OR}(TRIGRAM_FIELDNAME1:(FRANK INDIANO))^0.82 OR pkey:"" OR ((FIELDNAME1:FRANK~0.80) AND (FIELDNAME1:INDIANO~0.80))^0.80 Also the field type has whitespace,synonym,lowercase and stopwords in

Grouping and Sorting with shards

is an example, the id field is something like "identifier!date_format" doc { id: doc1!20130618_formatX docId: doc1 dateReceived: 20130620 } doc { id: doc1!20130621_formatY docId: doc1 dateReceived: 20130621 } doc { id: doc2!20130619_formatX docId: doc2 dateReceived: 20130619 } So

RE: Issues in the Fuzzy Query !

Hello, Can you share the exact params you are passing to solr? Thanks From: vibhoreng04 [vibhoren...@gmail.com] Sent: June 21, 2013 9:27 AM To: solr-user@lucene.apache.org Subject: Issues in the Fuzzy Query ! Hi All, I have been facing problems in the f

Solr replication is extremely slow(less then 1MB/s)

Solr replication is extremely slow(less then 1MB/s) When the replication is runinng,network and disk occupancy rate remained at a very low level. I've tried downloading a piece of the index file with browser. like this: http://master/solr/product/replication?command=filecontent&wt=filestream&ind

Solr shards & (reverse) proxy

Hello, I have this kind of url : "http://remoteserver/solr/leg0/select/?rows=10&version=2&fl=* &shards= remoteserver:80/solr/core0, ... remoteserver:80/solr/core5, ... remoteserver:80/solr/core9 &.. There is only ONE Solr instance with multiple cores (core0 to core9) on the same server.

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Yeah I'd rather just pay more up front, unless the "revisions" are more like additional volumes. On Fri, Jun 21, 2013 at 9:57 AM, Swati Swoboda wrote: > I'd be willing to pay $30-$40 initial cost, but would expect to be able to > get the revisions for no cost then. > > With the revision model, I

RE: The book: Solr 4.x Deep Dive - Early Access Release #1

I'd be willing to pay $30-$40 initial cost, but would expect to be able to get the revisions for no cost then. With the revision model, I'd pay the initial $10 and then $3-$9 per revision (depending on what's in the revision). It's an interesting model then, because I can choose to not purchase

Re: Solr, Shards, multi cores and (reverse proxy)

On 6/21/2013 3:32 AM, medley wrote: > I have this kind of url : > > "http://remoteserver/solr/leg0/select/?rows=10&version=2&fl=* > &shards= > remoteserver:80/solr/core0, > ... > remoteserver:80/solr/core5, > ... > remoteserver:80/solr/core9 > &.. > > There is only ONE Solr instance with

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Consider https://leanpub.com/ for publishing. I'm in no way affiliated with them, just have positive personal buying experience. One can and I regularly give more than what author requested as min price. For such a work of 1k pages, would definitely pay more than $10. Kind regards, Stevo Slavic.

Re: Solr Suggest does not work in solrcloud environment

Hi Simon, Good that it works. The reason as far as I could make out is that by itself/ standalone the SpellCheckComponent (used by the suggester) is not distributed. One way to explicitly distribute the search is to provide the shards: http://wiki.apache.org/solr/SpellCheckComponent#Distributed_S

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

On 6/21/2013 9:22 AM, Alexandre Rafalovitch wrote: I might be however confused regarding your strategy. I thought you were going to do several different volumes, rather than one large one. Or is this all a 'first' volume discussion so far. Pricing: $7.99 feels better for the book this size. U

RE: SolrCloud: no "timing" when no result in distributed mode

Seems to work fine for me on 4.3.0, maybe you can try a newer version. 4.3.1 is available. -Original Message- From: Elodie Sannier [mailto:elodie.sann...@kelkoo.fr] Sent: Friday, June 21, 2013 8:54 AM To: solr-user@lucene.apache.org >> "solr-user@lucene.apache.org" Subject: SolrCloud: no

Issues in the Fuzzy Query !

Hi All, I have been facing problems in the fuzzy queries.For an example if I query ((FIELDNAME1:FRANK~0.80) AND (FIELDNAME1:INDIANO~0.80))^0.80 , the parsed query changes my distance grade to ~0 etc.Also in the other cases the distance is changed in the pared query like ~1 and ~2.Can anyone tell

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Does Lulu actually supports recurrent charge? And if it is PDF only, I am not sure Lulu is the best option for you (their focus is on print after all). But I haven't done any research. The way I have seen this done is to have a single high-price for all the updates but give early/beta buyers a dis

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

Yes Jack ! I think Solr is still hot in market and there is less material available .So a prize tag of 10$ will be ok. -- View this message in context: http://lucene.472066.n3.nabble.com/The-book-Solr-4-x-Deep-Dive-Early-Access-Release-1-tp4072118p4072123.html Sent from the Solr - User mailing

Re: The book: Solr 4.x Deep Dive - Early Access Release #1

I think people will pay 10$ :) On 06/21/2013 03:04 PM, Jack Krupansky wrote: I’m expecting to self-publish the first Early Access Release for my book, Solr 4.x Deep Dive, on lulu.com sometime today. It is still far from finished and needs lots of work and missing a lot of important areas (Sol

The book: Solr 4.x Deep Dive - Early Access Release #1

I’m expecting to self-publish the first Early Access Release for my book, Solr 4.x Deep Dive, on lulu.com sometime today. It is still far from finished and needs lots of work and missing a lot of important areas (SolrCloud and distributed Solr in general, DIH, highlighting, core and collection A

SolrCloud: no "timing" when no result in distributed mode

Hello, I am using SolrCloud 4.2.1 with two shards, with the "debugQuery=true" parameter, when a query does not return documents then the "timing" debug information is not returned: curl -sS "http://localhost:8983/solr/select?q=dummy&debugQuery=true"; | grep -o '.*' If i use the "distrib=false"

AW: DataImportHandler: Problems with delta-import and CachedSqlEntityProcessor

Hi, thanks for the other ideas. I worked around the problem with the idea of Paul Noble. This is working really fine for me right now. My full-import is at around 40 minutes and my delta-import runs in less than 10 seconds, because it runs every minute. So that configuration seems to be prett

Re: Solr, Shards, multi cores and (reverse proxy)

If they are running on port 8983, then you just use localhost:8983 instead of remoteserver:80. Those URLs will be used from the solr server itself, so localhost should work. Upayavira On Fri, Jun 21, 2013, at 10:32 AM, medley wrote: > Hello, > > I have this kind of url : > > "http://remoteserve

Solr, Shards, multi cores and (reverse proxy)

Hello, I have this kind of url : "http://remoteserver/solr/leg0/select/?rows=10&version=2&fl=* &shards= remoteserver:80/solr/core0, ... remoteserver:80/solr/core5, ... remoteserver:80/solr/core9 &.. There is only ONE Solr instance with multiple cores (core0 to core9) on the same server.

Re: Informal poll on running Solr 4 on Java 7 with G1GC

On 6/20/2013 10:22 PM, William Bell wrote: > It would be good to see some CMS configs too... Can you send your java > params? I've got it documented on my personal wiki page: https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning Thanks, Shawn

Re: Queuing for Solr Updates?

On 6/20/2013 11:42 PM, William Bell wrote: > Is there a simpler way to kick off a DIH handler update when it is running? > > Scenario: > > 1. Doing an update using DIH > 2. We need to kick off another update. Cannot since DIH is already running. > So the program inserts into a table (ID=55) > 3.