Re: Calculating Solr document score by ignoring the field.

Sorry to repeat Jacks' previous answer but x times zero is always zero :) A index boost is just what the name suggests, a factor by which the document score is boosted (multiplied). Since it is in an index time value, it is stored alongside the document, so any future scoring of the document by an

Re: Calculating Solr document score by ignoring the field.

Ok thanks, I just wanted the know is it possible to ignore boost value or not during score calculation and as you said its not. Now I would have to focus on nutch to fix the issue and not to send boost=0 to Solr. Regards, Khan On Wed, Jul 10, 2013 at 12:14 PM, Daniel Collins wrote: > Sorry to r

Re: Solr Hangs During Updates for over 10 minutes

We are planning an upgrade to 4.4 but it's still weeks out. We offer a high availability search service and there are a number of changes in 4.4 that are not backward compatible. (i.e. Clusterstate.json and no solr.xml) So there must be lots of testing, additionally this upgrade cannot be performed

Re: Solr limitations

I understand, thanks. I just wanted to check in case there were scalability limitations with how SolrCloud operates.. On 9 Jul 2013 12:45, "Erick Erickson" wrote: > I think Jack was mostly thinking in "slam dunk" terms. I know of > SolrCloud demo clusters with 500+ nodes, and at that point > peop

Re: two types of answers in my query

This will work. Thanks. On Tue, Jul 9, 2013 at 4:37 PM, Jack Krupansky wrote: > Usually a car term and a car part term will look radically different. So, > simply use the edismax query parser and set "qf" to be both the car and car > part fields. If either matches, the document will be selected.

Disabling workd breaking for codes and SKUs

Some of the data in my index is SKUs and barcodes as follows ASDF3-DASDD-2133DD-21H44 I want to disable the wordbreaking for this type (maybe through Regex. Is there a possible way to do this?

Re: Norms

I don't know the full answer to your question, but here's what I can offer. Solr offers 2 types of normalisation, FieldNorm and QueryNorm. FieldNorm is as the name suggests field level normalisation, based on length of the field, and can be controlled by the omitNorms parameter on the field. In

Re: Solr Hangs During Updates for over 10 minutes

We had something similar in terms of update times suddenly spiking up for no obvious reason. We never got quite as bad as you in terms of the other knock on effects, but we certainly saw updates jumping from 10ms up to 3ms, all our external queues backed up and we rejected some updates, then a

Re: Solr 3.6 optimize and field cache question

Not a solution for the short term but sounds like a good use case to migrate to Solr 4.X and use DocValues instead of FieldCache for faceting. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-6-optimize-and-field-cache-question-tp4076398p4076822.html Sent from the Solr

Re: Solr Live Nodes not updating immediately

What do you have your ZK Timeout set to (zkClientTimeout in solr.xml or command line if you override it)? A kill of the raw process is bad, but ZK should spot that using its heartbeat mechanism, so unless your timeout is very large, it should be detecting the node is no longer available, and then

Switch to new leader transparently?

Hi there, I've built a SolrCloud cluster from example, but I have some question. When I send query to one leader (say http://xxx.xxx.xxx.xxx:8983/solr/collection1) and no problem everything will be fine. When I shutdown that leader, the other replica( http://xxx.xxx.xxx.xxx:9983/solr/collection1)

Re: Switch to new leader transparently?

You don't really need to direct any query specifically to a leader. It will automatically be routed to the right leader. You may put a load balancer on top to just fix the problem with querying a node that has gone away. Also, ZK aware SolrJ Java client that load-balances across all nodes in clust

Re: Disabling workd breaking for codes and SKUs

On 10 July 2013 14:02, Mysurf Mail wrote: > Some of the data in my index is SKUs and barcodes as follows > ASDF3-DASDD-2133DD-21H44 > > I want to disable the wordbreaking for this type (maybe through Regex. > Is there a possible way to do this? What fieldtype are you using for this in schema.xml?

Re: Switch to new leader transparently?

You can define a CloudSolrServer as like that: *private static CloudSolrServer solrServer;* and then define the addres of your zookeeper host: *private static String zkHost = "localhost:9983";* initialize your variable: *solrServer = new CloudSolrServer(zkHost);* You can get leader list as li

Re: Switch to new leader transparently?

Hi anshum Thanks for your response. My application is developed using C#, so I can't use CloudSolrServer with SolrJ. My problem is there is a setting in my application SolrUrl = http://xxx.xxx.xxx.xxx:8983/solr/collection1 When this Solr instance shutdown or crash, I have to change this setting

Re: Switch to new leader transparently?

You can check the source code of LBHttpSolrServer and try to implement something like that as your own. 2013/7/10 Floyd Wu > Hi anshum > Thanks for your response. > My application is developed using C#, so I can't use CloudSolrServer with > SolrJ. > > My problem is there is a setting in my appl

Re: Switch to new leader transparently?

Hi Furkan I'm using C#, SolrJ won't help on this, but its impl is a good reference for me. Thanks for your help. by the way, how to fetch/get cluster state from zk directly in plain http or tcp socket? In my SolrCloud cluster, I'm using standalone zk to coordinate. Floyd 2013/7/10 Furkan KAM

Re: Switch to new leader transparently?

By the this is not related to your question but this may help you for connecting Solr via C#: http://solrsharp.codeplex.com/ 2013/7/10 Floyd Wu > Hi Furkan > I'm using C#, SolrJ won't help on this, but its impl is a good reference > for me. Thanks for your help. > > by the way, how to fetch/get

Re: Solr Live Nodes not updating immediately

My zkClientTimeout is set to 15000 by default. I am using external zookeeper-3.4.5 which is also running in 3 machines. I am using only one shard with replication factor being set to 3. Normal shutdown updates the solr state as soon as the node gets down.. I am facing issue with abrupt shut down

Re: Solr Hangs During Updates for over 10 minutes

Hey Daniel, Thanks for the response. I think we'll give this a try to see if this helps. Jed. On 7/10/13 10:48 AM, "Daniel Collins" wrote: >We had something similar in terms of update times suddenly spiking up for >no obvious reason. We never got quite as bad as you in terms of the other >kn

Re: replication getting stuck on a file

Hmmm, that is kind of funny. I know this is ugly, but what happens if you 1> stop the slave 2> completely delete the data/index directory (directory too, not just contents) 3> fire it back up? inelegant at best, but if it cures your problem Erick On Tue, Jul 9, 2013 at 5:57 PM, Petersen, Rob

Re: join not working with UUIDs

What kind of field is root_id? If it's tokenized or not the same type as id, that could account for it. Best Erick On Tue, Jul 9, 2013 at 7:34 PM, Marcelo Elias Del Valle wrote: > Hello, > > I am trying to create a POC to test query joins. However, I was > surprised when I saw my test worked

Re: Switch to new leader transparently?

Floyd: The Apache Zookeeper project should have the relevant info on how to get the state from ZK directly. FWIW, Erick On Wed, Jul 10, 2013 at 6:41 AM, Furkan KAMACI wrote: > By the this is not related to your question but this may help you for > connecting Solr via C#: http://solrsharp.codepl

Re: Solr Hangs During Updates for over 10 minutes

Jed: I'm not sure changing Java runtime is any less scary than upgrading Solr Wait, I know! Ask your manager if you can do both at once . I have a t-shirt that says "I don't test, but when I do it's in production"... Erick On Wed, Jul 10, 2013 at 8:08 AM, Jed Glazner wrote: > Hey Daniel,

Re: Staggered Replication In Solr?

Thanks Shawn, We do have repeaters setup to replicate index to the 8 Slaves. We update documents to Master every 2hrs in a batch process. When on hard commit is replicated to repeaters and then to slaves. The concern is that during heavy traffic when slaves are busy serving request, when a new ind

Re: Solr Hangs During Updates for over 10 minutes

It is certainly 'more' possible, as we have additional code that revolves around reading the clusterstate.json and since solr decided to change the format of the clusterstate.json from 4.0 to 4.1 it requires additional code changes to our service since the solrj lib from 4.0 isn't compatible with a

Re: join not working with UUIDs

root_id is a dynamic field... But should the type of the field change according to the values? Because using the same configuration but using "room1" as value, it works. Let me compare the configurations: Indeed, one is text_general and the other is string... I will try to create a fixed fiel

Re: Solr limitations

Again, no hard limits, mostly performance-based limits and environmental factors of your own environment, as well as the fact that most people on this list will have deeper experience with smaller clusters, so if you decide to "go big", you will be in uncharted and untested territory. I would

Re: Switch to new leader transparently?

Hi Floyd, We use SolrNet to connect to Solr from a C# application. Since SolrNet is not aware about SolrCloud or ZK, we use a Http load balancer in front of the Solr nodes & query via the load balancer url. You could use something like HAProxy or Apache reverse proxy for load balancing. On the ot

simple date query

Hi, I'm trying to do something like startDate_tdt <= NOW >= endDate_tdt. Any ideas how I can implement this in a query? I don't think the normal range query will work. Regards, Marcos

RE: simple date query

hi - check the examples for range queries and date math: http://wiki.apache.org/solr/SolrQuerySyntax http://lucene.apache.org/solr/4_3_1/solr-core/org/apache/solr/util/DateMathParser.html -Original message- > From:Marcos Mendez > Sent: Wednesday 10th July 2013 15:47 > To: solr-user@lu

Re: join not working with UUIDs

Worked :D Thanks a lot! 2013/7/10 Marcelo Elias Del Valle > root_id is a dynamic field... But should the type of the field change > according to the values? Because using the same configuration but using > "room1" as value, it works. > > Let me compare the configurations: > >

Re: simple date query

You can't use two fields in one range query, but you can combine two range queries: startDate_tdt:[* TO NOW] AND endDate_tdt:[NOW TO *] -- Jack Krupansky -Original Message- From: Marcos Mendez Sent: Wednesday, July 10, 2013 9:31 AM To: solr-user@lucene.apache.org Subject: simple date

Securing SOLR REST API

Hello Everyone, I have been developing several solutions, mainly geospatial, that include solr. The availability of the restful services seem to bother a lot of people. Mainly IT security, of course. How can I guarantee that Solr services are only 'called' from my web html5/jquery based applica

Re: Securing SOLR REST API

Hi Guilherme, see - Steve On Jul 10, 2013, at 10:22 AM, "Pires, Guilherme" wrote: > Hello Everyone, > > I have been developing several solutions, mainly geospatial, that include > solr. > The availability of the restful services seem to bother a lot

Re: Securing SOLR REST API

Sent from my iPhone On Jul 10, 2013, at 10:22 AM, "Pires, Guilherme" wrote: > Hello Everyone, > > I have been developing several solutions, mainly geospatial, that include > solr. > The availability of the restful services seem to bother a lot of people. > Mainly IT security, of course. >

Commit different database rows to solr with same "id" value?

Hello, I am trying to use Solr to store fields from two different database tables, where the primary keys are in the format of "1, 2, 3, " In Java, we build different POJO classes for these two database tables: table1.java @SolrIndex(name="id") private String idTable1 table2.java

RE: Commit different database rows to solr with same "id" value?

Hi Jason, Assuming you're using DIH, why not build a new, unique id within the query to use as the 'doc_id' for SOLR? We do something like this in one of our collections. In MySQL, try this (don't know what it would be for any other db but there must be equivalents): select @rownum:=@rownum+1

Re: Solr Hangs During Updates for over 10 minutes

On 7/10/2013 6:57 AM, Jed Glazner wrote: So we'll do what we can quickly to see if we can 'band-aid' the problem until we can upgrade to solr 4.4 Speaking of band-aids - does anyone know of a way to change the socket timeout/connection timeout for distributed updates? If you need to change Htt

When not to use NRTCachingDirectory and what to use instead.

Hello all, The default directory implementation in Solr 4 is the NRTCachingDirectory (in the example solrconfig.xml file , see below). The Javadoc for NRTCachingDirectoy ( http://lucene.apache.org/core/4_3_1/core/org/apache/lucene/store/NRTCachingDirectory.html?is-external=true) says: "This cl

Re: What are the options for obtaining IDF at interactive speeds?

I didn't try indexing each term as a separate document (and if I had I probably would've just used tv.tf_idf instead of a functional query -- why not?). The regular functional query which required sending a separate request for each of thousands of terms was wy dominated by the overhead of each

Re: When not to use NRTCachingDirectory and what to use instead.

On 7/10/2013 9:59 AM, Tom Burton-West wrote: The Javadoc for NRTCachingDirectoy ( http://lucene.apache.org/core/4_3_1/core/org/apache/lucene/store/NRTCachingDirectory.html?is-external=true) says: "This class is likely only useful in a near real-time context, where indexing rate is lowish but

more than 1 join on the same query

Hello, I am playing with joins here just to test what I can do with them. I have been learning a lot, but I am still having some troubles with more complex queries. For example, suppose I have the following documents: - id = 1 - name = Humblebee - age = 1000 - id = 2 - type = arm -

AW: Solr Hangs During Updates for over 10 minutes

Hi Shawn this code is für the solrj lib which we already use. I'm talking about solr s internal communication from leader to replica via the DistributedCmdUpdate class. I want to force the leader to time out after a fixed period instead of waiting for 15 minutes für the server to figure out the

How to create optional 'fq' plugin?

I am trying to suppress the error messages received when a value is not passed to a query, Ex: /select?first_name=peter&fq=$first_name&q=*:* I don't want the above query to throw error or die whenever the variable first_name is not passed to the query hence I came up with a plugin to return nul

How to make a variable in 'fq' optional?

I am trying to make a variable in fq optional, Ex: /select?first_name=peter&fq=$first_name&q=*:* I don't want the above query to throw error or die whenever the variable first_name is not passed to the query instead return the value corresponding to rest of the query. I can use switch but its

How to make a variable in 'fq' optional?

I am trying to make a variable in fq optional, Ex: /select?first_name=peter&fq=$first_name&q=*:* I don't want the above query to throw error or die whenever the variable first_name is not passed to the query instead return the value corresponding to rest of the query. I can use switch but its

Re: more than 1 join on the same query

try fq = {!join from=root_id to=id}type:leg&attr1=right OR {!join from=root_id to=id}type:arm&attr1=left Dom 2013/7/10 Marcelo Elias Del Valle > Hello, > > I am playing with joins here just to test what I can do with them. I > have been learning a lot, but I am still having some troubles

Re: more than 1 join on the same query

This fq = {!join from=root_id to=id}type:leg&attr1=right OR {!join from=root_id to=id}type:arm&attr1=left works even if I have attr1=left1 in the second condition. My goal is to select bodies that matches both conditions. It's strange, but if I try fq = {!join from=root_id to=id}type:leg&attr1=rig

Re: more than 1 join on the same query

Sorry, I didn't check preciselyI guess in your sample &attr1 applies to the body, not the legs, that could explain your problem 2013/7/10 Marcelo Elias Del Valle > This > fq = {!join from=root_id to=id}type:leg&attr1=right OR {!join from=root_id > to=id}type:arm&attr1=left > works even if I

Re: Solr Hangs During Updates for over 10 minutes

On 7/10/2013 6:57 AM, Jed Glazner wrote: So, while it's 'just as risky' as you say, it's 'less risky' than a new version of java and is possible to implement without downtime. I believe that if you update one node at a time, there should be no downtime. I've not actually tried this, so it wou

Re: more than 1 join on the same query

Dominique, I tried also: fq = {!join from=root_id to=id}type:leg AND {!join from=root_id to=id}type:arm If I understood what you said correctly, that should return something too, right? It also got me 0 results... 2013/7/10 Dominique Debailleux > Sorry, I didn't check preciselyI guess in yo

Re: more than 1 join on the same query

Got puzzled now! If instead of AND I use &, it works: fq = {!join from=root_id to=id}type:leg & {!join from=root_id to=id}type:arm I am definitly missing something, I don't know what... Shouldn't both be the same? []s 2013/7/10 Marcelo Elias Del Valle > Dominique, > I tried also: > fq = {!jo

Re: Solr Hangs During Updates for over 10 minutes

+1 for G1. We just had a happy client this week switch to G1 after seeing stw pauses with CMS. I can't share their JVM metrics from SPM, but I can share ours: http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ (HBase, not Solr, but we've seen the same effect with ElasticSearch for

Re: more than 1 join on the same query

Be careful with URL encoding... that may be messing you up depending on how you are trying to submit the query (and the single & you were using as AND) fq={!join from=root_id to=id}type:arm AND attr1=left fq={!join from=root_id to=id}type:leg AND attr1=right -Yonik http://lucidworks.com On Wed,

How to make 'fq' optional?

I am trying to make a variable in fq optional, Ex: /select?first_name=peter&fq=$first_name&q=*:* I don't want the above query to throw error or die whenever the variable first_name is not passed to the query instead return the value corresponding to rest of the query. I can use switch but its

How to form / return filter (Query object)?

I was able to form a TermQuery as below, Query query=new TermQuery(new Term("id","value")); I am trying to form a filter query something that returns just the filter that can be used with any query type (q or fq). if (qstr == null || qstr.trim().length() < 1) {

Re: Solr limitations

Also, total index file size. At 200-300gb managing an index becomes a pain. Lance On 07/08/2013 07:28 AM, Jack Krupansky wrote: Other that the per-node/per-collection limit of 2 billion documents per Lucene index, most of the limits of Solr are performance-based limits - Solr can handle it, bu

Re: Commit different database rows to solr with same "id" value?

Thanks David. I am actually trying to commit the database row on the fly, not DIH. :) Anyway, if I understand you correctly, basically you are suggesting to modify the value of the primary key and pass the new value to "id" before committing to solr. This could probably be one solution. What if

Re: Solr limitations

For what it's worth, in SPM we keep track of nodes/server stats, of course, and that metric has been going up for those using SPM to monitor Solr clusters, which is a nice sign. Otis -- Solr & ElasticSearch Support -- http://sematext.com/ Solr Performance Monitoring -- http://sematext.com/spm O

amount of values in a multi value field - is denormalization always the best option?

Hello, I have asked a question recently about solr limitations and some about joins. It comes that this question is about both at the same time. I am trying to figure how to denormalize my data so I will need just 1 document in my index instead of performing a join. I figure one way of doi

Re: amount of values in a multi value field - is denormalization always the best option?

Simple answer: avoid "large number of values in a single document". There should only be a modest to moderate number of fields in a single document. Is the data relatively static, or subject to frequent updates? To update any field of a single document, even with atomic update, requires Solr to

solr postfilter question

Hey, I am trying to create a plugin which makes use of postfilter. I know that the collect function is called for every document matched, but is there a way i can access all the matched documents upto this point before collect is called on each of them? Thanks, Rohir

Re: solr postfilter question

On Wed, Jul 10, 2013 at 6:08 PM, Rohit Harchandani wrote: > Hey, > I am trying to create a plugin which makes use of postfilter. I know that > the collect function is called for every document matched, but is there a > way i can access all the matched documents upto this point before collect > is

Re: replication getting stuck on a file

I have seen this in 4.2.1 too. Once replication is finished, on Admin UI we see 100% and time and dlspeed information goes out of wack Same is reflected in mbeans. But whats actually happening in the background is auto-warmup of caches (in my case) May be some minor stats bug -- View this mess

Re: My latest solr blog post on Solr's PostFiltering

Hi Amit, Great article. I tried it and it works well. I am new to developing in solr and had a question? do you know if there is a way to access all the matched ids before collect is called? Thanks, Rohit On Sat, Nov 10, 2012 at 1:12 PM, Erick Erickson wrote: > That'll teach _me_ to look close

Re: amount of values in a multi value field - is denormalization always the best option?

Jack, When you say: "large number of values in a single document" you also mean a block in a block join, right? Exactly the same thing, agree? In my case, I have just 1 insert and no updates. Even in this case, you think a large document or block would be a really bad idea? I am more wor

Re: amount of values in a multi value field - is denormalization always the best option?

On Wed, Jul 10, 2013 at 5:37 PM, Marcelo Elias Del Valle wrote: > Hello, > > I have asked a question recently about solr limitations and some about > joins. It comes that this question is about both at the same time. > I am trying to figure how to denormalize my data so I will need just 1

Re: amount of values in a multi value field - is denormalization always the best option?

Join is a query operation - it has nothing to do with the number of values (fields and multivalued fields) in a Solr/Lucene document. Block "insert" isn't available yet anyway, so we don't have any clear assessments of its performance. Generally, any kind of large block of data is not a great

RE: Overseer queues confused me

can someone answer my question? Thanks in advance Best Regards, Illu Ying -Original Message- From: Illu.Y.Ying (mis.sh04.Newegg) 41417 [mailto:illu.y.y...@newegg.com] Sent: Wednesday, July 10, 2013 10:44 AM To: solr-user@lucene.apache.org Subject: Overseer queues confused me Hi there：

expunging deletes

Hi guys, Using solr 3.6.1 and the following settings, I am trying to run without optimizes. I used to optimize nightly, but sometimes the optimize took a very long time to complete and slowed down our indexing. We are continuously indexing our new or changed data all day and night. After a f

Re: expunging deletes

On 7/10/2013 5:58 PM, Petersen, Robert wrote: > Using solr 3.6.1 and the following settings, I am trying to run without > optimizes. I used to optimize nightly, but sometimes the optimize took a > very long time to complete and slowed down our indexing. We are continuously > indexing our new o

solr 4.3 solrj generating search terms that return no results

I'm having trouble with solrj generating a query like &q=kohler%5C+k for the search term 'Kohler k' I am using Solr 4.3 in cloud mode. When I remove the %5C everything is fine. I'm not sure why the %5C is being added when I call solrQuery.setQuery('Kohler k'); Any help is appreciated. -- View

Re: solr 4.3 solrj generating search terms that return no results

On 7/10/2013 6:34 PM, dboychuck wrote: > I'm having trouble with solrj generating a query like &q=kohler%5C+k for the > search term 'Kohler k' > > I am using Solr 4.3 in cloud mode. When I remove the %5C everything is fine. > I'm not sure why the %5C is being added when I call > solrQuery.setQuery

Re: solr 4.3 solrj generating search terms that return no results

solrQuery.setQuery(ClientUtils.escapeQueryChars(keyword)); It looks like using the solrj ClientUtils.escapeQueryChars function is escaping any spaces with %5C+ which returns 0 results at search time. -- View this message in context: http://lucene.472066.n3.nabble.com/solr-4-3-solrj-generating-

Re: Moving replica from node to node?

Thanks Mark. I assume you are referring to using the Core Admin API - CREATE and UNLOAD? Added https://issues.apache.org/jira/browse/SOLR-5032 Otis -- Solr & ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Mon, Jul 8, 2013 at 10:50 PM, Mark M

Re: Switch to new leader transparently?

Thanks Aloke, I will do some research. 2013/7/10 下午9:45 於 "Aloke Ghoshal" 寫道： > Hi Floyd, > > We use SolrNet to connect to Solr from a C# application. Since SolrNet is > not aware about SolrCloud or ZK, we use a Http load balancer in front of > the Solr nodes & query via the load balancer url. Yo

Indexing database in Solr using Data Import Handler

Im trying to index MySql database using Data Import Handler in solr. I have made two tables. The first table holds the metadata of a file. create table filemetadata ( id varchar(20) primary key , filename varchar(50), path varchar(200), size varchar(10), author varchar(50) ) ; The second table