SolrCloud: Creating more shard at runtime will lower down the load?

Hi, I have a question regarding the solr cloud. The load on our search server are increasing day by day as our no of visitors are keep on increasing. So I have a scenario, I want to slice the data at the Runtime, by creating the more shards of the data. *i)* Does it affect the current queries *

Re: sort on fields that are not mandatory in each document

Oh ok. Thank youAlessandro. On 5/27/2015 6:07 PM, Alessandro Benedetti wrote: Actually it is both field level and field type level. You decide based on your use case ( can happen that for the same field type , you want sortMissingFirst for one field, and sortMissingLast for another) . I want to

Re: sort on fields that are not mandatory in each document

Got it. Thank you Rajani. On 5/27/2015 5:34 PM, Rajani Maski wrote: Hi Derek, They are at the fieldType Level. You might find some reference examples in schema.xml using them. https://cwiki.apache.org/confluence/display/solr/Field+Type+Definitions+and+Properties On Wed, May 27, 2015 at 2:30 PM

Re: Difference in running Solr with Jetty internally or externally

Thank you Erick! I'll use the internal Jetty and start Solr with the bin/solr start scripts. Regards, Edwin On 26 May 2015 at 12:20, Erick Erickson wrote: > Actually, just use the new bin/solr start scripts and ignore whether > it's running Jetty under the covers or not I think. > > Best, >

Re: Ability to load solrcore.properties from zookeeper

: I am attempting to override some properties in my solrconfig.xml file by : specifying properties in a solrcore.properties file which is uploaded in : Zookeeper's collections/conf directory, though when I go to create a new : collection those properties are never loaded. One work-around is to spec

Re: docValues: Can we apply synonym

Ok and what synonym processor you is talking about maybe it could help ? With Regards Aman Tandon On Thu, May 28, 2015 at 4:01 AM, Reitzel, Charles < charles.reit...@tiaa-cref.org> wrote: > Sorry, my bad. The synonym processor I mention works differently. It's > an extension of the EDisMax qu

RE: docValues: Can we apply synonym

Sorry, my bad. The synonym processor I mention works differently. It's an extension of the EDisMax query processor and doesn't require field level synonym configs. -Original Message- From: Reitzel, Charles [mailto:charles.reit...@tiaa-cref.org] Sent: Wednesday, May 27, 2015 6:12 PM T

RE: docValues: Can we apply synonym

But the query analysis isn't on a specific field, it is applied to the query string. -Original Message- From: Aman Tandon [mailto:amantandon...@gmail.com] Sent: Wednesday, May 27, 2015 6:08 PM To: solr-user@lucene.apache.org Subject: Re: docValues: Can we apply synonym Hi Charles, The

Re: docValues: Can we apply synonym

Hi Charles, The problem here is that the docValues works only with primitives data type only like String, int, etc So how could we apply synonym on primitive data type. With Regards Aman Tandon On Thu, May 28, 2015 at 3:19 AM, Reitzel, Charles < charles.reit...@tiaa-cref.org> wrote: > Is there

RE: When is too many fields in "qf" is too many?

One request handler per view? I think if you are able to make the actual view in use for the current request a single value (vs. all views that the user could use over time), it would keep the qf list down to a manageable size (e.g. specified within the request handler XML). Not sure if th

RE: docValues: Can we apply synonym

Is there any reason you cannot apply the synonyms at query time? Applying synonyms at indexing time has problems, e.g. polluting the term frequency for synonyms added, preventing distance queries, ... Since city names often have multiple terms, e.g. New York, Den Hague, etc., I would recommen

Re: Clarification on Collections API for 5.x

One driving reason behind making such a statement in the upgrade note was that at some point in the 5.x line, we want to move to a "Zookeeper state as truth" mode where only the state inside ZK is considered as truth regardless of the local state of a node. Such a move will break back-compat (e.g.

Re: distributed search limitations via SolrCloud

Thanks a lot Erick... great inputs... Currently our deployment is on Tomcat 7 and I think SOLR 5.x does not support Tomcat but runs on its own Jetty server, right ? I will discuss this with the team. Thanks again. Regards Vishal On Wed, May 27, 2015 at 4:16 PM, Erick Erickson wrote: > I'd mov

Re: distributed search limitations via SolrCloud

I'd move to Solr 4.10.3 at least, but preferably Solr 5.x. Solr 5.2 is being readied for release as we speak, it'll probably be available in a week or so barring unforeseen problems and that's the one I'd go with by preference. Do be aware, though, that the 5.x Solr world deprecates using a war fi

Re: distributed search limitations via SolrCloud

Thanks a lot Erick... You are right we should not delay moving to sharding/SolrCloud process. As you all are expert... currently we are using SOLR 4.7.. Do you suggest we should move to latest SOLR release 5.1.0 ? or we can manage the above issue using SOLR 4.7 Regards Vishal On Wed, May 27, 201

Re: Clarification on Collections API for 5.x

Thanks for the clarification! On 5/27/15, 12:00 PM, "Erick Erickson" wrote: >Are you defining shard and replicas here? Or is this just a >single-node collection? In any case, this seems unnecessary. You'd get >the same thing by having your uploading the config set to ZK, then >just issuing a Col

Re: Start Scripts

I wonder if, Dean, you are using an older. Take a look in the bin/ directory of any newer Solr, preferably 5.x and you'll see quite substantial start scripts. Upayavira On Wed, May 27, 2015, at 07:11 PM, Erick Erickson wrote: > Hmmm, this is a little confused I think. > > bq: copies all necessar

Ability to load solrcore.properties from zookeeper

I am attempting to override some properties in my solrconfig.xml file by specifying properties in a solrcore.properties file which is uploaded in Zookeeper's collections/conf directory, though when I go to create a new collection those properties are never loaded. One work-around is to specify prop

Re: distributed search limitations via SolrCloud

Hard to say. I've seen 20M doc be the place you need to consider sharding/SolrCloud. I've seen 300M docs be the place you need to start sharding. That said I'm quite sure you'll need to shard before you get to 2B. There's no good reason to delay that process. You'll have to do something about the

distributed search limitations via SolrCloud

Currently, we have SOLR configured on single linux server (24 GB physical memory) with multiple cores. We are using SOLR joins (https://wiki.apache.org/solr/Join) across cores on this single server. But, as data will grow to ~2 billion we need to assess whether we’ll need to run SolrCloud as "In a

Copying index from one Solr cloud to other Solr cloud

I am using Solr cloud 4.7.2. We have around 100 collections spread across 16 Solr nodes. Also, there are 5 dedicated servers for running Zookeeper. I want to move all these collections data (or collections) to a completely different solr cloud. How to achieve this? The Zookeeper servers for this S

Re: Start Scripts

Hmmm, this is a little confused I think. bq: copies all necessary config file accordingly This is handled for you, with the caveat that you have to push the configs to Zookeeper as a separate step with the zk client. Once that's done, just use the Collections API to create the collection. See th

Re: Dynamic range on numbers

thanks erick. will give it a whirl later today and report back tonight or tomorrow. i imagine i'll have some more questions crop up :) best, -- *John Blythe* Product Manager & Lead Developer 251.605.3071 | j...@curvolabs.com www.curvolabs.com 58 Adams Ave Evansville, IN 47713 On Wed, May 27,

Re: Clarification on Collections API for 5.x

Are you defining shard and replicas here? Or is this just a single-node collection? In any case, this seems unnecessary. You'd get the same thing by having your uploading the config set to ZK, then just issuing a Collections CREATE command, specifying the node to use if desired. What you're doing

Re: HW requirements

Sznajder ForMailingList wrote: > Could you give me some hints wrt HW requirements for Solr if I need to > index about 400 Gigas of text? No. You are providing far too few data for us to guess. 400GB can be handled on a laptop or require 3 strong servers, depending on what you intend to do with

Re: Solr 5 Tutorial which does NOT use dist examples

bin/solr start -d techproducts that will set up a non-cloud (i.e. stand-alone) Solr that you can use as a model. Note that Solr uses "core discovery", which basically recursively descends from SOLR_HOME and any time it finds a "core.properties" file it assumes it's at the root of another core and l

Re: Dynamic range on numbers

1> tfloat 2> fq=dimField:[4.5 TO 5.5] or even use frange to set the lower and upper bounds via function Best, Erick On Wed, May 27, 2015 at 5:29 AM, John Blythe wrote: > hi all, > > i'm attempting to suggest products across a range to users based on > dimensions. if there is a "5x10mm Drill Set"

Re: Index optimize runs in background.

All strange of course. What do your Solr logs show when this happens? And how reproducible is this? Best, Erick On Wed, May 27, 2015 at 4:00 AM, Upayavira wrote: > In this case, optimising makes sense, once the index is generated, you > are not updating It. > > Upayavira > > On Wed, May 27, 2015

Re: Removing characters like '\n \n' from indexing

The other alternative is to use SolrJ to parse the documents and do your processing there. Here's an article on the pros/cons and an example program. https://lucidworks.com/blog/indexing-with-solrj/ Best, Erick On Wed, May 27, 2015 at 1:57 AM, Erik Hatcher wrote: > Edwin - > > There’s a bunch o

Start Scripts

Does anyone have any simple SOLR startup scripts that they're willing to share? I am looking for a script that creates a custom collection and copies all necessary config file accordingly. Thanks in advance, Dean

HW requirements

Hi , Could you give me some hints wrt HW requirements for Solr if I need to index about 400 Gigas of text? Thanks Benjamin

Re: Clarification on Collections API for 5.x

bump On 5/21/15, 9:06 AM, "Jim.Musil" wrote: >Hi, > >In the guide for moving from Solr 4.x to 5.x, it states the following: > >"Solr 5.0 only supports creating and removing SolrCloud collections >through the Collections >API, >unl

Re: Solr 5 Tutorial which does NOT use dist examples

Well, the cloud initialization steps are in the scripts that set those examples up. Based on your parameters too. To connect with Oracle to read data, you are probably looking for DIH (DataImportHandler) and Oracle as your keywords - not really connected to the scripts. But if you really don't kno

Re: Solr 5 Tutorial which does NOT use dist examples

Alex, I already have gone over all the great examples, esp the cloud. My main goal is to stand up my own collections, my own nodes, etc without relying on the cloud example's auto-magic configuration. I am trying to gain insight into where each configuration item needs to go, how to set up a nod

Re: Solr 5 Tutorial which does NOT use dist examples

Why exactly do you not want to use bundled examples. There is an example that spins several servers next to each other on one machine. Then you can examine everything in there. I am missing something I guess. Regards, Alex. Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:

Dynamic range on numbers

hi all, i'm attempting to suggest products across a range to users based on dimensions. if there is a "5x10mm Drill Set" for instance and a competitor sales something similar enough then i'd like to have it shown. the range, however, would need to be dynamic. i'm thinking for our initial testing p

Solr 5 Tutorial which does NOT use dist examples

I am looking for a good tutorial on Solr 5 that does not use the provided dist examples. I am trying to setup a 'test' environment - using embedded zookeeper - that would mimic what would need to be done in test/prod without having to spin up multiple machines to do so. Specifically, we are looki

Re: Index optimize runs in background.

In this case, optimising makes sense, once the index is generated, you are not updating It. Upayavira On Wed, May 27, 2015, at 06:14 AM, Modassar Ather wrote: > Our index has almost 100M documents running on SolrCloud of 5 shards and > each shard has an index size of about 170+GB (for the record,

Re: Native library of plugin is loaded for every core

In your solr home directory, create a lib directory, and put your jar there. Then you wont have to declare it in solrconfig.xml. That's what Alan is suggesting. Upayavira On Wed, May 27, 2015, at 09:39 AM, adfel70 wrote: > Hi Alan, thanks for the reply. > I am not sure what did you mean. Currentl

Re: Displaying a date field that is empty

Mmmm if you add a default value, you are going to index a "fake" date for the null one ( like a specific date you decide). Of course this default value should be different from any other possible date in your system and must not be misleading for users. The reason you want the date to be in the do

Re: sort on fields that are not mandatory in each document

Actually it is both field level and field type level. You decide based on your use case ( can happen that for the same field type , you want sortMissingFirst for one field, and sortMissingLast for another) . I want to add a bonus note, related the ""( empty ) and null concept. Be very careful you

Re: sort on fields that are not mandatory in each document

Hi Derek, They are at the fieldType Level. You might find some reference examples in schema.xml using them. https://cwiki.apache.org/confluence/display/solr/Field+Type+Definitions+and+Properties On Wed, May 27, 2015 at 2:30 PM, Derek Poh wrote: > Hi Ahmet > > The sortMissingLast and sortMissin

Re: sort on fields that are not mandatory in each document

Hi Ahmet The sortMissingLast and sortMissingFirst attributes are defined at the field or fieldType level? multiValued="false"/> positionIncrementGap="0"/> On 5/27/2015 4:43 PM, Ahmet Arslan wrote: Hi, I think you are looking for sortMissing* attributes: sortMissingLast and sortMissingFir

Re: Removing characters like '\n \n' from indexing

Edwin - There’s a bunch of built-in update processors you can use, including a script one that allows you to code it dynamically in JavaScript (or other JVM scripting language). See https://cwiki.apache.org/confluence/display/solr/Update+Request+Processors

Re: sort on fields that are not mandatory in each document

Hi, I think you are looking for sortMissing* attributes: sortMissingLast and sortMissingFirst attributes are optional attributes are currently supported on types that are sorted internally as strings and on numeric types. Ahmet On Wednesday, May 27, 2015 11:36 AM, Derek Poh wrote: Hi I am tryi

Re: Displaying a date field that is empty

Hi, One approach if to use default value for that field. Missing values represented by that constant value. Other approach is to use a query to capture documents that do not have that field: +*:* -field:[* TO *] I think, newly added exists(field) function query can be used too. Ahmet On We

Re: Native library of plugin is loaded for every core

Hi Alan, thanks for the reply. I am not sure what did you mean. Currently it is loaded from solrconfig.xml Is there any other way? -- View this message in context: http://lucene.472066.n3.nabble.com/Native-library-of-plugin-is-loaded-for-every-core-tp4207996p4208004.html Sent from the Solr -

sort on fields that are not mandatory in each document

Hi I am trying to sort on multiple fields. These fields donot necessary exist in every document. sort=sppddrank asc, ddrank asc From the sorted result, it seems that documents which donot have sppddrank field is at the top. How can I make the documents that have the sppddrank field to be on

Displaying a date field that is empty

Hi, Would like to check, is there any method to index date that are empty, and allow it to be searchable and be displayed as an empty field? Currently, for my csv records, there's some records which has no date in the date field, while others have. For those records with no dates, when I search f

Re: Native library of plugin is loaded for every core

Does it work if you load it via the solr home /lib directory, rather than from the /lib directory of each individual core? Alan Woodward www.flax.co.uk On 27 May 2015, at 08:45, adfel70 wrote: > Hi guys, need your help: > I added a custom plugins to Solr, to support my applicative needs (one i

Native library of plugin is loaded for every core

Hi guys, need your help: I added a custom plugins to Solr, to support my applicative needs (one index handler and 2 search components), all of them access a native library using JNI. The native library wrapper class loads the library using the regular pattern: public class YWrapper{ static

Re: Removing characters like '\n \n' from indexing

Hi Shawn, Thanks for your reply. So that means the only way for me is to write my own custom class in order for the removing characters like '\n' to work? Regards, Edwin On 27 May 2015 at 14:46, Shawn Heisey wrote: > On 5/26/2015 10:16 PM, Zheng Lin Edwin Yeo wrote: > > I tried to follow t