thanks Eric. i will try that. Some how i am not able to run a query on the shard directly because of kerberos. i even tried curl --negotiate.
Regards, Anil On 14 March 2016 at 22:30, Erick Erickson <erickerick...@gmail.com> wrote: > Try shards.info=true, but pinging the shard directly is the most certain. > > > Best, > Erick > > On Mon, Mar 14, 2016 at 9:48 AM, Anil <anilk...@gmail.com> wrote: > > HI Erik, > > > > we have used document routing to balance the shards load and for > > expand/collapse. it is mainly used for main_collection which holds one to > > many relationship records. In file_collection, it is only for load > > distribution. > > > > 25GB for entire solr service. each machine will act as shard for some > > collections. > > > > we have not stress tested our servers at least for solr service. i have > > read the the link you have shared, i will do something on it. thanks for > > sharing. > > > > i have checked other collections, where index size is max 90GB and 5 M as > > max number of documents. but for the particular file_collection_2014 , i > > see total index size across replicas is 147 GB. > > > > Can we get any hints if we run the query with debugQuery=true ? what is > > the effective way of load distribution ? Please advice. > > > > Regards, > > Anil > > > > On 14 March 2016 at 20:32, Erick Erickson <erickerick...@gmail.com> > wrote: > > > >> bq: The slowness is happening for file_collection. though it has 3 > shards, > >> documents are available in 2 shards. shard1 - 150M docs and shard2 has > 330M > >> docs , shard3 is empty. > >> > >> Well, this collection terribly balanced. Putting 330M docs on a single > >> shard is > >> pushing the limits, the only time I've seen that many docs on a shard, > >> particularly > >> with 25G of ram, they were very small records. My guess is that you will > >> find > >> the queries you send to that shard substantially slower than the 150M > >> shard, > >> although 150M could also be pushing your limits. You can measure this > >> by sending the query to the specific core (something like > >> > >> solr/files_shard1_replica1/query?(your queryhere)&distrib=false > >> > >> My bet is that your QTime will be significantly different with the two > >> shards. > >> > >> It also sounds like you're using implicit routing where you control > where > >> the > >> files go, it's easy to have unbalanced shards in that case, why did you > >> decide > >> to do it this way? There are valid reasons, but... > >> > >> In short, my guess is that you've simply overloaded your shard with > >> 330M docs. It's > >> not at all clear that even 150 will give you satisfactory performance, > >> have you stress > >> tested your servers? Here's the long form of sizing: > >> > >> > >> > https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ > >> > >> Best, > >> Erick > >> > >> On Mon, Mar 14, 2016 at 7:05 AM, Susheel Kumar <susheel2...@gmail.com> > >> wrote: > >> > For each of the solr machines/shards you have. Thanks. > >> > > >> > On Mon, Mar 14, 2016 at 10:04 AM, Susheel Kumar < > susheel2...@gmail.com> > >> > wrote: > >> > > >> >> Hello Anil, > >> >> > >> >> Can you go to Solr Admin Panel -> Dashboard and share all 4 memory > >> >> parameters under System / share the snapshot. ? > >> >> > >> >> Thanks, > >> >> Susheel > >> >> > >> >> On Mon, Mar 14, 2016 at 5:36 AM, Anil <anilk...@gmail.com> wrote: > >> >> > >> >>> HI Toke and Jack, > >> >>> > >> >>> Please find the details below. > >> >>> > >> >>> * How large are your 3 shards in bytes? (total index across > replicas) > >> >>> -- *146G. i am using CDH (cloudera), not sure how to > check > >> the > >> >>> index size of each collection on each shard* > >> >>> * What storage system do you use (local SSD, local spinning drives, > >> remote > >> >>> storage...)? *Local (hdfs) spinning drives* > >> >>> * How much physical memory does your system have? *we have 15 data > >> nodes. > >> >>> multiple services installed on each data node (252 GB RAM for each > data > >> >>> node). 25 gb RAM allocated for solr service.* > >> >>> * How much memory is free for disk cache? *i could not find.* > >> >>> * How many concurrent queries do you issue? *very less. i dont see > any > >> >>> concurrent queries to this file_collection for now.* > >> >>> * Do you update while you search? *Yes.. its very less.* > >> >>> * What does a full query (rows, faceting, grouping, highlighting, > >> >>> everything) look like? *for the file_collection, rows - 100, > >> highlights = > >> >>> false, no facets, expand = false.* > >> >>> * How many documents does a typical query match (hitcount)? *it > varies > >> >>> with > >> >>> each file. i have sort on int field to order commands in the query.* > >> >>> > >> >>> we have two sets of collections on solr cluster ( 17 data nodes) > >> >>> > >> >>> 1. main_collection - collection created per year. each collection > uses > >> 8 > >> >>> shards 2 replicas ex: main_collection_2016, main_collection_2015 etc > >> >>> > >> >>> 2. file_collection (where files having commands are indexed) - > >> collection > >> >>> created per 2 years. it uses 3 shards and 2 replicas. ex : > >> >>> file_collection_2014, file_collection_2016 > >> >>> > >> >>> The slowness is happening for file_collection. though it has 3 > shards, > >> >>> documents are available in 2 shards. shard1 - 150M docs and shard2 > has > >> >>> 330M > >> >>> docs , shard3 is empty. > >> >>> > >> >>> main_collection is looks good. > >> >>> > >> >>> please let me know if you need any additional details. > >> >>> > >> >>> Regards, > >> >>> Anil > >> >>> > >> >>> > >> >>> On 13 March 2016 at 21:48, Anil <anilk...@gmail.com> wrote: > >> >>> > >> >>> > Thanks Toke and Jack. > >> >>> > > >> >>> > Jack, > >> >>> > > >> >>> > Yes. it is 480 million :) > >> >>> > > >> >>> > I will share the additional details soon. thanks. > >> >>> > > >> >>> > > >> >>> > Regards, > >> >>> > Anil > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > On 13 March 2016 at 21:06, Jack Krupansky < > jack.krupan...@gmail.com> > >> >>> > wrote: > >> >>> > > >> >>> >> (We should have a wiki/doc page for the "usual list of suspects" > >> when > >> >>> >> queries are/appear slow, rather than need to repeat the same > >> mantra(s) > >> >>> for > >> >>> >> every inquiry on this topic.) > >> >>> >> > >> >>> >> > >> >>> >> -- Jack Krupansky > >> >>> >> > >> >>> >> On Sun, Mar 13, 2016 at 11:29 AM, Toke Eskildsen < > >> >>> t...@statsbiblioteket.dk> > >> >>> >> wrote: > >> >>> >> > >> >>> >> > Anil <anilk...@gmail.com> wrote: > >> >>> >> > > i have indexed a data (commands from files) with 10 fields > and > >> 3 of > >> >>> >> them > >> >>> >> > is > >> >>> >> > > text fields. collection is created with 3 shards and 2 > >> replicas. I > >> >>> >> have > >> >>> >> > > used document routing as well. > >> >>> >> > > >> >>> >> > > Currently collection holds 47,80,01,405 records. > >> >>> >> > > >> >>> >> > ...480 million, right? Funny digit grouping in India. > >> >>> >> > > >> >>> >> > > text search against text field taking around 5 sec. solr is > >> query > >> >>> just > >> >>> >> > and > >> >>> >> > > of two terms with fl as 7 fields > >> >>> >> > > >> >>> >> > > fileId:"file unique id" AND command_text:(system login) > >> >>> >> > > >> >>> >> > While not an impressive response time, it might just be that > your > >> >>> >> hardware > >> >>> >> > is not enough to handle that amount of documents. The usual > >> culprit > >> >>> is > >> >>> >> IO > >> >>> >> > speed, so chances are you have a system with spinning drives > and > >> not > >> >>> >> enough > >> >>> >> > RAM: Switch to SSD and/or add more RAM. > >> >>> >> > > >> >>> >> > To give better advice, we need more information. > >> >>> >> > > >> >>> >> > * How large are your 3 shards in bytes? > >> >>> >> > * What storage system do you use (local SSD, local spinning > >> drives, > >> >>> >> remote > >> >>> >> > storage...)? > >> >>> >> > * How much physical memory does your system have? > >> >>> >> > * How much memory is free for disk cache? > >> >>> >> > * How many concurrent queries do you issue? > >> >>> >> > * Do you update while you search? > >> >>> >> > * What does a full query (rows, faceting, grouping, > highlighting, > >> >>> >> > everything) look like? > >> >>> >> > * How many documents does a typical query match (hitcount)? > >> >>> >> > > >> >>> >> > - Toke Eskildsen > >> >>> >> > > >> >>> >> > >> >>> > > >> >>> > > >> >>> > >> >> > >> >> > >> >
