Hi David, I ran the *:* query 10 times against all 30 servers and the results (below) were similar across all of them. I agree working against a single server is easier troubleshooting, but I do not know where to start.
Server shard replica, Matches, Time, Pass 16 1 n2 2989421 78800 1 20 1 n1 2989559 63246 1 23 1 n8 2989619 55141 1 28 1 n6 2989619 65536 1 17 1 n4 2989818 56694 1 26 2 n10 2990088 63485 1 21 2 n18 2990145 68077 1 11 2 n16 2990145 62271 1 13 2 n12 2990242 68564 1 27 2 n14 2990242 63739 1 10 3 n26 2988056 69117 1 25 3 n24 2988056 73750 1 12 3 n28 2988096 61948 1 6 3 n20 2988123 62174 1 19 3 n22 2988123 65826 1 1 4 n30 2985457 60404 1 29 4 n34 2985457 68498 1 30 4 n38 2985604 72034 1 9 4 n36 2902757 65943 1 15 4 n32 2985948 67208 1 7 5 n48 2992278 63098 1 5 5 n42 2992363 69503 1 8 5 n44 2992363 66818 1 4 5 n40 2992397 66784 1 14 5 n46 2883495 58759 1 3 6 n56 2878221 52265 1 22 6 n58 2878221 53768 1 24 6 n52 2878326 62174 1 2 6 n50 2878326 53143 1 18 6 n54 2878326 59044 1 Results from 10 passes p-solr-8-16.obscured.com:8983/solr/content_shard1_replica_n2/ 69697.8 4599.8171896 Query time milliseconds [78800, 65549, 68045, 72151, 62774, 69168, 66459, 74336, 69028, 70668] p-solr-8-20.obscured.com:8983/solr/content_shard1_replica_n1/ 58310.5 4531.23621224 Query time milliseconds [63246, 59626, 61001, 59366, 53028, 58693, 58832, 64633, 54659, 50021] p-solr-8-23.obscured.com:8983/solr/content_shard1_replica_n8/ 57778.5 4659.23933348 Query time milliseconds [55141, 55194, 59100, 62614, 65425, 59261, 58961, 59259, 53799, 49031] p-solr-8-28.obscured.com:8983/solr/content_shard1_replica_n6/ 64944.1 3382.61379705 Query time milliseconds [65536, 67825, 69829, 60059, 63616, 67588, 68443, 60853, 62666, 63026] p-solr-8-17.obscured.com:8983/solr/content_shard1_replica_n4/ 58018.9 4821.9028851 Query time milliseconds [56694, 58900, 55404, 51590, 66034, 51256, 57109, 57515, 63530, 62157] p-solr-8-26.obscured.com:8983/solr/content_shard2_replica_n10/ 59366.6 5036.84751936 Query time milliseconds [63485, 53315, 64845, 62077, 54313, 52607, 65389, 55977, 63486, 58172] p-solr-8-21.obscured.com:8983/solr/content_shard2_replica_n18/ 61844.1 4623.13444537 Query time milliseconds [68077, 61117, 64284, 65393, 60580, 57495, 58068, 67454, 62370, 53603] p-solr-8-11.obscured.com:8983/solr/content_shard2_replica_n16/ 61179.1 4224.86040401 Query time milliseconds [62271, 66059, 67076, 55706, 60905, 58617, 56561, 66308, 57100, 61188] p-solr-8-13.obscured.com:8983/solr/content_shard2_replica_n12/ 69578.3 3986.83530998 Query time milliseconds [68564, 67411, 71644, 75938, 73772, 69780, 67438, 72479, 66368, 62389] p-solr-8-27.obscured.com:8983/solr/content_shard2_replica_n14/ 59808.2 4896.04649579 Query time milliseconds [63739, 59873, 65775, 50280, 63009, 60955, 55516, 64130, 60016, 54789] p-solr-8-10.obscured.com:8983/solr/content_shard3_replica_n26/ 66038.1 3363.29340743 Query time milliseconds [69117, 63766, 66189, 72059, 68751, 67644, 65027, 60859, 63306, 63663] p-solr-8-25.obscured.com:8983/solr/content_shard3_replica_n24/ 60566.8 6390.11408001 Query time milliseconds [73750, 58849, 59947, 58810, 55752, 51790, 54871, 60592, 66853, 64454] p-solr-8-12.obscured.com:8983/solr/content_shard3_replica_n28/ 63456.6 4591.56887252 Query time milliseconds [61948, 61372, 60570, 59961, 66818, 62124, 66880, 72592, 56477, 65824] p-solr-8-6.obscured.com:8983/solr/content_shard3_replica_n20/ 60869.6 2619.82752613 Query time milliseconds [62174, 59369, 62098, 66974, 59552, 58757, 59728, 59413, 62414, 58217] p-solr-8-19.obscured.com:8983/solr/content_shard3_replica_n22/ 57122.0 5111.33222469 Query time milliseconds [65826, 51309, 52595, 50432, 59987, 61371, 62022, 57667, 55639, 54372] p-solr-8-1.obscured.com:8983/solr/content_shard4_replica_n30/ 61678.3 4091.35454478 Query time milliseconds [60404, 55604, 62858, 67807, 64320, 63962, 66846, 58085, 58951, 57946] p-solr-8-29.obscured.com:8983/solr/content_shard4_replica_n34/ 64042.9 3646.86466403 Query time milliseconds [68498, 60725, 69115, 61942, 61398, 61874, 61255, 68909, 66059, 60654] p-solr-8-30.obscured.com:8983/solr/content_shard4_replica_n38/ 63116.5 4609.50824685 Query time milliseconds [72034, 66459, 56206, 56873, 64180, 60061, 63122, 63786, 63754, 64690] p-solr-8-9.obscured.com:8983/solr/content_shard4_replica_n36/ 63432.7 4300.8959803 Query time milliseconds [65943, 69316, 55002, 68917, 63472, 58904, 63211, 64688, 62819, 62055] p-solr-8-15.obscured.com:8983/solr/content_shard4_replica_n32/ 61906.5 2902.66395843 Query time milliseconds [67208, 65850, 57653, 60862, 59523, 59620, 62713, 61529, 61265, 62842] p-solr-8-7.obscured.com:8983/solr/content_shard5_replica_n48/ 59892.7 3490.10076423 Query time milliseconds [63098, 60924, 61086, 63765, 56105, 53150, 56886, 63446, 59949, 60518] p-solr-8-5.obscured.com:8983/solr/content_shard5_replica_n42/ 62159.0 5601.61476719 Query time milliseconds [69503, 56827, 55902, 61201, 56940, 65581, 71914, 63370, 63051, 57301] p-solr-8-8.obscured.com:8983/solr/content_shard5_replica_n44/ 63823.7 3858.87292267 Query time milliseconds [66818, 67892, 58788, 69172, 64827, 63549, 63268, 59215, 58640, 66068] p-solr-8-4.obscured.com:8983/solr/content_shard5_replica_n40/ 58128.5 4130.89318429 Query time milliseconds [66784, 56183, 61574, 55183, 57776, 57367, 60974, 52964, 53819, 58661] p-solr-8-14.obscured.com:8983/solr/content_shard5_replica_n46/ 61351.6 4037.90460511 Query time milliseconds [58759, 62324, 57665, 67046, 56590, 59296, 59116, 68687, 63815, 60218] p-solr-8-3.obscured.com:8983/solr/content_shard6_replica_n56/ 56639.2 4509.2680559 Query time milliseconds [52265, 49244, 51974, 60207, 57325, 58683, 63489, 54787, 61009, 57409] p-solr-8-22.obscured.com:8983/solr/content_shard6_replica_n58/ 57532.8 2690.59624949 Query time milliseconds [53768, 57966, 57145, 53659, 60496, 56692, 59881, 56240, 61887, 57594] p-solr-8-24.obscured.com:8983/solr/content_shard6_replica_n52/ 65245.2 3561.3285536 Query time milliseconds [62174, 63386, 59160, 68495, 70150, 64167, 70341, 66024, 64306, 64249] p-solr-8-2.obscured.com:8983/solr/content_shard6_replica_n50/ 58632.8 3571.57904077 Query time milliseconds [53143, 56298, 58081, 62260, 61433, 61427, 63434, 59773, 56407, 54072] p-solr-8-18.obscured.com:8983/solr/content_shard6_replica_n54/ 62589.2 4958.34174341 Query time milliseconds [59044, 64196, 68406, 63014, 61170, 59517, 63885, 72458, 57918, 56284] Thank you, Russ *Manzama*a MODERN GOVERNANCE company Russell Bahr Lead Infrastructure Engineer USA & CAN Office: +1 (541) 306 3271 USA & CAN Support: +1 (541) 706 9393 UK Office & Support: +44 (0)203 282 1633 AUS Office & Support: +61 (0) 2 8417 2339 543 NW York Drive, Suite 100, Bend, OR 97703 LinkedIn <http://www.linkedin.com/company/manzama> | Twitter <https://twitter.com/ManzamaInc> | Facebook <http://www.facebook.com/manzamainc> | YouTube <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> On Thu, Sep 5, 2019 at 8:50 PM David Smiley <david.w.smi...@gmail.com> wrote: > I suggest first working with a single machine to see if it responds > substantially slower with the new version. Just find one of yours and > issue it a query that will resolve locally (distrib=false param). Your > current collection level queries are internally issuing such queries, and > so with a little bit of sleuthing, looking at logs, you can find a shard > level query like this. If it's quick then there's some distributed aspect > to investigate. But you'll probably see the slowness here, and the problem > is better scoped and easier to diagnose. At this point look at timings > with debug=timing to see information on each of the components. That may > give you a strong clue. If it's in the QueryComponent which actually > executes the underlying search then you have some further digging to do. > Use a profiler like JVisualVM. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley >
