Hi! * Data: - Solr 3.4; - 30 shards ~ 13GB, 27-29M docs each shard.
* Machine parameters (Ubuntu 10.04 LTS): user@Solr:~$ uname -a Linux Solr 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux user@Solr:~$ cat /proc/cpuinfo processor : 0 - 3 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz stepping : 2 cpu MHz : 3458.000 cache size : 12288 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat bogomips : 6916.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: user@Solr:~$ cat /proc/meminfo MemTotal: 16992680 kB MemFree: 110424 kB Buffers: 9976 kB Cached: 11588380 kB SwapCached: 41952 kB Active: 9860764 kB Inactive: 6198668 kB Active(anon): 4062144 kB Inactive(anon): 398972 kB Active(file): 5798620 kB Inactive(file): 5799696 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 46873592 kB SwapFree: 46810712 kB Dirty: 36 kB Writeback: 0 kB AnonPages: 4424756 kB Mapped: 940660 kB Shmem: 40 kB Slab: 362344 kB SReclaimable: 350372 kB SUnreclaim: 11972 kB KernelStack: 2488 kB PageTables: 68568 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 55369932 kB Committed_AS: 5740556 kB VmallocTotal: 34359738367 kB VmallocUsed: 350532 kB VmallocChunk: 34359384964 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 10240 kB DirectMap2M: 17299456 kB - Apache Tomcat 6.0.32: <!-- java arguments --> -XX:+DisableExplicitGC -XX:PermSize=512M -XX:MaxPermSize=512M -Xmx12G -Xms3G -XX:NewSize=128M -XX:MaxNewSize=128M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=50 -XX:GCTimeRatio=9 -XX:MinHeapFreeRatio=25 -XX:MaxHeapFreeRatio=25 -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:/opt/search/tomcat/logs/gc.log Out search schema is: - 5 servers with configuration above; - one tomcat6 application on each server with 6 solr applications. - Full addresses are: 1) http://192.168.1.85:8080/solr1,http://192.168.1.85:8080/solr2,...,http://192.168.1.85:8080/solr6 2) http://192.168.1.86:8080/solr7,http://192.168.1.86:8080/solr8,...,http://192.168.1.86:8080/solr12 ... 5) http://192.168.1.89:8080/solr25,http://192.168.1.89:8080/solr26,...,http://192.168.1.89:8080/solr30 - At another server there is a additional "common" application with shards paramerter: <requestHandler name="search" class="solr.SearchHandler" default="true"> <lst name="defaults"> <str name="echoParams">explicit</str> <str name="shards">192.168.1.85:8080/solr1,192.168.1.85:8080/solr2,...,192.168.1.89:8080/solr30</str> <int name="rows">10</int> </lst> </requestHandler> - schema and solrconfig are identical for all shards, for first shard see attach; - on these servers are only search, indexation is on another (optimized to 2 segments shards replicate with ssh/rsync scripts). So now the major problem is huge performance on distributed search. Take look on, for example, these logs: This is on 30 shards: INFO: [] webapp=/solr path=/select/params={fl=*,score&ident=true&start=0&q=(barium)&rows=2000} status=0 QTime=40712 INFO: [] webapp=/solr path=/select/params={fl=*,score&ident=true&start=0&q=(pittances)&rows=2000} status=0 QTime=36097 INFO: [] webapp=/solr path=/select/params={fl=*,score&ident=true&start=0&q=(reliability)&rows=2000} status=0 QTime=75756 INFO: [] webapp=/solr path=/select/params={fl=*,score&ident=true&start=0&q=(blessing's)&rows=2000} status=0 QTime=30342 INFO: [] webapp=/solr path=/select/params={fl=*,score&ident=true&start=0&q=(reiterated)&rows=2000} status=0 QTime=55690 Sometimes QTime is more than 150000. But when we run identical queries on one shard separately, QTime is between 200 and 1500. Does ditributed solr search really slow or our architecture is non optimal? Or maybe need to use any third-party applications? Thanks for any replies. -- Best regards, Artem
