Hi Danilo, following my experience now SSD or RAM Disk is the only way to speed up queries. It depends on your storage occupation of your 41M docs. If you don't have Enterprise SSD you can add consumer SSD as a fast cache (linux caching modules "flashcache / bcache" are able to use cheap SSD as a data cache and have your data safe stored on SATA Disks).
I don't think you can increase performances without changing technology on the storage system. Regards. Dario Il giorno lun 22 feb 2021 alle ore 08:52 Danilo Tomasoni <tomas...@cosbi.eu> ha scritto: > Hello all, > we are running a solr instance with around 41 MLN documents on a SATA > class 10 disk with around 10.000 rpm. > We are experiencing very slow query responses (in the order of hours..) > with an average of 205 segments. > We made a test with a normal pc and an SSD disk, and there the same solr > instance with the same data and the same number of segments was around 45 > times faster. > Force optimize was also tried to improve the performances, but it was very > slow, so we abandoned it. > > Since we still don't have enterprise server ssd disks, we are now > wondering if in the meanwhile defragmenting the solrdata folder can help. > The idea is that due to many updates, each segment file is fragmented > across different phisical blocks. > Put in another way, each segment file is non-contiguous on disk, and this > can slow-down the solr response. > > What do you suggest? > Is this somewhat equivalent to force-optimize or it can be faster? > > Thank you. > Danilo > > Danilo Tomasoni > > Fondazione The Microsoft Research - University of Trento Centre for > Computational and Systems Biology (COSBI) > Piazza Manifattura 1, 38068 Rovereto (TN), Italy > tomas...@cosbi.eu< > https://webmail.cosbi.eu/owa/redir.aspx?C=VNXi3_8-qSZTBi-FPvMwmwSB3IhCOjY8nuCBIfcNIs_5SgD-zNPWCA..&URL=mailto%3acalabro%40cosbi.eu > > > http://www.cosbi.eu< > https://webmail.cosbi.eu/owa/redir.aspx?C=CkilyF54_imtLHzZqF1gCGvmYXjsnf4bzGynd8OXm__5SgD-zNPWCA..&URL=http%3a%2f%2fwww.cosbi.eu%2f > > > > As for the European General Data Protection Regulation 2016/679 on the > protection of natural persons with regard to the processing of personal > data, we inform you that all the data we possess are object of treatment in > the respect of the normative provided for by the cited GDPR. > It is your right to be informed on which of your data are used and how; > you may ask for their correction, cancellation or you may oppose to their > use by written request sent by recorded delivery to The Microsoft Research > – University of Trento Centre for Computational and Systems Biology Scarl, > Piazza Manifattura 1, 38068 Rovereto (TN), Italy. > P Please don't print this e-mail unless you really need to > -- Dario Rigolin Comperio srl - CTO Mobile: +39 347 7232652 - Office: +39 0425 471482 Skype: dario.rigolin
