Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Hi Shawn, I tried optimizing using this command... curl ' http://10.7.233.54:8088/solr/update?optimize=true&maxSegments=10&waitFlush=true' And i got this response within secs... 0840 Is this a valid response that one should get ? I checked the statistics link from /solr/admin page and it shows the number segments got updated. Would this be a good indication that optimization is complete ? At the same time - I even noticed the number of files in data/index directory hasn't reduced & all files are not updated. Since it took just couple of secs for the response(even with waitFlush=true) - i am doubting if optimization really happened , but details on statistics page shows me correct number of segments. On Tue, Mar 12, 2013 at 8:34 PM, Shawn Heisey-4 [via Lucene] < ml-node+s472066n4046834...@n3.nabble.com> wrote: > On 3/12/2013 4:17 PM, feroz_kh wrote: > > Do we really need to optimize in order to reformat ? > > The alternative would be to start with an empty index and just reindex > your data. That is actually the best way to go, if that option is > available to you. > > > If yes, What is the best way of optimizing index - Online or Offline ? > > Can we do it online ? If yes - > > 1. What is the http request which we can use to invoke optimization - > How > > long it takes ? > > 2. What is the command line command to invoked optimization - How long > this > > one takes ? > > The only way I know of to optimize an index that's offline is using > Luke, but it is difficult to find versions of Luke that work with > indexes after 4.0-ALPHA - the official Luke page doesn't have any newer > versions, and I have no idea why. Online is better. Solr 4.2 just got > released, you may want to consider skipping 4.1 and going with 4.2. > > There would be no major speed difference between doing it offline or > online. Whatever else the machine is doing might be a factor. I can > only make guesses about how long it will take. You say your index in > 3.5 is 14GB. I have experience with indexes that are 22GB in 3.5, which > takes 11 minutes to optimize. The equivalent index in 4.2 is 14GB and > takes 14 minutes, because of the extra compression/decompression step. > This is on RAID10, volumes with no RAID or with other RAID levels would > be slower. Also, if the structure of your index is significantly > different than mine, yours might go faster or slower than the size alone > would suggest. > > There is a curl command that optimizes the index in the wiki: > > > http://wiki.apache.org/solr/UpdateXmlMessages#Passing_commit_and_commitWithin_parameters_as_part_of_the_URL > > You would want to leave off the "maxSegments" option so it optimizes > down to one segment. Whether to include waitFlush is up to you, but if > you don't include it, you won't know exactly when it finishes unless you > are looking at the index directory. > > Thanks, > Shawn > > > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4046834.html > To unsubscribe from Upgrade Solr3.5 to Solr4.1 - Index Reformat ?, click > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4046391&code=ZmVyb3oua2gyMDAwQGdtYWlsLmNvbXw0MDQ2MzkxfDIwNzA2NTYxOTI=> > . > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4052969.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Hi Shawn, I tried optimizing using this command... curl 'http://localhost:/solr/update?optimize=true&maxSegments=10&waitFlush=true' And i got this response within secs... 0840 Is this a valid response that one should get ? I checked the statistics link from /solr/admin page and it shows the number segments got updated. Would this be a good indication that optimization is complete ? At the same time - I even noticed the number of files in data/index directory hasn't reduced & all files are not updated. Since it took just couple of secs for the response(even with waitFlush=true) - i am doubting if optimization really happened , but details on statistics page shows me correct number of segments. -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4052970.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Also, Is it absolutely necessary to set the maxSegments=1 , if we need to reformat the whole index ? -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4052991.html Sent from the Solr - User mailing list archive at Nabble.com.
Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Hello, We are planning to upgrade our solr servers from version 3.5 to 4.1. We have master slave configuration and the index size is quite big (i.e. around 14 GB ). 1. Do we really need to re-format the whole index , when we upgrade to 4.1 ? 2. What will be the consequences - if we do not re-format and simply upgrade war file and config files ( solrconfig.xml, schema.xml ) on all slaves and master together. (Shutdown all master & slaves and then upgrade & startup) ? 3. If re-formatting is neccessary - then what is the best tool to achieve it. ( How long does it usually take to re-format the index of size around 14GB ) ? Thanks, Feroz -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Thanks Shawn. So if we have new segments in 4.1 format and all old files in 3.5 format at the same time, then will it cause any performance degradation on slaves while reading index files ( which will contain both 3.5 formatted and 4.1 formatted files)? -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4046469.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Thanks Tomas! I see the latest available version is 4.1 - but you have suggested a 4.2 version, where can i grab 4.2 version from? -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4046471.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Thanks Tomas/Shawn! One more question related to backward compatibilty. Previously we had upgraded our solr master/slaves from 1.4 version to 3.5 version - We didn't reformat the whole index then. So i believe there will be some files with 1.4 format present in our index. Now when we upgrade from 3.5 to 4.1/or4.2 - Can we expect solr slave version 4.x to read both 1.4 and 3.5 formatted indices, without any issues ? Thanks, Feroz -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4046500.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Upgrade Solr3.5 to Solr4.1 - Index Reformat ?
Hi Shawn, Do we really need to optimize in order to reformat ? If yes, What is the best way of optimizing index - Online or Offline ? Can we do it online ? If yes - 1. What is the http request which we can use to invoke optimization - How long it takes ? 2. What is the command line command to invoked optimization - How long this one takes ? Offline - 1. What's offline way/tool of invoking index optimization ? If No, what is the best way to only reformat index - Online or Offline ? How long each takes ? What's the online and offline ways/tool to reformat index ? Thanks, Feroz. -- View this message in context: http://lucene.472066.n3.nabble.com/Upgrade-Solr3-5-to-Solr4-1-Index-Reformat-tp4046391p4046802.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr Index linear growth - Performance degradation.
We have 4 shards with 14GB index on each of them Each shard has a master and 3 slaves(each of them with 32GB RAM) We're expecting that the index size will grow to double or triple in near future. So we thought of merging our indexes to 28GB index so that each shard has 28GB index and also increased our RAM on each slave to 48GB. We made this changes locally and tested the server by sending same 10K realistic queries to each server with 14GB & 28GB index, we found that 1. For server with 14GB index(48GB RAM): search time was 480ms, number of index hits: 3.8GB 2. For server with 28GB index(48GB RAM): search time was 900ms, number of index hits: 7.2GB. So we saw that having the whole index in RAM doesn't help in sustaining the performance in terms of search time . Search time increased linearly to double when the index size was doubled. We were thinking of keeping only 4 shards configuration but it looks like now we have to add another shard or another slave to each shard. Is there way we can configure our servers so that the performance isn't affected even when index size doubles or triples ? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
Here's few list of queries --- parallel zur xml beschreibungsdatei gibt es eine die verbindung zwischen beiden sei ten geschieht die owner klasse muss sich aus der benutzer ein oder mehrere lieblingsfarben ausw hlen kann found sample questions at http bjs ojp but more important parents need to keep --- Here's the jvm ram assignment -Xms24576m -Xmx24576m -XX:NewSize=6168m -XX:MaxNewSize=6168m -XX:MaxPermSize=1024m I believe that's enough assigned there... - I am not dealing with adding new documents here Just testing the solr index search - i just the have the indexes. For 14GB index the RAM cache gets filled with 14 GB around For 28GB index the RAM cache gets filled with 28GB around The Document cache size is 200MB max and initial 20MB. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001011.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
1. So we have 24.5GB assigned to jvm which is half of the total memory, which is 48GB RAM.(If that's what you meant, and if i am getting that right ?) 2. Size of *.fdt and *fdx is around 300m and 50m respectively.So that's definitely less that 5%. Do you see a problem there ? Is there a way that we can force or tune in such a way that the response time remains constant or doesn't degrade a lot(i.e. almost doubling) when the index size is doubled ? Or we cannot do anything about it ? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001034.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
It looks like reducing the jvm heap allocation did help in lowering the response time to some extent. Thanks for the pointer. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001056.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
index hits == total number of documents found by search query. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001063.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
Its 7.2Gig Hits. (GB was typo) This is the total number of index hits - calculated by summing each "numFound" attribute from solr query response. We have RHEL Tikanga version. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001061.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
Its 7,200,000 hits == number of documents found by all 10K queries. We have RHEL tikanga version. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001069.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
The queries were extracted from production log. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001182.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Index linear growth - Performance degradation.
These are simple search queries and Its multithreaded . -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Index-linear-growth-Performance-degradation-tp4000934p4001184.html Sent from the Solr - User mailing list archive at Nabble.com.
