Re: Indexing in one collection affect index in another collection

Hi all, This issue is still surfacing in the new Soir 8.0.0. Can't really figure out what is the issue, as it occurs also in system with more memory. Anyone has any further insights on this? Regards, Edwin On Fri, 15 Feb 2019 at 18:40, Zheng Lin Edwin Yeo wrote: > Hi Shawn, > > This issue is

Re: Indexing in one collection affect index in another collection

Hi Shawn, This issue is also occurring in the new Solr 7.7.0, with only the same data size of 20 GB. Regards, Edwin On Fri, 8 Feb 2019 at 23:53, Zheng Lin Edwin Yeo wrote: > Hi Shawn, > > Thanks for your reply. > > Although the space in the OS disk cache could be the issue, but we didn't > fac

Re: Indexing in one collection affect index in another collection

Hi Shawn, Thanks for your reply. Although the space in the OS disk cache could be the issue, but we didn't face this problem previously, especially in our other setup using Solr 6.5.1, which contains much more data (more than 1 TB), as compared to our current setup in Solr 7.6.0, in which the dat

Re: Indexing in one collection affect index in another collection

On 2/6/2019 7:58 AM, Zheng Lin Edwin Yeo wrote: Hi everyone, Does anyone has further updates on this issue? It is my strong belief that all the software running on this server OTHER than Solr is competing with Solr for space in the OS disk cache, and that Solr's data is getting pushed out of

Re: Indexing in one collection affect index in another collection

ds. >>> >>> If subsequent queries are fast, then to me it does not seem like a >>> problem for a development machine. For production you may wish to store >>> the indices in ram and/or change from windows to linux, id it is important >>> that all qu

Re: Indexing in one collection affect index in another collection

;> >> Have a nice day >> Paul >> >> -Ursprüngliche Nachricht- >> Von: Shawn Heisey >> Gesendet: Dienstag, 29. Januar 2019 13:25 >> An: solr-user@lucene.apache.org >> Betreff: Re: Indexing in one collection affect index in another collection

Re: Indexing in one collection affect index in another collection

e > indices in ram and/or change from windows to linux, id it is important that > all queries including the first are very fast. > > Have a nice day > Paul > > -Ursprüngliche Nachricht- > Von: Shawn Heisey > Gesendet: Dienstag, 29. Januar 2019 13:25 > An: solr-us

Re: Indexing in one collection affect index in another collection

Hi Shawn, No worries, and thanks for your clarification. We make these changes in order to use the Unifed Highlighter, with hl.offsetSource = POSTING, and add "light" term vectors. The settings comes from what is written in the Solr guide on highlighting, which says the following: *Postings*: S

Re: Indexing in one collection affect index in another collection

On 1/29/2019 5:25 AM, Shawn Heisey wrote: Adding termVectors will make the index bigger.  Potentially much bigger. This will increase the overall RAM requirement of the server, especially if the server is handling software other than Solr.  Anything that makes the index bigger can affect perfor

Re: Indexing in one collection affect index in another collection

On 1/29/2019 5:06 AM, Zheng Lin Edwin Yeo wrote: My guess is after we change our searchFields_tcs schema which is: *From*: *To:* Adding termVectors will make the index bigger. Potentially much bigger. This will increase the overall RAM requirement of the server, especially if the server

AW: Indexing in one collection affect index in another collection

first are very fast. Have a nice day Paul -Ursprüngliche Nachricht- Von: Shawn Heisey Gesendet: Dienstag, 29. Januar 2019 13:25 An: solr-user@lucene.apache.org Betreff: Re: Indexing in one collection affect index in another collection On 1/29/2019 5:06 AM, Zheng Lin Edwin Yeo wrote

AW: Indexing in one collection affect index in another collection

. Januar 2019 13:31 An: 'solr-user@lucene.apache.org' Betreff: AW: Indexing in one collection affect index in another collection Hi If the reason for the difference in speed is that the index is being read from disk, I would expect that the first query would be slow, but subsequent q

Re: Indexing in one collection affect index in another collection

Hi Shawn, Thanks for you reply. However, we did not delete our index when the screenshot was taken. All the indexes are still in Solr. My guess is after we change our searchFields_tcs schema which is: *From*: *To:* The above change was done in order to use the Solr recommended unified highl

Re: Indexing in one collection affect index in another collection

On 1/26/2019 4:48 PM, Zheng Lin Edwin Yeo wrote: Thanks for your reply. Below are the replies to your email: 1) We have tried to set the heap size to be 8g previously when we faced the same issue, and changing to 7g does not help too. 2) We are using standard disk at the moment. 3) In the link

Re: Indexing in one collection affect index in another collection

Hi Shawn / Jan, Do we have any further insights about this problem? The same problem still happens even after we make the changes and re-index all the data. Regards, Edwin On Sun, 27 Jan 2019 at 07:48, Zheng Lin Edwin Yeo wrote: > Hi Shawn, > > Thanks for your reply. Below are the replies to y

Re: Indexing in one collection affect index in another collection

Hi Shawn, Thanks for your reply. Below are the replies to your email: 1) We have tried to set the heap size to be 8g previously when we faced the same issue, and changing to 7g does not help too. 2) We are using standard disk at the moment. 3) In the link is the screenshot of the process list t

Re: Indexing in one collection affect index in another collection

On 1/26/2019 9:40 AM, Zheng Lin Edwin Yeo wrote: We have tried to add -a "-XX:+AlwaysPreTouch" that starts Solr, but there is no noticeable difference in the performance. As for the screenshot, I have captured another one after we added -a "-XX:+AlwaysPreTouch", and it is sorted on the Working

Re: Indexing in one collection affect index in another collection

Hi Shawn, We have tried to add -a "-XX:+AlwaysPreTouch" that starts Solr, but there is no noticeable difference in the performance. As for the screenshot, I have captured another one after we added -a "-XX:+AlwaysPreTouch", and it is sorted on the Working Set column. Below is the link to the new

Re: Indexing in one collection affect index in another collection

On 1/25/2019 9:11 AM, Zheng Lin Edwin Yeo wrote: As requested, below is the link to the screenshot of the resource monitor of our system. https://drive.google.com/file/d/1_-Tqhk9YYp9w8injHU4ZPSvdFJOx8A5s/view?usp=sharing The wiki page says to sort on the Working Set column. Your screenshot sh

Re: Indexing in one collection affect index in another collection

Hi Jorn, I have set the heap size to 6GB, and the system has 32GB of RAM. The data is indexed from CSV file, so each field's data is like database type of data. Only the searchFields may have more data as it contains the important fields of the collection. But then again it is not as large as thi

Re: Indexing in one collection affect index in another collection

Hi Shawn, As requested, below is the link to the screenshot of the resource monitor of our system. https://drive.google.com/file/d/1_-Tqhk9YYp9w8injHU4ZPSvdFJOx8A5s/view?usp=sharing Regards, Edwin On Fri, 25 Jan 2019 at 23:35, Shawn Heisey wrote: > On 1/25/2019 7:47 AM, Zheng Lin Edwin Yeo wro

Re: Indexing in one collection affect index in another collection

On 1/25/2019 7:47 AM, Zheng Lin Edwin Yeo wrote: Below is the command that we used to start Solr: cd solr-7.5.0 bin\solr.cmd start -cloud -p 8983 -s solrMain\node1 -m 6g -z "localhost:2181,localhost:2182,localhost:2183" -Dsolr.ltr.enabled=true pause Can you gather the screenshot mentioned here

Re: Indexing in one collection affect index in another collection

Hi Jan, Below is the command that we used to start Solr: cd solr-7.5.0 bin\solr.cmd start -cloud -p 8983 -s solrMain\node1 -m 6g -z "localhost:2181,localhost:2182,localhost:2183" -Dsolr.ltr.enabled=true pause We also have a replica, and in this development setting, we put it in the same PC to s

Re: Indexing in one collection affect index in another collection

How do you start Solr, cause the solr.in.cmd you sent does not contain the memory settings. What other parameters do you start Solr with? -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 25. jan. 2019 kl. 15:28 skrev Zheng Lin Edwin Yeo : > > Hi Jan, > > We are usin

Re: Indexing in one collection affect index in another collection

Hi Jan, We are using 64 bit Java, version 1.8.0_191. We started Solr with 6 GB heap size. Besides Solr, we have ZooKeeper, IIS, Google Chrome and NotePad++ running on the machine. There is still 22 GB of memory left on the server, out of the 32 GB available on the machine. Regards, Edwin On Fr

Re: Indexing in one collection affect index in another collection

Which java version? 32 or 64 bit? You start Solr with default 512Mb heap size? Other software running on the machine? -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 25. jan. 2019 kl. 13:05 skrev Zheng Lin Edwin Yeo : > > Hi Jan and Shawn, > > For your info, this is

Re: Indexing in one collection affect index in another collection

Have you done a correct sizing wrt to memory / CPU? Check also the data model if you have a lot of queried stored fields that may contain a lot of data. You may also split those two collections on different nodes. > Am 23.01.2019 um 18:01 schrieb Zheng Lin Edwin Yeo : > > Hi, > > I am using

Re: Indexing in one collection affect index in another collection

Hi Jan and Shawn, For your info, this is another debug query. "debug":{ "rawquerystring":"johnny", "querystring":"johnny", "parsedquery":"searchFields_tcs:johnny", "parsedquery_toString":"searchFields_tcs:johnny", "explain":{ "192280":"\n12.8497505 = weight(searc

Re: Indexing in one collection affect index in another collection

Hi Jan and Shawn, Please focus on the strange issue that I have described above in more details, summary is as follows: 1. Index customers data, then queries from highlight, select, and all handlers are very fast (less than 50ms) 2. Now index policies data, then queries on polices are very fast

Re: Indexing in one collection affect index in another collection

Hi Jan, Referring to what you have mentioned that the highlighting takes up most of the time in the first query from the policies collection, the highlighting was very fast (less than 50ms) from the time it was indexed, till the time after customers collection gets indexed, in which it slowed down

Re: Indexing in one collection affect index in another collection

Hi Jan, Thanks for your reply. However, we are still getting a slow QTime of 517ms even after we set hl=false&fl=null. Below is the debug query: "debug":{ "rawquerystring":"cherry", "querystring":"cherry", "parsedquery":"searchFields_tcs:cherry", "parsedquery_toString":"search

Re: Indexing in one collection affect index in another collection

Looks like highlighting takes most of the time on the first query (680ms). You config seems to ask for a lot of highlighting here, like 100 snippets of max 10 characters etc. Sounds to me that this might be a highlighting configuration problem. Try to disable highlighting (hl=false) and see

Re: Indexing in one collection affect index in another collection

Thanks for your reply. Below are what you have requested about our Solr setup, configurations files ,schema and results of debug queries: Looking forward to your advice and support on our problem. 1. System configurations OS: Windows 10 Pro 64 bit System Memory: 32GB CPU: Intel(R) Core(TM) i7-47

Re: Indexing in one collection affect index in another collection

It would be useful if you can disclose the machine configuration, OS, memory, settings etc, as well as solr config including solr.in .sh, solrconfig.xml etc, so we can see the whole picture of memory, GC, etc. You could also specify debugQuery=true on a slow search and check the

Re: Indexing in one collection affect index in another collection

Hi Shawn, Unfortunately your reply of memory may not be valid. Please refer to my explanation below of the strange behaviors (is it much more like a BUG than anything else that is explainable): Note that we still have 18GB of free unused memory on the server. 1. We indexed the first collection c

Re: Indexing in one collection affect index in another collection

Hi Shawn, > If the two collections have data on the same server(s), I can see this > happening. More memory is consumed when there is additional data, and > when Solr needs more memory, performance might be affected. The > solution is generally to install more memory in the server. I have found

Re: Indexing in one collection affect index in another collection

Hi Shawn, Thanks for your reply. The log only shows a list the following and I don't see any other logs besides these. 2019-01-24 02:47:57.925 INFO (qtp2131952342-1330) [c:collectioin1 s:shard1 r:core_node4 x:policies_shard1_replica_n2] o.a.s.u.p.StatelessScriptUpdateProcessorFactory update-sc

Re: Indexing in one collection affect index in another collection

On 1/23/2019 10:01 AM, Zheng Lin Edwin Yeo wrote: I am using Solr 7.5.0, and currently I am facing an issue of when I am indexing in collection2, the indexing affects the records in collection1. Although the records are still intact, it seems that the settings of the termVecotrs get wipe out, and

Indexing in one collection affect index in another collection

Hi, I am using Solr 7.5.0, and currently I am facing an issue of when I am indexing in collection2, the indexing affects the records in collection1. Although the records are still intact, it seems that the settings of the termVecotrs get wipe out, and the index size of collection1 reduced from 3.3