Hi Jan,
We are using 64 bit Java, version 1.8.0_191.
We started Solr with 6 GB heap size.
Besides Solr, we have ZooKeeper, IIS, Google Chrome and NotePad++ running
on the machine. There is still 22 GB of memory left on the server, out of
the 32 GB available on the machine.
Regards,
Edwin
On Fri, 25 Jan 2019 at 21:04, Jan Høydahl <jan....@cominvent.com> wrote:
> Which java version? 32 or 64 bit? You start Solr with default 512Mb heap
> size?
> Other software running on the machine?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 25. jan. 2019 kl. 13:05 skrev Zheng Lin Edwin Yeo <edwinye...@gmail.com
> >:
> >
> > Hi Jan and Shawn,
> >
> > For your info, this is another debug query.
> >
> > "debug":{
> >
> > "rawquerystring":"johnny",
> >
> > "querystring":"johnny",
> >
> > "parsedquery":"searchFields_tcs:johnny",
> >
> > "parsedquery_toString":"searchFields_tcs:johnny",
> >
> > "explain":{
> >
> > "192280":"\n12.8497505 = weight(searchFields_tcs:johnny in
> > 75730) [SchemaSimilarity], result of:\n 12.8497505 =
> > score(doc=75730,freq=4.0 = termFreq=4.0\n), product of:\n 7.5108404
> > = idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq +
> > 0.5)) from:\n 473.0 = docFreq\n 865438.0 = docCount\n
> > 1.7108272 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 -
> > b + b * fieldLength / avgFieldLength)) from:\n 4.0 =
> > termFreq=4.0\n 1.2 = parameter k1\n 0.75 = parameter b\n
> > 26.66791 = avgFieldLength\n 25.0 = fieldLength\n”,
> >
> > "QParser":"LuceneQParser",
> >
> > "timing":{
> >
> > "time":350.0,
> >
> > "prepare":{
> >
> > "time":0.0,
> >
> > "query":{
> >
> > "time":0.0},
> >
> > "facet":{
> >
> > "time":0.0},
> >
> > "facet_module":{
> >
> > "time":0.0},
> >
> > "mlt":{
> >
> > "time":0.0},
> >
> > "highlight":{
> >
> > "time":0.0},
> >
> > "stats":{
> >
> > "time":0.0},
> >
> > "expand":{
> >
> > "time":0.0},
> >
> > "terms":{
> >
> > "time":0.0},
> >
> > "debug":{
> >
> > "time":0.0}},
> >
> > "process":{
> >
> > "time":348.0,
> >
> > "query":{
> >
> > "time":287.0},
> >
> > "facet":{
> >
> > "time":0.0},
> >
> > "facet_module":{
> >
> > "time":0.0},
> >
> > "mlt":{
> >
> > "time":0.0},
> >
> > "highlight":{
> >
> > "time":54.0},
> >
> > "stats":{
> >
> > "time":0.0},
> >
> > "expand":{
> >
> > "time":0.0},
> >
> > "terms":{
> >
> > "time":0.0},
> >
> > "debug":{
> >
> > "time":6.0}},
> >
> > "loadFieldValues":{
> >
> > "time":0.0}}}}
> >
> >
> > Regards,
> > Edwin
> >
> >
> > On Fri, 25 Jan 2019 at 19:52, Zheng Lin Edwin Yeo <edwinye...@gmail.com>
> > wrote:
> >
> >> Hi Jan and Shawn,
> >>
> >> Please focus on the strange issue that I have described above in more
> >> details, summary is as follows:
> >>
> >> 1. Index customers data, then queries from highlight, select, and all
> >> handlers are very fast (less than 50ms)
> >>
> >> 2. Now index policies data, then queries on polices are very fast BUT
> >> queries on customers becomes slow
> >>
> >> 3. If I reindex customers data, then again queries for customers are
> very
> >> fast BUT queries on policies becomes slow.
> >>
> >> How can you explain this behavior?
> >>
> >> We have never experienced such a strange behavior before Solr 7.
> >>
> >> Regards,
> >> Edwin
> >>
> >> On Fri, 25 Jan 2019 at 17:06, Zheng Lin Edwin Yeo <edwinye...@gmail.com
> >
> >> wrote:
> >>
> >>> Hi Jan,
> >>>
> >>> Referring to what you have mentioned that the highlighting takes up
> most
> >>> of the time in the first query from the policies collection, the
> >>> highlighting was very fast (less than 50ms) from the time it was
> indexed,
> >>> till the time after customers collection gets indexed, in which it
> slowed
> >>> down tremendously.
> >>>
> >>> Also, the slow down does not just affect on the highlight
> requestHandler.
> >>> It also affects other requestHandler like select and clustering as
> well.
> >>> All of them gets the QTime of more than 500ms. This is also proven in
> the
> >>> latest debug query that we have sent earlier, in which we have set
> hl=false
> >>> and fl=null.
> >>>
> >>> Any idea or explanation on this strange behavior?
> >>> Thank you for your support, as we look forward to shed some lights on
> >>> this issue and to resolve it.
> >>>
> >>> Regards,
> >>> Edwin
> >>>
> >>> On Thu, 24 Jan 2019 at 23:35, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Jan,
> >>>>
> >>>> Thanks for your reply.
> >>>>
> >>>> However, we are still getting a slow QTime of 517ms even after we set
> >>>> hl=false&fl=null.
> >>>>
> >>>> Below is the debug query:
> >>>>
> >>>> "debug":{
> >>>> "rawquerystring":"cherry",
> >>>> "querystring":"cherry",
> >>>> "parsedquery":"searchFields_tcs:cherry",
> >>>> "parsedquery_toString":"searchFields_tcs:cherry",
> >>>> "explain":{
> >>>> "46226513":"\n14.227914 = weight(searchFields_tcs:cherry in
> 5747763) [SchemaSimilarity], result of:\n 14.227914 =
> score(doc=5747763,freq=3.0 = termFreq=3.0\n), product of:\n 9.614556 =
> idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5))
> from:\n 400.0 = docFreq\n 6000000.0 = docCount\n 1.4798305 =
> tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b *
> fieldLength / avgFieldLength)) from:\n 3.0 = termFreq=3.0\n 1.2 =
> parameter k1\n 0.75 = parameter b\n 19.397041 = avgFieldLength\n
> 25.0 = fieldLength\n",
> >>>> "54088731":"\n13.937909 = weight(searchFields_tcs:cherry in
> 4840794) [SchemaSimilarity], result of:\n 13.937909 =
> score(doc=4840794,freq=3.0 = termFreq=3.0\n), product of:\n 9.614556 =
> idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5))
> from:\n 400.0 = docFreq\n 6000000.0 = docCount\n 1.4496675 =
> tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b *
> fieldLength / avgFieldLength)) from:\n 3.0 = termFreq=3.0\n 1.2 =
> parameter k1\n 0.75 = parameter b\n 19.397041 = avgFieldLength\n
> 27.0 = fieldLength\n",
> >>>> "QParser":"LuceneQParser",
> >>>> "timing":{
> >>>> "time":517.0,
> >>>> "prepare":{
> >>>> "time":0.0,
> >>>> "query":{
> >>>> "time":0.0},
> >>>> "facet":{
> >>>> "time":0.0},
> >>>> "facet_module":{
> >>>> "time":0.0},
> >>>> "mlt":{
> >>>> "time":0.0},
> >>>> "highlight":{
> >>>> "time":0.0},
> >>>> "stats":{
> >>>> "time":0.0},
> >>>> "expand":{
> >>>> "time":0.0},
> >>>> "terms":{
> >>>> "time":0.0},
> >>>> "debug":{
> >>>> "time":0.0}},
> >>>> "process":{
> >>>> "time":516.0,
> >>>> "query":{
> >>>> "time":15.0},
> >>>> "facet":{
> >>>> "time":0.0},
> >>>> "facet_module":{
> >>>> "time":0.0},
> >>>> "mlt":{
> >>>> "time":0.0},
> >>>> "highlight":{
> >>>> "time":0.0},
> >>>> "stats":{
> >>>> "time":0.0},
> >>>> "expand":{
> >>>> "time":0.0},
> >>>> "terms":{
> >>>> "time":0.0},
> >>>> "debug":{
> >>>> "time":500.0}}}}}
> >>>>
> >>>> Regards,
> >>>> Edwin
> >>>>
> >>>>
> >>>> On Thu, 24 Jan 2019 at 22:43, Jan Høydahl <jan....@cominvent.com>
> wrote:
> >>>>
> >>>>> Looks like highlighting takes most of the time on the first query
> >>>>> (680ms). You config seems to ask for a lot of highlighting here,
> like 100
> >>>>> snippets of max 100000 characters etc.
> >>>>> Sounds to me that this might be a highlighting configuration problem.
> >>>>> Try to disable highlighting (hl=false) and see if you get back your
> speed.
> >>>>> Also, I see fl=* in your config, which is really asking for all
> fields.
> >>>>> Are you sure you want that, that may also be slow. Try to ask for
> just the
> >>>>> fields you will be using.
> >>>>>
> >>>>> --
> >>>>> Jan Høydahl, search solution architect
> >>>>> Cominvent AS - www.cominvent.com
> >>>>>
> >>>>>> 24. jan. 2019 kl. 14:59 skrev Zheng Lin Edwin Yeo <
> >>>>> edwinye...@gmail.com>:
> >>>>>>
> >>>>>> Thanks for your reply.
> >>>>>>
> >>>>>> Below are what you have requested about our Solr setup,
> configurations
> >>>>>> files ,schema and results of debug queries:
> >>>>>>
> >>>>>> Looking forward to your advice and support on our problem.
> >>>>>>
> >>>>>> 1. System configurations
> >>>>>> OS: Windows 10 Pro 64 bit
> >>>>>> System Memory: 32GB
> >>>>>> CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz, 4 Core(s), 8 Logical
> >>>>>> Processor(s)
> >>>>>> HDD: 3.0 TB (free 2.1 TB) SATA
> >>>>>>
> >>>>>> 2. solrconfig.xml of customers and policies collection, and solr.in
> >>>>> ,cmd
> >>>>>> which can be download from the following link:
> >>>>>>
> >>>>>
> https://drive.google.com/file/d/1AATjonQsEC5B0ldz27Xvx5A55Dp5ul8K/view?usp=sharing
> >>>>>>
> >>>>>> 3. The debug queries from both collections
> >>>>>>
> >>>>>> *3.1. Debug Query From Policies ( which is Slow)*
> >>>>>>
> >>>>>> "debug":{
> >>>>>>
> >>>>>> "rawquerystring":"sherry",
> >>>>>>
> >>>>>> "querystring":"sherry",
> >>>>>>
> >>>>>> "parsedquery":"searchFields_tcs:sherry",
> >>>>>>
> >>>>>> "parsedquery_toString":"searchFields_tcs:sherry",
> >>>>>>
> >>>>>> "explain":{
> >>>>>>
> >>>>>> "31702988":"\n14.540428 = weight(searchFields_tcs:sherry in
> >>>>>> 3097315) [SchemaSimilarity], result of:\n 14.540428 =
> >>>>>> score(doc=3097315,freq=5.0 = termFreq=5.0\n), product of:\n
> >>>>>> 8.907154 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
> >>>>>> (docFreq + 0.5)) from:\n 812.0 = docFreq\n 6000000.0 =
> >>>>>> docCount\n 1.6324438 = tfNorm, computed as (freq * (k1 + 1)) /
> >>>>>> (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:\n
> >>>>>> 5.0 = termFreq=5.0\n 1.2 = parameter k1\n 0.75 = parameter
> >>>>>> b\n 19.397041 = avgFieldLength\n 31.0 = fieldLength\n”,..
> >>>>>>
> >>>>>> "QParser":"LuceneQParser",
> >>>>>>
> >>>>>> "timing":{
> >>>>>>
> >>>>>> "time":681.0,
> >>>>>>
> >>>>>> "prepare":{
> >>>>>>
> >>>>>> "time":0.0,
> >>>>>>
> >>>>>> "query":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "facet":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "facet_module":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "mlt":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "highlight":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "stats":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "expand":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "terms":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "debug":{
> >>>>>>
> >>>>>> "time":0.0}},
> >>>>>>
> >>>>>> "process":{
> >>>>>>
> >>>>>> "time":680.0,
> >>>>>>
> >>>>>> "query":{
> >>>>>>
> >>>>>> "time":19.0},
> >>>>>>
> >>>>>> "facet":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "facet_module":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "mlt":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "highlight":{
> >>>>>>
> >>>>>> "time":651.0},
> >>>>>>
> >>>>>> "stats":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "expand":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "terms":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "debug":{
> >>>>>>
> >>>>>> "time":8.0}},
> >>>>>>
> >>>>>> "loadFieldValues":{
> >>>>>>
> >>>>>> "time":12.0}}}}
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *3.2. Debug Query From Customers (which is fast because we index it
> >>>>> after
> >>>>>> indexing Policies):*
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> "debug":{
> >>>>>>
> >>>>>> "rawquerystring":"sherry",
> >>>>>>
> >>>>>> "querystring":"sherry",
> >>>>>>
> >>>>>> "parsedquery":"searchFields_tcs:sherry",
> >>>>>>
> >>>>>> "parsedquery_toString":"searchFields_tcs:sherry",
> >>>>>>
> >>>>>> "explain":{
> >>>>>>
> >>>>>> "S7900271B":"\n13.191501 = weight(searchFields_tcs:sherry in
> >>>>>> 2453665) [SchemaSimilarity], result of:\n 13.191501 =
> >>>>>> score(doc=2453665,freq=3.0 = termFreq=3.0\n), product of:\n
> 9.08604
> >>>>>> = idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq +
> >>>>>> 0.5)) from:\n 428.0 = docFreq\n 3784142.0 = docCount\n
> >>>>>> 1.4518428 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1
> -
> >>>>>> b + b * fieldLength / avgFieldLength)) from:\n 3.0 =
> >>>>>> termFreq=3.0\n 1.2 = parameter k1\n 0.75 = parameter b\n
> >>>>>> 20.22558 = avgFieldLength\n 28.0 = fieldLength\n”, ..
> >>>>>>
> >>>>>> "QParser":"LuceneQParser",
> >>>>>>
> >>>>>> "timing":{
> >>>>>>
> >>>>>> "time":38.0,
> >>>>>>
> >>>>>> "prepare":{
> >>>>>>
> >>>>>> "time":1.0,
> >>>>>>
> >>>>>> "query":{
> >>>>>>
> >>>>>> "time":1.0},
> >>>>>>
> >>>>>> "facet":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "facet_module":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "mlt":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "highlight":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "stats":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "expand":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "terms":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "debug":{
> >>>>>>
> >>>>>> "time":0.0}},
> >>>>>>
> >>>>>> "process":{
> >>>>>>
> >>>>>> "time":36.0,
> >>>>>>
> >>>>>> "query":{
> >>>>>>
> >>>>>> "time":1.0},
> >>>>>>
> >>>>>> "facet":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "facet_module":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "mlt":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "highlight":{
> >>>>>>
> >>>>>> "time":31.0},
> >>>>>>
> >>>>>> "stats":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "expand":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "terms":{
> >>>>>>
> >>>>>> "time":0.0},
> >>>>>>
> >>>>>> "debug":{
> >>>>>>
> >>>>>> "time":3.0}},
> >>>>>>
> >>>>>> "loadFieldValues":{
> >>>>>>
> >>>>>> "time":13.0}}}}
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Edwin
> >>>>>>
> >>>>>> On Thu, 24 Jan 2019 at 20:57, Jan Høydahl <jan....@cominvent.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> It would be useful if you can disclose the machine configuration,
> OS,
> >>>>>>> memory, settings etc, as well as solr config including solr.in <
> >>>>>>> http://solr.in/>.sh, solrconfig.xml etc, so we can see the whole
> >>>>> picture
> >>>>>>> of memory, GC, etc.
> >>>>>>> You could also specify debugQuery=true on a slow search and check
> the
> >>>>>>> timings section for clues. What QTime are you seeing on the slow
> >>>>> queries in
> >>>>>>> solr.log?
> >>>>>>> If that does not reveal the reason, I'd connect to your solr
> >>>>> instance with
> >>>>>>> a tool like jVisualVM or similar, to inspect what takes time. Or
> >>>>> better,
> >>>>>>> hook up to DataDog, SPM or some other cloud tool to get a full view
> >>>>> of the
> >>>>>>> system.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Jan Høydahl, search solution architect
> >>>>>>> Cominvent AS - www.cominvent.com
> >>>>>>>
> >>>>>>>> 24. jan. 2019 kl. 13:42 skrev Zheng Lin Edwin Yeo <
> >>>>> edwinye...@gmail.com
> >>>>>>>> :
> >>>>>>>>
> >>>>>>>> Hi Shawn,
> >>>>>>>>
> >>>>>>>> Unfortunately your reply of memory may not be valid. Please refer
> >>>>> to my
> >>>>>>>> explanation below of the strange behaviors (is it much more like a
> >>>>> BUG
> >>>>>>> than
> >>>>>>>> anything else that is explainable):
> >>>>>>>>
> >>>>>>>> Note that we still have 18GB of free unused memory on the server.
> >>>>>>>>
> >>>>>>>> 1. We indexed the first collection called customers (3.7 millioin
> >>>>> records
> >>>>>>>> from CSV data), index size is 2.09GB. The search in customers for
> >>>>> any
> >>>>>>>> keyword is returned within 50ms (QTime) for using highlight
> (unified
> >>>>>>>> highlighter, posting, light term vectors)
> >>>>>>>>
> >>>>>>>> 2. Then we indexed the second collection called policies (6
> million
> >>>>>>> records
> >>>>>>>> from CSV data), index size is 2.55GB. The search in policies for
> any
> >>>>>>>> keyword is returned within 50ms (QTime) for using highlight
> (unified
> >>>>>>>> highlighter, posting, light term vectors)
> >>>>>>>>
> >>>>>>>> 3. But now any search in customers for any keywords (not from
> cache)
> >>>>>>> takes
> >>>>>>>> as high as 1200ms (QTime). But still policies search remains very
> >>>>> fast
> >>>>>>>> (50ms).
> >>>>>>>>
> >>>>>>>> 4. So we decided to run the force optimize command on customers
> >>>>>>> collection (
> >>>>>>>>
> >>>>>>>
> >>>>>
> https://localhost:8983/edm/customers/update?optimize=true&numSegments=1&waitFlush=false
> >>>>>>> ),
> >>>>>>>> surprisingly after optimization the search on customers collection
> >>>>> for
> >>>>>>> any
> >>>>>>>> keywords become very fast again (less than 50ms). BUT strangely,
> the
> >>>>>>> search
> >>>>>>>> in policies collection become very slow (around 1200ms) without
> any
> >>>>>>> changes
> >>>>>>>> to the policies collection.
> >>>>>>>>
> >>>>>>>> 5. Based on above result, we decided to run the force optimize
> >>>>> command on
> >>>>>>>> policies collection (
> >>>>>>>>
> >>>>>>>
> >>>>>
> https://localhost:8983/edm/policies/update?optimize=true&numSegments=1&waitFlush=false
> >>>>>>> ).
> >>>>>>>> More surprisingly, after optimization the search on policies
> >>>>> collection
> >>>>>>> for
> >>>>>>>> any keywords become very fast again (less than 50ms). BUT more
> >>>>> strangely,
> >>>>>>>> the search in customers collection again become very slow (around
> >>>>> 1200ms)
> >>>>>>>> without any changes to the customers collection.
> >>>>>>>>
> >>>>>>>> What a strange and unexpected behavior! If this is not a bug, how
> >>>>> could
> >>>>>>> you
> >>>>>>>> explain the above very strange behavior in Solr 7.5. Could it be a
> >>>>> bug?
> >>>>>>>>
> >>>>>>>> We would appreciate any support or help on our above situation.
> >>>>>>>>
> >>>>>>>> Thank you.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Edwin
> >>>>>>>>
> >>>>>>>> On Thu, 24 Jan 2019 at 16:14, Zheng Lin Edwin Yeo <
> >>>>> edwinye...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Shawn,
> >>>>>>>>>
> >>>>>>>>>> If the two collections have data on the same server(s), I can
> see
> >>>>> this
> >>>>>>>>>> happening. More memory is consumed when there is additional
> >>>>> data, and
> >>>>>>>>>> when Solr needs more memory, performance might be affected. The
> >>>>>>>>>> solution is generally to install more memory in the server.
> >>>>>>>>>
> >>>>>>>>> I have found that even after we delete the index in collection2,
> >>>>> the
> >>>>>>> query
> >>>>>>>>> QTime for collection1 still remains slow. It does not goes back
> to
> >>>>> its
> >>>>>>>>> previous fast speed before we index collection2.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Edwin
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, 24 Jan 2019 at 11:13, Zheng Lin Edwin Yeo <
> >>>>> edwinye...@gmail.com
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Shawn,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>
> >>>>>>>>>> The log only shows a list the following and I don't see any
> >>>>> other logs
> >>>>>>>>>> besides these.
> >>>>>>>>>>
> >>>>>>>>>> 2019-01-24 02:47:57.925 INFO (qtp2131952342-1330)
> [c:collectioin1
> >>>>>>>>>> s:shard1 r:core_node4 x:collection1_shard1_replica_n2]
> >>>>>>>>>> o.a.s.u.p.StatelessScriptUpdateProcessorFactory
> >>>>>>> update-script#processAdd:
> >>>>>>>>>> id=13245417
> >>>>>>>>>> 2019-01-24 02:47:57.957 INFO (qtp2131952342-1330)
> [c:collectioin1
> >>>>>>>>>> s:shard1 r:core_node4 x:collection1_shard1_replica_n2]
> >>>>>>>>>> o.a.s.u.p.StatelessScriptUpdateProcessorFactory
> >>>>>>> update-script#processAdd:
> >>>>>>>>>> id=13245430
> >>>>>>>>>> 2019-01-24 02:47:57.957 INFO (qtp2131952342-1330)
> [c:collectioin1
> >>>>>>>>>> s:shard1 r:core_node4 x:collection1_shard1_replica_n2]
> >>>>>>>>>> o.a.s.u.p.StatelessScriptUpdateProcessorFactory
> >>>>>>> update-script#processAdd:
> >>>>>>>>>> id=13245435
> >>>>>>>>>>
> >>>>>>>>>> There is no change to the segments info. but the slowdown in the
> >>>>> first
> >>>>>>>>>> collection is very drastic.
> >>>>>>>>>> Before the indexing of collection2, the collection1 query QTime
> >>>>> are in
> >>>>>>>>>> the range of 4 to 50 ms. However, after indexing collection2,
> the
> >>>>>>>>>> collection1 query QTime increases to more than 1000 ms. The
> index
> >>>>> are
> >>>>>>> done
> >>>>>>>>>> in CSV format, and the size of the index is 3GB.
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Edwin
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 24 Jan 2019 at 01:09, Shawn Heisey <apa...@elyograg.org
> >
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> On 1/23/2019 10:01 AM, Zheng Lin Edwin Yeo wrote:
> >>>>>>>>>>>> I am using Solr 7.5.0, and currently I am facing an issue of
> >>>>> when I
> >>>>>>> am
> >>>>>>>>>>>> indexing in collection2, the indexing affects the records in
> >>>>>>>>>>> collection1.
> >>>>>>>>>>>> Although the records are still intact, it seems that the
> >>>>> settings of
> >>>>>>>>>>> the
> >>>>>>>>>>>> termVecotrs get wipe out, and the index size of collection1
> >>>>> reduced
> >>>>>>>>>>> from
> >>>>>>>>>>>> 3.3GB to 2.1GB after I do the indexing in collection2.
> >>>>>>>>>>>
> >>>>>>>>>>> This should not be possible. Indexing in one collection should
> >>>>> have
> >>>>>>>>>>> absolutely no effect on another collection.
> >>>>>>>>>>>
> >>>>>>>>>>> If logging has been left at its default settings, the solr.log
> >>>>> file
> >>>>>>>>>>> should have enough info to show what actually happened.
> >>>>>>>>>>>
> >>>>>>>>>>>> Also, the search in
> >>>>>>>>>>>> collection1, which was originall very fast, becomes very slow
> >>>>> after
> >>>>>>> the
> >>>>>>>>>>>> indexing is done is collection2.
> >>>>>>>>>>>
> >>>>>>>>>>> If the two collections have data on the same server(s), I can
> >>>>> see this
> >>>>>>>>>>> happening. More memory is consumed when there is additional
> >>>>> data, and
> >>>>>>>>>>> when Solr needs more memory, performance might be affected.
> The
> >>>>>>>>>>> solution is generally to install more memory in the server. If
> >>>>> the
> >>>>>>>>>>> system is working, there should be no need to increase the heap
> >>>>> size
> >>>>>>>>>>> when the memory size increases ... but there can be situations
> >>>>> where
> >>>>>>> the
> >>>>>>>>>>> heap is a little bit too small, where you WOULD want to
> increase
> >>>>> the
> >>>>>>>>>>> heap size.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Shawn
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
>
>
