Re: using let() with other streaming expressions

2019-08-16 Thread Joel Bernstein
Yes, the examples you show will fail because the "let" expression reads streams into an in-memory List. All the Streaming Expressions expect a TupleStream to be passed in rather that a List. There is an undocumented function that turns a List of tuples back into a Stream. The function is called "s

Re: Solr crash | GC issue

2019-08-16 Thread Rohan Kasat
Thanks Shawn. I saw that error when the solr crashed last time. Am waiting to see if it happens again and to capture the compete error log. Regards, Rohan Kasat On Fri, Aug 16, 2019 at 2:36 PM Shawn Heisey wrote: > On 8/16/2019 11:59 AM, Rohan Kasat wrote: > > I see the OOM file getting created

Re: Solr crash | GC issue

2019-08-16 Thread Shawn Heisey
On 8/16/2019 11:59 AM, Rohan Kasat wrote: I see the OOM file getting created with "not much heap space" as the error Can you get the precise error cause? I haven't ever seen that particular text before. If you can paste the entire error (which will be many lines), that can be helpful. Sh

Re: Solr crash | GC issue

2019-08-16 Thread Rohan Kasat
Thanks Shawn and Paul. I tried using the https://gceasy.io/ but was not able to understand much. I see the OOM file getting created with "not much heap space" as the error . Shawn, i have tried your CMS settings too and now will try increasing the heap memory, hope it works this time. Any things s

Re: Solr is very slow with term vectors

2019-08-16 Thread Walter Underwood
First, time fetching one million records with all the fields you need, both for display and for re-ranking. If that is slow, then no amount of cosine code tweaking will make it fast. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 16, 2019, at 9

Re: "Missing" Docs in Solr

2019-08-16 Thread Brian Lininger
It's just a single query that results in 0 hits, I had the same thought of just adding code to retry the query when we get 0 hits (assuming that we expect there to be hits). That's likely going to be the interim solution so that we can get more info when this occurs. It's hard to triage when it's

Re: "Missing" Docs in Solr

2019-08-16 Thread Alexandre Rafalovitch
is there several 0 results in a row as an anomaly. Or really just one? You could nearly add SolrJ code to rerun 0-result query with full debug on if it is a rare enough event. Regards, Alex On Fri, Aug 16, 2019, 12:05 PM Brian Lininger, wrote: > Yeah, I thought of those same problems at f

Re: Solr cloud questions

2019-08-16 Thread Shawn Heisey
On 8/15/2019 8:14 AM, Kojo wrote: I am starting to think that my setup has more than one problem. As I said before, I am not balancing my load to Solr nodes, and I have eight nodes. All of my web application requests go to one Solr node, the only one that dies. If I distribute the load across the

Re: Solr is very slow with term vectors

2019-08-16 Thread Jan Høydahl
I bet your main issue is assuming that this particular plugin is the only way to solve your ranking requirements. I would advise you to start looking into the various built-in Similarities and instead try to tweak one of those, and/or adding more ranking signals to your solution, perhaps see if

Re: Solr crash | GC issue

2019-08-16 Thread Shawn Heisey
On 8/16/2019 8:23 AM, Rohan Kasat wrote: I have a Solr Cloud setup of 3 solr servers 7.5 version. 24GB heap memory is allocated to each solr server and i have around 655 GB of data in indexes to be searched for. Few last 2-3 days, the solr servers are crashing and am able to see the heap memory

RE: Solr crash | GC issue

2019-08-16 Thread Paul Russell
For quick analysis we use https://gceasy.io Very information and quick turnaround. Paul --- Begin Message --- Hi All, I have a Solr Cloud setup of 3 solr servers 7.5 version. 24GB heap memory is allocated to each solr server and i have around 655 GB of data in indexes to be searched f

Re: "Missing" Docs in Solr

2019-08-16 Thread Brian Lininger
Yeah, I thought of those same problems at first and expected to find something but no luck. There are no errors in the solr log for the hour before/after the time that we saw the issue, the only warnings I see are "PERFORMANCE WARNING: Overlapping onDeckSearchers=2" but these are for other Collecti

Re: Solr cloud questions

2019-08-16 Thread Kojo
Ere, thanks for the advice. I don´t have this specific use case, but I am doing some operations that I think could be risky, due to the first time I am using. There is a page that groups by one specific attribute of documents distributed accros shards. I am using Composite ID to allow grouping cor

Solr crash | GC issue

2019-08-16 Thread Rohan Kasat
Hi All, I have a Solr Cloud setup of 3 solr servers 7.5 version. 24GB heap memory is allocated to each solr server and i have around 655 GB of data in indexes to be searched for. Few last 2-3 days, the solr servers are crashing and am able to see the heap memory is almost full but the CPU usage i

Re: Solr is very slow with term vectors

2019-08-16 Thread Jörn Franke
You would have to implement that I don’t think that Solr is threading the query parser magically for you, but maybe some people have more insight on this topic. > Am 16.08.2019 um 15:42 schrieb Vignan Malyala : > > How do I check that in solr? Can anyone share link on implementation of > threads

Re: Solr is very slow with term vectors

2019-08-16 Thread Vignan Malyala
How do I check that in solr? Can anyone share link on implementation of threads in solr? On Fri 16 Aug, 2019, 4:52 PM Jörn Franke, wrote: > Is your custom query parser multithreaded and leverages all cores? > > > Am 16.08.2019 um 13:12 schrieb Vignan Malyala : > > > > I want response time below

Re: HttpShardHandlerFactory

2019-08-16 Thread Shawn Heisey
On 8/16/2019 3:51 AM, Mark Robinson wrote: I am trying to understand the socket time out and connection time out in the HttpShardHandlerFactory:- 10 20 The shard handler is used when that Solr instance needs to make connections to another Solr i

Re: Solr is very slow with term vectors

2019-08-16 Thread Jörn Franke
Is your custom query parser multithreaded and leverages all cores? > Am 16.08.2019 um 13:12 schrieb Vignan Malyala : > > I want response time below 3 seconds. > And fyi I'm already using 32 cores. > My cache is already full too and obviously same requests don't occur in my > case. > > >> On Fri

Re: Solr is very slow with term vectors

2019-08-16 Thread Vignan Malyala
I want response time below 3 seconds. And fyi I'm already using 32 cores. My cache is already full too and obviously same requests don't occur in my case. On Fri 16 Aug, 2019, 11:47 AM Jörn Franke, wrote: > How much response time do you require? > I think you have to solve the issue in your cod

Re: "Missing" Docs in Solr

2019-08-16 Thread Alexandre Rafalovitch
I would take the server log for those 10 seconds (plus buffer) and really try to see if something happens in that period. I am thinking an unexpected commit, index large, alias switch. That may help you to narrow down the kind of error. Another option is whether you got empty result or a connecti

HttpShardHandlerFactory

2019-08-16 Thread Mark Robinson
Hello, I am trying to understand the socket time out and connection time out in the HttpShardHandlerFactory:- 10 20 1.Could some one please help me understand the effect of using such low values of 10 ms and 20ms as given above inside my /select ha

Re: "Missing" Docs in Solr

2019-08-16 Thread Zheng Lin Edwin Yeo
Hi, Did you encounter any error message during those occasions where you get 0 hits returned? Regards, Edwin On Fri, 16 Aug 2019 at 06:02, Brian Lininger wrote: > Hi All, > I'm seeing some odd behavior that I'm hoping someone might have encountered > before. We're using Solr 6.6.6 and very in

using let() with other streaming expressions

2019-08-16 Thread Viktors Belovs
Dear Solr Comunity, Recently I've been working with the 'let()' expression. And I got in a sort of trouble, when I was trying combining it with the different streaming expressions, as well as trying to re-assign variables. As an example: let( a=search(techproducts, q="cat:electronics", fl="id,

Re: Solr cloud questions

2019-08-16 Thread Ere Maijala
Does your web application, by any chance, allow deep paging or something like that which requires returning rows at the end of a large result set? Something like a query where you could have parameters like &rows=10&start=100 ? That can easily cause OOM with Solr when using a sharded index. It