Re: Query takes more time in Solr 8.5.1 compare to 6.1.0 version

[vishal patel]

Hi Jason

Thanks for reply.

I have checked jay's query using "terms" query parser and it is really helpful 
to us. After execute using "terms" query parser it will come within a 500 
milliseconds even though grouping is applied.
Jay's Query : 
https://drive.google.com/file/d/1bavCqwHfJxoKHFzdOEt-mSG8N0fCHE-w/view

Actually I want to apply same things in my query but my field "msg_id" is 
applied boost.group is also used in my query.
I am also upgrading Solr 8.5.1.


MY query is : 
https://drive.google.com/file/d/1Op_Ja292Bcnv0Ijxw6VdAxvGlfsdczmS/view

I got 30 seconds for above query. How can I use the "terms" query parser in my 
query?

Regards,
Vishal Patel

From: Jason Gerlowski 
Sent: Friday, May 22, 2020 2:59 AM
To: solr-user@lucene.apache.org 
Subject: Re: Query takes more time in Solr 8.5.1 compare to 6.1.0 version

Hi Jay,

I can't speak to why you're seeing a performance change between 6.x
and 8.x.  What I can suggest though is an alternative way of
formulating the query: you might get different performance if you run
your query using Solr's "terms" query parser:
https://lucene.apache.org/solr/guide/8_5/other-parsers.html#terms-query-parser
 It's not guaranteed to help, but there's a chance it'll work for you.
And knowing whether or not it helps might point others here towards
the cause of your slowdown.

Even if "terms" performs better for you, it's probably worth
understanding what's going on here of course.

Are all other queries running comparably?

Jason

On Thu, May 21, 2020 at 10:25 AM jay harkhani  wrote:
>
> Hello,
>
> Please refer below details.
>
> >Did you create Solrconfig.xml for the collection from scratch after 
> >upgrading and reindexing?
> Yes, We have created collection from scratch and also re-indexing.
>
> >Was it based on the latest template?
> Yes, It was as per latest template.
>
> >What happens if you reexecute the query?
> Not more visible difference. Minor change in milliseconds.
>
> >Are there other processes/containers running on the same VM?
> No
>
> >How much heap and how much total memory you have?
> My heap and total memory are same as Solr 6.1.0. heap memory 5 gb and total 
> memory 25gb. As per me there is no issue related to memory.
>
> >Maybe also you need to increase the corresponding caches in the config.
> We are not using cache in both version.
>
> Both version have same configuration.
>
> Regards,
> Jay Harkhani.
>
> 
> From: Jörn Franke 
> Sent: Thursday, May 21, 2020 7:05 PM
> To: solr-user@lucene.apache.org 
> Subject: Re: Query takes more time in Solr 8.5.1 compare to 6.1.0 version
>
> Did you create Solrconfig.xml for the collection from scratch after upgrading 
> and reindexing? Was it based on the latest template?
> If not then please try this. Maybe also you need to increase the 
> corresponding caches in the config.
>
> What happens if you reexecute the query?
>
> Are there other processes/containers running on the same VM?
>
> How much heap and how much total memory you have? You should only have a 
> minor fraction of the memory as heap and most of it „free“ (this means it is 
> used for file caches).
>
>
>
> > Am 21.05.2020 um 15:24 schrieb vishal patel :
> >
> > Any one is looking this issue?
> > I got same issue.
> >
> > Regards,
> > Vishal Patel
> >
> >
> >
> > 
> > From: jay harkhani 
> > Sent: Wednesday, May 20, 2020 7:39 PM
> > To: solr-user@lucene.apache.org 
> > Subject: Query takes more time in Solr 8.5.1 compare to 6.1.0 version
> >
> > Hello,
> >
> > Currently I upgrade Solr version from 6.1.0 to 8.5.1 and come across one 
> > issue. Query which have more ids (around 3000) and grouping is applied 
> > takes more time to execute. In Solr 6.1.0 it takes 677ms and in Solr 8.5.1 
> > it takes 26090ms. While take reading we have same solr schema and same no. 
> > of records in both solr version.
> >
> > Please refer below details for query, logs and thead dump (generate from 
> > Solr Admin while execute query).
> >
> > Query : 
> > https://drive.google.com/file/d/1bavCqwHfJxoKHFzdOEt-mSG8N0fCHE-w/view
> >
> > Logs and Thread dump stack trace
> > Solr 8.5.1 : 
> > https://drive.google.com/file/d/149IgaMdLomTjkngKHrwd80OSEa1eJbBF/view
> > Solr 6.1.0 : 
> > https://drive.google.com/file/d/13v1u__fM8nHfyvA0Mnj30IhdffW6xhwQ/view
> >
> > To analyse further more we found that if we remove grouping field or we 
> > reduce no. of ids from query it execute fast. Is anything change in 8.5.1 
> > version compare to 6.1.0 as in 6.1.0 even for large no. Ids along with 
> > grouping it works faster?
> >
> > Can someone please help to isolate this issue.
> >
> > Regards,
> > Jay Harkhani.

Re: Trouble starting Solr on Windows/Ubuntu

[Jan Høydahl]

You have a Core Dump which tells that the java process crash big time. Could be 
a permission issue between your windows file system and the WSL file system? 
Try do a chmod -R 777 solr-8.5.1 and then try again?

Jan Høydahl

> 22. mai 2020 kl. 23:32 skrev Stavros Macrakis :
> 
> I'm trying to follow the Solr Tutorial (
> https://lucene.apache.org/solr/guide/8_5/solr-tutorial.html#solr-tutorial).
> 
> Yesterday, "bin/solr start" worked fine -- I could see the status page on
> http://localhost:8993 . I even created a test config server/solr/test1
> through the Web interface.
> 
> Today, I'm getting an error message when I try to start Solr. This is from
> an Ubuntu top-level shell (I previously tried a shell buffer within Emacs
> under Ubuntu, which failed). I've rebooted Windows, and it still fails. See
> transcript and version info below.
> 
> What am I doing wrong? -- and is solr-user the right place to ask newbie
> questions like this?
> (None of the env variables mentioned in the error message are defined.)
> 
> transcript
> xxx:/mnt/c/solr-8.5.1$ bin/solr status -help
> 
> No Solr nodes are running.
> 
> xxx:/mnt/c/solr-8.5.1$ bin/solr start
> Waiting up to 180 seconds to see Solr running on port 8983 [|]  bin/solr:
> line 669:  8456 Aborted (core dumped) nohup "$JAVA"
> "${SOLR_START_OPTS[@]}" $SOLR_ADDL_ARGS -Dsolr.log.muteconsole
> "-XX:OnOutOfMemoryError=$SOLR_TIP/bin/oom_solr.sh $SOLR_PORT
> $SOLR_LOGS_DIR" -jar start.jar "${SOLR_JETTY_CONFIG[@]}"
> $SOLR_JETTY_ADDL_CONFIG > "$SOLR_LOGS_DIR/solr-$SOLR_PORT-console.log" 2>&1
> [/]  Still not seeing Solr listening on 8983 after 180 seconds!
> tail: cannot open '/mnt/c/solr-8.5.1/server/logs/solr.log' for reading: No
> such file or directory
> 
> xxx:/mnt/c/solr-8.5.1$ echo foo > /mnt/c/solr-8.5.1/server/logs/solr.log
> xxx:/mnt/c/solr-8.5.1$ cat /mnt/c/solr-8.5.1/server/logs/solr.log
> foo   <<< log file is writeable
> 
> versions 
> 
> xxx:/mnt/c/solr-8.5.1$ uname -a
> Linux DESKTOP-M6LDB7Q 4.4.0-18362-Microsoft #836-Microsoft Mon May 05
> 16:04:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
> xxx:/mnt/c/solr-8.5.1$ which java
> /usr/bin/java
> xxx:/mnt/c/solr-8.5.1$ java -version
> openjdk version "11.0.7" 2020-04-14
> OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04)
> OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04, mixed
> mode, sharing)
> xxx:/mnt/c/solr-8.5.1$ bin/solr -version
> 8.5.1

Sorting in other collection in Solr 8.5.1

[vishal patel]

Hi

I am upgrading Solr 8.5.1. I have created 2 shards and each has one replica.
I have created 2 collection one is form and second is actionscomment.forms 
related data are stored in form collection and actions of that forms are stored 
in actionscomment collection.
There are 10 lakh documents in form and 50 lakh documents in actionscomment 
collection.

form schema.xml






actionscomment schema.xml










We are showing form listing using form and actionscomment collection. We are 
showing only 250 records in form listing page. Our form listing columns are 
id,title,form created date and action names. id,title,form created date and 
action names come from form collection and action names come from 
actionscomment collection. We want to give the sorting functionality for all 
columns.It is easy to sort id, title and form created date because it is in 
same collection.

For action name sorting, I execute 2 query. First I execute query in 
actionscomment collection with sort field title and get the form_id list and 
using those form_ids I execute in form collection. But I do not get the proper 
sorting. Sometimes I got so many form ids and my second query length becomes 
larger.
How can I get data from form collection same as order of form id list came from 
actionscomment?

Regards,
Vishal Patel

Re: hl.preserveMulti in Unified highlighter?

[Anthony Groves]

Hi Walter,

I did something very similar to what David is suggesting when switching
from the PostingsHighlighter to the UnifiedHighlighter in Solr 7.

In order to include non-highlighted items (exact ordering) when using
preserveMulti, we used a custom PassageFormatter that ignored the start and
end offsets:
https://github.com/oreillymedia/ifpress-solr-plugin/blob/bf3b07c5be32fbcfa7b6fdfd439d511ef60dab68/src/main/java/com/ifactory/press/db/solr/highlight/HighlightFormatter.java#L35

I was actually surprised to see not much of a performance hit from
essentially removing the offset usage, but our highlighted fields aren't
extremely large :-)

Hope that helps!
Anthony

*Anthony Groves*  | Technical Lead, Search

O'Reilly Media, Inc.  | https://www.linkedin.com/in/anthonygroves/


On Fri, May 22, 2020 at 4:59 PM David Smiley 
wrote:

> Hi Walter,
>
> No, the UnifiedHighlighter does not behave as if this setting were true.
>
> The docs say:
>
> `hl.preserveMulti`::
> If `true`, multi-valued fields will return all values in the order they
> were saved in the index. If `false`, the default, only values that match
> the highlight request will be returned.
>
>
> The first sentence there is the essence of it.  Notice it's not conditional
> on wether there are highlights or not.  The UH won't return values lacking
> a highlight. Even hl.defaultSummary isn't triggered because *some* of the
> values have a highlight.
>
> As I look at the pertinent code right now, I imagine a solution would be to
> provide a custom PassageFormatter.  If we can assume for this use-case that
> you can use hl.bs.type=WHOLE as well, then a a simpler PassageFormatter
> could basically ignore the passage starts & ends and merely mark up the
> original content in entirety, which is a null concatenated sequence of all
> the values for this field for a document.
>
> ~ David
>
>
> On Fri, Mar 29, 2019 at 2:02 PM Walter Underwood 
> wrote:
>
> > We are testing 6.6.1.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> > > On Mar 29, 2019, at 11:02 AM, Walter Underwood 
> > wrote:
> > >
> > > In testing, hl.preserveMulti=true works with the unified highlighter.
> > But the documentation says that the parameter is only implemented in the
> > original highlighter.
> > >
> > > Is the documentation wrong? Can we trust this to keep working with
> > unified?
> > >
> > > wunder
> > > Walter Underwood
> > > wun...@wunderwood.org
> > > http://observer.wunderwood.org/  (my blog)
> > >
> > >> On Mar 26, 2019, at 12:08 PM, Walter Underwood  >
> > wrote:
> > >>
> > >> It looks like hl.preserveMulti is only implemented in the Original
> > highlighter. Has anyone looked at doing this for the Unified highlighter?
> > >>
> > >> We need to preserve order in the highlights for a multi-valued field.
> > >>
> > >> wunder
> > >> Walter Underwood
> > >> wun...@wunderwood.org 
> > >> http://observer.wunderwood.org/  (my blog)
> > >>
> > >
> >
> >
>

Re: hl.preserveMulti in Unified highlighter?

[Walter Underwood]

I’m a little amused that this thread has become active after almost two months 
of silence.

I think we just used the old highlighter. I don’t even remember now.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On May 23, 2020, at 9:14 AM, Anthony Groves  wrote:
> 
> Hi Walter,
> 
> I did something very similar to what David is suggesting when switching
> from the PostingsHighlighter to the UnifiedHighlighter in Solr 7.
> 
> In order to include non-highlighted items (exact ordering) when using
> preserveMulti, we used a custom PassageFormatter that ignored the start and
> end offsets:
> https://github.com/oreillymedia/ifpress-solr-plugin/blob/bf3b07c5be32fbcfa7b6fdfd439d511ef60dab68/src/main/java/com/ifactory/press/db/solr/highlight/HighlightFormatter.java#L35
> 
> I was actually surprised to see not much of a performance hit from
> essentially removing the offset usage, but our highlighted fields aren't
> extremely large :-)
> 
> Hope that helps!
> Anthony
> 
> *Anthony Groves*  | Technical Lead, Search
> 
> O'Reilly Media, Inc.  | https://www.linkedin.com/in/anthonygroves/
> 
> 
> On Fri, May 22, 2020 at 4:59 PM David Smiley 
> wrote:
> 
>> Hi Walter,
>> 
>> No, the UnifiedHighlighter does not behave as if this setting were true.
>> 
>> The docs say:
>> 
>> `hl.preserveMulti`::
>> If `true`, multi-valued fields will return all values in the order they
>> were saved in the index. If `false`, the default, only values that match
>> the highlight request will be returned.
>> 
>> 
>> The first sentence there is the essence of it.  Notice it's not conditional
>> on wether there are highlights or not.  The UH won't return values lacking
>> a highlight. Even hl.defaultSummary isn't triggered because *some* of the
>> values have a highlight.
>> 
>> As I look at the pertinent code right now, I imagine a solution would be to
>> provide a custom PassageFormatter.  If we can assume for this use-case that
>> you can use hl.bs.type=WHOLE as well, then a a simpler PassageFormatter
>> could basically ignore the passage starts & ends and merely mark up the
>> original content in entirety, which is a null concatenated sequence of all
>> the values for this field for a document.
>> 
>> ~ David
>> 
>> 
>> On Fri, Mar 29, 2019 at 2:02 PM Walter Underwood 
>> wrote:
>> 
>>> We are testing 6.6.1.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
 On Mar 29, 2019, at 11:02 AM, Walter Underwood 
>>> wrote:
 
 In testing, hl.preserveMulti=true works with the unified highlighter.
>>> But the documentation says that the parameter is only implemented in the
>>> original highlighter.
 
 Is the documentation wrong? Can we trust this to keep working with
>>> unified?
 
 wunder
 Walter Underwood
 wun...@wunderwood.org
 http://observer.wunderwood.org/  (my blog)
 
> On Mar 26, 2019, at 12:08 PM, Walter Underwood >> 
>>> wrote:
> 
> It looks like hl.preserveMulti is only implemented in the Original
>>> highlighter. Has anyone looked at doing this for the Unified highlighter?
> 
> We need to preserve order in the highlights for a multi-valued field.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org 
> http://observer.wunderwood.org/  (my blog)
> 
 
>>> 
>>> 
>> 

Re: Trouble starting Solr on Windows/Ubuntu

[Stavros Macrakis]

Jan,

Thanks for your suggestion!

Unfortunately, that doesn't fix the problem in Ubuntu under Windows 10.

Fortunately, I figured out how to start Solr on Windows 10 itself. It turns
out that solr.cmd depends on the Windows functions 'find' and 'timeout',
which were being shadowed by the Cygwin (Gnu) utilities of the same names.
It also turned out that I had an old 32-bit JRE in my Windows config. I
deleted that and updated the 64-bit JRE.

Thanks again,

 -s

On Sat, May 23, 2020 at 5:27 AM Jan Høydahl  wrote:

> You have a Core Dump which tells that the java process crash big time.
> Could be a permission issue between your windows file system and the WSL
> file system? Try do a chmod -R 777 solr-8.5.1 and then try again?
>
> Jan Høydahl
>
> > 22. mai 2020 kl. 23:32 skrev Stavros Macrakis :
> >
> > I'm trying to follow the Solr Tutorial (
> >
> https://lucene.apache.org/solr/guide/8_5/solr-tutorial.html#solr-tutorial
> ).
> >
> > Yesterday, "bin/solr start" worked fine -- I could see the status page on
> > http://localhost:8993 . I even created a test config server/solr/test1
> > through the Web interface.
> >
> > Today, I'm getting an error message when I try to start Solr. This is
> from
> > an Ubuntu top-level shell (I previously tried a shell buffer within Emacs
> > under Ubuntu, which failed). I've rebooted Windows, and it still fails.
> See
> > transcript and version info below.
> >
> > What am I doing wrong? -- and is solr-user the right place to ask newbie
> > questions like this?
> > (None of the env variables mentioned in the error message are defined.)
> >
> > transcript
> > xxx:/mnt/c/solr-8.5.1$ bin/solr status -help
> >
> > No Solr nodes are running.
> >
> > xxx:/mnt/c/solr-8.5.1$ bin/solr start
> > Waiting up to 180 seconds to see Solr running on port 8983 [|]  bin/solr:
> > line 669:  8456 Aborted (core dumped) nohup "$JAVA"
> > "${SOLR_START_OPTS[@]}" $SOLR_ADDL_ARGS -Dsolr.log.muteconsole
> > "-XX:OnOutOfMemoryError=$SOLR_TIP/bin/oom_solr.sh $SOLR_PORT
> > $SOLR_LOGS_DIR" -jar start.jar "${SOLR_JETTY_CONFIG[@]}"
> > $SOLR_JETTY_ADDL_CONFIG > "$SOLR_LOGS_DIR/solr-$SOLR_PORT-console.log"
> 2>&1
> > [/]  Still not seeing Solr listening on 8983 after 180 seconds!
> > tail: cannot open '/mnt/c/solr-8.5.1/server/logs/solr.log' for reading:
> No
> > such file or directory
> >
> > xxx:/mnt/c/solr-8.5.1$ echo foo > /mnt/c/solr-8.5.1/server/logs/solr.log
> > xxx:/mnt/c/solr-8.5.1$ cat /mnt/c/solr-8.5.1/server/logs/solr.log
> > foo   <<< log file is writeable
> >
> > versions 
> >
> > xxx:/mnt/c/solr-8.5.1$ uname -a
> > Linux DESKTOP-M6LDB7Q 4.4.0-18362-Microsoft #836-Microsoft Mon May 05
> > 16:04:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
> > xxx:/mnt/c/solr-8.5.1$ which java
> > /usr/bin/java
> > xxx:/mnt/c/solr-8.5.1$ java -version
> > openjdk version "11.0.7" 2020-04-14
> > OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04)
> > OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04,
> mixed
> > mode, sharing)
> > xxx:/mnt/c/solr-8.5.1$ bin/solr -version
> > 8.5.1
>

Re: Query takes more time in Solr 8.5.1 compare to 6.1.0 version

[Mikhail Khludnev]

Unfortunately {!terms} doesn't let one ^boost terms.

On Sat, May 23, 2020 at 10:13 AM vishal patel 
wrote:

> Hi Jason
>
> Thanks for reply.
>
> I have checked jay's query using "terms" query parser and it is really
> helpful to us. After execute using "terms" query parser it will come within
> a 500 milliseconds even though grouping is applied.
> Jay's Query :
> https://drive.google.com/file/d/1bavCqwHfJxoKHFzdOEt-mSG8N0fCHE-w/view
>
> Actually I want to apply same things in my query but my field "msg_id" is
> applied boost.group is also used in my query.
> I am also upgrading Solr 8.5.1.
>
>
> MY query is :
> https://drive.google.com/file/d/1Op_Ja292Bcnv0Ijxw6VdAxvGlfsdczmS/view
>
> I got 30 seconds for above query. How can I use the "terms" query parser
> in my query?
>
> Regards,
> Vishal Patel
> 
> From: Jason Gerlowski 
> Sent: Friday, May 22, 2020 2:59 AM
> To: solr-user@lucene.apache.org 
> Subject: Re: Query takes more time in Solr 8.5.1 compare to 6.1.0 version
>
> Hi Jay,
>
> I can't speak to why you're seeing a performance change between 6.x
> and 8.x.  What I can suggest though is an alternative way of
> formulating the query: you might get different performance if you run
> your query using Solr's "terms" query parser:
>
> https://lucene.apache.org/solr/guide/8_5/other-parsers.html#terms-query-parser
>  It's not guaranteed to help, but there's a chance it'll work for you.
> And knowing whether or not it helps might point others here towards
> the cause of your slowdown.
>
> Even if "terms" performs better for you, it's probably worth
> understanding what's going on here of course.
>
> Are all other queries running comparably?
>
> Jason
>
> On Thu, May 21, 2020 at 10:25 AM jay harkhani 
> wrote:
> >
> > Hello,
> >
> > Please refer below details.
> >
> > >Did you create Solrconfig.xml for the collection from scratch after
> upgrading and reindexing?
> > Yes, We have created collection from scratch and also re-indexing.
> >
> > >Was it based on the latest template?
> > Yes, It was as per latest template.
> >
> > >What happens if you reexecute the query?
> > Not more visible difference. Minor change in milliseconds.
> >
> > >Are there other processes/containers running on the same VM?
> > No
> >
> > >How much heap and how much total memory you have?
> > My heap and total memory are same as Solr 6.1.0. heap memory 5 gb and
> total memory 25gb. As per me there is no issue related to memory.
> >
> > >Maybe also you need to increase the corresponding caches in the config.
> > We are not using cache in both version.
> >
> > Both version have same configuration.
> >
> > Regards,
> > Jay Harkhani.
> >
> > 
> > From: Jörn Franke 
> > Sent: Thursday, May 21, 2020 7:05 PM
> > To: solr-user@lucene.apache.org 
> > Subject: Re: Query takes more time in Solr 8.5.1 compare to 6.1.0 version
> >
> > Did you create Solrconfig.xml for the collection from scratch after
> upgrading and reindexing? Was it based on the latest template?
> > If not then please try this. Maybe also you need to increase the
> corresponding caches in the config.
> >
> > What happens if you reexecute the query?
> >
> > Are there other processes/containers running on the same VM?
> >
> > How much heap and how much total memory you have? You should only have a
> minor fraction of the memory as heap and most of it „free“ (this means it
> is used for file caches).
> >
> >
> >
> > > Am 21.05.2020 um 15:24 schrieb vishal patel <
> vishalpatel200...@outlook.com>:
> > >
> > > Any one is looking this issue?
> > > I got same issue.
> > >
> > > Regards,
> > > Vishal Patel
> > >
> > >
> > >
> > > 
> > > From: jay harkhani 
> > > Sent: Wednesday, May 20, 2020 7:39 PM
> > > To: solr-user@lucene.apache.org 
> > > Subject: Query takes more time in Solr 8.5.1 compare to 6.1.0 version
> > >
> > > Hello,
> > >
> > > Currently I upgrade Solr version from 6.1.0 to 8.5.1 and come across
> one issue. Query which have more ids (around 3000) and grouping is applied
> takes more time to execute. In Solr 6.1.0 it takes 677ms and in Solr 8.5.1
> it takes 26090ms. While take reading we have same solr schema and same no.
> of records in both solr version.
> > >
> > > Please refer below details for query, logs and thead dump (generate
> from Solr Admin while execute query).
> > >
> > > Query :
> https://drive.google.com/file/d/1bavCqwHfJxoKHFzdOEt-mSG8N0fCHE-w/view
> > >
> > > Logs and Thread dump stack trace
> > > Solr 8.5.1 :
> https://drive.google.com/file/d/149IgaMdLomTjkngKHrwd80OSEa1eJbBF/view
> > > Solr 6.1.0 :
> https://drive.google.com/file/d/13v1u__fM8nHfyvA0Mnj30IhdffW6xhwQ/view
> > >
> > > To analyse further more we found that if we remove grouping field or
> we reduce no. of ids from query it execute fast. Is anything change in
> 8.5.1 version compare to 6.1.0 as in 6.1.0 even for large no. Ids along
> with grouping it works faster?
> > >
> > > Can someone please

Re: Terms faceting and EnumField

[Poornima Ponnuswamy]

I see a defect created for that. But I see its resolved. Am I doing anything 
wrong?

https://issues.apache.org/jira/browse/SOLR-7496


QUERY
http://localhost:8983/solr/activity01us/select?
&json.facet={ServiceRequestTypeCode:{type:terms, field:ServiceRequestTypeCode, 
limit:10}}
&facet=on&indent=on&wt=json&q=*&rows=0

Below is the field and field type that is defined in solr schema.



Below is the configuration for the enum
   

  servicerequestcorrective
  servicerequestplanned
  servicerequestinstallationandupgrade
  servicerequestrecall
  servicerequestother
  servicerequestinquiry
  servicerequestproactive
  servicerequestsystemupdate
  servicerequesticenteradmin
  servicerequestonwatch
  servicerequestfmi
  servicerequestapplication
   

 I am getting error
"Expected numeric field type 
:ServiceRequestTypeCode{type=ServiceRequestTypeCode,properties=indexed,stored,omitNorms,omitTermFreqAndPositions}"




From: Poornima Ponnuswamy 
Sent: Friday, May 22, 2020 7:01 AM
To: solr-user@lucene.apache.org 
Subject: Terms faceting and EnumField

Hello,

We have solr 6.6 version.
Below is the field and field type that is defined in solr schema.



Below is the configuration for the enum
   

  servicerequestcorrective
  servicerequestplanned
  servicerequestinstallationandupgrade
  servicerequestrecall
  servicerequestother
  servicerequestinquiry
  servicerequestproactive
  servicerequestsystemupdate
  servicerequesticenteradmin
  servicerequestonwatch
  servicerequestfmi
  servicerequestapplication
   

When I try to invoke using the below request,

http://localhost:8983/solr/activity01us/select?&json.facet={ServiceRequestTypeCode:{type:terms,
 field:ServiceRequestTypeCode, limit:10}}&facet=on&indent=on&wt=json&q=*

 I am getting error
"Expected numeric field type 
:ServiceRequestTypeCode{type=ServiceRequestTypeCode,properties=indexed,stored,omitNorms,omitTermFreqAndPositions}"

But when I try to do as below it works fine.


http://localhost:8983/solr/activity01us/select?facet.field=ServiceRequestTypeCode&facet=on&indent=on&q=*:*&wt=json

I would like to use json facet as it would help me in subfaceting.

Any help would be appreciated

Solr 8.5.1 startup error - lengthTag=109, too big.

[Zheng Lin Edwin Yeo]

Hi,

I'm trying to upgrade from Solr 8.2.1 to Solr 8.5.1, with Solr SSL
Authentication and Authorization.

However, I get the following error when I enable SSL. The Solr itself can
start up if there is no SSL.  The main error that I see is this

  java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.

What could be the reason that causes this?


INFO  - 2020-05-24 10:38:20.080;
org.apache.solr.util.configuration.SSLConfigurations; Setting
javax.net.ssl.keyStorePassword
INFO  - 2020-05-24 10:38:20.081;
org.apache.solr.util.configuration.SSLConfigurations; Setting
javax.net.ssl.trustStorePassword
Waiting up to 120 to see Solr running on port 8983
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
at org.eclipse.jetty.start.Main.start(Main.java:491)
at org.eclipse.jetty.start.Main.main(Main.java:77)d
Caused by: java.security.PrivilegedActionException: java.io.IOException:
DerInputStream.getLength(): lengthTag=109, too big.
at java.security.AccessController.doPrivileged(Native Method)
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
... 7 more
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109,
too big.
at sun.security.util.DerInputStream.getLength(Unknown Source)
at sun.security.util.DerValue.init(Unknown Source)
at sun.security.util.DerValue.(Unknown Source)
at sun.security.util.DerValue.(Unknown Source)
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source)
at java.security.KeyStore.load(Unknown Source)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1188)
at
org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:323)
at
org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at
org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
... 9 more
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
at org.eclipse.jetty.start.Main.start(Main.java:491)
at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.io.IOException:
DerInputStream.getLength(): lengthTag=109, too big.
at java.security.AccessController.doPrivileged(Native Method)
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
... 7 more
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109,
too big.
at sun.security.util.DerInputStream.getLength(Unknown Source)
at sun.security.util.DerValue.init(Unknown Source)
at sun.security.util.DerValue.(Unknown Source)
at sun.security.util.DerValue.(Unknown Source)
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source)
at java.security.KeyStore.load(Unknown Source)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org

Re: Sorting in other collection in Solr 8.5.1

[Zheng Lin Edwin Yeo]

Hi Vishal,

Maybe you can post your query that you are using which has the issue too.

Regards,
Edwin

On Sat, 23 May 2020 at 22:27, vishal patel 
wrote:

> Hi
>
> I am upgrading Solr 8.5.1. I have created 2 shards and each has one
> replica.
> I have created 2 collection one is form and second is actionscomment.forms
> related data are stored in form collection and actions of that forms are
> stored in actionscomment collection.
> There are 10 lakh documents in form and 50 lakh documents in
> actionscomment collection.
>
> form schema.xml
>  required="true" multiValued="false" docValues="true"/>
>  docValues="true"/>
>  docValues="true"/>
>  omitNorms="true"/>
>  docValues="true"/>
>
> actionscomment schema.xml
>  required="true" multiValued="false" docValues="true"/>
>  docValues="true"/>
>  docValues="true"/>
> 
>  omitNorms="true"/>
>  docValues="true"/>
>  stored="true" docValues="true"/>
>
>
>
> We are showing form listing using form and actionscomment collection. We
> are showing only 250 records in form listing page. Our form listing columns
> are id,title,form created date and action names. id,title,form created date
> and action names come from form collection and action names come from
> actionscomment collection. We want to give the sorting functionality for
> all columns.It is easy to sort id, title and form created date because it
> is in same collection.
>
> For action name sorting, I execute 2 query. First I execute query in
> actionscomment collection with sort field title and get the form_id list
> and using those form_ids I execute in form collection. But I do not get the
> proper sorting. Sometimes I got so many form ids and my second query length
> becomes larger.
> How can I get data from form collection same as order of form id list came
> from actionscomment?
>
> Regards,
> Vishal Patel
>

Re: hl.preserveMulti in Unified highlighter?

[David Smiley]

Better late than never?  I added some new mail filters to bring topics of
interest to my attention.

Any way; this seems like an important use-case.

Anthony:  You'd probably benefit from also setting hl.bs.type=WHOLE since
clearly you want whole values (no snippets/fragments of values).  If I get
around to implementing hl.preserveMulti for the UH, i'll have it make this
assumption likewise.

~ David


On Sat, May 23, 2020 at 1:48 PM Walter Underwood 
wrote:

> I’m a little amused that this thread has become active after almost two
> months of silence.
>
> I think we just used the old highlighter. I don’t even remember now.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On May 23, 2020, at 9:14 AM, Anthony Groves  wrote:
> >
> > Hi Walter,
> >
> > I did something very similar to what David is suggesting when switching
> > from the PostingsHighlighter to the UnifiedHighlighter in Solr 7.
> >
> > In order to include non-highlighted items (exact ordering) when using
> > preserveMulti, we used a custom PassageFormatter that ignored the start
> and
> > end offsets:
> >
> https://github.com/oreillymedia/ifpress-solr-plugin/blob/bf3b07c5be32fbcfa7b6fdfd439d511ef60dab68/src/main/java/com/ifactory/press/db/solr/highlight/HighlightFormatter.java#L35
> >
> > I was actually surprised to see not much of a performance hit from
> > essentially removing the offset usage, but our highlighted fields aren't
> > extremely large :-)
> >
> > Hope that helps!
> > Anthony
> >
> > *Anthony Groves*  | Technical Lead, Search
> >
> > O'Reilly Media, Inc.  | https://www.linkedin.com/in/anthonygroves/
> >
> >
> > On Fri, May 22, 2020 at 4:59 PM David Smiley 
> > wrote:
> >
> >> Hi Walter,
> >>
> >> No, the UnifiedHighlighter does not behave as if this setting were true.
> >>
> >> The docs say:
> >>
> >> `hl.preserveMulti`::
> >> If `true`, multi-valued fields will return all values in the order they
> >> were saved in the index. If `false`, the default, only values that match
> >> the highlight request will be returned.
> >>
> >>
> >> The first sentence there is the essence of it.  Notice it's not
> conditional
> >> on wether there are highlights or not.  The UH won't return values
> lacking
> >> a highlight. Even hl.defaultSummary isn't triggered because *some* of
> the
> >> values have a highlight.
> >>
> >> As I look at the pertinent code right now, I imagine a solution would
> be to
> >> provide a custom PassageFormatter.  If we can assume for this use-case
> that
> >> you can use hl.bs.type=WHOLE as well, then a a simpler PassageFormatter
> >> could basically ignore the passage starts & ends and merely mark up the
> >> original content in entirety, which is a null concatenated sequence of
> all
> >> the values for this field for a document.
> >>
> >> ~ David
> >>
> >>
> >> On Fri, Mar 29, 2019 at 2:02 PM Walter Underwood  >
> >> wrote:
> >>
> >>> We are testing 6.6.1.
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wun...@wunderwood.org
> >>> http://observer.wunderwood.org/  (my blog)
> >>>
>  On Mar 29, 2019, at 11:02 AM, Walter Underwood  >
> >>> wrote:
> 
>  In testing, hl.preserveMulti=true works with the unified highlighter.
> >>> But the documentation says that the parameter is only implemented in
> the
> >>> original highlighter.
> 
>  Is the documentation wrong? Can we trust this to keep working with
> >>> unified?
> 
>  wunder
>  Walter Underwood
>  wun...@wunderwood.org
>  http://observer.wunderwood.org/  (my blog)
> 
> > On Mar 26, 2019, at 12:08 PM, Walter Underwood <
> wun...@wunderwood.org
> >>>
> >>> wrote:
> >
> > It looks like hl.preserveMulti is only implemented in the Original
> >>> highlighter. Has anyone looked at doing this for the Unified
> highlighter?
> >
> > We need to preserve order in the highlights for a multi-valued field.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org 
> > http://observer.wunderwood.org/  (my blog)
> >
> 
> >>>
> >>>
> >>
>
>