Re: Using QT param with /select

[Mikhail Khludnev]

Hello, Atita.

My question here is that on Solr 6.2.6 to enable using 'qt' param I need to
> do handleSelect=false

Can you elaborate on that? What exactly happens?  Also, please clarify
whether you use SolrCloud or standalone?


On Mon, Mar 2, 2020 at 7:37 PM Atita Arora  wrote:

> Hi,
>
> I am working on improving the search app which is using 'qt' param heavily
> to redirect requests to different handlers based on the parameters as
> provided by the user.
>
> Also for A B testing of different configurations, we have used qt param to
> send request to different handlers.
> My question here is that on Solr 6.2.6 to enable using 'qt' param I need to
> do handleSelect=false but it is the default request handler on solr
> administration UI and used as the default endpoint in all the integration
> tests.
>
> It may sound weird but is there a way I can retain both the
> functionalities?
> No code changes to integration test code and making qt param work again.
>
> Big thanks for any pointers !!
>
> Sincerely,
> Atita
>


-- 
Sincerely yours
Mikhail Khludnev

Core keeps getting closed

[wezy]

Hello,

where having a problem ever since our upgrade to SOLR 7.5

First we have this error 
 

directly followed by 

 

the last error just keeps repeating.

Sometimes we go days without the error, sometimes it's every 2 minutes.
After restarting SOLR it works again, just untill these error's show up
again.

This is the schema of the core




  id
  
  
  
  
  
  
  
  
  

  
  
  
  
  
  
  
  
  
  
  


Below is the log
2020-03-10 08:29:13.502 INFO  (qtp1571967156-17) [   ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={indexInfo=false&wt=json&_=1583828953435} status=0 QTime=1
2020-03-10 08:29:13.507 INFO  (qtp1571967156-17) [   ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/logging
params={wt=json&_=1583828953488&since=0} status=0 QTime=0
2020-03-10 08:29:13.604 INFO  (qtp1571967156-19) [   ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={wt=json&_=1583828953435}
status=0 QTime=100
2020-03-10 08:29:14.164 INFO  (indexFetcher-44-thread-1) [   ]
o.a.s.h.IndexFetcher Master's generation: 9
2020-03-10 08:29:14.164 INFO  (indexFetcher-44-thread-1) [   ]
o.a.s.h.IndexFetcher Master's version: 1404994976024
2020-03-10 08:29:14.164 INFO  (indexFetcher-44-thread-1) [   ]
o.a.s.h.IndexFetcher Slave's generation: 9
2020-03-10 08:29:14.164 INFO  (indexFetcher-44-thread-1) [   ]
o.a.s.h.IndexFetcher Slave's version: 1404994976024
2020-03-10 08:29:14.164 INFO  (indexFetcher-44-thread-1) [   ]
o.a.s.h.IndexFetcher Slave in sync with master.
2020-03-10 08:29:14.351 INFO  (indexFetcher-63-thread-1) [   ]
o.a.s.h.IndexFetcher Master's generation: 62163
2020-03-10 08:29:14.351 INFO  (indexFetcher-63-thread-1) [   ]
o.a.s.h.IndexFetcher Master's version: 1575628893292
2020-03-10 08:29:14.351 INFO  (indexFetcher-63-thread-1) [   ]
o.a.s.h.IndexFetcher Slave's generation: 62162
2020-03-10 08:29:14.351 INFO  (indexFetcher-63-thread-1) [   ]
o.a.s.h.IndexFetcher Slave's version: 1575628893292
2020-03-10 08:29:14.351 INFO  (indexFetcher-63-thread-1) [   ]
o.a.s.h.IndexFetcher Slave in sync with master.
2020-03-10 08:29:14.575 INFO  (qtp1571967156-17) [  
x:belgicapress_titles_1] o.a.s.c.SolrCore [belgicapress_titles_1]  CLOSING
SolrCore org.apache.solr.core.SolrCore@73ba84ca
2020-03-10 08:29:14.575 INFO  (qtp1571967156-17) [  
x:belgicapress_titles_1] o.a.s.m.SolrMetricManager Closing metric reporters
for registry=solr.core.belgicapress_titles_1, tag=73ba84ca
2020-03-10 08:29:14.576 ERROR (qtp1571967156-17) [  
x:belgicapress_titles_1] o.a.s.c.SolrCore
java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory(MMapDirectory@/solr/periodicals/belgicapress_titles_1/data/snapshot_metadata
lockFactory=org.apache.lucene.store.NativeFSLockFactory@11145a3e;
maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
at
org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1632)
at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:568)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerW

Solr range search queries

[Niharika]

hello, 

I have  two field declared in schema xml as 
 
*

*

I want to generate a query to find all the result in specific range for
longitude and latitude

My query looks like 

*latitude:[47.010225655683485 TO 52.40241887397332] AND
longitude:[-2.021484375004 TO 14.63378906252]*

the problem here is:- i am not getting all the result i except, can anyone
suggest me what I can do here and why it is wrong?

PS:- I have already tried rounding of the decimals, I cannot change the type
from string in schema.xml. 

Thanks & Regards
Niharika 




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

How to query with FL field lower than some value?

[GTHell]

I use score:dist(2, v1, v2  , 0.8, ) in fl to calculate the distance.
It gave me the result I want but I only want the result score that is below
0.5.

How would I achieve that?



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Using QT param with /select

[Atita Arora]

Hi,
Thanks for looping back in.
We use Master-Slave!
I resolved this one with handleSelect=true , removing /select handler from
config and created another requesthandler (with a different name - other
than select) and marking that as default="true".
Now all the requests with qt param are handled by this request handler
while the one specifying qt goes to the designated handlers.
The application I am working on has different request handlers - like for
the different types of suggestions, spellchecking, default search request
etc and the code leverage solrj with qt param to dispatch requests to
different handlers.


I hope that was the right way to go about it.


On Tue, Mar 10, 2020 at 8:26 AM Mikhail Khludnev  wrote:

> Hello, Atita.
>
> My question here is that on Solr 6.2.6 to enable using 'qt' param I need to
> > do handleSelect=false
>
> Can you elaborate on that? What exactly happens?  Also, please clarify
> whether you use SolrCloud or standalone?
>
>
> On Mon, Mar 2, 2020 at 7:37 PM Atita Arora  wrote:
>
> > Hi,
> >
> > I am working on improving the search app which is using 'qt' param
> heavily
> > to redirect requests to different handlers based on the parameters as
> > provided by the user.
> >
> > Also for A B testing of different configurations, we have used qt param
> to
> > send request to different handlers.
> > My question here is that on Solr 6.2.6 to enable using 'qt' param I need
> to
> > do handleSelect=false but it is the default request handler on solr
> > administration UI and used as the default endpoint in all the integration
> > tests.
> >
> > It may sound weird but is there a way I can retain both the
> > functionalities?
> > No code changes to integration test code and making qt param work again.
> >
> > Big thanks for any pointers !!
> >
> > Sincerely,
> > Atita
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: How to query with FL field lower than some value?

[Munendra S N]

If you are looking to filter out all those products with greater than or
equal to 0.5, you could use
https://lucene.apache.org/solr/guide/8_4/other-parsers.html#function-range-query-parser
Function range query with the upper limit being 0.5 which could be added as
fq

Regards,
Munendra S N



On Tue, Mar 10, 2020 at 3:10 PM GTHell  wrote:

> I use score:dist(2, v1, v2  , 0.8, ) in fl to calculate the
> distance.
> It gave me the result I want but I only want the result score that is below
> 0.5.
>
> How would I achieve that?
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Atomic Update and Optimization and segments

[Kayak28]

Hello, Community:

Currently, my index grows up to almost 1T, and I would like to minimize my
index.

I know I have a few fields that are not used or rarely used, so I want to
delete them.
I have tried to delete these fields by the atomic update, sending the
following JSON for example.
{
"id":"1",
"text":{"set": null }
}
As a result, it generated a new segment, so segment count increased +1,
index size became bigger, and mac doc is increased +1.
I have expected this result, but my goal is to minimize my index, so I sent
an expungeDeleted request and optimize request, expecting to reduce the
index size and segment count.
But, the segment did not reduce, the index size did not change, and max doc
did not change.

As of Solr 8.4.1, is there any way to minimize segment count, index size
and max doc after atomic-updating?

Sincerely,
Kaya Ota

Re: Atomic Update and Optimization and segments

[Jörn Franke]

How do you do the atomic updates? I discovered a bug when doing them via DIH or 
Scriptupdateprocessor (only this one! The atomic one is fine) that leads to 
infinite index growth when doing atomic updates 

> Am 10.03.2020 um 13:28 schrieb Kayak28 :
> 
> Hello, Community:
> 
> Currently, my index grows up to almost 1T, and I would like to minimize my
> index.
> 
> I know I have a few fields that are not used or rarely used, so I want to
> delete them.
> I have tried to delete these fields by the atomic update, sending the
> following JSON for example.
> {
> "id":"1",
> "text":{"set": null }
> }
> As a result, it generated a new segment, so segment count increased +1,
> index size became bigger, and mac doc is increased +1.
> I have expected this result, but my goal is to minimize my index, so I sent
> an expungeDeleted request and optimize request, expecting to reduce the
> index size and segment count.
> But, the segment did not reduce, the index size did not change, and max doc
> did not change.
> 
> As of Solr 8.4.1, is there any way to minimize segment count, index size
> and max doc after atomic-updating?
> 
> Sincerely,
> Kaya Ota

Solr Search

[Suresh]

Hi Team,

 

Greetings of the day.

 

I would like to subscribe for knowledge and configuration about the Solr
search for better user experience.

 

Kindly guide me to further proceed.

 

 

Regards,



 

Re: How to query with FL field lower than some value?

[GTHell]

But the fq can't use the alias field call score. Anyway, is the performance
the same if I just sort the dist() function and get only 3 results? 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: How to query with FL field lower than some value?

[Munendra S N]

fq={!frange u=0.5 incu=false}dist(2, v1, v2  , 0.8, )

This will return only products where distance is less than 0.5(excluded).
Do you also have requirements to display the distance only then you would
field aliasing. As this truncates search results and default sorting is
done, this would be more performant

Regards,
Munendra S N



On Tue, Mar 10, 2020 at 7:51 PM GTHell  wrote:

> But the fq can't use the alias field call score. Anyway, is the performance
> the same if I just sort the dist() function and get only 3 results?
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Solr 8.2.0 - Schema issue

[Joe Obernberger]

Erick - tried this, had to run it async, but it's been running for over 
24 hours on one collection with:


{
  "responseHeader":{
"status":0,
"QTime":18326},
  "status":{
"state":"submitted",
"msg":"found [5] in submitted tasks"}}

I don't see anything in the logs.

-Joe

On 3/6/2020 1:43 PM, Joe Obernberger wrote:
Thank you Erick - I have no record of that, but will absolutely give 
the API RELOAD a shot!  Thank you!


-Joe

On 3/6/2020 10:26 AM, Erick Erickson wrote:
Didn’t we talk about reloading the collections that share the schema 
after the schema change via the collections API RELOAD command?


Best,
Erick

On Mar 6, 2020, at 05:34, Joe Obernberger 
 wrote:


Hi All - any ideas on this?  Anything I can try?

Thank you!

-Joe


On 2/26/2020 9:01 AM, Joe Obernberger wrote:
Hi All - I have several solr collections all with the same schema.  
If I add a field to the schema and index it into the collection on 
which I added the field, it works fine. However, if I try to add a 
document to a different solr collection that contains the new field 
(and is using the same schema), I get an error that the field 
doesn't exist.


If I restart the cluster, this problem goes away and I can add a 
document with the new field to any solr collection that has the 
schema.  Any work-arounds that don't involve a restart?


Thank you!

-Joe Obernberger

Re: Overseer & Backups - Questions

[Aroop Ganguly]

May we know how you are invoking backups ?

> On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh  wrote:
> 
> Hi everybody,
> 
> Quick Specs:
> - Solr 7.4 Solr Cloud
> - 30gb index on 8 shards Tlog/Pull
> 
> We run daily backups on our 30gb index and noticed that the overseer does
> not process other jobs on it's task list while the backup is being taken.
> They remain on the pending list (in ZK). Is this expected?
> 
> Also I was wondering if there was a safe way to cancel a currently running
> task or deleing pending tasks?
> 
> Regards,
> 
> Ash
> 
> -- 
> **
> ** Empowering the world to design
> Also, we're 
> hiring. Apply here! 
> 
>   
>    
>     
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: Overseer & Backups - Questions

[Ashwin Ramesh]

We use the collection API to invoke backups. The tasks we noticed that
stalled are ADDREPLICA. As expected when the backup completed a few hours
ago, the task then got completed. Is there some concurrency setting with
these tasks? Or is a backup a blocking task? We noticed that the index was
still being flushed to segments though.

Regards,

Ash

On Wed, Mar 11, 2020 at 3:18 AM Aroop Ganguly
 wrote:

> May we know how you are invoking backups ?
>
> > On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh 
> wrote:
> >
> > Hi everybody,
> >
> > Quick Specs:
> > - Solr 7.4 Solr Cloud
> > - 30gb index on 8 shards Tlog/Pull
> >
> > We run daily backups on our 30gb index and noticed that the overseer does
> > not process other jobs on it's task list while the backup is being taken.
> > They remain on the pending list (in ZK). Is this expected?
> >
> > Also I was wondering if there was a safe way to cancel a currently
> running
> > task or deleing pending tasks?
> >
> > Regards,
> >
> > Ash
> >
> > --
> > **
> > ** Empowering the world to design
> > Also, we're
> > hiring. Apply here! 
> >
> >  
> >  
> >   
> > 
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>

-- 
**
** Empowering the world to design
Also, we're 
hiring. Apply here! 
 
  
   
    

Re: Overseer & Backups - Questions

[Aroop Ganguly]

Backups on hdfs ?
These should not be blocking if invoked asynchronously, are u doing them async 
by passing the async flag?

> On Mar 10, 2020, at 3:19 PM, Ashwin Ramesh  wrote:
> 
> We use the collection API to invoke backups. The tasks we noticed that
> stalled are ADDREPLICA. As expected when the backup completed a few hours
> ago, the task then got completed. Is there some concurrency setting with
> these tasks? Or is a backup a blocking task? We noticed that the index was
> still being flushed to segments though.
> 
> Regards,
> 
> Ash
> 
> On Wed, Mar 11, 2020 at 3:18 AM Aroop Ganguly
>  wrote:
> 
>> May we know how you are invoking backups ?
>> 
>>> On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh 
>> wrote:
>>> 
>>> Hi everybody,
>>> 
>>> Quick Specs:
>>> - Solr 7.4 Solr Cloud
>>> - 30gb index on 8 shards Tlog/Pull
>>> 
>>> We run daily backups on our 30gb index and noticed that the overseer does
>>> not process other jobs on it's task list while the backup is being taken.
>>> They remain on the pending list (in ZK). Is this expected?
>>> 
>>> Also I was wondering if there was a safe way to cancel a currently
>> running
>>> task or deleing pending tasks?
>>> 
>>> Regards,
>>> 
>>> Ash
>>> 
>>> --
>>> **
>>> ** Empowering the world to design
>>> Also, we're
>>> hiring. Apply here! 
>>> 
>>>  
>>>  
>>>   
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> -- 
> **
> ** Empowering the world to design
> Also, we're 
> hiring. Apply here! 
> 
>   
>    
>     
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: Overseer & Backups - Questions

[Ashwin Ramesh]

Hey Aroop,

Yes we sent ASYNC=

Backups are taken to an EFS drive (AWS's managed NFS)

I also thought it was async and Solr can process multiple tasks at once.
But the ZK state definitely showed that only the backup task was in
progress while all the other tasks were queued up.

Regards,

Ash

On Wed, Mar 11, 2020 at 9:21 AM Aroop Ganguly
 wrote:

> Backups on hdfs ?
> These should not be blocking if invoked asynchronously, are u doing them
> async by passing the async flag?
>
> > On Mar 10, 2020, at 3:19 PM, Ashwin Ramesh 
> wrote:
> >
> > We use the collection API to invoke backups. The tasks we noticed that
> > stalled are ADDREPLICA. As expected when the backup completed a few hours
> > ago, the task then got completed. Is there some concurrency setting with
> > these tasks? Or is a backup a blocking task? We noticed that the index
> was
> > still being flushed to segments though.
> >
> > Regards,
> >
> > Ash
> >
> > On Wed, Mar 11, 2020 at 3:18 AM Aroop Ganguly
> >  wrote:
> >
> >> May we know how you are invoking backups ?
> >>
> >>> On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh 
> >> wrote:
> >>>
> >>> Hi everybody,
> >>>
> >>> Quick Specs:
> >>> - Solr 7.4 Solr Cloud
> >>> - 30gb index on 8 shards Tlog/Pull
> >>>
> >>> We run daily backups on our 30gb index and noticed that the overseer
> does
> >>> not process other jobs on it's task list while the backup is being
> taken.
> >>> They remain on the pending list (in ZK). Is this expected?
> >>>
> >>> Also I was wondering if there was a safe way to cancel a currently
> >> running
> >>> task or deleing pending tasks?
> >>>
> >>> Regards,
> >>>
> >>> Ash
> >>>
> >>> --
> >>> **
> >>> ** Empowering the world to design
> >>> Also, we're
> >>> hiring. Apply here! 
> >>>
> >>>  
> >>>  
> >>>   
> >>> 
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> > --
> > **
> > ** Empowering the world to design
> > Also, we're
> > hiring. Apply here! 
> >
> >  
> >  
> >   
> > 
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>

-- 
**
** Empowering the world to design
Also, we're 
hiring. Apply here!