RE: Index size increases disproportionately to size of added field when indexed=false

[Alessandro Benedetti]

Hi David,
good to know that sorting solved your problem.
I understand perfectly that given the urgency of your situation, having the
solution ready takes priority over continuing with the investigations.

I would recommend anyway to open a Jira issue in Apache Solr with all the
information gathered so far.
Your situation caught our attention and definitely changing the order of the
documents in input shouldn't affect the index size ( by such a greater
factor).
The fact that the optimize didn't change anything is even more suspicious.
It may be an indicator that in some edge cases ordering of input documents
is affecting one of the index data structure.
As a last thing when you have time I would suggest to :

1) index the ordering which gives you a small index - Optimize - Take note
of the size by index file extension

2) index the ordering which gives you a big index - Optimize - Take note of
the size by index file extension

And attach that to the Jira issue.
Whenever someone picks it up, that would definitely help.

Cheers




-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Plugins Documentation

[Emir Arnautović]

Hi,
It is not query parser but update request processor, but you can find simple 
buildable Solr plugin on 
https://github.com/od-bits/solr-multivaluefield-processor 
. You can also find 
a related blogpost that has some details how to build, configure and test it: 
http://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html 


HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 17 Feb 2018, at 23:50, Howe, David  wrote:
> 
> 
> You will need to use your favourite java tooling to take the code that you 
> have written and package it as a jar file.  In my case, we use maven so I 
> have set my custom extensions up as a maven project, and in my POM file 
> (which tells maven what dependencies your project has), I declare:
> 
>
>
>org.apache.solr
>solr-core
>6.4.0
>provided
>
>
> 
> This puts the Solr core classes on my classpath when I am compiling, but does 
> not package them up when I am building my jar file (scope=provided).
> 
> Once you have written your extension class, running mvn clean install will 
> create your jar file for you.  You then need to copy that jar file into the 
> solr/server/solr/lib directory so that Solr can find it and put it on the 
> classpath.  You should then be able to reference your custom extension in the 
> Solr config.
> 
> See https://maven.apache.org/ for details on maven.
> See https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html 
> for the basics on jar files.
> 
> If you use a tool like IntelliJ or Exclipse, they have maven and jar file 
> support built in.
> 
> Regards,
> 
> David
> 
> David Howe
> Java Domain Architect
> Postal Systems
> Level 16, 111 Bourke Street Melbourne VIC 3000
> 
> T  0391067904
> 
> M  0424036591
> 
> E  david.h...@auspost.com.au
> 
> W  auspost.com.au
> W  startrack.com.au
> 
> Australia Post is committed to providing our customers with excellent 
> service. If we can assist you in any way please telephone 13 13 18 or visit 
> our website.
> 
> The information contained in this email communication may be proprietary, 
> confidential or legally professionally privileged. It is intended exclusively 
> for the individual or entity to which it is addressed. You should only read, 
> disclose, re-transmit, copy, distribute, act in reliance on or commercialise 
> the information if you are authorised to do so. Australia Post does not 
> represent, warrant or guarantee that the integrity of this email 
> communication has been maintained nor that the communication is free of 
> errors, virus or interference.
> 
> If you are not the addressee or intended recipient please notify us by 
> replying direct to the sender and then destroy any electronic or paper copy 
> of this message. Any views expressed in this email communication are taken to 
> be those of the individual sender, except where the sender specifically 
> attributes those views to Australia Post and is authorised to do so.
> 
> Please consider the environment before printing this email.

Re: Getting the error - The field '*********' does not support spatial filtering

[Aakanksha Gupta]

Thanks David. That was helpful. I was able to get it to work with after I
removed my schema.xml and modified the managed-schema to add my fields. I'm
not sure if this is the right thing to do, but this is how I got it to work
:)

On Mon, Feb 19, 2018 at 7:54 AM, Howe, David 
wrote:

> Hi Aakanksha,
>
>
>
> We use the following for geo queries which works for us:
>
>
>
> /solr/core/select?defType=edismax&indent=on&ps=0&start=
> 0&wt=json&sow=true&hl=on&hl.fl=*&fq=%7B!geofilt%7D&pt=-6.
> 08165,145.8612430&d=10&sfield=geoLocation&sort=geodist()%
> 20asc&rows=10&fl=*,score,distance:geodist()
>
>
>
> This gives us the results closest to the provided point in order of their
> distance from the point.
>
>
>
> Our field definition is:
>
>
>
>   echo "$(date) Creating geoLocation field"
>
>   curl -X POST -H 'Content-type:application/json' --data-binary '{
>
> "add-field":{
>
>"name":"geoLocation",
>
>"type":"location",
>
>"stored":true,
>
>"indexed":true
>
> }
>
>   }' http://localhost:8983/solr/core/schema
>
>
>
> We are running Solr 7.1.0.
>
>
>
> Hope this helps.
>
>
>
> Regards,
>
>
>
> David
>
>
>
>
>
> *From:* Aakanksha Gupta [mailto:aakankshagupta2...@gmail.com]
> *Sent:* Monday, 19 February 2018 12:27 AM
> *To:* solr-user@lucene.apache.org
> *Subject:* Getting the error - The field '*' does not support
> spatial filtering
>
>
>
> Hi all,
>
> I'm a newbie to Solr. I'm trying to use it for GeoSpatial Search and I'm
> facing an issue while using it. I've tried using the new 'location' field
> type( docValues="true"/>) as well as the deprecated solr.LatLonType fieldtype,
> but I always get the error:
>
>
> org.apache.solr.common.SolrException: The field latlong does not support 
> spatial filtering
>
> Here's a snippet of my field definition in schema.xml in the conf folder
> of my core:
>
>  required="true" multiValued="false" />
> 
> 
>  docValues="false" />
>  multiValued="true"/>
> 
> 
> 
> 
> 
>  stored="true" />
>  stored="true" />
>
> And here are the field type definitions:
>  subFieldSuffix="_coordinate"/>
>  docValues="true"/>
>
> Here's the Query I'm running:
> http://localhost:8983/solr/geo2/select?wt=json&q=:&fq={!geofilt
> sfield=latlong}&pt=-6.08165,145.8612430&d=100
>
> http://localhost:8983/solr/geo2/select/?q=*:*&fq={!
> geofilt}&sfield=latlong2&pt=-6.08165,145.8612430&d=100&wt=json
>
> And here's the Java snippet I'm using to insert data:
> String urlString = "http://localhost:8983/solr/geo2";;
> SolrClient solr = new HttpSolrClient.Builder(urlString).build();
> SolrInputDocument document = new SolrInputDocument();
> document.addField("id", UUID.randomUUID().toString());
> document.addField("driverid", "1");
> document.addField("latlong", "-6.081689,145.391881");
> document.addField("time", "7:01:17");
> document.addField("timestamp", Long.valueOf("1518908477190"));
> document.addField("latlong2", "-6.081689,145.391881");
> document.addField("location_0_coordinate",
> Double.valueOf(-6.081689));
> document.addField("location_1_coordinate",
> Double.valueOf(145.391881));
> UpdateResponse response = solr.add(document);
> solr.commit();
> response.getQTime();
>
>
>
> I've attached my schema.xml file herewith. Can someone let me know what
> I'm doing wrong?
>
>
>
> David Howe
> Java Domain Architect
> Postal Systems
> Australia Post
>
> Level 16, 111 Bourke Street
> 
> Melbourne
> 
> VIC
> 
> 3000
> 
>
> T  0391067904
>
> M 0424036591
>
> E  david.h...@auspost.com.au
> [image: Australia Post website] 
> [image: StarTrack website] 
> [image: Follow us on Twitter]  [image: Like
> us on Facebook]  [image: Connect
> with us on LinkedIn] 
>
> Australia Post is committed to providing our customers with excellent
> service. If we can assist you in any way please telephone 13 13 18 or visit
> our website.
>
> The information contained in this email communication may be proprietary,
> confidential or legally professionally privileged. It is intended
> exclusively for the individual or entity to which it is addressed. You
> should only read, disclose, re-transmit, copy, distribute, act in reliance
> on or commercialise the information if you are authorised to do so.
> Australia Post does not represent, warrant or guaran

Re: SolrException: Error Instantiating queryParser, com.site.s.CustomQParserPlugin failed to instantiate org.apache.solr.search.QParserPlugin

[Emir Arnautović]

Hi,
Here is a blog post that shows how simple update processor is 
built/configured/deployed (you can ignore docValues part and why custom 
processor is done). You can follow that and see what you did wrong in your 
case: http://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html 


HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 17 Feb 2018, at 10:51, ~$alpha`  wrote:
> 
> 
>> I have created a custom query parser for Solr
> public class CustomQParserPlugin extends QParserPlugin {
> }
> 
>> Then I have imported the jar into Solr lib directory. i.e placed in
> server/lib directory
> 
>> I restarted the Solr
> 
>> getting below error
> SolrException: Error Instantiating queryParser,
> com.site.s.CustomQParserPlugin failed to instantiate
> org.apache.solr.search.QParserPlugin
> 
> I am getting the above error which means I am able to include my jar but no
> idea why I am getting error
> 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Help required with SolrJ

[Aakanksha Gupta]

Hi all,
I'm looking for some help with SolrJ for querying spatial data. I have the
following URL query working fine, where it returns the results which are
within 100km radius from the 'pt' provided in the URL and where the
timestamp field is between the two timestamps provided in the URL. It also
returns me the 'distance' for each result.

http://localhost:8983/solr/geoloc/select/?q=*:*&fq={!geofilt}&sfield=latlong&pt=-6.08165,145.8612430&d=100&wt=json&fq=timestamp:[151890840%20TO%20151891200]&fl=*,_dist_:geodist()

But I'm not sure how to build the SolrJ equivalent of this query using
SolrQuery. I tried something like

SolrQuery query = new SolrQuery();
query.setQuery("
&fq={!geofilt}&sfield=latlong&pt=-6.08165,145.8612430&d=100&wt=json&fq=timestamp:[151890840%20TO%20151891200]&fl=*,_dist_:geodist()")

and it throws errors. I've been trying to find some samples for 'SolrQuery'
class but without luck. Can someone give some sample usage of SolrQuery for
spatial data?

Re: Issue Using JSON Facet API Buckets in Solr 6.6

[Antelmo Aguilar]

Hi all,

I was wondering if the information I sent is sufficient to look into the
issue.  Let me know if you need anything else from me please.

Thanks,
Antelmo

On Thu, Feb 15, 2018 at 1:56 PM, Antelmo Aguilar  wrote:

> Hi,
>
> Here are two pastebins.  The first is the full complete response with the
> search parameters used.  The second is the stack trace from the logs:
>
> https://pastebin.com/rsHvKK63
>
> https://pastebin.com/8amxacAj
>
> I am not using any custom code or plugins with the Solr instance.
>
> Please let me know if you need anything else and thanks for looking into
> this.
>
> -Antelmo
>
> On Wed, Feb 14, 2018 at 12:56 PM, Yonik Seeley  wrote:
>
>> Could you provide the full stack trace containing "Invalid Date
>> String"  and the full request that causes it?
>> Are you using any custom code/plugins in Solr?
>> -Yonik
>>
>>
>> On Mon, Feb 12, 2018 at 4:55 PM, Antelmo Aguilar  wrote:
>> > Hi,
>> >
>> > I was using the following part of a query to get facet buckets so that I
>> > can use the information in the buckets for some post-processing:
>> >
>> > "json":
>> > "{\"filter\":[\"bundle:pop_sample\",\"has_abundance_data_b:
>> true\",\"has_geodata:true\",\"${project}\"],\"facet\":{\"ter
>> m\":{\"type\":\"terms\",\"limit\":-1,\"field\":\"${term:spec
>> ies_category}\",\"facet\":{\"collection_dates\":{\"type\":\
>> "terms\",\"limit\":-1,\"field\":\"collection_date\",\"facet\
>> ":{\"collection\":
>> > {\"type\":\"terms\",\"field\":\"collection_assay_id_s\",\"fa
>> cet\":{\"abnd\":\"sum(div(sample_size_i,
>> > collection_duration_days_i))\""
>> >
>> > Sorry if it is hard to read.  Basically what is was doing was getting
>> the
>> > following buckets:
>> >
>> > First bucket will be categorized by "Species category" by default
>> unless we
>> > pass in the request the "term" parameter which we will categories the
>> first
>> > bucket by whatever "term" is set to.  Then inside this first bucket, we
>> > create another buckets of the "Collection date" category.  Then inside
>> the
>> > "Collection date" category buckets, we would use some functions to do
>> some
>> > calculations and return those calculations inside the "Collection date"
>> > category buckets.
>> >
>> > This query is working fine in Solr 6.2, but I upgraded our instance of
>> Solr
>> > 6.2 to the latest 6.6 version.  However it seems that upgrading to Solr
>> 6.6
>> > broke the above query.  Now it complains when trying to create the
>> buckets
>> > of the "Collection date" category.  I get the following error:
>> >
>> > Invalid Date String:'Fri Aug 01 00:00:00 UTC 2014'
>> >
>> > It seems that when creating the buckets of a date field, it does some
>> > conversion of the way the date is stored and causes the error to appear.
>> > Does anyone have an idea as to why this error is happening?  I would
>> really
>> > appreciate any help.  Hopefully I was able to explain my issue well.
>> >
>> > Thanks,
>> > Antelmo
>>
>
>

7.2.1 ExactStatsCache seems no longer functioning

[Markus Jelsma]

Hello,

We're on 7.2.1 and rely on ExactStatsCache to work around the problem of not 
all nodes sharing the same maxDoc within a shard. But, it doesn't work, anymore!

I've looked things up in Jira but nothing so far. SOLR-10952 also doesn't cause 
it because with queryResultCache disabled, document scores don't match up, the 
ordering of search results is not constant for the same query in consecutive 
searches. 

We see this on a local machine, just with default similarity and classic query 
parser.

Any hints on what to do now?

Many thanks,
Markus

Re: Help required with SolrJ

[Shawn Heisey]

On 2/19/2018 6:44 AM, Aakanksha Gupta wrote:

http://localhost:8983/solr/geoloc/select/?q=*:*&fq={!geofilt}&sfield=latlong&pt=-6.08165,145.8612430&d=100&wt=json&fq=timestamp:[151890840%20TO%20151891200]&fl=*,_dist_:geodist()

But I'm not sure how to build the SolrJ equivalent of this query using
SolrQuery.


I haven't done anything with spatial yet.  But I do know how to 
translate Solr URLs into SolrJ code.  The code below constructs a query 
object equivalent to that URL.  If that URL works as-is, this code 
should do the same.


I did not include the "wt" parameter, which controls the format of the 
response.  With SolrJ, the transfer format defaults to binary and should 
not be changed.  It CAN be changed, but any other choice would be less 
efficient, and the programmer doesn't need to worry about it.


  query.setQuery("*:*");
  query.addFilterQuery("{!geofilt}");
query.addFilterQuery("timestamp:[151890840%20TO%20151891200]");
  query.set("sfield", "latlong");
  query.set("pt", "-6.08165,145.8612430");
  query.set("d", "100");
  query.setFields("*", "_dist_:geodist()");

I couldn't actually test this code, as I don't have any indexes with 
spatial data.


Thanks,
Shawn

Facet performance problem

[MOUSSA MZE Oussama-ext]

Hi

We have following environement :

3 nodes cluster
1 shard
Replication factor = 2
8GB per node

29 millions of documents

We've faceting over field "motifPresence" defined as follow:



Once the user selects motifPresence filter we executes search again with:

fq: (value1 OR value2 OR value3 OR ...)

The problem is: During facet filtering query is too slow and her response time 
is greater than main search (without facet filtering).

Thanks in advance!

Re: Help required with SolrJ

[Aakanksha Gupta]

Hi Shawn,
Thanks for the quick solution. It works. I just had to replace %20 to space
in query.addFilterQuery("timestamp:[151890840 TO 151891200]");

Thanks a ton! :)

On Mon, Feb 19, 2018 at 11:43 PM, Shawn Heisey 
wrote:

> On 2/19/2018 6:44 AM, Aakanksha Gupta wrote:
>
>> http://localhost:8983/solr/geoloc/select/?q=*:*&fq={!geofilt
>> }&sfield=latlong&pt=-6.08165,145.8612430&d=100&wt=json&fq=
>> timestamp:[151890840%20TO%20151891200]&fl=*,_dist_:geodist()
>> 
>>
>> But I'm not sure how to build the SolrJ equivalent of this query using
>> SolrQuery.
>>
>
> I haven't done anything with spatial yet.  But I do know how to translate
> Solr URLs into SolrJ code.  The code below constructs a query object
> equivalent to that URL.  If that URL works as-is, this code should do the
> same.
>
> I did not include the "wt" parameter, which controls the format of the
> response.  With SolrJ, the transfer format defaults to binary and should
> not be changed.  It CAN be changed, but any other choice would be less
> efficient, and the programmer doesn't need to worry about it.
>
>   query.setQuery("*:*");
>   query.addFilterQuery("{!geofilt}");
> query.addFilterQuery("timestamp:[151890840%20TO%20151891200]");
>   query.set("sfield", "latlong");
>   query.set("pt", "-6.08165,145.8612430");
>   query.set("d", "100");
>   query.setFields("*", "_dist_:geodist()");
>
> I couldn't actually test this code, as I don't have any indexes with
> spatial data.
>
> Thanks,
> Shawn
>
>

RE: 7.2.1 ExactStatsCache seems no longer functioning

[Markus Jelsma]

Correction, SOLR-10952 is actually the real case (i misconfigured a test 
previously, p).

Well, what's the solution around this problem then? I would assume the cache 
stores the results of a query that was already executed with ExactStats, but 
that doesn't seem to be the case. 

So, what to do?

Many thanks, 
Markus

 
 
-Original message-
> From:Markus Jelsma 
> Sent: Monday 19th February 2018 16:41
> To: Solr-user 
> Subject: 7.2.1 ExactStatsCache seems no longer functioning
> 
> Hello,
> 
> We're on 7.2.1 and rely on ExactStatsCache to work around the problem of not 
> all nodes sharing the same maxDoc within a shard. But, it doesn't work, 
> anymore!
> 
> I've looked things up in Jira but nothing so far. SOLR-10952 also doesn't 
> cause it because with queryResultCache disabled, document scores don't match 
> up, the ordering of search results is not constant for the same query in 
> consecutive searches. 
> 
> We see this on a local machine, just with default similarity and classic 
> query parser.
> 
> Any hints on what to do now?
> 
> Many thanks,
> Markus
> 

Reg Elevate component

[bbarani]

I was under the impression that elevate component would only elevate if the
document is part of the returned result set(at some position) for that
searched keyword. Is that true? 

I see that the results are elevated even if the elevated document doesn't
match with the keyword (score - 0) now. I want to know if something got
changed recently related to elevate component.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Reg Elevate component

[Erick Erickson]

bq: "... elevate component would only elevate if the document is part
of the returned result set..."

That's never been true that I know of. QEC is a sledgehammer, when you
say "I want this doc returned for this query" it's assumed you really
_mean_ it and the doc will be returned no matter what.

Are you saying you see different behavior from previous versions of Solr?

Best,
Erick

On Mon, Feb 19, 2018 at 8:36 AM, bbarani  wrote:
>
> I was under the impression that elevate component would only elevate if the
> document is part of the returned result set(at some position) for that
> searched keyword. Is that true?
>
> I see that the results are elevated even if the elevated document doesn't
> match with the keyword (score - 0) now. I want to know if something got
> changed recently related to elevate component.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Help required with SolrJ

[Erick Erickson]

Aakanksha:

Be a little careful here, filter queries with timestamps can be
tricky. The example you have is fine, but for end-points with finer
granularity may be best if you don't cache them, see:
https://lucidworks.com/2012/02/23/date-math-now-and-filter-queries/

Best,
Erick

On Mon, Feb 19, 2018 at 7:49 AM, Aakanksha Gupta
 wrote:
> Hi Shawn,
> Thanks for the quick solution. It works. I just had to replace %20 to space
> in query.addFilterQuery("timestamp:[151890840 TO 151891200]");
>
> Thanks a ton! :)
>
> On Mon, Feb 19, 2018 at 11:43 PM, Shawn Heisey 
> wrote:
>
>> On 2/19/2018 6:44 AM, Aakanksha Gupta wrote:
>>
>>> http://localhost:8983/solr/geoloc/select/?q=*:*&fq={!geofilt
>>> }&sfield=latlong&pt=-6.08165,145.8612430&d=100&wt=json&fq=
>>> timestamp:[151890840%20TO%20151891200]&fl=*,_dist_:geodist()
>>> 
>>>
>>> But I'm not sure how to build the SolrJ equivalent of this query using
>>> SolrQuery.
>>>
>>
>> I haven't done anything with spatial yet.  But I do know how to translate
>> Solr URLs into SolrJ code.  The code below constructs a query object
>> equivalent to that URL.  If that URL works as-is, this code should do the
>> same.
>>
>> I did not include the "wt" parameter, which controls the format of the
>> response.  With SolrJ, the transfer format defaults to binary and should
>> not be changed.  It CAN be changed, but any other choice would be less
>> efficient, and the programmer doesn't need to worry about it.
>>
>>   query.setQuery("*:*");
>>   query.addFilterQuery("{!geofilt}");
>> query.addFilterQuery("timestamp:[151890840%20TO%20151891200]");
>>   query.set("sfield", "latlong");
>>   query.set("pt", "-6.08165,145.8612430");
>>   query.set("d", "100");
>>   query.setFields("*", "_dist_:geodist()");
>>
>> I couldn't actually test this code, as I don't have any indexes with
>> spatial data.
>>
>> Thanks,
>> Shawn
>>
>>

Issue :-Unable to write response, client closed connection or we are shutting down org.eclipse.jetty.io.EofException: Closed

[Sidharth Aggarwal]

Hello Team,


Hello We are getting below error while downloading indexing data (basically 
tagging them)



o.a.s.s.HttpSolrCall Unable to write response, client closed connection or we 
are shutting down
org.eclipse.jetty.io.EofException: Closed
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:620)
at 
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:55)
at 
org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54)
at java.io.OutputStream.write(OutputStream.java:116)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:140)
at org.apache.solr.util.FastWriter.flushBuffer(FastWriter.java:154)
at org.apache.solr.response.TextResponseWriter.close(TextResponseWriter.java:93)
at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:73)
at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:809)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:538)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)




Kernal version : Linux qa-solr-lx21 4.4.103-92.56-default #1 SMP Wed Dec 27 
16:24:31 UTC 2017 (2fd2155) x86_64 x86_64 x86_64 GNU/Linux



Java Version : java version "1.8.0_131"  Java(TM) SE Runtime Environment (build 
1.8.0_131-b11)

Solr version :6.6

CPU :6



Please help to rectify this issue.


Regards,
Sidharth Aggarwal | Senior IT Operations Specialist - Internal Platforms
McKinsey & Company, Inc. | Vatika Business Park | Sector - 49 Sohna Road | 
Gurgaon 122018 | India
T +91 124 333 1378 | M +91 9278987563  | Internal 871 1378

++
This email is confidential and may be privileged. If you have received it
in error, please notify us immediately and then delete it.  Please do not
copy it, disclose its contents or use it for any purpose.
++

Re: Reg Elevate component

[bbarani]

I used to use elevate.xml before (in SOLR 4.1) and never noticed this
behavior before (may be I didn't check these specific use cases where
elevated documents doesn't contain any searched keyword) but I started
elevating id's via query param now (using elevateIds parameter) and I
started noticing this behavior so wanted to confirm...



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Facet performance problem

[Erick Erickson]

I'm confused here. What do you mean by "facet filtering"? Your
examples have no facets at all, just a _filter query_.

I'll assume you want to use filter query (fq), and faceting has
nothing to do with it. This is one of the tricky bits of docValues.
While it's _possible_ to search on a field that's defined as above,
it's very inefficient since there's no "inverted index" for the field,
you specified 'indexed="false" '. So the docValues are searched, and
it's essentially a table scan.

If you mean to search against this field, set indexed="true". You'll
have to completely reindex your corpus of course.

If you intend to facet, group or sort on this field, you should _also_
have docValues="true".

Best,
Erick

On Mon, Feb 19, 2018 at 7:47 AM, MOUSSA MZE Oussama-ext
 wrote:
> Hi
>
> We have following environement :
>
> 3 nodes cluster
> 1 shard
> Replication factor = 2
> 8GB per node
>
> 29 millions of documents
>
> We've faceting over field "motifPresence" defined as follow:
>
>  stored="true" required="false"/>
>
> Once the user selects motifPresence filter we executes search again with:
>
> fq: (value1 OR value2 OR value3 OR ...)
>
> The problem is: During facet filtering query is too slow and her response 
> time is greater than main search (without facet filtering).
>
> Thanks in advance!

Re: Reg Elevate component

[Erick Erickson]

Right, so I'd guess if you checked 4.1 you'd see the same behavior. If
not please let us know.

On Mon, Feb 19, 2018 at 9:17 AM, bbarani  wrote:
> I used to use elevate.xml before (in SOLR 4.1) and never noticed this
> behavior before (may be I didn't check these specific use cases where
> elevated documents doesn't contain any searched keyword) but I started
> elevating id's via query param now (using elevateIds parameter) and I
> started noticing this behavior so wanted to confirm...
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Getting the error - The field '*********' does not support spatial filtering

[Erick Erickson]

bq:  I was able to get it to work with after I removed my schema.xml
and modified the managed-schema to add my fields

That's fine. You have three options and they can be confusing:
1> use classic schema (i.e. schema.xml). Requires that you replace the
managed schema definition with classic in solrcofnig.xml, it'll look
something like:
 

2> use managed scheama

2a> without "field guessing". In this mode you take out stuff like
"add-unknown-fields-to-the-schema" in solrconfig.xml (really the
chain). You can hand-edit the managed-schema file freely. HOWEVER, you
should be very controlled when mixing hand-editing the managed-schema
file with using the managed-schema API to modify it. The
managed-schema API will overwrite the config with whatever is in
memory so your hand-edits may be lost.

2b> Allow Solr to try to infer fields based on the first one
encountered, "schemaless mode". This is the _default config in 7x.

See the "Schema Factory Definition in SolrConfig" section of the
reference guide.

Best,
Erick

On Mon, Feb 19, 2018 at 5:28 AM, Aakanksha Gupta
 wrote:
> Thanks David. That was helpful. I was able to get it to work with after I
> removed my schema.xml and modified the managed-schema to add my fields. I'm
> not sure if this is the right thing to do, but this is how I got it to work
> :)
>
> On Mon, Feb 19, 2018 at 7:54 AM, Howe, David 
> wrote:
>
>> Hi Aakanksha,
>>
>>
>>
>> We use the following for geo queries which works for us:
>>
>>
>>
>> /solr/core/select?defType=edismax&indent=on&ps=0&start=
>> 0&wt=json&sow=true&hl=on&hl.fl=*&fq=%7B!geofilt%7D&pt=-6.
>> 08165,145.8612430&d=10&sfield=geoLocation&sort=geodist()%
>> 20asc&rows=10&fl=*,score,distance:geodist()
>>
>>
>>
>> This gives us the results closest to the provided point in order of their
>> distance from the point.
>>
>>
>>
>> Our field definition is:
>>
>>
>>
>>   echo "$(date) Creating geoLocation field"
>>
>>   curl -X POST -H 'Content-type:application/json' --data-binary '{
>>
>> "add-field":{
>>
>>"name":"geoLocation",
>>
>>"type":"location",
>>
>>"stored":true,
>>
>>"indexed":true
>>
>> }
>>
>>   }' http://localhost:8983/solr/core/schema
>>
>>
>>
>> We are running Solr 7.1.0.
>>
>>
>>
>> Hope this helps.
>>
>>
>>
>> Regards,
>>
>>
>>
>> David
>>
>>
>>
>>
>>
>> *From:* Aakanksha Gupta [mailto:aakankshagupta2...@gmail.com]
>> *Sent:* Monday, 19 February 2018 12:27 AM
>> *To:* solr-user@lucene.apache.org
>> *Subject:* Getting the error - The field '*' does not support
>> spatial filtering
>>
>>
>>
>> Hi all,
>>
>> I'm a newbie to Solr. I'm trying to use it for GeoSpatial Search and I'm
>> facing an issue while using it. I've tried using the new 'location' field
>> type(> docValues="true"/>) as well as the deprecated solr.LatLonType fieldtype,
>> but I always get the error:
>>
>>
>> org.apache.solr.common.SolrException: The field latlong does not support 
>> spatial filtering
>>
>> Here's a snippet of my field definition in schema.xml in the conf folder
>> of my core:
>>
>> > required="true" multiValued="false" />
>> 
>> 
>> > docValues="false" />
>> > multiValued="true"/>
>> 
>> 
>> 
>> 
>> 
>> > stored="true" />
>> > stored="true" />
>>
>> And here are the field type definitions:
>> > subFieldSuffix="_coordinate"/>
>> > docValues="true"/>
>>
>> Here's the Query I'm running:
>> http://localhost:8983/solr/geo2/select?wt=json&q=:&fq={!geofilt
>> sfield=latlong}&pt=-6.08165,145.8612430&d=100
>>
>> http://localhost:8983/solr/geo2/select/?q=*:*&fq={!
>> geofilt}&sfield=latlong2&pt=-6.08165,145.8612430&d=100&wt=json
>>
>> And here's the Java snippet I'm using to insert data:
>> String urlString = "http://localhost:8983/solr/geo2";;
>> SolrClient solr = new HttpSolrClient.Builder(urlString).build();
>> SolrInputDocument document = new SolrInputDocument();
>> document.addField("id", UUID.randomUUID().toString());
>> document.addField("driverid", "1");
>> document.addField("latlong", "-6.081689,145.391881");
>> document.addField("time", "7:01:17");
>> document.addField("timestamp", Long.valueOf("1518908477190"));
>> document.addField("latlong2", "-6.081689,145.391881");
>> document.addField("location_0_coordinate",
>> Double.valueOf(-6.081689));
>> document.addField("location_1_coordinate",
>> Double.valueOf(145.391881));
>> UpdateResponse response = solr.add(document);
>> solr.commit();
>> response.getQTime();
>>
>>
>>
>> I've attached my schema.xml file herewith. Can someone let me know what
>> I'm doing wrong?
>>
>>
>>
>> David Howe
>> Java Domain Architect
>> Postal Systems
>> Australia Post
>>
>> Level 16, 111 Bourke Street
>> 
>> Melbourne
>> 

Re: Help required with SolrJ

[Shawn Heisey]

On 2/19/2018 8:49 AM, Aakanksha Gupta wrote:
> Thanks for the quick solution. It works. I just had to replace %20 to space
> in query.addFilterQuery("timestamp:[151890840 TO 151891200]");
> 
> Thanks a ton! :)

Right, I didn't even really look closely at what was in the fq
parameter, I just copied it. :)  Sorry about that -- if I'd looked
better, I would have seen that what I was sending wouldn't work.

SolrJ will handle the URL encoding for you, so it would have URL encoded
the URL encoding, and Solr would receive the fq parameter with the %20
intact.

Glad you figured it out even with my mistake!

Thanks,
Shawn

Need help with match contains query in SOLR

[bbarani]

Hi,

I have a requirement where I want to perform the 'contains' match and would
need your help to define the fieldtype and query for this requirement.

Value stored in SOLR:

transfer responsibility
transfer account


Now, I want the above document to be returned for the below keyword when I
search with additional phrases (when it matches the searched keyword
entirely only)
1.) transfer responsibility issues
1.) how to fix transfer responsibility issues 
2.) transfer account issues
3.) troubleshoot transfer account 
4.)how to do transfer responsibility
etc...

But should not return when searching for
1.) transfer issues
2.) account issues
3.) how to troubleshoot account 
4.)how to do transfer

Idea is to return the document only when the search word matches the whole
value stored in SOLR but the actual search word might have additional
keyword compared to the value stored in SOLR.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Hard commits blocked | non-solrcloud v6.6.2

[mmb1234]

The below solrconfig.xml settings resolved the TIMED_WAIT in
ConcurrentMergeScheduler.doStall(). Thanks to Shawn and Erik for their
pointers.

  
...

   30
   100
   30.0


   18
   6

300
...
  

  

  ${solr.autoCommit.maxTime:3}
  false


  ${solr.autoSoftCommit.maxTime:1}
  ${solr.autoSoftCommit.maxDocs:-1}
  true

  



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

[mmb1234]

FYI. This issue went away after solrconfig.xml was tuned.

"Hard commits blocked | non-solrcloud v6.6.2" thread has the details.
http://lucene.472066.n3.nabble.com/Hard-commits-blocked-non-solrcloud-v6-6-2-td4374386.html




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: solr cloud unique key query request is sent to all shards!

[Ganesh Sethuraman]

This works !. Both V1 and V2 version of the real time get works fine.
Just an added note, the performance (response time) also improved.

Thanks Tom

On Mon, Feb 19, 2018 at 1:17 AM, Tomas Fernandez Lobbe 
wrote:

> In real-time get, the parameter name is “id”, regardless of the name of
> the unique key.
>
> The request should be in your case: http://:8080/api/
> collections/col1/get?id=69749398
>
> See: https://lucene.apache.org/solr/guide/7_2/realtime-get.html
>
> Sent from my iPhone
>
> > On Feb 18, 2018, at 9:28 PM, Ganesh Sethuraman 
> wrote:
> >
> > I tried this real time get on my collection using the both V1 and V2 URL
> > for real time get, but did not work!!!
> >
> > http://:8080/api/collections/col1/get?myid:69749398
> >
> > it returned...
> >
> > {
> >  "doc":null}
> >
> > same issue with V1 URL as well, http://
> > :8080/solr/col1/get?myid:69749398
> >
> > however if i do q=myid:69749398 with "select" request handler seems to
> > fine. I checked my schema again and it is configured correctly.  Like
> below:
> >
> > myid
> >
> > Also i see that this implicit request handler is configured correctly Any
> > thoughts, what I might be missing?
> >
> >
> >
> > On Sun, Feb 18, 2018 at 11:18 PM, Tomas Fernandez Lobbe <
> tflo...@apple.com>
> > wrote:
> >
> >> I think real-time get should be directed to the correct shard. Try:
> >> [COLLECTION]/get?id=[YOUR_ID]
> >>
> >> Sent from my iPhone
> >>
> >>> On Feb 18, 2018, at 3:17 PM, Ganesh Sethuraman <
> ganeshmail...@gmail.com>
> >> wrote:
> >>>
> >>> Hi
> >>>
> >>> I am using Solr 7.2.1. I have 8 shards in two nodes (two different m/c)
> >>> using Solr Cloud. The data was indexed with a unique key (default
> >> composite
> >>> id) using the CSV update handler (batch indexing). Note that I do NOT
> >> have
> >>>  while indexing.   Then when I try to  query the
> >>> collection col1 based on my primary key (as below), I see that in the
> >>> 'debug' response that the query was sent to all the shards and when it
> >>> finds the document in one the shards it sends a GET FIELD to that shard
> >> to
> >>> get the data.  The problem is potentially high response time, and more
> >>> importantly scalability issue as unnecessarily all shards are being
> >> queried
> >>> to get one document (by unique key).
> >>>
> >>> http://:8080/solr/col1/select?debug=true&q=id:69749278
> >>>
> >>> Is there a way to query to reach the right shard based on the has of
> the
> >>> unique key?
> >>>
> >>> Regards
> >>> Ganesh
> >>
>

ZK session times out intermittently

[Roy Lim]

Hi all, here's my setup:

6 x Solr (3 primary shard, 3 secondary)
3 x ZK

The client is indexing over 16 million documents using 8 threads.  Auto-soft
commit is 3 minutes, auto-commit is 10 minutes.

The following timeout is observed in our client log, intermittently:



Thinking that this is a case where ZK could no longer establish connection
to Solr node it is communicating with, I went to the primary nodes and
correlated the timestamps.  They all are very similar to below:



Note the time gap of over 1 minute, which I can only surmise that ZK is
waiting this whole time for Solr to return, only to timeout.  Is that
reasonable?  Thing is I have no idea what is happening in during that time
and why Solr is taking so long.  Note the second statement signaling the
start of the soft commit, so I don't think this is a case of a long commit.

Finally, checking the GC logs, there are no long pauses either!

Hoping an expert can shed some light here.





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Registering

[Graham Berends]

Hi Solr/Lucene

My name is Graham Berends.

Just installed Solr on my Linux machine: solr-7.2.1

Regards
Graham

 Processor -
MemTotal: 4040920 kB
model name  : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz 64 bit
cpu cores   : 4
architecture : 64bit
system   : GNU/Linux
release  : 4.13.12-300.fc27.x86_64
distribution : Fedora 27 Twenty Severn