Re: Solr deep paging queries run very slow due to redundant q param

[Toke Eskildsen]

On Sat, 2017-10-14 at 16:52 -0700, Sundeep T wrote:
> In our scale environment, we see that the deep paging queries  using
> cursormark are running really slow. When we traced out the calls, we
> see that the second query which queries the individual id's of
> matched pages is sending the q param that is already sent by the
> first query again.

Use fq instead of q, so that it will be cached for subsequent calls.

>  If we remove the q param and directly query for ids, the query runs
> really fast.

Sure. But that requires you to know the IDs to be matched.

[Solr-request with heavy q resulting in 40M hits]

> This query results in a second query due to solr implementation of
> deep paging like below. In this query, we already know the ids to be
> matched.
>
> So, there is no reason to pass the q param again. We tried manually
> executing the below query without the q param and just passing the
> ids alone and that executes in 50ms.

You only have the first 50 IDs from the first query. You cannot page
forward without knowing the IDs that follow, which if why you need to
state the q again (or convert it to fq).

It is not a bug in Solr, it is a design decision. In order to provide
pagination without recomputing the result set, you would need a
guaranteed server-side state. Solr does not implement that pattern and
thanks for that.

- Toke Eskildsen, Royal Danish Library

Re: Facets based on sampling

[Emir Arnautović]

Hi John,
Did you mean “docValues don’t work for analysed fields” since it works for 
multivalue string (or other supported types) fields. What you need to do is to 
convert your analysed field to multivalue string field - that requires changes 
in indexing flow.

HTH,
Emir 
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 23 Oct 2017, at 21:08, John Davis  wrote:
> 
> Docvalues don't work for multivalued fields. I just started a separate
> thread with more debug info. It is a bit surprising why facet computation
> is so slow even when the query matches hundreds of docs.
> 
> On Mon, Oct 23, 2017 at 6:53 AM, alessandro.benedetti 
> wrote:
> 
>> Hi John,
>> first of all, I may state the obvious, but have you tried docValues ?
>> 
>> Apart from that a friend of mine ( Diego Ceccarelli) was discussing a
>> probabilistic implementation similar to the hyperloglog[1] to approximate
>> facets counting.
>> I didn't have time to take a look in details / implement anything yet.
>> But it is on our To Do list :)
>> He may add some info here.
>> 
>> Cheers
>> 
>> 
>> 
>> 
>> [1]
>> https://blog.yld.io/2017/04/19/hyperloglog-a-probabilistic-data-structure/
>> 
>> 
>> 
>> -
>> ---
>> Alessandro Benedetti
>> Search Consultant, R&D Software Engineer, Director
>> Sease Ltd. - www.sease.io
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>> 

Re: How to Efficiently Extract Learning to Rank Similarity Features From Solr?

[alessandro.benedetti]

i think this can be actually a good idea and I think that would require a new
feature type implementation.

Specifically I think you could leverage the existing data structures ( such
TermVector) to calculate the matrix and then perform the calculations you
need.
Or maybe there is space for even a new optional data structure in the index,
to support matrix calculation ( it's been a while I don't take a look to
codecs and index file formats).



-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Query Boosting and sort

[Renuka Srishti]

Sorry for late reply. Thanks Erick for the solution, you got my problem and
the picture is right. I have resolved my problem with the similar approach
you mentioned here.

Regards,
Renuka Srishti

On Sat, Sep 9, 2017 at 11:45 PM, Erick Erickson 
wrote:

> I'm a completely lost here.
>
> bq: "I have designed a screen where results are showing in tabular
> form, in each column I have applied sorting( using Solr sort
> parameter)"
>
> What does that mean? What is the relationship between columnA and
> columnB in the same row? Are they from the same doc? Different docs?
> What information is the user getting when they compare columnA and
> columnB in the same row? What information is the user getting when
> they compare columnA from row1 and row2?
>
> bq: "There is one keyword search box, in which I have applied the
> phrase boosting to maintain relevancy.( Most relevant result will show
> on the top).
>
> This seems like a totally unrelated statement. You "applied the phrase
> boosting to maintain relevancy". What relationship does that have to
> "in each column I have applied sorting"?
>
> ---
>  It _sounds_ like you have some kind of grid format where you have N
> docs (rows) and M columns (fields). So each row (doc) has some fields
> displayed. Now you want to sort results by different columns (I'm
> thinking like a spreadsheet where you can sort by clicking the
> different column heads).
>
> So assuming my mental picture is accurate, you have two choices:
>
> 1> you have the top N relevant documents that you want to display
> differently depending on the column selected. When the user switches
> columns you re-sort the top N docs. This is entirely an application
> level sort. You do a single search that returns the top N docs by
> relevance. When the user clicks on a column head you re-sort that
> result set without going to Solr at all.
>
> 2> when the user clicks a column head, you want to display the top N
> docs _from the entire index in sorted order_, which would be a new
> Solr query with sort=column.
>
> Best,
> Erick
>
> On Fri, Sep 8, 2017 at 9:50 PM, Renuka Srishti
>  wrote:
> > Thanks Rick and Erick for your response.
> > Here is the situation where I want to use both sort and phrase boosting:
> >
> >- I have designed a screen where results are showing in tabular form,
> in
> >each column I have applied sorting( using Solr sort parameter). There
> is
> >one keyword search box, in which I have applied the phrase boosting to
> >maintain relevancy.( Most relevant result will show on the top).
> >
> >
> >- Now If I applied keyword search and then I want to sort the result,
> >how I can achieve this?(Sorting completely overrides scoring).
> >
> > Thanks
> >
> > Renuka Srishti
> >
> > On Sat, Sep 9, 2017 at 1:38 AM, Erick Erickson 
> > wrote:
> >
> >> Sorting completely overrides scoring. By specifying a sort parameter
> >> you're effectively telling Solr that you don't care about scoring,
> >> order the docs by the sort criteria.
> >>
> >> On Fri, Sep 8, 2017 at 3:35 AM, Rick Leir  wrote:
> >> > Renuka,
> >> >
> >> > You have not told us nearly enough about your issue. What query?
> config?
> >> >
> >> > cheers -- Rick
> >> >
> >> >
> >> >
> >> > On 2017-09-08 05:42 AM, Renuka Srishti wrote:
> >> >>
> >> >> Hello All,
> >> >>
> >> >> I am trying to use sort parameter and phrase boosting together in
> >> search.
> >> >> But, if I use the sort parameter, it seems like Phrase Boosting does
> not
> >> >> work with it.
> >> >>
> >> >> Thanks
> >> >> Renuka Srishti
> >> >>
> >> >
> >>
>

Re: Stats component with function

[Renuka Srishti]

Hello All,

I am exploring the functions
 and I
think we can perform the above query if we extend
"org.apache.solr.search.function" package to support all available
functions. Are we planning to extend this package to support all functions
like it supports ord().If I am going wrong please correct me.

Thanks
Renuka Srishti

On Tue, Oct 17, 2017 at 2:05 PM, Renuka Srishti 
wrote:

> Hello All
>
> Here is the link
> 
> for the reference. I want to use sub() by passing the min and max value of
> the price, can we perform this type of action, with the query given in the
> link.
>
> Thanks
> Renuka Srishti
>

Re: Solr nodes going into recovery mode and eventually failing

[Emir Arnautović]

Hi Shamik,
Please see incline comments/questions.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 24 Oct 2017, at 07:41, shamik  wrote:
> 
> Thanks Emir and Zisis.
> 
> I added the maxRamMB for filterCache and reduced the size. I could the
> benefit immediately, the hit ratio went to 0.97. Here's the configuration:
> 
>  autowarmCount="128" maxRamMB="500" />
>  autowarmCount="128" />
>  autowarmCount=“0" />
[EA] Based on what you mentioned earlier, not all your filters are “cache 
friendly” and hit rate depends on your clients usage (or maybe how many 
concurrent clients you have) - in other words this hit ratio could be false 
positive. The one explanation is that you previously had count limitation and 
now memory and that memory ignores count (would need to check this) and that 
your cache items are on average smaller then estimated and more than 4K can fit 
into 500MB, but based on your query rate and commit interval, not sure if you 
needed more than 4K.

> 
> It seemed to be stable for few days, the cache hits and jvm pool utilization
> seemed to be well within expected range. But the OOM issue occurred on one
> of the nodes as the heap size reached 30gb. The hit ratio for query result
> cache and document cache at that point was recorded as 0.18 and 0.65. I'm
> not sure if the cache caused the memory spike at this point, with filter
> cache restricted to 500mb, it should be negligible. One thing I noticed is
> that the eviction rate now (with the addition of maxRamMB) is staying at 0.
[EA] Did you see evictions before? With 400rq/h and 10min commit intervals, you 
get 60-70 rq between two commits. With 4K cache size, each request should 
consume >60 cache entries in order to start evicting.

> Index hard commit happens at every 10 min, that's when the cache gets
> flushed. Based on the monitoring log, the spike happened on the indexing
> side where almost 8k docs went to pending state.
> 
> On the query performance standpoint, there have been occasional slow queries
> (1sec+), but nothing alarming so far. Same goes for deep paging, I haven't
> seen any evidence which points to that.
> 
> Based on the hit ratio, I can further scale down the query result and
> document cache, also change to FastLRUCache and add maxRamMB. For filter
> cache, I think this setting should be optimal enough to work on a 30gb heap
> space unless I'm wrong on the maxRamMB concept. I'll have to get a heap dump
> somehow, unfortunately, the whole process (of the node going down) happens
> so quickly, I’ve hardly any time to run a profiler.
[EA] You did not mention if you ruled out fieldCache and fieldValueCache?
I don’t have much experience with LTR, but see that there is another cache 
related to that. Do you use it? Could it be the component that consumes memory?

> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Accent insensitive search for greek characters

[Chitra]

Hi Alexandre,
   ICUTransformFilter is working fine for greek characters
alone as per requirement. but one case it's breaking( σ & ς are the lower
forms of Σ Sigma).

*Example:*

I indexed the terms πελάτης (indexed as πελατης) & πελάτηΣ (indexed as
πελατης).

I get the expected search results if I perform the search for πελάτηΣ (or)
πελάτης (or) any combinations of upper case & lower case Greek characters.
But if I search as πελατησ I won't get any search results.

In Greek, σ & ς are the lower forms of Σ Sigma. And this case is solved in
ICUFoldingFilter.


Is ICU Transliterator rule formed right? Kindly look at the below code


TokenStream tok = new ICUTransformFilter(tok,
> Transliterator.getInstance("Greek; Lower; NFD; [:Nonspacing Mark:] Remove;
> NFC;"));



Kindly help me to resolve this.


Regards,
Chitra

Date range queries no longer work 6.6 to 7.1

[Markus Jelsma]

Hello,

We just attempted an upgrade which went fine so far. But now we have a problem, 
date range queries can no longer find records from before the upgrade, only 
records after the upgrade, it probably has something to do with DatePointField. 
To be clear, we upgraded our old date fields to the new Point type.

I could not find anything in the changelog.

How can we fix this?

Thanks,
Markus

Re: Retrieve DocIdSet from Query in lucene 5.x

[David Smiley]

See SolrIndexSearcher.getDocSet.  It may not be identical to what you want
but following what it does on through to DocSetUtil.createDocSet may be
enlightening.

On Fri, Oct 20, 2017 at 5:10 PM Jamie Johnson  wrote:

> I am trying to migrate some old code that used to retrieve DocIdSets from
> filters, but with Filters being deprecated in Lucene 5.x I am trying to
> move away from those classes but I'm not sure the right way to do this
> now.  Are there any examples of doing this?
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Trouble Installing Solr 7.1.0 On Ubunti 17

[Dane Terrell]

same error
sudo solr-7.1.0/solr/bin/install_solr_service.sh /tmp/solr-7.1.0.tgz
is what I'm using. I downloaded and extracted the tar file in my tmp folder.
 Dane  

On Monday, October 23, 2017 11:42 PM, Yasufumi Mizoguchi 
 wrote:
 

 Hi,

Maybe, you have a wrong path. Try below.
$ sudo solr-7.1.0/bin/install_solr_service.sh 

Thanks,
Yasufumi.

2017-10-24 12:11 GMT+09:00 Dane Terrell :

> Hi I'm new to apache solr. I'm looking to install apache solr 7.1.0 on my
> localhost computer. I downloaded and extracted the tar file in my tmp
> folder. But when I try to run the script... sudo:
> solr-7.1.0/solr/bin/install_solr_service.sh: command not found
> or
> solr-7.1.0/solr/bin/install_solr_service.sh --strip-components=2
> I get the same error message. Can anyone help?
> Dane


   

Ways to weight SOLR search results without development

[Bryant, Eric]

Hi,

Is there a way to boost certain Drupal nodes to show up higher in SOLR search 
without having to do development of introduce conditional scripts? For example, 
if a person does a search for "valuation services", our SOLR instance right now 
returns articles sorted by frequency of keyword and density. But what if we 
want a node to show up higher but that doesn't have as high frequency or 
density for that keyword? We tried using the Keyword metadata field but that 
didn't seem to help. Any ideas?

Thanks,

Eric Bryant
Web & Digital Analytics Manager
Alvarez & Marsal
600 Madison Avenue, 5th Floor
New York, NY 10022
Direct: +1 212 328 8504
Mobile: +1 908 443 1665
www.alvarezandmarsal.com

LinkedIn  | 
Facebook | 
Twitter

This message is intended only for the use of the addressee(s) and may contain 
information that is PRIVILEGED and CONFIDENTIAL. If you are not the intended 
recipient(s), you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please erase all copies of the message and its attachments and notify 
us immediately.

SolrCloud not able to view cloud page - Loading of "/solr/zookeeper?wt=json" failed (HTTP-Status 500)

[Tarjono, C. A.]

Hi All,

Would like to check if anyone have seen this issue before, we started having 
this a few days ago:
[cid:image003.jpg@01D34D14.FC34F4D0]

The only error I can see in solr console is below:
5960847 [main-SendThread(172.16.130.132:2281)] WARN 
org.apache.zookeeper.ClientCnxn [ ] - Session 0x65f4e28b7370001 for server 
172.16.130.132/172.16.130.132:2281, unexpected error, closing socket connection 
and attempting reconnect java.io.IOException: Packet len30829010 is out of 
range! at 
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112) at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79) at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 
5960947 [zkCallback-2-thread-120] INFO 
org.apache.solr.common.cloud.ConnectionManager [ ] - Watcher 
org.apache.solr.common.cloud.ConnectionManager@4cf4d11e 
name:ZooKeeperConnection 
Watcher:172.16.129.132:2281,172.16.129.133:2281,172.16.129.134:2281,172.16.130.132:2281,172.16.130.133:2281,172.16.130.134:2281
 got event WatchedEvent state:Disconnected type:None path:null path:null 
type:None 5960947 [zkCallback-2-thread-120] INFO 
org.apache.solr.common.cloud.ConnectionManager [ ] - zkClient has disconnected

We cant find any corresponding error in zookeeper log.
Appreciate any input, thanks!

Best Regards,

Christopher Tarjono
Accenture Pte Ltd

+65 9347 2484
c.a.tarj...@accenture.com
[cid:image005.jpg@01D34D14.FC34F4D0]




This message is for the designated recipient only and may contain privileged, 
proprietary, or otherwise confidential information. If you have received it in 
error, please notify the sender immediately and delete the original. Any other 
use of the e-mail by you is prohibited. Where allowed by local law, electronic 
communications with Accenture and its affiliates, including e-mail and instant 
messaging (including content), may be scanned by our systems for the purposes 
of information security and assessment of internal compliance with Accenture 
policy.
__

www.accenture.com

Get Username within a plugin (Basic Authentication Plugin)

[startrekfan]

Hello,

I have set up an solr server with the "Basic Authentication Plugin". Now
I'd like to write a custom PostFilter(extends ExtendedQueryBase) plugin.
Can you explain me how I can get the username of the authenticated user
within my plugin?

Thanks

Solr-Edismax boost a property

[ruby]

I'm new to edismax query parser. Trying to figure out how to boost a specific
field over a copy fields.

I want to show all documents for which the *object_name* property contains
the search string first and then list all documents which has the value in
FTS_CopyField  and finally show the documents which have the value in
DTS_CopyField.

My search happens against two copyfields called FTS_CopyField and
DTS_CopyField. These two fields contain values from all properties/fields.

q=FTS_CopyField:oneapple OR DTS_CopyField:oneapple OR 

so far my search handler looks like:

   

edismax
 explicit
 10
 FTS_CopyField
 FTS_CopyField^200
 DTS_CopyField^10
 object_name^500   
 
 prop1:(best)^500  
 prop2:(candidate)^400  

 rating:(5)^500
recip(ms(NOW,last_mod_date),3.16e-11,1,1)^2
recip(ms(NOW,creation_date),3.16e-11,1,1)^1 
 AND
 fc
  
  


Is the obove the correct way of boosting object_name filed? 
Also how do I boost full match over partial match? if a document has
oneapple then I want to show it first and then other documents which have
words starting with oneapple.

Thank so much in advance.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Date range queries no longer work 6.6 to 7.1

[alessandro.benedetti]

I know it is obvious, but ...
 have you done a full re-indexing or you used the Index migration tool ?
In the latter case, it could be a bug of the tool itself.



-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr 6.6 - JSON Facet API - Unexpected counts returned for child focused facets

[Damien Hawes]

Suppose I have a set of documents that look likes this:

[
{
"id": "1",
"post_time": "2017-10-24T13:00:00Z",
"category": "information technology",
"url": "https://www.mywebsite.com";,
"blog_post": "solr",
"is_root_document": true,
"_childDocuments_": [{
"id": "comment-1",
"title": "awesome work",
"comment": "this was a great read",
"is_root_document": false
}]
},
{
"id": "2",
"post_time": "2017-10-24T13:00:00Z",
"url": "https://www.mywebsite.com";,
"category": "gardening",
"is_root_document": true,
"blog_post": "bob's garden",
"_childDocuments_": [{
"id": "comment-2",
"title": "green everywhere",
"comment": "where are the other colours?",
"is_root_document": false
}]
}
]

Now suppose I post the following query to Solr:

select?q={!parent which=is_root_document:true}&fq=category:"information
technology"&fl=*,[child parentFilter=is_root_document:true]
I get the following (expected) response:

[
{
"id": "1",
"post_time": "2017-10-24T13:00:00Z",
"category": "information technology",
"url": "https://www.mywebsite.com";,
"blog_post": "solr",
"is_root_document": true,
"_childDocuments_": [{
"id": "comment-1",
"title": "awesome work",
"comment": "this was a great read",
"is_root_document": false
}]
}
]

Now suppose I append the following json.facet argument to the above query:

select?q={!parent which=is_root_document:true}&fq=category:"information
technology"&fl=*,[child
parentFilter=is_root_document:true]&json.facet={"comment_titles": {"type":
"terms", "field": "title", "domain": { "blockChildren":
"category:\"information technology\"" }}}

The facet response that is returned to me is:

"facets": {
"count": 1,
"comment_titles": {
"buckets": [
{
"val": "awesome work",
"count": 1
},
{
"val": "green everywhere",
"count": 1
}
]
}
}

My question is: Why is Solr returning the "green everywhere" value, when
I’ve explicitly filtered it out from the document set with the
'fq=category:"information technology"' filter and domain statements? What
changes do I need to make to get my desired behaviour?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr-Edismax boost a property

[ruby]

I'm new to edismax query parser. Trying to figure out how to boost a specific
field over a copy fields.

I want to show all documents for which the *object_name* property contains
the search string first and then list all documents which has the value in
FTS_CopyField  and finally show the documents which have the value in
DTS_CopyField.

My search happens against two copyfields called FTS_CopyField and
DTS_CopyField. These two fields contain values from all properties/fields.

q=FTS_CopyField:oneapple OR DTS_CopyField:oneapple OR 

so far my search handler looks like:

   

edismax
 explicit
 10
 FTS_CopyField
 FTS_CopyField^200
 DTS_CopyField^10
 object_name^500   
 
 prop1:(best)^500  
 prop2:(candidate)^400  

 rating:(5)^500
recip(ms(NOW,last_mod_date),3.16e-11,1,1)^2
recip(ms(NOW,creation_date),3.16e-11,1,1)^1 
 AND
 fc
  
  


Is the obove the correct way of boosting object_name filed? 
Also how do I boost full match over partial match? if a document has
oneapple then I want to show it first and then other documents which have
words starting with oneapple.

Thank so much in advance.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: Date range queries no longer work 6.6 to 7.1

[Markus Jelsma]

No, we have not reindexed, we don't like to reindex one of our collections, it 
is huge! 

Didn't know about a index migration tool, i have not seen it in the changelog.

Thanks,
Markus
 
-Original message-
> From:alessandro.benedetti 
> Sent: Tuesday 24th October 2017 17:03
> To: solr-user@lucene.apache.org
> Subject: Re: Date range queries no longer work 6.6 to 7.1
> 
> I know it is obvious, but ...
>  have you done a full re-indexing or you used the Index migration tool ?
> In the latter case, it could be a bug of the tool itself.
> 
> 
> 
> -
> ---
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> 

Re: Replacing legacyCloud Behaviour in Solr7

[Erick Erickson]

Yes, it looks like you're on track, good luck!

On Mon, Oct 23, 2017 at 5:21 PM, Marko Babic  wrote:
> Thanks for the quick reply, Erick.
>
> To follow up:
>
> “
> Well, first you can explicitly set legacyCloud=true by using the
> Collections API CLUSTERPROP command. I don't recommend this, mind you,
> as legacyCloud will not be supported forever.
> “
>
> Yes, but like you say: we’ll have to deal with at some point, not much 
> benefit in punting.
>
> “
> I'm not following something here though. When you say:
> "The desired final state of a such a deployment is a fully configured
> cluster ready to accept updates."
> are there any documents already in the index or is this really a new 
> collection?
> “
>
> It’s a brand new collection with new configuration on fresh hardware which 
> we’ll then fully index from a source document store (we do this when we have 
> certain schema changes that require re-indexing or we want to experiment).
>
> “
> Not sure what you mean here. Configuration of what?  Just spinning up
> a Solr node pointing to the right ZooKeeper should be sufficient, or
> I'm not understanding at all.
> “
>
> Apologies, the way I stated that was all wrong: by “requires configuration” I 
> just meant to note the need to specify a shard and a node when adding a 
> replica (and not even the node as you point out to me below ☺).
>
> “
> I suspect you're really talking about the "node" parameter
> to ADDREPLCIA
> “
>
> Ah, yes: that is what I meant, sorry.
>
> It sounds like I haven’t missed too much in the documentation then, I’ll look 
> more into replica placement rules.
>
> Thank you so much again for your time and help.
>
> Marko
>
>
> On 10/23/17, 4:33 PM, "Erick Erickson"  wrote:
>
> Well, first you can explicitly set legacyCloud=true by using the
> Collections API CLUSTERPROP command. I don't recommend this, mind you,
> as legacyCloud will not be supported forever.
>
> I'm not following something here though. When you say:
> "The desired final state of a such a deployment is a fully configured
> cluster ready to accept updates."
> are there any documents already in the index or is this really a new 
> collection?
>
> and "adding new nodes requires explicit configuration"
>
> Not sure what you mean here. Configuration of what?  Just spinning up
> a Solr node pointing to the right ZooKeeper should be sufficient, or
> I'm not understanding at all.
>
> If not, your proposed outline seems right with one difference:
> "if a node needs to be added: provision a machine, start up Solr, use
> ADDREPLICA from Collections API passing shard number and coreNodeName"
>
> coreNodeName isn't something you ordinarily need to bother with. I'm
> being specific here where coreNodeName is usually something like
> core_node7. I suspect you're really talking about the "node" parameter
> to ADDREPLCIA, something like: 192.168.1.32:8983_solr, the entry from
> live_nodes.
>
> Now, all that said you may be better off just letting Solr add the
> replica where it wants, it'll usually put a new replica on a node
> without replicas so specifying the collection and shard should be
> sufficient. Also, note that there are replica placement rules that can
> help enforce this kind of thing.
>
> Best,
> Erick
>
> On Mon, Oct 23, 2017 at 3:12 PM, Marko Babic  
> wrote:
> > Hi everyone,
> >
> > I'm working on upgrading a set of clusters from Solr 4.10.4 to Solr 
> 7.1.0.
> >
> > Our deployment tooling no longer works given that legacyCloud defaults 
> to false (SOLR-8256) and I'm hoping to get some advice on what to do going 
> forward.
> >
> > Our setup is as follows:
> >   * we run in AWS with multiple independent Solr clusters, each with 
> its own Zookeeper tier
> >   * each cluster hosts only a single collection
> >   * each machine/node in the cluster has a single core / is a replica 
> for one shard in the collection
> >
> > We bring up new clusters as needed.  This is entirely automated and 
> basically works as follows:
> >   * we first provision and set up a fresh Zookeeper tier
> >   * then, we provision a Solr bootstrapper machine that uploads 
> collection config, specifies numShards and starts up
> >   * it's then easy provision the rest of the machines and have them 
> automatically join a shard in the collection by hooking them to the right 
> Zookeeper cluster and specifying numShards
> >   * if a node needs to be added to the cluster we just need to spin a 
> machine up and start up Solr
> >
> > The desired final state of a such a deployment is a fully configured 
> cluster ready to accept updates.
> >
> > Now that legacyCloud is false I'm not sure how to preserve this pretty 
> nice, hands-off deployment style as the bootstrapping performed by the first 
> node provisioned doesn't create a co

Re: Date range queries no longer work 6.6 to 7.1

[Erick Erickson]

Assuming you haven't indexed new data, you can switch the field
definitions back to the old types.

I'm not totally sure the indexupgrade tool will fix this up, it'll do
an optimize though which isn't always a good thing going forward.

Best,
Erick

On Tue, Oct 24, 2017 at 8:25 AM, Markus Jelsma
 wrote:
> No, we have not reindexed, we don't like to reindex one of our collections, 
> it is huge!
>
> Didn't know about a index migration tool, i have not seen it in the changelog.
>
> Thanks,
> Markus
>
> -Original message-
>> From:alessandro.benedetti 
>> Sent: Tuesday 24th October 2017 17:03
>> To: solr-user@lucene.apache.org
>> Subject: Re: Date range queries no longer work 6.6 to 7.1
>>
>> I know it is obvious, but ...
>>  have you done a full re-indexing or you used the Index migration tool ?
>> In the latter case, it could be a bug of the tool itself.
>>
>>
>>
>> -
>> ---
>> Alessandro Benedetti
>> Search Consultant, R&D Software Engineer, Director
>> Sease Ltd. - www.sease.io
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>

Re: SolrCloud not able to view cloud page - Loading of "/solr/zookeeper?wt=json" failed (HTTP-Status 500)

[Erick Erickson]

The mail server aggressively removes attachments and the like, you'll have
to put it somewhere and provide a link.

Did anything change in that time frame?

Best,
Erick

On Tue, Oct 24, 2017 at 7:11 AM, Tarjono, C. A. 
wrote:

> Hi All,
>
>
>
> Would like to check if anyone have seen this issue before, we started
> having this a few days ago:
>
>
>
> The only error I can see in solr console is below:
>
> 5960847 [main-SendThread(172.16.130.132:2281)] WARN org.apache.zookeeper.
> ClientCnxn [ ] – Session 0x65f4e28b7370001 for server
> 172.16.130.132/172.16.130.132:2281, unexpected error, closing socket
> connection and attempting reconnect java.io.IOException: Packet
> len30829010 is out of range! at org.apache.zookeeper.
> ClientCnxnSocket.readLength(ClientCnxnSocket.java:112) at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNI
> O.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(Clie
> ntCnxn.java:1081) 5960947 [zkCallback-2-thread-120] INFO
> org.apache.solr.common.cloud.ConnectionManager [ ] – Watcher
> org.apache.solr.common.cloud.ConnectionManager@4cf4d11e
> name:ZooKeeperConnection Watcher:172.16.129.132:2281,172.16.129.133:2281,
> 172.16.129.134:2281,172.16.130.132:2281,172.16.130.133:2281,172.16.130.
> 134:2281 got event WatchedEvent state:Disconnected type:None path:null
> path:null type:None 5960947 [zkCallback-2-thread-120] INFO
> org.apache.solr.common.cloud.ConnectionManager [ ] – zkClient has
> disconnected
>
>
>
> We cant find any corresponding error in zookeeper log.
>
> Appreciate any input, thanks!
>
>
>
> Best Regards,
>
>
>
> Christopher Tarjono
>
> *Accenture Pte Ltd*
>
>
>
> +65 9347 2484 <+65%209347%202484>
>
> c.a.tarj...@accenture.com
>
>
>
> --
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
> 
> __
>
> www.accenture.com
>

[ANNOUNCE] Apache Solr 5.5.5 released

[Steve Rowe]

24 October 2017, Apache Solr™ 5.5.5 available 

The Lucene PMC is pleased to announce the release of Apache Solr 5.5.5. 

Solr is the popular, blazing fast, open source NoSQL search platform from the 
Apache Lucene project. Its major features include powerful full-text search, 
hit highlighting, faceted search and analytics, rich document parsing, 
geospatial search, extensive REST APIs as well as parallel SQL. Solr is 
enterprise grade, secure and highly scalable, providing fault tolerant 
distributed search and indexing, and powers the search and navigation features 
of many of the world's largest internet sites. 

This release contains one bugfix. 

This release includes one critical and one important security fix. Details: 

* Fix for a 0-day exploit (CVE-2017-12629), details: https://s.apache.org/FJDl. 
RunExecutableListener has been disabled by default (can be enabled by 
-Dsolr.enableRunExecutableListener=true) and resolving external entities in the 
XML query parser (defType=xmlparser or {!xmlparser ... }) is disabled by 
default. 

* Fix for CVE-2017-7660: Security Vulnerability in secure inter-node 
communication 
in Apache Solr, details: https://s.apache.org/APTY 

Furthermore, this release includes Apache Lucene 5.5.5 which includes one 
security 
fix since the 5.5.4 release. 

The release is available for immediate download at: 

http://www.apache.org/dyn/closer.lua/lucene/solr/5.5.5 

Please read CHANGES.txt for a detailed list of changes: 

https://lucene.apache.org/solr/5_5_5/changes/Changes.html 

Please report any feedback to the mailing lists 
(http://lucene.apache.org/solr/discussion.html) 

Note: The Apache Software Foundation uses an extensive mirroring 
network for distributing releases. It is possible that the mirror you 
are using may not have replicated the release yet. If that is the 
case, please try another mirror. This also goes for Maven access.

RE: Date range queries no longer work 6.6 to 7.1

[Markus Jelsma]

We have switched back to 6.6 for now so we are fine for now. Although i didn't 
try range queries other than date, i assume other Point fields can also have 
this problem?

That would mean completely switch back to Trie if you don't can/want to fully 
reindex all data.

Suggestions? A forceMerge at least fixes nothing. I'll take a look at index 
upgrade tool.

Thanks,
Markus

-Original message-
> From:Erick Erickson 
> Sent: Tuesday 24th October 2017 17:30
> To: solr-user 
> Subject: Re: Date range queries no longer work 6.6 to 7.1
> 
> Assuming you haven't indexed new data, you can switch the field
> definitions back to the old types.
> 
> I'm not totally sure the indexupgrade tool will fix this up, it'll do
> an optimize though which isn't always a good thing going forward.
> 
> Best,
> Erick
> 
> On Tue, Oct 24, 2017 at 8:25 AM, Markus Jelsma
>  wrote:
> > No, we have not reindexed, we don't like to reindex one of our collections, 
> > it is huge!
> >
> > Didn't know about a index migration tool, i have not seen it in the 
> > changelog.
> >
> > Thanks,
> > Markus
> >
> > -Original message-
> >> From:alessandro.benedetti 
> >> Sent: Tuesday 24th October 2017 17:03
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Date range queries no longer work 6.6 to 7.1
> >>
> >> I know it is obvious, but ...
> >>  have you done a full re-indexing or you used the Index migration tool ?
> >> In the latter case, it could be a bug of the tool itself.
> >>
> >>
> >>
> >> -
> >> ---
> >> Alessandro Benedetti
> >> Search Consultant, R&D Software Engineer, Director
> >> Sease Ltd. - www.sease.io
> >> --
> >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >>
> 

Re: Facets based on sampling

[Erick Erickson]

bq:  It is a bit surprising why facet computation
 is so slow even when the query matches hundreds of docs.

The number of terms in the field over all docs also comes into play.
Say you're faceting over a field that has 100,000,000 unique values
across all docs, that's a lot of bookkeeping.

Best,
Erick


On Tue, Oct 24, 2017 at 1:08 AM, Emir Arnautović
 wrote:
> Hi John,
> Did you mean “docValues don’t work for analysed fields” since it works for 
> multivalue string (or other supported types) fields. What you need to do is 
> to convert your analysed field to multivalue string field - that requires 
> changes in indexing flow.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 23 Oct 2017, at 21:08, John Davis  wrote:
>>
>> Docvalues don't work for multivalued fields. I just started a separate
>> thread with more debug info. It is a bit surprising why facet computation
>> is so slow even when the query matches hundreds of docs.
>>
>> On Mon, Oct 23, 2017 at 6:53 AM, alessandro.benedetti 
>> wrote:
>>
>>> Hi John,
>>> first of all, I may state the obvious, but have you tried docValues ?
>>>
>>> Apart from that a friend of mine ( Diego Ceccarelli) was discussing a
>>> probabilistic implementation similar to the hyperloglog[1] to approximate
>>> facets counting.
>>> I didn't have time to take a look in details / implement anything yet.
>>> But it is on our To Do list :)
>>> He may add some info here.
>>>
>>> Cheers
>>>
>>>
>>>
>>>
>>> [1]
>>> https://blog.yld.io/2017/04/19/hyperloglog-a-probabilistic-data-structure/
>>>
>>>
>>>
>>> -
>>> ---
>>> Alessandro Benedetti
>>> Search Consultant, R&D Software Engineer, Director
>>> Sease Ltd. - www.sease.io
>>> --
>>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>>
>

Re: Ways to weight SOLR search results without development

[Charlie Hull]

On 24/10/2017 14:40, Bryant, Eric wrote:

Hi,

Is there a way to boost certain Drupal nodes to show up higher in SOLR search without 
having to do development of introduce conditional scripts? For example, if a person does 
a search for "valuation services", our SOLR instance right now returns articles 
sorted by frequency of keyword and density. But what if we want a node to show up higher 
but that doesn't have as high frequency or density for that keyword? We tried using the 
Keyword metadata field but that didn't seem to help. Any ideas?


Hi Eric,

How/if you can do this is going to depend on which Drupal plugins you're 
using to talk to Solr, and how they've been configured. We've seen 
several Drupal/Solr installations where boosting has been applied in an 
inconsistent way - our approach has usually been to turn it all off and 
start again.


Cheers

Charlie




Thanks,

Eric Bryant
Web & Digital Analytics Manager
Alvarez & Marsal
600 Madison Avenue, 5th Floor
New York, NY 10022
Direct: +1 212 328 8504
Mobile: +1 908 443 1665
www.alvarezandmarsal.com

LinkedIn  | 
Facebook | 
Twitter

This message is intended only for the use of the addressee(s) and may contain 
information that is PRIVILEGED and CONFIDENTIAL. If you are not the intended 
recipient(s), you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please erase all copies of the message and its attachments and notify 
us immediately.




--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

RE: LTR feature extraction performance issues

[Brian Yee]

Hi Alessandro,

Unfortunately some of my most important features are query dependent. I think I 
found an issue though. I don't think my features are being inserted into the 
cache. Notice "cumulative_inserts:0". There are a lot of lookups, but since 
there appear to be no values in the cache, the hitratio is 0.

stats:
cumulative_evictions:0
cumulative_hitratio:0
cumulative_hits:0
cumulative_inserts:0
cumulative_lookups:215319
evictions:0
hitratio:0
hits:0
inserts:0
lookups:3303
size:0
warmupTime:0


My configs look are as follows:



  

  
QUERY_DOC_FV
sparse
  

Would anyone have any idea why my features are not being inserted into the 
cache? Is there an additional config setting I need?


--Brian

-Original Message-
From: alessandro.benedetti [mailto:a.benede...@sease.io] 
Sent: Monday, October 23, 2017 10:01 AM
To: solr-user@lucene.apache.org
Subject: Re: LTR feature extraction performance issues

It strictly depends on the kind of features you are using.
At the moment there is just one cache for all the features.
This means that even if you have 1 query dependent feature and 100 document 
dependent feature, a different value for the query dependent one will 
invalidate the cache entry for the full vector[1].

You may look to optimise your features ( where possible).

[1]  https://issues.apache.org/jira/browse/SOLR-10448



-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

NullPointer on Suggest Component running with multiple shards

[mvogelsmeier]

Hello everyone,

I got a NullPointer-Exception running a select-SearchHandler with a suggest 
component on multiple shards (cloud mode).
It does work in the same configuration with a single shard (or non-cloud setup).
The suggest requestHandler does work in both setups.

Exception:

  true
  500
  16
  
*:*
edismax
_text_
all
on
_text_
xml
1508861012736
  


  java.lang.NullPointerException
at 
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1224)
at 
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1075)
at 
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:775)
at 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:754)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:429)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

Multi-Shard-Setup (Solr 6.6.2):
1 Collection with 2 Shards (each ~29k documents)

Single-Shard-Setup (Solr 6.6.2):
1 Collection with 1 Shard (58k documents)


Here is my configoverlay.json (everything else is default data-driven)
{
  "userProps":{"update.autoCreateFields":"false"},
  "searchComponent":{"suggest":{
  "name":"suggest",
  "class":"solr.SuggestComponent",
  "suggester":{
"name":"mySuggester",
"lookupImpl":"FuzzyLookupFactory",
"dictionaryImpl":"DocumentDictionaryFactory",
"field":"id",
"suggestAnalyzerFieldType":"string",
"buildOnStartup":"true"}}},
  "requestHandler":{
"/select":{
  "name":"/select",
  "class":"solr.SearchHandler",
  "defaults":{
"defType":"edismax",
"qf":"_text_"},
  "components":[
"query",
"facet",
"suggest",
"debug"]},
"/suggest":{
  "name":"/suggest",
  "class":"solr.SearchHandler",
  "defaults":{
"suggest":"true",
"suggest.dictionary":"mySuggester",
"suggest.count":5},
  "compon

Re: [ANNOUNCE] Apache Solr 5.5.5 released

[Steve Rowe]

Yes.  

--
Steve
www.lucidworks.com

> On Oct 24, 2017, at 12:25 PM, Moenieb Davids  wrote:
> 
> Solr 5.5.5?
> 
> On 24 Oct 2017 17:34, "Steve Rowe"  wrote:
> 
>> 24 October 2017, Apache Solr™ 5.5.5 available
>> 
>> The Lucene PMC is pleased to announce the release of Apache Solr 5.5.5.
>> 
>> Solr is the popular, blazing fast, open source NoSQL search platform from
>> the
>> Apache Lucene project. Its major features include powerful full-text
>> search,
>> hit highlighting, faceted search and analytics, rich document parsing,
>> geospatial search, extensive REST APIs as well as parallel SQL. Solr is
>> enterprise grade, secure and highly scalable, providing fault tolerant
>> distributed search and indexing, and powers the search and navigation
>> features
>> of many of the world's largest internet sites.
>> 
>> This release contains one bugfix.
>> 
>> This release includes one critical and one important security fix. Details:
>> 
>> * Fix for a 0-day exploit (CVE-2017-12629), details:
>> https://s.apache.org/FJDl.
>> RunExecutableListener has been disabled by default (can be enabled by
>> -Dsolr.enableRunExecutableListener=true) and resolving external entities
>> in the
>> XML query parser (defType=xmlparser or {!xmlparser ... }) is disabled by
>> default.
>> 
>> * Fix for CVE-2017-7660: Security Vulnerability in secure inter-node
>> communication
>> in Apache Solr, details: https://s.apache.org/APTY
>> 
>> Furthermore, this release includes Apache Lucene 5.5.5 which includes one
>> security
>> fix since the 5.5.4 release.
>> 
>> The release is available for immediate download at:
>> 
>> http://www.apache.org/dyn/closer.lua/lucene/solr/5.5.5
>> 
>> Please read CHANGES.txt for a detailed list of changes:
>> 
>> https://lucene.apache.org/solr/5_5_5/changes/Changes.html
>> 
>> Please report any feedback to the mailing lists
>> (http://lucene.apache.org/solr/discussion.html)
>> 
>> Note: The Apache Software Foundation uses an extensive mirroring
>> network for distributing releases. It is possible that the mirror you
>> are using may not have replicated the release yet. If that is the
>> case, please try another mirror. This also goes for Maven access.

Re: Trouble Installing Solr 7.1.0 On Ubunti 17

[Shawn Heisey]

On 10/23/2017 9:11 PM, Dane Terrell wrote:
> Hi I'm new to apache solr. I'm looking to install apache solr 7.1.0 on my 
> localhost computer. I downloaded and extracted the tar file in my tmp folder. 
> But when I try to run the script... sudo: 
> solr-7.1.0/solr/bin/install_solr_service.sh: command not found
> or
> solr-7.1.0/solr/bin/install_solr_service.sh --strip-components=2
> I get the same error message. Can anyone help?

It looks like install_solr_service.sh is not executable.

I created a file named 'fff' in my current directory, with this content:

#!/bin/sh
echo yay

Then I proceeded to try to run it with sudo.  It gave the same message
you got.  Then I made it executable, tried it again, and it worked:

root@smeagol:~# sudo ./fff
sudo: ./fff: command not found
root@smeagol:~# chmod +x fff
root@smeagol:~# sudo ./fff
yay

You have two choices to fix this problem.  You can make the script
executable, or you can add "bash" right after sudo and before the script
path.

Thanks,
Shawn

Re: NullPointer on Suggest Component running with multiple shards

[Janaka Thilakarathna]

unsubscribe

On Tue, Oct 24, 2017 at 9:48 PM,  wrote:

> Hello everyone,
>
> I got a NullPointer-Exception running a select-SearchHandler with a
> suggest component on multiple shards (cloud mode).
> It does work in the same configuration with a single shard (or non-cloud
> setup).
> The suggest requestHandler does work in both setups.
>
> Exception:
> 
>   true
>   500
>   16
>   
> *:*
> edismax
> _text_
> all
> on
> _text_
> xml
> 1508861012736
>   
> 
> 
>   java.lang.NullPointerException
> at org.apache.solr.handler.component.QueryComponent.
> unmarshalSortValues(QueryComponent.java:1224)
> at org.apache.solr.handler.component.QueryComponent.
> mergeIds(QueryComponent.java:1075)
> at org.apache.solr.handler.component.QueryComponent.
> handleRegularResponses(QueryComponent.java:775)
> at org.apache.solr.handler.component.QueryComponent.
> handleResponses(QueryComponent.java:754)
> at org.apache.solr.handler.component.SearchHandler.
> handleRequestBody(SearchHandler.java:429)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:529)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:361)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:305)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)
> at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
> at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> RewriteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:320)
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)
> at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:589)
>
> Multi-Shard-Setup (Solr 6.6.2):
> 1 Collection with 2 Shards (each ~29k documents)
>
> Single-Shard-Setup (Solr 6.6.2):
> 1 Collection with 1 Shard (58k documents)
>
>
> Here is my configoverlay.json (everything else is default data-driven)
> {
>   "userProps":{"update.autoCreateFields":"false"},
>   "searchComponent":{"suggest":{
>   "name":"suggest",
>   "class":"solr.SuggestComponent",
>   "suggester":{
> "name":"mySuggester",
> "lookupImpl":"FuzzyLookupFactory",
> "dictionaryImpl":"DocumentDictionaryFactory",
> "field":"id",
> "suggestAnalyzerFieldType":"string",
> "buildOnStartup":"true"}}},
>   "requestHandler":{
> "/select":{
>   "name":"/select",
>   "class":"solr.SearchHandler",
>   "defaults":{
> "defType":"edismax",
> "qf":"_text_"

Re: Date range queries no longer work 6.6 to 7.1

[Shawn Heisey]

On 10/24/2017 9:25 AM, Markus Jelsma wrote:
> No, we have not reindexed, we don't like to reindex one of our collections, 
> it is huge! 
>
> Didn't know about a index migration tool, i have not seen it in the changelog.

If you change the class on a field (in this case, probably Trie to
Point), then you must completely reindex.  Each of Solr's field classes
can only read data that was written by the same class.

I am not aware of any index migration tool that would be able to prevent
the need for a reindex.  Such a tool would be really cool, but probably
wouldn't be trivial to write.

Thanks,
Shawn

Re: Date range queries no longer work 6.6 to 7.1

[Shawn Heisey]

On 10/24/2017 9:38 AM, Markus Jelsma wrote:
> We have switched back to 6.6 for now so we are fine for now. Although i 
> didn't try range queries other than date, i assume other Point fields can 
> also have this problem?
>
> That would mean completely switch back to Trie if you don't can/want to fully 
> reindex all data.
>
> Suggestions? A forceMerge at least fixes nothing. I'll take a look at index 
> upgrade tool.

As I said in the earlier reply, data written by a Trie field class
cannot be read by a Point field class.  That's true for any of them --
Int, Float, Double, Long, etc.

Lucene's IndexUpgrader just performs a forceMerge on the index.  There's
nothing special about the job it does.  It is not capable of converting
one field class to another.  It doesn't know anything about Solr's field
classes.

Solr 7.x can still use Trie fields, but they will be gone by the 8.0
release.  Lucene 7.0 no longer contains the legacy numeric classes that
Trie fields are built with.  Solr has kept those around for one more
major version.

Thanks,
Shawn

Re: NullPointer on Suggest Component running with multiple shards

[Erick Erickson]

Please follow the instructions here to unsubscrie:
http://lucene.apache.org/solr/community.html#mailing-lists-irc. You
must use the _exact_ same e-mail as you used to subscribe.

If the initial try doesn't work and following the suggestions at the
"problems" link doesn't work for you, let us know. But note you need
to show us the _entire_ return header to allow anyone to diagnose the
problem.


Best,

Erick

On Tue, Oct 24, 2017 at 9:44 AM, Janaka Thilakarathna
 wrote:
> unsubscribe
>
> On Tue, Oct 24, 2017 at 9:48 PM,  wrote:
>
>> Hello everyone,
>>
>> I got a NullPointer-Exception running a select-SearchHandler with a
>> suggest component on multiple shards (cloud mode).
>> It does work in the same configuration with a single shard (or non-cloud
>> setup).
>> The suggest requestHandler does work in both setups.
>>
>> Exception:
>> 
>>   true
>>   500
>>   16
>>   
>> *:*
>> edismax
>> _text_
>> all
>> on
>> _text_
>> xml
>> 1508861012736
>>   
>> 
>> 
>>   java.lang.NullPointerException
>> at org.apache.solr.handler.component.QueryComponent.
>> unmarshalSortValues(QueryComponent.java:1224)
>> at org.apache.solr.handler.component.QueryComponent.
>> mergeIds(QueryComponent.java:1075)
>> at org.apache.solr.handler.component.QueryComponent.
>> handleRegularResponses(QueryComponent.java:775)
>> at org.apache.solr.handler.component.QueryComponent.
>> handleResponses(QueryComponent.java:754)
>> at org.apache.solr.handler.component.SearchHandler.
>> handleRequestBody(SearchHandler.java:429)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(
>> RequestHandlerBase.java:173)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>> at org.apache.solr.servlet.HttpSolrCall.execute(
>> HttpSolrCall.java:723)
>> at org.apache.solr.servlet.HttpSolrCall.call(
>> HttpSolrCall.java:529)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:361)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:305)
>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1691)
>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
>> ServletHandler.java:582)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:143)
>> at org.eclipse.jetty.security.SecurityHandler.handle(
>> SecurityHandler.java:548)
>> at org.eclipse.jetty.server.session.SessionHandler.
>> doHandle(SessionHandler.java:226)
>> at org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1180)
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(
>> ServletHandler.java:512)
>> at org.eclipse.jetty.server.session.SessionHandler.
>> doScope(SessionHandler.java:185)
>> at org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1112)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:141)
>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.
>> handle(ContextHandlerCollection.java:213)
>> at org.eclipse.jetty.server.handler.HandlerCollection.
>> handle(HandlerCollection.java:119)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:134)
>> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
>> RewriteHandler.java:335)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:134)
>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> at org.eclipse.jetty.server.HttpChannel.handle(
>> HttpChannel.java:320)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:251)
>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
>> AbstractConnection.java:273)
>> at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:95)
>> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
>> SelectChannelEndPoint.java:93)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> executeProduceConsume(ExecuteProduceConsume.java:303)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> produceConsume(ExecuteProduceConsume.java:148)
>> at org.eclipse.jetty.util.thread.strategy.
>> ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:671)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:589)
>>
>> Multi-Shard-Setup (Solr 6.6.2):
>> 1 Collection with 2 Shards (each ~29k documents)
>>
>> Single-Shard-Setup (Solr 6.6.2):
>> 1 Collection with 1 Shard (58k documents)
>>
>>
>> Here is my configoverlay.json (everything else is default data-driven)
>> {
>>   "userP

Re: [ANNOUNCE] Apache Solr 5.5.5 released

[Moenieb Davids]

Solr 5.5.5?

On 24 Oct 2017 17:34, "Steve Rowe"  wrote:

> 24 October 2017, Apache Solr™ 5.5.5 available
>
> The Lucene PMC is pleased to announce the release of Apache Solr 5.5.5.
>
> Solr is the popular, blazing fast, open source NoSQL search platform from
> the
> Apache Lucene project. Its major features include powerful full-text
> search,
> hit highlighting, faceted search and analytics, rich document parsing,
> geospatial search, extensive REST APIs as well as parallel SQL. Solr is
> enterprise grade, secure and highly scalable, providing fault tolerant
> distributed search and indexing, and powers the search and navigation
> features
> of many of the world's largest internet sites.
>
> This release contains one bugfix.
>
> This release includes one critical and one important security fix. Details:
>
> * Fix for a 0-day exploit (CVE-2017-12629), details:
> https://s.apache.org/FJDl.
> RunExecutableListener has been disabled by default (can be enabled by
> -Dsolr.enableRunExecutableListener=true) and resolving external entities
> in the
> XML query parser (defType=xmlparser or {!xmlparser ... }) is disabled by
> default.
>
> * Fix for CVE-2017-7660: Security Vulnerability in secure inter-node
> communication
> in Apache Solr, details: https://s.apache.org/APTY
>
> Furthermore, this release includes Apache Lucene 5.5.5 which includes one
> security
> fix since the 5.5.4 release.
>
> The release is available for immediate download at:
>
> http://www.apache.org/dyn/closer.lua/lucene/solr/5.5.5
>
> Please read CHANGES.txt for a detailed list of changes:
>
> https://lucene.apache.org/solr/5_5_5/changes/Changes.html
>
> Please report any feedback to the mailing lists
> (http://lucene.apache.org/solr/discussion.html)
>
> Note: The Apache Software Foundation uses an extensive mirroring
> network for distributing releases. It is possible that the mirror you
> are using may not have replicated the release yet. If that is the
> case, please try another mirror. This also goes for Maven access.

Re: Really slow facet performance in 6.6

[Günter Hipler]

have a look for more background information:

https://issues.apache.org/jira/browse/SOLR-8096

it's not only related to version 6.6. It's a question of design since 5.x

Günter


On 23.10.2017 21:06, John Davis wrote:

Hello,

We are seeing really slow facet performance with new solr release. This is
on an index of 2M documents. A few things we've tried:

1. method=uif however that didn't help much (the facet fields have
docValues=false since they are multi-valued). Debug info below.

2. changing query (q=) that selects what documents to compute facets on
didn't help a lot, except repeating the same query was fast presumably due
to exact cache hits.

Sample debug info:

“timing”: {
 “prepare”: {
 “debug”: {
 “time”: 0.0
 },
 “expand”: {
 “time”: 0.0
 },
 “facet”: {
 “time”: 0.0
 },
 “facet_module”: {
 “time”: 0.0
 },
 “highlight”: {
 “time”: 0.0
 },
 “mlt”: {
 “time”: 0.0
 },
 “query”: {
 “time”: 0.0
 },
 “stats”: {
 “time”: 0.0
 },
 “terms”: {
 “time”: 0.0
 },
 “time”: 0.0
 },
 “process”: {
 “debug”: {
 “time”: 87.0
 },
 “expand”: {
 “time”: 0.0
 },
 “facet”: {
 “time”: 9814.0
 },
 “facet_module”: {
 “time”: 0.0
 },
 “highlight”: {
 “time”: 0.0
 },
 “mlt”: {
 “time”: 0.0
 },
 “query”: {
 “time”: 20.0
 },
 “stats”: {
 “time”: 0.0
 },
 “terms”: {
 “time”: 0.0
 },
 “time”: 9922.0
 },
 “time”: 9923.0
 }
 },

"facet-debug": {
 "elapse": 8310,
 "sub-facet": [
 {
 "action": "field facet",
 "elapse": 8310,
 "maxThreads": 2,
 "processor": "SimpleFacets",
 "sub-facet": [
 {},
 {
 "appliedMethod": "UIF",
 "field": "school",
 "inputDocSetSize": 476,
 "requestedMethod": "UIF"
 },
 {
 "appliedMethod": "UIF",
 "elapse": 2575,
 "field": "work",
 "inputDocSetSize": 476,
 "requestedMethod": "UIF"
 },
 {
 "appliedMethod": "UIF",
 "elapse": 8310,
 "field": "level",
 "inputDocSetSize": 476,
 "requestedMethod": "UIF"
 }
 ]
 }

Thanks
John



--
Günter Hipler

Universität Basel | Universitätsbibliothek | Projekt swissbib

Schönbeinstrasse 18-20 | 4056 Basel | Schweiz

Tel +41 61 207 31 12 | Fax +41 61 207 31 03

E-Mail guenter.hip...@unibas.ch | http://www.ub.unibas.ch | 
https://www.swissbib.ch

Re: [ANNOUNCE] Apache Solr 5.5.5 released

[Moenieb Davids]

Hi Steve,

I have just started with Solr 7.*, so I am a bit confused with 5.5.5, same
with lucene.
Also, the sites register versions 7.*,
Apologies for my ignorance if I had missed anything or do not have a proper
understanding of the version management

Regards
Moenieb

On Tue, Oct 24, 2017 at 6:27 PM, Steve Rowe  wrote:

> Yes.
>
> --
> Steve
> www.lucidworks.com
>
> > On Oct 24, 2017, at 12:25 PM, Moenieb Davids 
> wrote:
> >
> > Solr 5.5.5?
> >
> > On 24 Oct 2017 17:34, "Steve Rowe"  wrote:
> >
> >> 24 October 2017, Apache Solr™ 5.5.5 available
> >>
> >> The Lucene PMC is pleased to announce the release of Apache Solr 5.5.5.
> >>
> >> Solr is the popular, blazing fast, open source NoSQL search platform
> from
> >> the
> >> Apache Lucene project. Its major features include powerful full-text
> >> search,
> >> hit highlighting, faceted search and analytics, rich document parsing,
> >> geospatial search, extensive REST APIs as well as parallel SQL. Solr is
> >> enterprise grade, secure and highly scalable, providing fault tolerant
> >> distributed search and indexing, and powers the search and navigation
> >> features
> >> of many of the world's largest internet sites.
> >>
> >> This release contains one bugfix.
> >>
> >> This release includes one critical and one important security fix.
> Details:
> >>
> >> * Fix for a 0-day exploit (CVE-2017-12629), details:
> >> https://s.apache.org/FJDl.
> >> RunExecutableListener has been disabled by default (can be enabled by
> >> -Dsolr.enableRunExecutableListener=true) and resolving external
> entities
> >> in the
> >> XML query parser (defType=xmlparser or {!xmlparser ... }) is disabled by
> >> default.
> >>
> >> * Fix for CVE-2017-7660: Security Vulnerability in secure inter-node
> >> communication
> >> in Apache Solr, details: https://s.apache.org/APTY
> >>
> >> Furthermore, this release includes Apache Lucene 5.5.5 which includes
> one
> >> security
> >> fix since the 5.5.4 release.
> >>
> >> The release is available for immediate download at:
> >>
> >> http://www.apache.org/dyn/closer.lua/lucene/solr/5.5.5
> >>
> >> Please read CHANGES.txt for a detailed list of changes:
> >>
> >> https://lucene.apache.org/solr/5_5_5/changes/Changes.html
> >>
> >> Please report any feedback to the mailing lists
> >> (http://lucene.apache.org/solr/discussion.html)
> >>
> >> Note: The Apache Software Foundation uses an extensive mirroring
> >> network for distributing releases. It is possible that the mirror you
> >> are using may not have replicated the release yet. If that is the
> >> case, please try another mirror. This also goes for Maven access.
>
>

Re: [ANNOUNCE] Apache Solr 5.5.5 released

[Alexandre Rafalovitch]

Stay on Solr 7, that's the latest and greatest (well 7.1 is)

5.5.5 is a maintenance release due to the security issue.

It is not just the tip of the spear, older releases sometimes get upgrades too.

Regards,
   Alex

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 October 2017 at 14:28, Moenieb Davids  wrote:
> Hi Steve,
>
> I have just started with Solr 7.*, so I am a bit confused with 5.5.5, same
> with lucene.
> Also, the sites register versions 7.*,
> Apologies for my ignorance if I had missed anything or do not have a proper
> understanding of the version management
>
> Regards
> Moenieb
>
> On Tue, Oct 24, 2017 at 6:27 PM, Steve Rowe  wrote:
>
>> Yes.
>>
>> --
>> Steve
>> www.lucidworks.com
>>
>> > On Oct 24, 2017, at 12:25 PM, Moenieb Davids 
>> wrote:
>> >
>> > Solr 5.5.5?
>> >
>> > On 24 Oct 2017 17:34, "Steve Rowe"  wrote:
>> >
>> >> 24 October 2017, Apache Solr™ 5.5.5 available
>> >>
>> >> The Lucene PMC is pleased to announce the release of Apache Solr 5.5.5.
>> >>
>> >> Solr is the popular, blazing fast, open source NoSQL search platform
>> from
>> >> the
>> >> Apache Lucene project. Its major features include powerful full-text
>> >> search,
>> >> hit highlighting, faceted search and analytics, rich document parsing,
>> >> geospatial search, extensive REST APIs as well as parallel SQL. Solr is
>> >> enterprise grade, secure and highly scalable, providing fault tolerant
>> >> distributed search and indexing, and powers the search and navigation
>> >> features
>> >> of many of the world's largest internet sites.
>> >>
>> >> This release contains one bugfix.
>> >>
>> >> This release includes one critical and one important security fix.
>> Details:
>> >>
>> >> * Fix for a 0-day exploit (CVE-2017-12629), details:
>> >> https://s.apache.org/FJDl.
>> >> RunExecutableListener has been disabled by default (can be enabled by
>> >> -Dsolr.enableRunExecutableListener=true) and resolving external
>> entities
>> >> in the
>> >> XML query parser (defType=xmlparser or {!xmlparser ... }) is disabled by
>> >> default.
>> >>
>> >> * Fix for CVE-2017-7660: Security Vulnerability in secure inter-node
>> >> communication
>> >> in Apache Solr, details: https://s.apache.org/APTY
>> >>
>> >> Furthermore, this release includes Apache Lucene 5.5.5 which includes
>> one
>> >> security
>> >> fix since the 5.5.4 release.
>> >>
>> >> The release is available for immediate download at:
>> >>
>> >> http://www.apache.org/dyn/closer.lua/lucene/solr/5.5.5
>> >>
>> >> Please read CHANGES.txt for a detailed list of changes:
>> >>
>> >> https://lucene.apache.org/solr/5_5_5/changes/Changes.html
>> >>
>> >> Please report any feedback to the mailing lists
>> >> (http://lucene.apache.org/solr/discussion.html)
>> >>
>> >> Note: The Apache Software Foundation uses an extensive mirroring
>> >> network for distributing releases. It is possible that the mirror you
>> >> are using may not have replicated the release yet. If that is the
>> >> case, please try another mirror. This also goes for Maven access.
>>
>>

Re: [ANNOUNCE] Apache Solr 5.5.5 released

[Erick Erickson]

Releasing something as far back as two major versions is rare, it's
got to be something critical. In this case a security issue.

It's only provided for people who are on 5x and cannot upgrade to
other versions of Solr that have the fix already.

Best,
Erick

On Tue, Oct 24, 2017 at 11:39 AM, Alexandre Rafalovitch
 wrote:
> Stay on Solr 7, that's the latest and greatest (well 7.1 is)
>
> 5.5.5 is a maintenance release due to the security issue.
>
> It is not just the tip of the spear, older releases sometimes get upgrades 
> too.
>
> Regards,
>Alex
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 24 October 2017 at 14:28, Moenieb Davids  wrote:
>> Hi Steve,
>>
>> I have just started with Solr 7.*, so I am a bit confused with 5.5.5, same
>> with lucene.
>> Also, the sites register versions 7.*,
>> Apologies for my ignorance if I had missed anything or do not have a proper
>> understanding of the version management
>>
>> Regards
>> Moenieb
>>
>> On Tue, Oct 24, 2017 at 6:27 PM, Steve Rowe  wrote:
>>
>>> Yes.
>>>
>>> --
>>> Steve
>>> www.lucidworks.com
>>>
>>> > On Oct 24, 2017, at 12:25 PM, Moenieb Davids 
>>> wrote:
>>> >
>>> > Solr 5.5.5?
>>> >
>>> > On 24 Oct 2017 17:34, "Steve Rowe"  wrote:
>>> >
>>> >> 24 October 2017, Apache Solr™ 5.5.5 available
>>> >>
>>> >> The Lucene PMC is pleased to announce the release of Apache Solr 5.5.5.
>>> >>
>>> >> Solr is the popular, blazing fast, open source NoSQL search platform
>>> from
>>> >> the
>>> >> Apache Lucene project. Its major features include powerful full-text
>>> >> search,
>>> >> hit highlighting, faceted search and analytics, rich document parsing,
>>> >> geospatial search, extensive REST APIs as well as parallel SQL. Solr is
>>> >> enterprise grade, secure and highly scalable, providing fault tolerant
>>> >> distributed search and indexing, and powers the search and navigation
>>> >> features
>>> >> of many of the world's largest internet sites.
>>> >>
>>> >> This release contains one bugfix.
>>> >>
>>> >> This release includes one critical and one important security fix.
>>> Details:
>>> >>
>>> >> * Fix for a 0-day exploit (CVE-2017-12629), details:
>>> >> https://s.apache.org/FJDl.
>>> >> RunExecutableListener has been disabled by default (can be enabled by
>>> >> -Dsolr.enableRunExecutableListener=true) and resolving external
>>> entities
>>> >> in the
>>> >> XML query parser (defType=xmlparser or {!xmlparser ... }) is disabled by
>>> >> default.
>>> >>
>>> >> * Fix for CVE-2017-7660: Security Vulnerability in secure inter-node
>>> >> communication
>>> >> in Apache Solr, details: https://s.apache.org/APTY
>>> >>
>>> >> Furthermore, this release includes Apache Lucene 5.5.5 which includes
>>> one
>>> >> security
>>> >> fix since the 5.5.4 release.
>>> >>
>>> >> The release is available for immediate download at:
>>> >>
>>> >> http://www.apache.org/dyn/closer.lua/lucene/solr/5.5.5
>>> >>
>>> >> Please read CHANGES.txt for a detailed list of changes:
>>> >>
>>> >> https://lucene.apache.org/solr/5_5_5/changes/Changes.html
>>> >>
>>> >> Please report any feedback to the mailing lists
>>> >> (http://lucene.apache.org/solr/discussion.html)
>>> >>
>>> >> Note: The Apache Software Foundation uses an extensive mirroring
>>> >> network for distributing releases. It is possible that the mirror you
>>> >> are using may not have replicated the release yet. If that is the
>>> >> case, please try another mirror. This also goes for Maven access.
>>>
>>>

Get Username within a plugin (Basic Authentication Plugin)

[startrekfan]

Hello,

I have set up an solr server with the "Basic Authentication Plugin". Now
I'd like to write a custom PostFilter(extends ExtendedQueryBase) plugin.
Can you explain me how I can get the username of the authenticated user
within my plugin?

Thanks

Some problems in SOLR-6.5.1

[SOLR4189]

Before two days we upgraded our SOLR servers from 4.10.1 version to 6.5.1. We
explored logs and saw too many errors like:

1)
org.apache.solr.common.SolrException; null:java.lang.NullPointerException
  at
org.apache.solr.search.grouping.distributed.responseprocessor.StoredFieldsShardResponseProcessor.process(StoredFieldsShardResponseProcessor.java:41)
  at
org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:771)
 . . .

We don't know from which queries it throws.

2) Second error or something strange that we saw in logs - sometimes SOLR
service restarts automatically without any error

Can somebody help to us? Does someone have problems like ours?




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Facets based on sampling

[John Davis]

On Tue, Oct 24, 2017 at 8:37 AM, Erick Erickson 
wrote:

> bq:  It is a bit surprising why facet computation
>  is so slow even when the query matches hundreds of docs.
>
> The number of terms in the field over all docs also comes into play.
> Say you're faceting over a field that has 100,000,000 unique values
> across all docs, that's a lot of bookkeeping.
>
>
100M unique values might be across all docs, and unless the faceting
implementation is really naive I cannot see how that can come into play
when the query matches a fraction of those.



> Best,
> Erick
>
>
> On Tue, Oct 24, 2017 at 1:08 AM, Emir Arnautović
>  wrote:
> > Hi John,
> > Did you mean “docValues don’t work for analysed fields” since it works
> for multivalue string (or other supported types) fields. What you need to
> do is to convert your analysed field to multivalue string field - that
> requires changes in indexing flow.
> >
> > HTH,
> > Emir
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> >
> >> On 23 Oct 2017, at 21:08, John Davis  wrote:
> >>
> >> Docvalues don't work for multivalued fields. I just started a separate
> >> thread with more debug info. It is a bit surprising why facet
> computation
> >> is so slow even when the query matches hundreds of docs.
> >>
> >> On Mon, Oct 23, 2017 at 6:53 AM, alessandro.benedetti <
> a.benede...@sease.io>
> >> wrote:
> >>
> >>> Hi John,
> >>> first of all, I may state the obvious, but have you tried docValues ?
> >>>
> >>> Apart from that a friend of mine ( Diego Ceccarelli) was discussing a
> >>> probabilistic implementation similar to the hyperloglog[1] to
> approximate
> >>> facets counting.
> >>> I didn't have time to take a look in details / implement anything yet.
> >>> But it is on our To Do list :)
> >>> He may add some info here.
> >>>
> >>> Cheers
> >>>
> >>>
> >>>
> >>>
> >>> [1]
> >>> https://blog.yld.io/2017/04/19/hyperloglog-a-
> probabilistic-data-structure/
> >>>
> >>>
> >>>
> >>> -
> >>> ---
> >>> Alessandro Benedetti
> >>> Search Consultant, R&D Software Engineer, Director
> >>> Sease Ltd. - www.sease.io
> >>> --
> >>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >>>
> >
>

Re: Facets based on sampling

[Toke Eskildsen]

John Davis  wrote:
> 100M unique values might be across all docs, and unless the faceting
> implementation is really naive I cannot see how that can come into play
> when the query matches a fraction of those.

Solr simple string faceting uses an int-array to hold counts for the different 
terms in the facet. This array has the same length as the number of unique 
terms, which means 100M in your case (divided among the shards). In order to 
extract top-n, those 100M entries are iterated. Nearly all of them are 0 when 
the query result is small, but the implementation still requires the iteration 
of all array-entries. With low- to medium-cardinality (let's say up to a few 
million) this is normally not noticeable, but as cardinality goes up it takes 
its toll.

Number-based faceting uses a hashmap instead, but this approach scales poorly 
when the result set gets large (millions). I had great results with a structure 
that tracked the updated counters with Solr 4.10 and is in the process of 
porting to Solr 7. No promises of when that will finish and especially not 
about when/if it will be production-quality. Detailed description of the 
tracking idea at 
https://sbdevel.wordpress.com/2014/03/17/fast-faceting-with-high-cardinality-and-small-result-set/

- Toke Eskildsen

BlendedTermQuery for Solr?

[James]

 

On my Solr 6.6 server I'd like to use BlendedTermQuery.

https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/search/BlendedTe
rmQuery.html

 

I know it is a Lucene class. Is there a Solr API available to access it? If
not, maybe some workaround?

 

Thanks!