Re: Error when searching with special characters

[Ere Maijala]

You need to make sure you encode things properly in the URL. You can't 
just place an ampersand there because it's the parameter delimiter in a 
URL. If you're unsure, use e.g. http://meyerweb.com/eric/tools/dencoder/ 
to encode your search terms. You'll see that "r&d" will become 
%22r%26d%22. Escaping the ampersand for Solr is another thing. If that's 
needed, you'll need to URL encode "r\&d" so that it becomes %22r%5C%26d%22.


--Ere

1.7.2016, 7.13, Zheng Lin Edwin Yeo kirjoitti:

Hi,

When I use defType=edismax, and using debug mode by setting debug=True, I
found that the search for "r&d" is actually done to search on just the
character "r".

http://localhost:8983/solr/collection1/highlight?q=
"r&d"&debugQuery=true&defType=edismax

  "debug":{
"rawquerystring":"\"r",
"querystring":"\"r",
"parsedquery":"(+DisjunctionMaxQuery((text:r)))/no_coord",
"parsedquery_toString":"+(text:r)"


Even if I search with escape character, it is of no help.

http://localhost:8983/solr/collection1/highlight?q=
"r\&d"&debugQuery=true&defType=edismax

  "debug":{
"rawquerystring":"\"r\\",
"querystring":"\"r\\",
"parsedquery":"(+DisjunctionMaxQuery((text:r)))/no_coord",
"parsedquery_toString":"+(text:r)",



But if I'm using other symbols like "r*d", then the search is ok.

http://localhost:8983/solr/collection1/highlight?q=
"r*d"&debugQuery=true&defType=edismax

  "debug":{
"rawquerystring":"\"r*d\"",
"querystring":"\"r*d\"",
"parsedquery":"(+DisjunctionMaxQuery((text:\"r d\")))/no_coord",
"parsedquery_toString":"+(text:\"r d\")",


What could be the reason behind this?


Regards,
Edwin


On 20 June 2016 at 02:12, Ahmet Arslan  wrote:


Hi,

It is better to create a failing junit test case before opening jira.

ahmet


On Sunday, June 19, 2016 4:44 PM, Zheng Lin Edwin Yeo <
edwinye...@gmail.com> wrote:


Yes, it throws the parse exception even if the query is properly escaped
for ampersand (&) for defType=lucene.

Should we treat this as a bug, and create a JIRA>

Regards,
Edwin



On 19 June 2016 at 08:07, Ahmet Arslan  wrote:



If properly escaped ampersand throws parse exception, this could be a bug.



On Saturday, June 18, 2016 7:12 PM, Zheng Lin Edwin Yeo <
edwinye...@gmail.com> wrote:
Hi,

It does not work with the back slash too.

But I found that it does not work for defType=lucene.
It will work if the defType=dismax or edismax.

What could be the reason that it did not work with the default
defType=lucene?

Regards,
Edwin



On 18 June 2016 at 01:04, Ahmet Arslan  wrote:


Hi,

May be URL encoding issue?
By the way, I would use back slash to escape special characters.

Ahmet

On Friday, June 17, 2016 10:08 AM, Zheng Lin Edwin Yeo <
edwinye...@gmail.com> wrote:



Hi,

I encountered this error when I tried to search with special characters,
like "&" and "#".

{
  "responseHeader":{
"status":400,
"QTime":0},
  "error":{
"msg":"org.apache.solr.search.SyntaxError: Cannot parse
'\"Research ': Lexical error at line 1, column 11.  Encountered: 
after : \"\\\"Research \"",
"code":400}}


I have done the search by putting inverted commands, like: q="Research &
Development"

What could be the issue here?

I'm facing this problem in both Solr 5.4.0 and Solr 6.0.1.


Regards,
Edwin











--
Ere Maijala
Kansalliskirjasto / The National Library of Finland

Re: error rendering solr spatial in geoserver

[David Smiley]

Sorry, good point Era; I forgot about that.  I filed an issue:
https://issues.apache.org/jira/browse/SOLR-9270
When I work on that I'll add an upgrading note to the 6x section.

~ David

On Wed, Jun 29, 2016 at 6:31 AM Ere Maijala  wrote:

> It would have been _really_ nice if this had been in the release notes.
> Made me also scratch my head for a while when upgrading to Solr 6.
> Additionally, this makes a rolling upgrade from Solr 5.x a bit more
> scary since you have to update the collection schema to make the Solr 6
> nodes work while making sure that no Solr 5 node reloads the configuration.
>
> --Ere
>
> 30.6.2016, 3.46, David Smiley kirjoitti:
> > For polygons in 6.0 you need to set
> >
> spatialContextFactory="org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory"
> > -- see
> > https://cwiki.apache.org/confluence/display/solr/Spatial+Search and the
> > example.  And of course as you probably already know, put the JTS jar on
> > Solr's classpath.  What likely tripped you up between 5x and 6x is the
> > change in value of the spatialContextFactory as a result in
> organizational
> > package moving "com.spatial4j.core" to "org.locationtech.spatial4j".
> >
> > On Wed, Jun 29, 2016 at 12:44 PM tkg_cangkul 
> wrote:
> >
> >> hi erick, thx for your reply.
> >>
> >> i've solve this problem.
> >> i got this error when i use solr 6.0.0
> >> so i try to downgrade my solr to version 5.5.0 and it's successfull
> >>
> >>
> >> On 29/06/16 22:39, Erick Erickson wrote:
> >>> There is not nearly enough information here to say anything very
> helpful.
> >>> What does your schema look like for this field?
> >>> What does the input look like?
> >>> How are you pulling data from geoserver?
> >>>
> >>> You might want to review:
> >>> http://wiki.apache.org/solr/UsingMailingLists
> >>>
> >>> Best,
> >>> Erick
> >>>
> >>> On Wed, Jun 29, 2016 at 2:31 AM, tkg_cangkul  >>> > wrote:
> >>>
> >>> hi, i try to load data spatial from solr with geoserver.
> >>> when i try to show the layer preview i've got this error message.
> >>>
> >>> error
> >>>
> >>>
> >>> anybody can help me pls?
> >>>
> >>>
> >>
> >> --
> > Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> > LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> > http://www.solrenterprisesearchserver.com
> >
>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Error when searching with special characters

[Zheng Lin Edwin Yeo]

Hi Ere,

Yes, this works.
Thanks for your reply.

Regards,
Edwin


On 1 July 2016 at 15:00, Ere Maijala  wrote:

> You need to make sure you encode things properly in the URL. You can't
> just place an ampersand there because it's the parameter delimiter in a
> URL. If you're unsure, use e.g. http://meyerweb.com/eric/tools/dencoder/
> to encode your search terms. You'll see that "r&d" will become %22r%26d%22.
> Escaping the ampersand for Solr is another thing. If that's needed, you'll
> need to URL encode "r\&d" so that it becomes %22r%5C%26d%22.
>
> --Ere
>
>
> 1.7.2016, 7.13, Zheng Lin Edwin Yeo kirjoitti:
>
>> Hi,
>>
>> When I use defType=edismax, and using debug mode by setting debug=True, I
>> found that the search for "r&d" is actually done to search on just the
>> character "r".
>>
>> http://localhost:8983/solr/collection1/highlight?q=
>> "r&d"&debugQuery=true&defType=edismax
>>
>>   "debug":{
>> "rawquerystring":"\"r",
>> "querystring":"\"r",
>> "parsedquery":"(+DisjunctionMaxQuery((text:r)))/no_coord",
>> "parsedquery_toString":"+(text:r)"
>>
>>
>> Even if I search with escape character, it is of no help.
>>
>> http://localhost:8983/solr/collection1/highlight?q=
>> "r\&d"&debugQuery=true&defType=edismax
>>
>>   "debug":{
>> "rawquerystring":"\"r\\",
>> "querystring":"\"r\\",
>> "parsedquery":"(+DisjunctionMaxQuery((text:r)))/no_coord",
>> "parsedquery_toString":"+(text:r)",
>>
>>
>>
>> But if I'm using other symbols like "r*d", then the search is ok.
>>
>> http://localhost:8983/solr/collection1/highlight?q=
>> "r*d"&debugQuery=true&defType=edismax
>>
>>   "debug":{
>> "rawquerystring":"\"r*d\"",
>> "querystring":"\"r*d\"",
>> "parsedquery":"(+DisjunctionMaxQuery((text:\"r d\")))/no_coord",
>> "parsedquery_toString":"+(text:\"r d\")",
>>
>>
>> What could be the reason behind this?
>>
>>
>> Regards,
>> Edwin
>>
>>
>> On 20 June 2016 at 02:12, Ahmet Arslan  wrote:
>>
>> Hi,
>>>
>>> It is better to create a failing junit test case before opening jira.
>>>
>>> ahmet
>>>
>>>
>>> On Sunday, June 19, 2016 4:44 PM, Zheng Lin Edwin Yeo <
>>> edwinye...@gmail.com> wrote:
>>>
>>>
>>> Yes, it throws the parse exception even if the query is properly escaped
>>> for ampersand (&) for defType=lucene.
>>>
>>> Should we treat this as a bug, and create a JIRA>
>>>
>>> Regards,
>>> Edwin
>>>
>>>
>>>
>>> On 19 June 2016 at 08:07, Ahmet Arslan  wrote:
>>>
>>>
>>>
>>> If properly escaped ampersand throws parse exception, this could be a
>>> bug.
>>>
>>>
>>>
>>> On Saturday, June 18, 2016 7:12 PM, Zheng Lin Edwin Yeo <
>>> edwinye...@gmail.com> wrote:
>>> Hi,
>>>
>>> It does not work with the back slash too.
>>>
>>> But I found that it does not work for defType=lucene.
>>> It will work if the defType=dismax or edismax.
>>>
>>> What could be the reason that it did not work with the default
>>> defType=lucene?
>>>
>>> Regards,
>>> Edwin
>>>
>>>
>>>
>>> On 18 June 2016 at 01:04, Ahmet Arslan 
>>> wrote:
>>>
>>> Hi,

 May be URL encoding issue?
 By the way, I would use back slash to escape special characters.

 Ahmet

 On Friday, June 17, 2016 10:08 AM, Zheng Lin Edwin Yeo <
 edwinye...@gmail.com> wrote:



 Hi,

 I encountered this error when I tried to search with special characters,
 like "&" and "#".

 {
   "responseHeader":{
 "status":400,
 "QTime":0},
   "error":{
 "msg":"org.apache.solr.search.SyntaxError: Cannot parse
 '\"Research ': Lexical error at line 1, column 11.  Encountered: 
 after : \"\\\"Research \"",
 "code":400}}


 I have done the search by putting inverted commands, like: q="Research &
 Development"

 What could be the issue here?

 I'm facing this problem in both Solr 5.4.0 and Solr 6.0.1.


 Regards,
 Edwin


>>>
>>>
>>>
>>>
>>>
>>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>

solrcloud goes down

[穆 俊峰]

Dear Mr. Experts!


I came across a problem when use solrcloud, it goes down. The version is 4.9.1.

we got the java core log, it looks like the http connection pool is exhausted 
and most threads are waiting to get a free connection.

I have increased http connection defaults for the SolrJ client, and also 
configed the connection defaults in solr.xml for all shard servers as below.


6
3
1
500




the java stack as below

"httpShardExecutor-3-thread-541" prio=10 tid=0x7f7b1c02b000 nid=0x20af 
waiting on condition [0x7f79fd49]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000605710068> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
at 
org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at 
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
at 
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
at 
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208)
at 
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
at 
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205)
at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)




besides, we use the singleton pattern to create and get the solr server 
connection, I wonder if this pattern is OK. Once I shutdown the zookeeper, the 
application can not do the solr query, the error as "no live SolrServers 
available to handle this request".

so I need to restart our connection to reconnect solrcloud.

as we use the singleton pattern, we do not use the method of "shutdown" to 
release the solrserver connection, will this be a problem? or we need to create 
the connection on demand every time?


please help me, look forward to your reply. Thanks a lot!


Best Regards!
Kent Mu

答复: solrcloud goes down

[穆 俊峰]

FYI



发件人: 穆 俊峰 
发送时间: 2016年7月1日 8:45
收件人: solr-user@lucene.apache.org
主题: solrcloud goes down

Dear Mr. Experts!


I came across a problem when use solrcloud, it goes down. The version is 4.9.1.

we got the java core log, it looks like the http connection pool is exhausted 
and most threads are waiting to get a free connection.

I have increased http connection defaults for the SolrJ client, and also 
configed the connection defaults in solr.xml for all shard servers as below.


6
3
1
500




the java stack as below

"httpShardExecutor-3-thread-541" prio=10 tid=0x7f7b1c02b000 nid=0x20af 
waiting on condition [0x7f79fd49]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000605710068> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)
at 
org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at 
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)
at 
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)
at 
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208)
at 
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
at 
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205)
at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162)
at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)




besides, we use the singleton pattern to create and get the solr server 
connection, I wonder if this pattern is OK. Once I shutdown the zookeeper, the 
application can not do the solr query, the error as "no live SolrServers 
available to handle this request".

so I need to restart our connection to reconnect solrcloud.

as we use the singleton pattern, we do not use the method of "shutdown" to 
release the solrserver connection, will this be a problem? or we need to create 
the connection on demand every time?


please help me, look forward to your reply. Thanks a lot!


Best Regards!
Kent Mu

Writing Custom handlers for solr in java

[star jlong]

Hi,

I'm a newbie on this fascinating technology called solr. Well, I have been 
doing some configurations and run with solr. Now I want to custom handlers for 
solr with java. Please can somebody share with an example of a custom handler 
for solr 5.2.0, so that with this example will be able to write more handlers.



Thanks,
Jstar.

Matching all terms in a multiValued field

[Ellis, Tom (Financial Markets IT)]

Hi There,

I'm trying to create search component for some document level security. A user 
will have a number of tags assigned to them, and these will be passed to the 
search component which will add a filter to whatever the user's original query 
was. Documents will be written with some or all of the users tags, and the 
query must only return documents that have a set of tags that are included in 
the users tags.

E.g. Alice is authorised to see 'confidential' and 'paid_source'

Bob is only authorised to see 'confidential'

Document 1 has tags confidential and paid_source - Alice should be able to see 
this document, but Bob should not.

So if I am creating a query for Bob, how can I write it so that he can't see 
Document 1? I.e. how do I create a query that checks the multiValued field for 
'confidential' but excludes documents that have anything else?

Cheers,

Tom Ellis
Consultant Developer - Excelian
Data Lake | Financial Markets IT
LLOYDS BANK COMMERCIAL BANKING


E: tom.el...@lloydsbanking.com
Website: www.lloydsbankcommercial.com
, , ,
Reduce printing. Lloyds Banking Group is helping to build the low carbon 
economy.
Corporate Responsibility Report: 
www.lloydsbankinggroup-cr.com/downloads



Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. 
Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank plc. 
Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England 
and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. Registered 
Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. 
Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered Office: 
Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 2299428. 
Telephone: 0345 603 1637

Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential 
Regulation Authority and regulated by the Financial Conduct Authority and 
Prudential Regulation Authority.

Cheltenham & Gloucester plc is authorised and regulated by the Financial 
Conduct Authority.

Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings 
is a division of Lloyds Bank plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in 
Scotland no. SC218813.

This e-mail (including any attachments) is private and confidential and may 
contain privileged material. If you have received this e-mail in error, please 
notify the sender and delete it (including any attachments) immediately. You 
must not copy, distribute, disclose or use any of the information in it or any 
attachments. Telephone calls may be monitored or recorded.

Date range warming.

[Callum Lamb]

We want to warm some FQ's. The main ones we want to do being date presets
like "last 6 months", "last year" .etc

The queries for the last 6 months get generated to look like this from site
(it's really 6 months -1 day):

*date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*

But since I have to represent this is in the firstSearcher section of
solrconfig.xml I need to use the date math features (Is there another way?
There doesn't seem to be a JVM system properties with the date in it, and I
don't want to have to restart solr everyday to update a solr env variable).

So I have this:

*date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*

Which should resolve to the same thing. Is there someway I can check this
for sure? I get the same results when I run them.

I have a couple questions though:

1. Is Solr smart enough to see that it if the current explicit queries that
come through are the same as my date math queries and re-use the fq in this
case? Is there a way to confirm this? I can go and change them to be the
same as well, not much of an issue, more curious than anything.

2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing all
the time I'm worried there some kind of checker than just sets cache=false
on all queries containing NOW or worse expands them to the current time and
caches that, and none of the fq's will ever match it (assuming solr just
does a strcmp for fq's).

Cheers,

Callum.

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at 
http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, 
privileged 
or otherwise protected under applicable law. Unauthorised disclosure, 
copying, distribution 
or use of the contents is prohibited and may be unlawful. If you have 
received this email in error,
including without appropriate authorisation, then please reply to the 
sender about the error 
and delete this email and any attachments.

Re: Date range warming.

[Callum Lamb]

Woops, Just realised it's meant to be:

date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-*1MILLISECOND*])

instead.

On Fri, Jul 1, 2016 at 11:52 AM, Callum Lamb  wrote:

> We want to warm some FQ's. The main ones we want to do being date presets
> like "last 6 months", "last year" .etc
>
> The queries for the last 6 months get generated to look like this from
> site (it's really 6 months -1 day):
>
> *date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
>
> But since I have to represent this is in the firstSearcher section of
> solrconfig.xml I need to use the date math features (Is there another way?
> There doesn't seem to be a JVM system properties with the date in it, and I
> don't want to have to restart solr everyday to update a solr env variable).
>
> So I have this:
>
> *date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
>
> Which should resolve to the same thing. Is there someway I can check this
> for sure? I get the same results when I run them.
>
> I have a couple questions though:
>
> 1. Is Solr smart enough to see that it if the current explicit queries
> that come through are the same as my date math queries and re-use the fq in
> this case? Is there a way to confirm this? I can go and change them to be
> the same as well, not much of an issue, more curious than anything.
>
> 2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing all
> the time I'm worried there some kind of checker than just sets cache=false
> on all queries containing NOW or worse expands them to the current time and
> caches that, and none of the fq's will ever match it (assuming solr just
> does a strcmp for fq's).
>
> Cheers,
>
> Callum.
>
>
>
>
>

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at 
http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, 
privileged 
or otherwise protected under applicable law. Unauthorised disclosure, 
copying, distribution 
or use of the contents is prohibited and may be unlawful. If you have 
received this email in error,
including without appropriate authorisation, then please reply to the 
sender about the error 
and delete this email and any attachments.

solr shutdown error

[穆 俊峰]

hi all,


solr version: 4.9.0


I create and get a HttpSolrServer connection by singleton pattern. I find the 
method of "shutdown" in solrj API, so I decided to use "shutdown" to release 
resources, but I got the error as below, It is due to singleton pattern?



Error executing query
org.apache.solr.client.solrj.SolrServerException: Error executing query
at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:100)
 ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) 
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at 
com.suning.review.business.ReviewSolrBusiness.getReviewCntByReviewType(ReviewSolrBusiness.java:1072)
 ~[classes:na]
at 
com.suning.review.business.ReviewSolrBusiness.getReviewCountFromSolr(ReviewSolrBusiness.java:675)
 ~[classes:na]
at 
com.suning.review.business.ReviewCmmdtyBusiness.getCmmdtysCnt(ReviewCmmdtyBusiness.java:736)
 [classes:na]
at 
com.suning.review.business.ReviewCmmdtyBusiness.singleCmmdtyCnts(ReviewCmmdtyBusiness.java:877)
 [classes:na]
at 
com.suning.review.service.impl.CommodityReviewServiceImpl.getCmmdtyReviewCount(CommodityReviewServiceImpl.java:372)
 ~[classes:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
~[na:1.7.0_25]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[na:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
at com.suning.rsf.util.MethodUtils.invokeMethod(MethodUtils.java:268) 
~[rsf-2.0.1.jar:2.0.1]
at 
com.suning.rsf.provider.ServiceExecutorImpl.invoke(ServiceExecutorImpl.java:171)
 ~[rsf-2.0.1.jar:2.0.1]
at 
com.suning.rsf.provider.ServiceExecutorImpl.invoke(ServiceExecutorImpl.java:89) 
~[rsf-2.0.1.jar:2.0.1]
at 
com.suning.rsf.protocol.tcp.TcpEntryPoint$MessageHandler$1.run(TcpEntryPoint.java:178)
 ~[rsf-2.0.1.jar:2.0.1]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_25]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34) 
~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) 
~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217) 
~[httpcore-4.4.1.jar:4.4.1]
at 
org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
 ~[httpclient-4.5.jar:4.5]
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
 ~[httpclient-4.5.jar:4.5]
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
 ~[httpclient-4.5.jar:4.5]
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
 ~[httpclient-4.5.jar:4.5]
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
 ~[httpclient-4.5.jar:4.5]
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
 ~[httpclient-4.5.jar:4.5]
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
 ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
 ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
 ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) 
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
... 17 common frames omitted



look forward to your reply!

Best Regards!
Kent Mu

questions for the method "shutdown" of CloudSolrServer

[穆 俊峰]

hi all,


I create a CloudSolrServer Connection by singleton pattern like this


public synchronized static CloudSolrServer getSolrCloudReadServer() {
if (reviewSolrCloudServer == null) {
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 1);
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500);
HttpClient client = HttpClientUtil.createClient(params);
LBHttpSolrServer lbServer = new LBHttpSolrServer(client);

lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut());
lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut());

reviewSolrCloudServer = new 
CloudSolrServer(ReviewProperties.getZkHost(),lbServer);
reviewSolrCloudServer.setDefaultCollection("commodityReview");

reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout());

reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout());
reviewSolrCloudServer.connect();
}
return reviewSolrCloudServer;
}



and I see the solrj API, the method to create an instance, and the method of 
shutdown

  public CloudSolrServer(String zkHost, LBHttpSolrServer lbServer) {
this.zkHost = zkHost;
this.lbServer = lbServer;
this.updatesToLeaders = true;
shutdownLBHttpSolrServer = false;
  }

  public void shutdown() {
if (zkStateReader != null) {
  synchronized(this) {
if (zkStateReader!= null)
  zkStateReader.close();
zkStateReader = null;
  }
}

if (shutdownLBHttpSolrServer) {
  lbServer.shutdown();
}

if (myClient!=null) {
  myClient.getConnectionManager().shutdown();
}

if(this.threadPool != null && !this.threadPool.isShutdown()) {
  this.threadPool.shutdown();
}
  }




this kind of method by create an instance with a LBHttpSolrServer does not need 
to release LBHttpSolrServer  and HttpClient??

look forward to your reply!

Best Regards!
Kent Mu

Solr 5.2.0 - support for multiValued field sorting?

[Harsh Darji]

Hi,
We are using solr 5.2.0. We have requirement to sort dynamic fields which 
contains multivalued = true.Can any one guide me how we can achieve it?If newer 
version of solr supports it we are ok to upgrade solr.
Regards,Harsh 

RE: Automatic Language Identification

[Allison, Timothy B.]

+1 to langdetect

In Tika 2.0, we're going to remove our own language detection code and allow 
users to select Optimaize (fork of langdetect), MIT Lincoln Lab’s Text.jl 
library or Yalder (https://github.com/kkrugler/yalder).  The first two are now 
available in Tika 1.13.

-Original Message-
From: Markus Jelsma [mailto:markus.jel...@openindex.io] 
Sent: Wednesday, June 22, 2016 8:27 AM
To: solr-user@lucene.apache.org; solr-user 
Subject: RE: Automatic Language Identification

Hello,

I recommend using the langdetect language detector, it supports many more 
languages and has much higher precission than Tika's detector.

Markus
 
 

Re: Date range warming.

[Callum Lamb]

Okay I figured it out. Answer here if anyone ever stumbled across this in
future.

With debugQuery on you can see filter_queries actually get processed into
what's in the parsed_filter_queries and it's those that get cached. In this
case solr converts them to unix timestamps TO unix_timestamp and It's these
that get cached. So you can see if they'll match by looking at those.

On Fri, Jul 1, 2016 at 12:00 PM, Callum Lamb  wrote:

> Woops, Just realised it's meant to be:
>
> date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-*1MILLISECOND*])
>
> instead.
>
> On Fri, Jul 1, 2016 at 11:52 AM, Callum Lamb  wrote:
>
>> We want to warm some FQ's. The main ones we want to do being date presets
>> like "last 6 months", "last year" .etc
>>
>> The queries for the last 6 months get generated to look like this from
>> site (it's really 6 months -1 day):
>>
>> *date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
>>
>> But since I have to represent this is in the firstSearcher section of
>> solrconfig.xml I need to use the date math features (Is there another way?
>> There doesn't seem to be a JVM system properties with the date in it, and I
>> don't want to have to restart solr everyday to update a solr env variable).
>>
>> So I have this:
>>
>> *date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
>>
>> Which should resolve to the same thing. Is there someway I can check this
>> for sure? I get the same results when I run them.
>>
>> I have a couple questions though:
>>
>> 1. Is Solr smart enough to see that it if the current explicit queries
>> that come through are the same as my date math queries and re-use the fq in
>> this case? Is there a way to confirm this? I can go and change them to be
>> the same as well, not much of an issue, more curious than anything.
>>
>> 2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing
>> all the time I'm worried there some kind of checker than just sets
>> cache=false on all queries containing NOW or worse expands them to the
>> current time and caches that, and none of the fq's will ever match it
>> (assuming solr just does a strcmp for fq's).
>>
>> Cheers,
>>
>> Callum.
>>
>>
>>
>>
>>
>

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at 
http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, 
privileged 
or otherwise protected under applicable law. Unauthorised disclosure, 
copying, distribution 
or use of the contents is prohibited and may be unlawful. If you have 
received this email in error,
including without appropriate authorisation, then please reply to the 
sender about the error 
and delete this email and any attachments.

SolrCloud not commit documents automatically

[Chet Rebman]

Hello all,

I am new to SolrCloud. I have read that solrCloud prefers to commit 
automatically, rather then recieve commit commands. However solrCloud never 
commits my documents.

I am running SolrCloud 6.1 with three solr servers and 3 zookeeper instances. I 
am using CloudSolrClient with all three zookeeper instances loaded into the 
constructor. The indexing procedes normally. After about 80,000 documents I get 
a ConcurrentModificationException. I believe this error is the side effect of 
the documents not being commited. I can wait for an hour and nothing is ever 
commited. As soon as I perform a 'update?commit=true' the documents appear and 
I can restart indexing as if nothing had happened. My solrconfing contains.


   
  ${solr.autoCommit.maxTime:15000}
  false
   
   
  ${solr.autoSoftCommit.maxTime:-1}
   


My understanding is this should performa a hard commit every 15 seconds.

I uploaded my config files to solrCloud with...
./zkCli.sh -cmd upconfig -zkhost  localhost:2181 -confname XXX  -confdir YYY

tia!

..Chet

Re: SolrCloud not commit documents automatically

[Shawn Heisey]

On 7/1/2016 10:01 AM, Chet Rebman wrote:
> 
>
>   ${solr.autoCommit.maxTime:15000}
>   false
>
>
>   ${solr.autoSoftCommit.maxTime:-1}
>
> 
>
> My understanding is this should performa a hard commit every 15 seconds.

Yes, that is exactly what it will do -- but none of those commits will
make new documents visible.  The autoCommit has openSearcher set to false.

Don't change openSearcher to true -- the reason this config is there has
nothing to do with document visibility -- it is about durability and
transaction log size.

http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

You'll need to use explicit commits, commitWithin, or autoSoftCommit for
document visibility.  Make these commits as infrequently as you can --
an interval of several minutes if that's possible.  Opening a searcher
is very resource-intensive, and not something you want to be doing
frequently.

Thanks,
Shawn

solr sort

[????/mamao]

data:1T
Solr Sort  occurs the memory overflow
query is normal

Re: solr sort

[Erick Erickson]

Please review:

http://wiki.apache.org/solr/UsingMailingLists

There is precisely no information here that would allow us to diagnose the root
of your problem. The only general advice would be use DocValues.

Erick

On Thu, Jun 30, 2016 at 11:09 PM, 闲鹤/mamao  wrote:
> data:1T
> Solr Sort  occurs the memory overflow
> query is normal

Re: Date range warming.

[Erick Erickson]

Glad you figured it out. BTW, your query can be a little simpler
since you can mix and match inclusive/exclusive end points as:

date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY})

Note curly brace rather than square bracket at the end.

As to whether the filter cache is actually used, if you look at the
admin UI>>core>>plugins/stats>>filterCache you can see
both the hit count for the filter cache for the current searcher as well
as cumulative stats. So a manual test would be
1> reopen the searcher (or start Solr)
2> submit a filter query and see if the hit count bumped.

Best,
Erick

On Fri, Jul 1, 2016 at 6:22 AM, Callum Lamb  wrote:
> Okay I figured it out. Answer here if anyone ever stumbled across this in
> future.
>
> With debugQuery on you can see filter_queries actually get processed into
> what's in the parsed_filter_queries and it's those that get cached. In this
> case solr converts them to unix timestamps TO unix_timestamp and It's these
> that get cached. So you can see if they'll match by looking at those.
>
> On Fri, Jul 1, 2016 at 12:00 PM, Callum Lamb  wrote:
>
>> Woops, Just realised it's meant to be:
>>
>> date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-*1MILLISECOND*])
>>
>> instead.
>>
>> On Fri, Jul 1, 2016 at 11:52 AM, Callum Lamb  wrote:
>>
>>> We want to warm some FQ's. The main ones we want to do being date presets
>>> like "last 6 months", "last year" .etc
>>>
>>> The queries for the last 6 months get generated to look like this from
>>> site (it's really 6 months -1 day):
>>>
>>> *date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
>>>
>>> But since I have to represent this is in the firstSearcher section of
>>> solrconfig.xml I need to use the date math features (Is there another way?
>>> There doesn't seem to be a JVM system properties with the date in it, and I
>>> don't want to have to restart solr everyday to update a solr env variable).
>>>
>>> So I have this:
>>>
>>> *date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
>>>
>>> Which should resolve to the same thing. Is there someway I can check this
>>> for sure? I get the same results when I run them.
>>>
>>> I have a couple questions though:
>>>
>>> 1. Is Solr smart enough to see that it if the current explicit queries
>>> that come through are the same as my date math queries and re-use the fq in
>>> this case? Is there a way to confirm this? I can go and change them to be
>>> the same as well, not much of an issue, more curious than anything.
>>>
>>> 2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing
>>> all the time I'm worried there some kind of checker than just sets
>>> cache=false on all queries containing NOW or worse expands them to the
>>> current time and caches that, and none of the fq's will ever match it
>>> (assuming solr just does a strcmp for fq's).
>>>
>>> Cheers,
>>>
>>> Callum.
>>>
>>>
>>>
>>>
>>>
>>
>
> --
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for our other offices can be found at
> http://www.mintel.com/office-locations.
>
> This email and any attachments may include content that is confidential,
> privileged
> or otherwise protected under applicable law. Unauthorised disclosure,
> copying, distribution
> or use of the contents is prohibited and may be unlawful. If you have
> received this email in error,
> including without appropriate authorisation, then please reply to the
> sender about the error
> and delete this email and any attachments.
>

Re: Solr 5.2.0 - support for multiValued field sorting?

[Erick Erickson]

First answer "what does sorting on a field with more than one value
mean?" Say I have
"aardvark" and "zebra". Where should the doc go, at the beginning or
end of the list?

For numerics at least you can do this, see:
https://lucidworks.com/blog/2015/09/10/minmax-on-multivalued-field/
in Solr 5.3+.

If it's a fixed sort kind of thing an alternative is to extract the
value at index
time that you want to sort by (once you've defined what sorting on this
field means) and put that in a separate, single-valued field.

Best,
Erick

On Fri, Jul 1, 2016 at 5:21 AM, Harsh Darji  wrote:
> Hi,
> We are using solr 5.2.0. We have requirement to sort dynamic fields which 
> contains multivalued = true.Can any one guide me how we can achieve it?If 
> newer version of solr supports it we are ok to upgrade solr.
> Regards,Harsh

Re: Matching all terms in a multiValued field

[Erick Erickson]

If the permissions are hierarchical, i.e. anyone who can see
paid_source and see confidential you could just index the
minimum-required clearance with the doc (think numeric codes here) and
form your fq as auth:[max_level_for_user TO *]. I admit I've rarely
seen security models that are that simple.

You can form a rather complex fq clause like
fq=auth:(confidential NOT (paid_source OR other_levels_bob_can't_see)]

On the plus side, that fq clause can be cached in the filterCache and re-used.
Hint: If you do this, be absolutely sure you form the filter clause exactly the
same way each time to insure it is re-used. I.e. even though this
is logically equivalent it wouldn't re-use the filterCache entry

fq=auth:(confidential NOT (other_levels_bob_can't_see OR paid_source)]

You could also write a "post filter", they were originally written
exactly to handle
ACLs, see: http://yonik.com/advanced-filter-caching-in-solr/
and
https://lucidworks.com/blog/2012/02/22/custom-security-filtering-in-solr/

Best,
Erick

On Fri, Jul 1, 2016 at 3:48 AM, Ellis, Tom (Financial Markets IT)
 wrote:
> Hi There,
>
> I'm trying to create search component for some document level security. A 
> user will have a number of tags assigned to them, and these will be passed to 
> the search component which will add a filter to whatever the user's original 
> query was. Documents will be written with some or all of the users tags, and 
> the query must only return documents that have a set of tags that are 
> included in the users tags.
>
> E.g. Alice is authorised to see 'confidential' and 'paid_source'
>
> Bob is only authorised to see 'confidential'
>
> Document 1 has tags confidential and paid_source - Alice should be able to 
> see this document, but Bob should not.
>
> So if I am creating a query for Bob, how can I write it so that he can't see 
> Document 1? I.e. how do I create a query that checks the multiValued field 
> for 'confidential' but excludes documents that have anything else?
>
> Cheers,
>
> Tom Ellis
> Consultant Developer - Excelian
> Data Lake | Financial Markets IT
> LLOYDS BANK COMMERCIAL BANKING
> 
>
> E: tom.el...@lloydsbanking.com
> Website: www.lloydsbankcommercial.com
> , , ,
> Reduce printing. Lloyds Banking Group is helping to build the low carbon 
> economy.
> Corporate Responsibility Report: 
> www.lloydsbankinggroup-cr.com/downloads
>
>
>
> Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. 
> Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds Bank 
> plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in 
> England and Wales no. 2065. Telephone 0207626 1500. Bank of Scotland plc. 
> Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. 
> SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc. Registered 
> Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales 
> 2299428. Telephone: 0345 603 1637
>
> Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential 
> Regulation Authority and regulated by the Financial Conduct Authority and 
> Prudential Regulation Authority.
>
> Cheltenham & Gloucester plc is authorised and regulated by the Financial 
> Conduct Authority.
>
> Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester 
> Savings is a division of Lloyds Bank plc.
>
> HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in 
> Scotland no. SC218813.
>
> This e-mail (including any attachments) is private and confidential and may 
> contain privileged material. If you have received this e-mail in error, 
> please notify the sender and delete it (including any attachments) 
> immediately. You must not copy, distribute, disclose or use any of the 
> information in it or any attachments. Telephone calls may be monitored or 
> recorded.

Re: 500 error querying solrcloud with >1 shard

[Erick Erickson]

Hmmm, don't see anything outstanding. What version of Solr?
Historically there have been some bugs when a shard doesn't
happen to host any docs, which sometimes occurs when you
are testing and just add a few docs to the collection, is that
a possibility?

Best,
Erick

On Thu, Jun 30, 2016 at 10:40 PM, Nuhaa All Bakry  wrote:
> ah i see
>
> https://ibin.co/2mWaie7IVxDF.png
>
> Collection A:
> https://ibin.co/2mWawhCm76cN.png
>
> Collection B:
> https://ibin.co/2mWb4BmlMom2.png
>
>
>
> --
> nuhaa
> http://about.me/nuhaa
>
>> On 1 Jul 2016, at 12:24 PM, Erick Erickson  wrote:
>>
>> None of the pasted images came through, the mail server is quite aggressive
>> about stripping them. You'll need to upload them somewhere and provide a
>> link.
>>
>> Best,
>> Erick
>>
>> On Thu, Jun 30, 2016 at 7:36 PM, nuhaa  wrote:
>>
>>> hi all,
>>>
>>> We have a solrcloud setup with zookeeper, and right now we're testing it
>>> with indexing and querying.
>>>
>>>
>>> *collection A:*
>>>
>>> *collection B:*
>>>
>>>
>>>
>>> I'm trying to figure out why on collection B, indexing works but querying
>>> doesn't. I believe by looking at Collection B > Schema > Load Term Info on
>>> a selected field, if I can see the value histogram, that means they're
>>> indexed successfully.
>>>
>>> On collection A, indexing and querying both work.
>>>
>>> This is what returned when querying on collection B:
>>>
>>> {
>>>  "responseHeader":{
>>>"zkConnected":true,
>>>"status":500,
>>>"QTime":17,
>>>"params":{
>>>  "q":"*:*",
>>>  "indent":"on",
>>>  "wt":"json",
>>>  "_":"1467275265841"}},
>>>  "error":{
>>>"trace":"java.lang.NullPointerException\n\tat
>>> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1209)\n\tat
>>> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1060)\n\tat
>>> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:759)\n\tat
>>> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:738)\n\tat
>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)\n\tat
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)\n\tat
>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)\n\tat
>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)\n\tat
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)\n\tat
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)\n\tat
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)\n\tat
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)\n\tat
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\n\tat
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\n\tat
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\tat
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\n\tat
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat
>>> org.eclipse.jetty.server.Server.handle(Server.java:518)\n\tat
>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)\n\tat
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)\n\tat
>>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat
>>> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat
>>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat
>>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)\n\tat
>>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)\n\tat
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)\n\tat
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)\n\tat
>>> java.lang.Thread.run(Thread.java:745)\n",
>>>"code":500}}
>>>
>>> Is there anything I need to do add on solrconfig.xml?? Or is there
>>> something possibly wrong with the zookeeper/solrcloud configurations?
>>> How can

Re: Writing Custom handlers for solr in java

[Erick Erickson]

What _kind_ of handler? A search component? An update handler? A query parser?

Basically I'd do a google search, there are a lot of examples out
there. Try to pick
one that's relatively recent.

On Fri, Jul 1, 2016 at 3:40 AM, star jlong  wrote:
> Hi,
>
> I'm a newbie on this fascinating technology called solr. Well, I have been 
> doing some configurations and run with solr. Now I want to custom handlers 
> for solr with java. Please can somebody share with an example of a custom 
> handler for solr 5.2.0, so that with this example will be able to write more 
> handlers.
>
>
>
> Thanks,
> Jstar.

SolrCloud Node fails that was hosting replicas of a collection

[Deeksha Sharma]

Currently I am building a SolrCloud cluster with 3 Zookeepers (ensemble) and 4 
solr instances. Cluster is hosting 4 collections and their replicas.


When one Solr node say Solr1 goes down (hosting 2 replicas of collection1 and 
collection2), I add a new node to the cluster and that node in Admin UI is 
brown in color which means the new node is down.


When I create the core on Admin UI to this new solr instance (these cores are 
the 2 replicas that Solr1 was hosting) , the new node becomes green (up and 
running).


Am I doing the right thing by adding the new node and adding cores to it via 
Admin UI or there is a better way of doing this?


Should Solr automatically host those 2 replicas to the newly added node or we 
have to manually add cores to it?

Re: SolrCloud Node fails that was hosting replicas of a collection

[Erick Erickson]

Please do _not_ use the admin UI core creating screen when dealing
with SolrCloud. It can work, but you have to get everything exactly
right.

Instead, you should be using the ADDREPLICA command from the
Collections API, see:
https://cwiki.apache.org/confluence/display/solr/Collections+API

Although I do have to ask why the Solr node is going down. If it's not something
permanent, the replicas should return to green after the node is re-started.

There are plans to provide a screen in the new Admin UI to allow you
to add replicas to a collection and the like, but that code hasn't
been added yet.

Best,
Erick

On Fri, Jul 1, 2016 at 12:18 PM, Deeksha Sharma  wrote:
> Currently I am building a SolrCloud cluster with 3 Zookeepers (ensemble) and 
> 4 solr instances. Cluster is hosting 4 collections and their replicas.
>
>
> When one Solr node say Solr1 goes down (hosting 2 replicas of collection1 and 
> collection2), I add a new node to the cluster and that node in Admin UI is 
> brown in color which means the new node is down.
>
>
> When I create the core on Admin UI to this new solr instance (these cores are 
> the 2 replicas that Solr1 was hosting) , the new node becomes green (up and 
> running).
>
>
> Am I doing the right thing by adding the new node and adding cores to it via 
> Admin UI or there is a better way of doing this?
>
>
> Should Solr automatically host those 2 replicas to the newly added node or we 
> have to manually add cores to it?
>
>

Question about Indexing Updated Documents

[Jamal, Sarfaraz]

Hi Guys,

I have a data-import handler set up that indexes all of the documents from a 
few small tables.

What is the best way to update the index when a single one of those documents 
change.

Is it possible to use SQL or must I post json or xml to solr?

Thanks you,

Sas

Re: Searching inside parent AND children, returning only parents.

[Mikhail Khludnev]

Yep, query syntax is a little bit challenging. Here I'm showing query
nesting to answer your question and also a better syntax for boolean
queries:

q={!parent which="parentDoc:true" v=$childq} (+parentDoc:true
+) &childq=+parentDoc:false +
30 июня 2016 г. 10:11 пользователь "Marre"  написал:

> I have been experimenting with solr for a couple of weeks but i've been
> stuck
> on a query i would like to execute for a couple of days now.
>
> I have a nested data structure where I'm using a fq like this:
>
> {!parent which="parentDoc:true"}parentDoc:false AND 
>
> This matches my child documents and returns the parent to those children. I
> am very pleased with that. BUT the problem i have is if there is a match
> directly inside the parent and nothing in the children. I will not get a
> response.
>
> I would make it so that in some way there is a OR condition of some sort
> making it so, any document can match the searched term AND parentDoc:false
> OR the above filter query.
>
> Is this even possible to execute within one query in solr, or do i have to
> make two? I have not found any information about this issue, making me to
> believe im just missing something trivial.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Searching-inside-parent-AND-children-returning-only-parents-tp4285078.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Question about Indexing Updated Documents

[Chris Hostetter]

If you are already using DIH, then you can use a 
deltaQuery to find "updated" documents and index only 
them.

https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler

Some people just parameterize their main DIH query and use request params 
to control which documents get indexed when doing a "full build"...

https://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport



-Hoss
http://www.lucidworks.com/

Re: Automatic Language Identification

[William Bell]

We should add a simple filter in Solr for this. The current way requires
indexing.

https://github.com/kkrugler/yalder is good, it would be a great filter:

if NOT english, fail the whole text.

On Fri, Jul 1, 2016 at 6:33 AM, Allison, Timothy B. 
wrote:

> +1 to langdetect
>
> In Tika 2.0, we're going to remove our own language detection code and
> allow users to select Optimaize (fork of langdetect), MIT Lincoln Lab’s
> Text.jl library or Yalder (https://github.com/kkrugler/yalder).  The
> first two are now available in Tika 1.13.
>
> -Original Message-
> From: Markus Jelsma [mailto:markus.jel...@openindex.io]
> Sent: Wednesday, June 22, 2016 8:27 AM
> To: solr-user@lucene.apache.org; solr-user 
> Subject: RE: Automatic Language Identification
>
> Hello,
>
> I recommend using the langdetect language detector, it supports many more
> languages and has much higher precission than Tika's detector.
>
> Markus
>
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

solr shutdown error

[Kent Mu]

hi all,


solr version: 4.9.0


I create and get a HttpSolrServer connection by singleton pattern. I find
the method of "shutdown" in solrj API, so I decided to use "shutdown" to
release resources, but I got the error as below, It is due to singleton
pattern?


*Error executing query*
*org.apache.solr.client.solrj.SolrServerException: Error executing query*
*at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:100)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]*
*at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]*
*at
com.suning.review.business.ReviewSolrBusiness.getReviewCntByReviewType(ReviewSolrBusiness.java:1072)
~[classes:na]*
*at
com.suning.review.business.ReviewSolrBusiness.getReviewCountFromSolr(ReviewSolrBusiness.java:675)
~[classes:na]*
*at
com.suning.review.business.ReviewCmmdtyBusiness.getCmmdtysCnt(ReviewCmmdtyBusiness.java:736)
[classes:na]*
*at
com.suning.review.business.ReviewCmmdtyBusiness.singleCmmdtyCnts(ReviewCmmdtyBusiness.java:877)
[classes:na]*
*at
com.suning.review.service.impl.CommodityReviewServiceImpl.getCmmdtyReviewCount(CommodityReviewServiceImpl.java:372)
~[classes:na]*
*at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_25]*
*at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_25]*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_25]*
*at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]*
*at com.suning.rsf.util.MethodUtils.invokeMethod(MethodUtils.java:268)
~[rsf-2.0.1.jar:2.0.1]*
*at
com.suning.rsf.provider.ServiceExecutorImpl.invoke(ServiceExecutorImpl.java:171)
~[rsf-2.0.1.jar:2.0.1]*
*at
com.suning.rsf.provider.ServiceExecutorImpl.invoke(ServiceExecutorImpl.java:89)
~[rsf-2.0.1.jar:2.0.1]*
*at
com.suning.rsf.protocol.tcp.TcpEntryPoint$MessageHandler$1.run(TcpEntryPoint.java:178)
~[rsf-2.0.1.jar:2.0.1]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]*
*at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]*
*Caused by: java.lang.IllegalStateException: Connection pool shut down*
*at org.apache.http.util.Asserts.check(Asserts.java:34)
~[httpcore-4.4.1.jar:4.4.1]*
*at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
~[httpcore-4.4.1.jar:4.4.1]*
*at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
~[httpcore-4.4.1.jar:4.4.1]*
*at
org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
~[httpclient-4.5.jar:4.5]*
*at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
~[httpclient-4.5.jar:4.5]*
*at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
~[httpclient-4.5.jar:4.5]*
*at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
~[httpclient-4.5.jar:4.5]*
*at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
~[httpclient-4.5.jar:4.5]*
*at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
~[httpclient-4.5.jar:4.5]*
*at
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]*
*at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]*
*at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]*
*at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]*
*... 17 common frames omitted*


look forward to your reply!

Best Regards!
Kent Mu

Re: solr shutdown error

[Shawn Heisey]

On 7/1/2016 10:59 PM, Kent Mu wrote:
> solr version: 4.9.0 I create and get a HttpSolrServer connection by
> singleton pattern. I find the method of "shutdown" in solrj API, so I
> decided to use "shutdown" to release resources, but I got the error as
> below, It is due to singleton pattern? 

If you call shutdown, that client object can no longer be used.  You
will need to make a new one.

The expected usage pattern for the client object is that you will create
one object when the program starts and use it for all threads until the
program exits.  Whether or not to use the singleton pattern for that is
up to you.  Usually the only time you would call shutdown is just before
program exit ... but if the program is exiting, shutdown isn't really
necessary.

If you are in a situation where client objects are being regularly
created, then you would want to do shutdown on each of them when you are
finished so you don't have a resource leak.  Creating many client
objects is not recommended.

Thanks,
Shawn