Re: Block join children query parser returns "Parent query yields document which is not matched by parents filter"

[Mikhail Khludnev]

Hello,
note
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-BlockJoinQueryParsers

> The syntax for this parser is: q={!child of=}.

Thus it should be:
{!child of='ps:[* TO *]'}ps:PS1

it yields doc 2 with Martin


On Wed, Aug 12, 2015 at 9:54 AM, MOIS Martin (MORPHO) <
martin.m...@morpho.com> wrote:

> Hello,
>
> currently I am evaluating the block join children query parser. But it
> always returns "Parent query yields document which is not matched by
> parents filter".
>
> What I have done so far:
>
> I have added the _root_ field to the schema:  type="string" indexed="true" stored="false"/>.
> I have created a collection: curl
> http://localhost:8983/solr/admin/collections?action=CREATE&name=nestedPerson&numShards=6&collection.configName=mim
> I have enrolled this documents to the collection: curl
> http://localhost:8983/solr/nestedPerson/update?commitWithin=3000 -d '
> 
> 
>1
>PS1
>false
>
>2
>CS1
> name="firstName">Martin
>
> 
> 
>3
>PS2
>true
>
>4
>CS2
> name="firstName">Stefan
>
> 
> '
>
> Now I issue the following query: {!child of=ps:PS1}firstName:Martin. This
> returns an error message:
> "error": {
> "msg": "Error from server at
> http://bigdata2:8983/solr/nestedPerson_shard1_replica1: Parent query
> yields document which is not matched by parents filter, docID=0",
> "trace":
> "org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at
> http://bigdata2:8983/solr/nestedPerson_shard1_replica1: Parent query
> yields document which is not matched by parents filter, docID=0\n\tat
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)\n\tat
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)\n\tat
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)\n\tat
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)\n\tat
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:218)\n\tat
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
> java.lang.Thread.run(Thread.java:745)\n",
> "code": 500
>   }
>
> Interestingly the block join parent query parser returns a valid result
> set for this query: {!parent which=ps:PS1}firstName:Martin
>
> Do you have any idea why the children query parser leads to an error and
> the parent query parser not?
>
> I am using Solr 5.2.0 (5.2.0 1683206 - anshumgupta - 2015-06-02 18:32:43).
> It is a cloud installation (using zookeeper) with 6 nodes (one shard per
> node).
>
> Best Regards,
> Martin Mois
> #
> " This e-mail and any attached documents may contain confidential or
> proprietary information. If you are not the intended recipient, you are
> notified that any dissemination, copying of this e-mail and any attachments
> thereto or use of their contents by any means whatsoever is strictly
> prohibited. If you have received this e-mail in error, please advise the
> sender immediately and delete this e-mail and all attached documents from
> your computer system."
> #
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

Re: Choosing the order of the fields to be displayed at output

[Zheng Lin Edwin Yeo]

Thank you for the information.

Regards,
Edwin

On 12 August 2015 at 11:48, Shawn Heisey  wrote:

> On 8/11/2015 9:36 PM, Zheng Lin Edwin Yeo wrote:
> > I'm using Solr 5.2.1. I understand that for JSON format, Solr writes out
> > the fields of each document in the order they are found in the index as
> it
> > is the fastest and most efficient for Solr to return the data.
> >
> > However, this causes confusion as each of the records has fields arranged
> > in different order, as user are allowed to update the field after the
> > document is index. Whenever a field is updated, that field will be
> > displayed at the bottom of the record.
> >
> > Is there a way to choose the order of the fields to be displayed at the
> > output, so that the order will be consistent for all the records?
>
> Solr simply returns the fields in the order that Java naturally stores
> the information, which from a user perspective, is not very predictable,
> and may change from one version of code to the next, or when Java is
> upgraded.
>
> I think that deciding information display order order is a job for
> client code.  The application that makes the request to Solr can pick
> the pieces that need to be displayed to the user and decide what order
> they should be in.
>
> Thanks,
> Shawn
>
>

RE: Performance warning overlapping onDeckSearchers

[Adrian Liew]

Thanks Shawn. Having said that increasing maxWarmingSearchers is usually wrong 
to solve this, are there any implications if we set maxWarmingSearchers to zero 
to resolve this problem?

Or do you think there are some other settings that are worthwhile tuning to 
cater to the above?

Best regards,
Adrian 

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Tuesday, August 11, 2015 11:02 PM
To: solr-user@lucene.apache.org
Subject: Re: Performance warning overlapping onDeckSearchers

On 8/11/2015 3:02 AM, Adrian Liew wrote:
> Has anyone come across this issue, [some_index] PERFORMANCE WARNING: 
> Overlapping onDeckSearchers=2?
> 
> I am currently using Solr v5.2.1.
> 
> What does this mean? Does this raise red flags?
> 
> I am currently encountering an issue whereby my Sitecore system is unable to 
> update the index appropriately. I am not sure if this is linked to the 
> warnings above.

https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F

What the wiki page doesn't explicitly state is that increasing 
maxWarmingSearchers is usually the wrong way to solve this, because that can 
actually make the problem *worse*.  It is implied by the things the page DOES 
say, but it is not stated.

Thanks,
Shawn

RE: Performance warning overlapping onDeckSearchers

[Adrian Liew]

Additionally,

I realized that my autowarmCount is set to zero for the following Cache entries 
except perSegFilter:





   



 



The link, 
https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F
 did suggest to reduce the autowarmCount or reduce warm up cache activity 
(which I am not sure where to begin doing this).

I suspect autowarmCount is not very large as the above. 

Let me know what you think.

Best regards,
Adrian Liew


-Original Message-
From: Adrian Liew [mailto:adrian.l...@avanade.com] 
Sent: Wednesday, August 12, 2015 3:32 PM
To: solr-user@lucene.apache.org
Subject: RE: Performance warning overlapping onDeckSearchers

Thanks Shawn. Having said that increasing maxWarmingSearchers is usually wrong 
to solve this, are there any implications if we set maxWarmingSearchers to zero 
to resolve this problem?

Or do you think there are some other settings that are worthwhile tuning to 
cater to the above?

Best regards,
Adrian 

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Tuesday, August 11, 2015 11:02 PM
To: solr-user@lucene.apache.org
Subject: Re: Performance warning overlapping onDeckSearchers

On 8/11/2015 3:02 AM, Adrian Liew wrote:
> Has anyone come across this issue, [some_index] PERFORMANCE WARNING: 
> Overlapping onDeckSearchers=2?
> 
> I am currently using Solr v5.2.1.
> 
> What does this mean? Does this raise red flags?
> 
> I am currently encountering an issue whereby my Sitecore system is unable to 
> update the index appropriately. I am not sure if this is linked to the 
> warnings above.

https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F

What the wiki page doesn't explicitly state is that increasing 
maxWarmingSearchers is usually the wrong way to solve this, because that can 
actually make the problem *worse*.  It is implied by the things the page DOES 
say, but it is not stated.

Thanks,
Shawn

Re: Deadlock-like behavior when new IndexWriter created

[Andrii Berezhynskyi]

Sure, Eric. Here it is


Attaching to process ID 9429, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.51-b03
Deadlock Detection:

No deadlocks found.

Thread 9578: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 9436: (state = BLOCKED)


Thread 9568: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
 - org.apache.solr.core.CloserThread.run() @bci=27, line=1037 (Interpreted
frame)


Thread 9566: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 -
org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(org.apache.solr.core.SolrCore,
boolean) @bci=89, line=157 (Interpreted frame)
 - org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(boolean)
@bci=9, line=665 (Interpreted frame)
 - org.apache.solr.handler.IndexFetcher.fetchLatestIndex(boolean, boolean)
@bci=1530, line=489 (Compiled frame)
 - org.apache.solr.handler.IndexFetcher.fetchLatestIndex(boolean) @bci=3,
line=254 (Compiled frame)
 -
org.apache.solr.handler.ReplicationHandler.doFetch(org.apache.solr.common.params.SolrParams,
boolean) @bci=92, line=380 (Compiled frame)
 - org.apache.solr.handler.ReplicationHandler$2.run() @bci=51, line=1037
(Compiled frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
 - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308
(Compiled frame)
 -
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
@bci=1, line=180 (Compiled frame)
 -
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
@bci=37, line=294 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 9560: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
@bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 9559: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Interpreted frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 9548: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.la

Solr MLT Interestingterms return different terms than Lucene MoreLikeThis for some of the documents

[Ali Nazemian]

Hi,

I am going to implement a searchcomponent for Solr to return document main
keywords with using the more like this interesting terms. The main part of
implemented component which uses mlt.retrieveInterestingTerms by lucene
docID does not work for all of the documents. I mean for some of the
documents solr interestingterms returns some useful terms as top tf-idf
terms; however, the implemented method returns null! But for other
documents both results (solr MLT interesting terms and the
mlt.retrieveInterestingTerms(docId)) are the same! Would you please help me
through solving this issue?

public List getKeywords(int docId) throws SyntaxError {
String[] fields = new String[keywordSourceFields.size()];
List terms = new ArrayList();
fields = keywordSourceFields.toArray(fields);
mlt.setFieldNames(fields);
mlt.setAnalyzer(indexSearcher.getSchema().getIndexAnalyzer());
mlt.setMinTermFreq(minTermFreq);
mlt.setMinDocFreq(minDocFreq);
mlt.setMinWordLen(minWordLen);
mlt.setMaxQueryTerms(maxNumKeywords);
mlt.setMaxNumTokensParsed(maxTokensParsed);
try {

  terms = Arrays.asList(mlt.retrieveInterestingTerms(docId));
} catch (IOException e) {
  LOGGER.error(e.getMessage());
  throw new RuntimeException();
}

return terms;
  }

*Note:*

I did define termVectors=true for all the required fields that I am going
to use for the purpose of generating interesting terms (fields array in the
corresponding method)
Best regards.
-- 

A.Nazemian

RE: High CPU DistributedQueue and OverseerAutoReplicaFailoverThread

[Markus Jelsma]

Hi - anyone to share some hints on the topic?

Regards,
M. 
 
-Original message-
> From:Markus Jelsma 
> Sent: Wednesday 5th August 2015 12:13
> To: solr-user 
> Subject: High CPU DistributedQueue and OverseerAutoReplicaFailoverThread
> 
> Hello  - we have a single Solr 5.2.1 node that (for now) contains four single 
> shard collections. Only two collections actually contain data and are 
> queried. The machine has some unusual latency that led me to sample the CPU 
> time with VisualVM. On that node we see that 
> DistributedQueue$LatchWachter.await() and OverseerAutoReplicaFailoverThread() 
> claiming a lot of CPU time, DistributedQueue takes the most of it. I have set 
> logging to DEBUG but no Zookeeper logging is printed, the connections are 
> stable and updates rarely come in. 
> 
> Is it because the node runs the overseer? If so, how can i prevent it from 
> taking quite a lot of CPU time on a node with four collections where no state 
> changes occur?
> 
> Any other thoughts?
> Markus
> 

Re: Performance warning overlapping onDeckSearchers

[Joel Bernstein]

You'll want to check to see if there are any static warming queries being
run as well.

How often are you committing and opening a new searcher? Are you using
autoCommits or explicitly committing?

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Aug 12, 2015 at 3:57 AM, Adrian Liew 
wrote:

> Additionally,
>
> I realized that my autowarmCount is set to zero for the following Cache
> entries except perSegFilter:
>
>   size="512"
>  initialSize="512"
>  autowarmCount="0"/>
>
> 
>   size="512"
>  initialSize="512"
>  autowarmCount="0"/>
>
> 
> size="512"
>initialSize="512"
>autowarmCount="0"/>
>
> 
>class="solr.search.LRUCache"
>   size="10"
>   initialSize="0"
>   autowarmCount="10"
>   regenerator="solr.NoOpRegenerator" />
>
>
> The link,
> https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F
> did suggest to reduce the autowarmCount or reduce warm up cache activity
> (which I am not sure where to begin doing this).
>
> I suspect autowarmCount is not very large as the above.
>
> Let me know what you think.
>
> Best regards,
> Adrian Liew
>
>
> -Original Message-
> From: Adrian Liew [mailto:adrian.l...@avanade.com]
> Sent: Wednesday, August 12, 2015 3:32 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Performance warning overlapping onDeckSearchers
>
> Thanks Shawn. Having said that increasing maxWarmingSearchers is usually
> wrong to solve this, are there any implications if we set
> maxWarmingSearchers to zero to resolve this problem?
>
> Or do you think there are some other settings that are worthwhile tuning
> to cater to the above?
>
> Best regards,
> Adrian
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Tuesday, August 11, 2015 11:02 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Performance warning overlapping onDeckSearchers
>
> On 8/11/2015 3:02 AM, Adrian Liew wrote:
> > Has anyone come across this issue, [some_index] PERFORMANCE WARNING:
> Overlapping onDeckSearchers=2?
> >
> > I am currently using Solr v5.2.1.
> >
> > What does this mean? Does this raise red flags?
> >
> > I am currently encountering an issue whereby my Sitecore system is
> unable to update the index appropriately. I am not sure if this is linked
> to the warnings above.
>
>
> https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F
>
> What the wiki page doesn't explicitly state is that increasing
> maxWarmingSearchers is usually the wrong way to solve this, because that
> can actually make the problem *worse*.  It is implied by the things the
> page DOES say, but it is not stated.
>
> Thanks,
> Shawn
>
>

Re: Highlighting

[Scott Derrick]

I was pretty sure I tried that, though I thought if you don't specify it 
just uses the search terms?


If I just search for "calvin" and don't specify a field, what do I 
assign hl.fl?


Scott

On 8/11/2015 7:27 PM, Erik Hatcher wrote:

Scott - doesn’t look you’ve specified hl.fl specifying which field(s) to 
highlight.

p.s. Erick Erickson surely likes your e-mail domain :)


—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com 





On Aug 11, 2015, at 9:02 PM, Scott Derrick  wrote:

I guess I really don't get Highlighting in Solr.

We are transitioning from Google Custom Search which generally sucks, but does 
return nicely formatted highlighted fragment.

I turn highlighting on hl=true in the query and I get a highlighting section 
returned at the bottom of the page, each identified by the document file name 
with a empty {} .  It doesn't matter what I search for, plain text, a field, I 
get a list of documents followed by an empty brace?

"highlighting": {
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10385B/A10385B.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10089/A10089.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./L3/L3.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10646/A10646.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./V03482/V03482.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10594/A10594.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./645A.66.043/645A.66.043.html":
 {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./352.48.001/352.48.001.html":
 {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./144.23.001/144.23.001.html":
 {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./L18512/L18512.html": {}
  }

I haven't made any changes to the default settings

   
  
  
  

  100

  

  
  

  
  70
  
  0.5
  
  [-\w ,/\n\"']{20,200}

  

  
  

  
  

  

  
  

  
  

  
  

  
  

  
  

  

  
  

  
  

  

  

  10
  .,!? 	


  

  

  
  WORD
  
  
  en
  US

  






---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Highlighting

[Scott Derrick]

yeah, I'm partial to it too! :-)

On 8/11/2015 7:29 PM, Erick Erickson wrote:

bq: Erick Erickson surely likes your e-mail domain :)

Yep, I envy that one!

On Tue, Aug 11, 2015 at 6:27 PM, Erik Hatcher  wrote:

Scott - doesn’t look you’ve specified hl.fl specifying which field(s) to 
highlight.

p.s. Erick Erickson surely likes your e-mail domain :)


—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com 





On Aug 11, 2015, at 9:02 PM, Scott Derrick  wrote:

I guess I really don't get Highlighting in Solr.

We are transitioning from Google Custom Search which generally sucks, but does 
return nicely formatted highlighted fragment.

I turn highlighting on hl=true in the query and I get a highlighting section 
returned at the bottom of the page, each identified by the document file name 
with a empty {} .  It doesn't matter what I search for, plain text, a field, I 
get a list of documents followed by an empty brace?

"highlighting": {
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10385B/A10385B.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10089/A10089.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./L3/L3.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10646/A10646.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./V03482/V03482.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10594/A10594.html": {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./645A.66.043/645A.66.043.html":
 {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./352.48.001/352.48.001.html":
 {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./144.23.001/144.23.001.html":
 {},
"/home/scott/workspace/mbel-work/tei2html/build/web/./L18512/L18512.html": {}
  }

I haven't made any changes to the default settings

   
  
  
  

  100

  

  
  

  
  70
  
  0.5
  
  [-\w ,/\n\"']{20,200}

  

  
  

  
  

  

  
  

  
  

  
  

  
  

  
  

  

  
  

  
  

  

  

  10
  .,!? 	


  

  

  
  WORD
  
  
  en
  US

  




---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Highlighting, all matches show empty {}

[Scott Derrick]

Tried submitting a filed for hl.fl still empty {}

here are the query terms

"responseHeader": {
"status": 0,
"QTime": 8,
"params": {
  "q": "mary or calvin",
  "hl": "true",
  "hl.simple.post": "",
  "indent": "true",
  "fl": "accession, title, author, date",
  "hl.fl": "*",
  "wt": "json",
  "hl.simple.pre": "",
  "_": "1439388969240"
}

here is one of the responses, there were 135

{
"date": "1886-07-06",
"author": "Mary Baker Eddy",
"accession": "L02634",
"title": [
  "Mary Baker Eddy to Josephine C. Woodbury, July 6, 1886"
]
},

here is the highlight section listing the first 10 matches, still empty {}

"highlighting": {

"/home/scott/workspace/mbel-work/tei2html/build/web/./L02634/L02634.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./A10720/A10720.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./L07894/L07894.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./L09828/L09828.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./A10636D/A10636D.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./L13943/L13943.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./A10594/A10594.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./A10385B/A10385B.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./A10879/A10879.html": 
{},


"/home/scott/workspace/mbel-work/tei2html/build/web/./L3/L3.html": 
{}

  }


 Original Message 
Subject: Re: Highlighting
From: Scott Derrick 
To: solr-user@lucene.apache.org
Date: 08/12/2015 06:39 AM


I was pretty sure I tried that, though I thought if you don't specify it
just uses the search terms?

If I just search for "calvin" and don't specify a field, what do I
assign hl.fl?

Scott

On 8/11/2015 7:27 PM, Erik Hatcher wrote:

Scott - doesn’t look you’ve specified hl.fl specifying which field(s)
to highlight.

p.s. Erick Erickson surely likes your e-mail domain :)


—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com 





On Aug 11, 2015, at 9:02 PM, Scott Derrick  wrote:

I guess I really don't get Highlighting in Solr.

We are transitioning from Google Custom Search which generally sucks,
but does return nicely formatted highlighted fragment.

I turn highlighting on hl=true in the query and I get a highlighting
section returned at the bottom of the page, each identified by the
document file name with a empty {} .  It doesn't matter what I search
for, plain text, a field, I get a list of documents followed by an
empty brace?

"highlighting": {
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10385B/A10385B.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10089/A10089.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./L3/L3.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10646/A10646.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./V03482/V03482.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./A10594/A10594.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./645A.66.043/645A.66.043.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./352.48.001/352.48.001.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./144.23.001/144.23.001.html":
{},
"/home/scott/workspace/mbel-work/tei2html/build/web/./L18512/L18512.html":
{}
  }

I haven't made any changes to the default settings

   
  
  
  

  100

  

  
  

  
  70
  
  0.5
  
  [-\w
,/\n\"']{20,200}

  

  
  

  
  

  

  
  

  
  

  
  

  
  

  
  

  

  
  

  
  

  

  

  10
  .,!? 	


  

  

  
  WORD
  
  
  en
  US

  






---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus




--
I do not want the peace which passeth understanding, I want the 
understanding which bringeth peace.

Helen Keller

Re: Highlighting, all matches show empty {}

[Scott Derrick]

I think the highlighter is actually running, but I'm not getting the 
results??


with this request

http://localhost:8983/solr/mbepp/select?q=concord&fl=accession%2C+title%2C+author%2C+date&wt=json&indent=true&hl=true&hl.fl=*


I get this response

{
  "responseHeader":{
"status":0,
"QTime":3,
"params":{
  "q":"concord",
  "hl":"true",
  "indent":"true",
  "fl":"accession, title, author, date",
  "hl.fl":"*",
  "wt":"json"}},
  "response":{"numFound":3,"start":0,"docs":[
  {
"date":"1890-02-26",
"author":"Mary Baker Eddy",
"accession":"L13943",
"title":["Mary Baker Eddy to Joseph E. Adams,"]},
  {
"date":"1896-01-13",
"author":"Mary Baker Eddy",
"accession":"L03453",
"title":["Mary Baker Eddy to Ira O. Knapp,"]},
  {
"date":"1902-06-15",
"author":"Mary Baker Eddy",
"accession":"A10145",
"title":["Message of the Pastor Emeritus to The First Church of 
Christ, Scientist, Boston, Mass., June 15, 1902"]}]

  },
  "highlighting":{

"/home/scott/workspace/mbel-work/tei2html/build/web/L13943/L13943.html":{},

"/home/scott/workspace/mbel-work/tei2html/build/web/L03453/L03453.html":{},

"/home/scott/workspace/mbel-work/tei2html/build/web/A10145/A10145.html":{}}}

When I ran the request.
In the admin plubins/Stats I set "Watch Changes" before processing the 
request.  Highlighting showed 2 changes, the gapFragmenter and HTMLFormatter


here are the reported changes

org.apache.solr.highlight.GapFragmenter
class: org.apache.solr.highlight.GapFragmenter
version: 5.2.1
description: GapFragmenter
stats: requests: Was: 117, Now: 156, Delta: 39

org.apache.solr.highlight.HtmlFormatter
class: org.apache.solr.highlight.HtmlFormatter
version:5.2.1
description:HtmlFormatter
stats: requests: Was: 117, Now: 156, Delta: 39

Looks to me like there were 39 fragments or something processed, yet you 
can see above the highlights are empty {}???


though all the the other libraries in the highlighter showed no changes.

which are these...

org.apache.solr.highlight.BreakIteratorBoundaryScanner
org.apache.solr.highlight.HtmlEncoder
org.apache.solr.highlight.RegexFragmenter
org.apache.solr.highlight.ScoreOrderFragmentsBuilder
org.apache.solr.highlight.SimpleBoundaryScanner
org.apache.solr.highlight.SimpleFragListBuilder
org.apache.solr.highlight.SingleFragListBuilder
org.apache.solr.highlight.WeightedFragListBuilder


Scott

 Original Message 
Subject: Highlighting, all matches show empty {}
From: Scott Derrick 
To: solr-user@lucene.apache.org
Date: 08/12/2015 08:20 AM


Tried submitting a filed for hl.fl still empty {}

here are the query terms

"responseHeader": {
 "status": 0,
 "QTime": 8,
 "params": {
   "q": "mary or calvin",
   "hl": "true",
   "hl.simple.post": "",
   "indent": "true",
   "fl": "accession, title, author, date",
   "hl.fl": "*",
   "wt": "json",
   "hl.simple.pre": "",
   "_": "1439388969240"
 }

here is one of the responses, there were 135

{
 "date": "1886-07-06",
 "author": "Mary Baker Eddy",
 "accession": "L02634",
 "title": [
   "Mary Baker Eddy to Josephine C. Woodbury, July 6, 1886"
 ]
},

here is the highlight section listing the first 10 matches, still empty {}

"highlighting": {

"/home/scott/workspace/mbel-work/tei2html/build/web/./L02634/L02634.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./A10720/A10720.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./L07894/L07894.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./L09828/L09828.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./A10636D/A10636D.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./L13943/L13943.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./A10594/A10594.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./A10385B/A10385B.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./A10879/A10879.html":
{},

"/home/scott/workspace/mbel-work/tei2html/build/web/./L3/L3.html":
{}
   }


 Original Message 
Subject: Re: Highlighting
From: Scott Derrick 
To: solr-user@lucene.apache.org
Date: 08/12/2015 06:39 AM


I was pretty sure I tried that, though I thought if you don't specify it
just uses the search terms?

If I just search for "calvin" and don't specify a field, what do I
assign hl.fl?

Scott

On 8/11/2015 7:27 PM, Erik Hatcher wrote:

Scott - doesn’t look you’ve specified hl.fl specifying which field(s)
to highlight.

p.s. Erick Erickson surely likes your e-mail domain :)


—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com 





On Aug 11, 2015, at 9:02 PM, Scott Derrick  wrote:

I guess I really don't get Highlighting in Solr.

We

Solr search with AND and OR

[meena.sri...@mathworks.com]

I am using Solrj as solr client 
We have a user query like 
"XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server :0"
AND "after 109"

or
"XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server :0"
"after 109"

currently 

if (syndrome.startsWith("\"") && syndrome.endsWith("\"")) {
syndrome = syndrome.replaceAll("\"", "");
syndrome = QueryParser.escape(syndrome);
syndrome = "\"" + syndrome + "\"";

}

is the code which escapes the query, but the problem with this is it removes
the double quotes before escaping and inserts it after escaping which
removes the double quote for AND query and its gets searched for 

"XIO\:++fatal+IO+error+11+\(Resource+temporarily+unavailable\)+on+X+server+\:0+AND+after+109"

This will result in wrong results 
Is there a war to interpret this request correctly and send it to SOLR. 
I was looking if there are any query parsers I need to use ? Any suggestions
would be useful


Thanks
Meena






--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-search-with-AND-and-OR-tp4222618.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Highlighting, all matches show empty {}

[Erick Erickson]

Well, the example you just showed shouldn't show any highlighting. Your query is
q=concord
so it's trying to highlight "concord" which isn't in any of your
documents. hl.q can be
used to highlight something other than your q parameter.

I did notice in some of your other examples that you seemed to be searching for
terms that were in the fields so I suspect this isn't really your root
problem though.

do note that fields _must_ be stored to have highlighting work. Is it
possible that your
matches are on fields that aren't stored?

Let's build it up slowly though, try searching on one term in one
field that you _know_
is stored and see if you get anything back. While the query with
hl.fl=* and fl=field1, field2,
should be fine, let's start as simply as possible and work up maybe?

Best,
Erick

On Wed, Aug 12, 2015 at 7:59 AM, Scott Derrick  wrote:
> I think the highlighter is actually running, but I'm not getting the
> results??
>
> with this request
>
> http://localhost:8983/solr/mbepp/select?q=concord&fl=accession%2C+title%2C+author%2C+date&wt=json&indent=true&hl=true&hl.fl=*
>
>
> I get this response
>
> {
>   "responseHeader":{
> "status":0,
> "QTime":3,
> "params":{
>   "q":"concord",
>   "hl":"true",
>   "indent":"true",
>   "fl":"accession, title, author, date",
>   "hl.fl":"*",
>   "wt":"json"}},
>   "response":{"numFound":3,"start":0,"docs":[
>   {
> "date":"1890-02-26",
> "author":"Mary Baker Eddy",
> "accession":"L13943",
> "title":["Mary Baker Eddy to Joseph E. Adams,"]},
>   {
> "date":"1896-01-13",
> "author":"Mary Baker Eddy",
> "accession":"L03453",
> "title":["Mary Baker Eddy to Ira O. Knapp,"]},
>   {
> "date":"1902-06-15",
> "author":"Mary Baker Eddy",
> "accession":"A10145",
> "title":["Message of the Pastor Emeritus to The First Church of
> Christ, Scientist, Boston, Mass., June 15, 1902"]}]
>   },
>   "highlighting":{
>
> "/home/scott/workspace/mbel-work/tei2html/build/web/L13943/L13943.html":{},
>
> "/home/scott/workspace/mbel-work/tei2html/build/web/L03453/L03453.html":{},
>
> "/home/scott/workspace/mbel-work/tei2html/build/web/A10145/A10145.html":{}}}
>
> When I ran the request.
> In the admin plubins/Stats I set "Watch Changes" before processing the
> request.  Highlighting showed 2 changes, the gapFragmenter and HTMLFormatter
>
> here are the reported changes
>
> org.apache.solr.highlight.GapFragmenter
> class: org.apache.solr.highlight.GapFragmenter
> version: 5.2.1
> description: GapFragmenter
> stats: requests: Was: 117, Now: 156, Delta: 39
>
> org.apache.solr.highlight.HtmlFormatter
> class: org.apache.solr.highlight.HtmlFormatter
> version:5.2.1
> description:HtmlFormatter
> stats: requests: Was: 117, Now: 156, Delta: 39
>
> Looks to me like there were 39 fragments or something processed, yet you can
> see above the highlights are empty {}???
>
> though all the the other libraries in the highlighter showed no changes.
>
> which are these...
>
> org.apache.solr.highlight.BreakIteratorBoundaryScanner
> org.apache.solr.highlight.HtmlEncoder
> org.apache.solr.highlight.RegexFragmenter
> org.apache.solr.highlight.ScoreOrderFragmentsBuilder
> org.apache.solr.highlight.SimpleBoundaryScanner
> org.apache.solr.highlight.SimpleFragListBuilder
> org.apache.solr.highlight.SingleFragListBuilder
> org.apache.solr.highlight.WeightedFragListBuilder
>
>
> Scott
>
>  Original Message 
> Subject: Highlighting, all matches show empty {}
> From: Scott Derrick 
> To: solr-user@lucene.apache.org
> Date: 08/12/2015 08:20 AM
>
>> Tried submitting a filed for hl.fl still empty {}
>>
>> here are the query terms
>>
>> "responseHeader": {
>>  "status": 0,
>>  "QTime": 8,
>>  "params": {
>>"q": "mary or calvin",
>>"hl": "true",
>>"hl.simple.post": "",
>>"indent": "true",
>>"fl": "accession, title, author, date",
>>"hl.fl": "*",
>>"wt": "json",
>>"hl.simple.pre": "",
>>"_": "1439388969240"
>>  }
>>
>> here is one of the responses, there were 135
>>
>> {
>>  "date": "1886-07-06",
>>  "author": "Mary Baker Eddy",
>>  "accession": "L02634",
>>  "title": [
>>"Mary Baker Eddy to Josephine C. Woodbury, July 6, 1886"
>>  ]
>> },
>>
>> here is the highlight section listing the first 10 matches, still empty {}
>>
>> "highlighting": {
>>
>> "/home/scott/workspace/mbel-work/tei2html/build/web/./L02634/L02634.html":
>> {},
>>
>> "/home/scott/workspace/mbel-work/tei2html/build/web/./A10720/A10720.html":
>> {},
>>
>> "/home/scott/workspace/mbel-work/tei2html/build/web/./L07894/L07894.html":
>> {},
>>
>> "/home/scott/workspace/mbel-work/tei2html/build/web/./L09828/L09828.html":
>> {},
>>
>>
>> "/home/scott/workspace/mbel-work/tei2html/build/web/./A10636D/A10636D.h

Re: Deadlock-like behavior when new IndexWriter created

[Erick Erickson]

Hmmm, thanks. This line:

org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(org.apache.solr.core.SolrCore,
boolean) @bci=89, line=157 (Interpreted frame)

is suspiciously in the region of SOLR-7836 but the rest of the
precursors for that JIRA aren't
present (i.e. Delete By Query and/or reloading a core) so I'm a bit puzzled.

It's vaguely possible that 7836 will fix this, but we're still
wrangling over the right way to fix it.
You might want to track that JIRA...

Erick

On Wed, Aug 12, 2015 at 12:59 AM, Andrii Berezhynskyi
 wrote:
> Sure, Eric. Here it is
>
>
> Attaching to process ID 9429, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 25.51-b03
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 9578: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
> @bci=20, line=215 (Compiled frame)
>  -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=78, line=2078 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=124, line=1093 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=809 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
> (Compiled frame)
>  -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=26, line=1127 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
>
>
> Thread 9436: (state = BLOCKED)
>
>
> Thread 9568: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
>  - org.apache.solr.core.CloserThread.run() @bci=27, line=1037 (Interpreted
> frame)
>
>
> Thread 9566: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
> imprecise)
>  -
> org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(org.apache.solr.core.SolrCore,
> boolean) @bci=89, line=157 (Interpreted frame)
>  - org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(boolean)
> @bci=9, line=665 (Interpreted frame)
>  - org.apache.solr.handler.IndexFetcher.fetchLatestIndex(boolean, boolean)
> @bci=1530, line=489 (Compiled frame)
>  - org.apache.solr.handler.IndexFetcher.fetchLatestIndex(boolean) @bci=3,
> line=254 (Compiled frame)
>  -
> org.apache.solr.handler.ReplicationHandler.doFetch(org.apache.solr.common.params.SolrParams,
> boolean) @bci=92, line=380 (Compiled frame)
>  - org.apache.solr.handler.ReplicationHandler$2.run() @bci=51, line=1037
> (Compiled frame)
>  - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
> (Compiled frame)
>  - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308
> (Compiled frame)
>  -
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> @bci=1, line=180 (Compiled frame)
>  -
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
> @bci=37, line=294 (Compiled frame)
>  -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1142 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
>
>
> Thread 9560: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
> @bci=20, line=215 (Compiled frame)
>  -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=78, line=2078 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=124, line=1093 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=809 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
> (Compiled frame)
>  -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=26, line=1127 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
>
>
> Thread 9559: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=175 (Compiled frame)
>  -
> java.util.concurrent.locks.Abstrac

Re: Highlighting, all matches show empty {}

[Scott Derrick]

Erick,

that explains it. I figured I didn't understand how solr handled 
highlight fragments.


Most of my documents are just text. or as solr specifies that content 
_text_, which is not stored, by default.


You mention the I was searching for concord and that its not in any 
documents.  But the results below clearly show 3 hits


>>"response":{"numFound":3,"start":0,"docs":[

the problem is the hits are in _text_

Is there a problem with storing _text_  so I can get a highlight 
fragment when a hit is found there?


Scott

 Original Message 
Subject: Re: Highlighting, all matches show empty {}
From: Erick Erickson 
To: solr-user@lucene.apache.org
Date: 08/12/2015 09:27 AM


Well, the example you just showed shouldn't show any highlighting. Your query is
q=concord
so it's trying to highlight "concord" which isn't in any of your
documents. hl.q can be
used to highlight something other than your q parameter.

I did notice in some of your other examples that you seemed to be searching for
terms that were in the fields so I suspect this isn't really your root
problem though.

do note that fields _must_ be stored to have highlighting work. Is it
possible that your
matches are on fields that aren't stored?

Let's build it up slowly though, try searching on one term in one
field that you _know_
is stored and see if you get anything back. While the query with
hl.fl=* and fl=field1, field2,
should be fine, let's start as simply as possible and work up maybe?

Best,
Erick

On Wed, Aug 12, 2015 at 7:59 AM, Scott Derrick  wrote:

I think the highlighter is actually running, but I'm not getting the
results??

with this request

http://localhost:8983/solr/mbepp/select?q=concord&fl=accession%2C+title%2C+author%2C+date&wt=json&indent=true&hl=true&hl.fl=*


I get this response

{
   "responseHeader":{
 "status":0,
 "QTime":3,
 "params":{
   "q":"concord",
   "hl":"true",
   "indent":"true",
   "fl":"accession, title, author, date",
   "hl.fl":"*",
   "wt":"json"}},
   "response":{"numFound":3,"start":0,"docs":[
   {
 "date":"1890-02-26",
 "author":"Mary Baker Eddy",
 "accession":"L13943",
 "title":["Mary Baker Eddy to Joseph E. Adams,"]},
   {
 "date":"1896-01-13",
 "author":"Mary Baker Eddy",
 "accession":"L03453",
 "title":["Mary Baker Eddy to Ira O. Knapp,"]},
   {
 "date":"1902-06-15",
 "author":"Mary Baker Eddy",
 "accession":"A10145",
 "title":["Message of the Pastor Emeritus to The First Church of
Christ, Scientist, Boston, Mass., June 15, 1902"]}]
   },
   "highlighting":{

"/home/scott/workspace/mbel-work/tei2html/build/web/L13943/L13943.html":{},

"/home/scott/workspace/mbel-work/tei2html/build/web/L03453/L03453.html":{},

"/home/scott/workspace/mbel-work/tei2html/build/web/A10145/A10145.html":{}}}

When I ran the request.
In the admin plubins/Stats I set "Watch Changes" before processing the
request.  Highlighting showed 2 changes, the gapFragmenter and HTMLFormatter

here are the reported changes

org.apache.solr.highlight.GapFragmenter
 class: org.apache.solr.highlight.GapFragmenter
 version: 5.2.1
 description: GapFragmenter
 stats: requests: Was: 117, Now: 156, Delta: 39

org.apache.solr.highlight.HtmlFormatter
 class: org.apache.solr.highlight.HtmlFormatter
 version:5.2.1
 description:HtmlFormatter
 stats: requests: Was: 117, Now: 156, Delta: 39

Looks to me like there were 39 fragments or something processed, yet you can
see above the highlights are empty {}???

though all the the other libraries in the highlighter showed no changes.

which are these...

 org.apache.solr.highlight.BreakIteratorBoundaryScanner
 org.apache.solr.highlight.HtmlEncoder
 org.apache.solr.highlight.RegexFragmenter
 org.apache.solr.highlight.ScoreOrderFragmentsBuilder
 org.apache.solr.highlight.SimpleBoundaryScanner
 org.apache.solr.highlight.SimpleFragListBuilder
 org.apache.solr.highlight.SingleFragListBuilder
 org.apache.solr.highlight.WeightedFragListBuilder


Scott

 Original Message 
Subject: Highlighting, all matches show empty {}
From: Scott Derrick 
To: solr-user@lucene.apache.org
Date: 08/12/2015 08:20 AM


Tried submitting a filed for hl.fl still empty {}

here are the query terms

"responseHeader": {
  "status": 0,
  "QTime": 8,
  "params": {
"q": "mary or calvin",
"hl": "true",
"hl.simple.post": "",
"indent": "true",
"fl": "accession, title, author, date",
"hl.fl": "*",
"wt": "json",
"hl.simple.pre": "",
"_": "1439388969240"
  }

here is one of the responses, there were 135

{
  "date": "1886-07-06",
  "author": "Mary Baker Eddy",
  "accession": "L02634",
  "title": [
"Mary Baker Eddy to Josephine C. Woodbury, July 6, 1886"
  ]
},

here i

Re: Highlighting, all matches show empty {}

[Erick Erickson]

bq: You mention the I was searching for concord and that its not in
any documents.  But the results below clearly show 3 hits

Right, as you figured out I _really_ meant "concord in any stored
fields you were including in the hl.fl parameter". That could have
been clearer.

bq: Is there a problem with storing _text_  so I can get a highlight
fragment when a hit is found there?

No, you can store the data in the _text_ field just fine, you'll have
to re-index after the change though. It's often more useful to a user
to see the highlights in specific fields though, so I wouldn't throw
the rest of the highlighting away.

You should probably see the FastVectorHighlighter though. If you don't
use FVH, highlighting re-analyzes the raw text to produce the snippets
which may be expensive for large text fields.

Best,
Erick


On Wed, Aug 12, 2015 at 8:46 AM, Scott Derrick  wrote:
> Erick,
>
> that explains it. I figured I didn't understand how solr handled highlight
> fragments.
>
> Most of my documents are just text. or as solr specifies that content
> _text_, which is not stored, by default.
>
> You mention the I was searching for concord and that its not in any
> documents.  But the results below clearly show 3 hits
>
>>>"response":{"numFound":3,"start":0,"docs":[
>
> the problem is the hits are in _text_
>
> Is there a problem with storing _text_  so I can get a highlight fragment
> when a hit is found there?
>
> Scott
>
>  Original Message 
> Subject: Re: Highlighting, all matches show empty {}
> From: Erick Erickson 
> To: solr-user@lucene.apache.org
> Date: 08/12/2015 09:27 AM
>
>> Well, the example you just showed shouldn't show any highlighting. Your
>> query is
>> q=concord
>> so it's trying to highlight "concord" which isn't in any of your
>> documents. hl.q can be
>> used to highlight something other than your q parameter.
>>
>> I did notice in some of your other examples that you seemed to be
>> searching for
>> terms that were in the fields so I suspect this isn't really your root
>> problem though.
>>
>> do note that fields _must_ be stored to have highlighting work. Is it
>> possible that your
>> matches are on fields that aren't stored?
>>
>> Let's build it up slowly though, try searching on one term in one
>> field that you _know_
>> is stored and see if you get anything back. While the query with
>> hl.fl=* and fl=field1, field2,
>> should be fine, let's start as simply as possible and work up maybe?
>>
>> Best,
>> Erick
>>
>> On Wed, Aug 12, 2015 at 7:59 AM, Scott Derrick  wrote:
>>>
>>> I think the highlighter is actually running, but I'm not getting the
>>> results??
>>>
>>> with this request
>>>
>>>
>>> http://localhost:8983/solr/mbepp/select?q=concord&fl=accession%2C+title%2C+author%2C+date&wt=json&indent=true&hl=true&hl.fl=*
>>>
>>>
>>> I get this response
>>>
>>> {
>>>"responseHeader":{
>>>  "status":0,
>>>  "QTime":3,
>>>  "params":{
>>>"q":"concord",
>>>"hl":"true",
>>>"indent":"true",
>>>"fl":"accession, title, author, date",
>>>"hl.fl":"*",
>>>"wt":"json"}},
>>>"response":{"numFound":3,"start":0,"docs":[
>>>{
>>>  "date":"1890-02-26",
>>>  "author":"Mary Baker Eddy",
>>>  "accession":"L13943",
>>>  "title":["Mary Baker Eddy to Joseph E. Adams,"]},
>>>{
>>>  "date":"1896-01-13",
>>>  "author":"Mary Baker Eddy",
>>>  "accession":"L03453",
>>>  "title":["Mary Baker Eddy to Ira O. Knapp,"]},
>>>{
>>>  "date":"1902-06-15",
>>>  "author":"Mary Baker Eddy",
>>>  "accession":"A10145",
>>>  "title":["Message of the Pastor Emeritus to The First Church of
>>> Christ, Scientist, Boston, Mass., June 15, 1902"]}]
>>>},
>>>"highlighting":{
>>>
>>>
>>> "/home/scott/workspace/mbel-work/tei2html/build/web/L13943/L13943.html":{},
>>>
>>>
>>> "/home/scott/workspace/mbel-work/tei2html/build/web/L03453/L03453.html":{},
>>>
>>>
>>> "/home/scott/workspace/mbel-work/tei2html/build/web/A10145/A10145.html":{}}}
>>>
>>> When I ran the request.
>>> In the admin plubins/Stats I set "Watch Changes" before processing the
>>> request.  Highlighting showed 2 changes, the gapFragmenter and
>>> HTMLFormatter
>>>
>>> here are the reported changes
>>>
>>> org.apache.solr.highlight.GapFragmenter
>>>  class: org.apache.solr.highlight.GapFragmenter
>>>  version: 5.2.1
>>>  description: GapFragmenter
>>>  stats: requests: Was: 117, Now: 156, Delta: 39
>>>
>>> org.apache.solr.highlight.HtmlFormatter
>>>  class: org.apache.solr.highlight.HtmlFormatter
>>>  version:5.2.1
>>>  description:HtmlFormatter
>>>  stats: requests: Was: 117, Now: 156, Delta: 39
>>>
>>> Looks to me like there were 39 fragments or something processed, yet you
>>> can
>>> see above the highlights are empty {}???
>>>
>>> though all the the other libraries in the highlighter showed no changes.
>>>
>>> which are these

Geospatial Predicate Question

[Jamie Johnson]

Can someone clarify the difference between isWithin and Contains in regards
to Solr's spatial support?  From
https://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4 I see that if
you are using point data you should use Intersects, but it is not clear
when to use isWithin and contains.  My guess is that you use isWithin when
you want to know if the query shape is within the shape that is indexed and
you use contains to know if the query shape contains the indexed shape.  Is
that right?

Indexed & stored

[Nagasharath]

As stored & indexed both default to "true" do we still need to specify in the 
schema as

Indexed="true" & stored="true"

dataImportHandler

[Scott Derrick]

I am trying to index a slew of web pages but want to restrict what gets 
indexed


I'm trying to use a dataImportHandler to do this.

my initial config to test this approach isn't doing what I expect



 
 


   
  
  
   

 

The FileListEntityProccessor is feeding me the files as expected

But the XPathEntityProcessor is only processing one  and and its 
coming up empty?


"entity:jcurrent",
[
  null,
  "--- row #1-",
  "file",
  "A10078.html",
  "fileSize",
  43635,
  "fileLastModified",
  "2015-08-12T22:44:19Z",
  "fileDir",
  "/var/www/web/A10078",
  "fileAbsolutePath",
  "/var/www/web/A10078/A10078.html",
  null,
  "-",
  "entity:x",
  [
"document#1",
[
  "query",
  "/var/www/web/A10078/A10078.html",
  "time-taken",
  "0:0:0.0",
  null,
  "--- row #1-",
  "p",
  "",
  "$forEach",
  "/html/body",
  null,
  "-"
],
"document#1",
[]
  ]
]
  ],