Search Specific Boosting

[Brett]

I'm implementing a feature where admins have the ability to control the 
order of the results by adding a boost to any specific search.


The search is a faceted interface (no text input) and which we take a 
hash of the search parameters (to form a unique search id) and then 
boost that field for the document.


The field is a wild card field so that it might look like this:

boost="90">true


The problem is that in these search results I am seeing is that my 
results are being grouped and the individual boost values are not having 
the granular effect I am looking for.


Say on a result set of 75 documents.  I see results with search boosts 
of 60-70 receiving the same score even though they were indexed with 
different boost values.  There are always more than one group.


Does anyone know what might be causing this?  Is there a better way to 
do what I am looking for?


Thanks,

Brett


Field Definition:

sortMissingLast="true" omitNorms="false" omitTermFreqAndPositions="true">

  


  
  

   
   
  

Re: Search Specific Boosting

[Brett]

Hi Erick,

Yeah, I've reviewed the debug output and can't make sense of why they 
are scoring the same.  I have double checked that they are being indexed 
with different boost values for the search field.  I've also increased 
the factors trying to get them be more granular so instead of boosting 
1,2,3,4,5 I did 100,200,300,400,500... Same result.


Here's and example of the debug output with two documents having 
different field boost values but receiving the same score.


Does anything stick out?  Any other ideas on how to get the results I am 
looking for?



69.694855 = (MATCH) product of: 104.54228 = (MATCH) sum of: 0.08869071 = 
(MATCH) MatchAllDocsQuery, product of: 0.08869071 = queryNorm 104.45359 
= (MATCH) weight(searchfe2684d248eab25404c3668711d4642e_boost:true in 
4016) [DefaultSimilarity], result of: 104.45359 = 
score(doc=4016,freq=1.0 = termFreq=1 ), product of: 0.48125002 = 
queryWeight, product of: 5.4261603 = idf(docFreq=81, maxDocs=6856) 
0.08869071 = queryNorm 217.04642 = fieldWeight in 4016, product of: 1.0 
= tf(freq=1.0), with freq of: 1.0 = termFreq=1 5.4261603 = 
idf(docFreq=81, maxDocs=6856) 40.0 = fieldNorm(doc=4016) 0.667 = 
coord(2/3)




69.694855 = (MATCH) product of: 104.54228 = (MATCH) sum of: 0.08869071 = 
(MATCH) MatchAllDocsQuery, product of: 0.08869071 = queryNorm 104.45359 
= (MATCH) weight(searchfe2684d248eab25404c3668711d4642e_boost:true in 
4106) [DefaultSimilarity], result of: 104.45359 = 
score(doc=4106,freq=1.0 = termFreq=1 ), product of: 0.48125002 = 
queryWeight, product of: 5.4261603 = idf(docFreq=81, maxDocs=6856) 
0.08869071 = queryNorm 217.04642 = fieldWeight in 4106, product of: 1.0 
= tf(freq=1.0), with freq of: 1.0 = termFreq=1 5.4261603 = 
idf(docFreq=81, maxDocs=6856) 40.0 = fieldNorm(doc=4106) 0.667 = 
coord(2/3)



On 1/11/2012 9:46 PM, Erick Erickson wrote:

Boosts are fairly coarse-grained. I suspect your boost factors are just
being rounded into the same buckets. Attaching&debugQuery=on and
looking at how the scores were calculated should help you figure out
if this is the case.

Best
Erick

On Wed, Jan 11, 2012 at 7:57 PM, Brett  wrote:

I'm implementing a feature where admins have the ability to control the
order of the results by adding a boost to any specific search.

The search is a faceted interface (no text input) and which we take a hash
of the search parameters (to form a unique search id) and then boost that
field for the document.

The field is a wild card field so that it might look like this:

true

The problem is that in these search results I am seeing is that my results
are being grouped and the individual boost values are not having the
granular effect I am looking for.

Say on a result set of 75 documents.  I see results with search boosts of
60-70 receiving the same score even though they were indexed with different
boost values.  There are always more than one group.

Does anyone know what might be causing this?  Is there a better way to do
what I am looking for?

Thanks,

Brett


Field Definition:


  


  
  


  

Confusion when using go-live and MapReduceIndexerTool

[Brett Hoerner]

java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of index shards into
Solr cluster took 2.31269488E8 secs
14/04/17 16:00:31 INFO hadoop.GoLive: Live merging failed

I'm digging into the code now, but wanted to send this out as a sanity
check.

Thanks,
Brett

Re: Confusion when using go-live and MapReduceIndexerTool

[Brett Hoerner]

https://gist.github.com/bretthoerner/0dc6bfdbf45a18328d4b


On Thu, Apr 17, 2014 at 11:31 AM, Mark Miller  wrote:

> Odd - might be helpful if you can share your sorlconfig.xml being used.
>
> --
> Mark Miller
> about.me/markrmiller
>
> On April 17, 2014 at 12:18:37 PM, Brett Hoerner (br...@bretthoerner.com)
> wrote:
>
> I'm doing HDFS input and output in my job, with the following:
>
> hadoop jar /mnt/faas-solr.jar \
> -D mapreduce.job.map.class=com.massrel.faassolr.SolrMapper \
> --update-conflict-resolver com.massrel.faassolr.SolrConflictResolver
> \
> --morphline-file /mnt/morphline-ignore.conf \
> --zk-host $ZKHOST \
> --output-dir hdfs://$MASTERIP:9000/output/ \
> --collection $COLLECTION \
> --go-live \
> --verbose \
> hdfs://$MASTERIP:9000/input/
>
> Index creation works,
>
> $ hadoop fs -ls -R hdfs://$MASTERIP:9000/output/results/part-0
> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data
> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index
> -rwxr-xr-x 1 hadoop supergroup 61 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0.fdt
> -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0.fdx
> -rwxr-xr-x 1 hadoop supergroup 1681 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0.fnm
> -rwxr-xr-x 1 hadoop supergroup 396 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0.si
> -rwxr-xr-x 1 hadoop supergroup 67 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.doc
> -rwxr-xr-x 1 hadoop supergroup 37 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.pos
> -rwxr-xr-x 1 hadoop supergroup 508 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.tim
> -rwxr-xr-x 1 hadoop supergroup 305 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.tip
> -rwxr-xr-x 1 hadoop supergroup 120 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene45_0.dvd
> -rwxr-xr-x 1 hadoop supergroup 351 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene45_0.dvm
> -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/segments_1
> -rwxr-xr-x 1 hadoop supergroup 110 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/index/segments_2
> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
> 10.98.33.114:9000/output/results/part-0/data/tlog
> -rw-r--r-- 1 hadoop supergroup 333 2014-04-17 16:00 hdfs://
>
> 10.98.33.114:9000/output/results/part-0/data/tlog/tlog.000
>
> But the go-live step fails, it's trying to use the HDFS path as the remote
> index path?
>
> 14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of output shards into
> Solr cluster...
> 14/04/17 16:00:31 INFO hadoop.GoLive: Live merge hdfs://
> 10.98.33.114:9000/output/results/part-0 into
> http://discover8-test-1d.i.massrel.com:8983/solr
> 14/04/17 16:00:31 ERROR hadoop.GoLive: Error sending live merge command
> java.util.concurrent.ExecutionException:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> directory '/mnt/solr_8983/home/hdfs:/
> 10.98.33.114:9000/output/results/part-0/data/index' does not exist
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126)
> at
>
> org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:867)
> at
>
> org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:609)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
>
> org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:596)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> directory '/mnt/solr_8983/home/hdfs:/
> 10.98.33.114:9000/output/results/part-0/data/index' does not exist
> at
>
>

Re: index merge question

[Brett Hoerner]

Sorry to bump this, I have the same issue and was curious about the sanity
of trying to work around it.

* I have a constant stream of realtime documents I need to continually
index. Sometimes they even overwrite very old documents (by using the same
unique ID).
* I also have a *huge* backlog of documents I'd like to get into a
SolrCloud cluster via Hadoop.

I understand that the MERGEINDEXES operation expects me to have unique
documents, but is it reasonable at all for me to be able to change that? In
a plain Solr instance I can add doc1, then add doc1 again with new fields
and the new update "wins" and I assume during segment merges the old update
is eventually removed. Does that mean it's possible for me to somehow
override a merge policy (or something like that?) to effectively do exactly
what my Hadoop conflict-resolver does? I already have logic there that
knows how to (1) decide which of 2 duplicate documents to keep and (2)
respect and "keep" deletes over anything else.

I'd love some pointers at what Solr/Lucene classes to look at if I wanted
to try my hand at this. I'm down in Lucene SegmentMerger right now but it
seems too low level to understand whatever Solr "knows" about enforcing a
single unique ID at merge (and search...? or update...?) time.

Thanks!



On Tue, Jun 11, 2013 at 11:10 AM, Mark Miller  wrote:

> Right - but that sounds a little different than what we were talking about.
>
> You had brought up the core admin merge cmd that let's you merge an index
> into a running Solr cluster.
>
> We are calling that the golive option in the map reduce indexing code. It
> has the limitations we have discussed.
>
> However, if you are only using map reduce to build indexes, there are
> facilities for dealing with duplicate id's - as you see in the
> documentation. The merges involved in that are different though - these are
> merges that happen as the final index is being constructed by the map
> reduce job. The final step is the golive step, where the indexes will be
> deployed to the running Solr cluster - this is what uses the core admin
> merge command, and if you are doing updates or adds outside of map reduce,
> you will face the issues we have discussed.
>
>
> - Mark
>
> On Jun 11, 2013, at 11:57 AM, James Thomas  wrote:
>
> > FWIW, the Solr included with Cloudera Search, by default, "ignores all
> but the most recent document version" during merges.
> > The conflict resolution is configurable however.  See the documentation
> for details.
> >
> http://www.cloudera.com/content/support/en/documentation/cloudera-search/cloudera-search-documentation-v1-latest.html
> > -- see the user guide pdf, " update-conflict-resolver" parameter
> >
> > James
> >
> > -Original Message-
> > From: anirudh...@gmail.com [mailto:anirudh...@gmail.com] On Behalf Of
> Anirudha Jadhav
> > Sent: Tuesday, June 11, 2013 10:47 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: index merge question
> >
> > From my experience the lucene mergeTool and the one invoked by coreAdmin
> is a pure lucene implementation and does not understand the concepts of a
> unique Key(solr land concept)
> >
> >  http://wiki.apache.org/solr/MergingSolrIndexes has a cautionary note
> at the end
> >
> > we do frequent index merges for which we externally run map/reduce (
> java code using lucene api's) jobs to merge & validate merged indices with
> sources.
> > -Ani
> >
> > On Tue, Jun 11, 2013 at 10:38 AM, Mark Miller 
> wrote:
> >> Yeah, you have to carefully manage things if you are map/reduce
> building indexes *and* updating documents in other ways.
> >>
> >> If your 'source' data for MR index building is the 'truth', you also
> have the option of not doing incremental index merging, and you could
> simply rebuild the whole thing every time - of course, depending your
> cluster size, that could be quite expensive.
> >
> >>
> >> - Mark
> >>
> >> On Jun 10, 2013, at 8:36 PM, Jamie Johnson  wrote:
> >>
> >>> Thanks Mark.  My question is stemming from the new cloudera search
> stuff.
> >>> My concern its that if while rebuilding the index someone updates a
> >>> doc that update could be lost from a solr perspective.  I guess what
> >>> would need to happen to ensure the correct information was indexed
> >>> would be to record the start time and reindex the information that
> changed since then?
> >>> On Jun 8, 2013 2:37 PM, "Mark Miller"  wrote:
> >>>
> 
>  On Jun 8, 2013, at 12:52 PM, Jamie Johnson  wrote:
> 
> > When merging through the core admin (
> > http://wiki.apache.org/solr/MergingSolrIndexes) what is the policy
> > for conflicts during the merge?  So for instance if I am merging
> > core 1 and core 2 into core 0 (first example), what happens if core
> > 1 and core 2
>  both
> > have a document with the same key, say core 1 has a newer version
> > of core 2?  Does the merge fail, does the newer document remain?
> 
>  You end up with both documents, both with that ID

Re: Confusion when using go-live and MapReduceIndexerTool

[Brett Hoerner]

Anyone have any thoughts on this?

In general, am I expected to be able to go-live from an unrelated cluster
of Hadoop machines to a SolrCloud that isn't running off of HDFS?

intput: HDFS
output: HDFS
go-live cluster: SolrCloud cluster on different machines running on plain
MMapDirectory

I'm back to looking at the code but holy hell is debugging Hadoop hard. :)


On Thu, Apr 17, 2014 at 12:33 PM, Brett Hoerner wrote:

> https://gist.github.com/bretthoerner/0dc6bfdbf45a18328d4b
>
>
> On Thu, Apr 17, 2014 at 11:31 AM, Mark Miller wrote:
>
>> Odd - might be helpful if you can share your sorlconfig.xml being used.
>>
>> --
>> Mark Miller
>> about.me/markrmiller
>>
>> On April 17, 2014 at 12:18:37 PM, Brett Hoerner (br...@bretthoerner.com)
>> wrote:
>>
>> I'm doing HDFS input and output in my job, with the following:
>>
>> hadoop jar /mnt/faas-solr.jar \
>> -D mapreduce.job.map.class=com.massrel.faassolr.SolrMapper \
>> --update-conflict-resolver com.massrel.faassolr.SolrConflictResolver
>> \
>> --morphline-file /mnt/morphline-ignore.conf \
>> --zk-host $ZKHOST \
>> --output-dir hdfs://$MASTERIP:9000/output/ \
>> --collection $COLLECTION \
>> --go-live \
>> --verbose \
>> hdfs://$MASTERIP:9000/input/
>>
>> Index creation works,
>>
>> $ hadoop fs -ls -R hdfs://$MASTERIP:9000/output/results/part-0
>> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data
>> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index
>> -rwxr-xr-x 1 hadoop supergroup 61 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0.fdt
>> -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0.fdx
>> -rwxr-xr-x 1 hadoop supergroup 1681 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0.fnm
>> -rwxr-xr-x 1 hadoop supergroup 396 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0.si
>> -rwxr-xr-x 1 hadoop supergroup 67 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.doc
>> -rwxr-xr-x 1 hadoop supergroup 37 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.pos
>> -rwxr-xr-x 1 hadoop supergroup 508 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.tim
>> -rwxr-xr-x 1 hadoop supergroup 305 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.tip
>> -rwxr-xr-x 1 hadoop supergroup 120 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene45_0.dvd
>> -rwxr-xr-x 1 hadoop supergroup 351 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene45_0.dvm
>> -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/segments_1
>> -rwxr-xr-x 1 hadoop supergroup 110 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/index/segments_2
>> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
>> 10.98.33.114:9000/output/results/part-0/data/tlog
>> -rw-r--r-- 1 hadoop supergroup 333 2014-04-17 16:00 hdfs://
>>
>> 10.98.33.114:9000/output/results/part-0/data/tlog/tlog.000
>>
>> But the go-live step fails, it's trying to use the HDFS path as the remote
>> index path?
>>
>> 14/04/17 16:00:31 INFO hadoop.GoLive: Live merging of output shards into
>> Solr cluster...
>> 14/04/17 16:00:31 INFO hadoop.GoLive: Live merge hdfs://
>> 10.98.33.114:9000/output/results/part-0 into
>> http://discover8-test-1d.i.massrel.com:8983/solr
>> 14/04/17 16:00:31 ERROR hadoop.GoLive: Error sending live merge command
>> java.util.concurrent.ExecutionException:
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
>> directory '/mnt/solr_8983/home/hdfs:/
>> 10.98.33.114:9000/output/results/part-0/data/index' does not exist
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>> at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126)
>> at
>>
>> org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:867)
>> at
>>
>> org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:609

Re: Confusion when using go-live and MapReduceIndexerTool

[Brett Hoerner]

I think I'm just misunderstanding the use of go-live. From mergeindexes
docs: "The indexes must exist on the disk of the Solr host, which may make
using this in a distributed environment cumbersome."

I'm guessing I'll have to write some sort of tool that pulls each completed
index out of HDFS and onto the respective SolrCloud machines and manually
do some kind of merge? I don't want to (can't) be running my Hadoop jobs on
the same nodes that SolrCloud is running on...

Also confusing to me: "no writes should be allowed on either core until the
merge is complete. If writes are allowed, corruption may occur on the
merged index." Is that saying that Solr will block writes, or is that
saying the end user has to ensure no writes are happening against the
collection during a merge? That seems... risky?


On Tue, Apr 22, 2014 at 9:29 AM, Brett Hoerner wrote:

> Anyone have any thoughts on this?
>
> In general, am I expected to be able to go-live from an unrelated cluster
> of Hadoop machines to a SolrCloud that isn't running off of HDFS?
>
> intput: HDFS
> output: HDFS
> go-live cluster: SolrCloud cluster on different machines running on plain
> MMapDirectory
>
> I'm back to looking at the code but holy hell is debugging Hadoop hard. :)
>
>
> On Thu, Apr 17, 2014 at 12:33 PM, Brett Hoerner wrote:
>
>> https://gist.github.com/bretthoerner/0dc6bfdbf45a18328d4b
>>
>>
>> On Thu, Apr 17, 2014 at 11:31 AM, Mark Miller wrote:
>>
>>> Odd - might be helpful if you can share your sorlconfig.xml being used.
>>>
>>> --
>>> Mark Miller
>>> about.me/markrmiller
>>>
>>> On April 17, 2014 at 12:18:37 PM, Brett Hoerner (br...@bretthoerner.com)
>>> wrote:
>>>
>>> I'm doing HDFS input and output in my job, with the following:
>>>
>>> hadoop jar /mnt/faas-solr.jar \
>>> -D mapreduce.job.map.class=com.massrel.faassolr.SolrMapper \
>>> --update-conflict-resolver com.massrel.faassolr.SolrConflictResolver
>>> \
>>> --morphline-file /mnt/morphline-ignore.conf \
>>> --zk-host $ZKHOST \
>>> --output-dir hdfs://$MASTERIP:9000/output/ \
>>> --collection $COLLECTION \
>>> --go-live \
>>> --verbose \
>>> hdfs://$MASTERIP:9000/input/
>>>
>>> Index creation works,
>>>
>>> $ hadoop fs -ls -R hdfs://$MASTERIP:9000/output/results/part-0
>>> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data
>>> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index
>>> -rwxr-xr-x 1 hadoop supergroup 61 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0.fdt
>>> -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0.fdx
>>> -rwxr-xr-x 1 hadoop supergroup 1681 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0.fnm
>>> -rwxr-xr-x 1 hadoop supergroup 396 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0.si
>>> -rwxr-xr-x 1 hadoop supergroup 67 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.doc
>>> -rwxr-xr-x 1 hadoop supergroup 37 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.pos
>>> -rwxr-xr-x 1 hadoop supergroup 508 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.tim
>>> -rwxr-xr-x 1 hadoop supergroup 305 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene41_0.tip
>>> -rwxr-xr-x 1 hadoop supergroup 120 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene45_0.dvd
>>> -rwxr-xr-x 1 hadoop supergroup 351 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/_0_Lucene45_0.dvm
>>> -rwxr-xr-x 1 hadoop supergroup 45 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/segments_1
>>> -rwxr-xr-x 1 hadoop supergroup 110 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/index/segments_2
>>> drwxr-xr-x - hadoop supergroup 0 2014-04-17 16:00 hdfs://
>>> 10.98.33.114:9000/output/results/part-0/data/tlog
>>> -rw-r--r-- 1 hadoop supergroup 333 2014-04-17 16:00 hdfs://
>>>
>>> 10.98.33.114:9000/

Is the act of *caching* an fq very expensive? (seems to cost 4 seconds in my example)

[Brett Hoerner]

If I run a query like this,

fq=text:lol
fq=created_at_tdid:[1400544000 TO 1400630400]

It takes about 6 seconds. Following queries take only 50ms or less, as
expected because my fqs are cached.

However, if I change the query to not cache my big range query:

fq=text:lol
fq={!cache=false}created_at_tdid:[1400544000 TO 1400630400]

It takes 2 seconds every time, which is a much better experience for my
"first query for that range."

What's odd to me is that I would expect both of these (first) queries to
have to do the same amount of work, expect the first one stuffs the
resulting bitset into a map at the end... which seems to have a 4 second
overhead?

Here's my filterCache from solrconfig:



Thanks.

Re: Is the act of *caching* an fq very expensive? (seems to cost 4 seconds in my example)

[Brett Hoerner]

In this case, I have >400 million documents, so I understand it taking a
while.

That said, I'm still not sure I understand why it would take *more* time.
In your example above, wouldn't it have to create an 11.92MB bitset even if
I *don't* cache the bitset? It seems the mere act of storing the work after
it's done (it has to be done in either case) is taking 4 whole seconds?



On Tue, Jun 3, 2014 at 3:59 PM, Shawn Heisey  wrote:

> On 6/3/2014 2:44 PM, Brett Hoerner wrote:
> > If I run a query like this,
> >
> > fq=text:lol
> > fq=created_at_tdid:[1400544000 TO 1400630400]
> >
> > It takes about 6 seconds. Following queries take only 50ms or less, as
> > expected because my fqs are cached.
> >
> > However, if I change the query to not cache my big range query:
> >
> > fq=text:lol
> > fq={!cache=false}created_at_tdid:[1400544000 TO 1400630400]
> >
> > It takes 2 seconds every time, which is a much better experience for my
> > "first query for that range."
> >
> > What's odd to me is that I would expect both of these (first) queries to
> > have to do the same amount of work, expect the first one stuffs the
> > resulting bitset into a map at the end... which seems to have a 4 second
> > overhead?
> >
> > Here's my filterCache from solrconfig:
> >
> >  >  size="64"
> >  initialSize="64"
> >  autowarmCount="32"/>
>
> I think that probably depends on how many documents you have in the
> single index/shard.  If you have one hundred million documents stored in
> the Lucene index, then each filter entry is 1250 bytes (11.92MB) in
> size - it is a bitset representing every document and whether it is
> included or excluded.  That data would need to be gathered and copied
> into the cache.  I suspect that it's the gathering that takes the most
> time ... several megabytes of memory is not very much for a modern
> processor to copy.
>
> As for how long this takes, I actually have no idea.  You have two
> filters here, so it would need to do everything twice.
>
> Thanks,
> Shawn
>
>

Re: Is the act of *caching* an fq very expensive? (seems to cost 4 seconds in my example)

[Brett Hoerner]

This is seemingly where it checks whether to use cache or not, the extra
work is really just a get (miss) and a put:


https://github.com/apache/lucene-solr/blob/trunk/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L1216

I suppose it's possible the put is taking 4 seconds, but that seems...
surprising to me.


On Tue, Jun 3, 2014 at 4:02 PM, Brett Hoerner 
wrote:

> In this case, I have >400 million documents, so I understand it taking a
> while.
>
> That said, I'm still not sure I understand why it would take *more* time.
> In your example above, wouldn't it have to create an 11.92MB bitset even if
> I *don't* cache the bitset? It seems the mere act of storing the work after
> it's done (it has to be done in either case) is taking 4 whole seconds?
>
>
>
> On Tue, Jun 3, 2014 at 3:59 PM, Shawn Heisey  wrote:
>
>> On 6/3/2014 2:44 PM, Brett Hoerner wrote:
>> > If I run a query like this,
>> >
>> > fq=text:lol
>> > fq=created_at_tdid:[1400544000 TO 1400630400]
>> >
>> > It takes about 6 seconds. Following queries take only 50ms or less, as
>> > expected because my fqs are cached.
>> >
>> > However, if I change the query to not cache my big range query:
>> >
>> > fq=text:lol
>> > fq={!cache=false}created_at_tdid:[1400544000 TO 1400630400]
>> >
>> > It takes 2 seconds every time, which is a much better experience for my
>> > "first query for that range."
>> >
>> > What's odd to me is that I would expect both of these (first) queries to
>> > have to do the same amount of work, expect the first one stuffs the
>> > resulting bitset into a map at the end... which seems to have a 4 second
>> > overhead?
>> >
>> > Here's my filterCache from solrconfig:
>> >
>> > > >  size="64"
>> >  initialSize="64"
>> >  autowarmCount="32"/>
>>
>> I think that probably depends on how many documents you have in the
>> single index/shard.  If you have one hundred million documents stored in
>> the Lucene index, then each filter entry is 1250 bytes (11.92MB) in
>> size - it is a bitset representing every document and whether it is
>> included or excluded.  That data would need to be gathered and copied
>> into the cache.  I suspect that it's the gathering that takes the most
>> time ... several megabytes of memory is not very much for a modern
>> processor to copy.
>>
>> As for how long this takes, I actually have no idea.  You have two
>> filters here, so it would need to do everything twice.
>>
>> Thanks,
>> Shawn
>>
>>
>

Re: Is the act of *caching* an fq very expensive? (seems to cost 4 seconds in my example)

[Brett Hoerner]

Yonik, I'm familiar with your blog posts -- and thanks very much for them.
:) Though I'm not sure what you're trying to show me with the q=*:* part? I
was of course using q=*:* in my queries, but I assume you mean to leave off
the text:lol bit?

I've done some Cluster changes, so these are my baselines:

q=*:*
fq=created_at_tdid:[1392768004 TO 1393944400] (uncached at this point)
~7.5 seconds

q=*:*
fq={!cache=false}created_at_tdid:[1392768005 TO 1393944400]
~7.5 seconds (I guess this is what you were trying to show me?)

The thing is, my queries always more "specific" than that, so given a
string:

q=*:*
fq=text:basketball
fq={!cache=false}created_at_tdid:[1392768007 TO 1393944400]
~5.2 seconds

q=*:*
fq=text:basketball
fq={!cache=false}created_at_tdid:[1392768005 TO 1393944400]
~1.6 seconds

Is there no hope for my first time fq searches being as fast as non-cached
fqs? It's a shame to have to chose either (1) super fast queries after
cached XOR (2) more responsive first time queries (by a large margin).

Thanks!



On Tue, Jun 3, 2014 at 4:20 PM, Yonik Seeley  wrote:

> On Tue, Jun 3, 2014 at 5:19 PM, Yonik Seeley 
> wrote:
> > So try:
> >   q=*:*
> >   fq=created_at_tdid:[1400544000 TO 1400630400]
>
> vs
>
> So try:
>   q=*:*
>   fq={!cache=false}created_at_tdid:[1400544000 TO 1400630400]
>
>
> -Yonik
> http://heliosearch.org - facet functions, subfacets, off-heap
> filters&fieldcache
>

"Fake" cached join query much faster than cached fq?

[Brett Hoerner]

The following two queries are doing the same thing, one using a "normal" fq
range query and another using a parent query. The cache is warm (these are
both hits) but the "normal" ones takes ~6 to 7.5sec while the parent query
hack takes ~1.2sec.

Is this expected? Is there anything "wrong" with my "normal fq" query? My
plan is to increase the size of my perSegFilter cache so I can use the hack
for faster queries... any thoughts here?

"responseHeader": { "status": 0, "QTime": 7657, "params": { "q": "*:*", "
facet.field": "terms_smnd", "debug": "true", "indent": "true", "fq": [
"created_at_tdid:[1392768001
TO 1393954400]", "text:coffee" ], "rows": "0", "wt": "json", "facet": "true",
"_": "1401906435914" } }, "response": { "numFound": 2432754, "start": 0, "
maxScore": 1, "docs": [] }

Full response example:
https://gist.githubusercontent.com/bretthoerner/60418f08a88093c30220/raw/0a61f013f763e68985c15c5ed6cad6fa253182b9/gistfile1.txt

 "responseHeader": { "status": 0, "QTime": 1210, "params": { "q": "*:*", "
facet.field": "terms_smnd", "debug": "true", "indent": "true", "fq": [
"{!cache=false}{!parent
which='created_at_tdid:[1392768001 TO 1393954400]'}", "text:coffee" ], "rows":
"0", "wt": "json", "facet": "true", "_": "1401906444521" } }, "response": {
"numFound": 2432754, "start": 0, "maxScore": 1, "docs": [] }

Full response example:
https://gist.githubusercontent.com/bretthoerner/9d82aa8fe59ffc7ff6ab/raw/560a395a0933870a5d2ac736b58805d8fab7f758/gistfile1.txt

Re: "Fake" cached join query much faster than cached fq?

[Brett Hoerner]

Thanks Mikhail, I'll try to profile it soon.

As for cardinality, on a single core:

created_at_tdid:[1392768001 TO 1393954400] = 241657215
text:coffee = 117593

Oddly enough, I just tried the query with &distrib=false and both return in
about 50ms... hmm.




On Thu, Jun 5, 2014 at 5:09 AM, Mikhail Khludnev  wrote:

> Brett,
>
> It's really interesting observation. I can only speculate. It's worth to
> check cache hit stats  and cache content via
> http://wiki.apache.org/solr/SolrCaching#showItems (the key question what
> are cached doc sets classes). Also if you tell the overall number of docs
> in the index, and cardinality of both filters, it might allow to guess
> something. Anyway, jvisualvm sampling can give an exact answer. Giving
> responses, it's enough to profile one of the slave nodes.
>
>
> On Wed, Jun 4, 2014 at 10:32 PM, Brett Hoerner 
> wrote:
>
> > The following two queries are doing the same thing, one using a "normal"
> fq
> > range query and another using a parent query. The cache is warm (these
> are
> > both hits) but the "normal" ones takes ~6 to 7.5sec while the parent
> query
> > hack takes ~1.2sec.
> >
> > Is this expected? Is there anything "wrong" with my "normal fq" query? My
> > plan is to increase the size of my perSegFilter cache so I can use the
> hack
> > for faster queries... any thoughts here?
> >
> > "responseHeader": { "status": 0, "QTime": 7657, "params": { "q": "*:*", "
> > facet.field": "terms_smnd", "debug": "true", "indent": "true", "fq": [
> > "created_at_tdid:[1392768001
> > TO 1393954400]", "text:coffee" ], "rows": "0", "wt": "json", "facet":
> > "true",
> > "_": "1401906435914" } }, "response": { "numFound": 2432754, "start": 0,
> "
> > maxScore": 1, "docs": [] }
> >
> > Full response example:
> >
> >
> https://gist.githubusercontent.com/bretthoerner/60418f08a88093c30220/raw/0a61f013f763e68985c15c5ed6cad6fa253182b9/gistfile1.txt
> >
> >  "responseHeader": { "status": 0, "QTime": 1210, "params": { "q": "*:*",
> "
> > facet.field": "terms_smnd", "debug": "true", "indent": "true", "fq": [
> > "{!cache=false}{!parent
> > which='created_at_tdid:[1392768001 TO 1393954400]'}", "text:coffee" ],
> > "rows":
> > "0", "wt": "json", "facet": "true", "_": "1401906444521" } },
> "response": {
> > "numFound": 2432754, "start": 0, "maxScore": 1, "docs": [] }
> >
> > Full response example:
> >
> >
> https://gist.githubusercontent.com/bretthoerner/9d82aa8fe59ffc7ff6ab/raw/560a395a0933870a5d2ac736b58805d8fab7f758/gistfile1.txt
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
>  
>

Confusion about location of + and - ?

[Brett Hoerner]

Can anyone explain the difference between these two queries?

  text:(+"happy") AND -user:("123456789") = numFound 2912224

But

  text:(+"happy") AND user:(-"123456789") = numFound 0

Now, you may just say "then just put - infront of your field, duh!" Well,

  text:(+"happy") = numFound 2912224
  user:(-"123456789") = numFound 465998192

(FWIW there is no user named 123456789 in my index)

As you can see, the queries work alone, but when combined with an AND I
always get 0 results. If I move the - before the field in my query, it
works. What am I missing here?

Thanks.

Re: Confusion about location of + and - ?

[Brett Hoerner]

Interesting, is there a performance impact to sending the *:*?


On Tue, Jul 1, 2014 at 2:53 PM, Jack Krupansky 
wrote:

> Yeah, there's a known bug that a negative-only query within parentheses
> doesn't match properly - you need to add a non-negative term, such as
> "*:*". For example:
>
>  text:(+"happy") AND user:(*:* -"123456789")
>
> -- Jack Krupansky
>
> -Original Message- From: Brett Hoerner
> Sent: Tuesday, July 1, 2014 2:51 PM
> To: solr-user@lucene.apache.org
> Subject: Confusion about location of + and - ?
>
>
> Can anyone explain the difference between these two queries?
>
>  text:(+"happy") AND -user:("123456789") = numFound 2912224
>
> But
>
>  text:(+"happy") AND user:(-"123456789") = numFound 0
>
> Now, you may just say "then just put - infront of your field, duh!" Well,
>
>  text:(+"happy") = numFound 2912224
>  user:(-"123456789") = numFound 465998192
>
> (FWIW there is no user named 123456789 in my index)
>
> As you can see, the queries work alone, but when combined with an AND I
> always get 0 results. If I move the - before the field in my query, it
> works. What am I missing here?
>
> Thanks.
>

Re: Confusion about location of + and - ?

[Brett Hoerner]

Also, does anyone have the Solr or Lucene bug # for this?


On Tue, Jul 1, 2014 at 3:06 PM, Brett Hoerner 
wrote:

> Interesting, is there a performance impact to sending the *:*?
>
>
> On Tue, Jul 1, 2014 at 2:53 PM, Jack Krupansky 
> wrote:
>
>> Yeah, there's a known bug that a negative-only query within parentheses
>> doesn't match properly - you need to add a non-negative term, such as
>> "*:*". For example:
>>
>>  text:(+"happy") AND user:(*:* -"123456789")
>>
>> -- Jack Krupansky
>>
>> -Original Message- From: Brett Hoerner
>> Sent: Tuesday, July 1, 2014 2:51 PM
>> To: solr-user@lucene.apache.org
>> Subject: Confusion about location of + and - ?
>>
>>
>> Can anyone explain the difference between these two queries?
>>
>>  text:(+"happy") AND -user:("123456789") = numFound 2912224
>>
>> But
>>
>>  text:(+"happy") AND user:(-"123456789") = numFound 0
>>
>> Now, you may just say "then just put - infront of your field, duh!" Well,
>>
>>  text:(+"happy") = numFound 2912224
>>  user:(-"123456789") = numFound 465998192
>>
>> (FWIW there is no user named 123456789 in my index)
>>
>> As you can see, the queries work alone, but when combined with an AND I
>> always get 0 results. If I move the - before the field in my query, it
>> works. What am I missing here?
>>
>> Thanks.
>>
>
>

Trouble with manually routed collection after upgrade to 4.6

[Brett Hoerner]

Hi, I've been using a collection on Solr 4.5.X for a few weeks and just did
an upgrade to 4.6 and am having some issues.

First: this collection is, I guess, implicitly routed. I do this for every
document insert using SolrJ:

  document.addField("_route_", shardId)

After upgrading the servers to 4.6 I now get the following on every
insert/delete when using either SolrJ 4.5.1 or 4.6:

  org.apache.solr.common.SolrException: No active slice servicing hash code
17b9dff6 in DocCollection

In the clusterstate *none* of my shards have a range set (they're all
null), but I thought this would be expected since I do routing myself.

Did the upgrade change something here? I didn't see anything related to
this in the upgrade notes.

Thanks,
Brett

Re: Trouble with manually routed collection after upgrade to 4.6

[Brett Hoerner]

Here's my clusterstate.json:

  https://gist.github.com/bretthoerner/a8120a8d89c93f773d70


On Mon, Nov 25, 2013 at 10:18 AM, Brett Hoerner wrote:

> Hi, I've been using a collection on Solr 4.5.X for a few weeks and just
> did an upgrade to 4.6 and am having some issues.
>
> First: this collection is, I guess, implicitly routed. I do this for every
> document insert using SolrJ:
>
>   document.addField("_route_", shardId)
>
> After upgrading the servers to 4.6 I now get the following on every
> insert/delete when using either SolrJ 4.5.1 or 4.6:
>
>   org.apache.solr.common.SolrException: No active slice servicing hash
> code 17b9dff6 in DocCollection
>
> In the clusterstate *none* of my shards have a range set (they're all
> null), but I thought this would be expected since I do routing myself.
>
> Did the upgrade change something here? I didn't see anything related to
> this in the upgrade notes.
>
> Thanks,
> Brett
>

Re: Trouble with manually routed collection after upgrade to 4.6

[Brett Hoerner]

Think I got it. For some reason this was in my clusterstate.json after the
upgrade (note that I was using 4.5.X just fine previously...):

 "router": {
   "name": "compositeId"
 },

I stopped all my nodes and manually edited this to me "implicit" (is there
a tool for this? I've always done it manually), started the cluster up
again and it's all good now.



On Mon, Nov 25, 2013 at 10:38 AM, Brett Hoerner wrote:

> Here's my clusterstate.json:
>
>   https://gist.github.com/bretthoerner/a8120a8d89c93f773d70
>
>
> On Mon, Nov 25, 2013 at 10:18 AM, Brett Hoerner wrote:
>
>> Hi, I've been using a collection on Solr 4.5.X for a few weeks and just
>> did an upgrade to 4.6 and am having some issues.
>>
>> First: this collection is, I guess, implicitly routed. I do this for
>> every document insert using SolrJ:
>>
>>   document.addField("_route_", shardId)
>>
>> After upgrading the servers to 4.6 I now get the following on every
>> insert/delete when using either SolrJ 4.5.1 or 4.6:
>>
>>   org.apache.solr.common.SolrException: No active slice servicing hash
>> code 17b9dff6 in DocCollection
>>
>> In the clusterstate *none* of my shards have a range set (they're all
>> null), but I thought this would be expected since I do routing myself.
>>
>> Did the upgrade change something here? I didn't see anything related to
>> this in the upgrade notes.
>>
>> Thanks,
>> Brett
>>
>
>

After upgrading indexer to SolrJ 4.6.1: o.a.solr.servlet.SolrDispatchFilter - Unknown type 19

[Brett Hoerner]

I have Solr 4.6.1 on the server and just upgraded my indexer app to SolrJ
4.6.1 and indexing ceased (indexer returned "No live servers for shard" but
the real root from the Solr servers is below). Note that SolrJ 4.6.1 is
fine for the query side, just not adding documents.



21:35:21.508 [qtp1418442930-22296231] ERROR
o.a.solr.servlet.SolrDispatchFilter - null:java.lang.RuntimeException:
Unknown type 19
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:232)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:139)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:131)
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:223)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:116)
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:188)
at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:114)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:158)
at
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:99)
at
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)

Re: After upgrading indexer to SolrJ 4.6.1: o.a.solr.servlet.SolrDispatchFilter - Unknown type 19

[Brett Hoerner]

On Fri, Feb 7, 2014 at 6:15 PM, Mark Miller  wrote:

> You have to update the other nodes to 4.6.1 as well.
>

I'm not sure I follow, all of the Solr instances in the cluster are 4.6.1
to my knowledge?

Thanks,
Brett

Re: After upgrading indexer to SolrJ 4.6.1: o.a.solr.servlet.SolrDispatchFilter - Unknown type 19

[Brett Hoerner]

Hmmm, I'm assembling into an uberjar that forces uniqueness of classes. I
verified 4.6.1 is definitely winning and included alone when it breaks.


On Sat, Feb 8, 2014 at 9:44 AM, Mark Miller  wrote:

> If that is the case we really have to dig in. Given the error, the first
> thing I would assume is that you have an old solrj jar or something before
> 4.6.1 involved with a 4.6.1 solrj jar or install.
>
> - Mark
>
> http://about.me/markrmiller
>
>
>
> On Feb 7, 2014, 7:15:24 PM, Mark Miller  wrote:
> Hey, yeah, blew it on this one. Someone just reported it the other day -
> the way that a bug was fixed was not back and forward compatible. The first
> implementation was wrong.
>
> You have to update the other nodes to 4.6.1 as well.
>
> I’m going to look at some scripting test that can help check for this type
> of thing.
>
> - Mark
>
> http://about.me/markrmiller
>
>
>
> On Feb 7, 2014, 7:01:24 PM, Brett Hoerner  wrote:
> I have Solr 4.6.1 on the server and just upgraded my indexer app to SolrJ
> 4.6.1 and indexing ceased (indexer returned "No live servers for shard" but
> the real root from the Solr servers is below). Note that SolrJ 4.6.1 is
> fine for the query side, just not adding documents.
>
>
>
> 21:35:21.508 [qtp1418442930-22296231] ERROR
> o.a.solr.servlet.SolrDispatchFilter - null:java.lang.RuntimeException:
> Unknown type 19
> at
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:232)
> at
>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:139)
> at
>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:131)
> at
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:223)
> at
>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:116)
> at
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:188)
> at
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:114)
> at
>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:158)
> at
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:99)
> at
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
> at
>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
>
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection$R

Re: After upgrading indexer to SolrJ 4.6.1: o.a.solr.servlet.SolrDispatchFilter - Unknown type 19

[Brett Hoerner]

Oh, I was talking about my indexer. That stack is from my Solr servers,
very weird since it's *supposed* to be 4.6.1 . I'll dig in more, thanks.


On Sat, Feb 8, 2014 at 10:21 AM, Mark Miller  wrote:

> If you look at the stack trace, the line numbers match 4.6.0 in the src,
> but not 4.6.1. That code couldn’t have been 4.6.1 it seems.
>
> - Mark
>
> http://about.me/markrmiller
>
> On Feb 8, 2014, at 11:12 AM, Brett Hoerner  wrote:
>
> > Hmmm, I'm assembling into an uberjar that forces uniqueness of classes. I
> > verified 4.6.1 is definitely winning and included alone when it breaks.
> >
> >
> > On Sat, Feb 8, 2014 at 9:44 AM, Mark Miller 
> wrote:
> >
> >> If that is the case we really have to dig in. Given the error, the first
> >> thing I would assume is that you have an old solrj jar or something
> before
> >> 4.6.1 involved with a 4.6.1 solrj jar or install.
> >>
> >> - Mark
> >>
> >> http://about.me/markrmiller
> >>
> >>
> >>
> >> On Feb 7, 2014, 7:15:24 PM, Mark Miller  wrote:
> >> Hey, yeah, blew it on this one. Someone just reported it the other day -
> >> the way that a bug was fixed was not back and forward compatible. The
> first
> >> implementation was wrong.
> >>
> >> You have to update the other nodes to 4.6.1 as well.
> >>
> >> I’m going to look at some scripting test that can help check for this
> type
> >> of thing.
> >>
> >> - Mark
> >>
> >> http://about.me/markrmiller
> >>
> >>
> >>
> >> On Feb 7, 2014, 7:01:24 PM, Brett Hoerner 
> wrote:
> >> I have Solr 4.6.1 on the server and just upgraded my indexer app to
> SolrJ
> >> 4.6.1 and indexing ceased (indexer returned "No live servers for shard"
> but
> >> the real root from the Solr servers is below). Note that SolrJ 4.6.1 is
> >> fine for the query side, just not adding documents.
> >>
> >>
> >>
> >> 21:35:21.508 [qtp1418442930-22296231] ERROR
> >> o.a.solr.servlet.SolrDispatchFilter - null:java.lang.RuntimeException:
> >> Unknown type 19
> >> at
> >> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:232)
> >> at
> >>
> >>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:139)
> >> at
> >>
> >>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:131)
> >> at
> >> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:223)
> >> at
> >>
> >>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:116)
> >> at
> >> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:188)
> >> at
> >>
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:114)
> >> at
> >>
> >>
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:158)
> >> at
> >>
> >>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:99)
> >> at
> >> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
> >> at
> >>
> >>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> >> at
> >>
> >>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >> at
> >>
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
> >> at
> >>
> >>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
> >> at
> >>
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
> >> at
> >>
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
> >> at
> >>
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> >> at
> >>
> >>
> org.eclipse.jetty.server.handler.ScopedHan

Re: After upgrading indexer to SolrJ 4.6.1: o.a.solr.servlet.SolrDispatchFilter - Unknown type 19

[Brett Hoerner]

Mark, you were correct. I realized I was still running a prerelease of
4.6.1 (by a handful of commits). Bounced them with proper 4.6.1 and we're
all good, sorry for the spam. :)


On Sat, Feb 8, 2014 at 10:29 AM, Brett Hoerner wrote:

> Oh, I was talking about my indexer. That stack is from my Solr servers,
> very weird since it's *supposed* to be 4.6.1 . I'll dig in more, thanks.
>
>
> On Sat, Feb 8, 2014 at 10:21 AM, Mark Miller wrote:
>
>> If you look at the stack trace, the line numbers match 4.6.0 in the src,
>> but not 4.6.1. That code couldn’t have been 4.6.1 it seems.
>>
>> - Mark
>>
>> http://about.me/markrmiller
>>
>> On Feb 8, 2014, at 11:12 AM, Brett Hoerner 
>> wrote:
>>
>> > Hmmm, I'm assembling into an uberjar that forces uniqueness of classes.
>> I
>> > verified 4.6.1 is definitely winning and included alone when it breaks.
>> >
>> >
>> > On Sat, Feb 8, 2014 at 9:44 AM, Mark Miller 
>> wrote:
>> >
>> >> If that is the case we really have to dig in. Given the error, the
>> first
>> >> thing I would assume is that you have an old solrj jar or something
>> before
>> >> 4.6.1 involved with a 4.6.1 solrj jar or install.
>> >>
>> >> - Mark
>> >>
>> >> http://about.me/markrmiller
>> >>
>> >>
>> >>
>> >> On Feb 7, 2014, 7:15:24 PM, Mark Miller  wrote:
>> >> Hey, yeah, blew it on this one. Someone just reported it the other day
>> -
>> >> the way that a bug was fixed was not back and forward compatible. The
>> first
>> >> implementation was wrong.
>> >>
>> >> You have to update the other nodes to 4.6.1 as well.
>> >>
>> >> I’m going to look at some scripting test that can help check for this
>> type
>> >> of thing.
>> >>
>> >> - Mark
>> >>
>> >> http://about.me/markrmiller
>> >>
>> >>
>> >>
>> >> On Feb 7, 2014, 7:01:24 PM, Brett Hoerner 
>> wrote:
>> >> I have Solr 4.6.1 on the server and just upgraded my indexer app to
>> SolrJ
>> >> 4.6.1 and indexing ceased (indexer returned "No live servers for
>> shard" but
>> >> the real root from the Solr servers is below). Note that SolrJ 4.6.1 is
>> >> fine for the query side, just not adding documents.
>> >>
>> >>
>> >>
>> >> 21:35:21.508 [qtp1418442930-22296231] ERROR
>> >> o.a.solr.servlet.SolrDispatchFilter - null:java.lang.RuntimeException:
>> >> Unknown type 19
>> >> at
>> >> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:232)
>> >> at
>> >>
>> >>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:139)
>> >> at
>> >>
>> >>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:131)
>> >> at
>> >> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:223)
>> >> at
>> >>
>> >>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:116)
>> >> at
>> >> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:188)
>> >> at
>> >>
>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:114)
>> >> at
>> >>
>> >>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:158)
>> >> at
>> >>
>> >>
>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:99)
>> >> at
>> >>
>> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
>> >> at
>> >>
>> >>
>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>> >> at
>> >>
>> >>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> >> at
>> >>
>> >>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
>> >> at
>> >>
>> >>
&

Solr mapred MTree merge stage hangs repeatably in 4.10 (but not 4.9)

[Brett Hoerner]

I have a very weird problem that I'm going to try to describe here to see
if anyone has any "ah-ha" moments or clues. I haven't created a small
reproducible project for this but I guess I will have to try in the future
if I can't figure it out. (Or I'll need to bisect by running long Hadoop
jobs...)

So, the facts:

* Have been successfully using Solr mapred to build very large Solr
clusters for months
* As of Solr 4.10 *some* job sizes repeatably hang in the MTree merge phase
in 4.10
* Those same jobs (same input, output, and Hadoop cluster itself) succeed
if I only change my Solr deps to 4.9
* The job *does succeed* in 4.10 if I use the same data to create more, but
smaller shards (e.g. 12x as many shards each 1/12th the size of the job
that fails)
* Creating my "normal size" shards (the size I want, that works in 4.9) the
job hangs with 2 mappers running, 0 reducers in the MTree merge phase
* There are no errors or warning in the syslog/stderr of the MTree mappers,
no errors ever echo'd back to the "interactive run" of the job (mapper says
100%, reduce says 0%, will stay forever)
* No CPU being used on the boxes running the merge, no GC happening, JVM
waiting on a futex, all threads blocked on various queues
* No disk usage problems, nothing else obviously wrong with any box in the
cluster

I diff'ed around between 4.10 and 4.9 and barely see any changes in mapred
contrib, mostly some test stuff. I didn't see any transitive dependency
changes in Solr/Lucene that look like they would affect me.

Re: Solr mapred MTree merge stage hangs repeatably in 4.10 (but not 4.9)

[Brett Hoerner]

)
... 12 more




[...snip...] another similar failure:




14/09/23 17:52:55 INFO mapreduce.Job: Task Id :
attempt_1411487144915_0006_r_46_0, Status : FAILED
Error: java.io.IOException: org.apache.solr.common.SolrException: Error
opening new searcher
at
org.apache.solr.hadoop.SolrRecordWriter.close(SolrRecordWriter.java:307)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:558)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1421)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:615)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1648)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1625)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
at
org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:150)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
at
org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
at org.apache.solr.hadoop.BatchWriter.close(BatchWriter.java:200)
at
org.apache.solr.hadoop.SolrRecordWriter.close(SolrRecordWriter.java:295)
... 8 more
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=d9019857 actual=632aa4e2
(resource=BufferedChecksumIndexInput(_1i_Lucene41_0.tip))
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:211)
at
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:268)
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.(BlockTreeTermsReader.java:125)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:197)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
at
org.apache.lucene.index.SegmentReader.(SegmentReader.java:108)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:237)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:426)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:292)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:277)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1476)
... 25 more


On Tue, Sep 16, 2014 at 12:54 PM, Brett Hoerner 
wrote:

> I have a very weird problem that I'm going to try to describe here to see
> if anyone has any "ah-ha" moments or clues. I haven't created a small
> reproducible project for this but I guess I will have to try in the future
> if I can't figure it out. (Or I'll need to bisect by running long Hadoop
> jobs...)
>
> So, the facts:
>
> * Have been successfully using Solr mapred

Re: Solr mapred MTree merge stage hangs repeatably in 4.10 (but not 4.9)

[Brett Hoerner]

To be clear, those exceptions are during the "main" mapred job that is
creating the many small indexes. If these errors above occur (they don't
fail the job), I am 99% sure that is when the MTree job later hangs.

On Tue, Sep 23, 2014 at 1:02 PM, Brett Hoerner 
wrote:

> I believe these are related (they are new to me), anyone seen anything
> like this in Solr mapred?
>
>
>
> Error: java.io.IOException:
> org.apache.solr.client.solrj.SolrServerException:
> org.apache.solr.client.solrj.SolrServerException:
> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
> problem?) : expected=5fb8f6da actual=8b048ec4
> (resource=BufferedChecksumIndexInput(_1e_Lucene41_0.tip))
> at
> org.apache.solr.hadoop.SolrRecordWriter.close(SolrRecordWriter.java:307)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:558)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.solr.client.solrj.SolrServerException:
> org.apache.solr.client.solrj.SolrServerException:
> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
> problem?) : expected=5fb8f6da actual=8b048ec4
> (resource=BufferedChecksumIndexInput(_1e_Lucene41_0.tip))
> at
> org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:223)
> at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
> at
> org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
> at org.apache.solr.hadoop.BatchWriter.close(BatchWriter.java:200)
> at
> org.apache.solr.hadoop.SolrRecordWriter.close(SolrRecordWriter.java:295)
> ... 8 more
> Caused by: org.apache.solr.client.solrj.SolrServerException:
> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
> problem?) : expected=5fb8f6da actual=8b048ec4
> (resource=BufferedChecksumIndexInput(_1e_Lucene41_0.tip))
> at
> org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:155)
> ... 12 more
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=5fb8f6da actual=8b048ec4
> (resource=BufferedChecksumIndexInput(_1e_Lucene41_0.tip))
> at
> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:211)
> at
> org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:268)
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.(BlockTreeTermsReader.java:125)
> at
> org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:197)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:108)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:282)
> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3315)
> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3306)
> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3020)
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3169)
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3136)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:582)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateP

Solr mapred MTree merge stage ~6x slower in 4.10

[Brett Hoerner]

As an update to this thread, it seems my MTree wasn't completely hanging,
it was just much slower in 4.10.

If I replace 4.9.0 with 4.10 in my jar the MTree merge stage is 6x (or
more) slower (in my case, 20 min becomes 2 hours). I hope to bisect this in
the future, but the jobs I'm running take a long time. I haven't tried to
see if the issue shows on smaller jobs yet (does 1 minute become 6
minutes?).

Brett




On Tue, Sep 16, 2014 at 12:54 PM, Brett Hoerner 
wrote:

> I have a very weird problem that I'm going to try to describe here to see
> if anyone has any "ah-ha" moments or clues. I haven't created a small
> reproducible project for this but I guess I will have to try in the future
> if I can't figure it out. (Or I'll need to bisect by running long Hadoop
> jobs...)
>
> So, the facts:
>
> * Have been successfully using Solr mapred to build very large Solr
> clusters for months
> * As of Solr 4.10 *some* job sizes repeatably hang in the MTree merge
> phase in 4.10
> * Those same jobs (same input, output, and Hadoop cluster itself) succeed
> if I only change my Solr deps to 4.9
> * The job *does succeed* in 4.10 if I use the same data to create more,
> but smaller shards (e.g. 12x as many shards each 1/12th the size of the job
> that fails)
> * Creating my "normal size" shards (the size I want, that works in 4.9)
> the job hangs with 2 mappers running, 0 reducers in the MTree merge phase
> * There are no errors or warning in the syslog/stderr of the MTree
> mappers, no errors ever echo'd back to the "interactive run" of the job
> (mapper says 100%, reduce says 0%, will stay forever)
> * No CPU being used on the boxes running the merge, no GC happening, JVM
> waiting on a futex, all threads blocked on various queues
> * No disk usage problems, nothing else obviously wrong with any box in the
> cluster
>
> I diff'ed around between 4.10 and 4.9 and barely see any changes in mapred
> contrib, mostly some test stuff. I didn't see any transitive dependency
> changes in Solr/Lucene that look like they would affect me.
>

Advice for using Solr 4.5 custom sharding to handle rolling time-oriented event data

[Brett Hoerner]

I'm interesting in using the new custom sharding features in the
collections API to search a rolling window of event data. I'd appreciate a
spot/sanity check of my plan/understanding.

Say I only care about the last 7 days of events and I have thousands per
second (billions per week).

Am I correct that I could create a new shard for each hour, and send events
that happen in those hour with the ID (uniqueKey) of
`new_event_hour!event_id` so that each hour block of events goes into one
shard?

I *always* query these events by the time in which they occurred, which is
another TrieInt field that I index with every document. So at query time I
would need to calculate the range the user cared about and send something
like _route_=hour1&_route_=hour2 if I wanted to only query those two
shards. (I *can* set multiple _route_ arguments in one query, right? And
Solr will handle merging results like it would with any other cores?)

Some scheduled task would drop and delete shards after they were more than
7 days old.

Does all of that make sense? Do you see a smarter way to do large
"time-oriented" search in SolrCloud?

Thanks!

Problems with maxShardsPerNode in 4.5

[Brett Hoerner]

It seems that changes in 4.5 collection configuration now require users to
set a maxShardsPerNode (or it defaults to 1).

Maybe this was the case before, but with the new CREATESHARD API it seems a
very restrictive. I've just created a very simple test collection on 3
machines where I set maxShardsPerNode at collection creation time to 1, and
I made 3 shards. Everything is good.

Now I want a 4th shard, it seems impossible to create because the cluster
"knows" I should only have 1 shard per node. Yet my problem doesn't require
more hardware, I just my new shard to exist on one of the existing servers.

So I try again -- I create a collection with 3 shards and set
maxShardsPerNode to 1000 (just as a silly test). Everything is good.

Now I add shard4 and it immediately tries to add 1000 replicas of shard4...

You can see my earlier email today about time-oriented data in 4.5 to see
what I'm trying to do. I was hoping to have 1 shard per hour/day with the
ability to easily add/drop them as I move the time window (say, a week of
data, 1 per day).

Am I missing something?

Thanks!

Re: Problems with maxShardsPerNode in 4.5

[Brett Hoerner]

Related, 1 more try:

Created collection starting with 4 shards on 1 box. Had to set
maxShardsPerNode to 4 to do this.

Now I want to "roll over" my time window, so to attempt to deal with the
problems noted above I delete the oldest shard first. That works fine.

Now I try to add my new shard, which works, but again it defaults to
"maxShardsPerNode" # of replicas, so I'm left with:

* [deleted by me] hour0
* hour1 - 1 replica
* hour2 - 1 replica
* hour3 - 1 replica
* hour4 - 4 replicas [ << the one I created after deleting hour0]

Still at a loss as to how I would create 1 new shard with 1 replica on any
server in 4.5?

Thanks!


On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner wrote:

> It seems that changes in 4.5 collection configuration now require users to
> set a maxShardsPerNode (or it defaults to 1).
>
> Maybe this was the case before, but with the new CREATESHARD API it seems
> a very restrictive. I've just created a very simple test collection on 3
> machines where I set maxShardsPerNode at collection creation time to 1, and
> I made 3 shards. Everything is good.
>
> Now I want a 4th shard, it seems impossible to create because the cluster
> "knows" I should only have 1 shard per node. Yet my problem doesn't require
> more hardware, I just my new shard to exist on one of the existing servers.
>
> So I try again -- I create a collection with 3 shards and set
> maxShardsPerNode to 1000 (just as a silly test). Everything is good.
>
> Now I add shard4 and it immediately tries to add 1000 replicas of shard4...
>
> You can see my earlier email today about time-oriented data in 4.5 to see
> what I'm trying to do. I was hoping to have 1 shard per hour/day with the
> ability to easily add/drop them as I move the time window (say, a week of
> data, 1 per day).
>
> Am I missing something?
>
> Thanks!
>

Re: Problems with maxShardsPerNode in 4.5

[Brett Hoerner]

Shalin,

Thanks for the fix. There's still part of the underlying issue that I
consider a bug or a documentation problem: how do I adjust maxShardsPerNode
after my collection has been created, and/or how can I disable it being
checked/used at all? It seems odd to me that I have to set it to an odd
number like 1000 just to get around this?

Thanks!


On Wed, Oct 2, 2013 at 12:04 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> Thanks for reporting this Brett. This is indeed a bug. A workaround is to
> specify replicationFactor=1 with the createShard command which will create
> only one replica even if maxShardsPerNode=1000 at collection level.
>
> I'll open an issue.
>
>
> On Wed, Oct 2, 2013 at 7:25 AM, Brett Hoerner  >wrote:
>
> > Related, 1 more try:
> >
> > Created collection starting with 4 shards on 1 box. Had to set
> > maxShardsPerNode to 4 to do this.
> >
> > Now I want to "roll over" my time window, so to attempt to deal with the
> > problems noted above I delete the oldest shard first. That works fine.
> >
> > Now I try to add my new shard, which works, but again it defaults to
> > "maxShardsPerNode" # of replicas, so I'm left with:
> >
> > * [deleted by me] hour0
> > * hour1 - 1 replica
> > * hour2 - 1 replica
> > * hour3 - 1 replica
> > * hour4 - 4 replicas [ << the one I created after deleting hour0]
> >
> > Still at a loss as to how I would create 1 new shard with 1 replica on
> any
> > server in 4.5?
> >
> > Thanks!
> >
> >
> > On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner  > >wrote:
> >
> > > It seems that changes in 4.5 collection configuration now require users
> > to
> > > set a maxShardsPerNode (or it defaults to 1).
> > >
> > > Maybe this was the case before, but with the new CREATESHARD API it
> seems
> > > a very restrictive. I've just created a very simple test collection on
> 3
> > > machines where I set maxShardsPerNode at collection creation time to 1,
> > and
> > > I made 3 shards. Everything is good.
> > >
> > > Now I want a 4th shard, it seems impossible to create because the
> cluster
> > > "knows" I should only have 1 shard per node. Yet my problem doesn't
> > require
> > > more hardware, I just my new shard to exist on one of the existing
> > servers.
> > >
> > > So I try again -- I create a collection with 3 shards and set
> > > maxShardsPerNode to 1000 (just as a silly test). Everything is good.
> > >
> > > Now I add shard4 and it immediately tries to add 1000 replicas of
> > shard4...
> > >
> > > You can see my earlier email today about time-oriented data in 4.5 to
> see
> > > what I'm trying to do. I was hoping to have 1 shard per hour/day with
> the
> > > ability to easily add/drop them as I move the time window (say, a week
> of
> > > data, 1 per day).
> > >
> > > Am I missing something?
> > >
> > > Thanks!
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

What's the purpose of the bits option in compositeId (Solr 4.5)?

[Brett Hoerner]

I'm curious what the later "shard-local" bits do, if anything?

I have a very large cluster (256 shards) and I'm sending most of my data
with a single "composite", e.g. 1234!, but I'm noticing the data
is being split among many of the shards.

My guess right now is that since I'm only using the default 16 bits my data
is being split across multiple shards (because of my high # of shards).

Thanks,
Brett

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

[Brett Hoerner]

Router is definitely compositeId.

To be clear, data isn't being spread evenly... it's like it's *almost*
working. It's just odd to me that I'm slamming in data that's 99% of one
_route_ key yet after a few minutes (from a fresh empty index) I have 2
shards with a sizeable amount of data (68M and 128M) and the rest are very
small as expected.

The fact that two are receiving so much makes me think my data is being
split into two shards. I'm trying to debug more now.


On Tue, Oct 8, 2013 at 5:45 PM, Yonik Seeley  wrote:

> On Tue, Oct 8, 2013 at 6:29 PM, Brett Hoerner 
> wrote:
> > I'm curious what the later "shard-local" bits do, if anything?
> >
> > I have a very large cluster (256 shards) and I'm sending most of my data
> > with a single "composite", e.g. 1234!, but I'm noticing the
> data
> > is being split among many of the shards.
>
> That shouldn't be the case.  All of your shards should have a lower
> hash value with all 0 bits and an upper hash value of all 1s (i.e.
> 0x to 0x)
> So you see any shards where that's not true?
>
> Also, is the router set to compositeId?
>
> -Yonik
>
> > My guess right now is that since I'm only using the default 16 bits my
> data
> > is being split across multiple shards (because of my high # of shards).
> >
> > Thanks,
> > Brett
>

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

[Brett Hoerner]

This is my clusterstate.json:
https://gist.github.com/bretthoerner/0098f741f48f9bb51433

And these are my core sizes (note large ones are sorted to the end):
https://gist.github.com/bretthoerner/f5b5e099212194b5dff6

I've only "heavily sent" 2 shards by now (I'm sharding by hour and it's
been running for 2). There *is* a little old data in my stream, but not
that much (like <5%). What's confusing to me is that 5 of them are rather
large, when I'd expect 2 of them to be.


On Tue, Oct 8, 2013 at 5:45 PM, Yonik Seeley  wrote:

> On Tue, Oct 8, 2013 at 6:29 PM, Brett Hoerner 
> wrote:
> > I'm curious what the later "shard-local" bits do, if anything?
> >
> > I have a very large cluster (256 shards) and I'm sending most of my data
> > with a single "composite", e.g. 1234!, but I'm noticing the
> data
> > is being split among many of the shards.
>
> That shouldn't be the case.  All of your shards should have a lower
> hash value with all 0 bits and an upper hash value of all 1s (i.e.
> 0x to 0x)
> So you see any shards where that's not true?
>
> Also, is the router set to compositeId?
>
> -Yonik
>
> > My guess right now is that since I'm only using the default 16 bits my
> data
> > is being split across multiple shards (because of my high # of shards).
> >
> > Thanks,
> > Brett
>

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

[Brett Hoerner]

I have a silly question, how do I query a single shard in SolrCloud? When I
hit solr/foo_shard1_replica1/select it always seems to do a full cluster
query.

I can't (easily) do a _route_ query before I know what each have.


On Tue, Oct 8, 2013 at 7:06 PM, Yonik Seeley  wrote:

> On Tue, Oct 8, 2013 at 7:31 PM, Brett Hoerner 
> wrote:
> > This is my clusterstate.json:
> > https://gist.github.com/bretthoerner/0098f741f48f9bb51433
> >
> > And these are my core sizes (note large ones are sorted to the end):
> > https://gist.github.com/bretthoerner/f5b5e099212194b5dff6
> >
> > I've only "heavily sent" 2 shards by now (I'm sharding by hour and it's
> > been running for 2). There *is* a little old data in my stream, but not
> > that much (like <5%). What's confusing to me is that 5 of them are rather
> > large, when I'd expect 2 of them to be.
>
> The cluster state looks fine at first glance... and each route key
> should map to a single shard.
> You could try a query to each of the big shards and see what IDs are in
> them.
>
> -Yonik
>

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

[Brett Hoerner]

Ignore me I forgot about shards= from the wiki.


On Tue, Oct 8, 2013 at 7:11 PM, Brett Hoerner wrote:

> I have a silly question, how do I query a single shard in SolrCloud? When
> I hit solr/foo_shard1_replica1/select it always seems to do a full cluster
> query.
>
> I can't (easily) do a _route_ query before I know what each have.
>
>
> On Tue, Oct 8, 2013 at 7:06 PM, Yonik Seeley  wrote:
>
>> On Tue, Oct 8, 2013 at 7:31 PM, Brett Hoerner 
>> wrote:
>> > This is my clusterstate.json:
>> > https://gist.github.com/bretthoerner/0098f741f48f9bb51433
>> >
>> > And these are my core sizes (note large ones are sorted to the end):
>> > https://gist.github.com/bretthoerner/f5b5e099212194b5dff6
>> >
>> > I've only "heavily sent" 2 shards by now (I'm sharding by hour and it's
>> > been running for 2). There *is* a little old data in my stream, but not
>> > that much (like <5%). What's confusing to me is that 5 of them are
>> rather
>> > large, when I'd expect 2 of them to be.
>>
>> The cluster state looks fine at first glance... and each route key
>> should map to a single shard.
>> You could try a query to each of the big shards and see what IDs are in
>> them.
>>
>> -Yonik
>>
>
>

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

[Brett Hoerner]

Thanks folks,

As an update for future readers --- the problem was on my side (my logic in
picking the _route_ was flawed) as expected. :)


On Tue, Oct 8, 2013 at 7:35 PM, Yonik Seeley  wrote:

> On Tue, Oct 8, 2013 at 8:27 PM, Shawn Heisey  wrote:
> > There is also the "distrib=false" parameter that will cause the request
> to
> > be handled directly by the core it is sent to rather than being
> > distributed/balanced by SolrCloud.
>
> Right - this is probably the best option for diagnosing what is in what
> index.
>
> -Yonik
>

SolrCloud facet query repeatably fails with "No live SolrServers" for some terms, not all

[Brett Hoerner]

An example:
https://gist.github.com/bretthoerner/2ffc362450bcd4c2487a

I'll note that all shards and replicas show as "Up" (green) in the Admin UI.

Does anyone know how this could happen? I can repeat this over and over
with the same terms. It was my understanding that something like a facet
query would need to go to *all* shards for any query (I'm using the default
SolrCloud sharding mechanism, nothing special).

How could a text field search for 'happy' always work and 'austin' always
return an error, shouldn't that "down server" be hit for a 'happy' query
also?

Thanks,
Brett

Problem with Solr 3.6.1 extracting ODT content using SolrCell's ExtractingRequestHandler

[Brett Melbourne]

Hi all,

I am encountering a problem where Solr 3.6.1 is not able to extract the text 
content from ODT (Open Office Document) files submitted to the 
ExtractingRequestHandler. I can reproduce this issue against the example schema 
running with jetty.

Executing a simple index request (based on the example in the wiki):
curl 
"http://localhost:8983/solr/update/extract?literal.id=doc1&uprefix=attr_&fmap.content=attr_content&commit=true";<http://localhost:8983/solr/update/extract?literal.id=doc1&uprefix=attr_&fmap.content=attr_content&commit=true%22>
 -F "myfile=@testfile.odt"
returns no errors, and does not generate any exceptions in the log/console.

A query for doc1 returns an empty attr_content field:
  

Oddly enough, executing an "extractOnly=true" request against the 
ExtractingRequestHandler with the same ODT file correctly returns the text of 
the file.

I am wondering:

* Is this a known issue? (I couldn't find any mention of this 
particular issue anywhere...)

* Are there any workarounds or does anyone have any suggestions?

Thanks,

Brett.

RE: Problem with Solr 3.6.1 extracting ODT content using SolrCell's ExtractingRequestHandler

[Brett Melbourne]

Hi Erik,

The document is committed successfully... it is just missing all the extracted 
content from Tika when I query for that document.

i.e. the mapped content field attr_content is empty (fmap.content=attr_content)



1.9162908

24





2009-04-16T11:32:00


2012-11-23T00:29:39.73


...




Brett.

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Sunday, November 25, 2012 9:27 PM
To: solr-user@lucene.apache.org
Subject: Re: Problem with Solr 3.6.1 extracting ODT content using SolrCell's 
ExtractingRequestHandler

Did you commit after you added the document but before you tried the search?

Best
Erick


On Fri, Nov 23, 2012 at 6:25 PM, Brett Melbourne < 
bmelbou...@halogensoftware.com> wrote:

> Hi all,
>
> I am encountering a problem where Solr 3.6.1 is not able to extract 
> the text content from ODT (Open Office Document) files submitted to 
> the ExtractingRequestHandler. I can reproduce this issue against the 
> example schema running with jetty.
>
> Executing a simple index request (based on the example in the wiki):
> curl "
> http://localhost:8983/solr/update/extract?literal.id=doc1&uprefix=attr
> _&fmap.content=attr_content&commit=true
> "<
> http://localhost:8983/solr/update/extract?literal.id=doc1&uprefix=attr
> _&fmap.content=attr_content&commit=true%22>
> -F "myfile=@testfile.odt"
> returns no errors, and does not generate any exceptions in the log/console.
>
> A query for doc1 returns an empty attr_content field:
>   
>
> Oddly enough, executing an "extractOnly=true" request against the 
> ExtractingRequestHandler with the same ODT file correctly returns the 
> text of the file.
>
> I am wondering:
>
> * Is this a known issue? (I couldn't find any mention of this
> particular issue anywhere...)
>
> * Are there any workarounds or does anyone have any suggestions?
>
> Thanks,
>
> Brett.
>
>

SolrCloud stops handling collection CREATE/DELETE (but responds HTTP 200)

[Brett Hoerner]

Hi,

I have a Cloud setup of 4 machines. I bootstrapped them with 1 collection,
which I called "default" and haven't used since. I'm using an external ZK
ensemble that was completely empty before I started this cloud.

Once I had all 4 nodes in the cloud I used the collection API to create the
real collections I wanted. I also tested that deleting works.

For example,

# this worked
curl "
http://localhost:8984/solr/admin/collections?action=CREATE&name=15678&numShards=4
"

# this worked
curl "http://localhost:8984/solr/admin/collections?action=DELETE&name=15678";

Next, I started my indexer service which happily sent many, many updates to
the cloud. Queries against the collections also work just fine.

Finally, a few hours later, I tried doing a create and a delete. Both
operations did nothing, although Solr replied with a "200 OK".

$ curl -i "
http://localhost:8984/solr/admin/collections?action=CREATE&name=15679&numShards=4
"
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Transfer-Encoding: chunked



03

There is nothing in the stdout/stderr logs, nor the Java logs (I have it
set to WARN).

I have tried bouncing the nodes and it doesn't change anything.

Any ideas? How can I further debug this or what else can I provide?

Re: SolrCloud stops handling collection CREATE/DELETE (but responds HTTP 200)

[Brett Hoerner]

For what it's worth this is the log output with DEBUG on,

Dec 07, 2012 2:00:48 PM org.apache.solr.handler.admin.CollectionsHandler
handleCreateAction
INFO: Creating Collection : action=CREATE&name=foo&numShards=4
Dec 07, 2012 2:01:03 PM org.apache.solr.core.SolrCore execute
INFO: [15671] webapp=/solr path=/admin/system params={wt=json} status=0
QTime=5
Dec 07, 2012 2:01:15 PM org.apache.solr.handler.admin.CollectionsHandler
handleDeleteAction
INFO: Deleting Collection : action=DELETE&name=default
Dec 07, 2012 2:01:20 PM org.apache.solr.core.SolrCore execute

Neither the CREATE or DELETE actually did anything, though. (Again, HTTP
200 OK)

Still stuck here, any ideas?

Brett


On Tue, Dec 4, 2012 at 7:19 PM, Brett Hoerner wrote:

> Hi,
>
> I have a Cloud setup of 4 machines. I bootstrapped them with 1 collection,
> which I called "default" and haven't used since. I'm using an external ZK
> ensemble that was completely empty before I started this cloud.
>
> Once I had all 4 nodes in the cloud I used the collection API to create
> the real collections I wanted. I also tested that deleting works.
>
> For example,
>
> # this worked
> curl "
> http://localhost:8984/solr/admin/collections?action=CREATE&name=15678&numShards=4
> "
>
> # this worked
> curl "
> http://localhost:8984/solr/admin/collections?action=DELETE&name=15678";
>
> Next, I started my indexer service which happily sent many, many updates
> to the cloud. Queries against the collections also work just fine.
>
> Finally, a few hours later, I tried doing a create and a delete. Both
> operations did nothing, although Solr replied with a "200 OK".
>
> $ curl -i "
> http://localhost:8984/solr/admin/collections?action=CREATE&name=15679&numShards=4
> "
> HTTP/1.1 200 OK
> Content-Type: application/xml; charset=UTF-8
> Transfer-Encoding: chunked
>
> 
> 
> 0 name="QTime">3
>
> There is nothing in the stdout/stderr logs, nor the Java logs (I have it
> set to WARN).
>
> I have tried bouncing the nodes and it doesn't change anything.
>
> Any ideas? How can I further debug this or what else can I provide?
>

RE: Problem with Solr 3.6.1 extracting ODT content using SolrCell's ExtractingRequestHandler

[Brett Melbourne]

Hi Erick,

Thanks for the reply!

I don't think there is a problem with my schema, because I can successfully 
extract text from other file types.

For example, Tika is able to extract the content from a docx:
 
FINEST: Trying class name 
org.apache.solr.handler.extraction.ExtractingRequestHandler
Dec 7, 2012 3:18:35 PM org.apache.solr.handler.extraction.SolrContentHandler 
newDocument
FINE: Doc: SolrInputDocument[{attr_meta=attr_meta(1.0)={[stream_content_type, 
application/xml, stream_size, 9935, Content-Type, 
application/vnd.openxmlformats-officedocument.wordprocessingml.document]}, 
attr_revision_number=attr_revision_number(1.0)={1}, 
attr_template=attr_template(1.0)={Normal.dotm}, 
attr_last_author=attr_last_author(1.0)={Brett Melbourne}, 
attr_page_count=attr_page_count(1.0)={1}, 
attr_application_name=attr_application_name(1.0)={Microsoft Office Word}, 
author=author(1.0)={Brett Melbourne}, 
last_modified=last_modified(1.0)={2012-12-07T19:18:00.000Z}, 
attr_application_version=attr_application_version(1.0)={12.}, 
attr_character_count_with_spaces=attr_character_count_with_spaces(1.0)={60}, 
attr_date=attr_date(1.0)={2012-12-07T19:17:00Z}, 
attr_total_time=attr_total_time(1.0)={1}, 
attr_publisher=attr_publisher(1.0)={}, attr_creator=attr_creator(1.0)={Brett 
Melbourne}, attr_word_count=attr_word_count(1.0)={9}, 
attr_xmptpg_npages=attr_xmptpg_npages(1.0)={1}, 
attr_creation_date=attr_creation_date(1.0)={2012-12-07T19:17:00Z}, 
attr_stream_content_type=attr_stream_content_type(1.0)={application/xml}, 
attr_line_count=attr_line_count(1.0)={1}, 
attr_character_count=attr_character_count(1.0)={52}, 
attr_stream_size=attr_stream_size(1.0)={9935}, 
content_type=content_type(1.0)={application/vnd.openxmlformats-officedocument.wordprocessingml.document},
 attr_paragraph_count=attr_paragraph_count(1.0)={1}, id=id(1.0)={doc3}, 
text=text(1.0)={ This is some text content that Solr should be able 
to parse.   }}]


The docx content in Solr is:



0
0

on
0
id:doc3
2.2
10





Microsoft Office Word


12.


52


60


2012-12-07T19:17:00Z


Brett Melbourne


2012-12-07T19:17:00Z


Brett Melbourne


1


stream_content_type
application/xml
stream_size
9935
Content-Type

application/vnd.openxmlformats-officedocument.wordprocessingml.document



1


1





1


application/xml


9935


Normal.dotm


1


9


1

Brett Melbourne


application/vnd.openxmlformats-officedocument.wordprocessingml.document


doc3
2012-12-07T19:18:00Z


This is some text content that Solr should be able to parse.







When I attempt to index an ODT, it apparently works fine.. however notice that 
the text field is empty:

Dec 7, 2012 4:18:43 PM org.apache.solr.handler.extraction.SolrContentHandler 
newDocument
FINE: Doc: SolrInputDocument[{attr_editing_cycles=attr_editing_cycles(1.0)={1}, 
attr_page_count=attr_page_count(1.0)={2}, 
attr_date=attr_date(1.0)={2010-09-16T15:51:00Z}, 
attr_creator=attr_creator(1.0)={droy}, 
attr_word_count=attr_word_count(1.0)={475}, 
attr_xmptpg_npages=attr_xmptpg_npages(1.0)={2}, 
attr_edit_time=attr_edit_time(1.0)={PT60S}, 
attr_creation_date=attr_creation_date(1.0)={2010-09-16T15:50:00Z}, 
attr_nbpara=attr_nbpara(1.0)={6}, 
attr_stream_content_type=attr_stream_content_type(1.0)={application/xml}, 
attr_initial_creator=attr_initial_creator(1.0)={droy}, 
attr_character_count=attr_character_count(1.0)={3177}, 
attr_stream_size=attr_stream_size(1.0)={9130}, 
attr_generator=attr_generator(1.0)={MicrosoftOffice/12.0 MicrosoftWord}, 
attr_nbword=attr_nbword(1.0)={475}, attr_nbpage=attr_nbpage(1.0)={2}, 
content_type=content_type(1.0)={application/vnd.oasis.opendocument.text}, 
attr_nbcharacter=attr_nbcharacter(1.0)={3177}, 
attr_paragraph_count=attr_paragraph_count(1.0)={6}, id=id(1.0)={doc4}, 
text=text(1.0)={  }}]


The corresponding document in solr looks like this:



0
0

on
0
id:doc4
2.2
10





3177


2010-09-16T15:50:00Z


droy


2010-09-16T15:51:00Z


PT60S


1


MicrosoftOffice/12.0 MicrosoftWord


droy


3177


2


6


475


2


6


application/xml


9130


475


2


application/vnd.oasis.opendocument.text

doc4








Brett.




-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, November 27, 2012 7:38 AM
To: solr-user@lucene.apache.org
Subject: Re: Problem with Solr 3.6.1 extracting ODT content using SolrCell's 
ExtractingRequestHandler

Not an issue that I know of. I expect you've got some obscure problem in your 
definitions, but I'm guession. Try modifying your schema so the glob pattern 
maps to a stored field, something like:
 remove 
all other fields except id, remove your mapping, and try it again.
If you query with fl=* you should see everything that was extracted. That'll 
tell you whether it is a problem with Solr/Tika or something in how you're 
using them.

Best
Erick


On Mon, Nov 26, 2012 at 10:19 AM, Brett Melbourne < 
bmelbou...@halogensoftware.com> wrote:

> Hi Erik,
>
> Th

Re: SolrCloud stops handling collection CREATE/DELETE (but responds HTTP 200)

[Brett Hoerner]

Thanks,

It looks like my cluster is in a wedged state after I tried to delete a
collection that didn't exist. There are about 80 items in the queue after
the delete op (that it can't get by). Is that a known bug?

I guess for now I'll just check that a collection exists before sending any
deletes. :)

Brett


On Fri, Dec 7, 2012 at 10:50 AM, Mark Miller  wrote:

> Anything in any of the other logs (the other nodes)? The key is getting
> the logs from the node designated as the overseer - it should hopefully
> have the error.
>
> Right now because you pass this stuff off to the overseer, you will always
> get back a 200 - there is a JIRA issue that addresses this though
> (collection API responses) and I hope to get it committed soon.
>
> - Mark
>
> On Dec 7, 2012, at 7:26 AM, Brett Hoerner  wrote:
>
> > For what it's worth this is the log output with DEBUG on,
> >
> > Dec 07, 2012 2:00:48 PM org.apache.solr.handler.admin.CollectionsHandler
> > handleCreateAction
> > INFO: Creating Collection : action=CREATE&name=foo&numShards=4
> > Dec 07, 2012 2:01:03 PM org.apache.solr.core.SolrCore execute
> > INFO: [15671] webapp=/solr path=/admin/system params={wt=json} status=0
> > QTime=5
> > Dec 07, 2012 2:01:15 PM org.apache.solr.handler.admin.CollectionsHandler
> > handleDeleteAction
> > INFO: Deleting Collection : action=DELETE&name=default
> > Dec 07, 2012 2:01:20 PM org.apache.solr.core.SolrCore execute
> >
> > Neither the CREATE or DELETE actually did anything, though. (Again, HTTP
> > 200 OK)
> >
> > Still stuck here, any ideas?
> >
> > Brett
> >
> >
> > On Tue, Dec 4, 2012 at 7:19 PM, Brett Hoerner  >wrote:
> >
> >> Hi,
> >>
> >> I have a Cloud setup of 4 machines. I bootstrapped them with 1
> collection,
> >> which I called "default" and haven't used since. I'm using an external
> ZK
> >> ensemble that was completely empty before I started this cloud.
> >>
> >> Once I had all 4 nodes in the cloud I used the collection API to create
> >> the real collections I wanted. I also tested that deleting works.
> >>
> >> For example,
> >>
> >> # this worked
> >> curl "
> >>
> http://localhost:8984/solr/admin/collections?action=CREATE&name=15678&numShards=4
> >> "
> >>
> >> # this worked
> >> curl "
> >> http://localhost:8984/solr/admin/collections?action=DELETE&name=15678";
> >>
> >> Next, I started my indexer service which happily sent many, many updates
> >> to the cloud. Queries against the collections also work just fine.
> >>
> >> Finally, a few hours later, I tried doing a create and a delete. Both
> >> operations did nothing, although Solr replied with a "200 OK".
> >>
> >> $ curl -i "
> >>
> http://localhost:8984/solr/admin/collections?action=CREATE&name=15679&numShards=4
> >> "
> >> HTTP/1.1 200 OK
> >> Content-Type: application/xml; charset=UTF-8
> >> Transfer-Encoding: chunked
> >>
> >> 
> >> 
> >> 0 >> name="QTime">3
> >>
> >> There is nothing in the stdout/stderr logs, nor the Java logs (I have it
> >> set to WARN).
> >>
> >> I have tried bouncing the nodes and it doesn't change anything.
> >>
> >> Any ideas? How can I further debug this or what else can I provide?
> >>
>
>

Have the SolrCloud collection REST endpoints move or changed for 4.1?

[Brett Hoerner]

I was using Solr 4.0 but ran into a few problems using SolrCloud. I'm
trying out 4.1 RC1 right now but the update URL I used to use is returning
HTTP 404.

For example, I would post my document updates to,

http://localhost:8983/solr/collection1

But that is 404ing now (collection1 exists according to the admin UI, all
shards are green and happy, and data dirs exist on the nodes).

I also tried the following,

http://localhost:8983/solr/collection1/update

And also received a 404 there.

A specific example from the Java client:

22:38:12.474 [pool-7-thread-14] ERROR com.massrel.faassolr.SolrBackend -
Error while flushing to Solr.
org.apache.solr.common.SolrException: Server at
http://backfill-2d.i.massrel.com:8983/solr/15724/update returned non ok
status:404, message:Not Found
 at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
 at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:438)
~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]

But I can hit that URL with a GET,

$ curl http://backfill-1d.i.massrel.com:8983/solr/15724/update


4002missing content
stream400


Thoughts?

Thanks.

Re: Have the SolrCloud collection REST endpoints move or changed for 4.1?

[Brett Hoerner]

I'm actually wondering if this other issue I've been having is a problem:

https://issues.apache.org/jira/browse/SOLR-4321

The fact that some nodes don't "get" pieces of a collection could explain
the 404.

That said, even when a node has "parts" of a collection it reports 404
sometimes. What's odd is that I can use curl to post a JSON document to the
same URL and it will return 200.

When I log every request I make from my indexer process (using solr4j) it's
about 50/50 between 404 and 200...


On Sat, Jan 19, 2013 at 5:22 PM, Brett Hoerner wrote:

> I was using Solr 4.0 but ran into a few problems using SolrCloud. I'm
> trying out 4.1 RC1 right now but the update URL I used to use is returning
> HTTP 404.
>
> For example, I would post my document updates to,
>
> http://localhost:8983/solr/collection1
>
> But that is 404ing now (collection1 exists according to the admin UI, all
> shards are green and happy, and data dirs exist on the nodes).
>
> I also tried the following,
>
> http://localhost:8983/solr/collection1/update
>
> And also received a 404 there.
>
> A specific example from the Java client:
>
> 22:38:12.474 [pool-7-thread-14] ERROR com.massrel.faassolr.SolrBackend -
> Error while flushing to Solr.
> org.apache.solr.common.SolrException: Server at
> http://backfill-2d.i.massrel.com:8983/solr/15724/update returned non ok
> status:404, message:Not Found
>  at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>  at
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:438)
> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
> at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>
> But I can hit that URL with a GET,
>
> $ curl http://backfill-1d.i.massrel.com:8983/solr/15724/update
> 
> 
> 400 name="QTime">2missing content
> stream400
> 
>
> Thoughts?
>
> Thanks.
>

Re: Have the SolrCloud collection REST endpoints move or changed for 4.1?

[Brett Hoerner]

So the ticket I created wasn't related, there is a working patch for that
now but my original issue remains, I get 404 when trying to post updates to
a URL that worked fine in Solr 4.0.


On Sat, Jan 19, 2013 at 5:56 PM, Brett Hoerner wrote:

> I'm actually wondering if this other issue I've been having is a problem:
>
> https://issues.apache.org/jira/browse/SOLR-4321
>
> The fact that some nodes don't "get" pieces of a collection could explain
> the 404.
>
> That said, even when a node has "parts" of a collection it reports 404
> sometimes. What's odd is that I can use curl to post a JSON document to the
> same URL and it will return 200.
>
> When I log every request I make from my indexer process (using solr4j)
> it's about 50/50 between 404 and 200...
>
>
>  On Sat, Jan 19, 2013 at 5:22 PM, Brett Hoerner wrote:
>
>> I was using Solr 4.0 but ran into a few problems using SolrCloud. I'm
>> trying out 4.1 RC1 right now but the update URL I used to use is returning
>> HTTP 404.
>>
>> For example, I would post my document updates to,
>>
>> http://localhost:8983/solr/collection1
>>
>> But that is 404ing now (collection1 exists according to the admin UI, all
>> shards are green and happy, and data dirs exist on the nodes).
>>
>> I also tried the following,
>>
>> http://localhost:8983/solr/collection1/update
>>
>> And also received a 404 there.
>>
>> A specific example from the Java client:
>>
>> 22:38:12.474 [pool-7-thread-14] ERROR com.massrel.faassolr.SolrBackend -
>> Error while flushing to Solr.
>> org.apache.solr.common.SolrException: Server at
>> http://backfill-2d.i.massrel.com:8983/solr/15724/update returned non ok
>> status:404, message:Not Found
>>  at
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>>  at
>> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:438)
>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>> at
>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>>
>> But I can hit that URL with a GET,
>>
>> $ curl http://backfill-1d.i.massrel.com:8983/solr/15724/update
>> 
>> 
>> 400> name="QTime">2missing content
>> stream400
>> 
>>
>> Thoughts?
>>
>> Thanks.
>>
>
>

Re: Have the SolrCloud collection REST endpoints move or changed for 4.1?

[Brett Hoerner]

Sorry, I take it back. It looks like fixing
https://issues.apache.org/jira/browse/SOLR-4321 fixed my issue after all.


On Sun, Jan 20, 2013 at 2:21 PM, Brett Hoerner wrote:

> So the ticket I created wasn't related, there is a working patch for that
> now but my original issue remains, I get 404 when trying to post updates to
> a URL that worked fine in Solr 4.0.
>
>
> On Sat, Jan 19, 2013 at 5:56 PM, Brett Hoerner wrote:
>
>> I'm actually wondering if this other issue I've been having is a problem:
>>
>> https://issues.apache.org/jira/browse/SOLR-4321
>>
>> The fact that some nodes don't "get" pieces of a collection could explain
>> the 404.
>>
>> That said, even when a node has "parts" of a collection it reports 404
>> sometimes. What's odd is that I can use curl to post a JSON document to the
>> same URL and it will return 200.
>>
>> When I log every request I make from my indexer process (using solr4j)
>> it's about 50/50 between 404 and 200...
>>
>>
>>  On Sat, Jan 19, 2013 at 5:22 PM, Brett Hoerner 
>> wrote:
>>
>>> I was using Solr 4.0 but ran into a few problems using SolrCloud. I'm
>>> trying out 4.1 RC1 right now but the update URL I used to use is returning
>>> HTTP 404.
>>>
>>> For example, I would post my document updates to,
>>>
>>> http://localhost:8983/solr/collection1
>>>
>>> But that is 404ing now (collection1 exists according to the admin UI,
>>> all shards are green and happy, and data dirs exist on the nodes).
>>>
>>> I also tried the following,
>>>
>>> http://localhost:8983/solr/collection1/update
>>>
>>> And also received a 404 there.
>>>
>>> A specific example from the Java client:
>>>
>>> 22:38:12.474 [pool-7-thread-14] ERROR com.massrel.faassolr.SolrBackend -
>>> Error while flushing to Solr.
>>> org.apache.solr.common.SolrException: Server at
>>> http://backfill-2d.i.massrel.com:8983/solr/15724/update returned non ok
>>> status:404, message:Not Found
>>>  at
>>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
>>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>>> at
>>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
>>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>>>  at
>>> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:438)
>>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>>> at
>>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
>>> ~[solr-solrj-4.0.0.jar:4.0.0 1394950 - rmuir - 2012-10-06 03:05:44]
>>>
>>> But I can hit that URL with a GET,
>>>
>>> $ curl http://backfill-1d.i.massrel.com:8983/solr/15724/update
>>> 
>>> 
>>> 400>> name="QTime">2missing content
>>> stream400
>>> 
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>
>>
>

Problem querying collection in Solr 4.1

[Brett Hoerner]

I have a collection in Solr 4.1 RC1 and doing a simple query like
text:"puppy dog" is causing an exception. Oddly enough, I CAN query for
text:puppy or text:"puppy", but adding the space breaks everything.

Schema and config: https://gist.github.com/f49da15e39e5609b75b1

This happens whether I query the whole collection or a single direct core.
I haven't tested whether this would happen outside of SolrCloud.

http://localhost:8984/solr/timeline/select?q=text%3A%22puppy+dog%22&wt=xml

http://localhost:8984/solr/timeline_shard4_replica1/select?q=text%3A%22puppy+dog%22&wt=xml

Jan 22, 2013 12:07:24 AM org.apache.solr.common.SolrException log
SEVERE: null:org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers
available to handle this request:[
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard2_replica1,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard1_replica2,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard3_replica2,
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard4_replica1,
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard1_replica1,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard2_replica2,
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard3_replica1,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard4_replica2]
 at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
 at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
 at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
 at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
 at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
 at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.eclipse.jetty.server.Server.handle(Server.java:365)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
 at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
 at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
 at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live
SolrServers available to handle this request:[
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard2_replica1,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard1_replica2,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard3_replica2,
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard4_replica1,
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard1_replica1,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard2_replica2,
http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard3_replica1,
http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard4_replica2]
 at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:325)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:171)
 at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:135)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.

Re: Problem querying collection in Solr 4.1

[Brett Hoerner]

Thanks, I'll check that out.

Turns out our problem was we had omitTermFreqAndPositions true but were
running queries like "puppy dog" which, I would imagine, require position.


On Mon, Jan 21, 2013 at 9:22 PM, Gopal Patwa  wrote:

> one thing I noticed in solrconfig xml that it set to use Lucene version 4.0
> index format but you  mention you are using it 4.1
>
>   LUCENE_40
>
>
>
> On Mon, Jan 21, 2013 at 4:26 PM, Brett Hoerner  >wrote:
>
> > I have a collection in Solr 4.1 RC1 and doing a simple query like
> > text:"puppy dog" is causing an exception. Oddly enough, I CAN query for
> > text:puppy or text:"puppy", but adding the space breaks everything.
> >
> > Schema and config: https://gist.github.com/f49da15e39e5609b75b1
> >
> > This happens whether I query the whole collection or a single direct
> core.
> > I haven't tested whether this would happen outside of SolrCloud.
> >
> >
> http://localhost:8984/solr/timeline/select?q=text%3A%22puppy+dog%22&wt=xml
> >
> >
> >
> http://localhost:8984/solr/timeline_shard4_replica1/select?q=text%3A%22puppy+dog%22&wt=xml
> >
> > Jan 22, 2013 12:07:24 AM org.apache.solr.common.SolrException log
> > SEVERE: null:org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> > available to handle this request:[
> >
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard2_replica1,
> >
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard1_replica2,
> >
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard3_replica2,
> >
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard4_replica1,
> >
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard1_replica1,
> >
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard2_replica2,
> >
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard3_replica1,
> >
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard4_replica2]
> >  at
> >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302)
> > at
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
> >  at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
> > at
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
> >  at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> >  at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
> > at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> >  at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
> >  at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
> >  at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> >  at
> >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> >  at org.eclipse.jetty.server.Server.handle(Server.java:365)
> > at
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
> >  at
> >
> >
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> > at
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
> >  at
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
> > at org.eclipse.jetty.http.HttpP

Is it possible to manually select a shard leader in a running SolrCloud?

[Brett Hoerner]

Hi,

I have a 5 server cluster running 1 collection with 20 shards, replication
factor of 2.

Earlier this week I had to do a rolling restart across the cluster, this
worked great and the cluster stayed up the whole time. The problem is that
the last node I restarted is now the leader of 0 shards, and is just
holding replicas.

I've noticed this node has abnormally high load average, while the other
nodes (who have the same number of shards, but more leaders on average) are
fine.

First, I'm wondering if that loud could be related to being a 5x replica
and 0x leader?

Second, I was wondering if I could somehow flag single shards to re-elect a
leader (or force a leader) so that I could more evenly distribute how many
leader shards each physical server has running?

Thanks.

Re: Is it possible to manually select a shard leader in a running SolrCloud?

[Brett Hoerner]

As an update, it looks like the heavy load is in part because the node
never "catches back up" with the other nodes. In SolrCloud UI it was yellow
for a long time, then eventually grey, then back to yellow and orange. It
never recovers as green.

I should note this collection is very busy, indexing 5k+ small documents
per second, but the nodes were all fine until I had to restart them and
they had to re-sync.

Here is the log since reboot: https://gist.github.com/396af4b217ce8f536db6

Any ideas?


On Sat, Feb 2, 2013 at 10:27 AM, Brett Hoerner wrote:

> Hi,
>
> I have a 5 server cluster running 1 collection with 20 shards, replication
> factor of 2.
>
> Earlier this week I had to do a rolling restart across the cluster, this
> worked great and the cluster stayed up the whole time. The problem is that
> the last node I restarted is now the leader of 0 shards, and is just
> holding replicas.
>
> I've noticed this node has abnormally high load average, while the other
> nodes (who have the same number of shards, but more leaders on average) are
> fine.
>
> First, I'm wondering if that loud could be related to being a 5x replica
> and 0x leader?
>
> Second, I was wondering if I could somehow flag single shards to re-elect
> a leader (or force a leader) so that I could more evenly distribute how
> many leader shards each physical server has running?
>
> Thanks.
>

Re: Is it possible to manually select a shard leader in a running SolrCloud?

[Brett Hoerner]

What is the inverse I'd use to re-create/load a core on another machine but
make sure it's also "known" to SolrCloud/as a shard?


On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale  wrote:

>
> To be more clear lets say bob it the leader of core 1. On bob do a
> /admin/cores?action=unload&name=core1. This removes the core/shard from
> bob, giving the other servers a chance to grab leader props.
>
> -Joey
>
> On Feb 2, 2013, at 11:27 AM, Brett Hoerner  wrote:
>
> > Hi,
> >
> > I have a 5 server cluster running 1 collection with 20 shards,
> replication
> > factor of 2.
> >
> > Earlier this week I had to do a rolling restart across the cluster, this
> > worked great and the cluster stayed up the whole time. The problem is
> that
> > the last node I restarted is now the leader of 0 shards, and is just
> > holding replicas.
> >
> > I've noticed this node has abnormally high load average, while the other
> > nodes (who have the same number of shards, but more leaders on average)
> are
> > fine.
> >
> > First, I'm wondering if that loud could be related to being a 5x replica
> > and 0x leader?
> >
> > Second, I was wondering if I could somehow flag single shards to
> re-elect a
> > leader (or force a leader) so that I could more evenly distribute how
> many
> > leader shards each physical server has running?
> >
> > Thanks.
>
>

Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

I have a SolrCloud cluster (2 machines, 2 Solr instances, 32 shards,
replication factor of 2) that I've been using for over a month now in
production.

Suddenly, the hourly cron I run that dispatches a delete by query
completely halts all indexing. Select queries still run (and quickly),
there is no CPU or disk I/O happening, but suddenly my indexer (which runs
at ~400 doc/sec steady) pauses, and everything blocks indefinitely.

To clarify some on the schema, this is a moving window of data (imagine
messages that don't matter after a 24 hour period) which are regularly
"chopped" off by my hourly cron (deleting messages over 24 hours old) to
keep the index size reasonable.

There are no errors (log level warn) in the logs. I'm not sure what to look
into. As I've said this has been running (delete included) for about a
month.

I'll also note that I have another cluster much like this one where I do
the very same thing... it has 4 machines, and indexes 10x the documents per
second, with more indexes... and yet I delete on a cron without issue...

Any ideas on where to start, or other information I could provide?

Thanks much.

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

4.1, I'll induce it again and run jstack.


On Wed, Mar 6, 2013 at 1:50 PM, Mark Miller  wrote:

> Which version of Solr?
>
> Can you use jconsole, visualvm, or jstack to get some stack traces and see
> where things are halting?
>
> - Mark
>
> On Mar 6, 2013, at 11:45 AM, Brett Hoerner  wrote:
>
> > I have a SolrCloud cluster (2 machines, 2 Solr instances, 32 shards,
> > replication factor of 2) that I've been using for over a month now in
> > production.
> >
> > Suddenly, the hourly cron I run that dispatches a delete by query
> > completely halts all indexing. Select queries still run (and quickly),
> > there is no CPU or disk I/O happening, but suddenly my indexer (which
> runs
> > at ~400 doc/sec steady) pauses, and everything blocks indefinitely.
> >
> > To clarify some on the schema, this is a moving window of data (imagine
> > messages that don't matter after a 24 hour period) which are regularly
> > "chopped" off by my hourly cron (deleting messages over 24 hours old) to
> > keep the index size reasonable.
> >
> > There are no errors (log level warn) in the logs. I'm not sure what to
> look
> > into. As I've said this has been running (delete included) for about a
> > month.
> >
> > I'll also note that I have another cluster much like this one where I do
> > the very same thing... it has 4 machines, and indexes 10x the documents
> per
> > second, with more indexes... and yet I delete on a cron without issue...
> >
> > Any ideas on where to start, or other information I could provide?
> >
> > Thanks much.
>
>

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

Here is a dump after the delete, indexing has been stopped:
https://gist.github.com/bretthoerner/c7ea3bf3dc9e676a3f0e

An interesting hint that I forgot to mention: it doesn't always happen on
the first delete. I manually ran the delete cron, and the server continued
to work. I waited about 5 minutes and ran it again and it stalled the
indexer (as seen from indexer process): http://i.imgur.com/1Tt35u0.png

Another thing I forgot to mention. To bring the cluster back to life I:

1) stop my indexer
2) stop server1, start server1
3) stop server2, start start2
4) manually rebalance half of the shards to be mastered on server2
(unload/create on server1)
5) restart indexer

And it works again until a delete eventually kills it.

To be clear again, select queries continue to work indefinitely.

Thanks,
Brett


On Wed, Mar 6, 2013 at 1:50 PM, Mark Miller  wrote:

> Which version of Solr?
>
> Can you use jconsole, visualvm, or jstack to get some stack traces and see
> where things are halting?
>
> - Mark
>
> On Mar 6, 2013, at 11:45 AM, Brett Hoerner  wrote:
>
> > I have a SolrCloud cluster (2 machines, 2 Solr instances, 32 shards,
> > replication factor of 2) that I've been using for over a month now in
> > production.
> >
> > Suddenly, the hourly cron I run that dispatches a delete by query
> > completely halts all indexing. Select queries still run (and quickly),
> > there is no CPU or disk I/O happening, but suddenly my indexer (which
> runs
> > at ~400 doc/sec steady) pauses, and everything blocks indefinitely.
> >
> > To clarify some on the schema, this is a moving window of data (imagine
> > messages that don't matter after a 24 hour period) which are regularly
> > "chopped" off by my hourly cron (deleting messages over 24 hours old) to
> > keep the index size reasonable.
> >
> > There are no errors (log level warn) in the logs. I'm not sure what to
> look
> > into. As I've said this has been running (delete included) for about a
> > month.
> >
> > I'll also note that I have another cluster much like this one where I do
> > the very same thing... it has 4 machines, and indexes 10x the documents
> per
> > second, with more indexes... and yet I delete on a cron without issue...
> >
> > Any ideas on where to start, or other information I could provide?
> >
> > Thanks much.
>
>

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

If there's anything I can try, let me know. Interestingly, I think I have
noticed that if I stop my indexer, do my delete, and restart the indexer
then I'm fine. Which goes along with the update thread contention theory.


On Wed, Mar 6, 2013 at 5:03 PM, Mark Miller  wrote:

> This is what I see:
>
> We currently limit the number of outstanding update requests at one time
> to avoid a crazy number of threads being used.
>
> It looks like a bunch of update requests are stuck in socket reads and are
> taking up the available threads. It looks like the deletes are hanging out
> waiting for a free thread.
>
> It seems the question is, why are the requests stuck in socket reads. I
> don't have an answer at the moment.
>
> We should probably get this into a JIRA issue though.
>
> - Mark
>
>
> On Mar 6, 2013, at 2:15 PM, Alexandre Rafalovitch 
> wrote:
>
> > It does not look like a deadlock, though it could be a distributed one.
> Or
> > it could be a livelock, though that's less likely.
> >
> > Here is what we used to recommend in similar situations for large Java
> > systems (BEA Weblogic):
> > 1) Do thread dump of both systems before anything. As simultaneous as you
> > can make it.
> > 2) Do the first delete. Do a thread dump every 2 minutes on both servers
> > (so, say 3 dumps in that 5 minute wait)
> > 3) Do the second delete and do thread dumps every 30 seconds on both
> > servers from just before and then during. Preferably all the way until
> the
> > problem shows itself. Every 5 seconds if the problem shows itself really
> > quick.
> >
> > That gives you a LOT of thread dumps. But it also gives you something
> that
> > allows to compare thread state before and after the problem starting
> > showing itself and to identify moving (or unnaturally still) threads. I
> > even wrote a tool long time ago that parsed those thread dumps
> > automatically and generated pretty deadlock graphs of those.
> >
> >
> > Regards,
> >   Alex.
> >
> >
> >
> >
> >
> > Personal blog: http://blog.outerthoughts.com/
> > LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> > - Time is the quality of nature that keeps events from happening all at
> > once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD book)
> >
> >
> > On Wed, Mar 6, 2013 at 5:04 PM, Mark Miller 
> wrote:
> >
> >> Thans Brett, good stuff (though not a good problem).
> >>
> >> We def need to look into this.
> >>
> >> - Mark
> >>
> >> On Mar 6, 2013, at 1:53 PM, Brett Hoerner 
> wrote:
> >>
> >>> Here is a dump after the delete, indexing has been stopped:
> >>> https://gist.github.com/bretthoerner/c7ea3bf3dc9e676a3f0e
> >>>
> >>> An interesting hint that I forgot to mention: it doesn't always happen
> on
> >>> the first delete. I manually ran the delete cron, and the server
> >> continued
> >>> to work. I waited about 5 minutes and ran it again and it stalled the
> >>> indexer (as seen from indexer process): http://i.imgur.com/1Tt35u0.png
> >>>
> >>> Another thing I forgot to mention. To bring the cluster back to life I:
> >>>
> >>> 1) stop my indexer
> >>> 2) stop server1, start server1
> >>> 3) stop server2, start start2
> >>> 4) manually rebalance half of the shards to be mastered on server2
> >>> (unload/create on server1)
> >>> 5) restart indexer
> >>>
> >>> And it works again until a delete eventually kills it.
> >>>
> >>> To be clear again, select queries continue to work indefinitely.
> >>>
> >>> Thanks,
> >>> Brett
> >>>
> >>>
> >>> On Wed, Mar 6, 2013 at 1:50 PM, Mark Miller 
> >> wrote:
> >>>
> >>>> Which version of Solr?
> >>>>
> >>>> Can you use jconsole, visualvm, or jstack to get some stack traces and
> >> see
> >>>> where things are halting?
> >>>>
> >>>> - Mark
> >>>>
> >>>> On Mar 6, 2013, at 11:45 AM, Brett Hoerner 
> >> wrote:
> >>>>
> >>>>> I have a SolrCloud cluster (2 machines, 2 Solr instances, 32 shards,
> >>>>> replication factor of 2) that I've been using for over a month now in
> >>>>> production.
> >>>>>
> >>>>> Suddenly, the hourly cron I run t

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

Here is the other server when it's locked:
https://gist.github.com/3529b7b6415756ead413

To be clear, neither is really "the replica", I have 32 shards and each
physical server is the leader for 16, and the replica for 16.

Also, related to the max threads hunch: my working cluster has many, many
fewer shards per Solr instance. I'm going to do some migration dancing on
this cluster today to have more Solr JVMs each with fewer cores, and see
how it affects the deletes.


On Wed, Mar 6, 2013 at 5:40 PM, Mark Miller  wrote:

> Any chance you can grab the stack trace of a replica as well? (also when
> it's locked up of course).
>
> - Mark
>
> On Mar 6, 2013, at 3:34 PM, Brett Hoerner  wrote:
>
> > If there's anything I can try, let me know. Interestingly, I think I have
> > noticed that if I stop my indexer, do my delete, and restart the indexer
> > then I'm fine. Which goes along with the update thread contention theory.
> >
> >
> > On Wed, Mar 6, 2013 at 5:03 PM, Mark Miller 
> wrote:
> >
> >> This is what I see:
> >>
> >> We currently limit the number of outstanding update requests at one time
> >> to avoid a crazy number of threads being used.
> >>
> >> It looks like a bunch of update requests are stuck in socket reads and
> are
> >> taking up the available threads. It looks like the deletes are hanging
> out
> >> waiting for a free thread.
> >>
> >> It seems the question is, why are the requests stuck in socket reads. I
> >> don't have an answer at the moment.
> >>
> >> We should probably get this into a JIRA issue though.
> >>
> >> - Mark
> >>
> >>
> >> On Mar 6, 2013, at 2:15 PM, Alexandre Rafalovitch 
> >> wrote:
> >>
> >>> It does not look like a deadlock, though it could be a distributed one.
> >> Or
> >>> it could be a livelock, though that's less likely.
> >>>
> >>> Here is what we used to recommend in similar situations for large Java
> >>> systems (BEA Weblogic):
> >>> 1) Do thread dump of both systems before anything. As simultaneous as
> you
> >>> can make it.
> >>> 2) Do the first delete. Do a thread dump every 2 minutes on both
> servers
> >>> (so, say 3 dumps in that 5 minute wait)
> >>> 3) Do the second delete and do thread dumps every 30 seconds on both
> >>> servers from just before and then during. Preferably all the way until
> >> the
> >>> problem shows itself. Every 5 seconds if the problem shows itself
> really
> >>> quick.
> >>>
> >>> That gives you a LOT of thread dumps. But it also gives you something
> >> that
> >>> allows to compare thread state before and after the problem starting
> >>> showing itself and to identify moving (or unnaturally still) threads. I
> >>> even wrote a tool long time ago that parsed those thread dumps
> >>> automatically and generated pretty deadlock graphs of those.
> >>>
> >>>
> >>> Regards,
> >>>  Alex.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Personal blog: http://blog.outerthoughts.com/
> >>> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> >>> - Time is the quality of nature that keeps events from happening all at
> >>> once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> book)
> >>>
> >>>
> >>> On Wed, Mar 6, 2013 at 5:04 PM, Mark Miller 
> >> wrote:
> >>>
> >>>> Thans Brett, good stuff (though not a good problem).
> >>>>
> >>>> We def need to look into this.
> >>>>
> >>>> - Mark
> >>>>
> >>>> On Mar 6, 2013, at 1:53 PM, Brett Hoerner 
> >> wrote:
> >>>>
> >>>>> Here is a dump after the delete, indexing has been stopped:
> >>>>> https://gist.github.com/bretthoerner/c7ea3bf3dc9e676a3f0e
> >>>>>
> >>>>> An interesting hint that I forgot to mention: it doesn't always
> happen
> >> on
> >>>>> the first delete. I manually ran the delete cron, and the server
> >>>> continued
> >>>>> to work. I waited about 5 minutes and ran it again and it stalled the
> >>>>> indexer (as seen from indexer process):
> http://i.imgur.com/1Tt35u0.png
> >>>>>
> >>>&g

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

As a side note, do you think that was a poor idea? I figured it's better to
spread the master "load" around?


On Thu, Mar 7, 2013 at 11:29 AM, Mark Miller  wrote:

>
> On Mar 7, 2013, at 9:03 AM, Brett Hoerner  wrote:
>
> > To be clear, neither is really "the replica", I have 32 shards and each
> > physical server is the leader for 16, and the replica for 16.
>
> Ah, interesting. That actually could be part of the issue - some brain
> cells are firing. I'm away from home till this weekend, but I can try and
> duplicate this when I get to my home base setup.
>
> - Mark

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

[Brett Hoerner]

As an update to this, I did my SolrCloud dance and made it 2xJVMs per
machine (2 machines still, the same ones) and spread the load around. Each
Solr instance now has 16 total shards (master for 8, replica for 8).

*drum roll* ... I can repeatedly run my delete script and nothing breaks. :)


On Thu, Mar 7, 2013 at 11:03 AM, Brett Hoerner wrote:

> Here is the other server when it's locked:
> https://gist.github.com/3529b7b6415756ead413
>
> To be clear, neither is really "the replica", I have 32 shards and each
> physical server is the leader for 16, and the replica for 16.
>
> Also, related to the max threads hunch: my working cluster has many, many
> fewer shards per Solr instance. I'm going to do some migration dancing on
> this cluster today to have more Solr JVMs each with fewer cores, and see
> how it affects the deletes.
>
>
> On Wed, Mar 6, 2013 at 5:40 PM, Mark Miller  wrote:
>
>> Any chance you can grab the stack trace of a replica as well? (also when
>> it's locked up of course).
>>
>> - Mark
>>
>> On Mar 6, 2013, at 3:34 PM, Brett Hoerner  wrote:
>>
>> > If there's anything I can try, let me know. Interestingly, I think I
>> have
>> > noticed that if I stop my indexer, do my delete, and restart the indexer
>> > then I'm fine. Which goes along with the update thread contention
>> theory.
>> >
>> >
>> > On Wed, Mar 6, 2013 at 5:03 PM, Mark Miller 
>> wrote:
>> >
>> >> This is what I see:
>> >>
>> >> We currently limit the number of outstanding update requests at one
>> time
>> >> to avoid a crazy number of threads being used.
>> >>
>> >> It looks like a bunch of update requests are stuck in socket reads and
>> are
>> >> taking up the available threads. It looks like the deletes are hanging
>> out
>> >> waiting for a free thread.
>> >>
>> >> It seems the question is, why are the requests stuck in socket reads. I
>> >> don't have an answer at the moment.
>> >>
>> >> We should probably get this into a JIRA issue though.
>> >>
>> >> - Mark
>> >>
>> >>
>> >> On Mar 6, 2013, at 2:15 PM, Alexandre Rafalovitch 
>> >> wrote:
>> >>
>> >>> It does not look like a deadlock, though it could be a distributed
>> one.
>> >> Or
>> >>> it could be a livelock, though that's less likely.
>> >>>
>> >>> Here is what we used to recommend in similar situations for large Java
>> >>> systems (BEA Weblogic):
>> >>> 1) Do thread dump of both systems before anything. As simultaneous as
>> you
>> >>> can make it.
>> >>> 2) Do the first delete. Do a thread dump every 2 minutes on both
>> servers
>> >>> (so, say 3 dumps in that 5 minute wait)
>> >>> 3) Do the second delete and do thread dumps every 30 seconds on both
>> >>> servers from just before and then during. Preferably all the way until
>> >> the
>> >>> problem shows itself. Every 5 seconds if the problem shows itself
>> really
>> >>> quick.
>> >>>
>> >>> That gives you a LOT of thread dumps. But it also gives you something
>> >> that
>> >>> allows to compare thread state before and after the problem starting
>> >>> showing itself and to identify moving (or unnaturally still) threads.
>> I
>> >>> even wrote a tool long time ago that parsed those thread dumps
>> >>> automatically and generated pretty deadlock graphs of those.
>> >>>
>> >>>
>> >>> Regards,
>> >>>  Alex.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Personal blog: http://blog.outerthoughts.com/
>> >>> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
>> >>> - Time is the quality of nature that keeps events from happening all
>> at
>> >>> once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
>> book)
>> >>>
>> >>>
>> >>> On Wed, Mar 6, 2013 at 5:04 PM, Mark Miller 
>> >> wrote:
>> >>>
>> >>>> Thans Brett, good stuff (though not a good problem).
>> >>>>
>> >>>> We def need to look into this.
>> >>>>
>> >>>> - Ma

URL Case Sensitive/Insensitive

[Moyer, Brett]

Hello, I'm new to Solr been using it for a few months. A recent question came 
up from our business partners about URL casing. Previously their URLs were 
upper case, they made a change and now all lower. Both pages/URLs are still 
accessible so there are duplicates in Solr. They are requesting all URLs be 
evaluated as lowercase. What is the best practice on URL case? Is there a 
negative to making all lowercase? I know I can drop the index and re-crawl to 
fix it, but long term how should URL case be treated? Thanks!

Brett Moyer

*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: URL Case Sensitive/Insensitive

[Moyer, Brett]

https://www.nuveen.com/mutual-funds/nuveen-high-yield-municipal-bond-fund
https://www.nuveen.com/mutual-funds/Nuveen-High-Yield-Municipal-Bond-Fund

Is there any issue if we just lowercase all URLs? I can't think of an issue 
that would be caused, but that's why I'm asking the Guru's!

Brett Moyer
   

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, December 11, 2018 12:41 PM
To: solr-user
Subject: Re: URL Case Sensitive/Insensitive

What do you mean by "url case"? No, I'm not being snarky.

The value returned in a doc is very different than the value searched.
The stored data is the original input without going through any
filters.

If you mean the value _returned_ by Solr from a stored field, then the
case is exactly whatever was input originally. To get it a consistent
case, I'd change it on the client side before sending  to Solr, or
use, say, a  ScriptUpdateProcessor to change it on the way in to Solr.

If you're talking about _searching_ the URL, you need to put the
appropriate filters in your analysis chain. Most distributions have a
"lowercase" type that is a keywordtokenizer and lowercasefilter That
still treats the searchable text as a single token, so for instance
you wouldn't be able to search for url:com with pre-and-post wildcards
which is not a good pattern. If you want to search sub-parts of a url,
you'll use one of the text-based types to break it up into tokens.
Even in this case, though, the returned data is still the original
case since it's the stored data that's returned.

Best,
Erick
On Tue, Dec 11, 2018 at 8:38 AM Moyer, Brett  wrote:
>
> Hello, I'm new to Solr been using it for a few months. A recent question came 
> up from our business partners about URL casing. Previously their URLs were 
> upper case, they made a change and now all lower. Both pages/URLs are still 
> accessible so there are duplicates in Solr. They are requesting all URLs be 
> evaluated as lowercase. What is the best practice on URL case? Is there a 
> negative to making all lowercase? I know I can drop the index and re-crawl to 
> fix it, but long term how should URL case be treated? Thanks!
>
> Brett Moyer
>
> *
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender immediately 
> and then delete it.
>
> TIAA
> *
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

FieldTypes and LowerCase

[Moyer, Brett]

I'm using the below FieldType/Field but when I index my documents, the URL is 
not being lower case. Any ideas? Do I have the below wrong?

Example: http://connect.rightprospectus.com/RSVP/TADF
Expect: http://connect.rightprospectus.com/rsvp/tadf



  
  





Brett Moyer


*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: FieldTypes and LowerCase

[Moyer, Brett]

Thanks Shawn, " Analysis only happens to indexed data" Being the case when the 
data gets Indexed, then wouldn't the Analyzer kickoff and lowercase the URL? 
The analyzer I have defined is not set for Index or Query, so as I understand 
it will fire during both events. If that is the case I still don't get why the 
Lowercase doesn't fire when the data is being indexed. 

Brett Moyer

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Thursday, March 14, 2019 10:44 AM
To: solr-user@lucene.apache.org
Subject: Re: FieldTypes and LowerCase

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


On 3/14/2019 7:47 AM, Moyer, Brett wrote:
> I'm using the below FieldType/Field but when I index my documents, the URL is 
> not being lower case. Any ideas? Do I have the below wrong?
>
> Example: http://connect.rightprospectus.com/RSVP/TADF
> Expect: http://connect.rightprospectus.com/rsvp/tadf
>
>  omitNorms="true">
> 
>
>
> 
> 
>
>  stored="true"/>

Analysis only happens to indexed data.

The data that you get back from Solr (stored data) is *always* EXACTLY
what Solr indexes, before analysis.

You'll need to lowercase the data before it reaches analysis.  This is
how it is designed to work ... that will not be changing.

If you were to configure an Update Processor chain that did the
lowercasing, that would affect stored data as well as indexed data.

Thanks,
Shawn
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: FieldTypes and LowerCase

[Moyer, Brett]

Ok I think I'm getting it. At Index/Query time the analyzers fire and "do 
stuff". Ex: "the sheep jumped over the MOON" that could be Tokened on spaces, 
lowercased etc. and that is stored in the Inverted Index, something you 
probably can't really see.

In solr the string above is what you see in its original form. When you search 
for "sheep" that would come back because the Inverted Index has it stored in 
that form, separated words based on spaces, right? Further if I searched for 
moon (lowercase) it would be found because the analyzer is also storing in the 
Inverted Index the lowercase form, right?

I'm getting closer I think. Ok so if I want to physically lowercase the URL and 
store it that way, I need to do it before it gets to the Index as you stated. 
Ok got it, Thanks!

Brett Moyer
Manager, Sr. Technical Lead | TFS Technology
  Public Production Support
  Digital Search & Discovery

8625 Andrew Carnegie Blvd | 4th floor
Charlotte, NC 28263
Tel: 704.988.4508
Fax: 704.988.4907
bmo...@tiaa.org 


-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Thursday, March 14, 2019 10:57 AM
To: solr-user@lucene.apache.org
Subject: Re: FieldTypes and LowerCase

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


On 3/14/2019 8:49 AM, Moyer, Brett wrote:
> Thanks Shawn, " Analysis only happens to indexed data" Being the case when 
> the data gets Indexed, then wouldn't the Analyzer kickoff and lowercase the 
> URL? The analyzer I have defined is not set for Index or Query, so as I 
> understand it will fire during both events. If that is the case I still don't 
> get why the Lowercase doesn't fire when the data is being indexed.

It does happen for both index and query.

It sounds like you are assuming that when index analysis happens, that
what you get back in search results will be affected by that analysis.

What you get back in search results is stored data -- that is never
affected by analysis.

What gets affected by analysis is indexed data -- the data that is
searched by queries.  Not the data that comes back in search results.

Thanks,
Shawn
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

IRA or IRA the Person

[Moyer, Brett]

Hello,

Looking for ideas on how to determine intent and drive results to a 
person result or an article result. We are a financial institution and we have 
IRA's Individual Retirement Accounts and we have a page that talks about an 
Advisor, IRA Black.

Our users are in a bad habit of only using single terms for search. A 
very common search term is "ira". The PERSON page ranks higher than the article 
on IRA's. With essentially no information from the user, what are some way we 
can detect and rank differently? Thanks!

Brett Moyer
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: IRA or IRA the Person

[Moyer, Brett]

Wow, thank you Trey, great information! We are a Fusion client, works well for 
us, we are leveraging the Signals Boosting. We were thinking omitNorms might be 
of help here, turning that off actually. The PERSON document ranks #1 always 
because it’s a tiny document with very short fields. I'll take a closer look at 
what you sent, Thank you!

Brett Moyer
Manager, Sr. Technical Lead | TFS Technology
  Public Production Support
  Digital Search & Discovery

8625 Andrew Carnegie Blvd | 4th floor
Charlotte, NC 28263
Tel: 704.988.4508
Fax: 704.988.4907
bmo...@tiaa.org 


-Original Message-
From: Trey Grainger [mailto:solrt...@gmail.com] 
Sent: Monday, April 01, 2019 1:15 PM
To: solr-user@lucene.apache.org
Subject: Re: IRA or IRA the Person

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


Hi Brett,

There are a couple of angles you can take here. If you are only concerned
about this specific term or a small number of other known terms like "IRA"
and want to spot fix it, you can use something like the query elevation
component in Solr (
https://lucene.apache.org/solr/guide/7_7/the-query-elevation-component.html)
to explicitly include or exclude documents.

Otherwise, if you are looking for a more data-driven approach to solving
this, you can leverage the aggregate click-streams for your users across
all of the searches on your platform to boost documents higher that are
more popular for any given search. We do this in our commercial product
(Lucidworks Fusion) through our Signals Boosting feature, but you could
implement something similar yourself with some work, as the general
architecture is fairly well-documented here:
https://doc.lucidworks.com/fusion-ai/4.2/user-guide/signals/index.html

If you do not have long-lived content OR your do not have sufficient
signals history, you could alternatively use something like Solr's Semantic
Knowledge Graph to automatically find term vectors that are the most
related to your terms within your content. In that case, if the "individual
retirement account" meaning is more common across your documents, you'd
probably end up with terms more related to that which could be used to do
data-driven boosts on your query to that concept (instead of the person, in
this case).

I gave a presentation at Activate ("the Search & AI Conference") last year
on some of the more data-driven approaches to parsing and understanding the
meaning of terms within queries, that included things like disambiguation
(similar to what you're doing here) and some additional approaches
leveraging a combination of query log mining, the semantic knowledge graph,
and the Solr Text Tagger. If you start handling these use cases in a more
systematic and data-driven way, you might want to check out some of the
techniques I mentioned there: Video:
https://www.youtube.com/watch?v=4fMZnunTRF8 | Slides:
https://www.slideshare.net/treygrainger/how-to-build-a-semantic-search-system


All the best,

Trey Grainger
Chief Algorithms Officer @ Lucidworks


On Mon, Apr 1, 2019 at 11:45 AM Moyer, Brett  wrote:

> Hello,
>
> Looking for ideas on how to determine intent and drive results to
> a person result or an article result. We are a financial institution and we
> have IRA's Individual Retirement Accounts and we have a page that talks
> about an Advisor, IRA Black.
>
> Our users are in a bad habit of only using single terms for
> search. A very common search term is "ira". The PERSON page ranks higher
> than the article on IRA's. With essentially no information from the user,
> what are some way we can detect and rank differently? Thanks!
>
> Brett Moyer
> *
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender
> immediately and then delete it.
>
> TIAA
> *
>
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

Solr spellcheck Collation JSON

[Moyer, Brett]

Hello,

Looks like a more recent Solr release introduced a bug for collation. 
Does anyone know of a way to correct it, or if a future release will address? 
Because of this change we had to make the app teams rewrite their code. Made us 
look bad because we can't control our code and introduced a bug their 
perspective) Thanks

Solr 7.4
--
"spellcheck": {
"suggestions": [
"acount",
{
"numFound": 1,
"startOffset": 0,
"endOffset": 6,
"suggestion": [
"account"
]
}
],
"collations": [
"collation", <-this is the bad line
"account"
]

Previous Solr versions
--
"spellcheck": {
"suggestions": [
"acount",
{
"numFound": 1,
"startOffset": 0,
"endOffset": 6,
"suggestion": [
"account"
]
}
],
"collations": [
"collation":"account" <--correct format
]

Brett Moyer
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

Auto-Suggest within Tier Architecture

[Moyer, Brett]

Hello,

Looking to see how others accomplished this goal. We have a 3 Tier 
architecture, Solr is down deep in T3 far from the end user. How do you make 
Auto-Suggest calls from the Internet Browser through the Tiers down to Solr in 
T3? We essentially created steps down each tier, but I'm looking to know what 
other approaches people have created. Did you put your solr in T1, I assume 
not, that would put it at risk. Thanks!

Brett Moyer
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

Indexed Data Size

[Moyer, Brett]

In our data/solr//data/index on the filesystem, we have files 
that go back 1 year. I don’t understand why and I doubt they are in use. Files 
with extensions like fdx,cfe,doc,pos,tip,dvm etc. Some of these are very large 
and running us out of server space. Our search indexes themselves are not 
large, in total we might have 50k documents.  How can I reduce this /data/solr 
space? Is this what the Solr Optimize command is for? Thanks!

Brett

*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: modify query response plugin

[Moyer, Brett]

Highlight? What about using the Highlighter? 
https://lucene.apache.org/solr/guide/6_6/highlighting.html

Brett Moyer
Manager, Sr. Technical Lead | TFS Technology
  Public Production Support
  Digital Search & Discovery

8625 Andrew Carnegie Blvd | 4th floor
Charlotte, NC 28263
Tel: 704.988.4508
Fax: 704.988.4907
bmo...@tiaa.org


-Original Message-
From: Maria Muslea  
Sent: Thursday, August 8, 2019 1:28 PM
To: solr-user@lucene.apache.org
Subject: Re: modify query response plugin

Thank you for your response. I believe that the Tagger is used for NER, which 
is different than what I am trying to do.
It is also available only with Solr 7 and I would need this to work with 
version 6.5.0.

I am trying to manipulate the data that I already have in the response, and I 
can't find a good example of a plugin that does something similar, so I can see 
how I can access the response and construct a new one.

Your help is greatly appreciated.

Thank you,
Maria

On Tue, Aug 6, 2019 at 3:19 PM Erik Hatcher  wrote:

> I think you’re looking for the Solr Tagger, described here:
> https://lucidworks.com/post/solr-tagger-improving-relevancy/
>
> > On Aug 6, 2019, at 16:04, Maria Muslea  wrote:
> >
> > Hi,
> >
> > I am trying to implement a plugin that will modify my query 
> > response. For example, I would like to execute a query that will return 
> > something like:
> >
> > {...
> > "description":"flights at LAX",
> > "highlight":"airport;11;3"
> > ...}
> > This is information that I have in my document, so I can return it.
> >
> > Now, I would like the plugin to intercept the result, do some 
> > processing
> on
> > it, and return something like:
> >
> > {...
> > "description":"flights at LAX",
> > "highlight":{
> >   "concept":"airport",
> >   "description":"flights at LAX"
> > ...}
> >
> > I looked at some RequestHandler implementations, but I can't find 
> > any sample code that would help me with this. Would this type of 
> > plugin be handled by a RequestHandler? Could you maybe point me to a 
> > sample plugin that does something similar?
> >
> > I would really appreciate your help.
> >
> > Thank you,
> > Maria
>
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: Indexed Data Size

[Moyer, Brett]

Thanks! We update each index nightly, we don’t clear, but bring in New and 
Deltas, delete expired/404. All our data are basically webpages, so none are 
very large. Some PDFs but again not too large. We are running Solr 7.5, 
hopefully you can access the links.

https://www.dropbox.com/s/lzd6hkoikhagujs/CoreOne.png?dl=0
https://www.dropbox.com/s/ae6rayb38q39u9c/CoreTwo.png?dl=0

Brett
-Original Message-
From: Erick Erickson  
Sent: Thursday, August 8, 2019 5:49 PM
To: solr-user@lucene.apache.org
Subject: Re: Indexed Data Size

On the surface, this makes no sense at all, so there’s something I don’t 
understand here ;). 

How often do you update your index? Having files from a long time ago is 
perfectly reasonable if you’re not updating regularly.

But your statement that some of these are huge for just a 50K document index is 
odd unless they’re _huge_ documents.

I wouldn’t optimize, unless you’re on Solr 7.5+ as that’ll create a single 
segment, see: 
https://lucidworks.com/post/segment-merging-deleted-documents-optimize-may-bad/
and
https://lucidworks.com/post/solr-and-optimizing-your-index-take-ii/

The extensions you mentioned are perfectly reasonable. Each segment is made up 
of multiple files. .fdt for instance contains stored data. See: 
https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/codecs/lucene62/package-summary.html

Can you give us a long listing of one of your index directories?

Best,
Erick

> On Aug 8, 2019, at 5:17 PM, Moyer, Brett  wrote:
> 
> In our data/solr//data/index on the filesystem, we have files 
> that go back 1 year. I don’t understand why and I doubt they are in use. 
> Files with extensions like fdx,cfe,doc,pos,tip,dvm etc. Some of these are 
> very large and running us out of server space. Our search indexes themselves 
> are not large, in total we might have 50k documents.  How can I reduce this 
> /data/solr space? Is this what the Solr Optimize command is for? Thanks!
> 
> Brett
> 
> **
> *** This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender immediately 
> and then delete it.
> 
> TIAA
> **
> ***

*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: Indexed Data Size

[Moyer, Brett]

Correct our indexes are small document wise, but for some ready we have a 
years' worth of files in the data/solr folders. There are no index. 
files.

The biggest is /data/solr/system_logs_shard1_replica_n1/data/index, files with 
the extensions I stated previously. Each is 5gb and there are a few hundred. 
Dated by to last 3 months. I don’t understand why there are so many files with 
such small indexes. Not sure how to clean them up. 

-Original Message-
From: Shawn Heisey  
Sent: Friday, August 9, 2019 9:11 AM
To: solr-user@lucene.apache.org
Subject: Re: Indexed Data Size

On 8/9/2019 6:12 AM, Moyer, Brett wrote:
> Thanks! We update each index nightly, we don’t clear, but bring in New and 
> Deltas, delete expired/404. All our data are basically webpages, so none are 
> very large. Some PDFs but again not too large. We are running Solr 7.5, 
> hopefully you can access the links.

Solr is saying that the entire size of the index directory is 95 MB for one of 
those indexes and the other is 30 MB.  Those sound to me like very small 
indexes, not very large like you indicated.  You were saying that the large 
files were in data/index, and did not mention anything about index. 
directories.

If you do have a bunch of index. directories in the "Data" 
directory mentioned on the Core overview page, you can safely delete all of the 
index and/or index.* directories under that directory EXCEPT the one that is 
indicated as the "Index" directory.  If you delete that one, you're deleting 
the actual live index ... and since you're not on Windows, the OS will let you 
delete it without complaining.

The directory locations are cut off on both screenshots, so I can't confirm 
anything there.

The larger core has about 2000 deleted docs and the smaller one has 40. 
Doing an optimize will not save much disk space or take very long.

Thanks,
Shawn
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: Indexed Data Size

[Moyer, Brett]

Turns out this is due to a job that indexes logs. We were able to clear some 
with another job. We are working through the value of these indexed logs. 
Thanks for all your help!

Brett Moyer
Manager, Sr. Technical Lead | TFS Technology
  Public Production Support
  Digital Search & Discovery

8625 Andrew Carnegie Blvd | 4th floor
Charlotte, NC 28263
Tel: 704.988.4508
Fax: 704.988.4907
bmo...@tiaa.org

-Original Message-
From: Shawn Heisey  
Sent: Friday, August 9, 2019 2:25 PM
To: solr-user@lucene.apache.org
Subject: Re: Indexed Data Size

On 8/9/2019 12:17 PM, Moyer, Brett wrote:
> The biggest is /data/solr/system_logs_shard1_replica_n1/data/index, files 
> with the extensions I stated previously. Each is 5gb and there are a few 
> hundred. Dated by to last 3 months. I don’t understand why there are so many 
> files with such small indexes. Not sure how to clean them up.

Can you get a screenshot of the core overview for that particular core? 
Solr should correctly calculate the size on the overview based on what files 
are actually in the index directory.

Thanks,
Shawn
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

Facet Advice

[Moyer, Brett]

Hello, looking for some advice, I have the suspicion we are doing Facets all 
wrong. We host financial information and recently "tagged" our pages with 
appropriate Facets. We have built a Flat design. Are we going at it the wrong 
way?

In Solr we have a "Tags" field, based on some magic we tagged each page on the 
site with a number of the below example Facets. We have the UI team sending 
queries in the form of 1) q=get a loan&fq=Tags:Retirement, 2) q=get a 
loan&fq=Tags:Retirement AND Tags:Move Money. This restricts the resultset 
hopefully guiding the user to their desired result. Something about it doesn’t 
seem right. Is this right with a flat single level pattern like what we have? 
Should each doc have multiple Fields to map to different values? Any help is 
appreciated. Thanks!

Example Facets:
Brokerage
Retirement
Open an Account
Move Money
Estate Planning
Etc..

Brett
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: Facet Advice

[Moyer, Brett]

Hello Shawn, thanks for reply. The results that come back are correct, but are 
we implementing the query correctly to filter by a selected facet? When I say 
wrong, it's more about the design/use of Facets in the Query. Is it proper to 
do fq=Tags:Retirement? Is using a Multivalued field correct for Facets? Why do 
you say the above are not Facets?

Here is an excerpt from our JSON:

"facet_counts": {
"facet_queries": {},
"facet_fields": {
"Tags": [
"Retirement",
1260,
"Locations & People",
1149,
"Advice and Tools",
1015,
"Careers",
156,
"Annuities",
101,
"Performance",

Brett Moyer
Manager, Sr. Technical Lead | TFS Technology
  Public Production Support
  Digital Search & Discovery

8625 Andrew Carnegie Blvd | 4th floor
Charlotte, NC 28263
Tel: 704.988.4508
Fax: 704.988.4907
bmo...@tiaa.org

-Original Message-
From: Shawn Heisey  
Sent: Tuesday, October 15, 2019 5:40 AM
To: solr-user@lucene.apache.org
Subject: Re: Facet Advice

On 10/14/2019 3:25 PM, Moyer, Brett wrote:
> Hello, looking for some advice, I have the suspicion we are doing Facets all 
> wrong. We host financial information and recently "tagged" our pages with 
> appropriate Facets. We have built a Flat design. Are we going at it the wrong 
> way?
> 
> In Solr we have a "Tags" field, based on some magic we tagged each page on 
> the site with a number of the below example Facets. We have the UI team 
> sending queries in the form of 1) q=get a loan&fq=Tags:Retirement, 2) q=get a 
> loan&fq=Tags:Retirement AND Tags:Move Money. This restricts the resultset 
> hopefully guiding the user to their desired result. Something about it 
> doesn’t seem right. Is this right with a flat single level pattern like what 
> we have? Should each doc have multiple Fields to map to different values? Any 
> help is appreciated. Thanks!
> 
> Example Facets:
> Brokerage
> Retirement
> Open an Account
> Move Money
> Estate Planning

The queries you mentioned above do not have facets, only the q and fq 
parameters.  You also have not mentioned what in the results is wrong to you.

If you restrict the query to only a certain value in the tag field, then facets 
will only count documents that match the full query -- users will not be able 
to see the count of documents that do NOT match the query, unless you use 
tagging/excluding with your filters.  This is part of the functionality called 
multi-select faceting.

http://yonik.com/multi-select-faceting/

Because your message doesn't say what in the results is wrong, we can only 
guess about how to help you.  I do not know if the above information will be 
helpful or not.

Thanks,
Shawn
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

Odd Edge Case for SpellCheck

[Moyer, Brett]

Hello, we have spellcheck running, using the index as the dictionary. An odd 
use case came up today wanted to get your thoughts and see if what we 
determined is correct. Use case: User sends a query for q=brokerage, spellcheck 
fires and returns "brokerage". Looking at the output I see that solr must have 
pulled the root word "brokage" then spellcheck said hey I need to fix that. Is 
that correct? There's no issue, it's just an unexpected outcome. Thanks!

"q":"brokerage",
"spellcheck":{
"suggestions":
[
  {"name":"brokage",{
"type":"str","value":"numFound":1,
"startOffset":0,
"endOffset":9,
"suggestion":["brokerage"]}}],
"collations":
[
  {"name":"collation","type":"str","value":"brokerage"}]}}

Brett Moyer
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: Odd Edge Case for SpellCheck

[Moyer, Brett]

Yes we are stemming, ahh so we shouldn't stem our words to be spelled?

Brett Moyer

-Original Message-
From: Jörn Franke  
Sent: Friday, November 22, 2019 8:34 AM
To: solr-user@lucene.apache.org
Subject: Re: Odd Edge Case for SpellCheck

Stemming involved ?

> Am 22.11.2019 um 14:23 schrieb Moyer, Brett :
> 
> Hello, we have spellcheck running, using the index as the dictionary. An odd 
> use case came up today wanted to get your thoughts and see if what we 
> determined is correct. Use case: User sends a query for q=brokerage, 
> spellcheck fires and returns "brokerage". Looking at the output I see that 
> solr must have pulled the root word "brokage" then spellcheck said hey I need 
> to fix that. Is that correct? There's no issue, it's just an unexpected 
> outcome. Thanks!
> 
> "q":"brokerage",
> "spellcheck":{
>"suggestions":
>[
>  {"name":"brokage",{
>"type":"str","value":"numFound":1,
>    "startOffset":0,
>"endOffset":9,
>"suggestion":["brokerage"]}}],
>"collations":
>[
>  {"name":"collation","type":"str","value":"brokerage"}]}}
> 
> Brett Moyer
> **
> *** This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender immediately 
> and then delete it.
> 
> TIAA
> **
> ***
*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*

RE: Odd Edge Case for SpellCheck

[Moyer, Brett]

This is a great help, thank you!

Brett Moyer

-Original Message-
From: Erick Erickson  
Sent: Monday, November 25, 2019 4:12 PM
To: solr-user@lucene.apache.org
Subject: Re: Odd Edge Case for SpellCheck

If you’re using direct spell checking, it looks for the _indexed_ term. So this 
means you get stemmed corrections if you’re stemming etc. Usually you should 
use a copyField to a field with minimal analysis and use that field for 
spellchecking.

Another way to thing about it is that if you use the admin/analysis page for 
terms in a field, the terms in the dictionary are what’s at the end of the 
indexed side of the page.

Best,
Erick

> On Nov 25, 2019, at 4:02 PM, Moyer, Brett  wrote:
> 
> Yes we are stemming, ahh so we shouldn't stem our words to be spelled?
> 
> Brett Moyer
> 
> -Original Message-
> From: Jörn Franke 
> Sent: Friday, November 22, 2019 8:34 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Odd Edge Case for SpellCheck
> 
> Stemming involved ?
> 
>> Am 22.11.2019 um 14:23 schrieb Moyer, Brett :
>> 
>> Hello, we have spellcheck running, using the index as the dictionary. An 
>> odd use case came up today wanted to get your thoughts and see if what we 
>> determined is correct. Use case: User sends a query for q=brokerage, 
>> spellcheck fires and returns "brokerage". Looking at the output I see that 
>> solr must have pulled the root word "brokage" then spellcheck said hey I 
>> need to fix that. Is that correct? There's no issue, it's just an unexpected 
>> outcome. Thanks!
>> 
>> "q":"brokerage",
>> "spellcheck":{
>>   "suggestions":
>>   [
>> {"name":"brokage",{
>>   "type":"str","value":"numFound":1,
>>   "startOffset":0,
>>   "endOffset":9,
>>   "suggestion":["brokerage"]}}],
>>   "collations":
>>   [
>> {"name":"collation","type":"str","value":"brokerage"}]}}
>> 
>> Brett Moyer
>> *
>> *
>> *** This e-mail may contain confidential or privileged information.
>> If you are not the intended recipient, please notify the sender immediately 
>> and then delete it.
>> 
>> TIAA
>> *
>> *
>> ***
> **
> *** This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender immediately 
> and then delete it.
> 
> TIAA
> **
> ***

*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA
*