Re: DOcValues

2015-04-03 Thread Tomoko Uchida
Hi,

According to line 430 in SImpleFacet.java (Solr 5.0.0), facet method is
forced to "fc" when we set docValues=true.
https://github.com/apache/lucene-solr/blob/lucene_solr_5_0_0/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L430

So we need not set facet.method to use doc values. Even if we specify
facet.method=enum, it might be ignored.
If my understanding is wrong, please correct that.

Regards,
Tomoko


2015-04-03 12:01 GMT+09:00 William Bell :

> If I set indexed=true and docvalues=true, when I
> facet=true&facet.field=manu_exact
> will it use docValues or the Indexed version?
>
> Also, does it help with "*Too many values for UnInvertedField faceting" ?*
>
>
> *Do I need to set facet.method when using docvalues?*
>
>  docValues="true" />
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>


Re: DOcValues

2015-04-03 Thread Shawn Heisey
On 4/3/2015 6:53 AM, Tomoko Uchida wrote:
> According to line 430 in SImpleFacet.java (Solr 5.0.0), facet method is
> forced to "fc" when we set docValues=true.
> https://github.com/apache/lucene-solr/blob/lucene_solr_5_0_0/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L430
> 
> So we need not set facet.method to use doc values. Even if we specify
> facet.method=enum, it might be ignored.
> If my understanding is wrong, please correct that.

That code certainly looks like facet.method=enum is ignored when
docValues are present.

As I understand it, the only disadvantage to facet.method=fc (when
docValues are not present) is that it uses a lot of heap memory in the
FieldCache (or whatever replaces FieldCache in 5.0).  That memory
structure makes subsequent facets much faster, but on a large index, the
memory required can be astronomical.  The enum method skips that
caching, relying on the operating system to cache the data in the index
itself.  If there's enough memory for good OS caching, enum can be
almost as fast as fc, with a much smaller Java heap.

On a field with docValues, the large memory structure is not required,
and an optimized code path is used.  Based on the comment in the java
code that you highlighted, it sounds like only fc will do docValues, but
no handling is present for the fcs method with docValues, which would
seem to contradict that comment a little bit.

Thanks,
Shawn



Example of sorting by custom function

2015-04-03 Thread Robert Krüger
Hi,

I have been looking around on the web for information on sorting by a
custom function but the results are inconclusive to me and some of it seems
so old that I suspect it's outdated. What I want to do is the following:

I have a field "fingerprint" in my schema that contains a binary data (e.g.
64 bytes) that can be used to compute a distance/similarity between two
records.

Now I want to be able to execute a query and sort its result by the
distance of the matching records from a given reference value, so the query
would look something like this

q=*:*&sort=my_distance_func(fingerprint, 0xadet54786eguizgig)

where

my_distance_func is my custom function
fingerprint is the field in my schema (type binary)
0xadet54786eguizgig is a reference value (byte array encoded in whatever
way) to which the distance shall be computed, which differs with each query.

Is ValueSourceParser the right way to look here? Is there a source code
example someone can point me to?

Thanks in advance,

Robert


Re: DOcValues

2015-04-03 Thread Tomoko Uchida
Dear Shawn,

Thank you for the detailed explanation!
Many users would need such guidelines about memory consumption (and
performance trade-offs) for facets.

Thanks,
Tomoko

2015-04-03 22:26 GMT+09:00 Shawn Heisey :

> On 4/3/2015 6:53 AM, Tomoko Uchida wrote:
> > According to line 430 in SImpleFacet.java (Solr 5.0.0), facet method is
> > forced to "fc" when we set docValues=true.
> >
> https://github.com/apache/lucene-solr/blob/lucene_solr_5_0_0/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L430
> >
> > So we need not set facet.method to use doc values. Even if we specify
> > facet.method=enum, it might be ignored.
> > If my understanding is wrong, please correct that.
>
> That code certainly looks like facet.method=enum is ignored when
> docValues are present.
>
> As I understand it, the only disadvantage to facet.method=fc (when
> docValues are not present) is that it uses a lot of heap memory in the
> FieldCache (or whatever replaces FieldCache in 5.0).  That memory
> structure makes subsequent facets much faster, but on a large index, the
> memory required can be astronomical.  The enum method skips that
> caching, relying on the operating system to cache the data in the index
> itself.  If there's enough memory for good OS caching, enum can be
> almost as fast as fc, with a much smaller Java heap.
>
> On a field with docValues, the large memory structure is not required,
> and an optimized code path is used.  Based on the comment in the java
> code that you highlighted, it sounds like only fc will do docValues, but
> no handling is present for the fcs method with docValues, which would
> seem to contradict that comment a little bit.
>
> Thanks,
> Shawn
>
>


distributed search on tables

2015-04-03 Thread avinash09
Hi,

I have a use case search all the name="*test*" from two tables (product and
department)
i need distributed result 5 result from product and 5 from department

but i am getting first all result from which is 434 and then department as
shared below

http://localhost:8983/solr/test_core/select?q=name:*test*&wt=json&facet=true&facet.field=table_name


{
responseHeader: {
status: 0,
QTime: 19
},
response: {
numFound: 569,
start: 0,
docs: [
{
name: "test 6576",
table_name: "product"
},
{
name: "test 6578",
table_name: "product"
},
{
name: "test 65760",
table_name: "product"
},
{
name: "test 657699",
table_name: "product"
},
{
name: "test 657699",
table_name: "product"
},
{
name: "test 657666",
table_name: "product"
},
{
name: "test 657689",
table_name: "product"
},
{
name: "test 6576yu",
table_name: "product"
},
{
name: "test 657607",
table_name: "product"
},
{
name: "test 657687",
table_name: "product"
}
]
},
facet_counts: {
facet_queries: { },
facet_fields: {
table_name: [
"product",
434,
"dealer",
135
]
},
facet_dates: { },
facet_ranges: { }
}
}

please help me



--
View this message in context: 
http://lucene.472066.n3.nabble.com/distributed-search-on-tables-tp4197456.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: Example of sorting by custom function

2015-04-03 Thread david.w.smi...@gmail.com
ValueSourceParser — yes.  You’ll find a ton of them in Solr to get ideas
from.

In your example you forgot the “asc” or “desc”.

~ David Smiley
Freelance Apache Lucene/Solr Search Consultant/Developer
http://www.linkedin.com/in/davidwsmiley

On Fri, Apr 3, 2015 at 9:44 AM, Robert Krüger  wrote:

> Hi,
>
> I have been looking around on the web for information on sorting by a
> custom function but the results are inconclusive to me and some of it seems
> so old that I suspect it's outdated. What I want to do is the following:
>
> I have a field "fingerprint" in my schema that contains a binary data (e.g.
> 64 bytes) that can be used to compute a distance/similarity between two
> records.
>
> Now I want to be able to execute a query and sort its result by the
> distance of the matching records from a given reference value, so the query
> would look something like this
>
> q=*:*&sort=my_distance_func(fingerprint, 0xadet54786eguizgig)
>
> where
>
> my_distance_func is my custom function
> fingerprint is the field in my schema (type binary)
> 0xadet54786eguizgig is a reference value (byte array encoded in whatever
> way) to which the distance shall be computed, which differs with each
> query.
>
> Is ValueSourceParser the right way to look here? Is there a source code
> example someone can point me to?
>
> Thanks in advance,
>
> Robert
>


Re: distributed search on tables

2015-04-03 Thread Erick Erickson
You can do what you want either by using two queries or using
grouping/field collapsing.

Best,
Erick

On Fri, Apr 3, 2015 at 8:03 AM, avinash09  wrote:
> Hi,
>
> I have a use case search all the name="*test*" from two tables (product and
> department)
> i need distributed result 5 result from product and 5 from department
>
> but i am getting first all result from which is 434 and then department as
> shared below
>
> http://localhost:8983/solr/test_core/select?q=name:*test*&wt=json&facet=true&facet.field=table_name
>
>
> {
> responseHeader: {
> status: 0,
> QTime: 19
> },
> response: {
> numFound: 569,
> start: 0,
> docs: [
> {
> name: "test 6576",
> table_name: "product"
> },
> {
> name: "test 6578",
> table_name: "product"
> },
> {
> name: "test 65760",
> table_name: "product"
> },
> {
> name: "test 657699",
> table_name: "product"
> },
> {
> name: "test 657699",
> table_name: "product"
> },
> {
> name: "test 657666",
> table_name: "product"
> },
> {
> name: "test 657689",
> table_name: "product"
> },
> {
> name: "test 6576yu",
> table_name: "product"
> },
> {
> name: "test 657607",
> table_name: "product"
> },
> {
> name: "test 657687",
> table_name: "product"
> }
> ]
> },
> facet_counts: {
> facet_queries: { },
> facet_fields: {
> table_name: [
> "product",
> 434,
> "dealer",
> 135
> ]
> },
> facet_dates: { },
> facet_ranges: { }
> }
> }
>
> please help me
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/distributed-search-on-tables-tp4197456.html
> Sent from the Solr - User mailing list archive at Nabble.com.


Measuring QPS

2015-04-03 Thread Davis, Daniel (NIH/NLM) [C]
I wanted to gather QPS for our production Solr instances, but I was surprised 
that the Admin UI did not contain this information.   We are running a mix of 
versions, but mostly 4.10 at this point.   We are not using SolrCloud at 
present; that's part of why I'm checking - I want to validate the size of our 
existing setup and what sort of SolrCloud setup would be needed to centralize 
several of them.

What is the best way to gather QPS information?

What is the best way to add information like this to the Admin UI, if I decide 
to take that step?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



Re: Measuring QPS

2015-04-03 Thread Shawn Heisey
On 4/3/2015 9:37 AM, Davis, Daniel (NIH/NLM) [C] wrote:
> I wanted to gather QPS for our production Solr instances, but I was surprised 
> that the Admin UI did not contain this information.   We are running a mix of 
> versions, but mostly 4.10 at this point.   We are not using SolrCloud at 
> present; that's part of why I'm checking - I want to validate the size of our 
> existing setup and what sort of SolrCloud setup would be needed to centralize 
> several of them.
>
> What is the best way to gather QPS information?
>
> What is the best way to add information like this to the Admin UI, if I 
> decide to take that step?

As of Solr 4.1 (three years ago), request rate information is available
in the admin UI and via JMX.  In the admin UI, choose a core from the
dropdown, click on Plugins/Stats, then QUERYHANDLER, and open the
handler you wish to examine.  You have avgRequestsPerSecond, which is
calculated for the entire runtime of the SolrCore, as well as
5minRateReqsPerSecond and 15minRateReqsPerSecond, which are far more
useful pieces of information.

https://issues.apache.org/jira/browse/SOLR-1972

Thanks,
Shawn



Re: Unable to update config file using zkcli or RELOAD

2015-04-03 Thread shamik
Thanks Shawn for the pointer, really appreciate it.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Unable-to-update-config-file-using-zkcli-or-RELOAD-tp4197376p4197494.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: DOcValues

2015-04-03 Thread Toke Eskildsen
William Bell  wrote:

[docValues activation?]

> Also, does it help with "*Too many values for UnInvertedField faceting" ?*

Yes. There is an internal limit using UnInverted (aka fc without docValues) of 
16M somewhere - I am not sure exactly what it takes to trigger it, but many 
unique values and/or references will do it at some point.

docValues scales quite a bit higher; we have successfully used it with 7 
billion references to 640 million unique values in a single shard (where it 
worked surprisingly well BTW).

As far as I can see, there is an internal limit of 2 billion unique values per 
shard for docValues. I would like to see that go away, but that's just part of 
an ongoing mission to get Solr to break free from the old "2 billion should be 
enough for everyone"-design.

- Toke Eskildsen

Re: sort param could not be parsed as a query, and is not a field that exists in the index: geodist()

2015-04-03 Thread Niraj
I am not able to see the logs, seems like don't have admin privilege for the
same.

Below is my schema details:
* 
   *


Below are the version specifications:
"responseHeader":{
"status":0,
"QTime":447},
  "mode":"solrcloud",
*  "lucene":{
"solr-spec-version":"4.4.0-cdh5.3.2",
"solr-impl-version":"4.4.0-cdh5.3.2 exported - jenkins - 2015-02-24
12:57:59",
"lucene-spec-version":"4.4.0-cdh5.3.2",
"lucene-impl-version":"4.4.0-cdh5.3.2 exported - jenkins - 2015-02-24
12:51:43"},*
  "jvm":{
"version":"1.7.0_55 24.55-b03",
"name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
"spec":{
  "vendor":"Oracle Corporation",
  "name":"Java Platform API Specification",
  "version":"1.7"},
"jre":{
  "vendor":"Oracle Corporation",
  "version":"1.7.0_55"},
"vm":{
  "vendor":"Oracle Corporation",
  "name":"Java HotSpot(TM) 64-Bit Server VM",
  "version":"24.55-b03"},
"processors":24,





--
View this message in context: 
http://lucene.472066.n3.nabble.com/sort-param-could-not-be-parsed-as-a-query-and-is-not-a-field-that-exists-in-the-index-geodist-tp4197350p4197500.html
Sent from the Solr - User mailing list archive at Nabble.com.


Config join parse in solrconfig.xml

2015-04-03 Thread Frank li
Hi,

I am starting using join parser with our solr. We have some default fields.
They are defined in solrconfig.xml:

  
   edismax
   explicit
   10
   all_text number party name all_code ent_name
   all_text number^3 name^5 party^3 all_code^2
ent_name^7
   id description market_sector_type parent ult_parent
ent_name title patent_title *_ls *_lms *_is *_texts *_ac *_as *_s *_ss *_ds
*_sms *_ss *_bs
   AND
 


I found out once I use join parser, it does not recognize the default
fields any more. How do I modify the configuration for this?

Thanks,

Fred


Re: DOcValues

2015-04-03 Thread Shalin Shekhar Mangar
The UnInvertedField method of faceting is no longer used in Solr.

See https://issues.apache.org/jira/browse/SOLR-7190

On Fri, Apr 3, 2015 at 10:33 AM, Toke Eskildsen 
wrote:

> William Bell  wrote:
>
> [docValues activation?]
>
> > Also, does it help with "*Too many values for UnInvertedField faceting"
> ?*
>
> Yes. There is an internal limit using UnInverted (aka fc without
> docValues) of 16M somewhere - I am not sure exactly what it takes to
> trigger it, but many unique values and/or references will do it at some
> point.
>
> docValues scales quite a bit higher; we have successfully used it with 7
> billion references to 640 million unique values in a single shard (where it
> worked surprisingly well BTW).
>
> As far as I can see, there is an internal limit of 2 billion unique values
> per shard for docValues. I would like to see that go away, but that's just
> part of an ongoing mission to get Solr to break free from the old "2
> billion should be enough for everyone"-design.
>
> - Toke Eskildsen




-- 
Regards,
Shalin Shekhar Mangar.


Re: Solr -indexing from csv file having 28 cols taking lot of time ..plz help i m new to solr

2015-04-03 Thread Toke Eskildsen
avinash09  wrote:
> regex="^(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),
> (.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)$"

A better solution seems to have been presented, but for the record I would like 
to note that the regexp above is quite an effective performance bomb: For each 
group, the evaluation time roughly doubles. Not a problem for 10 groups, but 
you have 28.

I made a little test and matching a single sample line with 20 groups took 120 
ms/match, 24 groups took 2 seconds and 28 groups took 30 seconds on my machine. 
If you had 50 groups, a single match would take 4 years.

The explanation is that Java regexps are greedy: Every one of your groups 
starts by matching to the end of the line, then a comma is reached in the 
regexp and it backtracks. The solution is fortunately both simple and 
applicable to many other regexps: Make your matches terminate as soon as 
possible.

In this case, instead of having groups with (.*), use ([^,]*) instead, which 
means that each group matches everything, except commas. The combined regexp 
then looks like this:
regex="^([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),...([^,]*)$"

The match speed for 28 groups with that regexp was about 0.002ms (average over 
1000 matches).

- Toke Eskildsen


Re: DOcValues

2015-04-03 Thread Shalin Shekhar Mangar
Sorry I should have been more clear. The UnInvertedField method of faceting
is not used in Solr since Solr 5.0.

On Fri, Apr 3, 2015 at 12:17 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> The UnInvertedField method of faceting is no longer used in Solr.
>
> See https://issues.apache.org/jira/browse/SOLR-7190
>
> On Fri, Apr 3, 2015 at 10:33 AM, Toke Eskildsen 
> wrote:
>
>> William Bell  wrote:
>>
>> [docValues activation?]
>>
>> > Also, does it help with "*Too many values for UnInvertedField faceting"
>> ?*
>>
>> Yes. There is an internal limit using UnInverted (aka fc without
>> docValues) of 16M somewhere - I am not sure exactly what it takes to
>> trigger it, but many unique values and/or references will do it at some
>> point.
>>
>> docValues scales quite a bit higher; we have successfully used it with 7
>> billion references to 640 million unique values in a single shard (where it
>> worked surprisingly well BTW).
>>
>> As far as I can see, there is an internal limit of 2 billion unique
>> values per shard for docValues. I would like to see that go away, but
>> that's just part of an ongoing mission to get Solr to break free from the
>> old "2 billion should be enough for everyone"-design.
>>
>> - Toke Eskildsen
>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
Regards,
Shalin Shekhar Mangar.


Re: DOcValues

2015-04-03 Thread Toke Eskildsen
Shalin Shekhar Mangar  wrote:
> The UnInvertedField method of faceting is no longer used in Solr.

True. Un-inversion still takes place for non-DV-fields though (see 
UnivertingReader, which seems to lead to FieldCacheImpl.SortedDocValuesCache). 
But the wrapping is far nicer as everything looks like DocValues now and it 
seems (guessing quite a bit here) that the old 16M-limitation is gone.

- Toke Eskildsen


Re: Config join parse in solrconfig.xml

2015-04-03 Thread Erick Erickson
You have to show us several more things:

1> what exactly does the query look like?
2> what do you expect?
3> output when you specify &debug=query
4> anything else that would help. You might review:

http://wiki.apache.org/solr/UsingMailingLists

Best,
Erick

On Fri, Apr 3, 2015 at 10:58 AM, Frank li  wrote:
> Hi,
>
> I am starting using join parser with our solr. We have some default fields.
> They are defined in solrconfig.xml:
>
>   
>edismax
>explicit
>10
>all_text number party name all_code ent_name
>all_text number^3 name^5 party^3 all_code^2
> ent_name^7
>id description market_sector_type parent ult_parent
> ent_name title patent_title *_ls *_lms *_is *_texts *_ac *_as *_s *_ss *_ds
> *_sms *_ss *_bs
>AND
>  
>
>
> I found out once I use join parser, it does not recognize the default
> fields any more. How do I modify the configuration for this?
>
> Thanks,
>
> Fred


Re: DOcValues

2015-04-03 Thread Shalin Shekhar Mangar
On Fri, Apr 3, 2015 at 12:52 PM, Toke Eskildsen 
wrote:

> Shalin Shekhar Mangar  wrote:
> > The UnInvertedField method of faceting is no longer used in Solr.
>
> True. Un-inversion still takes place for non-DV-fields though (see
> UnivertingReader, which seems to lead to
> FieldCacheImpl.SortedDocValuesCache). But the wrapping is far nicer as
> everything looks like DocValues now and it seems (guessing quite a bit
> here) that the old 16M-limitation is gone.
>
>
Yes, you are right. I didn't mean to imply that fields aren't un-inverted
at all.


> - Toke Eskildsen
>



-- 
Regards,
Shalin Shekhar Mangar.


Re: solr query latency spike when replicating index

2015-04-03 Thread Timothy Ehlers
Replication uses allot of disk io. What kind of hardware is this do you
have the specs?

Can you provide sar -d and sar -b outputs.

What is your GC throughput like?
On Apr 2, 2015 3:55 PM, "wei"  wrote:

> I noticed the solr query latency spike on slave node when replicating index
> from master. Especially when master just finished optimization, the slave
> node will copy the whole index, and the latency is really bad.
>
> Is there some way to fix it?
>
> Thanks,
> Wei
>


Re: Solr 5.0.0 and HDFS

2015-04-03 Thread Joseph Obernberger
Hi - I've been able to replicate the error, and I'm also getting the 
following error that might be related:
null:org.apache.solr.common.SolrException: CLUSTERSTATUS the collection 
time out:180s
at 
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:630)
at 
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:582)
at 
org.apache.solr.handler.admin.CollectionsHandler.handleClusterStatus(CollectionsHandler.java:932)
at 
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:256)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
at 
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:736)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:261)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

at org.eclipse.jetty.server.Server.handle(Server.java:368)
at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at 
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at 
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at 
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at 
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at 
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

at java.lang.Thread.run(Thread.java:745)


Is there anything I can do to debug this better?
Thank you!

-Joe

On 3/31/2015 3:13 PM, Joseph Obernberger wrote:
I've tried to replicate the issue starting from new, but so far it 
hasn't happened again.


-Joe

On 3/28/2015 2:10 PM, Mark Miller wrote:

Hmm...can you file a JIRA issue with this info?

- Mark

On Fri, Mar 27, 2015 at 6:09 PM Joseph Obernberger 


wrote:

I just started up a two shard cluster on two machines using HDFS. 
When I

started to index documents, the log shows errors like this. They repeat
when I execute searches.  All seems well - searches and indexing appear
to be working.
Possibly a configuration issue?
My HDFS config:

  true
  160
  true
  16384
  true
  false
  true
  name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64
  name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512
  name="solr.hdfs.home">hdfs://nameservice1:8020/solr5
  name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1
str>
  
Thank you!

-Joe


java.lang.IllegalStateException: file:
BlockDirectory(HdfsDirectory@799d5a0e
lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@49838b82) 
appears

both in delegate and in cache: cache=[_25.fnm, _2d.si, _2e.nvd, _2b.si,
_28.tvx, _2c.tvx, _1t.si, _27.nvd, _2b.tvd, _2d_Lucene50_0.pos, 
_23.nvd,

_28_Lucene50_0.doc, _28_Lucene50_0.dvd, _2d.fdt, _2c_Lucene50_0.pos,
_23.fdx, _2b_Lucene50_0.doc, _2d.nvm, _28.nvd, _23.fnm,
_2b_Lucene50_0.tim, _2e.fdt, _2d_Lucene50_0.doc, _2b_Lucene50_0.dvd,
_2d_Lucene50_0.dvd, _2b.nvd, _2g.tvx, _28_Lucene50_0.dvm,
_1v_Lucene50_0.tip, _2e_Lucene50_0.dvm, _2e_Lucene50_0.pos, _2g.fdx,
_2e.nvm, _2f.fdx, _1s.tvd, _23.nvm, _27.nvm, _1s_Lu

Re: solr query latency spike when replicating index

2015-04-03 Thread Shalin Shekhar Mangar
In Solr 5.0 you can throttle the replication and limit the bandwidth it
uses. The Sematext guys wrote a nice blog post about it. See
http://blog.sematext.com/2015/01/26/solr-5-replication-throttling/

On Thu, Apr 2, 2015 at 1:53 PM, wei  wrote:

> I noticed the solr query latency spike on slave node when replicating index
> from master. Especially when master just finished optimization, the slave
> node will copy the whole index, and the latency is really bad.
>
> Is there some way to fix it?
>
> Thanks,
> Wei
>



-- 
Regards,
Shalin Shekhar Mangar.