RE: Limit on # of collections -SolrCloud

[Tim Potter]

Hi Chris,

Thanks for the link to Patrick's github (looks like some good stuff in there).

One thing to try (and this isn't the final word on this, but is helpful) is to 
go into the tree view in the Cloud panel and find out which node is hosting the 
Overseer (/overseer_elect/leader). When restarting your cluster, make sure you 
restart this node last. We've seen instances where you end up restarting the 
overseer node each time as you restart the cluster, which causes all kinds of 
craziness. I'll bet you'll see better results by doing this, but let us know 
either way.

Also, at 600 cores per machine, I had to reduce the JVM's thread stack size to 
-Xss256k as there are an extreme number of threads allocated when starting up 
that many cores.

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Chris W 
Sent: Thursday, March 20, 2014 4:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Limit on # of collections -SolrCloud

The replication factor is two. I have equally sharded all collections
across all nodes. We have a 6 node cluster setup. 300* 6 shards and 2
replicas per shard. I have almost 600 cores per machine

Also one fact is that my zk timeout is in the order of 2-3 minutes. I see
zk responses very slow and a lot of outstanding requests (found that out
thanks to https://github.com/phunt/)




On Thu, Mar 20, 2014 at 2:53 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:

> Hours sounds too long indeed.  We recently had a client with several
> thousand collections, but restart wasn't taking hours...
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Mar 20, 2014 5:49 PM, "Erick Erickson"  wrote:
>
> > How many total replicas are we talking here?
> > As in how many shards and, for each shard,
> > how many replicas? I'm not asking for a long list
> > here, just if you have a bazillion replicas in aggregate.
> >
> > Hours is surprising.
> >
> > Best,
> > Erick
> >
> > On Thu, Mar 20, 2014 at 2:17 PM, Chris W 
> wrote:
> > > Thanks, Shalin. Making clusterstate.json on a collection basis sounds
> > > awesome.
> > >
> > >  I am not having problems with #2 . #3 is a major time hog in my
> > > environment. I have over 300 +collections and restarting the entire
> > cluster
> > > takes in the order of hours.  (2-3 hour). Can you explain more about
> the
> > > leaderVoteWait setting?
> > >
> > >
> > >
> > >
> > > On Thu, Mar 20, 2014 at 1:28 PM, Shalin Shekhar Mangar <
> > > shalinman...@gmail.com> wrote:
> > >
> > >> There are no arbitrary limits on the number of collections but yes
> > >> there are practical limits. For example, the cluster state can become
> > >> a bottleneck. There is a lot of work happening on finding and
> > >> addressing these problems. See
> > >> https://issues.apache.org/jira/browse/SOLR-5381
> > >>
> > >> Boot up time is because of:
> > >> 1) Core discovery, schema/config parsing etc
> > >> 2) Transaction log replay on startup
> > >> 3) Wait time for enough replicas to become available before leader
> > >> election happens
> > >>
> > >> You can't do much about 1 right now I think. For #2, you can keep your
> > >> transaction logs smaller by a hard commit before shutdown. For #3
> > >> there is a leaderVoteWait settings but I'd rather not touch that
> > >> unless it becomes a problem.
> > >>
> > >> On Fri, Mar 21, 2014 at 1:39 AM, Chris W 
> > wrote:
> > >> > Hi there
> > >> >
> > >> >  Is there a limit on the # of collections solrcloud can support? Can
> > >> > zk/solrcloud handle 1000s of collections?
> > >> >
> > >> > Also i see that the bootup time of solrcloud increases with increase
> > in #
> > >> > of cores. I do not have any expensive warm up queries. How do i
> > speedup
> > >> > solr startup?
> > >> >
> > >> > --
> > >> > Best
> > >> > --
> > >> > C
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Shalin Shekhar Mangar.
> > >>
> > >
> > >
> > >
> > > --
> > > Best
> > > --
> > > C
> >
>



--
Best
--
C

RE: Questions about commits and OOE

[Tim Potter]

Hi Metin,

I think removing the softCommit=true parameter on the client side will 
definitely help as NRT wasn't designed to re-open searchers after every 
document. Try every 1 second (or even every few seconds), I doubt your users 
will notice. To get an idea of what threads are running in your JVM process, 
you can use jstack.

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: OSMAN Metin 
Sent: Wednesday, December 04, 2013 7:36 AM
To: solr-user@lucene.apache.org
Subject: Questions about commits and OOE

Hi all,

let me first explain our situation :

We have


-   two virtual servers with each :

4x SolR 4.4.0 on Tomcat 6 (+ with mod_cluster 1.2.0), each JVM has -Xms2048m 
-Xmx2048m -XX:MaxPermSize=384m
1x Zookeeper 3.4.5 (Only one of the two Zookeeper is active.)
CentOS 6.4
Sun JDK 1.6.0-31
16 GB of RAM
4 vCPU


-   only one core and one shard

-   ~25 docs and 50-100 MB of index size

-   two load balancers (apache + mod_cluster) who are both connected to the 
8 SolR nodes

-   1 VIP pointing to these two LB

The commit configuration is

-   every update request do a soft commit (i.e. param softCommit=true in 
the http request)

-   autosoftcommit disabled

-   autocommit enabled every 15 seconds

The client application is a java app with SolRj client using the previous VIP 
as an endpoint.
We need NearRealTime modifications visible by the end users.
During the day, the client uses SolR with about 80% of select requests and 20% 
of update requests.
Every morning, the client is sending a massive bunch of updates (about 1 in 
a few minutes).

During this massive update, we have sometimes a peak of active threads 
exceeding the limit of 8192 process authorized for the user running the tomcat 
and zookeeper process.
When this happens, every hardCommit is failing with an "OutOfMemory : unable to 
create native thread" message.


Now, I have some questions :

-   Why are there some many threads created ? Is the softCommit on every 
update that opens a new thread ?

-   Once an OOE occurs, every hardcommit will be broken, even if the number 
of threads opened on the system is low. Is there any way to "free" the JVM ? 
The only solution we have found is to restart all the JVM.

-   When the OOE occurs, the SolR cloud console shows the leader node as 
active and the others as recovering

o   is the replication working at that moment ?

o   as all the hardcommits are failing but the softcommits not, am I very sure 
that I will not lose some updates when restarting all the nodes ?

By the way, we are planning to

-   disable the softCommit parameter on the client side and to enable the 
autosoftcommit instead.

-   create another server and make 3 zookeeper chorum instead of a unique 
zookeeper master.

-   skip the use of load balancers and let zookeeper decide which node will 
respond to the requests

Any help would be appreciated !

Metin OSMAN

RE: Setting routerField/shardKey on specific collection?

[Tim Potter]

Hi Daniel,

I'm not sure how this would apply to an existing collection (in your case 
collection1). Try using the collections API to create a new collection and pass 
the router.field parameter. Grep'ing over the code, the parameter is named: 
router.field (not routerField or routeField).

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Daniel Bryant 
Sent: Wednesday, December 04, 2013 9:40 AM
To: solr-user@lucene.apache.org
Subject: Setting routerField/shardKey on specific collection?

Hi,

I'm using Solr 4.6 and trying to specify a router.field (shard key) on a
specific collection so that all documents with the same value in the
specified field end up in the same collection.

However, I can't find an example of how to do this via the solr.xml? I
see in this ticket https://issues.apache.org/jira/browse/SOLR-5017 there
is a mention of a routeField property.

Should the solr.xml contain the following?


 


Any help would be greatly appreciate? I've been yak shaving all
afternoon reading various Jira tickets and wikis trying to get this to
work :-)

Best wishes,

Daniel


--
*Daniel Bryant  |  Software Development Consultant  | www.tai-dev.co.uk
*
daniel.bry...@tai-dev.co.uk   |  +44
(0) 7799406399  |  Twitter: @taidevcouk 

RE: starting up solr automatically

[Tim Potter]

Apologies for chiming in late on this one ... just wanted to mention what I've 
used with good success in the past is supervisord (http://supervisord.org/). 
It's easy to install and configure and has the benefit of restarting nodes if 
they crash (such as due to an OOM). I'll also mention that you should consider 
configuring the OOM killer for your JVM when using SolrCloud as an OOM'd 
process is like zombie in your cluster, causing all kinds of malice.

-XX:OnOutOfMemoryError="/home/solr/oom_killer.sh $x %p"

But, whether you use that or not, definitely take a look at supervisord if 
you're on Linux as it has been a great way to run SolrCloud in a good sized 
cluster for me.

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Greg Walters 
Sent: Thursday, December 05, 2013 1:23 PM
To: solr-user@lucene.apache.org
Subject: Re: starting up solr automatically

Eric,

Sorry about that, the entire OPTIONS= part can be dropped. That's there to 
support a war file that we deploy next to solr.

Greg

On Dec 5, 2013, at 1:51 PM, Eric Palmer  wrote:

> some progress but getting this error now
> sudo service jetty start
> Starting Jetty: -bash: line 1: cd: /var/lib/answers/atlascloud/solr45: No
> such file or directory
> STARTED Jetty Thu Dec  5 19:50:09 UTC 2013
> [ec2-user@ip-10-50-203-92 ~]$ java.lang.IllegalArgumentException: No such
> OPTIONS: jsp
> at org.eclipse.jetty.start.Config.getCombinedClasspath(Config.java:411)
> at org.eclipse.jetty.start.Config.getActiveClasspath(Config.java:388)
> at org.eclipse.jetty.start.Main.start(Main.java:509)
> at org.eclipse.jetty.start.Main.main(Main.java:96)
>
>
> On Thu, Dec 5, 2013 at 2:28 PM, Greg Walters wrote:
>
>> Eric,
>>
>> If you're using the script from the gist I posted make sure you're
>> sourcing the jetty file at line 140.
>>
>> Thanks,
>> Greg
>>
>> On Dec 5, 2013, at 1:21 PM, Eric Palmer  wrote:
>>
>>> Greg or anyone that can help, when I try to start jetty as a service
>>> sudo service jetty start
>>>
>>> I get this error
>>> ** ERROR: JETTY_HOME not set, you need to set it or install in a standard
>>> location
>>>
>>> same for
>>> sudo service jetty stop
>>> sudo service jetty check
>>> etc
>>>
>>> I have a file here and the permissions look right
>>> ls -al /etc/default/
>>> total 20
>>> drwxr-xr-x  2 root root 4096 Dec  5 19:18 .
>>> drwxr-xr-x 68 root root 4096 Dec  5 19:03 ..
>>> -rwxr-xr-x  1 root root  317 Dec  5 19:18 jetty
>>>
>>> the contents if the jetty file is
>>> JAVA_HOME=/usr/lib/jvm/jre
>>> JETTY_HOME=/home/ec2-user/solr/solr-4.5.1/example/
>>> JETTY_USER=ec2-user
>>> JETTY_LOGS=/home/ec2-user/solr/solr-4.5.1/example/logs
>>> JAVA_OPTIONS="\
>>> -Dsolr.solr.home=/home/ec2-user/solr/solr-4.5.1/example/solr/ \
>>> -Xms1g \
>>> -Djetty.port=8983 \
>>> -Dcollection.configName=collection1 \
>>> $JAVA_OPTIONS"
>>>
>>> Any ideas what I should check?
>>>
>>> Eric P
>>>
>>> thanks in advance
>>>
>>>
>>>
>>> On Thu, Dec 5, 2013 at 11:28 AM, Greg Walters >> wrote:
>>>
 Alan,

 Yes, that's intentional. There's two reasons for this:

 1: We make schema changes frequently (more frequently than I like)
 2: So far as I've noticed, it doesn't hurt anything and covers my butt
 when I've got to clear out all the solr related data from ZK while
>> testing

 Thanks,
 Greg

 On Dec 5, 2013, at 5:53 AM, Alan Woodward  wrote:

> Hi Greg,
>
> It looks as though your script below will bootstrap a collection
 configuration every time Solr is restarted, which probably isn't what
>> you
 want to do?  You only need to upload the config once.
>
> Alan Woodward
> www.flax.co.uk
>
>
> On 4 Dec 2013, at 21:26, Greg Walters wrote:
>
>> I almost forgot, you'll need a file to setup the environment a bit
>> too:
>>
>> **
>> JAVA_HOME=/usr/java/default
>> JAVA_OPTIONS="-Xmx15g \
>> -Xms15g \
>> -XX:+PrintGCApplicationStoppedTime \
>> -XX:+PrintGCDateStamps \
>> -XX:+PrintGCDetails \
>> -XX:+UseConcMarkSweepGC \
>> -XX:+UseParNewGC \
>> -XX:+UseTLAB \
>> -XX:+CMSParallelRemarkEnabled \
>> -XX:+CMSScavengeBeforeRemark \
>> -XX:+UseCMSInitiatingOccupancyOnly \
>> -XX:CMSInitiatingOccupancyFraction=50 \
>> -XX:CMSWaitDuration=30 \
>> -XX:GCTimeRatio=40 \
>> -Xloggc:/tmp/solr45_gc.log \
>> -Dbootstrap_conf=true \
>>

>> -Dbootstrap_confdir=/var/lib/answers/atlascloud/solr45/solr/wa-en-collection_1/conf/
 \
>> -Dcollection.configName=wa-en-collection \
>> -DzkHost= \
>> -DnumShards= \
>> -Dsolr.solr.home=/var/lib/answers/atlascloud/solr45/solr/ \
>>

>> -Dlog4j.configuration=file:///var/lib/answers/atlascloud/solr45/resources/log4j.properties
 \
>> -Djetty.port=9101 \
>> $JAVA_OPTIONS"
>> JETTY_HOME=/var/lib/answers/atlascloud/solr45/
>> JETTY_USER=tomcat
>> JETTY_LOGS=/

RE: Cloud graph gone after manually editing clusterstate.json

[Tim Potter]

Hi Michael,

Can you /get clusterstate.json again to see the contents? Also, maybe just a 
typo but you have `cate clusterstate.json` vs. `cat ..`

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: michael.boom 
Sent: Wednesday, December 11, 2013 6:37 AM
To: solr-user@lucene.apache.org
Subject: Cloud graph gone after manually editing clusterstate.json

HI,

Today I changed my ZK config, removing one instance in the quorum and then
restarted both all ZKs and all Solr instances.
After this operation i noticed that one of the shards in one collection was
missing the range ("range":null). Router for that collection was
compositeId.

So, I proceeded adding the missing range manually by editing
clusterstate.json
$ zkCli.sh -server zk1:9983 get /clusterstate.json > clusterstate.json
i did my edits, and then:
$ zkCli.sh -server zk1:9983 set /clusterstate.json "`cate
clusterstate.json`"

Everything fine, I check in the Admin - the clusterstate.json was updated,
but now when i try to see the graph view or radial graph i can't see
anything. Just white space.

Any idea why?
Thanks!





-
Thanks,
Michael
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Cloud-graph-gone-after-manually-editing-clusterstate-json-tp4106142.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Cloud graph gone after manually editing clusterstate.json

[Tim Potter]

I'm not sure at this point as what you're describing seems fine to me ... I'm 
not too familiar with Solr's UI implementation, but I suspect the cloud graph 
stuff may be client side, so are you seeing any JavaScript errors in the dev 
console in your browser?

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: michael.boom 
Sent: Wednesday, December 11, 2013 8:21 AM
To: solr-user@lucene.apache.org
Subject: RE: Cloud graph gone after manually editing clusterstate.json

Thanks for the reply Tim,

Yes, that was just a typo, i used "cat" not "cate".
As for the checks everything looks fine, my edits were:
1. updating the shard range
2. removed the header which looked log information, as below:
* removed header start here*
Connecting to solr3:9983
2013-12-11 16:15:05,372 [myid:] - INFO  [main:Environment@100] - Client
environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2013-12-11 16:15:05,376 [myid:] - INFO  [main:Environment@100] - Client
environment:host.name=solr3.internal
2013-12-11 16:15:05,377 [myid:] - INFO  [main:Environment@100] - Client
environment:java.version=1.7.0_25
2013-12-11 16:15:05,377 [myid:] - INFO  [main:Environment@100] - Client
environment:java.vendor=Oracle Corporation
2013-12-11 16:15:05,378 [myid:] - INFO  [main:Environment@100] - Client
environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2013-12-11 16:15:05,378 [myid:] - INFO  [main:Environment@100] - Client
environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/z$
2013-12-11 16:15:05,378 [myid:] - INFO  [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
2013-12-11 16:15:05,379 [myid:] - INFO  [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp
2013-12-11 16:15:05,379 [myid:] - INFO  [main:Environment@100] - Client
environment:java.compiler=
2013-12-11 16:15:05,380 [myid:] - INFO  [main:Environment@100] - Client
environment:os.name=Linux
2013-12-11 16:15:05,380 [myid:] - INFO  [main:Environment@100] - Client
environment:os.arch=amd64
2013-12-11 16:15:05,381 [myid:] - INFO  [main:Environment@100] - Client
environment:os.version=3.2.0-4-amd64
2013-12-11 16:15:05,381 [myid:] - INFO  [main:Environment@100] - Client
environment:user.name=solr
2013-12-11 16:15:05,382 [myid:] - INFO  [main:Environment@100] - Client
environment:user.home=/home/solr
2013-12-11 16:15:05,382 [myid:] - INFO  [main:Environment@100] - Client
environment:user.dir=/opt/zookeeper
2013-12-11 16:15:05,384 [myid:] - INFO  [main:ZooKeeper@438] - Initiating
client connection, connectString=solr3:9983 sessionTimeout=3
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@58a5f543
2013-12-11 16:15:05,412 [myid:] - INFO
[main-SendThread(solr3.productdb.internal:9983):ClientCnxn$SendThread@966] -
Opening socket connection to server solr3.internal/10.33.182.78:9983. Will
not attempt to authenticate $
2013-12-11 16:15:05,419 [myid:] - INFO
[main-SendThread(solr3.productdb.internal:9983):ClientCnxn$SendThread@849] -
Socket connection established to solr3.internal/10.33.182.78:9983,
initiating session
2013-12-11 16:15:05,427 [myid:] - INFO
[main-SendThread(solr3.productdb.internal:9983):ClientCnxn$SendThread@1207]
- Session establishment complete on server solr3.internal/10.33.182.78:9983,
sessionid = 0x142e187355000$

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
*<< i removed the above until here*
{
  "offers_collection_GB":{
"shards":{
  "shard1":{
"range":"8000-bfff",
"state":"active",
"replicas":{
.. and so on


Could this be the problem?



-
Thanks,
Michael
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Cloud-graph-gone-after-manually-editing-clusterstate-json-tp4106142p4106161.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Load existing HDFS files into solr?

[Tim Potter]

Hi Chen,

I'm not aware of any direct integration between the two at this time. You might 
ping the Hive user list with this question too. That said, I've been thinking 
whether it makes sense to build a Hive StorageHandler for Solr? That at least 
seems like a quick way to go. However, it might also be possible to just plug a 
Hive InputFormat into Mark's MapReduce/Solr stuff?

See: https://github.com/markrmiller/solr-map-reduce-example 

Cheers,

Timothy Potter
www.lucidworks.com


From: cynosure 
Sent: Thursday, December 12, 2013 12:11 AM
To: solr-user@lucene.apache.org
Subject: Load existing HDFS files into solr?

Folks,
Our current data is stored in hive tables. Is there a way to specify solr
to index the existing hdfs files directly? or I have to import each hive
table to solr?
Can any one point to me some reference?
Thank you very much!
Chen

RE: How can you move a shard from one SolrCloud node to another?

[Tim Potter]

Hi Chris,

The easiest approach is to just create a new core on the new machine that 
references the collection and shard you want to migrate. For example, say you 
split shard1 of a collection named "cloud", which results in having: shard1_0 
and shard1_1. Now let's say you want to migrate shard 1_0 over to the new 
machine. 

First, fire off a q=*:*&distrib=false query to the shard you're migrating so 
that you know how many docs it has (which will be used to verify the migration 
was clean below).

Next, bring up the new machine in cloud mode (-zkHost=?) and then go to the 
admin console on that server. Nav to the core admin page and create a new core, 
specifying the collection and shard1_0 in the form; note: the form leads you to 
believe you need to create the directory on the local system but you actually 
don't need to worry about doing that as the config will get pulled from ZK and 
the directory will get created on the fly (at least that's what happened in my 
env using branch_4x). 

When the new core initializes, it will use good ol' snapshot replication to 
pull the index from the leader. Verify the new core is happy by executing the 
q=*:*&distrib=false  query again. Once you're satisfied, you can unload the 
core you migrated.

Btw ... you can do all this with the core admin API instead of the Web UI if 
you want to script it.

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: cwhi 
Sent: Sunday, December 15, 2013 3:43 PM
To: solr-user@lucene.apache.org
Subject: How can you move a shard from one SolrCloud node to another?

Let's say I want to rebalance a SolrCloud collection.  I call SPLITSHARD to
split an existing shard, and then I'd like to move one of the subshards to a
new machine so the index is more balanced.  Can this be done?  If not, how
do you rebalance an existing SolrCloud collection?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-can-you-move-a-shard-from-one-SolrCloud-node-to-another-tp4106815.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: How can you move a shard from one SolrCloud node to another?

[Tim Potter]

Hi Yago,

When you create a new core (via API or Web UI), you specify the collection name 
and shard id, in my example "cloud" and "shard1_0". When the core initializes 
in SolrCloud mode, it recognizes that the collection exists and adds itself as 
a replica to the shard. Then the main replica recovery process kicks in; try 
PeerSync, realize too far out of date, try snapshot replication from leader. 
The following core API command led to the same result as using the UI:

curl -v 
"http://localhost:8986/solr/admin/cores?action=CREATE&collection=cloud&shard=shard1_0&name=cloud_shard1_0_replica3";

The only trick here is you need to set the name of the core, which from what I 
can tell can be arbitrary but I chose to use the same naming standard as the 
other cores

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Yago Riveiro 
Sent: Monday, December 16, 2013 9:32 AM
To: solr-user@lucene.apache.org
Subject: Re: How can you move a shard from one SolrCloud node to another?

Tim,

Can you explain how the replication snapshot is done using the coreAdminAPI?

--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Monday, December 16, 2013 at 4:23 PM, Tim Potter wrote:

> Hi Chris,
>
> The easiest approach is to just create a new core on the new machine that 
> references the collection and shard you want to migrate. For example, say you 
> split shard1 of a collection named "cloud", which results in having: shard1_0 
> and shard1_1. Now let's say you want to migrate shard 1_0 over to the new 
> machine.
>
> First, fire off a q=*:*&distrib=false query to the shard you're migrating so 
> that you know how many docs it has (which will be used to verify the 
> migration was clean below).
>
> Next, bring up the new machine in cloud mode (-zkHost=?) and then go to the 
> admin console on that server. Nav to the core admin page and create a new 
> core, specifying the collection and shard1_0 in the form; note: the form 
> leads you to believe you need to create the directory on the local system but 
> you actually don't need to worry about doing that as the config will get 
> pulled from ZK and the directory will get created on the fly (at least that's 
> what happened in my env using branch_4x).
>
> When the new core initializes, it will use good ol' snapshot replication to 
> pull the index from the leader. Verify the new core is happy by executing the 
> q=*:*&distrib=false query again. Once you're satisfied, you can unload the 
> core you migrated.
>
> Btw ... you can do all this with the core admin API instead of the Web UI if 
> you want to script it.
>
> Cheers,
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com (http://www.lucidworks.com)
>
> 
> From: cwhi mailto:chris.whi...@gmail.com)>
> Sent: Sunday, December 15, 2013 3:43 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: How can you move a shard from one SolrCloud node to another?
>
> Let's say I want to rebalance a SolrCloud collection. I call SPLITSHARD to
> split an existing shard, and then I'd like to move one of the subshards to a
> new machine so the index is more balanced. Can this be done? If not, how
> do you rebalance an existing SolrCloud collection?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/How-can-you-move-a-shard-from-one-SolrCloud-node-to-another-tp4106815.html
> Sent from the Solr - User mailing list archive at Nabble.com 
> (http://Nabble.com).
>
>

RE: SolrCloud Suggester java ClassNotFoundException: org.apache.solr.suggest.tst.TSTLookup

[Tim Potter]

There have been some recent refactorings in this area of the code. The 
following class name should work:

org.apache.solr.spelling.suggest.tst.TSTLookupFactory

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Trevor Handley 
Sent: Monday, December 16, 2013 11:27 AM
To: solr-user@lucene.apache.org
Subject: RE: SolrCloud Suggester java ClassNotFoundException: 
org.apache.solr.suggest.tst.TSTLookup

Also, I'm aware that there's two typos in my schema.xml attached. I forgot to 
remove the linebreak \ character from the two splitOnCaseChange sections.
This typo does not exist in the official schema.xml that solr is using.

-Original Message-
From: Trevor Handley [mailto:hand...@civicplus.com]
Sent: Monday, December 16, 2013 12:24 PM
To: solr-user@lucene.apache.org
Subject: SolrCloud Suggester java ClassNotFoundException: 
org.apache.solr.suggest.tst.TSTLookup

Hello, I'm working with SolrCloud and trying to integrate the Suggester 
functionality http://wiki.apache.org/solr/Suggester.

I've configured the requestHandler and searchComponent sections of 
solrconfig.xml, and added a new fieldtype and field to schema.xml. These 
documents are attached to this message.

Background: This is a change I'm trying to make to a currently working/stable 
version of Solr 4.6 that has nearly 1.5 million unique documents in the index. 
The whole architecture is a single SolrCloud collection with 2 core shards that 
are replicated for a total of 4 cores. The shard1_slice1 core and shard2_slice2 
core exist on one physical server, and shard1_slice2 core and shard2_slice1 
core exist on a separate physical server.

When I try to restart solr with suggester enabled then I get java error 
"java.lang.ClassNotFoundException: org.apache.solr.suggest.tst.TSTLookup"

I tried a few different suggester classes but they all fail to load with the 
same message here.
I verified that my .jar files do contain that class and are in the correct lib 
directory using a script that searches .jar files for a class name:

[solr@Searchnode-001 ~]$ ./findclass.sh /opt/solrcloud/lib/ TSTLookup 
/opt/solrcloud/lib/solr-core-4.6.0.jar:org/apache/solr/spelling/suggest/tst/TSTLookupFactory.class
/opt/solrcloud/lib/lucene-suggest-4.6-SNAPSHOT.jar:org/apache/lucene/search/suggest/tst/TSTLookup.class

And here's a listing of the jar files in my lib directory:
activation-1.1.jar
AlchemyAPIAnnotator-2.3.1.jar
apache-mime4j-core-0.7.2.jar
apache-mime4j-dom-0.7.2.jar
attributes-binder-1.2.0.jar
bcmail-jdk15-1.45.jar
bcprov-jdk15-1.45.jar
boilerpipe-1.1.0.jar
carrot2-mini-3.8.0.jar
commons-beanutils-1.7.0.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-digester-2.0.jar
dom4j-1.6.1.jar
fontbox-1.8.1.jar
hppc-0.5.2.jar
icu4j-49.1.jar
isoparser-1.0-RC-1.jar
jackson-core-asl-1.7.4.jar
jackson-mapper-asl-1.7.4.jar
jdom-1.0.jar
jempbox-1.8.1.jar
jetty-continuation-8.1.10.v20130312.jar
jetty-deploy-8.1.10.v20130312.jar
jetty-http-8.1.10.v20130312.jar
jetty-io-8.1.10.v20130312.jar
jetty-jmx-8.1.10.v20130312.jar
jetty-security-8.1.10.v20130312.jar
jetty-server-8.1.10.v20130312.jar
jetty-servlet-8.1.10.v20130312.jar
jetty-util-8.1.10.v20130312.jar
jetty-webapp-8.1.10.v20130312.jar
jetty-xml-8.1.10.v20130312.jar
jsonic-1.2.7.jar
juniversalchardet-1.0.3.jar
langdetect-1.1-20120112.jar
lucene-analyzers-common-4.6-SNAPSHOT.jar
lucene-analyzers-kuromoji-4.6-SNAPSHOT.jar
lucene-analyzers-phonetic-4.6-SNAPSHOT.jar
lucene-codecs-4.6-SNAPSHOT.jar
lucene-core-4.6-SNAPSHOT.jar
lucene-grouping-4.6-SNAPSHOT.jar
lucene-highlighter-4.6-SNAPSHOT.jar
lucene-join-4.6-SNAPSHOT.jar
lucene-memory-4.6-SNAPSHOT.jar
lucene-misc-4.6-SNAPSHOT.jar
lucene-queries-4.6-SNAPSHOT.jar
lucene-queryparser-4.6-SNAPSHOT.jar
lucene-spatial-4.6-SNAPSHOT.jar
lucene-suggest-4.6-SNAPSHOT.jar
mahout-collections-1.0.jar
mahout-math-0.6.jar
mail-1.4.1.jar
metadata-extractor-2.6.2.jar
morfologik-fsa-1.7.1.jar
morfologik-polish-1.7.1.jar
morfologik-stemming-1.7.1.jar
netcdf-4.2-min.jar
OpenCalaisAnnotator-2.3.1.jar
pdfbox-1.8.1.jar
poi-3.9.jar
poi-ooxml-3.9.jar
poi-ooxml-schemas-3.9.jar
poi-scratchpad-3.9.jar
rome-0.9.jar
servlet-api-3.0.jar
simple-xml-2.7.jar
solr-analysis-extras-4.6.0.jar
solr-cell-4.6.0.jar
solr-clustering-4.6.0.jar
solr-core-4.6.0.jar
solr-dataimporthandler-4.6.0.jar
solr-dataimporthandler-extras-4.6.0.jar
solr-langid-4.6.0.jar
solr-solrj-4.6.0.jar
solr-test-framework-4.6.0.jar
solr-uima-4.6.0.jar
solr-velocity-4.6.0.jar
Tagger-2.3.1.jar
tagsoup-1.2.1.jar
tika-core-1.4.jar
tika-parsers-1.4.jar
uimaj-core-2.3.1.jar
velocity-1.7.jar
velocity-tools-2.0.jar
vorbis-java-core-0.1.jar
vorbis-java-tika-0.1.jar
WhitespaceTokenizer-2.3.1.jar
xercesImpl-2.9.1.jar
xmlbeans-2.3.0.jar
xz-1.0.jar

This is how I start solr with jetty:
java -Dbootstrap_confdir=/opt/solrcloud/zkBootstrapConfigs/ 
-Dcollection.configName=CP_Search 
-DzkHost=zookeeper-001:2181,zookeeper-002:2181,zookeeper-003:2181 
-Dcom.sun.manage

RE: SolrCloud Suggester java ClassNotFoundException: org.apache.solr.suggest.tst.TSTLookup

[Tim Potter]

Awesome ... I'll update the Wiki to reflect the new class names.

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Trevor Handley 
Sent: Monday, December 16, 2013 11:44 AM
To: solr-user@lucene.apache.org
Subject: RE: SolrCloud Suggester java ClassNotFoundException: 
org.apache.solr.suggest.tst.TSTLookup

Brilliant, thanks Timothy!

Changing the solrconfig.xml lookupImpl (not className) to the 
org.apache.solr.spelling.suggest.tst.TSTLookupFactory fixed this issue for me.

Thanks, Trevor

-Original Message-
From: Tim Potter [mailto:tim.pot...@lucidworks.com]
Sent: Monday, December 16, 2013 12:32 PM
To: solr-user@lucene.apache.org
Subject: RE: SolrCloud Suggester java ClassNotFoundException: 
org.apache.solr.suggest.tst.TSTLookup

There have been some recent refactorings in this area of the code. The 
following class name should work:

org.apache.solr.spelling.suggest.tst.TSTLookupFactory

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Trevor Handley 
Sent: Monday, December 16, 2013 11:27 AM
To: solr-user@lucene.apache.org
Subject: RE: SolrCloud Suggester java ClassNotFoundException: 
org.apache.solr.suggest.tst.TSTLookup

Also, I'm aware that there's two typos in my schema.xml attached. I forgot to 
remove the linebreak \ character from the two splitOnCaseChange sections.
This typo does not exist in the official schema.xml that solr is using.

-Original Message-
From: Trevor Handley [mailto:hand...@civicplus.com]
Sent: Monday, December 16, 2013 12:24 PM
To: solr-user@lucene.apache.org
Subject: SolrCloud Suggester java ClassNotFoundException: 
org.apache.solr.suggest.tst.TSTLookup

Hello, I'm working with SolrCloud and trying to integrate the Suggester 
functionality http://wiki.apache.org/solr/Suggester.

I've configured the requestHandler and searchComponent sections of 
solrconfig.xml, and added a new fieldtype and field to schema.xml. These 
documents are attached to this message.

Background: This is a change I'm trying to make to a currently working/stable 
version of Solr 4.6 that has nearly 1.5 million unique documents in the index. 
The whole architecture is a single SolrCloud collection with 2 core shards that 
are replicated for a total of 4 cores. The shard1_slice1 core and shard2_slice2 
core exist on one physical server, and shard1_slice2 core and shard2_slice1 
core exist on a separate physical server.

When I try to restart solr with suggester enabled then I get java error 
"java.lang.ClassNotFoundException: org.apache.solr.suggest.tst.TSTLookup"

I tried a few different suggester classes but they all fail to load with the 
same message here.
I verified that my .jar files do contain that class and are in the correct lib 
directory using a script that searches .jar files for a class name:

[solr@Searchnode-001 ~]$ ./findclass.sh /opt/solrcloud/lib/ TSTLookup 
/opt/solrcloud/lib/solr-core-4.6.0.jar:org/apache/solr/spelling/suggest/tst/TSTLookupFactory.class
/opt/solrcloud/lib/lucene-suggest-4.6-SNAPSHOT.jar:org/apache/lucene/search/suggest/tst/TSTLookup.class

And here's a listing of the jar files in my lib directory:
activation-1.1.jar
AlchemyAPIAnnotator-2.3.1.jar
apache-mime4j-core-0.7.2.jar
apache-mime4j-dom-0.7.2.jar
attributes-binder-1.2.0.jar
bcmail-jdk15-1.45.jar
bcprov-jdk15-1.45.jar
boilerpipe-1.1.0.jar
carrot2-mini-3.8.0.jar
commons-beanutils-1.7.0.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-digester-2.0.jar
dom4j-1.6.1.jar
fontbox-1.8.1.jar
hppc-0.5.2.jar
icu4j-49.1.jar
isoparser-1.0-RC-1.jar
jackson-core-asl-1.7.4.jar
jackson-mapper-asl-1.7.4.jar
jdom-1.0.jar
jempbox-1.8.1.jar
jetty-continuation-8.1.10.v20130312.jar
jetty-deploy-8.1.10.v20130312.jar
jetty-http-8.1.10.v20130312.jar
jetty-io-8.1.10.v20130312.jar
jetty-jmx-8.1.10.v20130312.jar
jetty-security-8.1.10.v20130312.jar
jetty-server-8.1.10.v20130312.jar
jetty-servlet-8.1.10.v20130312.jar
jetty-util-8.1.10.v20130312.jar
jetty-webapp-8.1.10.v20130312.jar
jetty-xml-8.1.10.v20130312.jar
jsonic-1.2.7.jar
juniversalchardet-1.0.3.jar
langdetect-1.1-20120112.jar
lucene-analyzers-common-4.6-SNAPSHOT.jar
lucene-analyzers-kuromoji-4.6-SNAPSHOT.jar
lucene-analyzers-phonetic-4.6-SNAPSHOT.jar
lucene-codecs-4.6-SNAPSHOT.jar
lucene-core-4.6-SNAPSHOT.jar
lucene-grouping-4.6-SNAPSHOT.jar
lucene-highlighter-4.6-SNAPSHOT.jar
lucene-join-4.6-SNAPSHOT.jar
lucene-memory-4.6-SNAPSHOT.jar
lucene-misc-4.6-SNAPSHOT.jar
lucene-queries-4.6-SNAPSHOT.jar
lucene-queryparser-4.6-SNAPSHOT.jar
lucene-spatial-4.6-SNAPSHOT.jar
lucene-suggest-4.6-SNAPSHOT.jar
mahout-collections-1.0.jar
mahout-math-0.6.jar
mail-1.4.1.jar
metadata-extractor-2.6.2.jar
morfologik-fsa-1.7.1.jar
morfologik-polish-1.7.1.jar
morfologik-stemming-1.7.1.jar
netcdf-4.2-min.jar
OpenCalaisAnnotator-2.3.1.jar
pdfbox-1.8.1.jar
poi-3.9.jar
poi-ooxml-3.9.jar
poi-ooxm

RE: solr cloud - deleting and adding the same doc

[Tim Potter]

Yes, SolrCloud uses a transaction log to keep track of ordered updates to a 
document. The latest update will be immediately visible from the real-time get 
handler /get?id=X even without a commit.

Cheers,
Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: adfel70 
Sent: Tuesday, December 17, 2013 7:54 AM
To: solr-user@lucene.apache.org
Subject: solr cloud - deleting and adding the same doc

Hi
in SolrCloud, if I send 2 different requests to solr - one with delete
action of doc with id X and another with add action of doc with the same id
- is it guaranteed that the delete action will occur before the add action?

Is it guaranteed that after all actions are done, the index will have doc X
with its most updated state?

thanks.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-cloud-deleting-and-adding-the-same-doc-tp4107111.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Solr failure results in misreplication?

[Tim Potter]

Any chance you still have the logs from the servers hosting 1 & 2? I would open 
a JIRA ticket for this one as it sounds like something went terribly wrong on 
restart. 

You can update the /clusterstate.json to fix this situation.

Lastly, it's recommended to use an OOM killer script with SolrCloud so that you 
don't end up with zombie nodes hanging around in your cluster. I use something 
like: -XX:OnOutOfMemoryError="$SCRIPT_DIR/oom_solr.sh $x %p"

$x in start script is the port # and %p is the process ID ... My oom_solr.sh 
script is something like this:

#!/bin/bash
SOLR_PORT=$1
SOLR_PID=$2
NOW=$(date +"%F%T")
(
echo "Running OOM killer script for process $SOLR_PID for Solr on port 
89$SOLR_PORT"
kill -9 $SOLR_PID
echo "Killed process $SOLR_PID"
) | tee oom_killer-89$SOLR_PORT-$NOW.log

I use supervisord do handle the restart after the process gets killed by the 
OOM killer, which is why you don't see the restart in this script ;-)

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: youknow...@heroicefforts.net 
Sent: Tuesday, December 17, 2013 10:31 PM
To: solr-user@lucene.apache.org
Subject: Solr failure results in misreplication?

My client has a test cluster Solr 4.6 with three instances 1, 2, and 3 hosting 
shards 1, 2, and 3, respectively.  There is no replication in this cluster.  We 
started receiving OOME during indexing; likely the batches were too large.  The 
cluster was rebooted to restore the system.  However, upon reboot, instance 2 
now shows as a replica of shard 1 and its shard2 is down with a null range.  
Instance 2 is queryable shards.tolerant=true&distribute=false and returns a 
different set of records than instance 1 (as would be expected during normal 
operations).  Clusterstate.json is similar to the following:

mycollection:{
shard1:{
range:800-d554,
state:active,
replicas:{
instance1state:active...,
instance2state:active...
}
},
shard3:{state:active.},
shard2:{
range:null,
state:active,
replicas:{
instance2{state:down}
}
},
maxShardsPerNode:1,
replicationFactor:1
}

Any ideas on how this would come to pass?  Would manually correcting the 
clusterstate.json in Zk correct this situation?

RE: monitoring solr logs

[Tim Potter]

I'm using logstash4solr (http://logstash4solr.org) for something similar ...

I setup my Solr to use Log4J by passing the following on the command-line when 
starting Solr: 
-Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties

Then I use a custom Log4J appender that writes to RabbitMQ: 

https://github.com/plant42/rabbitmq-log4j-appender

You can then configure a RabbitMQ input for logstash - 
http://logstash.net/docs/1.3.2/inputs/rabbitmq

This decouples the log writes from log indexing in logstash4solr, which scales 
better for active Solr installations.

Btw ... I just log everything from Solr using this approach but you can use 
standard Log4J configuration settings to limit which classes / log levels to 
send to the RabbitMQ appender.

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: adfel70 
Sent: Monday, December 30, 2013 8:15 AM
To: solr-user@lucene.apache.org
Subject: monitoring solr logs

hi
i'm trying to figure out which solr and zookeeper logs i should monitor and
collect.
All the logs will be written to a file but I want to collect some of them
with logstash in order to be able to analyze them efficiently.
any inputs on logs of which classes i should collect?

thanks.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: monitoring solr logs

[Tim Potter]

We're (LucidWorks) are actively developing on logstash4solr so if you have 
issues, let us know. So far, so good for me but I upgraded to logstash 1.3.2 
even though the logstash4solr version includes 1.2.2 you can use the newer one. 
I'm not quite in production with my logstash4solr <- rabbit-mq <- log4j <- Solr 
solution yet though ;-)

Yeah, 50GB is too much logging for only 150K docs. Maybe start by filtering by 
log level (WARN and more severe). If a server crashes, you're likely to see 
some errors in the logstash side but sometimes you may have to SSH to the 
specific box and look at the local log (so definitely append all messages to 
the local Solr log too), I'm using something like the following for local 
logging:

log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=50MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.File=logs/solr.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3} %x - 
%m%n


Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: adfel70 
Sent: Monday, December 30, 2013 9:34 AM
To: solr-user@lucene.apache.org
Subject: RE: monitoring solr logs

Actually I was considering using logstash4solr, but it didn't seem mature
enough.
does it work fine? any known bugs?

are you collecting the logs in the same solr cluster you use for the
production systems?
if so, what will you do if for some reason solr is down and you would like
to analyze the logs to see what happend?

btw, i started a new solr cluster with 7 shards, replicationfactor=3 and run
indexing job of 400K docs,
it got stuck on 150K because I used Socketappender directly to write to
logstash and logstash disk got full.

that's why I moved to using AsyncAppender, and I plan on moving to using
rabbit.
but this is also why I wanted to filter some of the logs. indexing 150K docs
prodcued 50GB of logs.
this seemed too much.




Tim Potter wrote
> I'm using logstash4solr (http://logstash4solr.org) for something similar
> ...
>
> I setup my Solr to use Log4J by passing the following on the command-line
> when starting Solr:
> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
>
> Then I use a custom Log4J appender that writes to RabbitMQ:
>
> https://github.com/plant42/rabbitmq-log4j-appender
>
> You can then configure a RabbitMQ input for logstash -
> http://logstash.net/docs/1.3.2/inputs/rabbitmq
>
> This decouples the log writes from log indexing in logstash4solr, which
> scales better for active Solr installations.
>
> Btw ... I just log everything from Solr using this approach but you can
> use standard Log4J configuration settings to limit which classes / log
> levels to send to the RabbitMQ appender.
>
> Cheers,
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com
>
> 
> From: adfel70 <

> adfel70@

> >
> Sent: Monday, December 30, 2013 8:15 AM
> To:

> solr-user@.apache

> Subject: monitoring solr logs
>
> hi
> i'm trying to figure out which solr and zookeeper logs i should monitor
> and
> collect.
> All the logs will be written to a file but I want to collect some of them
> with logstash in order to be able to analyze them efficiently.
> any inputs on logs of which classes i should collect?
>
> thanks.
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
> Sent from the Solr - User mailing list archive at Nabble.com.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: monitoring solr logs

[Tim Potter]

I've just been using the Solr query form so far :P but have plans to try out 
Kibana too. Let me know how that goes for you and I'll do the same.


From: adfel70 
Sent: Monday, December 30, 2013 10:06 AM
To: solr-user@lucene.apache.org
Subject: RE: monitoring solr logs

And are you using any tool like kibana as a dashboard for the logs?



Tim Potter wrote
> We're (LucidWorks) are actively developing on logstash4solr so if you have
> issues, let us know. So far, so good for me but I upgraded to logstash
> 1.3.2 even though the logstash4solr version includes 1.2.2 you can use the
> newer one. I'm not quite in production with my logstash4solr <- rabbit-mq
> <- log4j <- Solr solution yet though ;-)
>
> Yeah, 50GB is too much logging for only 150K docs. Maybe start by
> filtering by log level (WARN and more severe). If a server crashes, you're
> likely to see some errors in the logstash side but sometimes you may have
> to SSH to the specific box and look at the local log (so definitely append
> all messages to the local Solr log too), I'm using something like the
> following for local logging:
>
> log4j.rootLogger=INFO, file
> log4j.appender.file=org.apache.log4j.RollingFileAppender
> log4j.appender.file.MaxFileSize=50MB
> log4j.appender.file.MaxBackupIndex=10
> log4j.appender.file.File=logs/solr.log
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3}
> %x - %m%n
>
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com
>
> 
> From: adfel70 <

> adfel70@

> >
> Sent: Monday, December 30, 2013 9:34 AM
> To:

> solr-user@.apache

> Subject: RE: monitoring solr logs
>
> Actually I was considering using logstash4solr, but it didn't seem mature
> enough.
> does it work fine? any known bugs?
>
> are you collecting the logs in the same solr cluster you use for the
> production systems?
> if so, what will you do if for some reason solr is down and you would like
> to analyze the logs to see what happend?
>
> btw, i started a new solr cluster with 7 shards, replicationfactor=3 and
> run
> indexing job of 400K docs,
> it got stuck on 150K because I used Socketappender directly to write to
> logstash and logstash disk got full.
>
> that's why I moved to using AsyncAppender, and I plan on moving to using
> rabbit.
> but this is also why I wanted to filter some of the logs. indexing 150K
> docs
> prodcued 50GB of logs.
> this seemed too much.
>
>
>
>
> Tim Potter wrote
>> I'm using logstash4solr (http://logstash4solr.org) for something similar
>> ...
>>
>> I setup my Solr to use Log4J by passing the following on the command-line
>> when starting Solr:
>> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
>>
>> Then I use a custom Log4J appender that writes to RabbitMQ:
>>
>> https://github.com/plant42/rabbitmq-log4j-appender
>>
>> You can then configure a RabbitMQ input for logstash -
>> http://logstash.net/docs/1.3.2/inputs/rabbitmq
>>
>> This decouples the log writes from log indexing in logstash4solr, which
>> scales better for active Solr installations.
>>
>> Btw ... I just log everything from Solr using this approach but you can
>> use standard Log4J configuration settings to limit which classes / log
>> levels to send to the RabbitMQ appender.
>>
>> Cheers,
>>
>> Timothy Potter
>> Sr. Software Engineer, LucidWorks
>> www.lucidworks.com
>>
>> 
>> From: adfel70 <
>
>> adfel70@
>
>> >
>> Sent: Monday, December 30, 2013 8:15 AM
>> To:
>
>> solr-user@.apache
>
>> Subject: monitoring solr logs
>>
>> hi
>> i'm trying to figure out which solr and zookeeper logs i should monitor
>> and
>> collect.
>> All the logs will be written to a file but I want to collect some of them
>> with logstash in order to be able to analyze them efficiently.
>> any inputs on logs of which classes i should collect?
>>
>> thanks.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
> Sent from the Solr - User mailing list archive at Nabble.com.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108744.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: side logging requests

[Tim Potter]

You can wire-in a custom UpdateRequestProcessor - 
http://wiki.apache.org/solr/UpdateRequestProcessor

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: elmerfudd 
Sent: Monday, December 30, 2013 10:26 AM
To: solr-user@lucene.apache.org
Subject: side logging requests

Hi all,
currently there are 2 things I want to accomplish.
I want that on demend Every doc (xml) that is sent to be indexed in solr
will be copied to a big log file (I want to control when to activate this
feature and when to deactivate)
same as for queries.
Also, I may need to manipulate the data before its written.

Is there any way achieving this without changing solr sourcecode? (So it
won't be affected by updates).

I thought of a possible way,
I posted before about making "transparent" request handler , is it possible
? If so, how?


thankkk you!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/side-logging-requests-tp4108752.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Solr Cloud Query Scaling

[Tim Potter]

Absolutely adding replicas helps you scale query load. Queries do not need to 
be routed to leaders; they can be handled by any replica in a shard. Leaders 
are only needed for handling update requests.

In general, a distributed query has two phases, driven by a controller node 
(what you called collator below). The controller is the Solr that received the 
query request from the client. In Phase 1, the controller distributes the query 
to one of the replicas for all shards and receives back the list of matching 
document IDs from each replica (only a page worth btw). 

The controller merges the results and sorts them to generate a final page of 
results to be returned to the client. In Phase 2, the controller collects all 
the fields from the documents to generate the final result set by querying the 
replicas involved in Phase 1.

The controller uses SolrJ's LBSolrServer to query the shards in Phase 1 so you 
get some basic load-balancing amongst replicas for a shard. I've not done any 
research to see how balanced that selection process is in production but I 
suspect if you have 3 replicas in a shard, then roughly 1/3 of the queries go 
to each.

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Sir Gilligan 
Sent: Thursday, January 09, 2014 11:02 AM
To: solr-user@lucene.apache.org
Subject: Solr Cloud Query Scaling

Question: Does adding replicas help with query load?

Scenario: 3 Physical Machines. 3 Shards
Query any machine, get results. Standard Solr Cloud stuff.

Update Scenario: 6 Physical Machines. 3 Shards.
M = Machine, S = Shard, -L = Leader
M1S1-L
M2S2
M3S3
M4S1
M5S2-L
M6S3-L

Incoming Query to M2S2. How will Solr Cloud (4.6.0) distribute the query?
Will M2S2 handle the query for shard 2? Or, will it send it to the
leader of S2 which is M5S2?
When the query is distributed, will it send it to the other leaders? OR,
will it send it to any shard?
Specifically:
Query sent to M2S2. Solr Cloud distributes the query. Could it possibly
send the query on to M3S3 and M4S1? Some kind of query load balance
functionality (maybe like a round robin to the shard members).
OR will M2S2 just be the collator, and send the query to the leaders?
OR something different that I have not described?

If queries do not have to be processed by leaders then we could add
three more physical machines (now total 9 machines) and handle more
query load.

Thank you.

RE: Trying to config solr cloud

[Tim Potter]

Hi Svante,

It seems like the TermVectorComponent is in the search component chain of your 
/select search handler but you haven't indexed docs with term vectors enabled 
(at least from what's in the schema you provided). Admittedly, the NamedList 
code could be a little more paranoid but I think the key is to check the 
component chain of your /select handler to make sure tvComponent isn't included 
(or re-index with term vectors enabled).

Cheers,

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: saka.csi...@gmail.com  on behalf of svante 
karlsson 
Sent: Tuesday, January 21, 2014 4:20 PM
To: solr-user@lucene.apache.org
Subject: Trying to config solr cloud

I've been playing around with solr 4.6.0 for some weeks and I'm trying to
get a solrcloud configuration running.

I've installed two physical machines and I'm trying to set up 4 shards on
each.

I installled a zookeeper on each host as well

I uploaded a config to zookeeper with
/opt/solr-4.6.0/example/cloud-scripts/zkcli.sh -cmd upconfig -zkhost
192.168.0.93:2181 -confdir /opt/solr/om5/conf/ -confname om5

The /opt/solr/om5 was where I kept my normal solr and I'm trying to reuse
that config.


now I start two hosts (one on each server)
java -DzkHost=192.168.0.93:2181,192.168.0.94:2181 -Dhost=192.168.0.93 -jar
start.jar
java -DzkHost=192.168.0.93:2181,192.168.0.94:2181 -Dhost=192.168.0.94 -jar
start.jar

and finally I'll run
curl '
http://192.168.0.93:8983/solr/admin/collections?action=CREATE&name=om5&numShards=8&replicationFactor=1&maxShardsPerNode=4
'

This gets me 8 shard in the web gui
http://192.168.0.94:8983/solr/#/~cloud

Now I add documents to this and that seems to work. I pushed 97 million
docs during the night. ( each shard reports a 8th of the documents )

But all questions returns http 500 in variants of the below result. I get
correct data in the body but always an error trace after that...

http://192.168.0.93:8983/solr/om5/select?q=*:*&rows=1&fl=id

returns



500
32



b1e5865c-3b01---0471b12d16ac




java.lang.NullPointerException at
org.apache.solr.common.util.NamedList.nameValueMapToList(NamedList.java:114)
at org.apache.solr.common.util.NamedList.(NamedList.java:80) at
org.apache.solr.handler.component.TermVectorComponent.finishStage(TermVectorComponent.java:453)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:317)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368) at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)

500



So I must be doing something wrong

RE: Solr Permgen Exceptions when creating/removing cores

[Tim Potter]

Hi Josh,

Try adding: -XX:+CMSPermGenSweepingEnabled as I think for some VM versions, 
permgen collection was disabled by default.

Also, I use: -XX:MaxPermSize=512m -XX:PermSize=256m with Solr, so 64M may be 
too small.


Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Josh 
Sent: Wednesday, February 26, 2014 12:27 PM
To: solr-user@lucene.apache.org
Subject: Solr Permgen Exceptions when creating/removing cores

We are using the Bitnami version of Solr 4.6.0-1 on a 64bit windows
installation with 64bit Java 1.7U51 and we are seeing consistent issues
with PermGen exceptions. We have the permgen configured to be 512MB.
Bitnami ships with a 32bit version of Java for windows and we are replacing
it with a 64bit version.

Passed in Java Options:

-XX:MaxPermSize=64M
-Xms3072M
-Xmx6144M
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+CMSClassUnloadingEnabled
-XX:NewRatio=3

-XX:MaxTenuringThreshold=8

This is our use case:

We have what we call a database core which remains fairly static and
contains the imported contents of a table from SQL server. We then have
user cores which contain the record ids of results from a text search
outside of Solr. We then query for the data we want from the database core
and limit the results to the content of the user core. This allows us to
combine facet data from Solr with the search results from another engine.
We are creating the user cores on demand and removing them when the user
logs out.

Our issue is the constant creation and removal of user cores combined with
the constant importing seems to push us over our PermGen limit. The user
cores are removed at the end of every session and as a test I made an
application that would loop creating the user core, import a set of data to
it, query the database core using it as a limiter and then remove the user
core. My expectation was in this scenario that all the permgen associated
with that user cores would be freed upon it's unload and allow permgen to
reclaim that memory during a garbage collection. This was not the case, it
would constantly go up until the application would exhaust the memory.

I also investigated whether the there was a connection between the two
cores left behind because I was joining them together in a query but even
unloading the database core after unloading all the user cores won't
prevent the limit from being hit or any memory to be garbage collected from
Solr.

Is this a known issue with creating and unloading a large number of cores?
Could it be configuration based for the core? Is there something other than
unloading that needs to happen to free the references?

Thanks

Notes: I've tried using tools to determine if it's a leak within Solr such
as Plumbr and my activities turned up nothing.

RE: Replicating Between Solr Clouds

[Tim Potter]

Unfortunately, there is no out-of-the-box solution for this at the moment. 

In the past, I solved this using a couple of different approaches, which 
weren't all that elegant but served the purpose and were simple enough to allow 
the ops folks to setup monitors and alerts if things didn't work.

1) use DIH's Solr entity processor to pull data from one Solr to another, see: 
http://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor

This only works if you store all fields, which in my use case was OK because I 
also did lots of partial document updates, which also required me to store all 
fields

2) use the replication handler's snapshot support to create snapshots on a 
regular basis and then move the files over the network

This one works but required the use of read and write aliases and two 
collections on the remote (slave) data center so that I could rebuild my write 
collection from the snapshots and then update the aliases to point the reads at 
the updated collection. Work on an automated backup/restore solution is 
planned, see https://issues.apache.org/jira/browse/SOLR-5750, but if you need 
something sooner, you can write a backup driver using SolrJ that uses 
CloudSolrServer to get the address of all the shard leaders, initiate the 
backup command on each leader, poll the replication details handler for 
snapshot completion on each shard, and then ship the files across the network. 
Obviously, this isn't a solution for NRT multi-homing ;-)

Lastly, these aren't the only ways to go about this, just wanted to share some 
high-level details about what has worked.

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: perdurabo 
Sent: Tuesday, March 04, 2014 1:04 PM
To: solr-user@lucene.apache.org
Subject: Replicating Between Solr Clouds

We are looking to setup a highly available failover site across a WAN for our
SolrCloud instance.  The main production instance is at colo center A and
consists of a 3-node ZooKeeper ensemble managing configs for a 4-node
SolrCloud running Solr 4.6.1.  We only have one collection among the 4 cores
and there are two shards in the collection, one master node and one replica
node for each shard.  Our search and indexing services address the Solr
cloud through a load balancer VIP, not a compound API call.

Anyway, the Solr wiki explains fairly well how to replicate single node Solr
collections, but I do not see an obvious way for replicating a SolrCloud's
indices over a WAN to another SolrCloud.  I need for a SolrCloud in another
data center to be able to replicate both shards of the collection in the
other data center over a WAN.  It needs to be able to replicate from a load
balancer VIP, not a single named server of the SolrCloud, which round robins
across all four nodes/2 shards for high availability.

I've searched high and low for a white paper or some discussion of how to do
this and haven't found anything.  Any ideas?

Thanks in advance.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Replicating-Between-Solr-Clouds-tp4121196.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: SOLRJ and SOLR compatibility

[Tim Potter]

Just my 2 cents on this while I wait for a build ... I think we have to ensure 
that an older client will work with a newer server or newer client will work 
with older server to support hot rolling upgrades. It's not unheard of these 
days for an org to have 10's (or even 100's) of Solr cloud servers. As Solr is 
a mission-critical technology, sometimes it can't just be taken off-line, so 
most need to upgrade servers one-by-one. This implies that during a hot rolling 
upgrade, there's going to be a mix of Solr server versions talking to each 
other and clients talking to different versions of servers. You can't take it 
out of the LB either since eventually, you'll have no nodes in your LB. 

I think ops folks will accept either solution (old client -> new server or new 
client -> old server), but we as a community need to pick one and build out the 
test suites that ensure SolrJ compatibility with different versions.

Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com


From: Michael Sokolov 
Sent: Tuesday, March 04, 2014 8:37 AM
To: solr-user@lucene.apache.org
Subject: Re: SOLRJ and SOLR compatibility

Does that mean newer clients work with older servers (I think so, from
reading this thread), or the other way round?  If so, I guess the advice
would be --  upgrade all your clients first?

-Mike


On 03/04/2014 10:00 AM, Mark Miller wrote:
> Yeah, sorry :(  the fix applied is only for compatibility in one direction. 
> Older code won’t know what this type 19 is.
>
> - Mark
>
> http://about.me/markrmiller
>
> On Mar 4, 2014, at 2:42 AM, Thomas Scheffler  
> wrote:
>
>> Am 04.03.2014 07:21, schrieb Thomas Scheffler:
>>> Am 27.02.2014 09:15, schrieb Shawn Heisey:
 On 2/27/2014 12:49 AM, Thomas Scheffler wrote:
>> What problems have you seen with mixing 4.6.0 and 4.6.1?  It's possible
>> that I'm completely ignorant here, but I have not heard of any.
> Actually bug reports arrive me that sound like
>
> "Unknown type 19"
 Aha!  I found it!  It was caused by the change applied for SOLR-5658,
 fixed in 4.7.0 (just released) by SOLR-5762.  Just my luck that there's
 a bug bad enough to contradict what I told you.

 https://issues.apache.org/jira/browse/SOLR-5658
 https://issues.apache.org/jira/browse/SOLR-5762

 I've added a comment that will help users find SOLR-5762 with a search
 for "Unknown type 19".

 If you use SolrJ 4.7.0, compatibility should be better.
>>> Hi,
>>>
>>> I am sorry to inform you that SolrJ 4.7.0 face the same issue with SOLR
>>> 4.5.1. I received a client stack trace this morning and still waiting
>>> for a Log-Output from the Server:
>> Here we go for the server side (4.5.1):
>>
>> Mrz 03, 2014 2:39:26 PM org.apache.solr.core.SolrCore execute
>> Information: [clausthal_test] webapp=/solr path=/select
>> params={fl=*,score&sort=mods.dateIssued+desc&q=%2BobjectType:"mods"+%2Bcategory:"clausthal_status\:published"&wt=javabin&version=2&rows=3}
>> hits=186 status=0 QTime=2
>> Mrz 03, 2014 2:39:38 PM
>> org.apache.solr.update.processor.LogUpdateProcessor finish
>> Information: [clausthal_test] webapp=/solr path=/update
>> params={wt=javabin&version=2} {} 0 0
>> Mrz 03, 2014 2:39:38 PM org.apache.solr.common.SolrException log
>> Schwerwiegend: java.lang.RuntimeException: Unknown type 19
>> at
>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:228)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:139)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:131)
>> at
>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:221)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:116)
>> at
>> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:186)
>> at
>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:112)
>> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:158)
>> at
>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:99)
>> at
>> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
>> at
>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>> at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
>> at
>> org.apache.solr.servlet.Sol

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Tim Potter]

Try adding shards.info=true and debug=track to your queries ... these will
give more detailed information about what's going behind the scenes.

On Mon, Oct 13, 2014 at 11:11 PM, S.L  wrote:

> Erick,
>
> I have upgraded to SolrCloud 4.10.1 with the same toplogy , 3 shards and 2
> replication factor with six cores altogether.
>
> Unfortunately , I still see the issue of intermittently no results being
> returned.I am not able to figure out whats going on here, I have included
> the logging information below.
>
> *Here's the query that I run.*
>
>
> http://server1.mydomain.com:8081/solr/dyCollection1/select/?q=*:*&fq=%28id:220a8dce-3b31-4d46-8386-da8405595c47%29&wt=json&distrib=true
>
>
>
> *Scenario 1: No result returned.*
>
> *Log Information for Scenario #1 .*
> 92860314 [http-bio-8081-exec-103] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server1.mydomain.com:8082/solr/dyCollection1_shard3_replica2/|http://server2.mydomain.com:8082/solr/dyCollection1_shard3_replica1/
> null
> 92860315 [http-bio-8081-exec-103] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server3.mydomain.com:8082/solr/dyCollection1_shard1_replica1/|http://server2.mydomain.com:8081/solr/dyCollection1_shard1_replica2/
> null
> 92860315 [http-bio-8081-exec-103] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server1.mydomain.com:8081/solr/dyCollection1_shard2_replica1/|http://server3.mydomain.com:8081/solr/dyCollection1_shard2_replica2/
> null
> 92860315 [http-bio-8081-exec-103] INFO  org.apache.solr.core.SolrCore  –
> [dyCollection1_shard2_replica1] webapp=/solr path=/select/
>
> params={q=*:*&distrib=true&wt=json&fq=(id:220a8dce-3b31-4d46-8386-da8405595c47)}
> hits=0 status=0 QTime=5
>
> *Scenario #2 : I get result back*
>
>
>
> *Log information for scenario #2.*92881911 [http-bio-8081-exec-177] INFO
> org.apache.solr.core.SolrCore  – [dyCollection1_shard2_replica1]
> webapp=/solr path=/select
>
> params={spellcheck=true&spellcheck.maxResultsForSuggest=5&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollations=5&spellcheck.maxCollationTries=10&distrib=false&wt=javabin&spellcheck.collate=true&version=2&rows=10&NOW=1413251927427&shard.url=
>
> http://server1.mydomain.com:8081/solr/dyCollection1_shard2_replica1/|http://server3.mydomain.com:8081/solr/dyCollection1_shard2_replica2/&fl=productURL,score&df=suggestAggregate&start=0&q=*:*&spellcheck.dictionary=direct&spellcheck.dictionary=wordbreak&spellcheck.count=10&isShard=true&fsv=true&fq=(id:220a8dce-3b31-4d46-8386-da8405595c47)&spellcheck.alternativeTermCount=5
> }
> hits=1 status=0 QTime=1
> 92881913 [http-bio-8081-exec-177] INFO  org.apache.solr.core.SolrCore  –
> [dyCollection1_shard2_replica1] webapp=/solr path=/select
>
> params={spellcheck=false&spellcheck.maxResultsForSuggest=5&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&ids=
>
> http://www.searcheddomain.com/p/ironwork-8-piece-comforter-set/-/A-15273248&spellcheck.maxCollations=5&spellcheck.maxCollationTries=10&distrib=false&wt=javabin&spellcheck.collate=true&version=2&rows=10&NOW=1413251927427&shard.url=http://server1.mydomain.com:8081/solr/dyCollection1_shard2_replica1/|http://server3.mydomain.com:8081/solr/dyCollection1_shard2_replica2/&df=suggestAggregate&q=*:*&spellcheck.dictionary=direct&spellcheck.dictionary=wordbreak&spellcheck.count=10&isShard=true&fq=(id:220a8dce-3b31-4d46-8386-da8405595c47)&spellcheck.alternativeTermCount=5
> }
> status=0 QTime=0
> 92881914 [http-bio-8081-exec-169] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server1.mydomain.com:8082/solr/dyCollection1_shard3_replica2/|http://server2.mydomain.com:8082/solr/dyCollection1_shard3_replica1/
> null
> 92881914 [http-bio-8081-exec-169] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server3.mydomain.com:8082/solr/dyCollection1_shard1_replica1/|http://server2.mydomain.com:8081/solr/dyCollection1_shard1_replica2/
> null
> 92881914 [http-bio-8081-exec-169] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server1.mydomain.com:8081/solr/dyCollection1_shard2_replica1/|http://server3.mydomain.com:8081/solr/dyCollection1_shard2_replica2/
> null
> 92881914 [http-bio-8081-exec-169] INFO
> org.apache.solr.handler.component.SpellCheckComponent  –
>
> http://server1.mydomain.com:8081/solr/dyCollection1_shard2_replica1/|http://server3.mydomain.com:8081/solr/dyCollection1_shard2_replica2/
> null
> 92881915 [http-bio-8081-exec-169] INFO  org.apache.solr.core.SolrCore  –
> [dyCollection1_shard2_replica1] webapp=/solr path=/select/
>
> params={q=*:*&distrib=true&wt=json&fq=(id:220a8dce-3b31-4d46-8386-da8405595c47)}
> hits=1 status=0 QTime=7
>
>
> *Autocommit and Soft commit settings.*
>
>  
>${solr.autoSoftCommit.maxTime:-1}
>  
>
>  
>${solr.autoCommit.maxTime:15000}
>
>true
>  
>
>
>
> On Tue, Oct 7, 2014 a

Re: Recovering from Out of Mem

[Tim Potter]

jfyi - the bin/solr script does the following:

-XX:OnOutOfMemoryError="$SOLR_TIP/bin/oom_solr.sh $SOLR_PORT" where
$SOLR_PORT is the port Solr is bound to, e.g. 8983

The oom_solr.sh script looks like:

SOLR_PORT=$1

SOLR_PID=`ps waux | grep start.jar | grep $SOLR_PORT | grep -v grep | awk
'{print $2}' | sort -r`

if [ "$SOLR_PID" == "" ]; then

  echo "Couldn't find Solr process running on port $SOLR_PORT!"

  exit

fi

NOW=$(date +"%F%T")

(

echo "Running OOM killer script for process $SOLR_PID for Solr on port
$SOLR_PORT"

kill -9 $SOLR_PID

echo "Killed process $SOLR_PID"

) | tee solr_oom_killer-$SOLR_PORT-$NOW.log


I usually run Solr behind a supervisor type process (supervisord or
upstart) that will restart it if the process dies.


On Tue, Oct 14, 2014 at 8:09 AM, Markus Jelsma  wrote:

> This will do:
> kill -9 `ps aux | grep -v grep | grep tomcat6 | awk '{print $2}'`
>
> pkill should also work
>
> On Tuesday 14 October 2014 07:02:03 Yago Riveiro wrote:
> > Boogie,
> >
> >
> >
> >
> > Any example for java_error.sh script?
> >
> >
> > —
> > /Yago Riveiro
> >
> > On Tue, Oct 14, 2014 at 2:48 PM, Boogie Shafer <
> boogie.sha...@proquest.com>
> >
> > wrote:
> > > a really simple approach is to have the OOM generate an email
> > > e.g.
> > > 1) create a simple script (call it java_oom.sh) and drop it in your
> tomcat
> > > bin dir echo `date` | mail -s "Java Error: OutOfMemory - $HOSTNAME"
> > > not...@domain.com 2) configure your java options (in setenv.sh or
> > > similar) to trigger heap dump and the email script when OOM occurs #
> > > config error behaviors
> > > CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError
> > > -XX:HeapDumpPath=$TOMCAT_DIR/temp/tomcat-dump.hprof
> > > -XX:OnError=$TOMCAT_DIR/bin/java_error.sh
> > > -XX:OnOutOfMemoryError=$TOMCAT_DIR/bin/java_oom.sh
> > > -XX:ErrorFile=$TOMCAT_DIR/temp/java_error%p.log"
> > > 
> > > From: Mark Miller 
> > > Sent: Tuesday, October 14, 2014 06:30
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Recovering from Out of Mem
> > > Best is to pass the Java cmd line option that kills the process on OOM
> and
> > > setup a supervisor on the process to restart it.  You need a somewhat
> > > recent release for this to work properly though. - Mark
> > >
> > >> On Oct 14, 2014, at 9:06 AM, Salman Akram
> > >>  wrote:
> > >>
> > >> I know there are some suggestions to avoid OOM issue e.g. setting
> > >> appropriate Max Heap size etc. However, what's the best way to recover
> > >> from
> > >> it as it goes into non-responding state? We are using Tomcat on back
> end.
> > >>
> > >> The scenario is that once we face OOM issue it keeps on taking queries
> > >> (doesn't give any error) but they just time out. So even though we
> have a
> > >> fail over system implemented but we don't have a way to distinguish if
> > >> these are real time out queries OR due to OOM.
> > >>
> > >> --
> > >> Regards,
> > >>
> > >> Salman Akram
>
>

Re: Frequent recovery of nodes in SolrCloud

[Tim Potter]

A couple of things to check:

1) How many znodes are under the /overseer/queue (which you can see in the
Cloud Tree panel in the Admin UI)
2) How often are you committing? The general advice is that your indexing
client(s) should not send commits and instead rely on auto-commit settings
in solrconfig.xml. I usually start with a hard auto-commit every 60secs
3) Anything in the logs telling you why a replica thinks it needs to
recover? Specifically, I'd search for ZooKeeper session expiration log
messages (grep expired solr.log)



On Thu, Oct 16, 2014 at 10:01 PM, Sachin Kale  wrote:

> Also, the PingRequestHandler is configured as:
>
> 
> server-enabled.txt
>
>
> On Fri, Oct 17, 2014 at 9:07 AM, Sachin Kale 
> wrote:
>
> > From ZooKeeper side, we have following configuration:
> > tickTime=2000
> > dataDir=/var/lib/zookeeper
> > clientPort=2181
> > initLimit=5
> > syncLimit=2
> > server.1=192.168.70.27:2888:3888
> > server.2=192.168.70.64:2889:3889
> > server.3=192.168.70.26:2889:3889
> >
> > Also, in solr.xml, we have zkClientTimeout set to 3.
> >
> > On Fri, Oct 17, 2014 at 7:27 AM, Erick Erickson  >
> > wrote:
> >
> >> And what is your zookeeper timeout? When it's too short that can lead
> >> to this behavior.
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Oct 16, 2014 at 4:34 PM, "Jürgen Wagner (DVT)"
> >>  wrote:
> >> > Hello,
> >> >   you have one shard and 11 replicas? Hmm...
> >> >
> >> > - Why you have to keep two nodes on some machines?
> >> > - Physical hardware or virtual machines?
> >> > - What is the size of this index?
> >> > - Is this all on a local network or are there links with potential
> >> outages
> >> > or failures in between?
> >> > - What is the query load?
> >> > - Have you had a look at garbage collection?
> >> > - Do you use the internal Zookeeper?
> >> > - How many nodes?
> >> > - Any observers?
> >> > - What kind of load does Zookeeper show?
> >> > - How much RAM do these nodes have available?
> >> > - Do some servers get into swapping?
> >> > - ...
> >> >
> >> > How about some more details in terms of sizing and topology?
> >> >
> >> > Cheers,
> >> > --Jürgen
> >> >
> >> >
> >> > On 16.10.2014 18:41, sachinpkale wrote:
> >> >
> >> > Hi,
> >> >
> >> > Recently we have shifted to SolrCloud (4.10.1) from traditional
> >> Master-Slave
> >> > configuration. We have only one collection and it has only only one
> >> shard.
> >> > Cloud Cluster contains total 12 nodes (on 8 machines. On 4 machiens,
> we
> >> have
> >> > two instances running on each) out of which one is leader.
> >> >
> >> > Whenever I see the cluster status using http://
> :/solr/#/~cloud,
> >> it
> >> > shows at least one (sometimes, it is 2-3) node status as recovering.
> We
> >> are
> >> > using HAProxy load balancer and there also many times, it is showing
> the
> >> > nodes are recovering. This is happening for all nodes in the cluster.
> >> >
> >> > What would be the problem here? How do I check this in logs?
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://lucene.472066.n3.nabble.com/Frequent-recovery-of-nodes-in-SolrCloud-tp4164541.html
> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
> >> > уважением
> >> > i.A. Jürgen Wagner
> >> > Head of Competence Center "Intelligence"
> >> > & Senior Cloud Consultant
> >> >
> >> > Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
> >> > Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864
> >> 1543
> >> > E-Mail: juergen.wag...@devoteam.com, URL: www.devoteam.de
> >> >
> >> > 
> >> > Managing Board: Jürgen Hatzipantelis (CEO)
> >> > Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
> >> > Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
> >> >
> >> >
> >>
> >
> >
>

Re: Recovering from Out of Mem

[Tim Potter]

You'd still want to kill it ... so you'll need to register a cmd script
with the JVM using -XX:OnOutOfMemoryError=kill.cmd and then you could
either

1) trap the PID at startup using something like:

title SolrCloud

for /F "tokens=2 delims= " %%A in ('TASKLIST /FI ^"WINDOWTITLE eq
SolrCloud^" /NH') do (

set /A SOLR_PID=%%A

echo !SOLR_PID!>solr.pid


or


2) if you keep track of the port (which all my Windows scripts do), then
you can do:


For /f "tokens=5" %%j in ('netstat -aon ^| find /i "listening" ^| find
":%SOLR_PORT%"') do (

  taskkill /t /f /pid %%j > nul 2>&1

)


On Fri, Oct 17, 2014 at 1:11 AM, Salman Akram <
salman.ak...@northbaysolutions.net> wrote:

> I know this might sound weird but any easy way to do it in Windows?
>
> On Tue, Oct 14, 2014 at 7:51 PM, Boogie Shafer  >
> wrote:
>
> > yago,
> >
> > you can put more complex restart logic as shown in the examples below or
> > just do something similar to the java_oom.sh i posted earlier where you
> > just spit out an email alert and deal with service restarts and
> > troubleshooting manually
> >
> >
> > e.g. something like the following for a java_error.sh will drop an email
> > with a timestamp
> >
> >
> >
> > echo `date` | mail -s "Java Error: General - $HOSTNAME"
> not...@domain.com
> >
> >
> > 
> > From: Tim Potter 
> > Sent: Tuesday, October 14, 2014 07:35
> > To: solr-user@lucene.apache.org
> > Subject: Re: Recovering from Out of Mem
> >
> > jfyi - the bin/solr script does the following:
> >
> > -XX:OnOutOfMemoryError="$SOLR_TIP/bin/oom_solr.sh $SOLR_PORT" where
> > $SOLR_PORT is the port Solr is bound to, e.g. 8983
> >
> > The oom_solr.sh script looks like:
> >
> > SOLR_PORT=$1
> >
> > SOLR_PID=`ps waux | grep start.jar | grep $SOLR_PORT | grep -v grep | awk
> > '{print $2}' | sort -r`
> >
> > if [ "$SOLR_PID" == "" ]; then
> >
> >   echo "Couldn't find Solr process running on port $SOLR_PORT!"
> >
> >   exit
> >
> > fi
> >
> > NOW=$(date +"%F%T")
> >
> > (
> >
> > echo "Running OOM killer script for process $SOLR_PID for Solr on port
> > $SOLR_PORT"
> >
> > kill -9 $SOLR_PID
> >
> > echo "Killed process $SOLR_PID"
> >
> > ) | tee solr_oom_killer-$SOLR_PORT-$NOW.log
> >
> >
> > I usually run Solr behind a supervisor type process (supervisord or
> > upstart) that will restart it if the process dies.
> >
> >
> > On Tue, Oct 14, 2014 at 8:09 AM, Markus Jelsma 
> > wrote:
> >
> > > This will do:
> > > kill -9 `ps aux | grep -v grep | grep tomcat6 | awk '{print $2}'`
> > >
> > > pkill should also work
> > >
> > > On Tuesday 14 October 2014 07:02:03 Yago Riveiro wrote:
> > > > Boogie,
> > > >
> > > >
> > > >
> > > >
> > > > Any example for java_error.sh script?
> > > >
> > > >
> > > > —
> > > > /Yago Riveiro
> > > >
> > > > On Tue, Oct 14, 2014 at 2:48 PM, Boogie Shafer <
> > > boogie.sha...@proquest.com>
> > > >
> > > > wrote:
> > > > > a really simple approach is to have the OOM generate an email
> > > > > e.g.
> > > > > 1) create a simple script (call it java_oom.sh) and drop it in your
> > > tomcat
> > > > > bin dir echo `date` | mail -s "Java Error: OutOfMemory - $HOSTNAME"
> > > > > not...@domain.com 2) configure your java options (in setenv.sh or
> > > > > similar) to trigger heap dump and the email script when OOM occurs
> #
> > > > > config error behaviors
> > > > > CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError
> > > > > -XX:HeapDumpPath=$TOMCAT_DIR/temp/tomcat-dump.hprof
> > > > > -XX:OnError=$TOMCAT_DIR/bin/java_error.sh
> > > > > -XX:OnOutOfMemoryError=$TOMCAT_DIR/bin/java_oom.sh
> > > > > -XX:ErrorFile=$TOMCAT_DIR/temp/java_error%p.log"
> > > > > 
> > > > > From: Mark Miller 
> > > > > Sent: Tuesday, October 14, 2014 06:30
> > > > > To: solr-user@lucene.apache.org
> > > > > Subject: Re: Recovering from Out of Mem
> > > > > Best is to pass the Java cmd line option that kills the process on
> > OOM
> > > and
> > > > > setup a supervisor on the process to restart it.  You need a
> somewhat
> > > > > recent release for this to work properly though. - Mark
> > > > >
> > > > >> On Oct 14, 2014, at 9:06 AM, Salman Akram
> > > > >>  wrote:
> > > > >>
> > > > >> I know there are some suggestions to avoid OOM issue e.g. setting
> > > > >> appropriate Max Heap size etc. However, what's the best way to
> > recover
> > > > >> from
> > > > >> it as it goes into non-responding state? We are using Tomcat on
> back
> > > end.
> > > > >>
> > > > >> The scenario is that once we face OOM issue it keeps on taking
> > queries
> > > > >> (doesn't give any error) but they just time out. So even though we
> > > have a
> > > > >> fail over system implemented but we don't have a way to
> distinguish
> > if
> > > > >> these are real time out queries OR due to OOM.
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >>
> > > > >> Salman Akram
> > >
> > >
> >
>
>
>
> --
> Regards,
>
> Salman Akram
>