Re: Filter cache pollution during sharded edismax queries

[Anca Kopetz]

Hi Ken,

Have you managed to find out why these entries were stored into filterCache and 
if they have an impact on the hit ratio ?
We noticed the same problem, there are entries of this type : 
item_+(+(title:western^10.0 | ... in our filterCache.

Thanks,
Anca

On 07/02/2013 09:01 PM, Ken Krugler wrote:

Hi all,

After upgrading from Solr 3.5 to 4.2.1, I noticed our filterCache hit ratio had 
dropped significantly.

Previously it was at 95+%, but now it's < 50%.

I enabled recording 100 entries for debugging, and in looking at them it seems 
that edismax (and faceting) is creating entries for me.

This is in a sharded setup, so it's a distributed search.

If I do a search for the string "bogus text" using edismax on two fields, I get 
an entry in each of the shard's filter caches that looks like:

item_+(((field1:bogus | field2:bogu) (field1:text | field2:text))~2):

Is this expected?

I have a similar situation happening during faceted search, even though my 
fields are single-value/untokenized strings, and I'm not using the enum facet 
method.

But I'll get many, many entries in the filterCache for facet values, and they all look like 
"item_::"

The net result of the above is that even with a very big filterCache size of 
2K, the hit ratio is still only 60%.

Thanks for any insights,

-- Ken

--
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr









Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Solution for "MM ignored in edismax queries with operators" ?

[Anca Kopetz]

Hi,

We found a possible solution for 
SOLR-2649<https://issues.apache.org/jira/browse/SOLR-2649> : MM ignored in edismax 
queries with operators. The details are here 
https://issues.apache.org/jira/browse/SOLR-2649?focusedCommentId=13822482&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13822482.

Any feedback is welcome.

Best regards,
Anca Kopetz



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

How to boost documents ?

[Anca Kopetz]

Hi,

How to boost documents that contain all search terms in several of its fields ?

Below you cand find a simplified example : 

The query with Min should match:
q=beautiful Christmas tree&mm=2&qf=title^12 description^2

There are two offers that match the query : 
offer1 {title:"Christmas tree", description:"a joy for children"}

offer2 {title:"Christmas tree", description:"beautiful for holidays"}}

The first offer ranks before the second, despite of the fact that the second one contains all the search terms. I tried to play with the boosts of
qf, but the results vary a lot.

Is there a way to add a boost on all search fields, the same way we do with pf on one field :
pf=title:2^3.0 ?

Thank you,
Anca
-- 






Anca Kopetz Software engineer

E anca.kop...@kelkoo.com  Y!Messenger kelkooancak
T +33 (0)4 56 09 07 55   

A 4/6 Rue des Meridiens 38130 Echirolles











Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.

PeerSync Recovery fails, starting Replication Recovery

[Anca Kopetz]

veryThread] INFO  org.apache.solr.handler.SnapPuller::211  -  No value set for 'pollInterval'. Timer Task not started.
2013-12-05 13:52:01,209 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:369  - Master's generation: 4382
2013-12-05 13:52:01,209 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:370  - Slave's generation: 4382
2013-12-05 13:52:01,209 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:371  - Starting replication process
2013-12-05 13:52:01,213 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:376  - Number of files in latest index in master: 158
2013-12-05 13:52:01,213 [RecoveryThread] INFO  org.apache.solr.core.CachingDirectoryFactory:get:360  - return new directory for /opt/kookel/data/searchSolrNode/solrindex/fr1_green/index.20131205135201213
2013-12-05 13:52:01,213 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:402  - Starting download to

org.apache.lucene.store.MMapDirectory@/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index.20131205135201213 lockFactory=org.apache.lucene.store.NativeFSLockFactory@697d36b7 fullCopy=true
2013-12-05 13:52:21,970 [main-EventThread] INFO  org.apache.solr.common.cloud.ZkStateReader:process:210  - A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 18)
2013-12-05 13:55:15,039 [main-EventThread] INFO  org.apache.solr.common.cloud.ZkStateReader:process:210  - A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 18)
2013-12-05 13:56:27,160 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:406  - Total time taken for download : 265 secs
2013-12-05 13:56:27,540 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:modifyIndexProps:886  - New index installed. Updating index properties... index=index.20131205135201213
2013-12-05 13:56:27,546 [RecoveryThread] INFO  org.apache.solr.handler.SnapPuller:fetchLatestIndex:461  - removing old index directory

org.apache.lucene.store.MMapDirectory@/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index.20131201033712724 lockFactory=org.apache.lucene.store.NativeFSLockFactory@53949838
2013-12-05 13:56:27,765 [RecoveryThread] INFO  org.apache.solr.search.SolrIndexSearcher::197  - Opening Searcher@30a544f3 main
2013-12-05 13:58:52,337 [RecoveryThread] INFO  org.apache.solr.core.CachingDirectoryFactory:closeCacheValue:235  - looking to close /opt/kookel/data/searchSolrNode/solrindex/fr1_green/index.20131201033712724[CachedDir<>]
2013-12-05 13:58:52,337 [RecoveryThread] INFO  org.apache.solr.core.CachingDirectoryFactory:close:304  - Closing directory: /opt/kookel/data/searchSolrNode/solrindex/fr1_green/index.20131201033712724
2013-12-05 13:58:52,337 [RecoveryThread] INFO  org.apache.solr.core.CachingDirectoryFactory:closeCacheValue:279  - Removing directory before core close: /opt/kookel/data/searchSolrNode/solrindex/fr1_green/index.20131201033712724
2013-12-05 13:58:54,172 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy:replay:506  - Replaying buffered documents. core=fr_green
2013-12-05 13:58:54,172 [recoveryExecutor-7-thread-2] WARN  org.apache.solr.update.UpdateLog:doReplay:1240  - Starting log replay tlog{file=/opt/kookel/data/searchSolrNode/solrindex/fr1_green/tlog/tlog.0004091 refcount=2} active=true starting pos=1103833347
2013-12-05 13:59:16,660 [recoveryExecutor-7-thread-2] INFO  org.apache.solr.search.SolrIndexSearcher::197  - Opening Searcher@5e48fb12 main
2013-12-05 14:01:11,908 [main-EventThread] INFO  org.apache.solr.common.cloud.ZkStateReader:process:210  - A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 18)
2013-12-05 14:01:22,846 [recoveryExecutor-7-thread-2] WARN  org.apache.solr.update.UpdateLog:run:1230  - Log replay finished. recoveryInfo=RecoveryInfo{adds=39345 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1103833347}
2013-12-05 14:01:22,846 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy:doRecovery:410  - Replication Recovery was successful - registering as Active. core=fr_green
2013-12-05 14:01:22,846 [RecoveryThread] INFO  org.apache.solr.cloud.ZkController:publish:1017  - publishing core=fr_green state=active
2013-12-05 14:01:22,850 [RecoveryThread] INFO  org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on descriptor - reading it from system property
2013-12-05 14:01:22,860 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy:doRecovery:495  - Finished recovery process. core=fr_green

Best regards,
Anca Kopetz
-- 






Anca Kopetz Software engineer

E anca.kop...@kelkoo.com  Y!Messenger kelkooancak
T +33 (0)4 56 09 07 55   

A 4/6 Rue des Meridie

Re: PeerSync Recovery fails, starting Replication Recovery

[Anca Kopetz]

Hi,

Thank you for the link, it does not seem to be the same problem ...

Best regards,
Anca

On 12/18/2013 11:41 PM, Furkan KAMACI wrote:

Hi Anca;

Could you check the conversation at here:
http://lucene.472066.n3.nabble.com/ColrCloud-IOException-occured-when-talking-to-server-at-td4061831.html

Thanks;
Furkan KAMACI


18 Aralık 2013 Çarşamba tarihinde Anca Kopetz  adlı
kullanıcı şöyle yazdı:

Hi,

In our SolrCloud cluster (2 shards, 8 replicas), the replicas go from

time to time into recovering state, and it takes more than 10 minutes to
finish to recover.

In logs, we see that "PeerSync Recovery" fails with the message :

PeerSync: core=fr_green url=http://solr-08/searchsolrnodefr too many

updates received since start - startingUpdates no longer overlaps with our
currentUpdates

Then "Replication Recovery" starts.

Is there something we can do to avoid the failure of "Peer Recovery" so

that the recovery process is more rapid (less than 10 minutes) ?

The full trace log is here :

2013-12-05 13:51:53,740 [http-8080-46] INFO

org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705
- It has been requested that we recover

2013-12-05 13:51:53,740 [http-8080-112] INFO

org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705
- It has been requested that we recover

2013-12-05 13:51:53,740 [http-8080-112] INFO

org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  -
[admin] webapp=null path=/admin/cores
params={action=REQUESTRECOVERY&core=fr_green&wt=javabin&version=2} status=0
QTime=0

2013-12-05 13:51:53,740 [Thread-1544] INFO

org.apache.solr.cloud.ZkController:publish:1017  - publishing core=fr_green
state=recovering

2013-12-05 13:51:53,741 [http-8080-46] INFO

org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  -
[admin] webapp=null path=/admin/cores
params={action=REQUESTRECOVERY&core=fr_green&wt=javabin&version=2} status=0
QTime=1

2013-12-05 13:51:53,740 [Thread-1543] INFO

org.apache.solr.cloud.ZkController:publish:1017  - publishing core=fr_green
state=recovering

2013-12-05 13:51:53,743 [Thread-1544] INFO

org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on
descriptor - reading it from system property

2013-12-05 13:51:53,746 [Thread-1543] INFO

org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on
descriptor - reading it from system property

2013-12-05 13:51:53,755 [Thread-1543] WARN

org.apache.solr.cloud.RecoveryStrategy:close:105  - Stopping recovery for
zkNodeName=solr-08_searchsolrnodefr_fr_greencore=fr_green

2013-12-05 13:51:53,756 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:run:216  - Starting recovery
process.  core=fr_green recoveringAfterStartup=false

2013-12-05 13:51:53,762 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:doRecovery:495  - Finished recovery
process. core=fr_green

2013-12-05 13:51:53,762 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:run:216  - Starting recovery
process.  core=fr_green recoveringAfterStartup=false

2013-12-05 13:51:53,765 [RecoveryThread] INFO

org.apache.solr.cloud.ZkController:publish:1017  - publishing core=fr_green
state=recovering

2013-12-05 13:51:53,765 [RecoveryThread] INFO

org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on
descriptor - reading it from system property

2013-12-05 13:51:53,767 [RecoveryThread] INFO

org.apache.solr.client.solrj.impl.HttpClientUtil:createClient:103  -
Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false

2013-12-05 13:51:54,777 [main-EventThread] INFO

org.apache.solr.common.cloud.ZkStateReader:process:210  - A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 18)

2013-12-05 13:51:56,804 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:doRecovery:356  - Attempting to
PeerSync from http://solr-02/searchsolrnodefr/fr_green/ core=fr_green -
recoveringAfterStartup=false

2013-12-05 13:51:56,806 [RecoveryThread] WARN

org.apache.solr.update.PeerSync:sync:232  - PeerSync: core=fr_green url=
http://solr-08/searchsolrnodefr too many updates received since start -
startingUpdates no longer overlaps with our currentUpdates

2013-12-05 13:51:56,806 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:doRecovery:394  - PeerSync Recovery
was not successful - trying replication. core=fr_green

2013-12-05 13:51:56,806 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:doRecovery:397  - Starting
Replication Recovery. core=fr_green

2013-12-05 13:51:56,806 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:doRecovery:399  - Begin buffering
updates. core=fr_green

2013-12-05 13:51:56,806 [RecoveryThread] INFO

org.apache.solr.cloud.RecoveryStrategy:replicate:127  - Attempting to
replicate from 

Re: PeerSync Recovery fails, starting Replication Recovery

[Anca Kopetz]

Hi,

We used to have many "Client session timeout" messages in solr logs.

INFO org.apache.zookeeper.ClientCnxn:run:1083 - Client session timed out, have 
not heard from server in 18461ms for sessionid 0x242047fc6d77804, closing 
socket connection and attempting reconnect

Then we set the zkClientTimeout to 30 seconds. Therefore there are no more 
messages of this type in the logs.

But we get some other messages :

Leader logs (solr-39):

2013-12-18 09:45:26,052 [http-8083-74] ERROR 
org.apache.solr.update.SolrCmdDistributor:log:119  - shard update error 
StdNode: 
http://solr-40/searchsolrnodees/es_blue/:org.apache.solr.client.solrj.SolrServerException:
 IOException occured when talking to server at: 
http://solr-40/searchsolrnodees/es_blue
...
Caused by: java.net.SocketException: Connection reset

2013-12-18 09:45:26,060 [http-8083-74] INFO  
org.apache.solr.client.solrj.impl.HttpClientUtil:createClient:103  - Creating new http 
client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-12-18 09:45:26,140 [http-8083-49] INFO  
org.apache.solr.handler.admin.CoreAdminHandler:handleWaitForStateAction:819  - 
Going to wait for coreNodeName: solr-40_searchsolrnodees_es_blue, state: 
recovering, checkLive: true, onlyIfLeader: true

Replica logs (solr-40) :

2013-12-18 09:45:26,083 [http-8083-65] INFO  
org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705  
- It has been requested that we recover
2013-12-18 09:45:26,091 [http-8083-65] INFO  
org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  - [admin] webapp=null 
path=/admin/cores 
params={action=REQUESTRECOVERY&core=es_blue&wt=javabin&version=2} status=0 
QTime=8
...
2013-12-18 09:45:29,190 [RecoveryThread] WARN  
org.apache.solr.update.PeerSync:sync:232  - PeerSync: core=es_blue 
url=http://dc2-s6-prod-solr-40.prod.dc2.kelkoo.net:8083/searchsolrnodees too 
many updates received since start - startingUpdates no longer overlaps with our 
currentUpdates
2013-12-18 09:45:29,191 [RecoveryThread] INFO  
org.apache.solr.cloud.RecoveryStrategy:doRecovery:394  - PeerSync Recovery was 
not successful - trying replication. core=es_blue
2013-12-18 09:45:29,191 [RecoveryThread] INFO  
org.apache.solr.cloud.RecoveryStrategy:doRecovery:397  - Starting Replication 
Recovery. core=es_blue

Therefore, if I understand it right, the leader does not manage to foward the documents 
to the replica due to a "Connection reset" problem, and it asks the replica to 
recover. The replica tries to recover, it fails, and it starts a replication recovery.

Why does the leader not retry to forward the documents to the replica when it 
gets an IOException in SolrCmdDistributor ?

Solr version is 4.5.1

For the Garbage Collector, we use the settings defined here 
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

Thank you,
Anca

On 12/19/2013 04:50 PM, Mark Miller wrote:

Sounds like you need to raise your ZooKeeper connection timeout.

Also, make sure you are using a concurrent garbage collector as a side note - 
stop the world pauses should be avoided. Just good advice :)

- Mark

On Dec 18, 2013, at 5:48 AM, Anca Kopetz 
<mailto:anca.kop...@kelkoo.com> wrote:



Hi,

In our SolrCloud cluster (2 shards, 8 replicas), the replicas go from time to 
time into recovering state, and it takes more than 10 minutes to finish to 
recover.

In logs, we see that "PeerSync Recovery" fails with the message :

PeerSync: core=fr_green url=http://solr-08/searchsolrnodefr too many updates 
received since start - startingUpdates no longer overlaps with our 
currentUpdates

Then "Replication Recovery" starts.

Is there something we can do to avoid the failure of "Peer Recovery" so that 
the recovery process is more rapid (less than 10 minutes) ?

The full trace log is here :

2013-12-05 13:51:53,740 [http-8080-46] INFO  
org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705  
- It has been requested that we recover
2013-12-05 13:51:53,740 [http-8080-112] INFO  
org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705  
- It has been requested that we recover
2013-12-05 13:51:53,740 [http-8080-112] INFO  
org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  - [admin] webapp=null 
path=/admin/cores 
params={action=REQUESTRECOVERY&core=fr_green&wt=javabin&version=2} status=0 
QTime=0
2013-12-05 13:51:53,740 [Thread-1544] INFO  
org.apache.solr.cloud.ZkController:publish:1017  - publishing core=fr_green 
state=recovering
2013-12-05 13:51:53,741 [http-8080-46] INFO  
org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  - [admin] webapp=null 
path=/admin/cores 
params={action=REQUESTRECOVERY&core=fr_green&wt=javabin&version=2} status=0 
QTime=1
2013-12-05 13:51:53,740 [Thread-1543] INFO  
org.apache.solr.cloud.ZkController:publish:1017  - publishing core=fr_green 
stat

Re: PeerSync Recovery fails, starting Replication Recovery

[Anca Kopetz]

Hi,

We do not use a NRT solution. We do a hardCommit (autocommit with
openSearcher=false) every 15 minutes, and updates with commitWithin
every 30 minutes. The updates are done one by one by many feeders, we do
not send batches of documents.

Solr version : 4.5.1

Yes, the problem is before the recovery, as I explained in the reply to
Mark's mail.

Best regards,
Anca Kopetz

On 12/19/2013 06:39 PM, Daniel Collins wrote:

Are you using a NRT solution, how often do you commit?  We see similar
issues with PeerSync, but then we have a very active NRT system and we
soft-commit sub-second, so since PeerSync has a limit of 100 versions
before it decides its too much to do, if we try and PeerSync whilst
indexing is running, we inevitably have to fallback to a full-sync as this
does.

What Solr version are you using?  There were issues with early 4.X (up to
4.3 wasn't it, I can't find the ticket now?) whhereby if PeerSync failed,
it did a fullCopy when what it should do is an incremental copy (i.e. only
the segments that are missing) which again might hurt you.

But surely your real problem isn't that the recovery taking a long time,
your problem is why did the system enter recovery in the first place (which
is the bit *just before* the trace you gave us!)
The first line is "It has been requested that we recover" (aren't Solr
trace message polite), which I know from recent experience means either the
leader thinks this replica is out of date or you just had a leadership
transfer.  As mark says, the root cause of that is probably a ZK timeout
issue



On 19 December 2013 15:50, Mark Miller  wrote:


Sounds like you need to raise your ZooKeeper connection timeout.

Also, make sure you are using a concurrent garbage collector as a side
note - stop the world pauses should be avoided. Just good advice :)

- Mark

On Dec 18, 2013, at 5:48 AM, Anca Kopetz  wrote:


Hi,

In our SolrCloud cluster (2 shards, 8 replicas), the replicas go from

time to time into recovering state, and it takes more than 10 minutes to
finish to recover.

In logs, we see that "PeerSync Recovery" fails with the message :

PeerSync: core=fr_green url=http://solr-08/searchsolrnodefr too many

updates received since start - startingUpdates no longer overlaps with our
currentUpdates

Then "Replication Recovery" starts.

Is there something we can do to avoid the failure of "Peer Recovery" so

that the recovery process is more rapid (less than 10 minutes) ?

The full trace log is here :

2013-12-05 13:51:53,740 [http-8080-46] INFO

  org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705
  - It has been requested that we recover

2013-12-05 13:51:53,740 [http-8080-112] INFO

  org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705
  - It has been requested that we recover

2013-12-05 13:51:53,740 [http-8080-112] INFO

  org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  -
[admin] webapp=null path=/admin/cores
params={action=REQUESTRECOVERY&core=fr_green&wt=javabin&version=2} status=0
QTime=0

2013-12-05 13:51:53,740 [Thread-1544] INFO

  org.apache.solr.cloud.ZkController:publish:1017  - publishing
core=fr_green state=recovering

2013-12-05 13:51:53,741 [http-8080-46] INFO

  org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  -
[admin] webapp=null path=/admin/cores
params={action=REQUESTRECOVERY&core=fr_green&wt=javabin&version=2} status=0
QTime=1

2013-12-05 13:51:53,740 [Thread-1543] INFO

  org.apache.solr.cloud.ZkController:publish:1017  - publishing
core=fr_green state=recovering

2013-12-05 13:51:53,743 [Thread-1544] INFO

  org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on
descriptor - reading it from system property

2013-12-05 13:51:53,746 [Thread-1543] INFO

  org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on
descriptor - reading it from system property

2013-12-05 13:51:53,755 [Thread-1543] WARN

  org.apache.solr.cloud.RecoveryStrategy:close:105  - Stopping recovery for
zkNodeName=solr-08_searchsolrnodefr_fr_greencore=fr_green

2013-12-05 13:51:53,756 [RecoveryThread] INFO

  org.apache.solr.cloud.RecoveryStrategy:run:216  - Starting recovery
process.  core=fr_green recoveringAfterStartup=false

2013-12-05 13:51:53,762 [RecoveryThread] INFO

  org.apache.solr.cloud.RecoveryStrategy:doRecovery:495  - Finished recovery
process. core=fr_green

2013-12-05 13:51:53,762 [RecoveryThread] INFO

  org.apache.solr.cloud.RecoveryStrategy:run:216  - Starting recovery
process.  core=fr_green recoveringAfterStartup=false

2013-12-05 13:51:53,765 [RecoveryThread] INFO

  org.apache.solr.cloud.ZkController:publish:1017  - publishing
core=fr_green state=recovering

2013-12-05 13:51:53,765 [RecoveryThread] INFO

  org.apache.solr.cloud.ZkController:publish:1021  - numShards not found on
de

Re: How to boost documents ?

[Anca Kopetz]

Hi,

Thank you for your response.

When I try the URL you sent me, I get the following error message :

org.apache.solr.search.SyntaxError: Infinite Recursion detected parsing query 
'beautiful Christmas tree'

Any idea what this means ?

Best regards,
Anca
On 12/16/2013 02:00 PM, Ahmet Arslan wrote:

Hi Anca,

Can you try following URL?



q=beautiful Christmas tree&mm=2&qf=title^12 
description^2&defType=dismax&bf=map(query($qq),0,0,0,100.0)&qq={!dismax qf='title 
description' mm=100%}beautiful Christmas tree

Modified from Jan's solution. See his original post [1] to a similar discussion.
[1] http://search-lucene.com/m/nK6t9j1fuc2





On Monday, December 16, 2013 12:19 PM, Anca Kopetz 
<mailto:anca.kop...@kelkoo.com> wrote:

Hi,

How to boost documents that contain all search terms in several of its fields ?

Below you cand find a simplified example :

The query with Min should match:
q=beautiful Christmas tree&mm=2&qf=title^12 description^2

There are two offers that match the query :
offer1 {title:"Christmas tree", description:"a joy for children"}

offer2 {title:"Christmas tree", description:"beautiful for holidays"}}

The first offer ranks before the second, despite of the fact that the second 
one contains all the search terms. I tried to play with the boosts of qf, but 
the results vary a lot.

Is there a way to add a boost on all search fields, the same way we do with pf 
on one field : pf=title:2^3.0 ?

Thank you,
Anca




Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: How to boost documents ?

[Anca Kopetz]

lr.search.QParser.getQuery(QParser.java:142)
   at 
org.apache.solr.search.FunctionQParser.parseNestedQuery(FunctionQParser.java:236)
   at 
org.apache.solr.search.ValueSourceParser$19.parse(ValueSourceParser.java:270)
   at 
org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:352)
   at 
org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:223)
   at 
org.apache.solr.search.ValueSourceParser$13.parse(ValueSourceParser.java:198)
   at 
org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:352)
   at org.apache.solr.search.FunctionQParser.parse(FunctionQParser.java:68)
   at org.apache.solr.search.QParser.getQuery(QParser.java:142)
   at 
org.apache.solr.search.ExtendedDismaxQParser.getBoostFunctions(ExtendedDismaxQParser.java:437)
   at 
org.apache.solr.search.ExtendedDismaxQParser.parse(ExtendedDismaxQParser.java:175)
   at org.apache.solr.search.QParser.getQuery(QParser.java:142)
...

Is this a bug ?

Thank you,
Anca
On 12/30/2013 02:30 PM, Anca Kopetz wrote:

Hi,

Thank you for your response.

When I try the URL you sent me, I get the following error message :

org.apache.solr.search.SyntaxError: Infinite Recursion detected parsing query 
'beautiful Christmas tree'

Any idea what this means ?

Best regards,
Anca
On 12/16/2013 02:00 PM, Ahmet Arslan wrote:

Hi Anca,

Can you try following URL?



q=beautiful Christmas tree&mm=2&qf=title^12 
description^2&defType=dismax&bf=map(query($qq),0,0,0,100.0)&qq={!dismax qf='title 
description' mm=100%}beautiful Christmas tree

Modified from Jan's solution. See his original post [1] to a similar discussion.
[1] http://search-lucene.com/m/nK6t9j1fuc2





On Monday, December 16, 2013 12:19 PM, Anca Kopetz 
<mailto:anca.kop...@kelkoo.com><mailto:anca.kop...@kelkoo.com><mailto:anca.kop...@kelkoo.com>
 wrote:

Hi,

How to boost documents that contain all search terms in several of its fields ?

Below you cand find a simplified example :

The query with Min should match:
q=beautiful Christmas tree&mm=2&qf=title^12 description^2

There are two offers that match the query :
offer1 {title:"Christmas tree", description:"a joy for children"}

offer2 {title:"Christmas tree", description:"beautiful for holidays"}}

The first offer ranks before the second, despite of the fact that the second 
one contains all the search terms. I tried to play with the boosts of qf, but 
the results vary a lot.

Is there a way to add a boost on all search fields, the same way we do with pf 
on one field : pf=title:2^3.0 ?

Thank you,
Anca




Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: PeerSync Recovery fails, starting Replication Recovery

[Anca Kopetz]

Hi,

We tried to understand why we get a "Connection reset" exception on the leader 
when it tries to foward the documents to one of its replica. We analyzed the GC logs and 
we did not see any long GC pauses around the time the exception was thrown.

For 24 hours of gc logs, the max full gc pause = 3 seconds, but it is not in the same 
time as the "Connection reset" exception.

My question again :

Why does the leader not retry to forward the documents to the replica when it gets an 
IOException in SolrCmdDistributor ? Instead, it sents a "recovery" request to 
the replica.

Solr version is 4.5.1 (we tried 4.6.0, but we had some problems with this version, 
detailed in the mail : "SolrCloud 4.6.0 - leader election issue").

Related to "Connection reset" exception, is there any other tests that we can 
do in order to the cause of it ?

Thank you,
Anca

On 12/20/2013 11:07 AM, Anca Kopetz wrote:

Hi,

We used to have many "Client session timeout" messages in solr logs.

INFO org.apache.zookeeper.ClientCnxn:run:1083 - Client session timed out, have 
not heard from server in 18461ms for sessionid 0x242047fc6d77804, closing 
socket connection and attempting reconnect

Then we set the zkClientTimeout to 30 seconds. Therefore there are no more 
messages of this type in the logs.

But we get some other messages :

Leader logs (solr-39):

2013-12-18 09:45:26,052 [http-8083-74] ERROR 
org.apache.solr.update.SolrCmdDistributor:log:119  - shard update error 
StdNode: 
http://solr-40/searchsolrnodees/es_blue/:org.apache.solr.client.solrj.SolrServerException:
 IOException occured when talking to server at: 
http://solr-40/searchsolrnodees/es_blue
...
Caused by: java.net.SocketException: Connection reset

2013-12-18 09:45:26,060 [http-8083-74] INFO  
org.apache.solr.client.solrj.impl.HttpClientUtil:createClient:103  - Creating new http 
client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-12-18 09:45:26,140 [http-8083-49] INFO  
org.apache.solr.handler.admin.CoreAdminHandler:handleWaitForStateAction:819  - 
Going to wait for coreNodeName: solr-40_searchsolrnodees_es_blue, state: 
recovering, checkLive: true, onlyIfLeader: true

Replica logs (solr-40) :

2013-12-18 09:45:26,083 [http-8083-65] INFO  
org.apache.solr.handler.admin.CoreAdminHandler:handleRequestRecoveryAction:705  
- It has been requested that we recover
2013-12-18 09:45:26,091 [http-8083-65] INFO  
org.apache.solr.servlet.SolrDispatchFilter:handleAdminRequest:658  - [admin] webapp=null 
path=/admin/cores 
params={action=REQUESTRECOVERY&core=es_blue&wt=javabin&version=2} status=0 
QTime=8
...
2013-12-18 09:45:29,190 [RecoveryThread] WARN  
org.apache.solr.update.PeerSync:sync:232  - PeerSync: core=es_blue 
url=http://dc2-s6-prod-solr-40.prod.dc2.kelkoo.net:8083/searchsolrnodees too 
many updates received since start - startingUpdates no longer overlaps with our 
currentUpdates
2013-12-18 09:45:29,191 [RecoveryThread] INFO  
org.apache.solr.cloud.RecoveryStrategy:doRecovery:394  - PeerSync Recovery was 
not successful - trying replication. core=es_blue
2013-12-18 09:45:29,191 [RecoveryThread] INFO  
org.apache.solr.cloud.RecoveryStrategy:doRecovery:397  - Starting Replication 
Recovery. core=es_blue

Therefore, if I understand it right, the leader does not manage to foward the documents 
to the replica due to a "Connection reset" problem, and it asks the replica to 
recover. The replica tries to recover, it fails, and it starts a replication recovery.

Why does the leader not retry to forward the documents to the replica when it 
gets an IOException in SolrCmdDistributor ?

Solr version is 4.5.1

For the Garbage Collector, we use the settings defined here 
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

Thank you,
Anca

On 12/19/2013 04:50 PM, Mark Miller wrote:

Sounds like you need to raise your ZooKeeper connection timeout.

Also, make sure you are using a concurrent garbage collector as a side note - 
stop the world pauses should be avoided. Just good advice :)

- Mark

On Dec 18, 2013, at 5:48 AM, Anca Kopetz 
<mailto:anca.kop...@kelkoo.com><mailto:anca.kop...@kelkoo.com><mailto:anca.kop...@kelkoo.com>
 wrote:



Hi,

In our SolrCloud cluster (2 shards, 8 replicas), the replicas go from time to 
time into recovering state, and it takes more than 10 minutes to finish to 
recover.

In logs, we see that "PeerSync Recovery" fails with the message :

PeerSync: core=fr_green url=http://solr-08/searchsolrnodefr too many updates 
received since start - startingUpdates no longer overlaps with our 
currentUpdates

Then "Replication Recovery" starts.

Is there something we can do to avoid the failure of "Peer Recovery" so that 
the recovery process is more rapid (less than 10 minutes) ?

The full trace log is here :

2013-12-05 13:51:53,740 [http-8080-46] INFO  
org.apache.solr.handler.adm

Re: How to boost documents ?

[Anca Kopetz]

Hi,

I tested the BoostQueryParser and it works on the simplified example.
But we need to keep the edismax Query parser, so I tried the following query 
and it seems to work (I defined a local bf='' for qq).

&q=beautiful Christmas tree
&mm=2
&qf=title^12 description^2
&defType=edismax
&bf=map(query($qq),0,0,0,100.0)
&qq={!edismax bf='' mm=100%}beautiful Christmas tree

Best regards,
Anca

On 01/07/2014 07:42 PM, Ahmet Arslan wrote:

Hi Hoss,

Thanks for the explanation. Very complicated stuff. I never understand 
NestedQParserPlugin.

We want all the documents with all terms (mm=100%) get an extra 1000 points, 
but change nothing else. How would you restructure the following query?

q=beautiful Christmas tree&mm=2&qf=title^12 
description^2&defType=dismax&bf=map(query($qq),0,0,0,100.0)&qq={!dismax qf='title 
description' mm=100%}beautiful Christmas tree

Ahmet


On Tuesday, January 7, 2014 8:12 PM, Chris Hostetter 
 wrote:


: http://localhost:8983/solr/collection1/select?q=ipod
: 
belkin&wt=xml&debugQuery=true&q.op=AND&defType=edismax&bf=map(query($qq),0,0,0,100.0)&qq={!edismax}power
:
: The error is :
: org.apache.solr.search.SyntaxError: Infinite Recursion detected parsing query
: 'power'
:
: And the stacktrace :
:
: ERROR - 2014-01-06 18:27:02.275; org.apache.solr.common.SolrException;
: org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError:
: Infinite Recursion detected parsing query 'power'

your "qq" param uses the edismax parser which goes looking for a "bf" -
since there is no local bf param, it finds the global one -- which
recursively refers to the "qq" param again.  Hence the infinite recursion.

You either need to override the bf param locally in your qq param, or
restructure your query slightly so the bf is not global

Perhaps something like this...

qq=ipod belkin
q={!query defType=edismax bf=$boostFunc v=$qq}
boostFunc=map(query($boostQuery),0,0,0,100.0)
boostQuery={!edismax}power

having said that however: instead of using "bf" you should probably
consider using the "boost" parser -- it's a multiplicitive boost instead
of an additive boost, and (in my opinion) makes the flow/params easier to
understand...

https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-BoostQueryParser

qq=ipod belkin
q={!boost defType=edismax b=$boostFunc v=$qq}
boostFunc=map(query($boostQuery),0,0,0,100.0)
boostQuery={!edismax}power



-Hoss
http://www.lucidworks.com/





Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Sharding configuration

[Anca Kopetz]

Hi,

We have a SolrCloud configuration of 10 servers, no sharding, 20
millions of documents, the index has 26 GB.
As the number of documents has increased recently, the performance of
the cluster decreased.

We thought of sharding the index, in order to measure the latency. What
is the best approach ?
- to use shard splitting and have several sub-shards on the same server
and in the same tomcat instance
- having several shards on the same server but on different tomcat instances
- having one shard on each server (for example 2 shards / 5 replicas on
10 servers)

What's the impact of these 3 configuration on performance ?

Thanks,
Anca

--

Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: Sharding configuration

[Anca Kopetz]

Hi,

We did some tests with 4 shards / 4 different tomcat instances on the
same server and the average latency was smaller than the one when having
only one shard.
We tested also é shards on different servers and the performance results
were also worse.

It seems that the sharding does not make any difference for our index in
terms of latency gains.

Thanks for your response,
Anca

On 10/28/2014 08:44 PM, Ramkumar R. Aiyengar wrote:

As far as the second option goes, unless you are using a large amount of
memory and you reach a point where a JVM can't sensibly deal with a GC
load, having multiple JVMs wouldn't buy you much. With a 26GB index, you
probably haven't reached that point. There are also other shared resources
at an instance level like connection pools and ZK connections, but those
are tunable and you probably aren't pushing them as well (I would imagine
you are just trying to have only a handful of shards given that you aren't
sharded at all currently).

That leaves single vs multiple machines. Assuming the network isn't a
bottleneck, and given the same amount of resources overall (number of
cores, amount of memory, IO bandwidth times number of machines), it
shouldn't matter between the two. If you are procuring new hardware, I
would say buy more, smaller machines, but if you already have the hardware,
you could serve as much as possible off a machine before moving to a
second. There's nothing which limits the number of shards as long as the
underlying machine has the sufficient amount of parallelism.

Again, this advice is for a small number of shards, if you had a lot more
(hundreds) of shards and significant volume of requests, things start to
become a bit more fuzzy with other limits kicking in.
On 28 Oct 2014 09:26, "Anca Kopetz"  wrote:


Hi,

We have a SolrCloud configuration of 10 servers, no sharding, 20
millions of documents, the index has 26 GB.
As the number of documents has increased recently, the performance of
the cluster decreased.

We thought of sharding the index, in order to measure the latency. What
is the best approach ?
- to use shard splitting and have several sub-shards on the same server
and in the same tomcat instance
- having several shards on the same server but on different tomcat
instances
- having one shard on each server (for example 2 shards / 5 replicas on
10 servers)

What's the impact of these 3 configuration on performance ?

Thanks,
Anca

--

Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à
l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
destinataire de ce message, merci de le détruire et d'en avertir
l'expéditeur.



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: Sharding configuration

[Anca Kopetz]

Hi,

You are right, it is a mistake in my phrase, for the tests with 4
shards/ 4 instances,  the latency was worse (therefore *bigger*) than
for the tests with one shard.

In our case, the query rate is high.

Thanks,
Anca

On 10/30/2014 03:48 PM, Shawn Heisey wrote:

On 10/30/2014 4:32 AM, Anca Kopetz wrote:

We did some tests with 4 shards / 4 different tomcat instances on the
same server and the average latency was smaller than the one when having
only one shard.
We tested also é shards on different servers and the performance results
were also worse.

It seems that the sharding does not make any difference for our index in
terms of latency gains.

That statement is confusing, because if latency goes down, that's good,
not worse.

If you're going to put multiple shards on one server, it should be done
with one solr/tomcat instance, not multiple.  One instance is perfectly
capable of dealing with many shards, and has a lot less overhead.  The
SolrCloud collection create command would need the maxShardsPerNode
parameter.

In order to see a gain in performance from multiple shards per server,
the server must have a lot of CPUs and the query rate must be fairly
low.  If the query rate is high, then all the CPUs will be busy just
handling simultaneous queries, so putting multiple shards per server
will probably slow things down.  When query rate is low, multiple CPUs
can handle each shard query simultaneously, speeding up the overall query.

Thanks,
Shawn



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Problem with CoreAdminRequest/GET_CORE_STATUS in SolrCloud

[Anca Kopetz]

Hi,

We have a solr cloud cluster with 2 different collections, each
collection having many nodes. We try to get the status of each
collection using CoreAdminRequest.
The code gets all live nodes from the cluster and sends a request to
each node until it gets a valid response.

We would like to have the status for each collection, but the code does
not filter the live nodes by collection name when the request path
contains "/admin/collections" or "/admin/cores".

CloudSolrServer.java
if (request.getPath().equals("/admin/collections") ||
request.getPath().equals("/admin/cores")) {
  Set liveNodes = clusterState.getLiveNodes();
  for (String liveNode : liveNodes) {
int splitPointBetweenHostPortAndContext = liveNode.indexOf("_");
theUrlList.add("http://";
+ liveNode.substring(0,
splitPointBetweenHostPortAndContext) + "/"
+ URLDecoder.decode(liveNode,
"UTF-8").substring(splitPointBetweenHostPortAndContext + 1));
  }
}


What is the explanation for this implementation ? Is it possible that
the code filters the live nodes by collection name ?

Best regards,
Anca

Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

"optimize" index : impact on performance

[Anca Kopetz]

Hi,

We are trying to improve the performance of our Solr Search application in 
terms of QPS (queries per second).

We tuned SOLR settings (e.g. mergeFactor=3), launched several benchmarks and 
had better performance results, but still unsatisfactory for our traffic volume.
Then we optimized the index to 1 segment / 0 deleted docs and we got +40% of 
QPS compared to the previous test.

Therefore we thought of optimizing the index every two hours, as our index is 
evolving due to frequent commits (every 30 minutes) and thus the performance 
results are degrading.

1. Is this a good practice ?
2. Instead of executing an "optimize" many times a day, are there any other 
parameters that we can tune and test in order to gain in average QPS?

We want to avoid the solution of adding more servers to our SolrCloud cluster.

Some details of our system :

SolrCloud cluster: 8 nodes on 8 dedicated servers; 2 shards / 4 replicas
Hardware configuration: 2 Processors (16CPU cores) per server; 24GB of memory; 
6GB allocated to JVM
Index: 13M documents, 15GB
Search algorithm : grouping, faceting, filter queries
Solr version 4.4

Best regards,
Anca Kopetz



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

"optimize" index : impact on performance [Republished]

[Anca Kopetz]

Hi,

[I am sending again my message to the mailing list, as well as Shawn's reply. 
Thanks Shawn for your explanations]

We are trying to improve the performance of our Solr Search application in 
terms of QPS (queries per second).

We tuned SOLR settings (e.g. mergeFactor=3), launched several benchmarks and 
had better performance results, but still unsatisfactory for our traffic volume.
Then we optimized the index to 1 segment / 0 deleted docs and we got +40% of 
QPS compared to the previous test.

Therefore we thought of optimizing the index every two hours, as our index is 
evolving due to frequent commits (every 30 minutes) and thus the performance 
results are degrading.

1. Is this a good practice ?
2. Instead of executing an "optimize" many times a day, are there any other 
parameters that we can tune and test in order to gain in average QPS?

We want to avoid the solution of adding more servers to our SolrCloud cluster.

Some details of our system :

SolrCloud cluster: 8 nodes on 8 dedicated servers; 2 shards / 4 replicas
Hardware configuration: 2 Processors (16CPU cores) per server; 24GB of memory; 
6GB allocated to JVM
Index: 13M documents, 15GB
Search algorithm : grouping, faceting, filter queries
Solr version 4.4



Please read and follow this note about thread hijacking:

http://people.apache.org/~hossman/#threadhijack

Optimizing that frequently with an index that large *might* cause more
problems than it solves.  You'd have to actually try it to see whether
it works for you, though.  Here's some information explaining why it may
be a problem:

Optimizing a 15GB index is likely to take up to 15 minutes, depending on
how fast the I/O subsystem on your servers is.  It probably won't happen
in less than 5 minutes unless you're running on SSD, which also
mitigates some of the impact described in the next paragraph.

Performance will be lower, potentially a LOT lower, for those few
minutes while an optimize is occurring.  Solr has to read the index,
process each document, and write it back out.  It does happen quite
fast, but that's a lot of I/O.  Because it's continually going back and
forth between the old copy and the new copy, the OS disk cache will have
critical data evicted for the entire process, unless you have enough
free RAM so *twice* the index can fit in the cache, and from your
mentioned stats, you don't.

FYI, commits every 30 minutes are NOT frequent.  Commits happening one
or more times every *second* are frequent.

If you can share your solrconfig.xml, there might be some suggestions we
can make so things will generally work better.  The list doesn't accept
attachments.  It's better if you use a paste website like
http://www.fpaste.org/, choose the proper language for highlighting, and
set the "delete after" setting to something that will work for you.
Making it a paste that never gets deleted will mean that your message
will retain usefulness for others as long as archives exist, but you
might not want it available that long.

Properly tuning your garbage collection is important.  The default
garbage collector is, risking a pun, garbage.

http://wiki.apache.org/solr/SolrPerformanceProblems#GC_pause_problems
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

Thanks,
Shawn





Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: "optimize" index : impact on performance [Republished]

[Anca Kopetz]

Hi,

We already did some benchmarks during optimize and we haven't noticed a big 
impact on overall performance of search. The benchmarks' results were almost 
the same with vs. without running optimization. We have enough free RAM for the 
two OS disk caches during optimize (15 GB represents the total size of our 
index, which is split into 2 shards, therefore there are 7.5GB of index per 
server that has 24GB of memory). But we will launch them again, maybe we missed 
smth.

Here you can fing our solconfig.xml file http://fpaste.org/30154/.

Thanks for the urls on GC tunning, we will test them.

Best regards,
Anca

On 08/05/2013 10:42 AM, Anca Kopetz wrote:

Please read and follow this note about thread hijacking:

http://people.apache.org/~hossman/#threadhijack<http://people.apache.org/%7Ehossman/#threadhijack><http://people.apache.org/%7Ehossman/#threadhijack>

Optimizing that frequently with an index that large *might* cause more
problems than it solves.  You'd have to actually try it to see whether
it works for you, though.  Here's some information explaining why it may
be a problem:

Optimizing a 15GB index is likely to take up to 15 minutes, depending on
how fast the I/O subsystem on your servers is.  It probably won't happen
in less than 5 minutes unless you're running on SSD, which also
mitigates some of the impact described in the next paragraph.

Performance will be lower, potentially a LOT lower, for those few
minutes while an optimize is occurring.  Solr has to read the index,
process each document, and write it back out.  It does happen quite
fast, but that's a lot of I/O.  Because it's continually going back and
forth between the old copy and the new copy, the OS disk cache will have
critical data evicted for the entire process, unless you have enough
free RAM so *twice* the index can fit in the cache, and from your
mentioned stats, you don't.

FYI, commits every 30 minutes are NOT frequent.  Commits happening one
or more times every *second* are frequent.

If you can share your solrconfig.xml, there might be some suggestions we
can make so things will generally work better.  The list doesn't accept
attachments.  It's better if you use a paste website like
http://www.fpaste.org/, choose the proper language for highlighting, and
set the "delete after" setting to something that will work for you.
Making it a paste that never gets deleted will mean that your message
will retain usefulness for others as long as archives exist, but you
might not want it available that long.

Properly tuning your garbage collection is important.  The default
garbage collector is, risking a pun, garbage.

http://wiki.apache.org/solr/SolrPerformanceProblems#GC_pause_problems
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

Thanks,
Shawn





Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.