Re: Indexing timeout issues with SolrCloud 7.1
>From the error list, i can see multiple errors: 1. Failure to recover replica 2. Peer sync error 3. Failure to download file On 24 Feb 2018 03:10, "Tom Peters" wrote: I included the last 25 lines from the logs from each of the five nodes during that time period. I _think_ I'm running into issues with bulking up deleteByQuery. Quick background: we have objects in our system that may have multiple availability windows. So when we index an object, will store it as separate documents each with their own begins and expires date. At index time we don't know if the all of the windows are still valid or not, so we remove all of them with a deleteByQuery (e.g. deleteByQuery=object_id:12345) and then index one or more documents. I ran an isolated test a number of times where I indexed 1500 documents in this manner (deletes then index). In Solr 3.4, it takes about 15s to complete. In Solr 7.1, it's taking about 5m. If I remove the deleteByQuery, the indexing times are nearly identical. When run in normal production mode where we have lots of processes indexing at once (~20 or so), it starts to cause lots of issues (which you see below). Please let me know if anything I mentioned is unclear. Thanks! solr2-a: 2018-02-23 04:09:36.551 ERROR (updateExecutor-2-thread-2672- processing-http:solr2-b:8080//solr//mycollection_shard1_replica_n1 x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. ErrorReportingConcurrentUpdateSolrClient error 2018-02-23 04:09:36.551 ERROR (updateExecutor-2-thread-2692- processing-http:solr2-d:8080//solr//mycollection_shard1_replica_n11 x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. ErrorReportingConcurrentUpdateSolrClient error 2018-02-23 04:09:36.551 ERROR (updateExecutor-2-thread-2711- processing-http:solr2-e:8080//solr//mycollection_shard1_replica_n4 x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. ErrorReportingConcurrentUpdateSolrClient error 2018-02-23 04:09:36.552 ERROR (qtp1595212853-32739) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://solr2-b:8080/solr/ mycollection_shard1_replica_n1/ 2018-02-23 04:09:36.552 ERROR (qtp1595212853-32739) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://solr2-d:8080/solr/ mycollection_shard1_replica_n11/ 2018-02-23 04:09:36.552 ERROR (qtp1595212853-32739) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://solr2-e:8080/solr/ mycollection_shard1_replica_n4/ 2018-02-23 04:09:38.217 ERROR (updateExecutor-2-thread-2712- processing-http:solr2-e:8080//solr//mycollection_shard1_replica_n4 x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. ErrorReportingConcurrentUpdateSolrClient error 2018-02-23 04:09:38.217 ERROR (updateExecutor-2-thread-2726- processing-http:solr2-d:8080//solr//mycollection_shard1_replica_n11 x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. ErrorReportingConcurrentUpdateSolrClient error 2018-02-23 04:09:38.217 ERROR (updateExecutor-2-thread-2727- processing-http:solr2-b:8080//solr//mycollection_shard1_replica_n1 x:mycollection_shard1_replica_n6 r:core_node9 n:solr2-a.vam.be.cmh. mycollection.com:8080_solr s:shard1 c:mycollection) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u. ErrorReportingConcurrentUpdateSolrClient error 2018-02-23 04:09:38.218 ERROR (qtp1595212853-32260) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://solr2-b:8080/solr/ mycollection_shard1_replica_n1/ 2018-02-23 04:09:38.218 ERROR (qtp1595212853-32260) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://solr2-d:8080/solr/ mycollection_shard1_replica_n11/ 2018-02-23 04:09:38.218 ERROR (qtp1595212853-32260) [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6] o.a.s.u.p.DistributedUpdateProc
Re: Could not find collection when upgrading to Solr 7
On 2/24/2018 3:13 AM, Marvin Bredal Lillehaug wrote: We have a multicore Solr-instance that currently is running version 6.6.2. For local development we run ./solr start -v -d $solrDir/server -h localhost -p $SOLR_PORT -c -f -s $solrHomeDir -a "-Dbootstrap_conf=true -Djetty.host=localhost" -m 4096m Are you aware that the -c option, without the -z option, will start an embedded zookeeper server and put Solr into SolrCloud mode? The embedded ZK server is not recommended except for setting up a proof of concept. For production, the ZK ensemble needs at least three servers. Setting jetty.host to localhost will make it so that Solr cannot be accessed remotely, it will only work on the local machine. This defeats one of the primary purposes of SolrCloud -- a highly available fault tolerant cluster. The bootstrap_conf setting is intended to be used precisely once -- in a situation where you want to convert a non-cloud install to SolrCloud. It is not meant to be defined every time you start Solr. I would strongly recommended that you do not use bootstrap at all. Instead, start up a new install, create your collection how you desire, and then run your indexing so you create the index fresh. On Solr 6 everything starts fine, and cores and collections are created on startup. When running the same setup on Solr 7.x.x it crash with Could not find collection : nvdb-transactionlog when running CoreContainer.repairCoreProperty. Logs for Solr 6 and 7 attached. That error probably means that the collection was not found in the ZooKeeper database. Are you doing any kind of configuration or changes on the embedded ZK server before you start Solr? Does the user that's running Solr have enough permissions to create and write to the ZK data directory? The problem *might* be the bootstrap_conf setting, but I am not sure whether that is correct. Unless something is very broken in your install, then I would suspect the bootstrap option is causing a problem. Thanks, Shawn
Solt with no text search
0
down vote
favorite
I have successfuly implemented the custom search in solr by extending
CustomScoreProvider. But the problem is i need to pass some user info like
31,1 which i need in custom sort.
q={!mycustomparser}31+1+*:*
and it is acting as a text search. Is there any way that i can skip this
text search and can just pass the user info.
some code.
public class MyCustomParserPlugin extends QParserPlugin {
@Override
public QParser createParser(String querystring, SolrParams slocParams,
SolrParams params, SolrQueryRequest req) {
return new MyCustomParser(querystring, slocParams, params, req);
}
private static class MyCustomParser extends QParser{
private Query inQuery;
public MyCustomParser(String querystring, SolrParams slocParams,
SolrParams params, SolrQueryRequest req) {
super(querystring, slocParams, params, req);
try {
QParser parser = getParser(querystring, getReq());
this.inQuery = parser.parse();
}catch(SyntaxError ex) {
throw new RuntimeException("error parsing query", ex);
}
}
@Override
public Query parse() throws SyntaxError {
return new MatchingQuery(inQuery);
}
}
}
=
public class MatchingQuery extends CustomScoreQuery {
private Query subQuery;
public MatchingQuery(Query subQuery) {
super(subQuery);
this.subQuery = subQuery;
}
@Override
protected CustomScoreProvider getCustomScoreProvider(LeafReaderContext
context) {
return new MyCustomSortClz(subQuery, context);
}
}
=
public class MyCustomSortClz extends CustomScoreProvider {
// ---> my sort
}
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Could not find collection when upgrading to Solr 7
This is for doing local development and running tests. So starting with embedded zookeeper in SolrCloud mode is intentional. The idea is to just run the script setting up the folder structure for the cores and everything is up and running, ready for indexing and querying locally. The applications using Solr also use Zookeeper for configuration and coordination of their own stuff. There are no persmission issues. I have put the scripts here https://github.com/computerlove/start-solr The file config specifies what version to use. When running run-dev-solr.sh that version is downloaded, solrhome is copied to the right place and the instance is started. On Sat, Feb 24, 2018 at 1:20 PM, Shawn Heisey wrote: > On 2/24/2018 3:13 AM, Marvin Bredal Lillehaug wrote: > >> We have a multicore Solr-instance that currently is running version 6.6.2. >> For local development we run >> >> ./solr start -v -d $solrDir/server -h localhost -p $SOLR_PORT -c -f -s >> $solrHomeDir -a "-Dbootstrap_conf=true -Djetty.host=localhost" -m 4096m >> >> > Are you aware that the -c option, without the -z option, will start an > embedded zookeeper server and put Solr into SolrCloud mode? The embedded > ZK server is not recommended except for setting up a proof of concept. For > production, the ZK ensemble needs at least three servers. > > Setting jetty.host to localhost will make it so that Solr cannot be > accessed remotely, it will only work on the local machine. This defeats > one of the primary purposes of SolrCloud -- a highly available fault > tolerant cluster. > > The bootstrap_conf setting is intended to be used precisely once -- in a > situation where you want to convert a non-cloud install to SolrCloud. It > is not meant to be defined every time you start Solr. I would strongly > recommended that you do not use bootstrap at all. Instead, start up a new > install, create your collection how you desire, and then run your indexing > so you create the index fresh. > > On Solr 6 everything starts fine, and cores and collections are created on >> startup. >> When running the same setup on Solr 7.x.x it crash with Could not find >> collection : nvdb-transactionlog when running >> CoreContainer.repairCoreProperty. >> Logs for Solr 6 and 7 attached. >> > > That error probably means that the collection was not found in the > ZooKeeper database. Are you doing any kind of configuration or changes on > the embedded ZK server before you start Solr? Does the user that's running > Solr have enough permissions to create and write to the ZK data directory? > The problem *might* be the bootstrap_conf setting, but I am not sure > whether that is correct. Unless something is very broken in your install, > then I would suspect the bootstrap option is causing a problem. > > Thanks, > Shawn > > -- med vennlig hilsen, Marvin B. Lillehaug
Solr Phrase Count : How to get count of a phrase in a text field solr
Hi All, I want to get the count of a phrase from a document . Currently im using Shingle Filter factory but it consuming a large disk space. Any alternate ways or any way to optimize this. currently it consuming 40GB for just 46K records my schema setting is given below -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Solt with no text search
0
down vote
favorite
I have successfuly implemented the custom search in solr by extending
CustomScoreProvider. But the problem is i need to pass some user info like
31,1 which i need in custom sort.
*q={!mycustomparser}31+1+*:**
and it is acting as a text search. Is there any way that i can skip this
text search and can just pass the user info.
*some code.*
public class MyCustomParserPlugin extends QParserPlugin {
@Override
public QParser createParser(String querystring, SolrParams slocParams,
SolrParams params, SolrQueryRequest req) {
return new MyCustomParser(querystring, slocParams, params, req);
}
private static class MyCustomParser extends QParser{
private Query inQuery;
public MyCustomParser(String querystring, SolrParams slocParams,
SolrParams params, SolrQueryRequest req) {
super(querystring, slocParams, params, req);
try {
QParser parser = getParser(querystring, getReq());
this.inQuery = parser.parse();
}catch(SyntaxError ex) {
throw new RuntimeException("error parsing query", ex);
}
}
@Override
public Query parse() throws SyntaxError {
return new MatchingQuery(inQuery);
}
}
}
=
public class MatchingQuery extends CustomScoreQuery {
private Query subQuery;
public MatchingQuery(Query subQuery) {
super(subQuery);
this.subQuery = subQuery;
}
@Override
protected CustomScoreProvider getCustomScoreProvider(LeafReaderContext
context) {
return new MyCustomSortClz(subQuery, context);
}
}
=
public class MyCustomSortClz extends CustomScoreProvider {
// ---> my sort
}
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
