RE: Unable to delete zookeeper queue
Hi, Does anyone know a working solution to delete zookeeper queue data? Please help!! Regards, Vinodh From: Kommu, Vinodh K. Sent: Tuesday, March 31, 2020 12:55 PM To: solr-user@lucene.apache.org Subject: Unable to delete zookeeper queue All, For some reason one of our zookeeper queue was filled with way bigger number so when I tried to delete queues with "rmr /overseer/queue" command, it's throwing - Packet len19029055 is out of range! exception. Later I have increased maxbuffer size to 50M and tried the same rmr command but still getting following error. Since the queues are not getting deleted, solr cluster status is not healthy which apparently marks all replicas as down even nodes are up & running. Looks like it is a known bug with zookeeper. Is there a way to delete zookeeper queues forcefully? Error snippet: Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/queue at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114) at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:588) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282) Queue output: get /overseer/queue null cZxid = 0x30017 ctime = Wed Feb 06 22:29:14 EST 2019 mZxid = 0x30017 mtime = Wed Feb 06 22:29:14 EST 2019 pZxid = 0x341869 cversion = 1420613 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 0 numChildren = 1119355 Regards, Vinodh DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
Re: Unable to delete zookeeper queue
Maybe you need I inc on zk server and zk client Jute Max bufffer to execute this . You can better ask the ZK mailing list > Am 01.04.2020 um 14:53 schrieb Kommu, Vinodh K. : > > Hi, > > Does anyone know a working solution to delete zookeeper queue data? Please > help!! > > > Regards, > Vinodh > > From: Kommu, Vinodh K. > Sent: Tuesday, March 31, 2020 12:55 PM > To: solr-user@lucene.apache.org > Subject: Unable to delete zookeeper queue > > All, > > For some reason one of our zookeeper queue was filled with way bigger number > so when I tried to delete queues with "rmr /overseer/queue" command, it's > throwing - Packet len19029055 is out of range! exception. Later I have > increased maxbuffer size to 50M and tried the same rmr command but still > getting following error. Since the queues are not getting deleted, solr > cluster status is not healthy which apparently marks all replicas as down > even nodes are up & running. Looks like it is a known bug with zookeeper. Is > there a way to delete zookeeper queues forcefully? > > Error snippet: > > Exception in thread "main" > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss for /overseer/queue >at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) >at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) >at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114) >at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49) >at > org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703) >at > org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:588) >at > org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360) >at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323) >at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282) > > > Queue output: > > get /overseer/queue > null > cZxid = 0x30017 > ctime = Wed Feb 06 22:29:14 EST 2019 > mZxid = 0x30017 > mtime = Wed Feb 06 22:29:14 EST 2019 > pZxid = 0x341869 > cversion = 1420613 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 0 > numChildren = 1119355 > > > Regards, > Vinodh > > DTCC DISCLAIMER: This email and any files transmitted with it are > confidential and intended solely for the use of the individual or entity to > whom they are addressed. If you have received this email in error, please > notify us immediately and delete the email and any attachments from your > system. The recipient should check this email and any attachments for the > presence of viruses. The company accepts no liability for any damage caused > by any virus transmitted by this email.
Re: Request Tracking in Solr
Hi Prakhar, Newer versions of Solr offer an "Audit Logging" plugin for use cases similar to yours. https://lucene.apache.org/solr/guide/8_1/audit-logging.html If don't think that's available as far back as 5.2.1 though. Just thought I'd mention it in case upgrading is an option. Best, Jason On Wed, Apr 1, 2020 at 2:29 AM Prakhar Kumar wrote: > > Hello Folks, > > I'm looking for a way to track requests in Solr from a > particular user/client. Suppose, I've created a user, say *Client1*, using > the basic authentication/authorization plugin. Now I want to get a count of > the number of requests/queries made by *Client1* on the Solr server. > Looking forward to some valuable suggestions. > > P.S. We are using Solr 5.2.1 > > -- > Kind Regards, > Prakhar Kumar > Sr. Enterprise Software Engineer > > *HotWax Systems* > *Enterprise open source experts* > cell: +91-89628-81820 > office: 0731-409-3684 > http://www.hotwaxsystems.com
Re: Solrcloud 7.6 OOM due to unable to create native threads
We have defined a “search feed” as a file of JSONL objects, one per line. The feed files can be stored in S3, reloaded, sent to two clusters, etc. Each destination can keep its own log of failures and retries. We’ve been doing this for full batch feeds and incrementals for a few years. We’ve been using a Java program to load those, but I just wrote a multi-threaded Python thingy that uses the JSON update handlers. That is pretty simple code. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Mar 31, 2020, at 11:19 PM, S G wrote: > > One approach could be to buffer the messages in Kafka before pushing to > Solr. > And then use "Kafka mirror" to replicate the messages to the other DC. > Now both DCs' Kafka pipelines are in sync by the mirror and you can run > storm/spark/flink etc jobs to consume local Kafka and publish to local Solr > clusters. > This moves the responsibility of DR-sync to something designed specifically > for this purpose - Kafka mirror. > However do not use more than an year old version of Kafka as they had lot > of issues with mirroring. > > > On Mon, Mar 30, 2020 at 11:43 PM Raji N wrote: > >> Hi Eric, >> >> What are you recommendations for SolrCloud DR strategy. >> >> Thanks, >> Raji >> >> On Sun, Mar 29, 2020 at 6:25 PM Erick Erickson >> wrote: >> >>> I don’t recommend CDCR at this point, I think there better approaches. >>> >>> The root problem is that CDCR uses tlog files as a queueing mechanism. >>> If the connection between the DCs is broken for any reason, the tlogs >> grow >>> without limit. This could probably be fixed, but a better alternative is >> to >>> use something designed to insure messages (updates) are delivered to >>> separate DCs rathe than try to have CDCR re-invent that wheel. >>> >>> Best, >>> Erick >>> On Mar 29, 2020, at 6:47 PM, S G wrote: Is CDCR even recommended to be used in production? Or it was abandoned before it could become production ready ? Thanks SG On Sun, Mar 29, 2020 at 5:18 AM Erick Erickson < >> erickerick...@gmail.com> wrote: > What that error usually means is that there are a zillion threads >>> running. > > Try taking a thread dump. It’s _probable_ that it’s CDCR, but > take a look at the thread dump to see if you have lots of > threads that are running. Any by “lots” here, I mean 100s of threads > that reference the same component, in this case that have cdcr in > the stack trace. > > CDCR is not getting active work at this point, you might want to > consider another replication strategy if you’re not willing to fix > the code. > > Best, > Erick > >> On Mar 29, 2020, at 4:17 AM, Raji N wrote: >> >> Hi All, >> >> We running solrcloud 7.6 (with the patch # >> > >>> >> https://issues.apache.org/jira/secure/attachment/12969150)/SOLR-11724.patchon >> production on 7 hosts in containers. The container memory is 48GB , >>> heap >> is 24GB. >> ulimit -v >> >> unlimited >> >> ulimit -m >> >> unlimited >> We don't have any custom code in solr. We have set up bidirectional >>> CDCR >> between primary and secondary Datacenter. Our secondary DC is very > unstable >> and many times many instances are down. >> >> We get below exception quite often. Is this because the CDCR >> connection > is >> broken. >> >> WARN (cdcr-update-log-synchronizer-80-thread-1) [ ] >> o.a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception >> >> java.lang.OutOfMemoryError: unable to create new native thread >> >> at java.lang.Thread.start0(Native Method) ~[?:1.8.0_211] >> >> at java.lang.Thread.start(Thread.java:717) >> ~[?:1.8.0_211] >> >> at >> > >>> >> org.apache.http.impl.client.IdleConnectionEvictor.start(IdleConnectionEvictor.java:96) >> ~[httpclient-4.5.3.jar:4.5.3] >> >> at >> > >>> >> org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1219) >> ~[httpclient-4.5.3.jar:4.5.3] >> >> at >> > >>> >> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:319) >> ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f >> - nknize - 2018-12-07 14:47:53] >> >> at >> > >>> >> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330) >> ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f >> - nknize - 2018-12-07 14:47:53] >> >> at >> > >>> >> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268) >> ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f >> - nknize - 201
Required operator (+) is being ignored when using default conjunction operator AND
Using solr 8.3.0 it seems like required operator isn't functioning properly
when default conjunction operator is AND.
Steps to reproduce:
20 docs
all have text field
17 have the value A
13 have the value B
10 have both A and B (the intersection)
===the data===
[
{
"id": "0",
"_text_": [
"abc",
"123",
"xyz"
]
},
{
"id": "1",
"_text_": [
"abc",
"123",
"xyz"
]
},
{
"id": "2",
"_text_": [
"abc",
"123",
"xyz"
]
},
{
"id": "3",
"_text_": [
"abc",
"123"
]
},
{
"id": "4",
"_text_": [
"abc",
"123"
]
},
{
"id": "5",
"_text_": [
"abc",
"123",
"xyz"
]
},
{
"id": "6",
"_text_": [
"abc",
"123"
]
},
{
"id": "7",
"_text_": [
"abc",
"123"
]
},
{
"id": "8",
"_text_": [
"abc",
"123",
"xyz"
]
},
{
"id": "9",
"_text_": [
"abc",
"123"
]
},
{
"id": "10",
"_text_": [
"abc",
"xyz"
]
},
{
"id": "11",
"_text_": [
"abc"
]
},
{
"id": "12",
"_text_": [
"abc",
"xyz"
]
},
{
"id": "13",
"_text_": [
"abc",
"xyz"
]
},
{
"id": "14",
"_text_": [
"abc",
"xyz"
]
},
{
"id": "15",
"_text_": [
"abc",
"xyz"
]
},
{
"id": "16",
"_text_": [
"abc",
"xyz"
]
},
{
"id": "17",
"_text_": [
"xyz",
"123"
]
},
{
"id": "18",
"_text_": [
"def",
"123",
"xyz"
]
},
{
"id": "19",
"_text_": [
"def",
"123"
]
}
]
==
default operator is set to AND
my query is:
http://localhost:8983/solr/new_core/select?debug.explain.structured=true&debugQuery=on&q=%7B!q.op%3DAND%7D%20%2Babc%20OR%20123&rows=20
the response:
{
"responseHeader":{
"status":0,
"QTime":7,
"params":{
"q":"{!q.op=AND} +abc OR 123",
"rows":"20",
"debug.explain.structured":"true",
"debugQuery":"on"}},
"response":{"numFound":20,"start":0,"docs":[
{
"id":"3",
"_version_":1662786291343818752},
{
"id":"4",
"_version_":1662786291343818753},
{
"id":"6",
"_version_":1662786291344867329},
{
"id":"7",
"_version_":1662786291345915904},
{
"id":"9",
"_version_":1662786291346964480},
{
"id":"0",
"_version_":1662786291339624448},
{
"id":"1",
"_version_":1662786291342770176},
{
"id":"2",
"_version_":1662786291342770177},
{
"id":"5",
"_version_":1662786291344867328},
{
"id":"8",
"_version_":1662786291345915905},
{
"id":"17",
"_version_":1662786291350110209},
{
"id":"19",
"_version_":1662786291351158784},
{
"id":"18",
"_version_":1662786291350110210},
{
"id":"11",
"_version_":1662786291348013056},
{
"id":"10",
"_version_":1662786291346964481},
{
"id":"12",
"_version_":1662786291348013057},
{
"id":"13",
"_version_":1662786291348013058},
{
"id":"14",
"_version_":1662786291349061632},
{
"id":"15",
"_version_":1662786291349061633},
{
"id":"16",
"_version_":1662786291350110208}]
},
"debug":{
"rawquerystring":"{!q.op=AND} +abc OR 123",
"querystring":"{!q.op=AND} +abc OR 123",
"parsedquery":"_text_:abc _text_:123",
"parsedquery_toString":"_text_:abc _text_:123",
"explain":{
"3":{
"match":true,
"value":0.29721633,
"description":"sum of:",
"details":[{
"match":true,
"value":0.08681979,
"description":"weight(_text_:abc in 3) [SchemaSimilarity],
result of:",
"details":[{
"match":true,
"value":0.08681979,
"description":"score(freq=1.0), computed as boost * idf *
tf from:"
Re: Required operator (+) is being ignored when using default conjunction operator AND
: Using solr 8.3.0 it seems like required operator isn't functioning properly : when default conjunction operator is AND. You're mixing the "prefix operators" with the "infix operators" which is always a recipe for disaster. The use of q.op=AND vs q.op=OR in these examples only complicates the issue because q.op isn't really overriding any sort of implicit "infix operator" when clauses exist w/o an infix operator between them, it is overriding the implicit MUST/SHOULD/MUST_NOT given to each clause as parsed ... but in general setting q.op-AND really only makes sense when you expect/intend to only be using "infix operators" This write up i did several years ago is still very accurate -- the bottom line is you REALLY don't want to mix infix and prefix operators.. https://lucidworks.com/post/why-not-and-or-and-not/ ...because the results of mixing them really only "make sense" given the context that the parser goes left to right (ie: no precedence) and has no explicit "prefix" operator syntax for "SHOULD" -Hoss http://www.lucidworks.com/
SolrCloud 8.2.0 - adding a field
Hi All - I'm trying this:
curl -X POST -H 'Content-type:application/json' --data-binary
'{"add-field":{"name":"Para450","type":"text_general","stored":"false","indexed":"true","docValues":"false","multiValued":"false"}}'
http://ursula.querymasters.com:9100/api/cores/UNCLASS/schema
This results in:
{
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"no core retrieved for UNCLASS",
"code":404}}
I've also tried going to api/c:
curl -X POST -H 'Content-type:application/json' --data-binary
'{"add-field":{"name":"Para450","type":"text_general","stored":"false","indexed":"true","docValues":"false","multiValued":"false"}}'
http://ursula.querymasters.com:9100/api/c/UNCLASS/schema
results in:
{
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"no such collection or alias",
"code":400}}
What am I doing wrong? The schema UNCLASS does exist in Zookeeper.
Thanks!
-Joe
Re: SolrCloud 8.2.0 - adding a field
Nevermind - I see that I need to specify an existing collection not a
schema. There is no collection called UNCLASS - only a schema.
-Joe
On 4/1/2020 4:52 PM, Joe Obernberger wrote:
Hi All - I'm trying this:
curl -X POST -H 'Content-type:application/json' --data-binary
'{"add-field":{"name":"Para450","type":"text_general","stored":"false","indexed":"true","docValues":"false","multiValued":"false"}}'
http://ursula.querymasters.com:9100/api/cores/UNCLASS/schema
This results in:
{
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"no core retrieved for UNCLASS",
"code":404}}
I've also tried going to api/c:
curl -X POST -H 'Content-type:application/json' --data-binary
'{"add-field":{"name":"Para450","type":"text_general","stored":"false","indexed":"true","docValues":"false","multiValued":"false"}}'
http://ursula.querymasters.com:9100/api/c/UNCLASS/schema
results in:
{
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"no such collection or alias",
"code":400}}
What am I doing wrong? The schema UNCLASS does exist in Zookeeper.
Thanks!
-Joe
