CDCR - how to deal with the transaction log files

Once enabled CDCR, update log stores an unlimited number of entries. This is causing the tlog folder getting bigger and bigger, as well as the open files are growing. How can one reduce the number of open files and also to reduce the tlog files? If it’s not taken care properly, sooner or later t

Re: Slowly running OOM due to Query instances?!

Around 300MB will be spent for your filterCache and query cache (taking avg size for query string.. https://teaspoon-consulting.com/articles/solr-cache-tuning.html). So during continous indexing and queries (complex) your cache thus heap utilization may go up. On Fri, Jul 7, 2017 at 9:41 AM, Mark

Re: index new discovered fileds of different types

Thaer Whoa, hold everything! You said RDF, meaning resource description framework? If so, you have exactly​ three fields: subject, predicate, and object. Maybe they are text type, or for exact matches you might want string fields. Add an ID field, which could be automatically generated by Solr,

Re: Slowly running OOM due to Query instances?!

I suspect this is related to caching in some way, possibly one of the following: 1) You have large queries in a large cache. 2) Your custom query parsers have a bug that is causing a leak with the caches. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jul 7, 2017 at 9:41 AM, Markus Jelsma

Re: Max document per shard ( included deleted documents )

The deleted records will be automatically cleaned up in the background. You don’t have to do anything. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Jul 7, 2017, at 1:25 PM, calamita.agost...@libero.it wrote: > > > Sorry , I know that size is f

Re[2]: Re[2]: Re[2]: Max document per shard ( included deleted documents )

Sorry , I  know that size is for shard and not for collection. My doubt  is: if every day I insert 10M documents in a shard and delete 10M of documents (the old ones  ) after  20 days I have to add a new shard or not ? Number of undeleted documents is always the same. ( 100M  for example ) Than

Re: Re[2]: Re[2]: Max document per shard ( included deleted documents )

You seem to be confusing shards with collections. You can have 100 shards each with 100M documents for a total of 10B documents in the _collection_, but no individual shard has more than 100M docs. Best, Erick On Fri, Jul 7, 2017 at 10:02 AM, wrote: > > Ok. I will never have more than 100

Re[2]: Re[2]: Max document per shard ( included deleted documents )

Ok. I will  never  have  more than  100 Million of document per shard in the same time, because I delete old  documents every  night To keep last  10 days   I don't understand  if I have add shards after months  of indexing  ( insert  and delete can reach 2B after a few  months  ) or  leave the

Re: Re[2]: Max document per shard ( included deleted documents )

Stop.. 2 billion is _per shard_ not per collection. You'll probably never have that many in practice as the search performance would be pretty iffy. Every filterCache entry would occupy up to .25G for instance. So just don't expect to fit 2B docs per shard unless you've tested the heck out of it an

Re: uploading solr.xml to zk

Actually it is corrected in the latest docs... On Fri, Jul 7, 2017 at 9:35 AM, Erick Erickson wrote: > Blast, you're right that's a doc problem. I'll change the current docs > but I'm afraid that'll live on in older docs. > > It should be: > > bin/solr zk cp blah blah > > (note the "zk" bit) > >

Re: uploading solr.xml to zk

Blast, you're right that's a doc problem. I'll change the current docs but I'm afraid that'll live on in older docs. It should be: bin/solr zk cp blah blah (note the "zk" bit) Sorry about that. On Fri, Jul 7, 2017 at 12:31 PM, wrote: > The documentation says > > If you for example would like

Re: index new discovered fileds of different types

I'd recommend "managed schema" rather than schemaless. They're related but distinct. The problem is that schemaless makes assumptions based on the first field it finds. So if it finds a field with a "1" in it, it guesses "int". That'll break if the next doc has a 1.0 since it doesn't parse to an i

Re: uploading solr.xml to zk

Not quite right. That should be: (note the ZK in the command). I'm surprised that you didn't get an error message if that's the exact command. solr-6.4.1/bin/solr zk cp file:/home/user1/solr/nodes/day1/solr/solr.xml zk:/solr.xml -z localhost:9983 Second possibility is that you have a chroot in t

Re: behind nginx

Hi Eric, with tomcat probe and jmeter i do test load and i see there is no session and the memory can easily recovered where gc occured, so there is no worry, And as he has said eric "don't chnge any thing" -- View this message in context: http://lucene.472066.n3.nabble.com/behind-nginx-tp4344

[ANNOUNCE] [SECURITY] CVE-2017-7660: Security Vulnerability in secure inter-node communication in Apache Solr

CVE-2017-7660: Security Vulnerability in secure inter-node communication in Apache Solr Severity: Important Vendor: The Apache Software Foundation Versions Affected: Solr 5.3 to 5.5.4 Solr 6.0 to 6.5.1 Description: Solr uses a PKI based mechanism to secure inter-node communication when securit

RE: Slowly running OOM due to Query instances?!

Hello, Sorry, i forgot to mention we already run 6.6. Now i am looking at the sampler again on a freshly restarted instance. This node has almost 6000 uncollectable TermQuery instances at this moment. There are just 3.4 queries per second on this node. The final query is complex but i cannot

RE: Slowly running OOM due to Query instances?!

Hello, There has not been a change in code recently. An increase in QPS and a broader spectrum of queries (a world wide network of users started using our platform) happened a month ago. All still happy with 400 MB, nothing changed. The index does increase continously, all the time so i did exp

Re: Slowly running OOM due to Query instances?!

With generated Query’s, one has to be really careful with .equals and .hashCode implementations. That may not be applicable here, but something that has bitten me with caching. Note that there were fixes made in Solr 6.6 with PayloadScoreQuery in this regard. See LUCENE-7808 and LUCENE-7481

Re: Slowly running OOM due to Query instances?!

What changed in the system? Has there been a code change, increased QPS or different types of queries being run? Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jul 7, 2017 at 8:07 AM, Markus Jelsma wrote: > Hello, > > No, 800 MB heap is fine. Up till now, heap consumption averages at 4

Re: Network segmentation of replica

Anyone that might be able to test this, or already did, please help clarify.. On Fri, 7 Jul 2017 at 00:42, Dave wrote: > Sorry that should have read have not tested in solr cloud. > > > On Jul 6, 2017, at 6:37 PM, Dave wrote: > > > > I have tested that out in solr cloud, but for solr master sla

RE: Slowly running OOM due to Query instances?!

Hello, No, 800 MB heap is fine. Up till now, heap consumption averages at 400 MB for past few months, which is fine. FilterCache and QueryCache are always full. If restarted, they fill up very quickly. FilterCache is just 512 entries large, QueryCache about 2k. All Solr instances run on separa

Re: Slowly running OOM due to Query instances?!

Xms 800m sounds low regardless did you know how much total cache consumption may go based on your current solrconfig.xml settings. Also 2 shards and 3 replca's are on 6 such machines i assume. Thanks, Susheel On Fri, Jul 7, 2017 at 7:01 AM, Markus Jelsma wrote: > Hello, > > This morning i spott

Slowly running OOM due to Query instances?!

Hello, This morning i spotted our QTime suddenly go up. This has been going on for a few hours by now and coincides with a serious increase in heap consumption. No node ran out of memory so far but either that is going to happen soon, or the nodes become unusable in another manner. I restarted

Help with updateHandler commit stats

Hi, I'm taking a look to UpdateHandler stats... and i see when autosoftcommit occurs (every 10 secs) both metrics, "commits" and "soft autocommits" increments by one. ¿is this normal? My config is: autoCommit: 180 secs autoSoftCommit: 10 secs Thanks!

RE: uploading solr.xml to zk

Thanks for the reply This is the exact command on a RHEL 6 machine solr-6.4.1/bin/solr cp file:/home/user1/solr/nodes/day1/solr/solr.xml zk:/solr.xml -z localhost:9983 I am following the documentation of 6.4.1 I am assuming if the solr.xml is present in zookeeper, we can point to an empty di

Re: index new discovered fileds of different types

Hi Jan, Thanks!, I am exploring the schemaless option based on Furkan suggestion. I need the the flexibility because not all fields are known. We get the data from RDF database (which changes continuously). To be more specific, we have a database and all changes on it are sent to a kafka queue. an

Re: uploading solr.xml to zk

> ERROR: cp is not a valid command! Can you write the exact command you typed again? Once solr.xml is in zookeeper, solr will find it automatically. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 7. jul. 2017 kl. 21.31 skrev im...@elogic.pk: > > The documentation s

Re: index new discovered fileds of different types

If you do not need the flexibility of dynamic fields, don’t use them. Sounds to me that you really want a field “price” to be float and a field “birthdate” to be of type date etc. If so, simply create your schema (either manually, through Schema API or using schemaless) up front and index each fi

Re: help on implicit routing

http://lucene.apache.org/solr/guide/6_6/shards-and-indexing-data-in-solrcloud.html -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 6. jul. 2017 kl. 03.15 skrev im...@elogic.pk: > >

Re: Solr 6.5.1 crashing when too many queries with error or high memory usage are queried

You have not told us how many documents you have, how many shards, how big the docs are, physical RAM, Java heap, what typical search latency is etc. If you have tried to squeeze too many docs into a single node it might get overloaded faster, thus sharding would help. If you return too much con

Re: recovery information for replica in recovering state

It is 250gb data. It takes around 40 minutes. And yes, recovery completes correctly. On Thu, 6 Jul 2017 at 23:32 Rick Leir wrote: > Novin, How long is recovery taking for you? I assume the recovery > completes correctly. > Cheers-- Rick > > On July 6, 2017 7:59:03 AM EDT, Novin Novin wrote: > >

RE: help on implicit routing

Thanks that was helpful, can this be done without modifying the document also when posting data through the post utility or a java client? Regards, Imran Sent from Mail for Windows 10 From: Susheel Kumar Sent: Thursday, July 6, 2017 7:52 AM To: solr-user@lucene.apache.org Subject: Re: help on i

uploading solr.xml to zk

The documentation says If you for example would like to keep your solr.xml in ZooKeeper to avoid having to copy it to every node's so lr_home directory, you can push it to ZooKeeper with the bin/solr utility (Unix example): bin/solr cp file:local/file/path/to/solr.xml zk:/solr.xml -z localhost:2