Right, every individual shard is limited to 2B records. That does include deleted docs. But I've never seen a shard (a Lucene index actually) perform satisfactorily at that scale so while this is a limit people usually add shards long before that.
There is no technical reason to optimize every time, normal segment merging will eventually remove the data associated with deleted documents. You'll carry forward a number of deleted docs, but I usually see it stabilize around 10%-15%. You don't necessarily have to re-index, you can split existing shards. But from your e-mail, it looks like you think you have to do something explicit to reclaim the resources associated with deleted documents. You do not have to do this. Optimize is really a special heavyweight merge. Normal merging happens when you do a commit and that process also reclaims the deleted document resources. Best, Erick On Thu, Jul 6, 2017 at 11:59 AM, <calamita.agost...@libero.it> wrote: > Hi, > > I'm working on an application that index CDR ( Call Detail Record ) in > SolrCloud with 1 collection and 3 shards. > > Every day the application index 30 millions of CDR. > > I have a purge application that delete records older than 10 days, and call > OPTIMIZE, so the collection will keep only 300 millions of CDR. > > Do you know if there is a limit on max number of documents per shard , > included deleted documents ? > > I read in some blogs that there is a limit of 2 Billions per shard included > deleted documents, that is I can have an empty collection, but if I already > indexed 6 Billions of CDR ( 2 per 3 shards ) in that collection, I'll get an > error. Is it true ? Have I recreate the collection ? > > I see that when I call delete records, apache solr free space on disk. > > Thanks. > > Agostino > > >
