[
https://issues.apache.org/jira/browse/SOLR-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999472#comment-16999472
]
Andy Vuong commented on SOLR-14044:
-----------------------------------
Our current shared store deletion path involves the use of an asynchronous
process manager, BlobDeleteManager, in which we enqueue eligible files for
deletion and this has been scoped to one deletion flow, which is the
“activation” and “deactivation” of segment files as indexing progresses locally
on a node.
More concretely: in the indexing path of a shared collection, we resolve the
differences between our local index and source of truth (blob) and after we’re
synced, track files missing locally (were formerly present and then likely
deleted locally as indexing happens) to be deleted.
Eligible files to be deleted refer to the files that are being “tracked”, and
have aged beyond a deletion threshold. That threshold is currently 30 seconds.
A good heuristic hasn’t been explored yet but we haven’t seen an issue with
this value so far in our load tests.
BlobDeleteManager uses a thread pool of 5 deleter threads and a BlockingQueue,
allowing up to 200 deletion tasks at a time.
> Support shard/collection deletion in shared storage
> ---------------------------------------------------
>
> Key: SOLR-14044
> URL: https://issues.apache.org/jira/browse/SOLR-14044
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Andy Vuong
> Priority: Major
>
> The Solr Cloud deletion APIs for collections and shards are not currently
> supported by shared storage but are an essential functionality required by
> the shared storage design. Deletion of objects from shared storage currently
> only happens in the indexing path (on pushes) and after the index file
> listings between the local solr process and external store have been resolved.
>
> This task is to track supporting the delete shard/collection API commands and
> its scope does not include cleaning up so called “orphaned” index files from
> blob (i.e. files that are no longer referenced by any core.metadata file on
> the external store). This will be designed/covered in another subtask.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
