[
https://issues.apache.org/jira/browse/SOLR-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110552#comment-17110552
]
ASF subversion and git services commented on SOLR-13101:
--------------------------------------------------------
Commit ad3f0a30181aa4d7ba099ee67fd45fa98cf9c2c0 in lucene-solr's branch
refs/heads/jira/SOLR-13101 from Bilal Waheed
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ad3f0a3 ]
SOLR-13101: SHARED replica's distributed indexing (#1430)
* SOLR-13101: SHARED replica's distributed indexing
The basic purpose of this change was to refactor most of the SHARED replica
logic out of DistributedZkUpdateProcessor. Along with that refactoring I came
across couple of functional issues in the logic that have been fixed too.
Functional fixes:
-If a replica looses its leadership in the middle of indexing batch, it should
still push its changes to the shared store.
-SHARED replica does not need to process soft commits and does not need to
broadcast hard commits to all the shards of a collection.
-Previously we would pull from the shared store even when the doc being
added/deleted is not meant for the current core/shard.
-Previously DistributedZkUpdateProcessor#processDelete was running its pull
logic even before the request had been setup
(DistributedZkUpdateProcessor#setupRequest).
-DistributedZkUpdateProcessorTest have been deleted in favor of new
SharedStoreDistributedIndexingTest SharedCoreIndexingBatchProcessorTest.
Refactoring:
-Most of DistributedZkUpdateProcessor's SHARED replica logic is moved into a
new SharedCoreIndexingBatchProcessor. The purpose of this class is to pull from
the shared store at the start of an indexing batch (if the core is stale) and
push to the shared store at the end of a successfully committed indexing batch.
-CoreUpdateTracker has been deleted and its only persistShardIndexToSharedStore
method has been renamed to pushCoreToSharedStore and moved to CorePusher.
-BlobStoreUtilsTest#syncLocalCoreWithSharedStore is renamed to
pullCoreFromSharedStore and moved into a new CorePuller class and the tests to
CorePullerTests.
-I did rename phrase "blob store" to "shared store" at some places in the
changed classes. But it was not meant to be an exhaustive attempt.
* -Throw error for pull request for an unknown.
-Log warning when indexing a non-active shard.
* Address CR feedback.
> Shared storage support in SolrCloud
> -----------------------------------
>
> Key: SOLR-13101
> URL: https://issues.apache.org/jira/browse/SOLR-13101
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
> Reporter: Yonik Seeley
> Priority: Major
> Time Spent: 15h 40m
> Remaining Estimate: 0h
>
> Solr should have first-class support for shared storage (blob/object stores
> like S3, google cloud storage, etc. and shared filesystems like HDFS, NFS,
> etc).
> The key component will likely be a new replica type for shared storage. It
> would have many of the benefits of the current "pull" replicas (not indexing
> on all replicas, all shards identical with no shards getting out-of-sync,
> etc), but would have additional benefits:
> - Any shard could become leader (the blob store always has the index)
> - Better elasticity scaling down
> - durability not linked to number of replcias.. a single replica could be
> common for write workloads
> - could drop to 0 replicas for a shard when not needed (blob store always
> has index)
> - Allow for higher performance write workloads by skipping the transaction
> log
> - don't pay for what you don't need
> - a commit will be necessary to flush to stable storage (blob store)
> - A lot of the complexity and failure modes go away
> An additional component a Directory implementation that will work well with
> blob stores. We probably want one that treats local disk as a cache since
> the latency to remote storage is so large. I think there are still some
> "locking" issues to be solved here (ensuring that more than one writer to the
> same index won't corrupt it). This should probably be pulled out into a
> different JIRA issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
