Jackie-Jiang commented on code in PR #9062:
URL: https://github.com/apache/pinot/pull/9062#discussion_r934993468
##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java:
##########
@@ -55,12 +55,16 @@ public enum Strategy {
@JsonPropertyDescription("Column for upsert comparison, default to time
column")
private final String _comparisonColumn;
+ @JsonPropertyDescription("whether use snapshot for fast upsert metadata
recovery")
+ private boolean _useSnapshot;
+
@JsonCreator
public UpsertConfig(@JsonProperty(value = "mode", required = true) Mode mode,
@JsonProperty("partialUpsertStrategies") @Nullable Map<String, Strategy>
partialUpsertStrategies,
@JsonProperty("defaultPartialUpsertStrategy") @Nullable Strategy
defaultPartialUpsertStrategy,
@JsonProperty("comparisonColumn") @Nullable String comparisonColumn,
- @JsonProperty("hashFunction") @Nullable HashFunction hashFunction) {
+ @JsonProperty("hashFunction") @Nullable HashFunction hashFunction,
+ @JsonProperty("useSnapshot") @Nullable boolean useSnapshot) {
Review Comment:
`enableSnapshot` might be better
##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java:
##########
@@ -55,12 +55,16 @@ public enum Strategy {
@JsonPropertyDescription("Column for upsert comparison, default to time
column")
private final String _comparisonColumn;
+ @JsonPropertyDescription("whether use snapshot for fast upsert metadata
recovery")
+ private boolean _useSnapshot;
+
@JsonCreator
public UpsertConfig(@JsonProperty(value = "mode", required = true) Mode mode,
Review Comment:
We already got too many keys in this config. Suggest making a PR similar to
#8743 to clean them up first.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java:
##########
@@ -213,6 +215,16 @@ public List<StarTreeV2> getStarTrees() {
return _starTreeIndexContainer != null ?
_starTreeIndexContainer.getStarTrees() : null;
}
+ @Nullable
+ public void setValidDocSnapshots(ImmutableRoaringBitmap validDocSnapshots) {
Review Comment:
I feel a easier way to manage the snapshot is to add 3 methods into this
class:
- `@Nullable ImmutableRoaringBitmap loadValidDocIdsSnapshot()`
- `void persistValidDocIdsSnapshot()`
- `void deleteValidDocIdsSnapshot()`
One scenario we want to handle is that when snapshot is enabled then
disabled, we should be able to delete the existing snapshots.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java:
##########
@@ -213,6 +215,16 @@ public List<StarTreeV2> getStarTrees() {
return _starTreeIndexContainer != null ?
_starTreeIndexContainer.getStarTrees() : null;
}
+ @Nullable
+ public void setValidDocSnapshots(ImmutableRoaringBitmap validDocSnapshots) {
+ _validDocSnapshots = validDocSnapshots;
+ }
+
+ @Nullable
+ public ImmutableRoaringBitmap getValidDocSnapshots() {
Review Comment:
(minor) Suggest renaming it to `getValidDocIdsSnapshot()`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SegmentMetadataImpl.java:
##########
@@ -403,6 +408,10 @@ public Map<String, ColumnMetadata> getColumnMetadataMap() {
return _columnMetadataMap;
}
+ public boolean isUpsertSnapshotEnabled() {
Review Comment:
We should not rely on segment metadata to determine whether snapshot is
enabled. The upsert config might change anytime, and we want to use/drop
snapshot accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
