(pinot) branch master updated: Add segment metadata deletion code to segment deletion path (#12350)

2024-02-01 Thread snlee
This is an automated email from the ASF dual-hosted git repository. snlee pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 041e04078f Add segment metadata deletion code to s

Re: [PR] Add segment metadata deletion code to segment deletion path [pinot]

2024-02-01 Thread via GitHub
snleee merged PR #12350: URL: https://github.com/apache/pinot/pull/12350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

Re: [PR] Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction [pinot]

2024-02-01 Thread via GitHub
vvivekiyer commented on code in PR #12347: URL: https://github.com/apache/pinot/pull/12347#discussion_r1475548266 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawThetaSketchAggregationFunction.java: ## @@ -53,7 +53,8 @@ public String

Re: [PR] Detect expired messages in Kafka. Log and set a guage. [pinot]

2024-02-01 Thread via GitHub
vrajat commented on code in PR #12157: URL: https://github.com/apache/pinot/pull/12157#discussion_r1475535154 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -140,6 +142,8 @@ public class PinotLLCRealti

Re: [I] [multistage] attach default limit 10 to queries that doesn't have a limit [pinot]

2024-02-01 Thread via GitHub
walterddr commented on issue #12304: URL: https://github.com/apache/pinot/issues/12304#issuecomment-1922815466 Hi @dttung2905 thank you for the interest. yes this is indeed a relatively good beginner task to get to know the pinot codebase. a bit of a explanation: the code you

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on PR #12346: URL: https://github.com/apache/pinot/pull/12346#issuecomment-1922809960 @Jackie-Jiang @tibrewalpratik17 Please take a final look on this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475530240 ## pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/ValidDocIdMetadataInfo.java: ## @@ -28,14 +28,19 @@ public class ValidDocIdMetadataInfo { priv

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475530142 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: ##

[I] `ISO_8859_1` breaking Hangul in CLP `_logtype` column [pinot]

2024-02-01 Thread via GitHub
intr3p1d opened a new issue, #12352: URL: https://github.com/apache/pinot/issues/12352 # Cause `clp-ffi-java` [internally use](https://github.com/y-scope/clp-ffi-java/blob/c4a74dbdeb09bd4e7e3d119826dddbe5005ccf53/src/main/java/com/yscope/clp/compressorfrontend/EncodedMessage.java#L30-

Re: [PR] Add metrics for no-HA segments monitoring [pinot]

2024-02-01 Thread via GitHub
chenboat commented on PR #12336: URL: https://github.com/apache/pinot/pull/12336#issuecomment-1922689613 > > the approach looks good. i was wondering if we should have some definitions. e.g. when nReplica equals the following, how should we call each status. HighAvailability is not a very w

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475383825 ## pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/ValidDocIdsBitmapResponse.java: ## @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Softwar

Re: [I] [multistage] attach default limit 10 to queries that doesn't have a limit [pinot]

2024-02-01 Thread via GitHub
dttung2905 commented on issue #12304: URL: https://github.com/apache/pinot/issues/12304#issuecomment-1922563716 Hi @walterddr, I'm just following the issue and I think applying the limit by default is a really good idea too. Although I'm quite new to the pinot code base, I think the

Re: [PR] Added pinot-error-code header in query response [pinot]

2024-02-01 Thread via GitHub
walterddr commented on code in PR #12338: URL: https://github.com/apache/pinot/pull/12338#discussion_r1475351199 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java: ## @@ -348,4 +349,25 @@ private static HttpRequesterIdentity makeHttpIde

Re: [PR] [bugfix] fix getBigDecimal() scale throwing rounding error [pinot]

2024-02-01 Thread via GitHub
walterddr merged PR #12326: URL: https://github.com/apache/pinot/pull/12326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

(pinot) branch master updated (33074e1e7e -> 0a4398634b)

2024-02-01 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 33074e1e7e Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED (#12351) add 0a4398634b

Re: [PR] [multistage][test] adding tests for IN/NOT-IN operation [pinot]

2024-02-01 Thread via GitHub
walterddr commented on code in PR #12349: URL: https://github.com/apache/pinot/pull/12349#discussion_r1475300114 ## pinot-query-runtime/src/test/resources/queries/FilterAggregates.json: ## @@ -75,6 +75,9 @@ }, { "sql": "SELECT bool_col, COALESCE(min(double

Re: [PR] Fix partition handling by always using string values [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on PR #12115: URL: https://github.com/apache/pinot/pull/12115#issuecomment-1922445010 @jackjlli Thanks for the suggestion! Going forward, will add a section to list all the public API changes in the PR description. The backward incompatible changes (non API changes)

Re: [I] Everytime I add a table [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on issue #9840: URL: https://github.com/apache/pinot/issues/9840#issuecomment-1922434936 @KKcorps Can you help address the above request? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12227: URL: https://github.com/apache/pinot/pull/12227#discussion_r1475228145 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java: ## @@ -0,0 +1,368 @@ +/** + * Licensed to

(pinot) branch master updated: Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED (#12351)

2024-02-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 33074e1e7e Work around the problem of Helix sendi

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12351: URL: https://github.com/apache/pinot/pull/12351#discussion_r1475219741 ## pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java: ## @@ -46,6 +50,13 @@ * 3. Delete an existed segm

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang merged PR #12351: URL: https://github.com/apache/pinot/pull/12351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475204057 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475202125 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475202125 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475202125 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475197414 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: ##

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475187328 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475184076 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java:

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
klsince commented on code in PR #12351: URL: https://github.com/apache/pinot/pull/12351#discussion_r1475184165 ## pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java: ## @@ -46,6 +50,13 @@ * 3. Delete an existed segment.

Re: [PR] Fluent test framework [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12215: URL: https://github.com/apache/pinot/pull/12215#discussion_r1475168985 ## pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/AbstractAggregationFunctionTest.java: ## @@ -0,0 +1,121 @@ +/** + * Licensed to the Ap

Re: [I] Pinot hitting stuck partitions while consuming transactionally [pinot]

2024-02-01 Thread via GitHub
dang-stripe commented on issue #12318: URL: https://github.com/apache/pinot/issues/12318#issuecomment-1922264881 Thanks @snleee. I think increasing the fetch timeout or pausing + resume is a reasonable workaround. I don't see anything in the Kafka consumer library that'd allow us to g

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
codecov-commenter commented on PR #12351: URL: https://github.com/apache/pinot/pull/12351#issuecomment-1922262814 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on PR #12351: URL: https://github.com/apache/pinot/pull/12351#issuecomment-1922258767 Submitted a Helix issue to verify the observation: https://github.com/apache/helix/issues/2751 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add segment metadata deletion code to segment deletion path [pinot]

2024-02-01 Thread via GitHub
swaminathanmanish commented on code in PR #12350: URL: https://github.com/apache/pinot/pull/12350#discussion_r1475083432 ## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/SegmentDeletionManagerTest.java: ## @@ -357,6 +358,9 @@ public void createTableA

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on PR #12351: URL: https://github.com/apache/pinot/pull/12351#issuecomment-1922135433 Related to #12343 which has more context. cc @gortiz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Add segment metadata deletion code to segment deletion path [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12350: URL: https://github.com/apache/pinot/pull/12350#discussion_r1475077573 ## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/SegmentDeletionManagerTest.java: ## @@ -357,6 +358,9 @@ public void createTableAndSegmentFi

[PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang opened a new pull request, #12351: URL: https://github.com/apache/pinot/pull/12351 Helix might send 2 state transitions when a segment is dropped from a server. This behavior breaks the `onConsumingToDropped()` callback because it is only invoked when processing `CONSUMING -> D

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1475063849 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: ##

Re: [PR] Cleaning replication delay metrics on cleanupMetrics [pinot]

2024-02-01 Thread via GitHub
zhtaoxiang commented on code in PR #12343: URL: https://github.com/apache/pinot/pull/12343#discussion_r1475062186 ## pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java: ## @@ -151,12 +156,14 @@ public void onBecomeDroppedF

Re: [PR] [multistage][test] adding tests for IN/NOT-IN operation [pinot]

2024-02-01 Thread via GitHub
codecov-commenter commented on PR #12349: URL: https://github.com/apache/pinot/pull/12349#issuecomment-1922070292 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12349?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Add segment metadata deletion code to segment deletion path [pinot]

2024-02-01 Thread via GitHub
codecov-commenter commented on PR #12350: URL: https://github.com/apache/pinot/pull/12350#issuecomment-1922068621 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12350?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (4f05fd71ae -> a796baa1b5)

2024-02-01 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments in repository https://gitbox.apache.org/repos/asf/pinot.git omit 4f05fd71ae Release all segments of a table in releaseAndRemoveAllSegments meth

Re: [PR] Add segment metadata deletion code to segment deletion path [pinot]

2024-02-01 Thread via GitHub
swaminathanmanish commented on PR #12350: URL: https://github.com/apache/pinot/pull/12350#issuecomment-1921999009 @snleee - Could you take a look? Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Add segment metadata deletion code to segment deletion path [pinot]

2024-02-01 Thread via GitHub
swaminathanmanish opened a new pull request, #12350: URL: https://github.com/apache/pinot/pull/12350 **Whats in the PR:** Added segment metadata deletion from remote store, in the segment deletion path. **Why its needed:** Through [PR](https://github.com/apache/pinot/pull/10034)

Re: [PR] Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction [pinot]

2024-02-01 Thread via GitHub
codecov-commenter commented on PR #12347: URL: https://github.com/apache/pinot/pull/12347#issuecomment-1921968434 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] [multistage][test] adding tests for IN/NOT-IN operation [pinot]

2024-02-01 Thread via GitHub
walterddr opened a new pull request, #12349: URL: https://github.com/apache/pinot/pull/12349 follow up with #12305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction [pinot]

2024-02-01 Thread via GitHub
jackjlli commented on code in PR #12347: URL: https://github.com/apache/pinot/pull/12347#discussion_r1474880961 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java: ## @@ -1024,7 +1024,8 @@ public Comparab

(pinot) branch fix-DistinctCountThetaSketchAggregationFunction updated (577e5e2c58 -> db75e449ad)

2024-02-01 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch fix-DistinctCountThetaSketchAggregationFunction in repository https://gitbox.apache.org/repos/asf/pinot.git discard 577e5e2c58 Fix backward compatible issue in DistinctCountThetaSketchAggregation

(pinot) branch master updated: Implement IN and NOT_IN filters in FilterOperand class (#12285 fix) (#12305)

2024-02-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 04cf96e03e Implement IN and NOT_IN filters in Fil

Re: [PR] Implement IN and NOT_IN filters in FilterOperand class (#12285 fix) [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang merged PR #12305: URL: https://github.com/apache/pinot/pull/12305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] Multistage query failure when select a CASE/IN expression [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang closed issue #12285: Multistage query failure when select a CASE/IN expression URL: https://github.com/apache/pinot/issues/12285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] MinimizeDataMovement does not work correctly for upsert tables [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on issue #11894: URL: https://github.com/apache/pinot/issues/11894#issuecomment-1921800664 Should be solved with #11953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] MinimizeDataMovement does not work correctly for upsert tables [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang closed issue #11894: MinimizeDataMovement does not work correctly for upsert tables URL: https://github.com/apache/pinot/issues/11894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

(pinot) branch master updated (76379fb86a -> a8b1fa3dce)

2024-02-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 76379fb86a making nonLeaderForTables exhaustive (#12345) add a8b1fa3dce Maintain pool selection for the minimizeDa

Re: [PR] Maintain pool selection for the minimizeDataMovement instance partition assignment strategy [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang merged PR #11953: URL: https://github.com/apache/pinot/pull/11953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch maintain-pool-selection-for-minimizeDataMovement deleted (was a286bd868f)

2024-02-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch maintain-pool-selection-for-minimizeDataMovement in repository https://gitbox.apache.org/repos/asf/pinot.git was a286bd868f Enhance algorithm The revisions that were on this branch are sti

(pinot) branch master updated: making nonLeaderForTables exhaustive (#12345)

2024-02-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 76379fb86a making nonLeaderForTables exhaustive (

Re: [PR] making nonLeaderForTables exhaustive [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang merged PR #12345: URL: https://github.com/apache/pinot/pull/12345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] [bug][multistage] IN/NOT_IN function issue [pinot]

2024-02-01 Thread via GitHub
Jackie-Jiang commented on issue #12035: URL: https://github.com/apache/pinot/issues/12035#issuecomment-1921768562 Find a failing query (`NOT IN` within `FILTER` clause): ``` SET useStarTree = false; SELECT OriginStateName, Dest, count(ArrivalDelayGroups) FILTER ( WHERE OriginStateName

Re: [PR] Added pinot-error-code header in query response [pinot]

2024-02-01 Thread via GitHub
ankitsultana commented on code in PR #12338: URL: https://github.com/apache/pinot/pull/12338#discussion_r1474714391 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java: ## @@ -348,4 +349,25 @@ private static HttpRequesterIdentity makeHttp

Re: [PR] Detect expired messages in Kafka. Log and set a guage. [pinot]

2024-02-01 Thread via GitHub
ege-st commented on code in PR #12157: URL: https://github.com/apache/pinot/pull/12157#discussion_r1474666166 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -140,6 +142,8 @@ public class PinotLLCRealti

Re: [PR] Enhancing metadata API to return upsert partition to primary key count map for both controller and server APIs [pinot]

2024-02-01 Thread via GitHub
ege-st commented on code in PR #12334: URL: https://github.com/apache/pinot/pull/12334#discussion_r1474617679 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/models/DummyTableUpsertMetadataManager.java: ## @@ -72,6 +73,11 @@ public PartitionUpsertMeta

Re: [PR] Bump com.google.api.grpc:proto-google-iam-v1 from 1.0.0 to 1.28.0 [pinot]

2024-02-01 Thread via GitHub
codecov-commenter commented on PR #12348: URL: https://github.com/apache/pinot/pull/12348#issuecomment-1921193849 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12348?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

(pinot) branch dependabot/maven/com.google.api.grpc-proto-google-iam-v1-1.28.0 created (now 2e9a103cfa)

2024-02-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.api.grpc-proto-google-iam-v1-1.28.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 2e9a103cfa Bump com.google.api.grpc:proto-google-iam-v1 f

[PR] Bump com.google.api.grpc:proto-google-iam-v1 from 1.0.0 to 1.28.0 [pinot]

2024-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12348: URL: https://github.com/apache/pinot/pull/12348 Bumps [com.google.api.grpc:proto-google-iam-v1](https://github.com/googleapis/sdk-platform-java) from 1.0.0 to 1.28.0. Release notes Sourced from https://github.com/googleapis/sdk-pl

(pinot) branch dependabot/maven/com.google.api.grpc-proto-google-iam-v1-1.27.0 deleted (was dc81f8da5d)

2024-02-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.api.grpc-proto-google-iam-v1-1.27.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was dc81f8da5d Bump com.google.api.grpc:proto-google-iam-v1 f

Re: [PR] Bump com.google.api.grpc:proto-google-iam-v1 from 1.0.0 to 1.27.0 [pinot]

2024-02-01 Thread via GitHub
dependabot[bot] commented on PR #12321: URL: https://github.com/apache/pinot/pull/12321#issuecomment-1921086737 Superseded by #12348. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Bump com.google.api.grpc:proto-google-iam-v1 from 1.0.0 to 1.27.0 [pinot]

2024-02-01 Thread via GitHub
dependabot[bot] closed pull request #12321: Bump com.google.api.grpc:proto-google-iam-v1 from 1.0.0 to 1.27.0 URL: https://github.com/apache/pinot/pull/12321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] making nonLeaderForTables exhaustive [pinot]

2024-02-01 Thread via GitHub
gortiz commented on code in PR #12345: URL: https://github.com/apache/pinot/pull/12345#discussion_r1474244625 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java: ## @@ -63,29 +69,25 @@ protected final void runTask(Pr

Re: [PR] making nonLeaderForTables exhaustive [pinot]

2024-02-01 Thread via GitHub
gortiz commented on PR #12345: URL: https://github.com/apache/pinot/pull/12345#issuecomment-1921052465 I've applied changes suggested by @Jackie-Jiang and also modified the javadoc to be more specific. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] making nonLeaderForTables exhaustive [pinot]

2024-02-01 Thread via GitHub
gortiz commented on code in PR #12345: URL: https://github.com/apache/pinot/pull/12345#discussion_r1474244132 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java: ## @@ -63,29 +69,25 @@ protected final void runTask(Pr

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1474232231 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: ##

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1474232231 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: ##

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1474232231 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: ##

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1474230137 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java: ## @@ -465,10 +466,64 @@ public Response downloadSegment( } } + @GET

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-01 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1474229548 ## pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/ValidDocIdMetadataInfo.java: ## @@ -28,14 +28,16 @@ public class ValidDocIdMetadataInfo { priv

Re: [PR] Cleaning replication delay metrics on cleanupMetrics [pinot]

2024-02-01 Thread via GitHub
gortiz commented on PR #12343: URL: https://github.com/apache/pinot/pull/12343#issuecomment-1920959709 I've changed the code to call `RealtimeTableDataManager.onConsumingToDropped()` from `SegmentOnlineOfflineStateModelFactory.onBecomeOfflineFromConsuming`, which seems closer to what Jacki

Re: [PR] making nonLeaderForTables exhaustive [pinot]

2024-02-01 Thread via GitHub
gortiz commented on code in PR #12345: URL: https://github.com/apache/pinot/pull/12345#discussion_r1474102380 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTask.java: ## @@ -63,29 +69,25 @@ protected final void runTask(Pr

Re: [PR] Cleaning replication delay metrics on cleanupMetrics [pinot]

2024-02-01 Thread via GitHub
gortiz commented on PR #12343: URL: https://github.com/apache/pinot/pull/12343#issuecomment-1920737404 > When segment is dropped from server (due to rebalance or segment removal), TableDataManager.onConsumingToDropped() will be invoked via the helix state transition from CONSUMING to DROPPE

Re: [PR] Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction [pinot]

2024-02-01 Thread via GitHub
vvivekiyer commented on code in PR #12347: URL: https://github.com/apache/pinot/pull/12347#discussion_r1473959548 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java: ## @@ -1024,7 +1024,8 @@ public Compar