Re: [I] Controller fails to startup if DataDir cannot be created [pinot]

2025-02-25 Thread via GitHub
Jackie-Jiang commented on issue #15127: URL: https://github.com/apache/pinot/issues/15127#issuecomment-2684203699 I believe data dir is pointing to the deep store. Without it, server won't be able to download segment. So the check should be mandatory for the cluster to function. -- This

(pinot) branch master updated (1d251b8b1f -> ad7780d20e)

2025-02-25 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 1d251b8b1f Make KafkaConsumerFactory compatible with multi-topics ingestion (#15094) add ad7780d20e Implement Mdc

Re: [PR] Optimize lookup table in join operator [pinot]

2025-02-25 Thread via GitHub
Jackie-Jiang commented on code in PR #14972: URL: https://github.com/apache/pinot/pull/14972#discussion_r1971092931 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/join/LookupTable.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software F

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
gortiz merged PR #15072: URL: https://github.com/apache/pinot/pull/15072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

Re: [PR] Optimize lookup table in join operator [pinot]

2025-02-25 Thread via GitHub
Jackie-Jiang commented on code in PR #14972: URL: https://github.com/apache/pinot/pull/14972#discussion_r1971085572 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/join/DoubleLookupTable.java: ## @@ -0,0 +1,65 @@ +/** + * Licensed to the Apache Softw

Re: [PR] Optimize lookup table in join operator [pinot]

2025-02-25 Thread via GitHub
yashmayya commented on code in PR #14972: URL: https://github.com/apache/pinot/pull/14972#discussion_r1971010755 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/join/LookupTable.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foun

Re: [I] `ArrayIndexOutOfBoundsException` error for multi-stage query [pinot]

2025-02-25 Thread via GitHub
yashmayya commented on issue #15121: URL: https://github.com/apache/pinot/issues/15121#issuecomment-2683905618 @dang-stripe are there any more errors in the server logs that could provide more context on the actual issue causing the `ArrayIndexOutOfBoundsException `? I agree that we need to

Re: [PR] Add minimizeDataMovement to Rebalacne API [pinot]

2025-02-25 Thread via GitHub
somandal commented on code in PR #15110: URL: https://github.com/apache/pinot/pull/15110#discussion_r1970742135 ## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java: ## @@ -504,6 +505,208 @@ public void testMirr

[I] Controller fails to startup if DataDir cannot be created [pinot]

2025-02-25 Thread via GitHub
rseetham opened a new issue, #15127: URL: https://github.com/apache/pinot/issues/15127 The controller has a property `controller.data.dir`. When the controller is started, we initialize a class called `ControllerFilePathProvider`. It reads this properpy and tried to load the PinotFS with th

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-02-25 Thread via GitHub
xiangfu0 commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r1970704248 ## pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java: ## @@ -146,6 +147,7 @@ public abstract class BaseBrokerStarter implements S

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-02-25 Thread via GitHub
lnbest0707-uber commented on PR #14920: URL: https://github.com/apache/pinot/pull/14920#issuecomment-2683545261 This is a great feature. And I believe we could expand its usage beyond failure recovery. E.g. we could combine the feature with this proposal https://github.com/apache/pinot/i

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-02-25 Thread via GitHub
Jackie-Jiang commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r1970679582 ## pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java: ## @@ -392,6 +394,11 @@ public void start() timeSeriesReque

Re: [PR] Add minimizeDataMovement to Rebalacne API [pinot]

2025-02-25 Thread via GitHub
J-HowHuang commented on PR #15110: URL: https://github.com/apache/pinot/pull/15110#issuecomment-2683530503 > Is it possible to add a rebalance specific test case as well for the new flag? > > `OfflineClusterIntegrationTest` had some tests with dry-run, but can probably be enhance

Re: [PR] Add minimizeDataMovement to Rebalacne API [pinot]

2025-02-25 Thread via GitHub
J-HowHuang commented on code in PR #15110: URL: https://github.com/apache/pinot/pull/15110#discussion_r1970687275 ## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java: ## @@ -604,7 +814,8 @@ public void testMirr

Re: [PR] Update the segment operation throttle defaults to Integer.MAX_VALUE [pinot]

2025-02-25 Thread via GitHub
somandal commented on code in PR #15126: URL: https://github.com/apache/pinot/pull/15126#discussion_r1970673685 ## pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/SegmentOperationsThrottlerTest.java: ## @@ -308,11 +308,13 @@ public void testServingQueriesD

Re: [I] `ArrayIndexOutOfBoundsException` error for multi-stage query [pinot]

2025-02-25 Thread via GitHub
Jackie-Jiang commented on issue #15121: URL: https://github.com/apache/pinot/issues/15121#issuecomment-2683502195 @gortiz From observability perspective, the exception message doesn't include the stack trace of the original exception (i.e `java.lang.ArrayIndexOutOfBoundsException`). We shou

Re: [I] `ArrayIndexOutOfBoundsException` error for multi-stage query [pinot]

2025-02-25 Thread via GitHub
Jackie-Jiang commented on issue #15121: URL: https://github.com/apache/pinot/issues/15121#issuecomment-2683498189 @gortiz @yashmayya Can you help take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Update the segment operation throttle defaults to Integer.MAX_VALUE [pinot]

2025-02-25 Thread via GitHub
codecov-commenter commented on PR #15126: URL: https://github.com/apache/pinot/pull/15126#issuecomment-2683489052 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15126?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Update the segment operation throttle defaults to Integer.MAX_VALUE [pinot]

2025-02-25 Thread via GitHub
klsince commented on code in PR #15126: URL: https://github.com/apache/pinot/pull/15126#discussion_r1970661505 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -252,26 +252,28 @@ public static class Instance { // Preprocess throttle configs

Re: [PR] Update the segment operation throttle defaults to Integer.MAX_VALUE [pinot]

2025-02-25 Thread via GitHub
somandal commented on PR #15126: URL: https://github.com/apache/pinot/pull/15126#issuecomment-2683470357 Quickstart test failures seem to be happening for other PRs as well, so it is unrelated to this change ``` Standalone test failed: Cannot get correct result for count star query

[PR] Update the segment operation throttle defaults to Integer.MAX_VALUE [pinot]

2025-02-25 Thread via GitHub
somandal opened a new pull request, #15126: URL: https://github.com/apache/pinot/pull/15126 Update the segment operation throttle defaults to Integer.MAX_VALUE. Setting the default to this is more effective in disabling throttling for scenarios where max.parallel.refresh.threads and STATE_T

Re: [I] `ArrayIndexOutOfBoundsException` error for multi-stage query [pinot]

2025-02-25 Thread via GitHub
dang-stripe commented on issue #15121: URL: https://github.com/apache/pinot/issues/15121#issuecomment-2683208509 I've narrowed it down to this part of the query. * when i comment the sum agg out it works * when i switch to count/min/max it works * when i switch to count(*) filter

Re: [PR] Add `DATE_TRUNC` Optimizer [pinot]

2025-02-25 Thread via GitHub
ashishjayamohan commented on code in PR #14385: URL: https://github.com/apache/pinot/pull/14385#discussion_r1970474848 ## pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java: ## @@ -411,6 +427,94 @@ && isStringLiteral(dateTimeC

Re: [PR] Add `DATE_TRUNC` Optimizer [pinot]

2025-02-25 Thread via GitHub
ashishjayamohan commented on code in PR #14385: URL: https://github.com/apache/pinot/pull/14385#discussion_r1970466933 ## pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java: ## @@ -411,6 +427,94 @@ && isStringLiteral(dateTimeC

(pinot) branch master updated: Make KafkaConsumerFactory compatible with multi-topics ingestion (#15094)

2025-02-25 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 1d251b8b1f Make KafkaConsumerFactory compatible

Re: [PR] Make KafkaConsumerFactory compatible with multi-topics ingestion [pinot]

2025-02-25 Thread via GitHub
chenboat merged PR #15094: URL: https://github.com/apache/pinot/pull/15094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Add minimizeDataMovement to Rebalacne API [pinot]

2025-02-25 Thread via GitHub
klsince commented on code in PR #15110: URL: https://github.com/apache/pinot/pull/15110#discussion_r1970330120 ## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java: ## @@ -604,7 +814,8 @@ public void testMirrorS

Re: [PR] Add predownload functionality to Pinot [pinot]

2025-02-25 Thread via GitHub
lnbest0707-uber commented on PR #14686: URL: https://github.com/apache/pinot/pull/14686#issuecomment-2682921049 > > `feature` `performance` #14592 > > Add a pre-download feature to enable "graceful" node replacement on Pinot. With this feature, during node replacement, admins would replac

Re: [PR] Add predownload functionality to Pinot [pinot]

2025-02-25 Thread via GitHub
chenboat commented on PR #14686: URL: https://github.com/apache/pinot/pull/14686#issuecomment-2682903021 > `feature` `performance` #14592 > > Add a pre-download feature to enable "graceful" node replacement on Pinot. With this feature, during node replacement, admins would replace the

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
gortiz commented on PR #15072: URL: https://github.com/apache/pinot/pull/15072#issuecomment-2682089120 I've changed some interfaces and improved javadocs as requested. I hope it will be easier to read now -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
gortiz commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969850229 ## pinot-tools/src/main/resources/log4j2.xml: ## @@ -23,7 +23,12 @@ logs -%d{/MM/dd HH:mm:ss.SSS} %p [%c{1}] [%t] %m%n + Review Comment: It i

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
gortiz commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969845432 ## pinot-spi/src/main/java/org/apache/pinot/spi/trace/LoggerConstants.java: ## @@ -0,0 +1,65 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
gortiz commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969812087 ## pinot-spi/src/main/java/org/apache/pinot/spi/executor/DecoratorExecutorService.java: ## @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
gortiz commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969779895 ## pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java: ## @@ -140,8 +142,25 @@ public synchronized void shutDown() { @Overri

Re: [PR] The delete function behaviour for GCSPinotFS is different from that of S3PinotFS, LocalPinotFS. [pinot]

2025-02-25 Thread via GitHub
codecov-commenter commented on PR #15123: URL: https://github.com/apache/pinot/pull/15123#issuecomment-2681691382 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15123?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Bump software.amazon.awssdk:bom from 2.30.25 to 2.30.26 [pinot]

2025-02-25 Thread via GitHub
dependabot[bot] closed pull request #15115: Bump software.amazon.awssdk:bom from 2.30.25 to 2.30.26 URL: https://github.com/apache/pinot/pull/15115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.30.27 created (now 1f550b728a)

2025-02-25 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.30.27 in repository https://gitbox.apache.org/repos/asf/pinot.git at 1f550b728a Bump software.amazon.awssdk:bom from 2.30.25 to 2.30.27 N

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.30.26 deleted (was 61603ac05f)

2025-02-25 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.30.26 in repository https://gitbox.apache.org/repos/asf/pinot.git was 61603ac05f Bump software.amazon.awssdk:bom from 2.30.25 to 2.30.26 T

Re: [PR] Bump software.amazon.awssdk:bom from 2.30.25 to 2.30.26 [pinot]

2025-02-25 Thread via GitHub
dependabot[bot] commented on PR #15115: URL: https://github.com/apache/pinot/pull/15115#issuecomment-2681599935 Superseded by #15125. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Bump software.amazon.awssdk:bom from 2.30.25 to 2.30.27 [pinot]

2025-02-25 Thread via GitHub
dependabot[bot] opened a new pull request, #15125: URL: https://github.com/apache/pinot/pull/15125 Bumps software.amazon.awssdk:bom from 2.30.25 to 2.30.27. Most Recent Ignore Conditions Applied to This Pull Request | Dependency Name | Ignore Conditions | | --- | --- |

[PR] Bump com.gradle:develocity-maven-extension from 1.23.1 to 1.23.2 [pinot]

2025-02-25 Thread via GitHub
dependabot[bot] opened a new pull request, #15124: URL: https://github.com/apache/pinot/pull/15124 Bumps com.gradle:develocity-maven-extension from 1.23.1 to 1.23.2. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-na

(pinot) branch dependabot/maven/com.gradle-develocity-maven-extension-1.23.2 created (now e49d9bc112)

2025-02-25 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.gradle-develocity-maven-extension-1.23.2 in repository https://gitbox.apache.org/repos/asf/pinot.git at e49d9bc112 Bump com.gradle:develocity-maven-extension from

[PR] The delete function behaviour for GCSPinotFS is different from that of S3PinotFS, LocalPinotFS. [pinot]

2025-02-25 Thread via GitHub
9aman opened a new pull request, #15123: URL: https://github.com/apache/pinot/pull/15123 ## Issue GCS pinot FS returns `true` while trying to delete a directory/ file even when the directory/ file does not exist, when the forceDelete flag is set to `true`. This behavior is **wron

[PR] Update collaborators list [pinot]

2025-02-25 Thread via GitHub
yashmayya opened a new pull request, #15122: URL: https://github.com/apache/pinot/pull/15122 - The new list was generated with the same method used previously (https://github.com/apache/pinot/pull/13346, https://github.com/apache/pinot/pull/14086): ``` > git shortlog --numbered --

(pinot) branch yashmayya-patch-1 created (now 2be73175f6)

2025-02-25 Thread yashmayya
This is an automated email from the ASF dual-hosted git repository. yashmayya pushed a change to branch yashmayya-patch-1 in repository https://gitbox.apache.org/repos/asf/pinot.git at 2be73175f6 Update collaborators list This branch includes the following new commits: new 2be73175f

(pinot) 01/01: Update collaborators list

2025-02-25 Thread yashmayya
This is an automated email from the ASF dual-hosted git repository. yashmayya pushed a commit to branch yashmayya-patch-1 in repository https://gitbox.apache.org/repos/asf/pinot.git commit 2be73175f609926f25f055ad57429ff25171b280 Author: Yash Mayya AuthorDate: Tue Feb 25 16:27:34 2025 +0530

(pinot) branch master updated (49f0ebae53 -> fffa7365e1)

2025-02-25 Thread yashmayya
This is an automated email from the ASF dual-hosted git repository. yashmayya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 49f0ebae53 Decouple tracking and storage in MinionEventObserver using MinionTaskObserverStorageManager (#15044)

Re: [PR] Fix stale host / port bug for failure detector [pinot]

2025-02-25 Thread via GitHub
yashmayya merged PR #15116: URL: https://github.com/apache/pinot/pull/15116 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
yashmayya commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969459862 ## pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java: ## @@ -140,8 +142,25 @@ public synchronized void shutDown() { @Ove

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
yashmayya commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969390425 ## pinot-spi/src/main/java/org/apache/pinot/spi/trace/LoggerConstants.java: ## @@ -0,0 +1,65 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Implement MdcExecutor to manage MDC context for query execution [pinot]

2025-02-25 Thread via GitHub
yashmayya commented on code in PR #15072: URL: https://github.com/apache/pinot/pull/15072#discussion_r1969360467 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/server/ServerPlanRequestUtils.java: ## @@ -40,6 +40,7 @@ import org.apache.pinot.common.util