Re: [PR] [multistage] Support Physical Optimizer E2E [pinot]

2025-05-02 Thread via GitHub
siddharthteotia commented on PR #15698: URL: https://github.com/apache/pinot/pull/15698#issuecomment-2848466737 Apologies if these questions have been discussed earlier. In that case, you can just point to the discussion or doc. I have a couple of questions out of curiosity --

Re: [PR] [multistage] Support Physical Optimizer E2E [pinot]

2025-05-02 Thread via GitHub
ankitsultana commented on PR #15698: URL: https://github.com/apache/pinot/pull/15698#issuecomment-2848380935 Not yet, but I hope to run it at some point. FWIW I don't expect this new optimizer to add any new overheads of its own. Startree folks have seen some bottlenecks from how Calc

Re: [PR] Pinot-spi japicmp test [pinot]

2025-05-02 Thread via GitHub
siddharthteotia commented on PR #15705: URL: https://github.com/apache/pinot/pull/15705#issuecomment-2848344933 This was a test PR to detect SPI incompatible changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Pinot-spi japicmp test [pinot]

2025-05-02 Thread via GitHub
siddharthteotia closed pull request #15705: Pinot-spi japicmp test URL: https://github.com/apache/pinot/pull/15705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

(pinot) branch master updated: Metrics that Tracks the Progress of Table Rebalance Jobs (#15650)

2025-05-02 Thread somandal
This is an automated email from the ASF dual-hosted git repository. somandal pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 78cb37c189 Metrics that Tracks the Progress of

Re: [PR] [re-open] Metrics that Tracks the Progress of each Table Rebalance Job [pinot]

2025-05-02 Thread via GitHub
somandal merged PR #15650: URL: https://github.com/apache/pinot/pull/15650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Log IS update failures in ensureAllPartitionsConsuming [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on code in PR #15673: URL: https://github.com/apache/pinot/pull/15673#discussion_r2072252978 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1208,33 +1208,45 @@ public void ensur

Re: [PR] Logical table CRUD operations. [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on code in PR #15515: URL: https://github.com/apache/pinot/pull/15515#discussion_r2072247997 ## pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java: ## @@ -216,15 +273,32 @@ public boolean registerTableConfigChangeListener(T

Re: [PR] Retry postStopConsumedMsg for segment init errors [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on PR #15652: URL: https://github.com/apache/pinot/pull/15652#issuecomment-2848289152 We do have `RealtimeSegmentValidationManager._segmentAutoResetOnErrorAtValidation` but seems it is always skipped for pauseless. What will happen for non-pauseless case when t

Re: [PR] Retry postStopConsumedMsg for segment init errors [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on PR #15652: URL: https://github.com/apache/pinot/pull/15652#issuecomment-2848284910 > > Can we simply throw the exception out and not mark IS as OFFLINE? > > @Jackie-Jiang if we donot mark IS offline, Realtime segment validation manager (RVM) job cannot restar

Re: [PR] Fallback for bad values in periodic controller task configurations [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on code in PR #15466: URL: https://github.com/apache/pinot/pull/15466#discussion_r2072235191 ## pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java: ## @@ -367,6 +367,8 @@ private static long getRandomInitialDelayInSeconds() {

(pinot) branch master updated (8af019e32c -> 2c39c8f5e1)

2025-05-02 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 8af019e32c Bump software.amazon.awssdk:bom from 2.31.33 to 2.31.34 (#15697) add 2c39c8f5e1 Bump parquet.version f

(pinot) branch dependabot/maven/parquet.version-1.15.2 deleted (was e021842ace)

2025-05-02 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/parquet.version-1.15.2 in repository https://gitbox.apache.org/repos/asf/pinot.git was e021842ace Bump parquet.version from 1.15.1 to 1.15.2 The revisions that were on this

Re: [PR] Bump parquet.version from 1.15.1 to 1.15.2 [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang merged PR #15696: URL: https://github.com/apache/pinot/pull/15696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072234211 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.34 deleted (was 12f586f574)

2025-05-02 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.34 in repository https://gitbox.apache.org/repos/asf/pinot.git was 12f586f574 Bump software.amazon.awssdk:bom from 2.31.33 to 2.31.34 The r

(pinot) branch master updated (c36d88f7bf -> 8af019e32c)

2025-05-02 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from c36d88f7bf Japicmp implementation in pinot-spi (#15684) add 8af019e32c Bump software.amazon.awssdk:bom from 2.31.3

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.33 to 2.31.34 [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang merged PR #15697: URL: https://github.com/apache/pinot/pull/15697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Fail the config file reading if there are duplicated keys in the file [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on PR #15648: URL: https://github.com/apache/pinot/pull/15648#issuecomment-2848268990 > > I think duplicate key is the standard way of configuring array in `configuration2` to avoid problem of escaping special character. Please make sure the change is backward compati

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072233262 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072233837 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072233756 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1352,26 +1380,55 @@ static boolean isExternalViewConv

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072232226 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -239,6 +239,10 @@ TableRebalanceContext

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072232915 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

[PR] Pinot-spi japicmp test [pinot]

2025-05-02 Thread via GitHub
matvj250 opened a new pull request, #15705: URL: https://github.com/apache/pinot/pull/15705 Deleting utils.builder.ControllerRequestURLBuilder.forTableGetServerInstances() to test japicmp in apache/pinot. It has no uses, so the removal shouldn't cause any compiler errors, only errors in ja

[I] Add quantileIndex Support for Fast Percentile Queries in Apache Pinot [pinot]

2025-05-02 Thread via GitHub
scheler opened a new issue, #15704: URL: https://github.com/apache/pinot/issues/15704 Requesting comments on the following proposal to add `quantileIndex` support for fast percentile queries. **Description**: Currently, Apache Pinot does not provide a native, automatic way to optim

Re: [PR] [multistage] Support Physical Optimizer E2E [pinot]

2025-05-02 Thread via GitHub
siddharthteotia commented on PR #15698: URL: https://github.com/apache/pinot/pull/15698#issuecomment-2848227917 Do we have data points on overhead of optimizer itself ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Create proper response when all segments are pruned on broker [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15703: URL: https://github.com/apache/pinot/pull/15703#issuecomment-2848226031 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15703?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Japicmp implementation in pinot-spi [pinot]

2025-05-02 Thread via GitHub
siddharthteotia merged PR #15684: URL: https://github.com/apache/pinot/pull/15684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pi

(pinot) branch master updated: Japicmp implementation in pinot-spi (#15684)

2025-05-02 Thread siddteotia
This is an automated email from the ASF dual-hosted git repository. siddteotia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c36d88f7bf Japicmp implementation in pinot-sp

Re: [PR] Do not modify cached schema in TableDataManager [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on PR #15690: URL: https://github.com/apache/pinot/pull/15690#issuecomment-2848192573 @ankitsultana Yes it is, but I don't think it will break AFAICT. It is mostly for future proof -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Create proper response when all segments are pruned on broker [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on PR #15703: URL: https://github.com/apache/pinot/pull/15703#issuecomment-2848183119 cc @albertobastos -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[PR] Create proper response when all segments are pruned on broker [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang opened a new pull request, #15703: URL: https://github.com/apache/pinot/pull/15703 Another attempt over #15502 Add broker side empty response handling similar to server side (`ResultsBlockUtils`) so that when all segments are pruned on the broker side, we can still get

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848145603 > Before this PR, if we set externalViewStabilizationTimeoutInMs=1ms, regular run will fail after 1ms wait, while bestEfforts continue the next step after 1ms. > > In

(pinot) branch master updated (a7c99db6d9 -> f9ca123c9a)

2025-05-02 Thread somandal
This is an automated email from the ASF dual-hosted git repository. somandal pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from a7c99db6d9 Do not modify cached schema in TableDataManager (#15690) add f9ca123c9a [Test] Clean-up and move reba

Re: [PR] [Test] Clean-up and move rebalance related tests from offline integration to TableRebalanceIntegrationTest [pinot]

2025-05-02 Thread via GitHub
somandal merged PR #15702: URL: https://github.com/apache/pinot/pull/15702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848117820 Before this PR, if we set `externalViewStabilizationTimeoutInMs=1ms`, regular run will fail after 1ms wait, while bestEfforts continue the next step after 1ms. In th

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072141619 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExternalV

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072139850 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848092083 Also, my understanding is there is no behavior change for bestEfforts, right? Or am I misunderstanding that? The timeout scenario you talked about can happen today too, right? -- This

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848089933 > Imagine that we set `externalViewStabilizationTimeoutInMs` super short (relative to 1hr today), say 1ms, a regular rebalance with `bestEfforts=false` should finish as usual. Howeve

Re: [PR] [Test] Clean-up and move rebalance related tests from offline integration to TableRebalanceIntegrationTest [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15702: URL: https://github.com/apache/pinot/pull/15702#issuecomment-2848089761 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15702?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848078621 Imagine that we set `externalViewStabilizationTimeoutInMs` super short (relative to 1hr today), say 1ms, a regular rebalance with `bestEfforts=false` should finish as usual. Howeve

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848044561 > @yashmayya @somandal , for `bestEfforts=true`, it stops waiting convergence once reaching the first timeout period. Should we change it to keeping waiting until converge, and continue

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2848036611 @yashmayya @somandal , for `bestEfforts=true`, it stops waiting convergence once reaching the first timeout period. Should we change it to keeping waiting until converge, and continue

Re: [PR] [re-open] Metrics that Tracks the Progress of each Table Rebalance Job [pinot]

2025-05-02 Thread via GitHub
somandal commented on PR #15650: URL: https://github.com/apache/pinot/pull/15650#issuecomment-2848032345 @J-HowHuang Looks like this is consistently failing due to: ``` [INFO] [INFO] Results: [INFO] Error: Failures: Error: PauselessRealtimeIngestionWithDe

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072100032 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExternalV

Re: [I] [flaky-test] PauselessRealtimeIngestionWithDedupIntegrationTest#BaseDedupIntegrationTest [pinot]

2025-05-02 Thread via GitHub
itschrispeck closed issue #15701: [flaky-test] PauselessRealtimeIngestionWithDedupIntegrationTest#BaseDedupIntegrationTest URL: https://github.com/apache/pinot/issues/15701 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] [flaky-test] PauselessRealtimeIngestionWithDedupIntegrationTest#BaseDedupIntegrationTest [pinot]

2025-05-02 Thread via GitHub
itschrispeck commented on issue #15701: URL: https://github.com/apache/pinot/issues/15701#issuecomment-2848027074 nevermind, this was just fixed: https://github.com/apache/pinot/commit/2b12cbdf4892f7e8402a180e7f456e4a7eac7920 -- This is an automated message from the Apache Git Service. T

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072095129 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

Re: [PR] Local setup [pinot]

2025-05-02 Thread via GitHub
noob-se7en commented on PR #15700: URL: https://github.com/apache/pinot/pull/15700#issuecomment-2847985244 Sigh I again created PR here by mistake -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Local setup [pinot]

2025-05-02 Thread via GitHub
noob-se7en closed pull request #15700: Local setup URL: https://github.com/apache/pinot/pull/15700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commi

[PR] [Test] Clean-up and move rebalance related tests from offline integration to TableRebalanceIntegrationTest [pinot]

2025-05-02 Thread via GitHub
somandal opened a new pull request, #15702: URL: https://github.com/apache/pinot/pull/15702 Clean up the offline table rebalance integration tests, add a test to pre-checks, and move them to TableRebalanceIntegrationTest -- This is an automated message from the Apache Git Service. To resp

[I] [flaky-test] PauselessRealtimeIngestionWithDedupIntegrationTest#BaseDedupIntegrationTest [pinot]

2025-05-02 Thread via GitHub
itschrispeck opened a new issue, #15701: URL: https://github.com/apache/pinot/issues/15701 Seen this a few times, e.g. error: ``` Error: PauselessRealtimeIngestionWithDedupIntegrationTest>BaseDedupIntegrationTest.setUp:79->BaseDedupIntegrationTest.createDedupConfigsWithReplicas:89->

[PR] Local setup [pinot]

2025-05-02 Thread via GitHub
noob-se7en opened a new pull request, #15700: URL: https://github.com/apache/pinot/pull/15700 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat`

Re: [PR] [multistage] Support Physical Optimizer E2E [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15698: URL: https://github.com/apache/pinot/pull/15698#issuecomment-2847867046 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15698?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2072001496 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExternalV

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071998555 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

Re: [PR] Do not modify cached schema in TableDataManager [pinot]

2025-05-02 Thread via GitHub
ankitsultana commented on PR #15690: URL: https://github.com/apache/pinot/pull/15690#issuecomment-2847839982 @Jackie-Jiang : was this introduced by #15461 ? I wanted to see if one of our builds is impacted or not. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071989698 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1303,47 +1303,75 @@ private IdealState waitForExterna

Re: [PR] Do not modify cached schema in TableDataManager [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang merged PR #15690: URL: https://github.com/apache/pinot/pull/15690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Fixes Integration test [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang merged PR #15699: URL: https://github.com/apache/pinot/pull/15699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: Do not modify cached schema in TableDataManager (#15690)

2025-05-02 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new a7c99db6d9 Do not modify cached schema in TableDa

(pinot) branch master updated: Fixes Integration test (#15699)

2025-05-02 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2b12cbdf48 Fixes Integration test (#15699) 2b12cb

Re: [PR] Do not modify cached schema in TableDataManager [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on code in PR #15690: URL: https://github.com/apache/pinot/pull/15690#discussion_r2071973613 ## pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java: ## @@ -297,6 +298,7 @@ private TableDataManager createTableDataM

Re: [PR] Adding null check for fromZNRecord while creating a new InstancePartitions from ZNRecord [pinot]

2025-05-02 Thread via GitHub
ankitsultana commented on code in PR #15695: URL: https://github.com/apache/pinot/pull/15695#discussion_r2071969846 ## pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitions.java: ## @@ -119,6 +120,7 @@ public void setInstances(int partitionId, int rep

Re: [PR] Do not modify cached schema in TableDataManager [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15690: URL: https://github.com/apache/pinot/pull/15690#issuecomment-2847771138 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15690?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Do not modify cached schema in TableDataManager [pinot]

2025-05-02 Thread via GitHub
xiangfu0 commented on code in PR #15690: URL: https://github.com/apache/pinot/pull/15690#discussion_r2071909950 ## pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java: ## @@ -297,6 +298,7 @@ private TableDataManager createTableDataManag

Re: [PR] Fixes Integration test [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15699: URL: https://github.com/apache/pinot/pull/15699#issuecomment-2847700051 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15699?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] MSE support for case-insensitiveness without breaking compatibility [pinot]

2025-05-02 Thread via GitHub
Jackie-Jiang commented on code in PR #15692: URL: https://github.com/apache/pinot/pull/15692#discussion_r2071895037 ## pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java: ## @@ -102,7 +103,7 @@ public class BrokerRoutingManager implements Routi

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071886886 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1387,9 +1444,13 @@ private static boolean isExternalV

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
J-HowHuang commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071879467 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1352,26 +1380,55 @@ static boolean isExternalViewConv

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
noob-se7en commented on PR #15398: URL: https://github.com/apache/pinot/pull/15398#issuecomment-2847621451 https://github.com/apache/pinot/pull/15699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Fixes Integration test [pinot]

2025-05-02 Thread via GitHub
noob-se7en opened a new pull request, #15699: URL: https://github.com/apache/pinot/pull/15699 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
noob-se7en commented on PR #15398: URL: https://github.com/apache/pinot/pull/15398#issuecomment-2847590539 yep just saw as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
KKcorps commented on PR #15398: URL: https://github.com/apache/pinot/pull/15398#issuecomment-2847582010 @noob-se7en can you please fix this test to address the error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
KKcorps commented on PR #15398: URL: https://github.com/apache/pinot/pull/15398#issuecomment-2847576916 @ankitsultana that is due to a different PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
ankitsultana commented on PR #15398: URL: https://github.com/apache/pinot/pull/15398#issuecomment-2847570504 @KKcorps @noob-se7en : has this broken tests on master? Seeing this in my PR: ``` Error: Failures: Error: PauselessRealtimeIngestionWithDedupIntegrationTest>B

Re: [PR] Fail the config file reading if there are duplicated keys in the file [pinot]

2025-05-02 Thread via GitHub
xiangfu0 commented on PR #15648: URL: https://github.com/apache/pinot/pull/15648#issuecomment-2847503767 > I think duplicate key is the standard way of configuring array in `configuration2` to avoid problem of escaping special character. Please make sure the change is backward compatible

Re: [PR] [multistage] Add Pushdown and Worker Rules [pinot]

2025-05-02 Thread via GitHub
ankitsultana commented on code in PR #15658: URL: https://github.com/apache/pinot/pull/15658#discussion_r2071687778 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/physical/v2/opt/rules/AggregatePushdownRule.java: ## @@ -0,0 +1,290 @@ +/** + * Licensed to the

Re: [PR] [multistage] Add Pushdown and Worker Rules [pinot]

2025-05-02 Thread via GitHub
ankitsultana commented on code in PR #15658: URL: https://github.com/apache/pinot/pull/15658#discussion_r2071684420 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/physical/v2/PRelToPlanNodeConverter.java: ## @@ -0,0 +1,379 @@ +/** + * Licensed to the Apache

[PR] [multistage] Support Physical Optimizer E2E [pinot]

2025-05-02 Thread via GitHub
ankitsultana opened a new pull request, #15698: URL: https://github.com/apache/pinot/pull/15698 # Summary Completes the first E2E working version of the new Physical Optimizer. There's quite a bit of refactoring and feature porting that still needs to be done in order to make this us

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
yashmayya commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071601489 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1387,9 +1444,13 @@ private static boolean isExternalVi

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
somandal commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071597362 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1387,9 +1444,13 @@ private static boolean isExternalVie

Re: [PR] Log IS update failures in ensureAllPartitionsConsuming [pinot]

2025-05-02 Thread via GitHub
noob-se7en commented on code in PR #15673: URL: https://github.com/apache/pinot/pull/15673#discussion_r2071591203 ## pinot-common/src/main/java/org/apache/pinot/common/utils/helix/IdealStateGroupCommit.java: ## @@ -105,7 +105,7 @@ public static synchronized void setMinNumCharsI

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
yashmayya commented on PR #15618: URL: https://github.com/apache/pinot/pull/15618#issuecomment-2847158387 Thanks for this enhancement @J-HowHuang, I think this will be super useful to have! Let's also update the `externalViewStabilizationTimeoutInMs` API param docs for the `POST /tables/{ta

Re: [PR] Dynamic timeout for EV convergence in table rebalance [pinot]

2025-05-02 Thread via GitHub
yashmayya commented on code in PR #15618: URL: https://github.com/apache/pinot/pull/15618#discussion_r2071513651 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -1352,26 +1380,55 @@ static boolean isExternalViewConve

(pinot) branch master updated (c423ae04db -> 55eb236edc)

2025-05-02 Thread kharekartik
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from c423ae04db Add updateTargetTier check to the rebalance configs pre-check for table rebalancer (#15689) add 5

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
KKcorps merged PR #15398: URL: https://github.com/apache/pinot/pull/15398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

Re: [PR] Adds Intgeration tests for Dedup and Pauseless [pinot]

2025-05-02 Thread via GitHub
noob-se7en commented on code in PR #15398: URL: https://github.com/apache/pinot/pull/15398#discussion_r2071560925 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/PauselessRealtimeIngestionWithDedupIntegrationTest.java: ## @@ -0,0 +1,33 @@ +/** + * Lic

Re: [PR] Adding null check for fromZNRecord while creating a new InstancePartitions from ZNRecord [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15695: URL: https://github.com/apache/pinot/pull/15695#issuecomment-2847033144 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15695?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Improve byte-based gRPC block splitting [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15694: URL: https://github.com/apache/pinot/pull/15694#issuecomment-2847025160 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15694?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.34 created (now 12f586f574)

2025-05-02 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.34 in repository https://gitbox.apache.org/repos/asf/pinot.git at 12f586f574 Bump software.amazon.awssdk:bom from 2.31.33 to 2.31.34 N

[PR] Bump software.amazon.awssdk:bom from 2.31.33 to 2.31.34 [pinot]

2025-05-02 Thread via GitHub
dependabot[bot] opened a new pull request, #15697: URL: https://github.com/apache/pinot/pull/15697 Bumps software.amazon.awssdk:bom from 2.31.33 to 2.31.34. Most Recent Ignore Conditions Applied to This Pull Request | Dependency Name | Ignore Conditions | | --- | --- |

(pinot) branch dependabot/maven/parquet.version-1.15.2 created (now e021842ace)

2025-05-02 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/parquet.version-1.15.2 in repository https://gitbox.apache.org/repos/asf/pinot.git at e021842ace Bump parquet.version from 1.15.1 to 1.15.2 No new revisions were added

[PR] Bump parquet.version from 1.15.1 to 1.15.2 [pinot]

2025-05-02 Thread via GitHub
dependabot[bot] opened a new pull request, #15696: URL: https://github.com/apache/pinot/pull/15696 Bumps `parquet.version` from 1.15.1 to 1.15.2. Updates `org.apache.parquet:parquet-avro` from 1.15.1 to 1.15.2 Release notes Sourced from https://github.com/apache/parquet-mr/releas

Re: [PR] Execute Queries on Logical Tables in SSE [pinot]

2025-05-02 Thread via GitHub
codecov-commenter commented on PR #15634: URL: https://github.com/apache/pinot/pull/15634#issuecomment-2846996310 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15634?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] Adding null check for fromZNRecord while creating a new InstancePartitions from ZNRecord [pinot]

2025-05-02 Thread via GitHub
tarun11Mavani opened a new pull request, #15695: URL: https://github.com/apache/pinot/pull/15695 We see a null pointer exception while deleting the tables in our integration environment. The null pointer exception is due to ZNRecord being null when calling fromZNRecord to create a new Inst

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-05-02 Thread via GitHub
albertobastos commented on code in PR #15571: URL: https://github.com/apache/pinot/pull/15571#discussion_r2071398844 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcSendingMailbox.java: ## @@ -156,19 +176,30 @@ private StreamObserver getContentObserver()

[PR] Improve byte-based gRPC block splitting [pinot]

2025-05-02 Thread via GitHub
albertobastos opened a new pull request, #15694: URL: https://github.com/apache/pinot/pull/15694 A few hours after [byte-based gRPC block splitting PR](https://github.com/apache/pinot/pull/15571) was merged @Jackie-Jiang point out some concerns and considerations on what have been done.

  1   2   >