Re: [PR] Avoid double zkMetadata updates when uploading segments for a Tier table when enableParallelPushProtection is set [pinot]

2025-05-16 Thread via GitHub
codecov-commenter commented on PR #15829: URL: https://github.com/apache/pinot/pull/15829#issuecomment-2888124869 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15829?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] [New Feature] Case-Sensitive Text Indexing [pinot]

2025-05-16 Thread via GitHub
xiangfu0 commented on PR #15803: URL: https://github.com/apache/pinot/pull/15803#issuecomment-2888035772 > What do you think about just reusing the existing Analyzer class config to enable this? I see case sensitivity eventually being necessary for multiple analyzers, but the config only en

Re: [PR] Modify OFFLINE and REALTIME completed segments to calculate default partitionId based on same logic as consuming segments rather than 0 [pinot]

2025-05-16 Thread via GitHub
somandal commented on code in PR #15817: URL: https://github.com/apache/pinot/pull/15817#discussion_r2093804392 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java: ## @@ -448,10 +472,10 @@ public static int get

Re: [PR] Modify OFFLINE and REALTIME completed segments to calculate default partitionId based on same logic as consuming segments rather than 0 [pinot]

2025-05-16 Thread via GitHub
somandal commented on code in PR #15817: URL: https://github.com/apache/pinot/pull/15817#discussion_r2093803467 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java: ## @@ -439,6 +440,29 @@ public Map> getNonTie

Re: [PR] Logical time boundary [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang commented on code in PR #15776: URL: https://github.com/apache/pinot/pull/15776#discussion_r2093797708 ## pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java: ## @@ -418,17 +420,74 @@ public synchronized void includeServerToRouting(S

Re: [PR] Modify OFFLINE and REALTIME completed segments to calculate default partitionId based on same logic as consuming segments rather than 0 [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang commented on code in PR #15817: URL: https://github.com/apache/pinot/pull/15817#discussion_r2093793828 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java: ## @@ -439,6 +440,29 @@ public Map> getNo

[PR] Add memory allocation stats per query on server [pinot]

2025-05-16 Thread via GitHub
vvivekiyer opened a new pull request, #15828: URL: https://github.com/apache/pinot/pull/15828 This PR adds memoryAllocated stats per query (similar to threadCPUTime). At the leaf servers, this collects the total heap memory allocated for query execution (runner + worker threads) and

(pinot) branch master updated: [multistage] Bug Fixes and Improvements to Physical Optimizer (#15813)

2025-05-16 Thread ankitsultana
This is an automated email from the ASF dual-hosted git repository. ankitsultana pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2de0888e96 [multistage] Bug Fixes and Impro

Re: [PR] [multistage] Bug Fixes and Improvements to Physical Optimizer [pinot]

2025-05-16 Thread via GitHub
ankitsultana merged PR #15813: URL: https://github.com/apache/pinot/pull/15813 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] [multistage] Bug Fixes and Improvements to Physical Optimizer [pinot]

2025-05-16 Thread via GitHub
ankitsultana commented on code in PR #15813: URL: https://github.com/apache/pinot/pull/15813#discussion_r2093740781 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/physical/v2/ExchangeStrategy.java: ## @@ -18,50 +18,100 @@ */ package org.apache.pinot.query

Re: [PR] [multistage] Bug Fixes and Improvements to Physical Optimizer [pinot]

2025-05-16 Thread via GitHub
wirybeaver commented on code in PR #15813: URL: https://github.com/apache/pinot/pull/15813#discussion_r2093696160 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/physical/v2/ExchangeStrategy.java: ## @@ -18,50 +18,100 @@ */ package org.apache.pinot.query.p

Re: [I] IngestionAggregation is prone to silent breakage [pinot]

2025-05-16 Thread via GitHub
jadami10 commented on issue #15827: URL: https://github.com/apache/pinot/issues/15827#issuecomment-2887702131 Great suggestion for 2. That seems like the best compromise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Tables with _realtime at the end of their name are not routeable [pinot]

2025-05-16 Thread via GitHub
jadami10 commented on issue #15826: URL: https://github.com/apache/pinot/issues/15826#issuecomment-2887700769 Solid link, thank you! I can take up verifying in the next few weeks when we pull a bit closer to latest upstream. I think either way, we're just going to disallow _offline/re

Re: [I] IngestionAggregation is prone to silent breakage [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang commented on issue #15827: URL: https://github.com/apache/pinot/issues/15827#issuecomment-2887697872 I think we should do 2 things: 1. Enhance the validation to validate everything required to enable metrics aggregation 2. Check `continueOnError` from `IngestionConfig` in

[I] IngestionAggregation is prone to silent breakage [pinot]

2025-05-16 Thread via GitHub
jadami-stripe opened a new issue, #15827: URL: https://github.com/apache/pinot/issues/15827 https://github.com/apache/pinot/pull/12464 attempted to address this, but it only did so for 1 case. [MutableSegmentImpl](https://sourcegraph.com/github.com/apache/pinot/-/blob/pinot-segment-local/sr

Re: [PR] [New Feature] Case-Sensitive Text Indexing [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang commented on code in PR #15803: URL: https://github.com/apache/pinot/pull/15803#discussion_r2093680517 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/TextIndexConfig.java: ## @@ -98,6 +100,21 @@ static DocIdTranslatorMode of(String mode) {

Re: [I] IngestionAggregation is prone to silent breakage [pinot]

2025-05-16 Thread via GitHub
jadami10 commented on issue #15827: URL: https://github.com/apache/pinot/issues/15827#issuecomment-2887687294 cc @Jackie-Jiang, thoughts here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Tables with _realtime at the end of their name are not routeable [pinot]

2025-05-16 Thread via GitHub
ankitsultana commented on issue #15826: URL: https://github.com/apache/pinot/issues/15826#issuecomment-2887662530 I had reported this here: https://github.com/apache/pinot/pull/14830#issuecomment-2788228861 This should be fixed now with #15692 but I haven't verified it yet. -- This

[I] Tables with _realtime at the end of their name are not routeable [pinot]

2025-05-16 Thread via GitHub
jadami10 opened a new issue, #15826: URL: https://github.com/apache/pinot/issues/15826 I'm not sure where yet, but some time between 1.2 and latest master, this has broken. We have a user making a table, `tableA_realtime`. On Pinot 1.2, this worked fine. But on latest master, we see

Re: [I] Kafka Offset Lag Metrics Increase Kafka Cluster Traffic [pinot]

2025-05-16 Thread via GitHub
jadami10 commented on issue #15808: URL: https://github.com/apache/pinot/issues/15808#issuecomment-2887655066 I took a cursory look at the implementation. I don't have a strong opinion other than `DEFAULT_ENABLE_OFFSET_LAG_METRIC` should be false. And it seems all of the offset fetching in

Re: [PR] [New Feature] Case-Sensitive Text Indexing [pinot]

2025-05-16 Thread via GitHub
itschrispeck commented on PR #15803: URL: https://github.com/apache/pinot/pull/15803#issuecomment-2887653480 What do you think about just reusing the existing Analyzer class config to enable this? I see case sensitivity eventually being necessary for multiple analyzers, but the config only

Re: [PR] [bugfix] Remove deleting segments from table status info [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang commented on code in PR #15725: URL: https://github.com/apache/pinot/pull/15725#discussion_r2093537489 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableViews.java: ## @@ -141,27 +145,40 @@ public TableView getExternalView( public S

Re: [I] taskMaxNumFiles is being ignored [pinot]

2025-05-16 Thread via GitHub
KKcorps commented on issue #15666: URL: https://github.com/apache/pinot/issues/15666#issuecomment-2887563944 It is being ignored because `tableMaxNumTasks` is set to `-1` In this case, it will run only one task and uses desiredSegmentSize You can set it to a positive value and `ta

Re: [I] Kafka Offset Lag Metrics Increase Kafka Cluster Traffic [pinot]

2025-05-16 Thread via GitHub
KKcorps commented on issue #15808: URL: https://github.com/apache/pinot/issues/15808#issuecomment-2887554602 One path I see is making it periodic and only enable it via a config I had attempted an implementation early on here - https://github.com/apache/pinot/pull/14142 If this app

Re: [PR] [Flaky-test] Fix QueryRunnerTest.testSqlWithExceptionMsgChecker() [pinot]

2025-05-16 Thread via GitHub
codecov-commenter commented on PR #15825: URL: https://github.com/apache/pinot/pull/15825#issuecomment-2887517192 ### :x: Unsupported file format > Upload processing failed due to unsupported file format. Please review the parser error message: > `Error parsing JUnit XML in /home/

[PR] [Flaky-test] Fix QueryRunnerTest.testSqlWithExceptionMsgChecker() [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang opened a new pull request, #15825: URL: https://github.com/apache/pinot/pull/15825 Fix the following error: ``` Error: Failures: Error:QueryRunnerTest.testSqlWithExceptionMsgChecker:215 Exception should contain: Timeout, but found: Received 1 error from servers: T

(pinot) branch master updated (56f1ce734b -> d77676d7d9)

2025-05-16 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 56f1ce734b Verify if skipped submodule POMs contain hardcoded versions (#15816) add d77676d7d9 Bump org.jline:jli

(pinot) branch master updated: remove Nullable on primitive type (#15809)

2025-05-16 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c47f0b66dc remove Nullable on primitive type (#15

Re: [PR] remove Nullable on primitive type [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang merged PR #15809: URL: https://github.com/apache/pinot/pull/15809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch dependabot/maven/org.jline-jline-3.30.1 deleted (was 0844b6fa9a)

2025-05-16 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.jline-jline-3.30.1 in repository https://gitbox.apache.org/repos/asf/pinot.git was 0844b6fa9a Bump org.jline:jline from 3.30.0 to 3.30.1 The revisions that were on this

Re: [PR] Bump org.jline:jline from 3.30.0 to 3.30.1 [pinot]

2025-05-16 Thread via GitHub
Jackie-Jiang merged PR #15820: URL: https://github.com/apache/pinot/pull/15820 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Maven Enforcer Plugin Test 5: Hardcoded version in any submodule POM is not allowed (skipped module example) [pinot]

2025-05-16 Thread via GitHub
codecov-commenter commented on PR #15824: URL: https://github.com/apache/pinot/pull/15824#issuecomment-2887351708 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15824?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Maven Enforcer Plugin Test 4: Hardcoded version in any submodule POM is not allowed (non-skipped module example) [pinot]

2025-05-16 Thread via GitHub
codecov-commenter commented on PR #15823: URL: https://github.com/apache/pinot/pull/15823#issuecomment-2887345423 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15823?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] remove Nullable on primitive type [pinot]

2025-05-16 Thread via GitHub
codecov-commenter commented on PR #15809: URL: https://github.com/apache/pinot/pull/15809#issuecomment-2887323285 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15809?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Support for Virtual DataSource [pinot]

2025-05-16 Thread via GitHub
praveenc7 commented on code in PR #15350: URL: https://github.com/apache/pinot/pull/15350#discussion_r2093419556 ## pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableConfigAndSchemaCache.java: ## @@ -0,0 +1,112 @@ +/** + * Licensed to the Apache Software Fo

Re: [PR] [multistage] Bug Fixes and Improvements to Physical Optimizer [pinot]

2025-05-16 Thread via GitHub
ankitsultana commented on code in PR #15813: URL: https://github.com/apache/pinot/pull/15813#discussion_r2093414409 ## pinot-query-planner/src/test/resources/queries/PhysicalOptimizerPlans.json: ## @@ -307,6 +307,59 @@ } ] }, + "physical_opt_group_trim_enabled":

[PR] Maven Enforcer Plugin Test 5: Hardcoded version in any submodule POM is not allowed (skipped module example) [pinot]

2025-05-16 Thread via GitHub
leujean02 opened a new pull request, #15824: URL: https://github.com/apache/pinot/pull/15824 Testing Maven Enforce Plugin with custom rules to validate dependency declarations and enforce the [Dependency Management Guidelines](https://docs.pinot.apache.org/developers/developers-and-contribu

[PR] Maven Enforcer Plugin Test 4: Hardcoded version in any submodule POM is not allowed (non-skipped module example) [pinot]

2025-05-16 Thread via GitHub
leujean02 opened a new pull request, #15823: URL: https://github.com/apache/pinot/pull/15823 Testing Maven Enforce Plugin with custom rules to validate dependency declarations and enforce the [Dependency Management Guidelines](https://docs.pinot.apache.org/developers/developers-and-contribu

Re: [PR] Verify if skipped submodule POMs contain hardcoded versions [pinot]

2025-05-16 Thread via GitHub
siddharthteotia merged PR #15816: URL: https://github.com/apache/pinot/pull/15816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pi

Re: [PR] Fix up rebalance batching partitionId calculation to account for COMPLETED segment partitioning differently as done in RealtimeSegmentAssignment [pinot]

2025-05-16 Thread via GitHub
codecov-commenter commented on PR #15822: URL: https://github.com/apache/pinot/pull/15822#issuecomment-2887114705 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15822?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] Fix up rebalance batching partitionId calculation to account for COMPLETED segment partitioning differently as done in RealtimeSegmentAssignment [pinot]

2025-05-16 Thread via GitHub
somandal opened a new pull request, #15822: URL: https://github.com/apache/pinot/pull/15822 The rebalance batching needs to fetch the partitionId of the segments for `strictReplicaGroup` instance selector enabled tables. The logic added today blindly uses the CONSUMING segment logic for REA

Re: [PR] Modify OFFLINE and REALTIME completed segments to calculate default partitionId based on same logic as consuming segments rather than 0 [pinot]

2025-05-16 Thread via GitHub
somandal commented on PR #15817: URL: https://github.com/apache/pinot/pull/15817#issuecomment-2886991851 This PR will need more work to get the change right for partitionColumn = null and not just allocating partitionId = 0, e.g. other code paths like reassignInstances also need to be handl

[PR] make dedup table use strict replica group assignment too [pinot]

2025-05-16 Thread via GitHub
klsince opened a new pull request, #15778: URL: https://github.com/apache/pinot/pull/15778 This PR ensures dedup table to use strict replica group assignment too. Like upsert table, for dedup table, the segments from same partition must be hosted on same server as well. -- This is an aut

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.44 created (now ec6ad75dac)

2025-05-16 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.44 in repository https://gitbox.apache.org/repos/asf/pinot.git at ec6ad75dac Bump software.amazon.awssdk:bom from 2.31.43 to 2.31.44 N

(pinot) branch dependabot/maven/org.jline-jline-3.30.1 created (now 0844b6fa9a)

2025-05-16 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.jline-jline-3.30.1 in repository https://gitbox.apache.org/repos/asf/pinot.git at 0844b6fa9a Bump org.jline:jline from 3.30.0 to 3.30.1 No new revisions were added