[PR] [Bugfix] Fix URI construction so that AddSchema command line tool works [pinot]

2024-06-05 Thread via GitHub
suyashpatel98 opened a new pull request, #13320: URL: https://github.com/apache/pinot/pull/13320 Fix URI construction so that AddSchema command line tool works when override flag is set to true Verified that when the override flag is set to true the schema is overridden successfull

(pinot) branch master updated (f4a4bf389a -> 6290bc001a)

2024-06-05 Thread vvivekiyer
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from f4a4bf389a Custom configuration property reader for segment metadata files (#12440) add 6290bc001a Interning

Re: [PR] Interning for OnHeapByteDictionary [pinot]

2024-06-05 Thread via GitHub
vvivekiyer merged PR #12342: URL: https://github.com/apache/pinot/pull/12342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

Re: [PR] Clean Google Dependencies [pinot]

2024-06-05 Thread via GitHub
abhioncbr commented on code in PR #13297: URL: https://github.com/apache/pinot/pull/13297#discussion_r1628677484 ## pom.xml: ## @@ -224,12 +224,12 @@ 26.40.0 -1.23.0 -2.10.1 -33.1.0-jre -1.44.1 -3.25.2 -1.61.1 +3.25.2 +1.61.1 Revi

Re: [I] Realtime servers reporting lag for partitions they don't own after rebalancing w/o including consuming segments [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang commented on issue #11448: URL: https://github.com/apache/pinot/issues/11448#issuecomment-2151192409 1. What is the typical freshness lag during segment commit? If we don't want to read IS for every segment commit, we need to configure a timeout longer than the common freshness

Re: [PR] Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes [pinot]

2024-06-05 Thread via GitHub
klsince commented on code in PR #13314: URL: https://github.com/apache/pinot/pull/13314#discussion_r1628544264 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/CompletionServiceHelper.java: ## @@ -129,13 +129,21 @@ private CompletionServiceResponse collectRespo

Re: [I] Error while performing schema evolution [pinot]

2024-06-05 Thread via GitHub
suyashpatel98 commented on issue #13312: URL: https://github.com/apache/pinot/issues/13312#issuecomment-2151061084 @Jackie-Jiang Please assign this to me. I will raise the PR soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [I] Realtime servers reporting lag for partitions they don't own after rebalancing w/o including consuming segments [pinot]

2024-06-05 Thread via GitHub
jadami10 commented on issue #11448: URL: https://github.com/apache/pinot/issues/11448#issuecomment-2151049470 I wish there was a smaller ZK node we could look at. But given that we (and likely others) have use cases that want to meet SLC thresholds < 30s, then that's roughly how quickly we'

Re: [PR] Clean Google Dependencies [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang commented on code in PR #13297: URL: https://github.com/apache/pinot/pull/13297#discussion_r1628489765 ## pom.xml: ## @@ -1395,7 +1389,7 @@ net.openhft posix -2.26ea0 +2.25ea0 Review Comment: Is this change intended?

Re: [I] Realtime servers reporting lag for partitions they don't own after rebalancing w/o including consuming segments [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang commented on issue #11448: URL: https://github.com/apache/pinot/issues/11448#issuecomment-2151037009 Server doesn't read IS on its own. Controller periodically scan all IS/EV and send messages to servers. In order to detect that the next segment doesn't come to this server, s

Re: [I] Realtime servers reporting lag for partitions they don't own after rebalancing w/o including consuming segments [pinot]

2024-06-05 Thread via GitHub
jadami10 commented on issue #11448: URL: https://github.com/apache/pinot/issues/11448#issuecomment-2151032441 @Jackie-Jiang, I wasn't able to track this down, but do servers consistently parse changes between idealstate/external view? Is there a way to do it when that happens and have it al

Re: [I] Realtime servers reporting lag for partitions they don't own after rebalancing w/o including consuming segments [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang commented on issue #11448: URL: https://github.com/apache/pinot/issues/11448#issuecomment-2151027072 The current solution should be able to eventually remove the lag, but could take up to 15 minutes (10 minutes timeout + 5 minutes scheduling delay). There are 2 potential enha

Re: [PR] Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes [pinot]

2024-06-05 Thread via GitHub
klsince commented on code in PR #13314: URL: https://github.com/apache/pinot/pull/13314#discussion_r1628481078 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/ServerSegmentMetadataReader.java: ## @@ -235,8 +236,18 @@ public List getValidDocIdsMetadataFromServ

(pinot) branch master updated: Custom configuration property reader for segment metadata files (#12440)

2024-06-05 Thread xbli
This is an automated email from the ASF dual-hosted git repository. xbli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new f4a4bf389a Custom configuration property reader for

Re: [PR] Custom configuration property reader for segment metadata files [pinot]

2024-06-05 Thread via GitHub
klsince merged PR #12440: URL: https://github.com/apache/pinot/pull/12440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

(pinot) branch master updated: Fix Logging Location for CPU-Based Query Killing (#13318)

2024-06-05 Thread jiaguo
This is an automated email from the ASF dual-hosted git repository. jiaguo pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 3d07b90bd7 Fix Logging Location for CPU-Based Que

Re: [PR] Fix Logging Location for CPU-Based Query Killing [pinot]

2024-06-05 Thread via GitHub
jasperjiaguo merged PR #13318: URL: https://github.com/apache/pinot/pull/13318 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Fix Logging Location for CPU-Based Query Killing [pinot]

2024-06-05 Thread via GitHub
codecov-commenter commented on PR #13318: URL: https://github.com/apache/pinot/pull/13318#issuecomment-2150896993 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13318?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[I] Data ingestion CPU efficiency improvements [pinot]

2024-06-05 Thread via GitHub
lnbest0707-uber opened a new issue, #13319: URL: https://github.com/apache/pinot/issues/13319 Pinot data ingestion from Kafka is following the 1 thread per Kafka partition mechanism. The scaling up is relying on increasing number of Kafka topic partitions. However, due to the nature of inge

[PR] fix logging [pinot]

2024-06-05 Thread via GitHub
PraveenVora opened a new pull request, #13318: URL: https://github.com/apache/pinot/pull/13318 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat`

Re: [PR] [Backfill] allow externally partitioned segment uploads for upsert tables [pinot]

2024-06-05 Thread via GitHub
klsince commented on code in PR #13107: URL: https://github.com/apache/pinot/pull/13107#discussion_r1628157120 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/UploadedRealtimeSegmentNameGenerator.java: ## @@ -0,0 +1,62 @@ +/** + * Licensed to the Apa

Re: [PR] Bump net.openhft:chronicle-core from 2.25ea15 to 2.26ea0 [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang commented on PR #13274: URL: https://github.com/apache/pinot/pull/13274#issuecomment-2150569516 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

(pinot) branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 updated (65963672c1 -> fc7eb4a0a8)

2024-06-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git discard 65963672c1 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16

Re: [PR] Disabling checkstyle plugin and Spotless plugin for Java version > 21.0 [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang closed pull request #13252: Disabling checkstyle plugin and Spotless plugin for Java version > 21.0 URL: https://github.com/apache/pinot/pull/13252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Multi stage metrics [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang commented on code in PR #13035: URL: https://github.com/apache/pinot/pull/13035#discussion_r1628117693 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java: ## @@ -54,7 +54,14 @@ public enum ServerTimer implements AbstractMetrics.Timer {

Re: [PR] Clean Google Dependencies [pinot]

2024-06-05 Thread via GitHub
abhioncbr closed pull request #13297: Clean Google Dependencies URL: https://github.com/apache/pinot/pull/13297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Error while performing schema evolution [pinot]

2024-06-05 Thread via GitHub
aishwaryathondapu commented on issue #13312: URL: https://github.com/apache/pinot/issues/13312#issuecomment-2150541038 yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.66 deleted (was 1e52d710af)

2024-06-05 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.66 in repository https://gitbox.apache.org/repos/asf/pinot.git was 1e52d710af Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.66 The r

Re: [PR] Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.66 [pinot]

2024-06-05 Thread via GitHub
Jackie-Jiang merged PR #13316: URL: https://github.com/apache/pinot/pull/13316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.66 (#13316)

2024-06-05 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 065f325b53 Bump software.amazon.awssdk:bom from 2

Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]

2024-06-05 Thread via GitHub
shounakmk219 commented on PR #13290: URL: https://github.com/apache/pinot/pull/13290#issuecomment-2150188180 ## Controller metrics validations: 1. https://github.com/apache/pinot/assets/25409127/cb1a6438-8797-4de0-99f1-cff12ec7bd51";> 2. https://github.com/apache/pinot/ass

Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]

2024-06-05 Thread via GitHub
shounakmk219 commented on code in PR #13290: URL: https://github.com/apache/pinot/pull/13290#discussion_r1627830416 ## docker/images/pinot/etc/jmx_prometheus_javaagent/configs/server.yml: ## @@ -73,6 +66,14 @@ rules: table: "$1$3" tableType: "$4" partition: "$5" +

Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]

2024-06-05 Thread via GitHub
shounakmk219 commented on code in PR #13290: URL: https://github.com/apache/pinot/pull/13290#discussion_r1627823043 ## docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml: ## @@ -112,6 +118,12 @@ rules: - pattern: "\"org\\.apache\\.pinot\\.common\\.metrics\"<>(

Re: [PR] Fix few metric rules which were affected by the database prefix handling [pinot]

2024-06-05 Thread via GitHub
shounakmk219 commented on code in PR #13290: URL: https://github.com/apache/pinot/pull/13290#discussion_r1627809416 ## docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml: ## @@ -59,6 +59,12 @@ rules: labels: database: "$2" table: "$1$3" +- pattern:

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.65 deleted (was 874a2ddb43)

2024-06-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.65 in repository https://gitbox.apache.org/repos/asf/pinot.git was 874a2ddb43 Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.65 T

Re: [PR] Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.65 [pinot]

2024-06-05 Thread via GitHub
dependabot[bot] closed pull request #13311: Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.65 URL: https://github.com/apache/pinot/pull/13311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.25.66 created (now 1e52d710af)

2024-06-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.25.66 in repository https://gitbox.apache.org/repos/asf/pinot.git at 1e52d710af Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.66 N

Re: [PR] Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.65 [pinot]

2024-06-05 Thread via GitHub
dependabot[bot] commented on PR #13311: URL: https://github.com/apache/pinot/pull/13311#issuecomment-2149608954 Superseded by #13316. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Bump software.amazon.awssdk:bom from 2.25.64 to 2.25.66 [pinot]

2024-06-05 Thread via GitHub
dependabot[bot] opened a new pull request, #13316: URL: https://github.com/apache/pinot/pull/13316 Bumps software.amazon.awssdk:bom from 2.25.64 to 2.25.66. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softwa

Re: [PR] [partial-upsert] configure early release of _partitionGroupConsumerSemaphore in RealtimeSegmentDataManager [pinot]

2024-06-05 Thread via GitHub
rohityadav1993 commented on code in PR #13256: URL: https://github.com/apache/pinot/pull/13256#discussion_r1627386271 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java: ## @@ -94,6 +94,9 @@ public enum ConsistencyMode { @JsonPropertyDescription("W

Re: [PR] Fix the NPE from IS update metrics [pinot]

2024-06-05 Thread via GitHub
gortiz commented on PR #13313: URL: https://github.com/apache/pinot/pull/13313#issuecomment-2149310502 I've created a NOOP registry in https://github.com/apache/pinot/pull/13032, so this kind of issues shouldn't be problematic in the future. -- This is an automated message from the Apache

(pinot) branch master updated: Add metrics to count joins and window functions (#13032)

2024-06-05 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new bdef83b749 Add metrics to count joins and window

Re: [PR] Add metrics to count joins and window functions [pinot]

2024-06-05 Thread via GitHub
gortiz merged PR #13032: URL: https://github.com/apache/pinot/pull/13032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

Re: [PR] Multi stage metrics [pinot]

2024-06-05 Thread via GitHub
codecov-commenter commented on PR #13035: URL: https://github.com/apache/pinot/pull/13035#issuecomment-2149272592 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13035?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u