Re: [PR] feat: change display of rebalance response [pinot]

2025-03-24 Thread via GitHub
himanish-star commented on code in PR #15331: URL: https://github.com/apache/pinot/pull/15331#discussion_r2011416497 ## pinot-controller/src/main/resources/app/components/Homepage/Operations/RebalanceServer/RebalanceServerResponses/RebalanceServerRebalanceSummaryResponse.tsx: ##

(pinot) branch master updated: Fix tooltip wording for a couple of rebalance options in the UI (#15336)

2025-03-24 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new fb8587bb92 Fix tooltip wording for a couple of re

(pinot) branch master updated (1c6553c5d8 -> c2487916e0)

2025-03-24 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 1c6553c5d8 update rebalance result order (#15305) add c2487916e0 Clean up HLC related code (#15318) No new revisi

Re: [PR] feat: change display of rebalance response [pinot]

2025-03-24 Thread via GitHub
himanish-star commented on code in PR #15331: URL: https://github.com/apache/pinot/pull/15331#discussion_r2011416002 ## pinot-controller/src/main/resources/app/components/Homepage/Operations/RebalanceServer/RebalanceResponse.tsx: ## @@ -0,0 +1,63 @@ +import {RebalanceServerSecti

(pinot) branch master updated: [spark connector] replace the use of circe with jackson (#15321)

2025-03-24 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 4baf65d4ec [spark connector] replace the use of c

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2011075925 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java: ## @@ -147,9 +159,18 @@ private BrokerData getBrokerData(Map> responses)

Re: [PR] Fix real-time table segment download [pinot]

2025-03-24 Thread via GitHub
9aman commented on PR #15316: URL: https://github.com/apache/pinot/pull/15316#issuecomment-2750110690 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Adds log to debug slow ingestion while upsert snapshot [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15334: URL: https://github.com/apache/pinot/pull/15334#discussion_r2011068916 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -849,17 +848,22 @@ public void takeSnapsh

Re: [PR] Added support to apply record enrichers before complex transformations [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15359: URL: https://github.com/apache/pinot/pull/15359#issuecomment-2749649822 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15359?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2010697074 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/IngestionConfigUtils.java: ## @@ -101,11 +101,38 @@ public static List> getStreamConfigMaps(TableConfig tabl

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-03-24 Thread via GitHub
xiangfu0 commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2011190125 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java: ## @@ -124,6 +135,7 @@ public BrokerCache(Properties properties, String con

Re: [PR] Enhance index and field config validation to block adding bloomfilter on boolean column [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15283: URL: https://github.com/apache/pinot/pull/15283#discussion_r2011061402 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java: ## @@ -1003,6 +1003,7 @@ private static void validateIndexingConfi

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-03-24 Thread via GitHub
xiangfu0 commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2011190226 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java: ## @@ -147,9 +159,18 @@ private BrokerData getBrokerData(Map> responses) {

Re: [PR] Allow More Flexibility in Time Predicate Optimizer Function Ordering [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15362: URL: https://github.com/apache/pinot/pull/15362#issuecomment-2749827806 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15362?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] Allow More Flexibility in Time Predicate Optimizer Function Ordering [pinot]

2025-03-24 Thread via GitHub
ashishjayamohan opened a new pull request, #15362: URL: https://github.com/apache/pinot/pull/15362 - Follow up from: https://github.com/apache/pinot/pull/14385#discussion_r1966265135 - Previously, the time predicate optimizers only optimized predicates in the form of `func(x) = literal`.

(pinot) branch master updated (a1d89a713b -> 477f4eeae7)

2025-03-24 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from a1d89a713b Add some comments about why using Semaphore as the consumer lock (#15361) add 477f4eeae7 Close `ChunkR

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2011075925 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java: ## @@ -147,9 +159,18 @@ private BrokerData getBrokerData(Map> responses)

Re: [PR] Close `ChunkReaderContext` to improve projection/scan based filter memory for raw columns [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang merged PR #15139: URL: https://github.com/apache/pinot/pull/15139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Adds log to debug slow ingestion while upsert snapshot [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15334: URL: https://github.com/apache/pinot/pull/15334#discussion_r2011068589 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -849,17 +848,22 @@ public void takeSnapsh

Re: [PR] [Minor] Add some comments about why using Semaphore as the consumer lock [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15361: URL: https://github.com/apache/pinot/pull/15361#issuecomment-2749637404 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15361?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-03-24 Thread via GitHub
xiangfu0 commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2011044073 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java: ## @@ -147,9 +159,18 @@ private BrokerData getBrokerData(Map> responses) {

Re: [PR] Make stats system more resilient in MSE [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15312: URL: https://github.com/apache/pinot/pull/15312#discussion_r2011055715 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -1475,6 +1476,23 @@ public static class MultiStageQueryRunner { public stati

Re: [PR] Consumes segments in strict order of sequence number [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15261: URL: https://github.com/apache/pinot/pull/15261#discussion_r2010958163 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java: ## @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Fou

Re: [PR] Add Tenants Info to Rebalance API summary [pinot]

2025-03-24 Thread via GitHub
somandal commented on code in PR #15284: URL: https://github.com/apache/pinot/pull/15284#discussion_r2011049284 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -699,6 +714,26 @@ private RebalanceSummaryResult calcul

Re: [PR] Extend POSSIBLE_QUERY_OPTIONS in PinotConnection to all query level o… [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15281: URL: https://github.com/apache/pinot/pull/15281#discussion_r2011049029 ## pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotConnection.java: ## @@ -70,10 +66,10 @@ public class PinotConnection extends Abstract

Re: [PR] Add an interface for SecretStore [pinot]

2025-03-24 Thread via GitHub
nayanika-u commented on code in PR #15226: URL: https://github.com/apache/pinot/pull/15226#discussion_r2011042834 ## pinot-spi/src/main/java/org/apache/pinot/spi/secretstore/SecretStore.java: ## @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Add an interface for SecretStore [pinot]

2025-03-24 Thread via GitHub
nayanika-u commented on code in PR #15226: URL: https://github.com/apache/pinot/pull/15226#discussion_r2011038392 ## pinot-spi/src/main/java/org/apache/pinot/spi/secretstore/SecretStore.java: ## @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] [Minor] Add some comments about why using Semaphore as the consumer lock [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang merged PR #15361: URL: https://github.com/apache/pinot/pull/15361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Initialize OOM protection in GrpcQueryServer [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15271: URL: https://github.com/apache/pinot/pull/15271#discussion_r2010926464 ## pinot-core/src/test/java/org/apache/pinot/core/transport/grpc/GrpcQueryServerTest.java: ## @@ -0,0 +1,69 @@ +package org.apache.pinot.core.transport.grpc; + +im

Re: [PR] Add Tenants Info to Rebalance API summary [pinot]

2025-03-24 Thread via GitHub
J-HowHuang commented on code in PR #15284: URL: https://github.com/apache/pinot/pull/15284#discussion_r2011028375 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -699,6 +714,26 @@ private RebalanceSummaryResult calc

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-03-24 Thread via GitHub
deepthi912 commented on PR #15335: URL: https://github.com/apache/pinot/pull/15335#issuecomment-2749513216 @klsince I agree with your point especially on the server side, there is risk of not solving the OOM problem when let's say ingestion is happening on the table when it's trying to comm

Re: [PR] Fix needReload rebalance pre-check to only fetch status from currently assigned servers [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15360: URL: https://github.com/apache/pinot/pull/15360#issuecomment-2749499907 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15360?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Precheck Rebalance Config [pinot]

2025-03-24 Thread via GitHub
somandal commented on code in PR #15317: URL: https://github.com/apache/pinot/pull/15317#discussion_r2010970056 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalancePreChecker.java: ## @@ -279,6 +299,48 @@ private RebalancePreChecker

(pinot) branch dependabot/maven/kotlin.stdlib.version-2.1.20 updated (dd1aaccc05 -> e9e81fbdb9)

2025-03-24 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/kotlin.stdlib.version-2.1.20 in repository https://gitbox.apache.org/repos/asf/pinot.git discard dd1aaccc05 Bump kotlin.stdlib.version from 2.0.21 to 2.1.20 add 4ce0ca

Re: [PR] Precheck Rebalance Config [pinot]

2025-03-24 Thread via GitHub
somandal commented on code in PR #15317: URL: https://github.com/apache/pinot/pull/15317#discussion_r2010963378 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalancePreChecker.java: ## @@ -279,6 +299,48 @@ private RebalancePreChecker

Re: [PR] Consumes segments in strict order of sequence number [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15261: URL: https://github.com/apache/pinot/pull/15261#discussion_r2010941704 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java: ## @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends

Re: [PR] Fix needReload rebalance pre-check to only fetch status from currently assigned servers [pinot]

2025-03-24 Thread via GitHub
somandal commented on PR #15360: URL: https://github.com/apache/pinot/pull/15360#issuecomment-2749421784 cc @J-HowHuang @klsince @Jackie-Jiang @deepthi912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Fix needReload rebalance pre-check to only fetch status from currently assigned servers [pinot]

2025-03-24 Thread via GitHub
somandal opened a new pull request, #15360: URL: https://github.com/apache/pinot/pull/15360 This PR enhances the needReload pre-check done as part of table rebalancer to only fetch the status from the list of servers currently assigned based on the current assignment (ideal state). W

Re: [I] Segment Not Getting Uploaded to S3 (Deep Storage) [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on issue #15295: URL: https://github.com/apache/pinot/issues/15295#issuecomment-2749407605 cc @swaminathanmanish @KKcorps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Unable to change the default storage path in pinot [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on issue #15301: URL: https://github.com/apache/pinot/issues/15301#issuecomment-2749410586 You might get more help from the Pinot Slack troubleshooting channel -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Warn about numGroups above threshold [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on PR #15278: URL: https://github.com/apache/pinot/pull/15278#issuecomment-2749387891 What is the intention of this PR? Within a segment the group count should not go over the `numGroupsLimit`, will it ever hit the warning limit? Logging a warning is not very user f

Re: [PR] HELM: Add TLS-only support and publish 0.3.3 Chart version [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15356: URL: https://github.com/apache/pinot/pull/15356#discussion_r2010900795 ## helm/pinot/templates/broker/ingress-v1.yaml: ## @@ -20,7 +20,7 @@ {{- if .Values.broker.ingress.v1.enabled }} {{- $ingressPath := .Values.broker.ingress.v1.pa

Re: [PR] Simplify the Helix state model [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on PR #15341: URL: https://github.com/apache/pinot/pull/15341#issuecomment-2749138455 Closing as it might not work as expected -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/'s getChildren response size [pinot]

2025-03-24 Thread via GitHub
rhodo commented on PR #15345: URL: https://github.com/apache/pinot/pull/15345#issuecomment-2749267004 > One concern is that constantly reading the segment list of a large table itself may put pressure on the controller and zookeeper. Is it possible to track this metric when segments are add

Re: [PR] Add Tenants Info to Rebalance API summary [pinot]

2025-03-24 Thread via GitHub
somandal commented on code in PR #15284: URL: https://github.com/apache/pinot/pull/15284#discussion_r2010790936 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -699,6 +714,26 @@ private RebalanceSummaryResult calcul

Re: [PR] Add Tenants Info to Rebalance API summary [pinot]

2025-03-24 Thread via GitHub
somandal commented on code in PR #15284: URL: https://github.com/apache/pinot/pull/15284#discussion_r2010790936 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -699,6 +714,26 @@ private RebalanceSummaryResult calcul

Re: [PR] Simplify the Helix state model [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang closed pull request #15341: Simplify the Helix state model URL: https://github.com/apache/pinot/pull/15341 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Bump kotlin.stdlib.version from 2.0.21 to 2.1.20 [pinot]

2025-03-24 Thread via GitHub
dependabot[bot] commented on PR #15325: URL: https://github.com/apache/pinot/pull/15325#issuecomment-2749101595 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency by

Re: [PR] Add metrics to track external view's node size [pinot]

2025-03-24 Thread via GitHub
xiangfu0 merged PR #15338: URL: https://github.com/apache/pinot/pull/15338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

(pinot) branch master updated (07d5b81964 -> d772d20eb6)

2025-03-24 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 07d5b81964 Bump com.google.cloud:libraries-bom from 26.56.0 to 26.57.0 (#15355) add d772d20eb6 Add metrics to tr

Re: [PR] Bump kotlin.stdlib.version from 2.0.21 to 2.1.20 [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang closed pull request #15325: Bump kotlin.stdlib.version from 2.0.21 to 2.1.20 URL: https://github.com/apache/pinot/pull/15325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Added support to apply record enrichers before complex transformations [pinot]

2025-03-24 Thread via GitHub
rajagopr opened a new pull request, #15359: URL: https://github.com/apache/pinot/pull/15359 Make https://github.com/apache/pinot/pull/15267 prod ready. Added support to apply enrichments to incoming records before the record is passed on to the Complex transformer. With this change, a

Re: [PR] Bump kotlin.stdlib.version from 2.0.21 to 2.1.20 [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on PR #15325: URL: https://github.com/apache/pinot/pull/15325#issuecomment-2749072374 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

(pinot) branch dependabot/maven/com.google.cloud-libraries-bom-26.57.0 deleted (was 132b2ed6ed)

2025-03-24 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.google.cloud-libraries-bom-26.57.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 132b2ed6ed Bump com.google.cloud:libraries-bom from 26.56.0 to 26.57.

(pinot) branch master updated: Bump com.google.cloud:libraries-bom from 26.56.0 to 26.57.0 (#15355)

2025-03-24 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 07d5b81964 Bump com.google.cloud:libraries-bom fr

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-03-24 Thread via GitHub
deepthi912 commented on code in PR #15335: URL: https://github.com/apache/pinot/pull/15335#discussion_r2010712060 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMServerHandler.java: ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (AS

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-03-24 Thread via GitHub
deepthi912 commented on code in PR #15335: URL: https://github.com/apache/pinot/pull/15335#discussion_r2010704030 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java: ## @@ -132,6 +132,8 @@ public enum ServerMeter implements AbstractMetrics.Meter {

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2010689889 ## pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/SegmentZKMetadata.java: ## @@ -309,6 +309,7 @@ public void setSizeThresholdToFlushSegment(in

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2010698203 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/IngestionConfigUtils.java: ## @@ -101,11 +101,38 @@ public static List> getStreamConfigMaps(TableConfig tabl

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2010692538 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -456,10 +453,9 @@ private boolean addS

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2010692538 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -456,10 +453,9 @@ private boolean addS

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-03-24 Thread via GitHub
klsince commented on code in PR #15335: URL: https://github.com/apache/pinot/pull/15335#discussion_r2010670961 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMServerHandler.java: ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-03-24 Thread via GitHub
klsince commented on PR #15335: URL: https://github.com/apache/pinot/pull/15335#issuecomment-2748925301 is it safe to recover the server process from the OOM error here? To my knowledge, it's usually not a good idea (e.g. https://stackoverflow.com/questions/3058198/can-the-jvm-recover-from-

Re: [PR] Allow controller to skip PinotFS setup test during boot [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15358: URL: https://github.com/apache/pinot/pull/15358#issuecomment-2748800153 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15358?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-03-24 Thread via GitHub
Jackie-Jiang commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2010531335 ## pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java: ## @@ -147,9 +159,18 @@ private BrokerData getBrokerData(Map> responses)

Re: [PR] add MetricsExecutor to track number of tasks started and finished in MSE [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15357: URL: https://github.com/apache/pinot/pull/15357#issuecomment-2748696875 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15357?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] Allow controller to skip PinotFS setup test during boot [pinot]

2025-03-24 Thread via GitHub
rseetham opened a new pull request, #15358: URL: https://github.com/apache/pinot/pull/15358 This is to fix the issue #15127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Close `ChunkReaderContext` to improve projection/scan based filter memory for raw columns [pinot]

2025-03-24 Thread via GitHub
itschrispeck commented on code in PR #15139: URL: https://github.com/apache/pinot/pull/15139#discussion_r2008339976 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/ChunkReaderContext.java: ## @@ -52,11 +51,8 @@ public ChunkReader

[PR] add MetricsExecutor to track number of tasks started and finished in MSE [pinot]

2025-03-24 Thread via GitHub
gortiz opened a new pull request, #15357: URL: https://github.com/apache/pinot/pull/15357 This PR adds 4 new metrics to track the number of tasks started and finished in both submission and runner MSE thread runners. These metrics will be useful for analyzing the impact of different c

Re: [PR] HELM: Add TLS-only support and publish 0.3.3 Chart version [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15356: URL: https://github.com/apache/pinot/pull/15356#issuecomment-2748363223 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15356?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] HELM: Add TLS-only support and publish 0.3.3 Chart version [pinot]

2025-03-24 Thread via GitHub
PovilasV1 opened a new pull request, #15356: URL: https://github.com/apache/pinot/pull/15356 Reference issue - https://github.com/apache/pinot/issues/10091 This PR improves the Helm chart's TLS configuration capabilities by: Adding configurable port settings for all components (

Re: [PR] Revert "Consolidate Auth and Validation in SSE Request Handler" [pinot]

2025-03-24 Thread via GitHub
gortiz commented on PR #15346: URL: https://github.com/apache/pinot/pull/15346#issuecomment-2747876372 > Specifically, it removed the use of accessControl.authorize with the brokerRequest parameter ([previous usage](https://github.com/apache/pinot/pull/15240/files#diff-06e2bf5ea89ac7b1f67fa

Re: [I] roundDecimal() fails for null FLOAT values [pinot]

2025-03-24 Thread via GitHub
mghildiy commented on issue #15255: URL: https://github.com/apache/pinot/issues/15255#issuecomment-2748110417 CSV I ingested has null value defined as: ``` floatfield 1.43 , ``` I assume this is the standard way of having nulls in pinot. -- This is an automate

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.6 deleted (was c0b1aa7cae)

2025-03-24 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.6 in repository https://gitbox.apache.org/repos/asf/pinot.git was c0b1aa7cae Bump software.amazon.awssdk:bom from 2.31.5 to 2.31.6 The revi

(pinot) branch master updated: Bump software.amazon.awssdk:bom from 2.31.5 to 2.31.6 (#15354)

2025-03-24 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 8d7841b859 Bump software.amazon.awssdk:bom from 2

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.5 to 2.31.6 [pinot]

2025-03-24 Thread via GitHub
tibrewalpratik17 merged PR #15354: URL: https://github.com/apache/pinot/pull/15354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@p

Re: [I] roundDecimal() fails for null FLOAT values [pinot]

2025-03-24 Thread via GitHub
pauladams12345 commented on issue #15255: URL: https://github.com/apache/pinot/issues/15255#issuecomment-2748088393 @Jackie-Jiang > Thanks for reporting the issue. Have you tried this on a DOUBLE dimension with null values? Yes, `floatDecimal` works correctly on DOUBLE columns.

Re: [PR] Fix array agg functions to use existing result holder when present [pinot]

2025-03-24 Thread via GitHub
yashmayya merged PR #15308: URL: https://github.com/apache/pinot/pull/15308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

(pinot) branch master updated (52c5330f4e -> 23d5146e92)

2025-03-24 Thread yashmayya
This is an automated email from the ASF dual-hosted git repository. yashmayya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 52c5330f4e Initialize MSE worker thread context for TSE queries (#15353) add 23d5146e92 Fix array agg function

Re: [PR] Fix array agg functions to use existing result holder when present [pinot]

2025-03-24 Thread via GitHub
yashmayya commented on code in PR #15308: URL: https://github.com/apache/pinot/pull/15308#discussion_r2010088867 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/array/ArrayAggDistinctDoubleFunction.java: ## @@ -39,14 +39,15 @@ public void aggregate(i

Re: [I] Inconsistent results given by array_agg vs group_by [pinot]

2025-03-24 Thread via GitHub
yashmayya closed issue #15238: Inconsistent results given by array_agg vs group_by URL: https://github.com/apache/pinot/issues/15238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Allows consumption during build for Dedup [pinot]

2025-03-24 Thread via GitHub
noob-se7en commented on code in PR #15296: URL: https://github.com/apache/pinot/pull/15296#discussion_r2010054241 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1723,6 +1725,15 @@ public RealtimeSegmentDataManager(

(pinot) branch master updated: Initialize MSE worker thread context for TSE queries (#15353)

2025-03-24 Thread yashmayya
This is an automated email from the ASF dual-hosted git repository. yashmayya pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 52c5330f4e Initialize MSE worker thread contex

Re: [PR] Initialize MSE worker thread context for TSE queries [pinot]

2025-03-24 Thread via GitHub
yashmayya merged PR #15353: URL: https://github.com/apache/pinot/pull/15353 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

Re: [PR] Revert "Consolidate Auth and Validation in SSE Request Handler" [pinot]

2025-03-24 Thread via GitHub
gortiz merged PR #15346: URL: https://github.com/apache/pinot/pull/15346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

(pinot) branch master updated (23e06e3f4f -> 50958ca138)

2025-03-24 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 23e06e3f4f feat(KafkaConfluentSchemaRegistryJsonMessageDecoder): added the JSON confluent schema decoder (#15273)

[PR] Bump com.google.cloud:libraries-bom from 26.56.0 to 26.57.0 [pinot]

2025-03-24 Thread via GitHub
dependabot[bot] opened a new pull request, #15355: URL: https://github.com/apache/pinot/pull/15355 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.56.0 to 26.57.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bom/

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.6 created (now c0b1aa7cae)

2025-03-24 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.6 in repository https://gitbox.apache.org/repos/asf/pinot.git at c0b1aa7cae Bump software.amazon.awssdk:bom from 2.31.5 to 2.31.6 No n

(pinot) branch dependabot/maven/com.google.cloud-libraries-bom-26.57.0 created (now 132b2ed6ed)

2025-03-24 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.cloud-libraries-bom-26.57.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 132b2ed6ed Bump com.google.cloud:libraries-bom from 26.56.0 to 26

[PR] Bump software.amazon.awssdk:bom from 2.31.5 to 2.31.6 [pinot]

2025-03-24 Thread via GitHub
dependabot[bot] opened a new pull request, #15354: URL: https://github.com/apache/pinot/pull/15354 Bumps software.amazon.awssdk:bom from 2.31.5 to 2.31.6. Most Recent Ignore Conditions Applied to This Pull Request | Dependency Name | Ignore Conditions | | --- | --- | |

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-03-24 Thread via GitHub
gortiz commented on code in PR #15335: URL: https://github.com/apache/pinot/pull/15335#discussion_r2010013239 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMeter.java: ## @@ -132,6 +132,8 @@ public enum ServerMeter implements AbstractMetrics.Meter { TOTA

Re: [PR] Initialize MSE worker thread context for TSE queries [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15353: URL: https://github.com/apache/pinot/pull/15353#issuecomment-2747855990 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15353?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/'s getChildren response size [pinot]

2025-03-24 Thread via GitHub
vrajat commented on PR #15345: URL: https://github.com/apache/pinot/pull/15345#issuecomment-2747816587 One concern is that constantly reading the segment list of a large table itself may put pressure on the controller and zookeeper. Is it possible to track this metric when segments are adde

Re: [PR] switch log.debug to log.info on query request [pinot]

2025-03-24 Thread via GitHub
gortiz commented on PR #15264: URL: https://github.com/apache/pinot/pull/15264#issuecomment-2747803634 > We need to apply the same throttling as the QueryLogger does. This change can flood the log for super high QPS use cases (imagine over 10k query lines per second) I disagree. Not

[PR] Initialize MSE worker thread context for TSE queries [pinot]

2025-03-24 Thread via GitHub
yashmayya opened a new pull request, #15353: URL: https://github.com/apache/pinot/pull/15353 - https://github.com/apache/pinot/pull/15180 didn't add the MSE worker thread context initialization for TSE queries. - This isn't an issue for actual TSE queries in real clusters since "strict"

Re: [PR] Support jsonKeyValueArrayToMap function [pinot]

2025-03-24 Thread via GitHub
codecov-commenter commented on PR #15352: URL: https://github.com/apache/pinot/pull/15352#issuecomment-2747525719 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15352?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] Support jsonKeyValueArrayToMap function [pinot]

2025-03-24 Thread via GitHub
xiangfu0 opened a new pull request, #15352: URL: https://github.com/apache/pinot/pull/15352 Extract an array of key-value maps to a map. E.g. - input: `[{"key": "k1", "value": "v1"}, {"key": "k2", "value": "v2"}, {"key": "k3", "value": "v3"}]` - output: `{"k1": "v1", "k2": "v2", "k

Re: [PR] Consumes segments in strict order of sequence number [pinot]

2025-03-24 Thread via GitHub
noob-se7en commented on code in PR #15261: URL: https://github.com/apache/pinot/pull/15261#discussion_r2009576026 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java: ## @@ -0,0 +1,241 @@ +/** + * Licensed to the Apache Softwar

Re: [PR] Consumes segments in strict order of sequence number [pinot]

2025-03-24 Thread via GitHub
noob-se7en commented on code in PR #15261: URL: https://github.com/apache/pinot/pull/15261#discussion_r2009563467 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -885,4 +921,21 @@ private void validate(TableConfig tabl