(pinot) branch master updated (4a937c1c02 -> f3c4a29706)

2025-04-05 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 4a937c1c02 Bump com.google.cloud:libraries-bom from 26.58.0 to 26.59.0 (#15424) add f3c4a29706 Bump org.asynchttp

Re: [PR] Upgrade Calcite to 1.39.0 [pinot]

2025-04-05 Thread via GitHub
yashmayya commented on code in PR #15263: URL: https://github.com/apache/pinot/pull/15263#discussion_r2000856836 ## pinot-query-planner/src/test/resources/queries/JoinPlans.json: ## @@ -405,10 +405,10 @@ "\nLogicalProject(col1=[$0])", "\n LogicalJoin(condi

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.5 deleted (was 0e26c60d51)

2025-04-05 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.5 in repository https://gitbox.apache.org/repos/asf/pinot.git was 0e26c60d51 Bump software.amazon.awssdk:bom from 2.31.4 to 2.31.5 The revi

Re: [PR] Fix Direct Memory OOM on Server [pinot]

2025-04-05 Thread via GitHub
gortiz commented on PR #15335: URL: https://github.com/apache/pinot/pull/15335#issuecomment-2771487196 Idea: instead (or maybe complementary) of trying to free Netty's memory when we are getting out of it... could we try to limit the amount of memory being used by Netty? By default Netty's

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2025722585 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -140,10 +156,20 @@ public TableRebalancer(HelixManager h

[PR] Precheck Rebalance Config [pinot]

2025-04-05 Thread via GitHub
J-HowHuang opened a new pull request, #15317: URL: https://github.com/apache/pinot/pull/15317 ## Description Some options in `RebalanceConfig` in rebalance API need to be double checked before execution: - if `bestEffort=true`, flag it - if `downtime=true`, but replication > 1,

Re: [PR] Consumes segments in strict order of sequence number [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on code in PR #15261: URL: https://github.com/apache/pinot/pull/15261#discussion_r2007452776 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java: ## @@ -0,0 +1,251 @@ +/** + * Licensed to the Apache Softwar

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2019839835 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSummaryResult.java: ## @@ -227,19 +224,14 @@ public void increaseNumServerP

Re: [PR] Fix real-time table segment download [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15316: URL: https://github.com/apache/pinot/pull/15316#discussion_r2012517925 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -564,77 +564,74 @@ private void doAddConsumingSegment(

Re: [PR] Deprecates AllowConsumptionDuringCommit Config [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15452: URL: https://github.com/apache/pinot/pull/15452#issuecomment-2775859150 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15452?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Query thread context and cid [pinot]

2025-04-05 Thread via GitHub
yashmayya commented on code in PR #15180: URL: https://github.com/apache/pinot/pull/15180#discussion_r2000150562 ## pinot-query-planner/src/main/java/org/apache/pinot/query/function/InternalMseFunctions.java: ## @@ -0,0 +1,60 @@ +/** + * Licensed to the Apache Software Foundatio

(pinot) branch dependabot/maven/org.apache-apache-34 created (now e860823792)

2025-04-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache-apache-34 in repository https://gitbox.apache.org/repos/asf/pinot.git at e860823792 Bump org.apache:apache from 33 to 34 No new revisions were added by this

Re: [PR] Fix up parsing of controller timers tableRebalanceExecutionTimeMs and cronSchedulerJobExecutionTimeMs [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15330: URL: https://github.com/apache/pinot/pull/15330#discussion_r2007876312 ## docker/images/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml: ## @@ -48,6 +48,21 @@ rules: table: "$2$4" tableType: "$5" taskType: "$6" +

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2019854520 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSummaryResult.java: ## @@ -138,6 +134,7 @@ public int getSegmentsUnchanged(

Re: [PR] Migrate H3 v3 api to v4 [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15313: URL: https://github.com/apache/pinot/pull/15313#issuecomment-2738013289 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15313?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Improved retention management for hybrid tables [pinot]

2025-04-05 Thread via GitHub
rajagopr commented on code in PR #15257: URL: https://github.com/apache/pinot/pull/15257#discussion_r2006159617 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/BrokerServiceHelper.java: ## @@ -0,0 +1,116 @@ +/** + * Licensed to the Apache Software Foundation (

Re: [PR] Emit metrics for each query error code returned by the broker [pinot]

2025-04-05 Thread via GitHub
gortiz commented on code in PR #15387: URL: https://github.com/apache/pinot/pull/15387#discussion_r2018026339 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMetrics.java: ## @@ -84,4 +91,23 @@ protected BrokerMeter[] getMeters() { protected BrokerGauge[]

Re: [PR] Enhance rebalance progress stats [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15266: URL: https://github.com/apache/pinot/pull/15266#discussion_r2021940809 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -271,17 +315,279 @@ public static TableRe

[PR] Close grpc connection after query in ClusterTest [pinot]

2025-04-05 Thread via GitHub
shounakmk219 opened a new pull request, #15394: URL: https://github.com/apache/pinot/pull/15394 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] [Refactor] Move segment reload logic into TableDataManager [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang opened a new pull request, #15441: URL: https://github.com/apache/pinot/pull/15441 - Move segment reload handling from `HelixInstanceDataManager` into `BaseTableDataManager` - Rename `SegmentRefreshSemaphore` to `SegmentReloadSemaphore` and move it into `segment-local` modul

[PR] Deprecates AllowConsumptionDuringCommit Config [pinot]

2025-04-05 Thread via GitHub
noob-se7en opened a new pull request, #15452: URL: https://github.com/apache/pinot/pull/15452 Deprecate `_allowDedupConsumptionDuringCommit` and `_allowPartialUpsertConsumptionDuringCommit`. Because: 1. These configs are confusing because the name and behaviour of these config ar

Re: [PR] Close `ChunkReaderContext` to improve projection/scan based filter memory for raw columns [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15139: URL: https://github.com/apache/pinot/pull/15139#discussion_r2008120713 ## pinot-core/src/main/java/org/apache/pinot/core/common/BlockDocIdIterator.java: ## @@ -47,4 +47,11 @@ public interface BlockDocIdIterator { * @see {https://

Re: [PR] Enhance rebalance progress stats [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15266: URL: https://github.com/apache/pinot/pull/15266#discussion_r2021934047 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -567,6 +577,15 @@ private RebalanceResult doRebalance(Ta

Re: [PR] Make MSE leaf stage use cached TableConfig and Schema from TableDataManager [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15461: URL: https://github.com/apache/pinot/pull/15461#issuecomment-2777381756 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15461?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
J-HowHuang commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2021903313 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -120,16 +130,22 @@ */ public class TableRebalancer

Re: [PR] Moving dependency version declarations to maven properties [pinot]

2025-04-05 Thread via GitHub
gviedma commented on code in PR #15437: URL: https://github.com/apache/pinot/pull/15437#discussion_r2027080536 ## pom.xml: ## @@ -271,6 +271,24 @@ 2.5.3 0.11.0-pre2 0.3.2 +3.49.2 +2.4.21 +2.0.2 +1.3.0 +0.19.0 Review Comment: Suggest replaci

Re: [PR] Fix starvation in consumer lock [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on code in PR #15404: URL: https://github.com/apache/pinot/pull/15404#discussion_r2019977505 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -474,23 +473,26 @@ protected void doAddOnlineSegment(Str

[PR] Fix up parsing of controller timers tableRebalanceExecutionTimeMs and cronSchedulerJobExecutionTimeMs [pinot]

2025-04-05 Thread via GitHub
somandal opened a new pull request, #15330: URL: https://github.com/apache/pinot/pull/15330 Today the two controller timers `tableRebalanceExecutionTimeMs` and `cronSchedulerJobExecutionTimeMs` are showing up in Prometheus with the following type of metric names: - `pinot_controller

(pinot) branch master updated (9db45e364b -> 4f91c27120)

2025-04-05 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 9db45e364b [multistage] Adding Basic Constructs for Physical Optimization (#15371) add 4f91c27120 Adding ArrowRe

(pinot) branch dependabot/maven/org.apache.httpcomponents.core5-httpcore5-5.3.4 deleted (was 95a829d0e2)

2025-04-05 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a change to branch dependabot/maven/org.apache.httpcomponents.core5-httpcore5-5.3.4 in repository https://gitbox.apache.org/repos/asf/pinot.git was 95a829d0e2 Bump org.apache.httpcomponents.core5:httpcore5 fr

[PR] Bump org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25 from 1.3.0 to 1.4.0 [pinot]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #15397: URL: https://github.com/apache/pinot/pull/15397 Bumps org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25 from 1.3.0 to 1.4.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_s

Re: [PR] Allows consumption during build for Dedup [pinot]

2025-04-05 Thread via GitHub
KKcorps commented on code in PR #15296: URL: https://github.com/apache/pinot/pull/15296#discussion_r2011357076 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1066,7 +1067,7 @@ protected SegmentBuildDescriptor buil

Re: [PR] Fix real-time table segment download [pinot]

2025-04-05 Thread via GitHub
9aman commented on code in PR #15316: URL: https://github.com/apache/pinot/pull/15316#discussion_r2007038286 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -564,77 +562,59 @@ private void doAddConsumingSegment(String

Re: [PR] Enforce schema for all tables [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on PR #15333: URL: https://github.com/apache/pinot/pull/15333#issuecomment-2744111672 @jadami10 We enforcement is on schema existence and also schema name. We already have log and metrics tracking invliad schema since Pinot 1.1, which is done in #11574 -- This is

Re: [PR] Refresh segment consumption end time [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on code in PR #15419: URL: https://github.com/apache/pinot/pull/15419#discussion_r2021626001 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -747,6 +747,14 @@ public void run() { _segm

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.12 deleted (was 032aff0d1f)

2025-04-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.12 in repository https://gitbox.apache.org/repos/asf/pinot.git was 032aff0d1f Bump software.amazon.awssdk:bom from 2.31.11 to 2.31.12 T

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2012551388 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -867,28 +868,31 @@ private SegmentZKMe

Re: [PR] Fix real-time table segment download [pinot]

2025-04-05 Thread via GitHub
9aman commented on code in PR #15316: URL: https://github.com/apache/pinot/pull/15316#discussion_r2007012278 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/IngestionConfigUtils.java: ## @@ -100,8 +100,9 @@ public static List> getStreamConfigMaps(TableConfig tableCon

Re: [PR] [bugfix] Make clientId to be unique for PartitionGroupMetadataFetcher [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on PR #15393: URL: https://github.com/apache/pinot/pull/15393#issuecomment-2764325852 Seems like `AdminClient` will also try to register JXM with the same client id, which can cause the same problem. It was introduced in #14678. cc @suvodeep-pyne -- This is an aut

Re: [PR] Add metrics to track segment operation throttle thresholds and count of number of segments undergoing operation [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15392: URL: https://github.com/apache/pinot/pull/15392#discussion_r2021451659 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/BaseSegmentOperationsThrottler.java: ## @@ -41,17 +46,22 @@ public abstract class BaseSegment

Re: [PR] Add off-heap distinct set implementations [pinot]

2025-04-05 Thread via GitHub
siddharthteotia commented on PR #15205: URL: https://github.com/apache/pinot/pull/15205#issuecomment-2774494424 We would like to leverage this as we had done some experiments in the past with off-heap hashsets and the results were good. Do you have user docs ? -- This is an automa

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2012909684 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -453,6 +453,8 @@ public static class Broker { public static final String CO

Re: [PR] Bump net.alchim31.maven:scala-maven-plugin from 4.9.2 to 4.9.5 [pinot]

2025-04-05 Thread via GitHub
tibrewalpratik17 merged PR #15415: URL: https://github.com/apache/pinot/pull/15415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@p

Re: [PR] Enhance rebalance progress stats [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15266: URL: https://github.com/apache/pinot/pull/15266#discussion_r2021943999 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -271,17 +315,279 @@ public static TableRe

Re: [PR] feat: change display of rebalance response [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15331: URL: https://github.com/apache/pinot/pull/15331#issuecomment-2741165656 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15331?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

(pinot) branch master updated (8783eb1017 -> 6bbd5ddf86)

2025-04-05 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 8783eb1017 Bump software.amazon.awssdk:bom from 2.31.4 to 2.31.5 (#15344) add 6bbd5ddf86 Bump com.mycila:license-

Re: [PR] Cleanup controller and MSE broker entry points to keep error codes an… [pinot]

2025-04-05 Thread via GitHub
gortiz commented on code in PR #15277: URL: https://github.com/apache/pinot/pull/15277#discussion_r2005691321 ## pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java: ## @@ -278,53 +204,69 @@ private QueryEnvironment.QueryPlannerResult getQueryPlannerR

Re: [PR] [spark connector] replace the use of circe with jackson [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang merged PR #15321: URL: https://github.com/apache/pinot/pull/15321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: add getter for precheckcontext (#15339)

2025-04-05 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2f3612f45d add getter for precheckcontext (#15339

Re: [PR] Metrics for monitoring PROPERTYSTORE/SEGMENTS/'s getChildren response size [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15345: URL: https://github.com/apache/pinot/pull/15345#issuecomment-2744499959 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15345?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

(pinot) branch master updated (aaccf139ee -> 45263975aa)

2025-04-05 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from aaccf139ee Make more StartMinionCommand -config alias (#15307) add 45263975aa Migrate H3 v3 api to v4 (#15313) No

Re: [PR] Improve consumer lock [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on code in PR #15405: URL: https://github.com/apache/pinot/pull/15405#discussion_r2020090409 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java: ## @@ -71,39 +70,34 @@ public ConsumerCoordinator(boolean enforceC

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.11 to 2.31.14 [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang merged PR #15448: URL: https://github.com/apache/pinot/pull/15448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: Bump com.azure:azure-sdk-bom from 1.2.32 to 1.2.33 (#15412)

2025-04-05 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 5779c6ecff Bump com.azure:azure-sdk-bom from 1.2.

Re: [PR] Allows consumption during build for Dedup [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on code in PR #15296: URL: https://github.com/apache/pinot/pull/15296#discussion_r2004285251 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1723,6 +1725,15 @@ public RealtimeSegmentDataManager(

(pinot) branch master updated (404e50e887 -> 2196306385)

2025-04-05 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 404e50e887 Allow lazy init of the record reader in CompactedPinotSegmentRecordReader (#15428) add 2196306385 Bump

Re: [PR] Enforce schema for all tables [pinot]

2025-04-05 Thread via GitHub
jadami10 commented on PR #15333: URL: https://github.com/apache/pinot/pull/15333#issuecomment-2743553013 though what might be useful here is if you have a specific log to check for for Pinot 1.2 and 1.3 for everyone to be sure before they just try upgrading and find out live. -- This is

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
J-HowHuang commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2027597651 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -806,6 +848,153 @@ private List getServerTag(String se

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.5 created (now 0e26c60d51)

2025-04-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.5 in repository https://gitbox.apache.org/repos/asf/pinot.git at 0e26c60d51 Bump software.amazon.awssdk:bom from 2.31.4 to 2.31.5 No n

[I] Upgrade Pinot service source version to Java 21 [pinot]

2025-04-05 Thread via GitHub
gortiz opened a new issue, #15343: URL: https://github.com/apache/pinot/issues/15343 Since 1.0.0, Pinot dropped support for Java 8 and since then the minimum source and runtime Java version is 11. Here I propose increasing that minimum to Java 21 for Pinot 1.4.0. *This proposal doesn't

Re: [PR] Bump org.jacoco:jacoco-maven-plugin from 0.8.12 to 0.8.13 [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang merged PR #15449: URL: https://github.com/apache/pinot/pull/15449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Enhance rebalance progress stats [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15266: URL: https://github.com/apache/pinot/pull/15266#discussion_r2025459940 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -284,4 +328,306 @@ public static TableReb

Re: [PR] Improved retention management for hybrid tables [pinot]

2025-04-05 Thread via GitHub
rajagopr commented on code in PR #15257: URL: https://github.com/apache/pinot/pull/15257#discussion_r1999407556 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java: ## @@ -118,11 +127,20 @@ private void manageRetentionForTable

Re: [PR] Fix up parsing of controller timers tableRebalanceExecutionTimeMs and cronSchedulerJobExecutionTimeMs [pinot]

2025-04-05 Thread via GitHub
suddendust commented on code in PR #15330: URL: https://github.com/apache/pinot/pull/15330#discussion_r2006887838 ## docker/images/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml: ## @@ -48,6 +48,21 @@ rules: table: "$2$4" tableType: "$5" taskType: "$6"

Re: [PR] Improved retention management for hybrid tables [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15257: URL: https://github.com/apache/pinot/pull/15257#discussion_r2006134057 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java: ## @@ -199,6 +225,70 @@ private void manageRetentionForRe

Re: [PR] Minor Refactoring [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15419: URL: https://github.com/apache/pinot/pull/15419#discussion_r2025393695 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -747,6 +747,14 @@ public void run() { _se

(pinot) branch master updated: Disable reingestion for Pauseless dedup (#15383)

2025-04-05 Thread kharekartik
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 5550c24f1e Disable reingestion for Pauseless

Re: [PR] Add Disk Utilization Info into Rebalance API [pinot]

2025-04-05 Thread via GitHub
klsince commented on code in PR #15175: URL: https://github.com/apache/pinot/pull/15175#discussion_r2003988748 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalancePreChecker.java: ## @@ -42,19 +47,27 @@ public class DefaultRebalance

(pinot) branch dependabot/maven/kotlin.stdlib.version-2.1.20 deleted (was e9e81fbdb9)

2025-04-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/kotlin.stdlib.version-2.1.20 in repository https://gitbox.apache.org/repos/asf/pinot.git was e9e81fbdb9 Bump kotlin.stdlib.version from 2.0.21 to 2.1.20 The revisions

Re: [PR] Improve exceptions broker [pinot]

2025-04-05 Thread via GitHub
gortiz closed pull request #14994: Improve exceptions broker URL: https://github.com/apache/pinot/pull/14994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.2 to 2.31.3 [pinot]

2025-04-05 Thread via GitHub
tibrewalpratik17 merged PR #15309: URL: https://github.com/apache/pinot/pull/15309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@p

Re: [PR] Bump org.checkerframework:checker-qual from 3.49.1 to 3.49.2 [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang merged PR #15434: URL: https://github.com/apache/pinot/pull/15434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
J-HowHuang commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2019504487 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSummaryResult.java: ## @@ -240,6 +240,92 @@ public Map getServerSegmentC

Re: [PR] switch log.debug to log.info on query request [pinot]

2025-04-05 Thread via GitHub
albertobastos commented on code in PR #15264: URL: https://github.com/apache/pinot/pull/15264#discussion_r2007299497 ## pinot-broker/src/main/java/org/apache/pinot/broker/querylog/QueryLogger.java: ## @@ -88,9 +87,12 @@ public void log(QueryLogParams params) { queryLogBui

Re: [PR] Minor Refactoring [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15419: URL: https://github.com/apache/pinot/pull/15419#issuecomment-2767239399 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15419?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Close grpc connection after query in ClusterTest [pinot]

2025-04-05 Thread via GitHub
codecov-commenter commented on PR #15394: URL: https://github.com/apache/pinot/pull/15394#issuecomment-2760405821 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15394?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Support size based threshold for pauseless consumption [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang commented on code in PR #15347: URL: https://github.com/apache/pinot/pull/15347#discussion_r2012554061 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1170,14 +1178,15 @@ public void ensur

Re: [PR] Add BrokerGrpcServer for a streaming response fetching endpoint [pinot]

2025-04-05 Thread via GitHub
xiangfu0 commented on code in PR #15088: URL: https://github.com/apache/pinot/pull/15088#discussion_r2011189976 ## pinot-common/src/main/java/org/apache/pinot/common/compression/CompressionFactory.java: ## @@ -0,0 +1,49 @@ +/** + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Add metrics to track external view's node size [pinot]

2025-04-05 Thread via GitHub
klsince commented on code in PR #15338: URL: https://github.com/apache/pinot/pull/15338#discussion_r2006511133 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java: ## @@ -233,29 +236,34 @@ public void testSegmen

Re: [PR] roundDecimal() fails for null FLOAT values [pinot]

2025-04-05 Thread via GitHub
mghildiy commented on code in PR #15377: URL: https://github.com/apache/pinot/pull/15377#discussion_r2019686047 ## pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/RoundDecimalTransformFunction.java: ## @@ -90,8 +90,12 @@ public double[] transformToDoub

Re: [PR] Unblocks helix thread for online segment [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on PR #15246: URL: https://github.com/apache/pinot/pull/15246#issuecomment-2737976979 Closing this in favour of https://github.com/apache/pinot/pull/15261 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] [multistage] Adding Basic Constructs for Physical Optimization [pinot]

2025-04-05 Thread via GitHub
ankitsultana commented on code in PR #15371: URL: https://github.com/apache/pinot/pull/15371#discussion_r2025365497 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/physical/v2/TableScanMetadata.java: ## @@ -0,0 +1,70 @@ +/** + * Licensed to the Apache Softwar

Re: [PR] Direct mse query server [pinot]

2025-04-05 Thread via GitHub
gortiz commented on code in PR #15445: URL: https://github.com/apache/pinot/pull/15445#discussion_r2028243221 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/server/QueryServer.java: ## @@ -159,37 +164,49 @@ public void submit(Worker.QueryRequest request, St

[PR] Fix Direct Memory OOM on Server [pinot]

2025-04-05 Thread via GitHub
deepthi912 opened a new pull request, #15335: URL: https://github.com/apache/pinot/pull/15335 **Context:** Some expensive queries from broker caused servers to go into OOM state causing the consuming segments to stop consumption and we had to increase the direct max memory on the servers an

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
somandal commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2025329513 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSummaryResult.java: ## @@ -306,18 +297,120 @@ public Map getServerSegmentC

(pinot) branch master updated (1a211d8244 -> 624167dea9)

2025-04-05 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 1a211d8244 Bump com.puppycrawl.tools:checkstyle from 10.21.4 to 10.22.0 (#15413) add 624167dea9 Bump org.ow2.asm:

Re: [PR] Add metrics to track segment operation throttle thresholds and count of number of segments undergoing operation [pinot]

2025-04-05 Thread via GitHub
klsince commented on code in PR #15392: URL: https://github.com/apache/pinot/pull/15392#discussion_r2021462537 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/BaseSegmentOperationsThrottler.java: ## @@ -41,17 +46,22 @@ public abstract class BaseSegmentO

[PR] Add getter for PreCheckContext [pinot]

2025-04-05 Thread via GitHub
J-HowHuang opened a new pull request, #15339: URL: https://github.com/apache/pinot/pull/15339 For the change in https://github.com/apache/pinot/pull/15175, protected members in `PreCheckContext` should be private and use proper getters. -- This is an automated message from the Apache Git

[PR] Update RebalanceResult field order [pinot]

2025-04-05 Thread via GitHub
J-HowHuang opened a new pull request, #15305: URL: https://github.com/apache/pinot/pull/15305 Moved `preChecksResult` and `rebalanceSummaryResult` to the top. As it should be looked up first. ``` { "jobId": "2982d8ba-f578-4172-b393-0916ba6a3324", "status": "NO_OP", "desc

(pinot) branch dependabot/maven/net.alchim31.maven-scala-maven-plugin-4.9.5 created (now 087ec18d0a)

2025-04-05 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/net.alchim31.maven-scala-maven-plugin-4.9.5 in repository https://gitbox.apache.org/repos/asf/pinot.git at 087ec18d0a Bump net.alchim31.maven:scala-maven-plugin from 4

Re: [PR] (Minor) fix logs [pinot]

2025-04-05 Thread via GitHub
noob-se7en commented on code in PR #15438: URL: https://github.com/apache/pinot/pull/15438#discussion_r2025647189 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -621,7 +621,7 @@ private void commitSegm

Re: [PR] Moving dependency version declarations to maven properties [pinot]

2025-04-05 Thread via GitHub
gviedma commented on code in PR #15437: URL: https://github.com/apache/pinot/pull/15437#discussion_r2027112915 ## pom.xml: ## @@ -271,6 +271,24 @@ 2.5.3 0.11.0-pre2 0.3.2 +3.49.2 +2.4.21 +2.0.2 +1.3.0 +0.19.0 +8.5.15 +4.0.0 +2.4 +

Re: [PR] Substitute TransferableBlock with MseBlock [pinot]

2025-04-05 Thread via GitHub
gortiz commented on code in PR #15245: URL: https://github.com/apache/pinot/pull/15245#discussion_r2024207226 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/MseBlock.java: ## @@ -0,0 +1,178 @@ +/** + * Licensed to the Apache Software Foundation (ASF)

Re: [I] HDFS as Deep Storage Not Working in Pinot 1.3.0 [pinot]

2025-04-05 Thread via GitHub
minwoo-jung commented on issue #15215: URL: https://github.com/apache/pinot/issues/15215#issuecomment-2742102845 @xiangfu0 **Thank you for quickly creating the patch and building the Docker image.** I will test it with the provided patch version. -- This is an automated message from

[PR] Bump org.apache.kafka:kafka-clients from 2.8.2 to 3.7.1 [pinot]

2025-04-05 Thread via GitHub
dependabot[bot] opened a new pull request, #15337: URL: https://github.com/apache/pinot/pull/15337 Bumps org.apache.kafka:kafka-clients from 2.8.2 to 3.7.1. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.ap

Re: [PR] Consuming Rebalance Summary [pinot]

2025-04-05 Thread via GitHub
klsince commented on code in PR #15368: URL: https://github.com/apache/pinot/pull/15368#discussion_r2025635256 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -270,6 +275,7 @@ public String load(String instanceId) {

Re: [PR] Make stats system more resilient in MSE [pinot]

2025-04-05 Thread via GitHub
albertobastos commented on code in PR #15312: URL: https://github.com/apache/pinot/pull/15312#discussion_r2005236269 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingMultiStreamConsumer.java: ## @@ -244,11 +244,14 @@ private E readBlockO

Re: [PR] Bump org.webjars:swagger-ui from 5.20.1 to 5.20.3 [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang merged PR #15451: URL: https://github.com/apache/pinot/pull/15451 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Fix wording to explain minAvailableReplicas better in rebalance UI [pinot]

2025-04-05 Thread via GitHub
Jackie-Jiang merged PR #15367: URL: https://github.com/apache/pinot/pull/15367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] roundDecimal() fails for null FLOAT values [pinot]

2025-04-05 Thread via GitHub
mghildiy commented on issue #15255: URL: https://github.com/apache/pinot/issues/15255#issuecomment-2745307119 I have added a schema via UI: ``` { "schemaName": "my-test-schema", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name":

  1   2   >