(pinot) branch master updated: Fix up elapsed time calculation for rebalance progress stats when carry over are present (#15674)

2025-04-30 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c3e118f1fa Fix up elapsed time calculation for r

Re: [PR] Fix up elapsed time calculation for rebalance progress stats when carry over are present [pinot]

2025-04-30 Thread via GitHub
xiangfu0 merged PR #15674: URL: https://github.com/apache/pinot/pull/15674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Fix Select-Star failures during schema evolution and segment reload [pinot]

2025-04-30 Thread via GitHub
praveenc7 commented on PR #15350: URL: https://github.com/apache/pinot/pull/15350#issuecomment-2844116537 > Is there a design doc for this? I’m thinking of solving this for the general case—whenever a newly-added column is accessed, we could inject a virtual column on the fly whose

Re: [PR] Japicmp implementation in pinot-spi [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15684: URL: https://github.com/apache/pinot/pull/15684#issuecomment-2843974027 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15684?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Allow segment pre-processing failure during segment loading phase [pinot]

2025-04-30 Thread via GitHub
xiangfu0 commented on PR #15672: URL: https://github.com/apache/pinot/pull/15672#issuecomment-2843960969 > hey @xiangfu0 would it make sense to put this behind a config and disable it by default? so that this behavior change doesn't take anyone by surprise? I was also thinking about t

Re: [PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
xiangfu0 merged PR #15676: URL: https://github.com/apache/pinot/pull/15676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Japicmp implementation in pinot-spi [pinot]

2025-04-30 Thread via GitHub
xiangfu0 commented on PR #15684: URL: https://github.com/apache/pinot/pull/15684#issuecomment-2843933424 For the jar file to compare, can we download it as part of the CI or use the previous release version? -- This is an automated message from the Apache Git Service. To respond to the me

(pinot) branch master updated: Set conservative default for gRPC query sever worker thread count (#15676)

2025-04-30 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 8c2cefb4d5 Set conservative default for gRPC que

Re: [PR] Add `maxBytesSize` config to limit mutable json index heap usage [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15685: URL: https://github.com/apache/pinot/pull/15685#issuecomment-2843911096 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15685?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[PR] Add `maxBytesSize` config to limit mutable json index heap usage [pinot]

2025-04-30 Thread via GitHub
itschrispeck opened a new pull request, #15685: URL: https://github.com/apache/pinot/pull/15685 Add `maxBytesSize` config to limit mutable json index heap usage. When data is unpredictable, we often see JSON index can spike to use a substantial amount of heap. This PR adds a config to limit

Re: [PR] Fix Select-Star failures during schema evolution and segment reload [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on PR #15350: URL: https://github.com/apache/pinot/pull/15350#issuecomment-2843797064 Is there a design doc for this? I'm thinking of solving this for general case whenever the new added column is accessed. We can potentially add a virtual column on the fly, where a

Re: [PR] Japicmp implementation in pinot-spi [pinot]

2025-04-30 Thread via GitHub
UOETianleZhang commented on code in PR #15684: URL: https://github.com/apache/pinot/pull/15684#discussion_r2069704603 ## pinot-spi/pom.xml: ## @@ -124,6 +124,72 @@ true + +com.github.siom79.japicmp +japicmp-maven-plugin +

Re: [PR] Move Table Routing logic in SSE to TableRoute class [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15388: URL: https://github.com/apache/pinot/pull/15388#discussion_r2069700732 ## pinot-query-planner/src/main/java/org/apache/pinot/query/routing/table/TableRouteProvider.java: ## @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Fo

Re: [PR] Move Table Routing logic in SSE to TableRoute class [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15388: URL: https://github.com/apache/pinot/pull/15388#discussion_r206973 ## pinot-core/src/main/java/org/apache/pinot/core/transport/TableRouteInfo.java: ## @@ -0,0 +1,198 @@ +/** + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Move Table Routing logic in SSE to TableRoute class [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15388: URL: https://github.com/apache/pinot/pull/15388#discussion_r2069697456 ## pinot-core/src/main/java/org/apache/pinot/core/transport/TableRouteInfo.java: ## @@ -0,0 +1,198 @@ +/** + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Move Table Routing logic in SSE to TableRoute class [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15388: URL: https://github.com/apache/pinot/pull/15388#discussion_r2069696864 ## pinot-core/src/main/java/org/apache/pinot/core/transport/TableRouteInfo.java: ## @@ -0,0 +1,198 @@ +/** + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Changes to Enable Pluggable forward Index and Map Index Filter Operator for MAP column [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15526: URL: https://github.com/apache/pinot/pull/15526#discussion_r2069618976 ## pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java: ## @@ -238,6 +248,8 @@ private BaseFilterOperator constructPhysicalOperator(FilterCont

Re: [PR] Japicmp implementation in pinot-spi [pinot]

2025-04-30 Thread via GitHub
matvj250 commented on code in PR #15684: URL: https://github.com/apache/pinot/pull/15684#discussion_r2069642124 ## pinot-spi/pom.xml: ## @@ -124,6 +124,68 @@ true + +com.github.siom79.japicmp +japicmp-maven-plugin +0.23.

Re: [PR] Reduce segment metadata memory overhead. [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15591: URL: https://github.com/apache/pinot/pull/15591#discussion_r2069598453 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BigNumberOfSegmentsIntegrationTest.java: ## @@ -0,0 +1,266 @@ +/** + * Licensed to the

Re: [PR] Japicmp implementation in pinot-spi [pinot]

2025-04-30 Thread via GitHub
UOETianleZhang commented on code in PR #15684: URL: https://github.com/apache/pinot/pull/15684#discussion_r2069552185 ## pinot-spi/pom.xml: ## @@ -124,6 +124,68 @@ true + +com.github.siom79.japicmp +japicmp-maven-plugin +

[PR] Japicmp implementation in pinot-spi [pinot]

2025-04-30 Thread via GitHub
matvj250 opened a new pull request, #15684: URL: https://github.com/apache/pinot/pull/15684 Implementing japicmp in the pinot-spi module to fail builds (i.e. running mvn clean) when incompatible SPI changes are made, such as removing methods, changing method return types, etc. Compatible SP

Re: [PR] Metrics that Tracks the Progress of each Table Rebalance Job [pinot]

2025-04-30 Thread via GitHub
J-HowHuang commented on PR #15518: URL: https://github.com/apache/pinot/pull/15518#issuecomment-2843278700 This PR was reverted. See https://github.com/apache/pinot/pull/15649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[I] Disallow multiple TableRebalance jobs from running for the same table at the same time [pinot]

2025-04-30 Thread via GitHub
somandal opened a new issue, #15683: URL: https://github.com/apache/pinot/issues/15683 Today we allow running multiple TableRebalancer jobs for the same table at the same time. TableRebalance is idempotent, but ideally we should not allow this behavior. There are some periodic tasks such as

(pinot) branch master updated: [ui] Fix: Prevent Runtime Errors by Validating Add Schema JSON Fields (#15680)

2025-04-30 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ce9f20a4c1 [ui] Fix: Prevent Runtime Errors by Va

Re: [PR] [bugfix][ui] Fix: Prevent Runtime Errors by Validating Add Schema JSON Fields [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang merged PR #15680: URL: https://github.com/apache/pinot/pull/15680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated (7bd61f05a7 -> 75fe05f37f)

2025-04-30 Thread yashmayya
This is an automated email from the ASF dual-hosted git repository. yashmayya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 7bd61f05a7 SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocato

Re: [PR] Add type coercion (implicit cast) rules for TIMESTAMP / LONG in the multi-stage engine [pinot]

2025-04-30 Thread via GitHub
yashmayya merged PR #15679: URL: https://github.com/apache/pinot/pull/15679 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

(pinot) branch master updated: SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run (#15681)

2025-04-30 Thread somandal
This is an automated email from the ASF dual-hosted git repository. somandal pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7bd61f05a7 SegmentRelocator should skip rebalan

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal merged PR #15681: URL: https://github.com/apache/pinot/pull/15681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069322103 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -120,17 +121,23 @@ public SegmentRelocator(PinotHelixR

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069318369 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -196,6 +204,14 @@ private void rebalanceTable(String t

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069316146 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -196,6 +204,14 @@ private void rebalanceTable(Stri

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069314466 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -120,17 +121,23 @@ public SegmentRelocator(PinotHe

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069311235 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -196,6 +203,10 @@ private void rebalanceTable(String t

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069310613 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -120,17 +121,23 @@ public SegmentRelocator(PinotHelixR

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069310084 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -80,6 +80,7 @@ public class SegmentRelocator extends

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069274941 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -120,17 +121,23 @@ public SegmentRelocator(PinotHe

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-04-30 Thread via GitHub
Jackie-Jiang commented on code in PR #15571: URL: https://github.com/apache/pinot/pull/15571#discussion_r2069226305 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcSendingMailbox.java: ## @@ -156,19 +176,30 @@ private StreamObserver getContentObserver()

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
yashmayya commented on code in PR #15681: URL: https://github.com/apache/pinot/pull/15681#discussion_r2069210950 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/relocation/SegmentRelocator.java: ## @@ -80,6 +80,7 @@ public class SegmentRelocator extends

[PR] Skip IngestionBasedConsumptionStatusCheck for Kinesis [pinot]

2025-04-30 Thread via GitHub
noob-se7en opened a new pull request, #15682: URL: https://github.com/apache/pinot/pull/15682 As per Kinesis code (from `software.amazon.awssdk.services.kinesis.model`): ``` `/** * * The ending sequence number for the range. Shards that are in the OPEN state have an ending

Re: [PR] Allow segment pre-processing failure during segment loading phase [pinot]

2025-04-30 Thread via GitHub
somandal commented on PR #15672: URL: https://github.com/apache/pinot/pull/15672#issuecomment-2842675339 hey @xiangfu0 would it make sense to put this behind a config and disable it by default? so that this behavior change doesn't take anyone by surprise? -- This is an automated message f

Re: [PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
rhodo commented on code in PR #15676: URL: https://github.com/apache/pinot/pull/15676#discussion_r2069122363 ## pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcQueryServer.java: ## @@ -74,6 +75,8 @@ public class GrpcQueryServer extends PinotQueryServerGrpc.Pin

Re: [PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15681: URL: https://github.com/apache/pinot/pull/15681#issuecomment-2842656629 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15681?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Parallelize segment metadata file generation. [pinot]

2025-04-30 Thread via GitHub
priyen-stripe commented on PR #15030: URL: https://github.com/apache/pinot/pull/15030#issuecomment-2842643714 @abhishekbafna what type of improvements did you see? any stats? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add type coercion (implicit cast) rules for TIMESTAMP / LONG in the multi-stage engine [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15679: URL: https://github.com/apache/pinot/pull/15679#issuecomment-2842611132 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15679?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Don't replace environment variables and system properties in get table configs REST API [pinot]

2025-04-30 Thread via GitHub
priyen-stripe commented on PR #14002: URL: https://github.com/apache/pinot/pull/14002#issuecomment-2842595139 @yashmayya why was this marked as backwards incompatible? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[PR] SegmentRelocator should skip rebalancing tables that haven't completed rebalance since the last relocator run [pinot]

2025-04-30 Thread via GitHub
somandal opened a new pull request, #15681: URL: https://github.com/apache/pinot/pull/15681 Today when `SegmentRelocator` runs, it issues a table rebalance request for each table without checking whether the last rebalance it had issued completed or not. For small rebalances that move a few

Re: [PR] Use Kafka Admin client to Fetch offsets instead of Kafka Consumer [pinot]

2025-04-30 Thread via GitHub
noob-se7en commented on code in PR #15641: URL: https://github.com/apache/pinot/pull/15641#discussion_r2068954594 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1323,6 +1323,9 @@ private void closePartitionMetadata

[PR] [bugfix][ui] Fix: Prevent Runtime Errors by Validating Add Schema JSON Fields [pinot]

2025-04-30 Thread via GitHub
saptarshi1996 opened a new pull request, #15680: URL: https://github.com/apache/pinot/pull/15680 ### Prevent Runtime Errors by Validating Schema Fields This pull request addresses a potential runtime error in the isObjEmpty function by adding a validation check to ensure that each fie

[PR] Add type coercion (implicit cast) rules for TIMESTAMP / LONG in the multi-stage engine [pinot]

2025-04-30 Thread via GitHub
yashmayya opened a new pull request, #15679: URL: https://github.com/apache/pinot/pull/15679 - These type coercions / implicit casts are added to improve the user experience for new users to the multi-stage engine since the single-stage engine already treats `TIMESTAMP` / `LONG` types as in

Re: [PR] Adding segment level QueryCache for expensive predicates [pinot]

2025-04-30 Thread via GitHub
Copilot commented on code in PR #15606: URL: https://github.com/apache/pinot/pull/15606#discussion_r2068765761 ## pinot-common/src/main/java/org/apache/pinot/common/cache/InMemoryQueryCache.java: ## @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.32 deleted (was f634714cb9)

2025-04-30 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.32 in repository https://gitbox.apache.org/repos/asf/pinot.git was f634714cb9 Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.32 The

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.32 [pinot]

2025-04-30 Thread via GitHub
xiangfu0 merged PR #15678: URL: https://github.com/apache/pinot/pull/15678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Bump nl.jqno.equalsverifier:equalsverifier from 3.19.3 to 3.19.4 [pinot]

2025-04-30 Thread via GitHub
xiangfu0 merged PR #15677: URL: https://github.com/apache/pinot/pull/15677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

(pinot) branch master updated (1d2a4a4ac4 -> 38a2a6e272)

2025-04-30 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 1d2a4a4ac4 Bump nl.jqno.equalsverifier:equalsverifier from 3.19.3 to 3.19.4 (#15677) add 38a2a6e272 Bump softwar

(pinot) branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.19.4 deleted (was 7070648aad)

2025-04-30 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.19.4 in repository https://gitbox.apache.org/repos/asf/pinot.git was 7070648aad Bump nl.jqno.equalsverifier:equalsverifier from 3.1

(pinot) branch master updated (7b22966633 -> 1d2a4a4ac4)

2025-04-30 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 7b22966633 byte-based splitting for mse query blocks (#15571) add 1d2a4a4ac4 Bump nl.jqno.equalsverifier:equalsv

Re: [PR] Include stats on cancel [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15609: URL: https://github.com/apache/pinot/pull/15609#discussion_r2068646994 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java: ## @@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestCon

Re: [PR] Include stats on cancel [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15609: URL: https://github.com/apache/pinot/pull/15609#discussion_r2068639112 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java: ## @@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestCon

Re: [PR] Include stats on cancel [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15609: URL: https://github.com/apache/pinot/pull/15609#discussion_r2068633803 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java: ## @@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestCon

Re: [PR] Include stats on cancel [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15609: URL: https://github.com/apache/pinot/pull/15609#discussion_r2068522578 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java: ## @@ -18,35 +18,65 @@ */ package org.apache.pinot.query.ru

Re: [PR] Include stats on cancel [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15609: URL: https://github.com/apache/pinot/pull/15609#discussion_r2068519625 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java: ## @@ -18,35 +18,65 @@ */ package org.apache.pinot.query.ru

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.32 [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15678: URL: https://github.com/apache/pinot/pull/15678#issuecomment-2841739295 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15678?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

(pinot) branch master updated (83e20eea73 -> 7b22966633)

2025-04-30 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 83e20eea73 Codecov tests (#15560) add 7b22966633 byte-based splitting for mse query blocks (#15571) No new revis

Re: [PR] Codecov tests [pinot]

2025-04-30 Thread via GitHub
gortiz merged PR #15560: URL: https://github.com/apache/pinot/pull/15560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

(pinot) branch master updated: Codecov tests (#15560)

2025-04-30 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 83e20eea73 Codecov tests (#15560) 83e20eea73 is d

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-04-30 Thread via GitHub
gortiz merged PR #15571: URL: https://github.com/apache/pinot/pull/15571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

Re: [PR] Bump nl.jqno.equalsverifier:equalsverifier from 3.19.3 to 3.19.4 [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15677: URL: https://github.com/apache/pinot/pull/15677#issuecomment-2841731242 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15677?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.31 [pinot]

2025-04-30 Thread via GitHub
dependabot[bot] closed pull request #15667: Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.31 URL: https://github.com/apache/pinot/pull/15667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.31 deleted (was a27cb7d323)

2025-04-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.31 in repository https://gitbox.apache.org/repos/asf/pinot.git was a27cb7d323 Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.31 T

Re: [PR] Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.31 [pinot]

2025-04-30 Thread via GitHub
dependabot[bot] commented on PR #15667: URL: https://github.com/apache/pinot/pull/15667#issuecomment-2841645752 Superseded by #15678. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.31.32 created (now f634714cb9)

2025-04-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.31.32 in repository https://gitbox.apache.org/repos/asf/pinot.git at f634714cb9 Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.32 N

[PR] Bump software.amazon.awssdk:bom from 2.31.30 to 2.31.32 [pinot]

2025-04-30 Thread via GitHub
dependabot[bot] opened a new pull request, #15678: URL: https://github.com/apache/pinot/pull/15678 Bumps software.amazon.awssdk:bom from 2.31.30 to 2.31.32. Most Recent Ignore Conditions Applied to This Pull Request | Dependency Name | Ignore Conditions | | --- | --- |

(pinot) branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.19.4 created (now 7070648aad)

2025-04-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.19.4 in repository https://gitbox.apache.org/repos/asf/pinot.git at 7070648aad Bump nl.jqno.equalsverifier:equalsverifier from

[PR] Bump nl.jqno.equalsverifier:equalsverifier from 3.19.3 to 3.19.4 [pinot]

2025-04-30 Thread via GitHub
dependabot[bot] opened a new pull request, #15677: URL: https://github.com/apache/pinot/pull/15677 Bumps [nl.jqno.equalsverifier:equalsverifier](https://github.com/jqno/equalsverifier) from 3.19.3 to 3.19.4. Changelog Sourced from https://github.com/jqno/equalsverifier/blob/main/C

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-04-30 Thread via GitHub
albertobastos commented on code in PR #15571: URL: https://github.com/apache/pinot/pull/15571#discussion_r2068400368 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -1508,6 +1508,13 @@ public static class MultiStageQueryRunner { public stat

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15571: URL: https://github.com/apache/pinot/pull/15571#discussion_r2068389246 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -1508,6 +1508,13 @@ public static class MultiStageQueryRunner { public static fina

Re: [I] Upgrade Pinot service source version to Java 21 [pinot]

2025-04-30 Thread via GitHub
yupeng9 commented on issue #15343: URL: https://github.com/apache/pinot/issues/15343#issuecomment-2841464104 the plugins like segment-uploader and segment-writer to push segments into pinot server. -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
codecov-commenter commented on PR #15676: URL: https://github.com/apache/pinot/pull/15676#issuecomment-2841396037 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15676?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
xiangfu0 commented on code in PR #15676: URL: https://github.com/apache/pinot/pull/15676#discussion_r2068283430 ## pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcQueryServer.java: ## @@ -74,6 +75,8 @@ public class GrpcQueryServer extends PinotQueryServerGrpc.

Re: [PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
xiangfu0 commented on code in PR #15676: URL: https://github.com/apache/pinot/pull/15676#discussion_r2068283430 ## pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcQueryServer.java: ## @@ -74,6 +75,8 @@ public class GrpcQueryServer extends PinotQueryServerGrpc.

Re: [PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
xiangfu0 commented on code in PR #15676: URL: https://github.com/apache/pinot/pull/15676#discussion_r2068276988 ## pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcQueryServer.java: ## @@ -74,6 +75,8 @@ public class GrpcQueryServer extends PinotQueryServerGrpc.

[PR] Set conservative default for gRPC query sever worker thread count [pinot]

2025-04-30 Thread via GitHub
rhodo opened a new pull request, #15676: URL: https://github.com/apache/pinot/pull/15676 Previously, the gRPC query server defaulted to using `2 * num_cores` threads for query workers. This PR adjusts the default to a lower cap (min(8, 2 * num_cores)) to reduce memory usage during request s

Re: [PR] Direct mse query server [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15445: URL: https://github.com/apache/pinot/pull/15445#discussion_r2068194890 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/server/QueryServer.java: ## @@ -329,13 +364,12 @@ private void submitStage( * applying the submi

Re: [PR] Direct mse query server [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15445: URL: https://github.com/apache/pinot/pull/15445#discussion_r2068182383 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/server/QueryServer.java: ## @@ -130,6 +124,15 @@ public void start() { } } + private > Ser

Re: [PR] Direct mse query server [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15445: URL: https://github.com/apache/pinot/pull/15445#discussion_r2068157362 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/server/QueryServer.java: ## @@ -159,37 +164,49 @@ public void submit(Worker.QueryRequest request, St

Re: [PR] Codecov tests [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15560: URL: https://github.com/apache/pinot/pull/15560#discussion_r2068142357 ## .github/workflows/pinot_tests.yml: ## @@ -78,19 +78,9 @@ jobs: fail-fast: false matrix: testset: [ 1, 2 ] -java: [ 11, 21 ] +ja

Re: [I] Upgrade Pinot service source version to Java 21 [pinot]

2025-04-30 Thread via GitHub
gortiz commented on issue #15343: URL: https://github.com/apache/pinot/issues/15343#issuecomment-2841182478 @yupeng9 do you mean pinot-connectors or ingestion plugins? pinot-connectors, like the pinot jdbc driver, should be independent of this change and still support java 8/11 -- This

Re: [I] Upgrade Pinot service source version to Java 21 [pinot]

2025-04-30 Thread via GitHub
gortiz commented on issue #15343: URL: https://github.com/apache/pinot/issues/15343#issuecomment-2841177739 Datasketches 8.0.0 supports only Java 21 with `--enable-preview` https://github.com/apache/datasketches-java (https://lists.apache.org/thread/z7nr5wgkrrqnx61ytyqrblhnjpg5knly) and the

Re: [PR] Codecov tests [pinot]

2025-04-30 Thread via GitHub
gortiz commented on code in PR #15560: URL: https://github.com/apache/pinot/pull/15560#discussion_r2068117526 ## .github/workflows/pinot_tests.yml: ## @@ -78,19 +78,9 @@ jobs: fail-fast: false matrix: testset: [ 1, 2 ] -java: [ 11, 21 ] +ja

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-04-30 Thread via GitHub
albertobastos commented on code in PR #15571: URL: https://github.com/apache/pinot/pull/15571#discussion_r2068026931 ## pinot-query-runtime/src/test/java/org/apache/pinot/query/mailbox/MailboxServiceTest.java: ## @@ -87,7 +87,7 @@ public void testLocalHappyPathSendFirst() S

Re: [PR] byte-based splitting for mse query blocks [pinot]

2025-04-30 Thread via GitHub
albertobastos commented on code in PR #15571: URL: https://github.com/apache/pinot/pull/15571#discussion_r2067999120 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcSendingMailbox.java: ## @@ -52,24 +58,32 @@ public class GrpcSendingMailbox implements Se