Re: [PR] Canonicalize BigDecimal values during ingestion [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on code in PR #14958: URL: https://github.com/apache/pinot/pull/14958#discussion_r1936812149 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SpecialValueTransformer.java: ## @@ -18,112 +18,119 @@ */ package org.apac

Re: [PR] Supports Force Committing Segments in Batches [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on code in PR #14811: URL: https://github.com/apache/pinot/pull/14811#discussion_r1936800273 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1848,15 +1860,130 @@ private boolean

Re: [PR] Fix configuration handling during MergeRollupTask execution [pinot]

2025-01-30 Thread via GitHub
swaminathanmanish merged PR #14856: URL: https://github.com/apache/pinot/pull/14856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@

(pinot) branch master updated (5fd881b149 -> 9c8ab8f397)

2025-01-30 Thread manishswaminathan
This is an automated email from the ASF dual-hosted git repository. manishswaminathan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 5fd881b149 Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 (#14948) add 9c8ab8f397 Fix configura

Re: [PR] Make HelixMaxStateTransitions extendable in BaseControllerStarter [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14960: URL: https://github.com/apache/pinot/pull/14960#issuecomment-2626424703 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14960?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Canonicalize BigDecimal values during ingestion [pinot]

2025-01-30 Thread via GitHub
jadami10 commented on code in PR #14958: URL: https://github.com/apache/pinot/pull/14958#discussion_r193674 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SpecialValueTransformer.java: ## @@ -18,112 +18,119 @@ */ package org.apache.p

Re: [PR] Make HelixMaxTransitions per instance extendable [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14959: URL: https://github.com/apache/pinot/pull/14959#issuecomment-2626416391 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14959?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Canonicalize BigDecimal values during ingestion [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on code in PR #14958: URL: https://github.com/apache/pinot/pull/14958#discussion_r1936746829 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SpecialValueTransformer.java: ## @@ -18,112 +18,119 @@ */ package org.apac

Re: [PR] Canonicalize BigDecimal values during ingestion [pinot]

2025-01-30 Thread via GitHub
jadami10 commented on code in PR #14958: URL: https://github.com/apache/pinot/pull/14958#discussion_r1936729841 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SpecialValueTransformer.java: ## @@ -18,112 +18,119 @@ */ package org.apache.p

[PR] Make HelixMaxStateTransitions extendable in BaseControllerStarter [pinot]

2025-01-30 Thread via GitHub
deepthi912 opened a new pull request, #14960: URL: https://github.com/apache/pinot/pull/14960 Refactor to a separate method for HelixMaxTransitions and make it extendable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Make HelixMaxTransitions per instance extendable [pinot]

2025-01-30 Thread via GitHub
deepthi912 closed pull request #14959: Make HelixMaxTransitions per instance extendable URL: https://github.com/apache/pinot/pull/14959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Make HelixMaxTransitions per instance extendable [pinot]

2025-01-30 Thread via GitHub
deepthi912 opened a new pull request, #14959: URL: https://github.com/apache/pinot/pull/14959 Refactor to a separate method and make HelixMaxTransitions extendable for other classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Fix configuration handling during MergeRollupTask execution [pinot]

2025-01-30 Thread via GitHub
swaminathanmanish commented on code in PR #14856: URL: https://github.com/apache/pinot/pull/14856#discussion_r1936681099 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java: #

Re: [PR] Improve error message when colocated join is requested but workermanager is disabled [pinot]

2025-01-30 Thread via GitHub
yashmayya commented on code in PR #14947: URL: https://github.com/apache/pinot/pull/14947#discussion_r1936676324 ## pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java: ## @@ -152,7 +152,8 @@ private WorkerManager getWorkerManager(SqlNodeAndOptions sq

Re: [PR] Canonicalize BigDecimal values during ingestion [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14958: URL: https://github.com/apache/pinot/pull/14958#issuecomment-2626202861 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14958?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Add segment StarTree index rebuild throttler configurable via ZK cluster configs [pinot]

2025-01-30 Thread via GitHub
somandal commented on PR #14943: URL: https://github.com/apache/pinot/pull/14943#issuecomment-2626160486 > Say in the future we want to add a `TextIndexThrottler`, are we keep appending parameters? They are all under the same topic, and should be managed by a single class. Within that class

[PR] Canonicalize BigDecimal values during ingestion [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang opened a new pull request, #14958: URL: https://github.com/apache/pinot/pull/14958 For the same value, `BigDecimal` could have multiple format with different trailing zeros. It could cause similar problems as Negative zero and NaN, where `compareTo() == 0` doesn't mean `equals(

(pinot) branch hotfix-upload-20250130 updated: safe copy in local pinot fs (#14883)

2025-01-30 Thread jtao
This is an automated email from the ASF dual-hosted git repository. jtao pushed a commit to branch hotfix-upload-20250130 in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/hotfix-upload-20250130 by this push: new 99051be29f safe

Re: [PR] Improve error handling in controller [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14951: URL: https://github.com/apache/pinot/pull/14951#issuecomment-2626101248 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14951?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [I] Improve GroupIdGenerators performance for high cardinality group by keys [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on issue #14685: URL: https://github.com/apache/pinot/issues/14685#issuecomment-2626097887 I know where the confusion is coming from. The initial result holder capacity should just match the initial group map capacity because the map is simply generating the index in

Re: [PR] Supports Force Committing Segments in Batches [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on code in PR #14811: URL: https://github.com/apache/pinot/pull/14811#discussion_r1936533387 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1848,15 +1860,130 @@ private boolean

Re: [PR] Add `DATE_TRUNC` Optimizer [pinot]

2025-01-30 Thread via GitHub
ashishjayamohan commented on PR #14385: URL: https://github.com/apache/pinot/pull/14385#issuecomment-2626077604 Sounds good, just let me know when. I'm in PST. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Add segment StarTree index rebuild throttler configurable via ZK cluster configs [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on PR #14943: URL: https://github.com/apache/pinot/pull/14943#issuecomment-2625998365 Say in the future we want to add a `TextIndexThrottler`, are we keep appending parameters? They are all under the same topic, and should be managed by a single class. Within that cla

Re: [I] Improve GroupIdGenerators performance for high cardinality group by keys [pinot]

2025-01-30 Thread via GitHub
shauryachats commented on issue #14685: URL: https://github.com/apache/pinot/issues/14685#issuecomment-2625985589 > I wouldn't add too many configs for the same purpose because there is no way to make user understand them. I agree with the sentiment, but in this case, I feel a new con

Re: [PR] Add segment StarTree index rebuild throttler configurable via ZK cluster configs [pinot]

2025-01-30 Thread via GitHub
somandal commented on PR #14943: URL: https://github.com/apache/pinot/pull/14943#issuecomment-2625965600 > Star-tree build is part of the segment pre-process. Can we integrate it into `SegmentPreprocessThrottler`? @Jackie-Jiang I didn't want to put it in the same class as I'll need 2

Re: [PR] Additional check to avoid downloading segments from deep store [pinot]

2025-01-30 Thread via GitHub
klsince commented on code in PR #14957: URL: https://github.com/apache/pinot/pull/14957#discussion_r1936459425 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -396,6 +396,18 @@ protected void replaceSegmentIfCrcMismatch(SegmentDat

(pinot) branch hotfix-upload-20250130 created (now 269f483c3c)

2025-01-30 Thread jtao
This is an automated email from the ASF dual-hosted git repository. jtao pushed a change to branch hotfix-upload-20250130 in repository https://gitbox.apache.org/repos/asf/pinot.git at 269f483c3c add server status to all instance tables (#14720) No new revisions were added by this update

(pinot) branch hotfix-upload created (now 5fd881b149)

2025-01-30 Thread jtao
This is an automated email from the ASF dual-hosted git repository. jtao pushed a change to branch hotfix-upload in repository https://gitbox.apache.org/repos/asf/pinot.git at 5fd881b149 Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 (#14948) No new revisions were added by this up

Re: [PR] FIX: Modifying the helm chart used for Pinot Quickstart [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14952: URL: https://github.com/apache/pinot/pull/14952#issuecomment-2625797699 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14952?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [I] Group By Truncates at Limit in Combine Operator [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on issue #14956: URL: https://github.com/apache/pinot/issues/14956#issuecomment-2625778367 Duplicate of #11706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Adds log for failure to acquire partitionGroupConsumerSemaphore [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on code in PR #14955: URL: https://github.com/apache/pinot/pull/14955#discussion_r1936394721 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1652,7 +1652,11 @@ public RealtimeSegmentDataManage

Re: [I] Group By Truncates at Limit in Combine Operator [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on issue #14956: URL: https://github.com/apache/pinot/issues/14956#issuecomment-2625777547 This is tracked under #11924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Additional check to avoid downloading segments from deep store [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on code in PR #14957: URL: https://github.com/apache/pinot/pull/14957#discussion_r1936388971 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -396,6 +396,18 @@ protected void replaceSegmentIfCrcMismatch(Segme

Re: [PR] Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang merged PR #14948: URL: https://github.com/apache/pinot/pull/14948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] Server replica allowed to overwrite deep store segment after segment was already committed [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on issue #14786: URL: https://github.com/apache/pinot/issues/14786#issuecomment-2625726698 @KKcorps I don't think they uploaded the segment. Segment is just committed from a different server. @dang-stripe Can you confirm? -- This is an automated message from the Apa

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.30.9 deleted (was e0e6589329)

2025-01-30 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.30.9 in repository https://gitbox.apache.org/repos/asf/pinot.git was e0e6589329 Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 The revi

(pinot) branch master updated: Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 (#14948)

2025-01-30 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 5fd881b149 Bump software.amazon.awssdk:bom from 2

Re: [PR] Bump nl.jqno.equalsverifier:equalsverifier from 3.18.1 to 3.18.2 [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang merged PR #14949: URL: https://github.com/apache/pinot/pull/14949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.18.2 deleted (was f751aa0700)

2025-01-30 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.18.2 in repository https://gitbox.apache.org/repos/asf/pinot.git was f751aa0700 Bump nl.jqno.equalsverifier:equalsverifier from 3.18

Re: [PR] FIX: Modifying the helm chart used for Pinot Quickstart [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang commented on PR #14952: URL: https://github.com/apache/pinot/pull/14952#issuecomment-2625718678 @abhioncbr Can you help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] [multi-stage] [optimization] Move inequi join out of hashjoin when there is no join key [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang closed issue #9728: [multi-stage] [optimization] Move inequi join out of hashjoin when there is no join key URL: https://github.com/apache/pinot/issues/9728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Separate NonEquiJoinOperator from HashJoinOperator [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang merged PR #14942: URL: https://github.com/apache/pinot/pull/14942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Additional check to avoid downloading segments from deep store [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14957: URL: https://github.com/apache/pinot/pull/14957#issuecomment-2625676686 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14957?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Additional check to avoid downloading segments from deep store [pinot]

2025-01-30 Thread via GitHub
deepthi912 commented on code in PR #14957: URL: https://github.com/apache/pinot/pull/14957#discussion_r1936329399 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -396,6 +396,18 @@ protected void replaceSegmentIfCrcMismatch(Segment

[PR] Additional check to avoid downloading segments from deep store [pinot]

2025-01-30 Thread via GitHub
deepthi912 opened a new pull request, #14957: URL: https://github.com/apache/pinot/pull/14957 Continuation to : https://github.com/apache/pinot/pull/10089 This avoids downloading from deep store when we can just simply use the existing directory in addition to crc check. -- This i

Re: [PR] Adds log for failure to acquire partitionGroupConsumerSemaphore [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14955: URL: https://github.com/apache/pinot/pull/14955#issuecomment-2625584190 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14955?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

[I] Group By Truncates at Limit in Combine Operator [pinot]

2025-01-30 Thread via GitHub
ankitsultana opened a new issue, #14956: URL: https://github.com/apache/pinot/issues/14956 Queries such as the following return incorrect results in most cases because the Combine Operator truncates the IndexedTable at the Query Limit. Moreover there are no warnings in the response:

[PR] Adds log for failure to acquire partitionGroupConsumerSemaphore [pinot]

2025-01-30 Thread via GitHub
noob-se7en opened a new pull request, #14955: URL: https://github.com/apache/pinot/pull/14955 Adding log to debug issue when a consuming segment takes extremely long time to complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Supports Force Committing Segments in Batches [pinot]

2025-01-30 Thread via GitHub
noob-se7en commented on PR #14811: URL: https://github.com/apache/pinot/pull/14811#issuecomment-2625414478 How are we going to come up with a good value for batch size param? h... IMO It should be `<= _serverConfig.getProperty(MAX_PARALLEL_SEGMENT_BUILDS) * num_of_server_instance

Re: [PR] Supports Force Committing Segments in Batches [pinot]

2025-01-30 Thread via GitHub
noob-se7en commented on PR #14811: URL: https://github.com/apache/pinot/pull/14811#issuecomment-2625412070 > How are we going to come up with a good value for batch size param? h... IMO It should be ```<= _serverConfig.getProperty(MAX_PARALLEL_SEGMENT_BUILDS) * num_of_server_

Re: [PR] Fix potential NPE in OffHeapSingleTreeBuilder.close() [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14954: URL: https://github.com/apache/pinot/pull/14954#issuecomment-2625411709 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14954?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [I] Improve error handling and reporting in query side [pinot]

2025-01-30 Thread via GitHub
real-mj-song commented on issue #14950: URL: https://github.com/apache/pinot/issues/14950#issuecomment-2625398726 > MSE errors are large and don't include the actual error code. Instead, error code 200 is always returned. 💯 agreed. We definitely need a throwable/exception class contai

Re: [PR] Map BadQueryRequestException to QueryException.QUERY_VALIDATION_ERROR [pinot]

2025-01-30 Thread via GitHub
real-mj-song commented on PR #14917: URL: https://github.com/apache/pinot/pull/14917#issuecomment-2625386989 @gortiz Oh nice! Your `PinotRuntimeException` and `QException` were exactly what I was looking for 💯 I was not planning to make more changes, but once your PR is merged, I can m

(pinot) branch master updated (11964592da -> 89189747fa)

2025-01-30 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 11964592da Add broker setting to override default implicit query response limit (#14452) add 89189747fa Fix poten

Re: [PR] Fix potential NPE in OffHeapSingleTreeBuilder.close() [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang merged PR #14954: URL: https://github.com/apache/pinot/pull/14954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[PR] Fix potential NPE in OffHeapSingleTreeBuilder.close() [pinot]

2025-01-30 Thread via GitHub
Jackie-Jiang opened a new pull request, #14954: URL: https://github.com/apache/pinot/pull/14954 `_starTreeRecordBuffer` might not be set when star-tree build is interrupted (e.g. throws exception) -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
noob-se7en commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1936067264 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java: ## @@ -0,0 +1,526 @@ +/** + * Licensed to the Apache Software Foundatio

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
noob-se7en commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1936058894 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java: ## @@ -0,0 +1,526 @@ +/** + * Licensed to the Apache Software Foundatio

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
noob-se7en commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1936058894 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java: ## @@ -0,0 +1,526 @@ +/** + * Licensed to the Apache Software Foundatio

Re: [PR] Supports Force Committing Segments in Batches [pinot]

2025-01-30 Thread via GitHub
sajjad-moradi commented on code in PR #14811: URL: https://github.com/apache/pinot/pull/14811#discussion_r1935978585 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1848,15 +1860,126 @@ private boolean

Re: [PR] [timeseries] Remove Time Series Specific Code from V1 Engine [pinot]

2025-01-30 Thread via GitHub
ankitsultana commented on code in PR #14841: URL: https://github.com/apache/pinot/pull/14841#discussion_r1935928210 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/AggregationFunctionType.java: ## @@ -220,7 +220,8 @@ public enum AggregationFunctionType { S

[PR] Push auth before query validation. [pinot]

2025-01-30 Thread via GitHub
bziobrowski opened a new pull request, #14953: URL: https://github.com/apache/pinot/pull/14953 This PR pushes authorization (user <-> table) before sql validation that might produce error messages leaking schema details, in both single- and multi-stage query engine. -- This is an automa

Re: [PR] [timeseries] Remove Time Series Specific Code from V1 Engine [pinot]

2025-01-30 Thread via GitHub
ankitsultana commented on code in PR #14841: URL: https://github.com/apache/pinot/pull/14841#discussion_r1935922216 ## pinot-core/src/main/java/org/apache/pinot/core/operator/timeseries/TimeSeriesOperatorUtils.java: ## @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software Fo

Re: [PR] Map BadQueryRequestException to QueryException.QUERY_VALIDATION_ERROR [pinot]

2025-01-30 Thread via GitHub
gortiz commented on PR #14917: URL: https://github.com/apache/pinot/pull/14917#issuecomment-2624804967 Hi there! I'm working on improving query error management. I've created https://github.com/apache/pinot/issues/14950 to try to keep track of it and also started to create some exception hi

[PR] FIX: Modifying the helm chart used for Pinot Quickstart [pinot]

2025-01-30 Thread via GitHub
subhashini-26 opened a new pull request, #14952: URL: https://github.com/apache/pinot/pull/14952 This bugfix fixes the issue with #14696 The following error is encountered when trying to deploy Apache Pinot on k8s clusters using Helm chart with the instructions mentioned [here](https

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
shounakmk219 commented on code in PR #14829: URL: https://github.com/apache/pinot/pull/14829#discussion_r1935676993 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -483,92 +489,34 @@ public void registerTaskGenerator(P

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
shounakmk219 commented on code in PR #14829: URL: https://github.com/apache/pinot/pull/14829#discussion_r1935669405 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -483,92 +489,34 @@ public void registerTaskGenerator(P

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
shounakmk219 commented on code in PR #14829: URL: https://github.com/apache/pinot/pull/14829#discussion_r1935663271 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -483,92 +489,34 @@ public void registerTaskGenerator(P

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on PR #14829: URL: https://github.com/apache/pinot/pull/14829#issuecomment-2624436814 Add missing license headers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1935539149 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java: ## @@ -0,0 +1,526 @@ +/** + * Licensed to the Apache Software Foundation (

Re: [I] Improve error handling and reporting in query side [pinot]

2025-01-30 Thread via GitHub
gortiz commented on issue #14950: URL: https://github.com/apache/pinot/issues/14950#issuecomment-2624350803 Opened #14951 to fix the first two problems -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[PR] Improve error handling in controller [pinot]

2025-01-30 Thread via GitHub
gortiz opened a new pull request, #14951: URL: https://github.com/apache/pinot/pull/14951 This PR is trying to fix the two first controller errors listed in #14950: - On errors, Pinot controller query endpoints inconsistently return broker-like JSON error payloads (if the error is detecte

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1935512278 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java: ## @@ -0,0 +1,526 @@ +/** + * Licensed to the Apache Software Foundation (

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1935509704 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/reingestion/utils/SimpleRealtimeSegmentDataManager.java: ## @@ -0,0 +1,575 @@ +/** + * Licensed to t

[I] Improve error handling and reporting in query side [pinot]

2025-01-30 Thread via GitHub
gortiz opened a new issue, #14950: URL: https://github.com/apache/pinot/issues/14950 Error handling and reporting can be improved in Pinot. Sometimes, error messages are not easy for users to understand, and the fact that each query engine notifies errors differently makes it even more conf

Re: [PR] Improve error message when colocated join is requested but workermanager is disabled [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14947: URL: https://github.com/apache/pinot/pull/14947#issuecomment-2624321945 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14947?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

(pinot) branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.18.2 created (now f751aa0700)

2025-01-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/nl.jqno.equalsverifier-equalsverifier-3.18.2 in repository https://gitbox.apache.org/repos/asf/pinot.git at f751aa0700 Bump nl.jqno.equalsverifier:equalsverifier from

[PR] Bump nl.jqno.equalsverifier:equalsverifier from 3.18.1 to 3.18.2 [pinot]

2025-01-30 Thread via GitHub
dependabot[bot] opened a new pull request, #14949: URL: https://github.com/apache/pinot/pull/14949 Bumps [nl.jqno.equalsverifier:equalsverifier](https://github.com/jqno/equalsverifier) from 3.18.1 to 3.18.2. Release notes Sourced from https://github.com/jqno/equalsverifier/release

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.30.9 created (now e0e6589329)

2025-01-30 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.30.9 in repository https://gitbox.apache.org/repos/asf/pinot.git at e0e6589329 Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 No n

[PR] Bump software.amazon.awssdk:bom from 2.30.7 to 2.30.9 [pinot]

2025-01-30 Thread via GitHub
dependabot[bot] opened a new pull request, #14948: URL: https://github.com/apache/pinot/pull/14948 Bumps software.amazon.awssdk:bom from 2.30.7 to 2.30.9. Most Recent Ignore Conditions Applied to This Pull Request | Dependency Name | Ignore Conditions | | --- | --- | |

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
codecov-commenter commented on PR #14946: URL: https://github.com/apache/pinot/pull/14946#issuecomment-2624223402 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14946?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Add broker setting to override default implicit query response limit [pinot]

2025-01-30 Thread via GitHub
gortiz commented on PR #14452: URL: https://github.com/apache/pinot/pull/14452#issuecomment-2624226925 I've merged this change, but the default value of 10 means we break backward compatibility for MSE. I'm going to create a PR to change that to -1 -- This is an automated message from the

(pinot) branch master updated (7293122c4f -> 11964592da)

2025-01-30 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 7293122c4f Skip block split for local exchange (#14940) add 11964592da Add broker setting to override default impl

Re: [PR] Add broker setting to override default implicit query response limit [pinot]

2025-01-30 Thread via GitHub
gortiz merged PR #14452: URL: https://github.com/apache/pinot/pull/14452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[PR] Improve error message when colocated join is requested but workermanager is disabled [pinot]

2025-01-30 Thread via GitHub
gortiz opened a new pull request, #14947: URL: https://github.com/apache/pinot/pull/14947 Simple change to include the property that needs to be changed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
yashmayya commented on code in PR #14946: URL: https://github.com/apache/pinot/pull/14946#discussion_r1935408022 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java: ## @@ -112,6 +118,7 @@ public MultiStageBrokerRequestHandler

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
yashmayya commented on code in PR #14946: URL: https://github.com/apache/pinot/pull/14946#discussion_r1935408022 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java: ## @@ -112,6 +118,7 @@ public MultiStageBrokerRequestHandler

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
yashmayya commented on code in PR #14946: URL: https://github.com/apache/pinot/pull/14946#discussion_r1935404586 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java: ## @@ -112,6 +118,7 @@ public MultiStageBrokerRequestHandler

Re: [PR] Include information on whether a query is using SSQE or MSQE in QueryLogger [pinot]

2025-01-30 Thread via GitHub
gortiz commented on code in PR #14938: URL: https://github.com/apache/pinot/pull/14938#discussion_r1935392420 ## pinot-broker/src/main/java/org/apache/pinot/broker/querylog/QueryLogger.java: ## @@ -255,6 +257,12 @@ void doFormat(StringBuilder builder, QueryLogger logger, QueryL

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
gortiz commented on code in PR #14946: URL: https://github.com/apache/pinot/pull/14946#discussion_r1935378577 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java: ## @@ -112,6 +118,7 @@ public MultiStageBrokerRequestHandler(Pi

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
gortiz commented on code in PR #14946: URL: https://github.com/apache/pinot/pull/14946#discussion_r1935371135 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java: ## @@ -186,11 +216,11 @@ protected BrokerResponse handleRequest

Re: [PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
gortiz commented on code in PR #14946: URL: https://github.com/apache/pinot/pull/14946#discussion_r1935366628 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java: ## @@ -176,7 +184,28 @@ protected BrokerResponse handleRequest(

[PR] Enforce query timeout on query compilation in the multi-stage query engine [pinot]

2025-01-30 Thread via GitHub
yashmayya opened a new pull request, #14946: URL: https://github.com/apache/pinot/pull/14946 - In some rare scenarios, query compilation itself can take a very long time in the multi-stage query engine (see https://github.com/apache/pinot/issues/13617 for example). - In these cases, the

Re: [PR] Add broker setting to override default implicit query response limit [pinot]

2025-01-30 Thread via GitHub
bziobrowski commented on code in PR #14452: URL: https://github.com/apache/pinot/pull/14452#discussion_r1935359600 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java: ## @@ -140,7 +140,9 @@ public abstract class BaseSing

Re: [PR] Add broker setting to override default implicit query response limit [pinot]

2025-01-30 Thread via GitHub
bziobrowski commented on code in PR #14452: URL: https://github.com/apache/pinot/pull/14452#discussion_r1935353600 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java: ## @@ -308,6 +313,10 @@ protected BrokerResponse hand

Re: [PR] Pauseless Consumption #3: Disaster Recovery with Reingestion [pinot]

2025-01-30 Thread via GitHub
noob-se7en commented on code in PR #14920: URL: https://github.com/apache/pinot/pull/14920#discussion_r1935222572 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ReIngestionResource.java: ## @@ -0,0 +1,526 @@ +/** + * Licensed to the Apache Software Foundatio

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on code in PR #14829: URL: https://github.com/apache/pinot/pull/14829#discussion_r1935310977 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -483,92 +489,34 @@ public void registerTaskGenerator(PinotT

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on code in PR #14829: URL: https://github.com/apache/pinot/pull/14829#discussion_r1935317644 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -483,92 +489,34 @@ public void registerTaskGenerator(PinotT

Re: [PR] Fix configuration handling during MergeRollupTask execution [pinot]

2025-01-30 Thread via GitHub
davecromberge commented on code in PR #14856: URL: https://github.com/apache/pinot/pull/14856#discussion_r1935312657 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskUtils.java: ## @@ -78

Re: [PR] Track the actor that triggers the minion task [pinot]

2025-01-30 Thread via GitHub
KKcorps commented on code in PR #14829: URL: https://github.com/apache/pinot/pull/14829#discussion_r1935300985 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -483,92 +489,34 @@ public void registerTaskGenerator(PinotT

  1   2   >