Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
tibrewalpratik17 commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843246982 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskGenerator.java:

Re: [PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
9aman commented on code in PR #14460: URL: https://github.com/apache/pinot/pull/14460#discussion_r1843179831 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -816,6 +816,27 @@ public void run() { //

Re: [PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
codecov-commenter commented on PR #14460: URL: https://github.com/apache/pinot/pull/14460#issuecomment-2477964407 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14460?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Add a server API to list segments out of date with table config [pinot]

2024-11-14 Thread via GitHub
vrajat commented on PR #14451: URL: https://github.com/apache/pinot/pull/14451#issuecomment-2477951638 This API is for refresh while IIUC that PR is for reload. I should make it clear that this PR is for refresh in the title. On Fri, Nov 15, 2024 at 10:00 AM Yash Mayya ***@***.***>

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
vvivekiyer commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843156063 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskExecutor.java: ##

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
vvivekiyer commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843179005 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java: ##

Re: [PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
9aman commented on code in PR #14460: URL: https://github.com/apache/pinot/pull/14460#discussion_r1843178346 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java: ## @@ -1015,7 +1076,12 @@ private SegmentCompletionProtoc

Re: [PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
9aman commented on code in PR #14460: URL: https://github.com/apache/pinot/pull/14460#discussion_r1843176391 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java: ## @@ -586,8 +621,21 @@ public SegmentCompletionProtocol.

Re: [PR] Add a server API to list segments out of date with table config [pinot]

2024-11-14 Thread via GitHub
yashmayya commented on PR #14451: URL: https://github.com/apache/pinot/pull/14451#issuecomment-2477939670 What's the difference between this and https://github.com/apache/pinot/pull/13789? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
vvivekiyer commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843154185 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java: ##

Re: [PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
9aman commented on code in PR #14460: URL: https://github.com/apache/pinot/pull/14460#discussion_r1843172802 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java: ## @@ -116,28 +116,36 @@ protected StreamPartitionMsgOffs

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
vvivekiyer commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843155788 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java: ## @@ -112,7 +112,7 @@ import org.apache.pinot.segment.l

Re: [PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
9aman commented on code in PR #14460: URL: https://github.com/apache/pinot/pull/14460#discussion_r1843166842 ## pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java: ## @@ -485,8 +485,7 @@ private void setUpPinotController() { // Helix res

[PR] Pauseless Consumption [pinot]

2024-11-14 Thread via GitHub
9aman opened a new pull request, #14460: URL: https://github.com/apache/pinot/pull/14460 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat` 6. `

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
vvivekiyer commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843156063 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskExecutor.java: ##

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
vvivekiyer commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1843154185 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java: ##

Re: [PR] Handle kafka backward incompatible props for AWS [pinot]

2024-11-14 Thread via GitHub
KKcorps merged PR #14446: URL: https://github.com/apache/pinot/pull/14446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

(pinot) branch master updated: Handle kafka backward incompatible props for AWS (#14446)

2024-11-14 Thread kharekartik
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 63c4c448e7 Handle kafka backward incompatibl

Re: [PR] Allows multiple requests per server per request ID [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on code in PR #13742: URL: https://github.com/apache/pinot/pull/13742#discussion_r1843021732 ## pinot-connectors/pinot-spark-common/src/main/scala/org/apache/pinot/connector/spark/common/reader/PinotServerDataFetcher.scala: ## @@ -40,9 +40,9 @@ import scal

(pinot) branch master updated (c119e046b4 -> 88f4916eee)

2024-11-14 Thread lqc
This is an automated email from the ASF dual-hosted git repository. lqc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from c119e046b4 add metric of startup time for all components (#14445) add 88f4916eee Add error log sampling to CLPLogMess

Re: [PR] Add error log sampling to CLPLogMessageDecoder [pinot]

2024-11-14 Thread via GitHub
deemoliu merged PR #14366: URL: https://github.com/apache/pinot/pull/14366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

(pinot) branch master updated: add metric of startup time for all components (#14445)

2024-11-14 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c119e046b4 add metric of startup time for all com

Re: [PR] add metric of startup time for all components [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang merged PR #14445: URL: https://github.com/apache/pinot/pull/14445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] Handle `IS NULL/IS NOT NULL` filtering when ExpressionTransformer outputs `null` [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #14433: URL: https://github.com/apache/pinot/issues/14433#issuecomment-2477587325 Correct. I feel the real issue is that null is not properly set in the `ExpressionTransformer` -- This is an automated message from the Apache Git Service. To respond to the m

Re: [I] Nullity Error When Querying `MAX` On Non-Numeric Column [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #14459: URL: https://github.com/apache/pinot/issues/14459#issuecomment-2477585296 `"null"` is the actual string value, which cannot be parsed as number -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Nullity Error When Querying `MAX` On Non-Numeric Column [pinot]

2024-11-14 Thread via GitHub
ashishjayamohan commented on issue #14459: URL: https://github.com/apache/pinot/issues/14459#issuecomment-2477574900 Correct, I updated to not be NPE. I think it would be better to state this error more informatively (rather than "null," state type) -- This is an automated message from th

Re: [I] Nullity Error When Querying `MAX` On Non-Numeric Column [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #14459: URL: https://github.com/apache/pinot/issues/14459#issuecomment-2477572653 Hmm, it is actually a `NumberFormatException` which is expected because `MAX` only works on numbers as of now -- This is an automated message from the Apache Git Service. To r

[I] NPE When Querying `MAX` On Non-Numeric Column [pinot]

2024-11-14 Thread via GitHub
ashishjayamohan opened a new issue, #14459: URL: https://github.com/apache/pinot/issues/14459 When querying the `MAX` of a column, we use `NonScanBasedAggregationOperator` to use the stored metadata. This produces a NPE on non-numeric columns as these columns do not have a stored max value.

Re: [PR] log segment name at best effort upon query exception to help locate the error segment [pinot]

2024-11-14 Thread via GitHub
klsince merged PR #14440: URL: https://github.com/apache/pinot/pull/14440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

(pinot) branch master updated (d02adb9148 -> 255022ee16)

2024-11-14 Thread xbli
This is an automated email from the ASF dual-hosted git repository. xbli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from d02adb9148 use offheapfststore as before (#14457) add 255022ee16 log segment name at best effort upon query exceptio

Re: [PR] Add error log sampling to CLPLogMessageDecoder [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14366: URL: https://github.com/apache/pinot/pull/14366#discussion_r1842871884 ## pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogMessageDecoder.java: ## @@ -38,18 +38,27 @@ * detail

Re: [PR] Add support for storing non-CLP-encodable values in a separate column; Replace CLP row value that are too large to store in FixedByteMVMutableForwardIndex with an error message. [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14365: URL: https://github.com/apache/pinot/pull/14365#discussion_r1842925372 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java: ## @@ -525,6 +526,9 @@ public boolean index(GenericRo

Re: [PR] Add immtuable CLPForwardIndex creator and related classes [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14288: URL: https://github.com/apache/pinot/pull/14288#discussion_r1842942533 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/stats/MutableNoDictionaryColStatistics.java: ## @@ -125,4 +125,13 @@ public CLPS

Re: [I] Handle `IS NULL/IS NOT NULL` filtering when ExpressionTransformer outputs `null` [pinot]

2024-11-14 Thread via GitHub
itschrispeck commented on issue #14433: URL: https://github.com/apache/pinot/issues/14433#issuecomment-2477467201 Current behavior is that the value is an empty string e.g. `select col1 from tbl where col1 is not null` gives: ``` "resultTable": { "dataSchema": { "colu

Re: [PR] Add immtuable CLPForwardIndex creator and related classes [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14288: URL: https://github.com/apache/pinot/pull/14288#discussion_r1842942533 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/stats/MutableNoDictionaryColStatistics.java: ## @@ -125,4 +125,13 @@ public CLPS

Re: [PR] Add immtuable CLPForwardIndex creator and related classes [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14288: URL: https://github.com/apache/pinot/pull/14288#discussion_r1842942533 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/stats/MutableNoDictionaryColStatistics.java: ## @@ -125,4 +125,13 @@ public CLPS

Re: [PR] Add error log sampling to CLPLogMessageDecoder [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14366: URL: https://github.com/apache/pinot/pull/14366#discussion_r1842915588 ## pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogMessageDecoder.java: ## @@ -38,18 +38,27 @@ * detail

Re: [PR] Add error log sampling to CLPLogMessageDecoder [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #14366: URL: https://github.com/apache/pinot/pull/14366#discussion_r1842875147 ## pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogMessageDecoder.java: ## @@ -38,18 +38,27 @@ * detail

(pinot) branch master updated (79f3ecba6f -> d02adb9148)

2024-11-14 Thread xbli
This is an automated email from the ASF dual-hosted git repository. xbli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 79f3ecba6f Bump org.codehaus.mojo:versions-maven-plugin from 2.17.1 to 2.18.0 (#14454) add d02adb9148 use offheapfs

Re: [PR] use offheapfststore as before [pinot]

2024-11-14 Thread via GitHub
klsince merged PR #14457: URL: https://github.com/apache/pinot/pull/14457 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

Re: [PR] Add `optimizeDictionaryType` config to automatically choose dictionary type [pinot]

2024-11-14 Thread via GitHub
deemoliu commented on code in PR #1: URL: https://github.com/apache/pinot/pull/1#discussion_r1842855965 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java: ## @@ -182,6 +182,14 @@ public void init(Seg

Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

2024-11-14 Thread via GitHub
dang-stripe commented on PR #13832: URL: https://github.com/apache/pinot/pull/13832#issuecomment-2477352667 @Jackie-Jiang curator was pulled in by hadoop-common which some pinot modules depended on so it showed up in our pinot uberjar whereas it didn't exist before. the uberjar is at the be

Re: [PR] add metric of startup time for all components [pinot]

2024-11-14 Thread via GitHub
jadami10 commented on PR #14445: URL: https://github.com/apache/pinot/pull/14445#issuecomment-2477317856 thanks jackie. fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] log segment name at best effort upon query exception to help locate the error segment [pinot]

2024-11-14 Thread via GitHub
codecov-commenter commented on PR #14440: URL: https://github.com/apache/pinot/pull/14440#issuecomment-2477310174 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14440?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] [Dependency] Cleanup unnecessary dependencies and exclusions [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on PR #13832: URL: https://github.com/apache/pinot/pull/13832#issuecomment-2477311081 @jadami10 @dang-stripe You should be able to override `curator.version` property to avoid the version conflict. -- This is an automated message from the Apache Git Service. To resp

Re: [I] Unsupported Range index types should fail with table validation [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #14431: URL: https://github.com/apache/pinot/issues/14431#issuecomment-2477305662 I think this is a bug. Range index should be applicable to TIMESTAMP type -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #14372: URL: https://github.com/apache/pinot/issues/14372#issuecomment-2477291589 Can you double check the topic existence and kafka settings? This seems a Kafka issue to me. You may also get more help in the slack troubleshooting channel -- This is an a

Re: [I] Handle `IS NULL/IS NOT NULL` filtering when ExpressionTransformer outputs `null` [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #14433: URL: https://github.com/apache/pinot/issues/14433#issuecomment-2477299198 cc @swaminathanmanish What is the current behavior? What is the value produced when the expression returns `null`? We should just use the setting in table config inste

Re: [PR] log segment name at best effort upon query exception to help locate the error segment [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on code in PR #14440: URL: https://github.com/apache/pinot/pull/14440#discussion_r1842792247 ## pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java: ## @@ -180,4 +181,20 @@ public List getChildOperators() { * Invoke

(pinot) branch dependabot/maven/org.codehaus.mojo-versions-maven-plugin-2.18.0 deleted (was 8352499865)

2024-11-14 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.codehaus.mojo-versions-maven-plugin-2.18.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 8352499865 Bump org.codehaus.mojo:versions-maven-plugin from

(pinot) branch master updated (476415f0c7 -> a49f3129d7)

2024-11-14 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 476415f0c7 Fix flaky kafka consumer tests (#14458) add a49f3129d7 Fix flaky upsert integration tests (#14447) No

Re: [PR] add metric of startup time for all components [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on PR #14445: URL: https://github.com/apache/pinot/pull/14445#issuecomment-2477250861 @jadami10 Can you take a look at the conflict? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Spark upsert table backfill support [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on PR #14443: URL: https://github.com/apache/pinot/pull/14443#issuecomment-2477244810 @pengding-stripe No worry, the failure is not related to this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] [Flaky-test] UpsertTableIntegrationTest.testUpsertCompactionWithSoftDelete [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on issue #13594: URL: https://github.com/apache/pinot/issues/13594#issuecomment-2477238926 Should be fixed with #14447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] [Flaky-test] UpsertTableIntegrationTest.testUpsertCompactionWithSoftDelete [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang closed issue #13594: [Flaky-test] UpsertTableIntegrationTest.testUpsertCompactionWithSoftDelete URL: https://github.com/apache/pinot/issues/13594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

(pinot) branch master updated (a49f3129d7 -> 79f3ecba6f)

2024-11-14 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from a49f3129d7 Fix flaky upsert integration tests (#14447) add 79f3ecba6f Bump org.codehaus.mojo:versions-maven-plugin

Re: [PR] Bump org.codehaus.mojo:versions-maven-plugin from 2.17.1 to 2.18.0 [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang merged PR #14454: URL: https://github.com/apache/pinot/pull/14454 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Fix flaky upsert integration tests [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang merged PR #14447: URL: https://github.com/apache/pinot/pull/14447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] [Flaky-test] Fix flaky kafka consumer tests [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang merged PR #14458: URL: https://github.com/apache/pinot/pull/14458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: Fix flaky kafka consumer tests (#14458)

2024-11-14 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 476415f0c7 Fix flaky kafka consumer tests (#14458

Re: [PR] Fix flaky upsert integration tests [pinot]

2024-11-14 Thread via GitHub
klsince commented on code in PR #14447: URL: https://github.com/apache/pinot/pull/14447#discussion_r1842698258 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/UpsertTableSegmentPreloadIntegrationTest.java: ## @@ -207,19 +208,24 @@ public void testSegm

Re: [PR] [Flaky-test] Fix flaky kafka consumer tests [pinot]

2024-11-14 Thread via GitHub
codecov-commenter commented on PR #14458: URL: https://github.com/apache/pinot/pull/14458#issuecomment-2477207336 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14458?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] log segment name at best effort upon query exception to help locate the error segment [pinot]

2024-11-14 Thread via GitHub
klsince commented on code in PR #14440: URL: https://github.com/apache/pinot/pull/14440#discussion_r1842756773 ## pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseSingleBlockCombineOperator.java: ## @@ -92,6 +92,9 @@ protected void processSegments() {

Re: [PR] Fix flaky upsert integration tests [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on code in PR #14447: URL: https://github.com/apache/pinot/pull/14447#discussion_r1842734884 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/UpsertTableIntegrationTest.java: ## @@ -102,14 +99,16 @@ public void setUp() start

Re: [PR] use offheapfststore as before [pinot]

2024-11-14 Thread via GitHub
codecov-commenter commented on PR #14457: URL: https://github.com/apache/pinot/pull/14457#issuecomment-2477164425 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14457?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Fix flaky upsert integration tests [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on code in PR #14447: URL: https://github.com/apache/pinot/pull/14447#discussion_r1842720764 ## pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java: ## @@ -361,16 +360,32 @@ public static void build

Re: [PR] Fix flaky upsert integration tests [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on code in PR #14447: URL: https://github.com/apache/pinot/pull/14447#discussion_r1842720764 ## pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java: ## @@ -361,16 +360,32 @@ public static void build

[PR] [Flaky-test] Fix flaky kafka consumer tests [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang opened a new pull request, #14458: URL: https://github.com/apache/pinot/pull/14458 Avoid reusing the same working dir in `EmbeddedZooKeeper` Example failure: ``` Error: org.apache.pinot.plugin.stream.kafka30.KafkaPartitionLevelConsumerBackwardCompatibilityTest.tear

Re: [PR] Lucene 9.12 Upgrade [pinot]

2024-11-14 Thread via GitHub
klsince commented on code in PR #14361: URL: https://github.com/apache/pinot/pull/14361#discussion_r1842684709 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/LuceneFSTIndexReader.java: ## @@ -52,8 +51,7 @@ public LuceneFSTIndexReader(Pi

Re: [PR] use offheapfststore as before [pinot]

2024-11-14 Thread via GitHub
klsince commented on PR #14457: URL: https://github.com/apache/pinot/pull/14457#issuecomment-2477095250 This was the init logic for OffHeapFSTStore in the older lucene lib, and the offset it from `this.in.getFilePointer();`. ``` public OffHeapFSTStore() { } public FSTSt

[PR] use offheapfststore as before [pinot]

2024-11-14 Thread via GitHub
klsince opened a new pull request, #14457: URL: https://github.com/apache/pinot/pull/14457 This PR changes to use OffHeapFSTStore, which was changed to OnHeapFSTStore in recent PR https://github.com/apache/pinot/pull/14361/files. The Offheap store does a simple slice to get the FST data, wh

[I] [Flaky-test] HybridClusterIntegrationTest.testReload() [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang opened a new issue, #14456: URL: https://github.com/apache/pinot/issues/14456 Observed: https://github.com/apache/pinot/actions/runs/11836706347/job/32982172166?pr=14454 ``` Error: Failures: Error: HybridClusterIntegrationTest.testReload:248->BaseClusterIntegr

Re: [PR] Enhance funnel functions to accept a new option for maxStepDuration [pinot]

2024-11-14 Thread via GitHub
Jackie-Jiang commented on code in PR #14434: URL: https://github.com/apache/pinot/pull/14434#discussion_r1842657273 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/window/FunnelBaseAggregationFunction.java: ## @@ -58,12 +62,35 @@ public Funne

(pinot) branch master updated: Add instance level consumer dir usage metric (#14430)

2024-11-14 Thread jiaguo
This is an automated email from the ASF dual-hosted git repository. jiaguo pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ed431a174e Add instance level consumer dir usage

Re: [PR] Add instance level consumer dir usage metric [pinot]

2024-11-14 Thread via GitHub
jasperjiaguo merged PR #14430: URL: https://github.com/apache/pinot/pull/14430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Spark upsert table backfill support [pinot]

2024-11-14 Thread via GitHub
pengding-stripe commented on PR #14443: URL: https://github.com/apache/pinot/pull/14443#issuecomment-2476982174 @Jackie-Jiang there is an integration test failure but I don't think that's related to my change, how can I retrigger that test? -- This is an automated message from the Apache

Re: [PR] Spark upsert table backfill support [pinot]

2024-11-14 Thread via GitHub
pengding-stripe commented on code in PR #14443: URL: https://github.com/apache/pinot/pull/14443#discussion_r1842608129 ## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java: #

Re: [PR] Spark upsert table backfill support [pinot]

2024-11-14 Thread via GitHub
pengding-stripe commented on code in PR #14443: URL: https://github.com/apache/pinot/pull/14443#discussion_r1842602704 ## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java: #

Re: [PR] Spark upsert table backfill support [pinot]

2024-11-14 Thread via GitHub
pengding-stripe commented on code in PR #14443: URL: https://github.com/apache/pinot/pull/14443#discussion_r1842592785 ## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java: #

Re: [PR] Spark upsert table backfill support [pinot]

2024-11-14 Thread via GitHub
pengding-stripe commented on code in PR #14443: URL: https://github.com/apache/pinot/pull/14443#discussion_r1842573810 ## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java: #

[PR] Bump org.codehaus.mojo:versions-maven-plugin from 2.17.1 to 2.18.0 [pinot]

2024-11-14 Thread via GitHub
dependabot[bot] opened a new pull request, #14454: URL: https://github.com/apache/pinot/pull/14454 Bumps [org.codehaus.mojo:versions-maven-plugin](https://github.com/mojohaus/versions) from 2.17.1 to 2.18.0. Release notes Sourced from https://github.com/mojohaus/versions/releases"

Re: [PR] Add broker setting to override default implicit query response limit [pinot]

2024-11-14 Thread via GitHub
bziobrowski commented on PR #14452: URL: https://github.com/apache/pinot/pull/14452#issuecomment-2476489469 I renamed the variables (but not setting keys) because, in my opinion, existing query response limit is badly named. It is not a default but rather an override because it applies even

Re: [PR] Add broker setting to override default implicit query response limit [pinot]

2024-11-14 Thread via GitHub
codecov-commenter commented on PR #14452: URL: https://github.com/apache/pinot/pull/14452#issuecomment-2476507087 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14452?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] Allow override of segment name when uploading a segment tar file [pinot]

2024-11-14 Thread via GitHub
codecov-commenter commented on PR #14455: URL: https://github.com/apache/pinot/pull/14455#issuecomment-2476369355 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14455?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u

Re: [PR] add metric of startup time for all components [pinot]

2024-11-14 Thread via GitHub
jadami10 commented on PR #14445: URL: https://github.com/apache/pinot/pull/14445#issuecomment-2476368759 fixed both, ty! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
tibrewalpratik17 commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842186746 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java: #

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.29.12 created (now 6139db26bb)

2024-11-14 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.29.12 in repository https://gitbox.apache.org/repos/asf/pinot.git at 6139db26bb Bump software.amazon.awssdk:bom from 2.29.11 to 2.29.12 N

[PR] Allow override of segment name when uploading a segment tar file [pinot]

2024-11-14 Thread via GitHub
vrajat opened a new pull request, #14455: URL: https://github.com/apache/pinot/pull/14455 In segment tar upload operation, the name of the segment is the file (or dir) name. Allow the client to specify a segment name instead. This feature is useful when writing tests or to simulate producti

(pinot) branch dependabot/maven/org.codehaus.mojo-versions-maven-plugin-2.18.0 created (now 8352499865)

2024-11-14 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.codehaus.mojo-versions-maven-plugin-2.18.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 8352499865 Bump org.codehaus.mojo:versions-maven-plugin f

[PR] Bump software.amazon.awssdk:bom from 2.29.11 to 2.29.12 [pinot]

2024-11-14 Thread via GitHub
dependabot[bot] opened a new pull request, #14453: URL: https://github.com/apache/pinot/pull/14453 Bumps software.amazon.awssdk:bom from 2.29.11 to 2.29.12. Most Recent Ignore Conditions Applied to This Pull Request | Dependency Name | Ignore Conditions | | --- | --- |

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842042118 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842080029 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/RefreshSegmentMinionClusterIntegrationTest.java: ## @@ -0,0 +1,414 @@ +/** + * Lic

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842031808 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842031808 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842042118 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842042118 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842042118 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842042118 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentrefresh/SegmentRefreshTaskExecutor.java:

Re: [PR] Minion Task to support automatic Segment Refresh [pinot]

2024-11-14 Thread via GitHub
swaminathanmanish commented on code in PR #14300: URL: https://github.com/apache/pinot/pull/14300#discussion_r1842031808 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskExecutor.java:

[PR] Add broker setting to override default implicit query response limit [pinot]

2024-11-14 Thread via GitHub
bziobrowski opened a new pull request, #14452: URL: https://github.com/apache/pinot/pull/14452 By default single stage query engine (aka v1) uses implicit limit of 10 rows when no limit is set. This PR adds `pinot.broker.default.query.response.limit` that allows for overriding it. It

  1   2   >