[pinot] branch master updated: Fix PerfBenchmarkRunner to skip the tmp dir (#9298)

2022-08-29 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new d1a71d8518 Fix PerfBenchmarkRunner to skip the tm

[GitHub] [pinot] Jackie-Jiang merged pull request #9298: Fix PerfBenchmarkRunner to skip the tmp dir

2022-08-29 Thread GitBox
Jackie-Jiang merged PR #9298: URL: https://github.com/apache/pinot/pull/9298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

[GitHub] [pinot] xiangfu0 commented on pull request #9294: Infer parquet reader type based on file metadata (wip)

2022-08-29 Thread GitBox
xiangfu0 commented on PR #9294: URL: https://github.com/apache/pinot/pull/9294#issuecomment-1231215626 Can you add a sample data file with decimal field and test the file is correctly parsed. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [pinot] snleee closed issue #9297: Flaky test for `RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments()`

2022-08-29 Thread GitBox
snleee closed issue #9297: Flaky test for `RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments()` URL: https://github.com/apache/pinot/issues/9297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [pinot] snleee commented on issue #9297: Flaky test for `RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments()`

2022-08-29 Thread GitBox
snleee commented on issue #9297: URL: https://github.com/apache/pinot/issues/9297#issuecomment-1231181492 #9253 @Jackie-Jiang already created the same issue and he actually added the above test to prove the flaky test. closing this ticket and we can address the fix in the above. -- Th

[GitHub] [pinot] snleee commented on issue #9297: Flaky test for `RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments()`

2022-08-29 Thread GitBox
snleee commented on issue #9297: URL: https://github.com/apache/pinot/issues/9297#issuecomment-1231178474 For this test, we run Helix's `AutoRebalanceStrategy.computePartitionAssignment()` for completed segments. We pass `node.capacity = targetSize (which we compute from "distFloor"`; howev

[GitHub] [pinot] siddharthteotia merged pull request #9282: [Cleanup] Break IntermediateResultsBlock into different results blocks

2022-08-29 Thread GitBox
siddharthteotia merged PR #9282: URL: https://github.com/apache/pinot/pull/9282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pino

[GitHub] [pinot] gortiz commented on issue #9292: return pre-sorted value on IntermediateResultsBlock

2022-08-29 Thread GitBox
gortiz commented on issue #9292: URL: https://github.com/apache/pinot/issues/9292#issuecomment-1231168693 Also, right now brokers temporally store the data from servers in a hash map, which implies to hash is twice (once when it is being read from the server and another one when data from d

[GitHub] [pinot] snleee commented on issue #9297: Flaky test for `RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments()`

2022-08-29 Thread GitBox
snleee commented on issue #9297: URL: https://github.com/apache/pinot/issues/9297#issuecomment-1231167798 Most of the time, the test passes with the `[29, 30, 30, 30, 29, 29, 30, 29, 29, 29]` -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [pinot] snleee opened a new issue, #9297: Flaky test for `RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments()`

2022-08-29 Thread GitBox
snleee opened a new issue, #9297: URL: https://github.com/apache/pinot/issues/9297 ``` java.lang.AssertionError: Expect at least: 29 segments assigned per instance, got: [30, 30, 30, 30, 29, 29, 29, 29, 30, 28] Expected :true Actual :false at org.testng.Assert

[GitHub] [pinot] snleee commented on a diff in pull request #9295: Enable Consistent Data Push for Standalone Segment Push Job Runners

2022-08-29 Thread GitBox
snleee commented on code in PR #9295: URL: https://github.com/apache/pinot/pull/9295#discussion_r957981334 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SegmentUploadIntegrationTest.java: ## @@ -81,10 +89,15 @@ protected List getBloomFilterColumns()

[GitHub] [pinot] yuanbenson closed pull request #9295: Enable Consistent Data Push for Standalone Segment Push Job Runners

2022-08-29 Thread GitBox
yuanbenson closed pull request #9295: Enable Consistent Data Push for Standalone Segment Push Job Runners URL: https://github.com/apache/pinot/pull/9295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[pinot] branch master updated: Add capabilities to ingest from another stream without disabling the realtime table (#9289)

2022-08-29 Thread sajjad
This is an automated email from the ASF dual-hosted git repository. sajjad pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new a273af0262 Add capabilities to ingest from anothe

[GitHub] [pinot] sajjad-moradi merged pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
sajjad-moradi merged PR #9289: URL: https://github.com/apache/pinot/pull/9289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9296: [WIP] Add Support for Cast Function on MV Columns

2022-08-29 Thread GitBox
siddharthteotia commented on code in PR #9296: URL: https://github.com/apache/pinot/pull/9296#discussion_r957847415 ## pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CastTransformFunction.java: ## @@ -84,6 +84,22 @@ public void init(List arguments, M

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9296: [WIP] Add Support for Cast Function on MV Columns

2022-08-29 Thread GitBox
siddharthteotia commented on code in PR #9296: URL: https://github.com/apache/pinot/pull/9296#discussion_r957846649 ## pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CastTransformFunction.java: ## @@ -84,6 +84,22 @@ public void init(List arguments, M

[GitHub] [pinot] SabrinaZhaozyf opened a new pull request, #9296: [WIP] Add Support for Cast Function on MV Columns

2022-08-29 Thread GitBox
SabrinaZhaozyf opened a new pull request, #9296: URL: https://github.com/apache/pinot/pull/9296 Label: `feature` This PR adds support for Cast transform function for mv columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9282: [Cleanup] Break IntermediateResultsBlock into different results blocks

2022-08-29 Thread GitBox
Jackie-Jiang commented on code in PR #9282: URL: https://github.com/apache/pinot/pull/9282#discussion_r957828740 ## pinot-core/src/main/java/org/apache/pinot/core/plan/maker/InstancePlanMakerImplV2.java: ## @@ -273,14 +273,13 @@ public Plan makeStreamingInstancePlan(List indexS

[GitHub] [pinot] npawar commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
npawar commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957827576 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java: ## @@ -77,14 +77,29 @@ public Response pauseConsumption( @PO

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9282: [Cleanup] Break IntermediateResultsBlock into different results blocks

2022-08-29 Thread GitBox
Jackie-Jiang commented on code in PR #9282: URL: https://github.com/apache/pinot/pull/9282#discussion_r957827047 ## pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java: ## @@ -34,24 +34,21 @@ public class InstanceResponseOperator extends B

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9282: [Cleanup] Break IntermediateResultsBlock into different results blocks

2022-08-29 Thread GitBox
Jackie-Jiang commented on code in PR #9282: URL: https://github.com/apache/pinot/pull/9282#discussion_r957812873 ## pinot-core/src/main/java/org/apache/pinot/core/operator/AcquireReleaseColumnsSegmentOperator.java: ## @@ -58,8 +58,8 @@ public AcquireReleaseColumnsSegmentOperator

[GitHub] [pinot] yuanbenson opened a new pull request, #9295: Consistent Data Push: Standalone Job Runners

2022-08-29 Thread GitBox
yuanbenson opened a new pull request, #9295: URL: https://github.com/apache/pinot/pull/9295 ### Description: This PR addresses https://github.com/apache/pinot/issues/9268 for the standalone execution framework segment push jobs: `SegmentMetadataPushJobRunner`, `SegmentTarPushJobRunner`,

[GitHub] [pinot] KKcorps commented on pull request #9288: Add Spark Job Launcher tool

2022-08-29 Thread GitBox
KKcorps commented on PR #9288: URL: https://github.com/apache/pinot/pull/9288#issuecomment-1230831952 It is failing in some cases like local environment but multi threaded. Working on fixing those post which we can merge. -- This is an automated message from the Apache Git Service. To res

[GitHub] [pinot] walterddr commented on a diff in pull request #9282: [Cleanup] Break IntermediateResultsBlock into different results blocks

2022-08-29 Thread GitBox
walterddr commented on code in PR #9282: URL: https://github.com/apache/pinot/pull/9282#discussion_r957726100 ## pinot-core/src/main/java/org/apache/pinot/core/plan/maker/InstancePlanMakerImplV2.java: ## @@ -273,14 +273,13 @@ public Plan makeStreamingInstancePlan(List indexSegm

[pinot] branch master updated: Force commit consuming segments (#9197)

2022-08-29 Thread sajjad
This is an automated email from the ASF dual-hosted git repository. sajjad pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 5ecca80cc8 Force commit consuming segments (#9197

[GitHub] [pinot] sajjad-moradi merged pull request #9197: Force commit consuming segments

2022-08-29 Thread GitBox
sajjad-moradi merged PR #9197: URL: https://github.com/apache/pinot/pull/9197 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.

[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
sajjad-moradi commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957671202 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1074,6 +1077,15 @@ IdealState ensureAll

[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
sajjad-moradi commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957671413 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1144,21 +1156,33 @@ IdealState ensureAl

[pinot] branch master updated: Fix issue with auth AccessType in Schema resource (#9293)

2022-08-29 Thread sajjad
This is an automated email from the ASF dual-hosted git repository. sajjad pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 8767437931 Fix issue with auth AccessType in Sche

[GitHub] [pinot] sajjad-moradi merged pull request #9293: Fix issue with auth AccessType in Schema REST endpoints

2022-08-29 Thread GitBox
sajjad-moradi merged PR #9293: URL: https://github.com/apache/pinot/pull/9293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.

[GitHub] [pinot] npawar commented on pull request #9114: Add support for Base64 Encode/Decode Scalar Functions

2022-08-29 Thread GitBox
npawar commented on PR #9114: URL: https://github.com/apache/pinot/pull/9114#issuecomment-1230676758 > @npawar Changes to user doc has been merged here ([pinot-contrib/pinot-docs#109](https://github.com/pinot-contrib/pinot-docs/pull/109)). Not sure why it hasn't been updated on documentatio

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
mcvsubbu commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957635296 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -1074,6 +1077,15 @@ IdealState ensureAllParti

[GitHub] [pinot] saurabhd336 opened a new pull request, #9294: Infer parquet reader type based on file metadata (wip)

2022-08-29 Thread GitBox
saurabhd336 opened a new pull request, #9294: URL: https://github.com/apache/pinot/pull/9294 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat`

[GitHub] [pinot] walterddr commented on issue #9204: Proper support for Array/List type

2022-08-29 Thread GitBox
walterddr commented on issue #9204: URL: https://github.com/apache/pinot/issues/9204#issuecomment-1230635716 updated the descriptions, thanks @Jackie-Jiang , i think we can actually support these out of MV columns. see: previous issue https://github.com/apache/pinot/issues/6083. I th

[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
sajjad-moradi commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957621762 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java: ## @@ -77,14 +77,29 @@ public Response pauseConsumption(

[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
sajjad-moradi commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957614663 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java: ## @@ -77,14 +77,29 @@ public Response pauseConsumption(

[GitHub] [pinot] sajjad-moradi opened a new pull request, #9293: Fix issue with auth AccessType in Schema REST endpoints

2022-08-29 Thread GitBox
sajjad-moradi opened a new pull request, #9293: URL: https://github.com/apache/pinot/pull/9293 AccessType for a couple of Schema REST endpoints were configured incorrectly. This PR fixes those issue. -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
mcvsubbu commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957559540 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java: ## @@ -77,14 +77,29 @@ public Response pauseConsumption( @

[GitHub] [pinot] SabrinaZhaozyf commented on pull request #9114: Add support for Base64 Encode/Decode Scalar Functions

2022-08-29 Thread GitBox
SabrinaZhaozyf commented on PR #9114: URL: https://github.com/apache/pinot/pull/9114#issuecomment-1230513674 @npawar Changes to user doc has been merged here (https://github.com/pinot-contrib/pinot-docs/pull/109). Not sure why it hasn't been updated on documentation yet. -- This is an au

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9289: Add capabilities to ingest from another stream without disabling the realtime table

2022-08-29 Thread GitBox
mcvsubbu commented on code in PR #9289: URL: https://github.com/apache/pinot/pull/9289#discussion_r957479004 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java: ## @@ -77,14 +77,29 @@ public Response pauseConsumption( @

[GitHub] [pinot] walterddr opened a new issue, #9292: return pre-sorted value on IntermediateResultsBlock

2022-08-29 Thread GitBox
walterddr opened a new issue, #9292: URL: https://github.com/apache/pinot/issues/9292 Currently, if a query is run with the ORDER BY clause. data is - in selection only, data is sorted and trimmed to the limit - in aggregation/group-by/distinct, data is unsorted but trimmed to the lim

[GitHub] [pinot] walterddr commented on a diff in pull request #8927: Proper null handling in SELECT, ORDER BY, DISTINCT, and GROUP BY

2022-08-29 Thread GitBox
walterddr commented on code in PR #8927: URL: https://github.com/apache/pinot/pull/8927#discussion_r957435868 ## pinot-core/src/main/java/org/apache/pinot/core/query/selection/SelectionOperatorUtils.java: ## @@ -227,16 +229,42 @@ public static void mergeWithOrdering(PriorityQue

[pinot] branch master updated: Improve server query cancellation and timeout checking during execution (#9286)

2022-08-29 Thread siddteotia
This is an automated email from the ASF dual-hosted git repository. siddteotia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 15b16e8f10 Improve server query cancellation

[GitHub] [pinot] siddharthteotia merged pull request #9286: Improve server query cancellation and timeout checking during execution

2022-08-29 Thread GitBox
siddharthteotia merged PR #9286: URL: https://github.com/apache/pinot/pull/9286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pino