[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #10964: Allow custom segment grouping in MergeRollupTask based on lineage metadata

2023-06-26 Thread via GitHub
sajjad-moradi commented on code in PR #10964: URL: https://github.com/apache/pinot/pull/10964#discussion_r1242797215 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/lineage/DefaultLineageManager.java: ## @@ -105,6 +107,11 @@ public void updateLineageForR

[GitHub] [pinot] jtao15 commented on a diff in pull request #10964: Allow custom segment grouping in MergeRollupTask based on lineage metadata

2023-06-26 Thread via GitHub
jtao15 commented on code in PR #10964: URL: https://github.com/apache/pinot/pull/10964#discussion_r1243152915 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/lineage/LineageManager.java: ## @@ -70,4 +71,14 @@ void updateLineageForRevertReplaceSegments(Ta

[GitHub] [pinot] wirybeaver opened a new issue, #10981: json_extract_scalar times out when the expression contains array

2023-06-26 Thread via GitHub
wirybeaver opened a new issue, #10981: URL: https://github.com/apache/pinot/issues/10981 json_extract_scala times out even though the returned number of rows is only 1. ``` SELECT json_extract_scalar(direct_tags, '$.root_workflow[0].tag_value', 'STRING') FROM foo where event

[GitHub] [pinot] walterddr commented on issue #10919: Vector embeddings support in Pinot

2023-06-26 Thread via GitHub
walterddr commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1608653404 CPU solutions only make sense in certain scenarios IMO and I am not sure if those are fit. - Q: can it perform significantly better in specific use cases, for example ANNS use

[GitHub] [pinot] abhioncbr opened a new issue, #10980: Failing local execution of Pinot.

2023-06-26 Thread via GitHub
abhioncbr opened a new issue, #10980: URL: https://github.com/apache/pinot/issues/10980 The local startup of Pinot(built from the `master` branch) using the command `./bin/pinot-admin.sh QuickStart -type batch` is failing with the following exception. ``` TargetExternalViewCalcWor

[pinot] branch master updated: Add minion observability for segment upload/download failures (#10978)

2023-06-26 Thread mcvsubbu
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2d30e28bed Add minion observability for segment

[GitHub] [pinot] mcvsubbu merged pull request #10978: Add minion observability for segment upload/download failures

2023-06-26 Thread via GitHub
mcvsubbu merged PR #10978: URL: https://github.com/apache/pinot/pull/10978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10891: Ensure min/max value generation in the segment metadata.

2023-06-26 Thread via GitHub
Jackie-Jiang commented on code in PR #10891: URL: https://github.com/apache/pinot/pull/10891#discussion_r1243059714 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/columnminmaxvalue/ColumnMinMaxValueGenerator.java: ## @@ -119,65 +121,126

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10891: Ensure min/max value generation in the segment metadata.

2023-06-26 Thread via GitHub
Jackie-Jiang commented on code in PR #10891: URL: https://github.com/apache/pinot/pull/10891#discussion_r1243058344 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java: ## @@ -570,12 +570,14 @@ public static v

[pinot] branch master updated: Make TransformBlockValSet::getNullBitmap return the cache. (#10967)

2023-06-26 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new a62cb0b271 Make TransformBlockValSet::getNullBitm

[GitHub] [pinot] Jackie-Jiang merged pull request #10967: Make TransformBlockValSet::getNullBitmap return the cache.

2023-06-26 Thread via GitHub
Jackie-Jiang merged PR #10967: URL: https://github.com/apache/pinot/pull/10967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] Jackie-Jiang commented on pull request #10961: Backfill segments into real-time table

2023-06-26 Thread via GitHub
Jackie-Jiang commented on PR #10961: URL: https://github.com/apache/pinot/pull/10961#issuecomment-1608598720 cc @snleee -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[pinot] branch master updated: Provide results in CompletableFuture for java clients and expose metrics (#10326)

2023-06-26 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 48df78e476 Provide results in CompletableFuture f

[GitHub] [pinot] Jackie-Jiang merged pull request #10326: Provide results in CompletableFuture for java clients and expose metrics

2023-06-26 Thread via GitHub
Jackie-Jiang merged PR #10326: URL: https://github.com/apache/pinot/pull/10326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] Jackie-Jiang closed pull request #10613: [feature] [null support # 10] Add null support in all transform function and pass the bitmap to upstream

2023-06-26 Thread via GitHub
Jackie-Jiang closed pull request #10613: [feature] [null support # 10] Add null support in all transform function and pass the bitmap to upstream URL: https://github.com/apache/pinot/pull/10613 -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [pinot] GSharayu commented on pull request #10978: Add minion observability for segment upload/download failures

2023-06-26 Thread via GitHub
GSharayu commented on PR #10978: URL: https://github.com/apache/pinot/pull/10978#issuecomment-1608576705 The changes look good to me. Thanks for adding them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [pinot] shenyu0127 commented on pull request #10967: Make ProjectionBlockValSet::getNullBitmap return a copy of the cache.

2023-06-26 Thread via GitHub
shenyu0127 commented on PR #10967: URL: https://github.com/apache/pinot/pull/10967#issuecomment-1608572593 Had an offline sync with @Jackie-Jiang , we will revert the change and make `TransformBlockValSet::getNullBitmap` return the cache. - For API design perspective, returning the ca

[GitHub] [pinot] walterddr commented on issue #10901: [multistage][feature] support explain on physical plan

2023-06-26 Thread via GitHub
walterddr commented on issue #10901: URL: https://github.com/apache/pinot/issues/10901#issuecomment-160850 yeah that's OK. let me know if you have any problems. - i think changing the syntatic plugin part might be a bit different and I am not even sure calcite's parser allows that pl

[GitHub] [pinot] zhouxiz9 commented on pull request #10979: Skip generating rollup task with all merged segments

2023-06-26 Thread via GitHub
zhouxiz9 commented on PR #10979: URL: https://github.com/apache/pinot/pull/10979#issuecomment-1608456603 Hi @jtao15 could you help review this change? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [pinot] zhouxiz9 opened a new pull request, #10979: Skip generating rollup task with all merged segments

2023-06-26 Thread via GitHub
zhouxiz9 opened a new pull request, #10979: URL: https://github.com/apache/pinot/pull/10979 This PR adds a filtering step during the minion task generation process to skip generating rollup task with all merged segments. * For non rollup task, the task generation process stays the sam

[GitHub] [pinot] codecov-commenter commented on pull request #10978: Add minion observability for segment upload/download failures

2023-06-26 Thread via GitHub
codecov-commenter commented on PR #10978: URL: https://github.com/apache/pinot/pull/10978#issuecomment-1608406049 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10978?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #10964: Allow custom segment grouping in MergeRollupTask based on lineage metadata

2023-06-26 Thread via GitHub
sajjad-moradi commented on code in PR #10964: URL: https://github.com/apache/pinot/pull/10964#discussion_r1242797215 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/lineage/DefaultLineageManager.java: ## @@ -105,6 +107,11 @@ public void updateLineageForR

[GitHub] [pinot] codecov-commenter commented on pull request #10977: allow to add custom context into query trace

2023-06-26 Thread via GitHub
codecov-commenter commented on PR #10977: URL: https://github.com/apache/pinot/pull/10977#issuecomment-1608287569 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10977?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] mcvsubbu opened a new pull request, #10978: Add minion observability for segment upload/download failures

2023-06-26 Thread via GitHub
mcvsubbu opened a new pull request, #10978: URL: https://github.com/apache/pinot/pull/10978 Currently, minions do not provide observability into upload or download failures. Added mechanism to log errors and bump metrics when either upload or download fails, so that operators can set alerts

[GitHub] [pinot] tibrewalpratik17 commented on pull request #10873: Use pinot-query-planner utils to extract table names in pinot-controller

2023-06-26 Thread via GitHub
tibrewalpratik17 commented on PR #10873: URL: https://github.com/apache/pinot/pull/10873#issuecomment-1608257834 > for now, I think it is better to fix the routing issue (e.g. if you can't find any table, route it to any random tenant) > later we should fix this issue --> we should not ev

[GitHub] [pinot] klsince opened a new pull request, #10977: allow to add custom context into query trace

2023-06-26 Thread via GitHub
klsince opened a new pull request, #10977: URL: https://github.com/apache/pinot/pull/10977 Allow custom context to be added into query trace and returned as part of the TraceInfo field in query response. -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [pinot] itschrispeck opened a new issue, #10976: _luceneRealtimeReaders is constantly increasing, causing text index creation lag

2023-06-26 Thread via GitHub
itschrispeck opened a new issue, #10976: URL: https://github.com/apache/pinot/issues/10976 I added some logs to track [`_luceneRealtimeReaders`](https://github.com/apache/pinot/blob/release-0.12.1/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/Re

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
xiangfu0 commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242589834 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/MaxAggregationFunction.java: ## @@ -295,6 +295,22 @@ public Double merge(Double intermedia

[GitHub] [pinot] eupraxis1981 opened a new issue, #10974: Clean up table management CLI commands

2023-06-26 Thread via GitHub
eupraxis1981 opened a new issue, #10974: URL: https://github.com/apache/pinot/issues/10974 # What's the problem? Cannot add schema and tables separately via CLI. Current AddTable CLI command assumes table(s) and associated schema are uploaded together. * Prevents using AddSchema

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
vvivekiyer commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242589650 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/NewAggregateOperator.java: ## @@ -0,0 +1,378 @@ +/** + * Licensed to the Apache Software

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
vvivekiyer commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242588866 ## pinot-core/src/main/java/org/apache/pinot/core/common/IntermediateStageBlockValSet.java: ## @@ -0,0 +1,250 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] [pinot] walterddr commented on pull request #10873: Use pinot-query-planner utils to extract table names in pinot-controller

2023-06-26 Thread via GitHub
walterddr commented on PR #10873: URL: https://github.com/apache/pinot/pull/10873#issuecomment-1607977608 > Yes @walterddr exactly! I have updated the QueryGenerator logic to swap values in `BETWEEN` clause if this case arises. But let me know if we want to follow any other route to solve t

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
vvivekiyer commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242566160 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/NewAggregateOperator.java: ## @@ -0,0 +1,378 @@ +/** + * Licensed to the Apache Software

[GitHub] [pinot] robertzych commented on a diff in pull request #10463: Segment compaction for upsert real-time tables

2023-06-26 Thread via GitHub
robertzych commented on code in PR #10463: URL: https://github.com/apache/pinot/pull/10463#discussion_r1242551885 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -489,14 +489,16 @@ static void setDefaultTimeValueIfInva

[GitHub] [pinot] walterddr commented on pull request #10873: Use pinot-query-planner utils to extract table names in pinot-controller

2023-06-26 Thread via GitHub
walterddr commented on PR #10873: URL: https://github.com/apache/pinot/pull/10873#issuecomment-1607950700 for now, I think it is better to fix the routing issue (e.g. if you can't find any table, route it to any random tenant) later we should fix this issue --> we should not even dispatch

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
vvivekiyer commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242534682 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionFactory.java: ## @@ -309,8 +313,8 @@ public static AggregationFunctio

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
vvivekiyer commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242529302 ## pinot-common/src/main/java/org/apache/pinot/common/request/context/ExpressionContext.java: ## @@ -74,8 +83,27 @@ public LiteralContext getLiteral(){ return _

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #10845: [Multistage] Runtime changes for leveraging V1 Aggregation Functions

2023-06-26 Thread via GitHub
vvivekiyer commented on code in PR #10845: URL: https://github.com/apache/pinot/pull/10845#discussion_r1242513304 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/PhysicalPlanVisitor.java: ## @@ -96,10 +102,85 @@ public MultiStageOperator visitMailboxSend

[GitHub] [pinot] mcvsubbu opened a new issue, #10973: Minion upload after purge may leave segment in an inconsistent state

2023-06-26 Thread via GitHub
mcvsubbu opened a new issue, #10973: URL: https://github.com/apache/pinot/issues/10973 This is a race condition that we have observed during the following sequence of events. - Minion purge task downloads a segment and purges some records - The task tries to upload the segment back to

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10948: Batch null check

2023-06-26 Thread via GitHub
Jackie-Jiang commented on code in PR #10948: URL: https://github.com/apache/pinot/pull/10948#discussion_r1242497965 ## pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/IngestionJobLauncher.java: ## @@ -143,7 +144,7 @@ private static void kickoffIngestionJob(SegmentG

[GitHub] [pinot] tibrewalpratik17 commented on pull request #10873: Use pinot-query-planner utils to extract table names in pinot-controller

2023-06-26 Thread via GitHub
tibrewalpratik17 commented on PR #10873: URL: https://github.com/apache/pinot/pull/10873#issuecomment-1607848525 > ah. i think that's b/c when you run a query similar to > > ``` > SELECT * FROM tbl WHERE longCol BETWEEN 0 AND -1 > ``` > > ^ this query is simplified by calc

[GitHub] [pinot] walterddr commented on pull request #10873: Use pinot-query-planner utils to extract table names in pinot-controller

2023-06-26 Thread via GitHub
walterddr commented on PR #10873: URL: https://github.com/apache/pinot/pull/10873#issuecomment-1607758742 ah. i think that's b/c when you run a query similar to ``` SELECT * FROM tbl WHERE longCol BETWEEN 0 AND -1 ``` ^ this query is simplified by calcite into `SELECT * FROM empt

[GitHub] [pinot] walterddr commented on issue #10921: Improve Syntax Error Messages

2023-06-26 Thread via GitHub
walterddr commented on issue #10921: URL: https://github.com/apache/pinot/issues/10921#issuecomment-1607754155 I think the error messages you shared are generated internally. but it is not piped back to the end-user. it has to do with 1. the boxing of the controller handler error on

[GitHub] [pinot] walterddr commented on a diff in pull request #10971: [multistage][feature] pipeline breaker on broker reduce

2023-06-26 Thread via GitHub
walterddr commented on code in PR #10971: URL: https://github.com/apache/pinot/pull/10971#discussion_r1241284837 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java: ## @@ -189,73 +191,60 @@ int submit(long requestId, DispatchableSub

[pinot] branch master updated: Changed Lists.of to ImmutableList.of for test in java8 (#10972)

2023-06-26 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 8170d435c5 Changed Lists.of to ImmutableList.of fo

[GitHub] [pinot] walterddr merged pull request #10972: Changed Lists.of to ImmutableList.of for java8

2023-06-26 Thread via GitHub
walterddr merged PR #10972: URL: https://github.com/apache/pinot/pull/10972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

[GitHub] [pinot] tibrewalpratik17 commented on pull request #10873: Use pinot-query-planner utils to extract table names in pinot-controller

2023-06-26 Thread via GitHub
tibrewalpratik17 commented on PR #10873: URL: https://github.com/apache/pinot/pull/10873#issuecomment-1607232924 hey @walterddr i was looking into the integration test failure and this is only happening for queries with `BETWEEN` clause. We lose the table information from RelNode at this

[GitHub] [pinot] gortiz commented on a diff in pull request #10963: Update dockerfile with recent jdk distro changes

2023-06-26 Thread via GitHub
gortiz commented on code in PR #10963: URL: https://github.com/apache/pinot/pull/10963#discussion_r1241776384 ## docker/images/pinot-base/pinot-base-build/openjdk.dockerfile: ## @@ -16,33 +16,37 @@ # specific language governing permissions and limitations # under the License.