[GitHub] [pinot] KKcorps opened a new pull request, #10406: Handle empty validDocId snapshots during reloading

KKcorps opened a new pull request, #10406: URL: https://github.com/apache/pinot/pull/10406 If a segment has `validDocIds` as empty and `enableSnapshot: true`, we persist an empty `validDocIdSnapshot` file on disk. During the next rest however, if we find the `validDocIdSnapshot` file

[GitHub] [pinot] cbalci commented on pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on PR #10394: URL: https://github.com/apache/pinot/pull/10394#issuecomment-1463367064 Thanks for the reviews @KKcorps @GSharayu @jackjlli ! I tried to address/resolve your comments, ptal. -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [pinot] cbalci commented on pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on PR #10394: URL: https://github.com/apache/pinot/pull/10394#issuecomment-1463366238 from @KKcorps : > Can you try to test it in cluster mode while running on a proper YARN, AWS EMR or DataProc cluster. Good suggestion. I ended up testing this in our YARN environm

[GitHub] [pinot] walterddr commented on a diff in pull request #10336: [Multistage] Allow queries on multiple tables of same tenant to be executed from controller UI

walterddr commented on code in PR #10336: URL: https://github.com/apache/pinot/pull/10336#discussion_r1131946143 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java: ## @@ -170,10 +171,19 @@ private String getMultiStageQueryRespons

[GitHub] [pinot] walterddr commented on a diff in pull request #10336: [Multistage] Allow queries on multiple tables of same tenant to be executed from controller UI

walterddr commented on code in PR #10336: URL: https://github.com/apache/pinot/pull/10336#discussion_r1131946143 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java: ## @@ -170,10 +171,19 @@ private String getMultiStageQueryRespons

[GitHub] [pinot] walterddr commented on a diff in pull request #10336: [Multistage] Allow queries on multiple tables of same tenant to be executed from controller UI

walterddr commented on code in PR #10336: URL: https://github.com/apache/pinot/pull/10336#discussion_r1131945053 ## pinot-common/src/main/java/org/apache/pinot/common/utils/request/RequestUtils.java: ## @@ -48,6 +48,14 @@ public class RequestUtils { private RequestUtils() {

[GitHub] [pinot] walterddr commented on a diff in pull request #10336: [Multistage] Allow queries on multiple tables of same tenant to be executed from controller UI

walterddr commented on code in PR #10336: URL: https://github.com/apache/pinot/pull/10336#discussion_r1131945053 ## pinot-common/src/main/java/org/apache/pinot/common/utils/request/RequestUtils.java: ## @@ -48,6 +48,14 @@ public class RequestUtils { private RequestUtils() {

[GitHub] [pinot] walterddr commented on a diff in pull request #10336: [Multistage] Allow queries on multiple tables of same tenant to be executed from controller UI

walterddr commented on code in PR #10336: URL: https://github.com/apache/pinot/pull/10336#discussion_r1131943722 ## pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java: ## @@ -130,6 +131,8 @@ public static void setMaxLinesOfStackTrace(int maxLinesOf

[GitHub] [pinot] swaminathanmanish commented on a diff in pull request #10359: RebalanceStatus API changes

swaminathanmanish commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1131622528 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -0,0 +1,168 @@ +/** + * Licensed

[GitHub] [pinot] swaminathanmanish commented on a diff in pull request #10359: RebalanceStatus API changes

swaminathanmanish commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1131622528 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -0,0 +1,168 @@ +/** + * Licensed

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #10405: [Refactor] Introduce BaseProjectOperator and ValueBlock

Jackie-Jiang opened a new pull request, #10405: URL: https://github.com/apache/pinot/pull/10405 Both `Projection` and `Transform` in pinot are SQL project operation. This PR introduce the `BaseProjectOperator` base class that represent the executor for SQL project, which generates the `Va

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131851353 ## pinot-connectors/pinot-spark-3-connector/README.md: ## @@ -0,0 +1,69 @@ + +# Spark-Pinot Connector + +Spark-pinot connector to read and write data from/to Pinot. Rev

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131850729 ## pinot-connectors/pinot-spark-3-connector/src/test/resources/schema/spark-schema.json: ## @@ -0,0 +1,105 @@ +{ + "type" : "struct", + "fields" : [ { +"name" : "f

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131848657 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ + +# Read Model + +Connector can scan offline, hybrid and realtime tables.

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131847893 ## pinot-connectors/pinot-spark-3-connector/src/main/scala/org/apache/pinot/connector/spark/v3/datasource/PinotDataSource.scala: ## @@ -0,0 +1,51 @@ +/** + * Licensed to

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131800899 ## pinot-connectors/pinot-spark-3-connector/README.md: ## @@ -0,0 +1,69 @@ +

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131798826 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ +

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131798826 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ +

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131798826 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ +

[GitHub] [pinot] ankitsultana commented on a diff in pull request #10401: [multistage] early terminate from query dispatcher

ankitsultana commented on code in PR #10401: URL: https://github.com/apache/pinot/pull/10401#discussion_r1131779563 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java: ## @@ -43,10 +46,16 @@ public class OpChainSchedulerServ

[GitHub] [pinot] cbalci commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

cbalci commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r113130 ## pinot-connectors/pinot-spark-3-connector/pom.xml: ## @@ -0,0 +1,324 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

[pinot] branch master updated: Enhance select order-by combine to use merge sort (#10357)

This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 6bd8a7dcff Enhance select order-by combine to use

[GitHub] [pinot] Jackie-Jiang merged pull request #10357: Enhance select order-by combine to use merge sort

Jackie-Jiang merged PR #10357: URL: https://github.com/apache/pinot/pull/10357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] codecov-commenter commented on pull request #10403: Make startReplaceSegments() more robust for NPE

codecov-commenter commented on PR #10403: URL: https://github.com/apache/pinot/pull/10403#issuecomment-1462933089 ## [Codecov](https://codecov.io/gh/apache/pinot/pull/10403?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Sof

[GitHub] [pinot] richardstartin closed pull request #10404: improve regexp like evaluation against dictionary

richardstartin closed pull request #10404: improve regexp like evaluation against dictionary URL: https://github.com/apache/pinot/pull/10404 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [pinot] richardstartin commented on pull request #10404: improve regexp like evaluation against dictionary

richardstartin commented on PR #10404: URL: https://github.com/apache/pinot/pull/10404#issuecomment-1462932810 On second thoughts, this seems like an objectively bad idea wrt the rest of the query engine. If some other index filters down to 100 rows, scanning a 100k+ cardinality dictionary

[GitHub] [pinot] richardstartin opened a new pull request, #10404: improve regexp like evaluation against dictionary

richardstartin opened a new pull request, #10404: URL: https://github.com/apache/pinot/pull/10404 This was suggest on [slack](https://apache-pinot.slack.com/archives/C011C9JHN7R/p1677947603760289) and was how I assumed this predicate evaluator worked (despite having modified it myself twic

[pinot] 01/01: improve regexp like evaluation against dictionary by scanning dictionary to build set of matching dictIds

This is an automated email from the ASF dual-hosted git repository. richardstartin pushed a commit to branch rgs/optimise-regexp-like in repository https://gitbox.apache.org/repos/asf/pinot.git commit 7aca15f3f48490b909c917d891cae152eef7192f Author: Richard Startin AuthorDate: Thu Mar 9 22:36:16

[pinot] branch rgs/optimise-regexp-like created (now 7aca15f3f4)

This is an automated email from the ASF dual-hosted git repository. richardstartin pushed a change to branch rgs/optimise-regexp-like in repository https://gitbox.apache.org/repos/asf/pinot.git at 7aca15f3f4 improve regexp like evaluation against dictionary by scanning dictionary to build

[GitHub] [pinot] jackjlli commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

jackjlli commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131646222 ## pinot-connectors/pinot-spark-3-connector/src/test/resources/schema/pinot-schema.json: ## @@ -0,0 +1,75 @@ +{ + "schemaName" : "schemaName", + "dimensionFieldSpecs

[GitHub] [pinot] walterddr opened a new issue, #10402: [multistage] fix stats/trace

walterddr opened a new issue, #10402: URL: https://github.com/apache/pinot/issues/10402 See TODO from #10390 1. if not enableTrace, disable all complexity in intermediary Operators, only MailboxSend and MailboxReceive should be dealing with metadata (e.g. no need to record operator

[GitHub] [pinot] klsince commented on a diff in pull request #10191: [Index SPI] IndexType

klsince commented on code in PR #10191: URL: https://github.com/apache/pinot/pull/10191#discussion_r1131613995 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexCreator.java: ## @@ -0,0 +1,62 @@ +/** + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] [pinot] swaminathanmanish commented on a diff in pull request #10359: RebalanceStatus API changes

swaminathanmanish commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1131622528 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -0,0 +1,168 @@ +/** + * Licensed

[GitHub] [pinot] swaminathanmanish commented on a diff in pull request #10359: RebalanceStatus API changes

swaminathanmanish commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1131616096 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -622,7 +622,8 @@ public SuccessResponse re

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10357: Enhance select order-by combine to use merge sort

Jackie-Jiang commented on code in PR #10357: URL: https://github.com/apache/pinot/pull/10357#discussion_r1131492493 ## pinot-core/src/main/java/org/apache/pinot/core/query/selection/SelectionOperatorService.java: ## @@ -78,6 +77,8 @@ public SelectionOperatorService(QueryContext

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10357: Enhance select order-by combine to use merge sort

Jackie-Jiang commented on code in PR #10357: URL: https://github.com/apache/pinot/pull/10357#discussion_r1131489683 ## pinot-core/src/main/java/org/apache/pinot/core/query/selection/SelectionOperatorUtils.java: ## @@ -192,37 +192,81 @@ public static DataSchema getResultTableDat

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10357: Enhance select order-by combine to use merge sort

Jackie-Jiang commented on code in PR #10357: URL: https://github.com/apache/pinot/pull/10357#discussion_r1131485629 ## pinot-core/src/main/java/org/apache/pinot/core/query/selection/SelectionOperatorUtils.java: ## @@ -192,37 +192,81 @@ public static DataSchema getResultTableDat

[GitHub] [pinot] Jackie-Jiang commented on pull request #10389: feat(ui): persist oidc auth details

Jackie-Jiang commented on PR #10389: URL: https://github.com/apache/pinot/pull/10389#issuecomment-1462617118 cc @joshigaurava @apucher -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[pinot] branch master updated: Let applyAnd to be applied using different window sizes (#10372)

This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 221db828cf Let applyAnd to be applied using diffe

[GitHub] [pinot] Jackie-Jiang merged pull request #10372: Let applyAnd to be applied using different window sizes

Jackie-Jiang merged PR #10372: URL: https://github.com/apache/pinot/pull/10372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131260556 ## pinot-connectors/pinot-spark-3-connector/src/main/scala/org/apache/pinot/connector/spark/v3/datasource/PinotDataSource.scala: ## @@ -0,0 +1,51 @@ +/** + * Licensed

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131257095 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ + +# Read Model + +Connector can scan offline, hybrid and realtime tables

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131260556 ## pinot-connectors/pinot-spark-3-connector/src/main/scala/org/apache/pinot/connector/spark/v3/datasource/PinotDataSource.scala: ## @@ -0,0 +1,51 @@ +/** + * Licensed

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131260556 ## pinot-connectors/pinot-spark-3-connector/src/main/scala/org/apache/pinot/connector/spark/v3/datasource/PinotDataSource.scala: ## @@ -0,0 +1,51 @@ +/** + * Licensed

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131258298 ## pinot-connectors/pinot-spark-3-connector/pom.xml: ## @@ -0,0 +1,324 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instanc

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131257095 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ + +# Read Model + +Connector can scan offline, hybrid and realtime tables

[GitHub] [pinot] GSharayu commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

GSharayu commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1131253107 ## pinot-connectors/pinot-spark-3-connector/documentation/read_model.md: ## @@ -0,0 +1,140 @@ + +# Read Model + +Connector can scan offline, hybrid and realtime tables

[GitHub] [pinot] gortiz commented on a diff in pull request #10372: Let applyAnd to be applied using different window sizes

gortiz commented on code in PR #10372: URL: https://github.com/apache/pinot/pull/10372#discussion_r113113 ## pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java: ## @@ -87,7 +94,7 @@ public int next() { int limit; int

[GitHub] [pinot] gortiz commented on a diff in pull request #10372: Let applyAnd to be applied using different window sizes

gortiz commented on code in PR #10372: URL: https://github.com/apache/pinot/pull/10372#discussion_r1131137892 ## pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java: ## @@ -47,15 +47,17 @@ public final class MVScanDocIdIterator impleme

[GitHub] [pinot] gortiz commented on pull request #10352: Encapsulate changes in IndexLoadingConfig and SegmentGeneratorConfig

gortiz commented on PR #10352: URL: https://github.com/apache/pinot/pull/10352#issuecomment-1462180991 About the tests failing: I've execute them locally and they seem to work. I will do another fake push to force CI to run again, but I don't think these problems are related to this PR --

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10359: RebalanceStatus API changes

saurabhd336 commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1130914572 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -0,0 +1,168 @@ +/** + * Licensed to the

[GitHub] [pinot] codecov-commenter commented on pull request #10389: feat(ui): persist auth details

codecov-commenter commented on PR #10389: URL: https://github.com/apache/pinot/pull/10389#issuecomment-1461872284 # [Codecov](https://codecov.io/gh/apache/pinot/pull/10389?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Soft

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10359: RebalanceStatus API changes

saurabhd336 commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1130868632 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -622,7 +622,8 @@ public SuccessResponse reloadSe

[GitHub] [pinot] KKcorps commented on pull request #10394: Pinot Spark Connector for Spark3

KKcorps commented on PR #10394: URL: https://github.com/apache/pinot/pull/10394#issuecomment-1461770973 Thanks for the contribution! Can you try to test it in cluster mode while running on a proper YARN, AWS EMR or DataProc cluster. Some of the times spark jobs inside pinot fail in th

[GitHub] [pinot] KKcorps commented on a diff in pull request #10394: Pinot Spark Connector for Spark3

KKcorps commented on code in PR #10394: URL: https://github.com/apache/pinot/pull/10394#discussion_r1130794374 ## pinot-connectors/pinot-spark-3-connector/pom.xml: ## @@ -0,0 +1,324 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance

[GitHub] [pinot] gortiz commented on pull request #10191: [Index SPI] IndexType

gortiz commented on PR #10191: URL: https://github.com/apache/pinot/pull/10191#issuecomment-1461684549 > Should we rename this method to getName()? I think `id` is a better description for this concept. Unlike `name`s, `id`s usually imply unicity and stability in time, while `name` is

[GitHub] [pinot] gortiz commented on a diff in pull request #10191: [Index SPI] IndexType

gortiz commented on code in PR #10191: URL: https://github.com/apache/pinot/pull/10191#discussion_r1130730468 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexType.java: ## @@ -0,0 +1,130 @@ +/** + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [pinot] gortiz commented on a diff in pull request #10191: [Index SPI] IndexType

gortiz commented on code in PR #10191: URL: https://github.com/apache/pinot/pull/10191#discussion_r1130723174 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/FieldIndexConfigs.java: ## @@ -0,0 +1,131 @@ +/** + * Licensed to the Apache Software Foundation (A