[GitHub] [pinot] KKcorps opened a new issue, #8526: Make Pinot Spark 3.X compatible

2022-04-12 Thread GitBox
KKcorps opened a new issue, #8526: URL: https://github.com/apache/pinot/issues/8526 Current pinot used Apache spark 2.4.0 for running Spark Ingestion tasks. We need to upgrade it to Spark 3.x so that the latest spark version can be supported. We will also need to make a lot of depend

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849107128 ## pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/PredicateComparisonRewriter.java: ## @@ -42,6 +44,73 @@ public PinotQuery rewrite(PinotQuery pi

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849107128 ## pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/PredicateComparisonRewriter.java: ## @@ -42,6 +44,73 @@ public PinotQuery rewrite(PinotQuery pi

[GitHub] [pinot] saurabhd336 commented on pull request #8502: Add zk metadata to segment's metadata.properties file

2022-04-12 Thread GitBox
saurabhd336 commented on PR #8502: URL: https://github.com/apache/pinot/pull/8502#issuecomment-1097585574 @npawar @KKcorps Have addressed the comments. At this moment, I'm unable to tag it to issue https://github.com/apache/pinot/issues/8489 -- This is an automated message from the Apache

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849096065 ## pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/PredicateComparisonRewriter.java: ## @@ -42,6 +44,73 @@ public PinotQuery rewrite(PinotQuery pi

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849092454 ## pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/PredicateComparisonRewriter.java: ## @@ -42,6 +44,73 @@ public PinotQuery rewrite(PinotQuery pi

[GitHub] [pinot] KKcorps commented on issue #8512: [Feature Request] Handling optional parts in date format

2022-04-12 Thread GitBox
KKcorps commented on issue #8512: URL: https://github.com/apache/pinot/issues/8512#issuecomment-1097575529 From my benchmarks, java time seems 20-30% slower than Joda time. ``` Benchmark Mode Cnt Score Error Units BenchmarkJoda.benchmarkJav

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8502: Add zk metadata to segment's metadata.properties file

2022-04-12 Thread GitBox
saurabhd336 commented on code in PR #8502: URL: https://github.com/apache/pinot/pull/8502#discussion_r849083233 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java: ## @@ -53,11 +55,13 @@ public class RealtimeSegme

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8502: Add zk metadata to segment's metadata.properties file

2022-04-12 Thread GitBox
saurabhd336 commented on code in PR #8502: URL: https://github.com/apache/pinot/pull/8502#discussion_r849082596 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java: ## @@ -619,6 +621,12 @@ private void writeMe

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849077605 ## pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java: ## @@ -2246,6 +2246,19 @@ public void testFlattenAndOr() { } }

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849076944 ## pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java: ## @@ -2254,9 +2267,9 @@ public void testFlattenAndOr() { List opera

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r849076687 ## pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java: ## @@ -2246,6 +2246,19 @@ public void testFlattenAndOr() { } }

[GitHub] [pinot] siddharthteotia commented on pull request #8398: Allow disabling dict generation for High cardinality columns

2022-04-12 Thread GitBox
siddharthteotia commented on PR #8398: URL: https://github.com/apache/pinot/pull/8398#issuecomment-1097555078 Shall we merge this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [pinot] KKcorps closed issue #8363: Auth Token should not go into Logs

2022-04-12 Thread GitBox
KKcorps closed issue #8363: Auth Token should not go into Logs URL: https://github.com/apache/pinot/issues/8363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[pinot] branch master updated: Make getConfigMapWithPrefix auto append tailing dot if missing (#8522)

2022-04-12 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 900f01f288 Make getConfigMapWithPrefix auto appe

[GitHub] [pinot] xiangfu0 merged pull request #8522: Make IngestionConfigUtils.getConfigMapWithPrefix method auto append trailing dot if missing

2022-04-12 Thread GitBox
xiangfu0 merged PR #8522: URL: https://github.com/apache/pinot/pull/8522 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[GitHub] [pinot] WangCHX commented on a diff in pull request #8498: Support st_contains using H3 index

2022-04-12 Thread GitBox
WangCHX commented on code in PR #8498: URL: https://github.com/apache/pinot/pull/8498#discussion_r849048478 ## pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java: ## @@ -207,20 +219,20 @@ private BaseFilterOperator constructPhysicalOperator(FilterContext fi

[GitHub] [pinot] WangCHX commented on a diff in pull request #8498: Support st_contains using H3 index

2022-04-12 Thread GitBox
WangCHX commented on code in PR #8498: URL: https://github.com/apache/pinot/pull/8498#discussion_r849041657 ## pinot-core/src/test/java/org/apache/pinot/queries/H3IndexQueriesTest.java: ## @@ -210,11 +226,47 @@ public void testH3Index() Assert.assertNotNull(aggregationRes

[GitHub] [pinot] yupeng9 commented on pull request #8498: Support st_contains using H3 index

2022-04-12 Thread GitBox
yupeng9 commented on PR #8498: URL: https://github.com/apache/pinot/pull/8498#issuecomment-1097514961 High-level comment is that using H3 index can speed up query evaluation, but we need to ensure it does not change query results. H3's Polyfill returns a set of H3 for approximation, but we

[GitHub] [pinot] yupeng9 commented on a diff in pull request #8498: Support st_contains using H3 index

2022-04-12 Thread GitBox
yupeng9 commented on code in PR #8498: URL: https://github.com/apache/pinot/pull/8498#discussion_r849035448 ## pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java: ## @@ -207,20 +219,20 @@ private BaseFilterOperator constructPhysicalOperator(FilterContext fi

[pinot] branch minimize-instance-movement updated (2fe95b6461 -> 24fbc0cc44)

2022-04-12 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch minimize-instance-movement in repository https://gitbox.apache.org/repos/asf/pinot.git discard 2fe95b6461 Reuse InstanceReplicaGroupPartitionSelector add 24fbc0cc44 Reuse InstanceReplicaGroup

[GitHub] [pinot] timsants commented on a diff in pull request #8509: Fix table creation bug for invalid realtime consumer props

2022-04-12 Thread GitBox
timsants commented on code in PR #8509: URL: https://github.com/apache/pinot/pull/8509#discussion_r848989982 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -1385,7 +1385,9 @@ public void addTable(TableConfig tableCo

[GitHub] [pinot] timsants commented on a diff in pull request #8509: Fix table creation bug for invalid realtime consumer props

2022-04-12 Thread GitBox
timsants commented on code in PR #8509: URL: https://github.com/apache/pinot/pull/8509#discussion_r848989982 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -1385,7 +1385,9 @@ public void addTable(TableConfig tableCo

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8491: Add connection based FailureDetector

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8491: URL: https://github.com/apache/pinot/pull/8491#discussion_r848983278 ## pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java: ## @@ -214,87 +219,178 @@ private void processInstanceConfigChange() { l

[GitHub] [pinot] Jackie-Jiang commented on issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

2022-04-12 Thread GitBox
Jackie-Jiang commented on issue #8519: URL: https://github.com/apache/pinot/issues/8519#issuecomment-1097437319 @mcvsubbu The issue here is that for new servers, the `shutdownInProgress` flag is not set, and broker will start routing queries to it whenever the segment becomes ONLINE/CONSUMI

[GitHub] [pinot] mcvsubbu commented on issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

2022-04-12 Thread GitBox
mcvsubbu commented on issue #8519: URL: https://github.com/apache/pinot/issues/8519#issuecomment-1097433786 But then only those segments that are already loaded (and are ONLINE or CONSUMING in externalview on the new server) should get queries right? Is it a performance issue that you are f

[GitHub] [pinot] mcvsubbu commented on pull request #8525: Set 'shutdownInProgress' in server config when server starts before the startup check

2022-04-12 Thread GitBox
mcvsubbu commented on PR #8525: URL: https://github.com/apache/pinot/pull/8525#issuecomment-1097432547 > But we already do this here: https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java#L491 never mind, my

[GitHub] [pinot] kkrugler commented on a diff in pull request #8491: Add connection based FailureDetector

2022-04-12 Thread GitBox
kkrugler commented on code in PR #8491: URL: https://github.com/apache/pinot/pull/8491#discussion_r848975615 ## pinot-broker/src/main/java/org/apache/pinot/broker/failuredetector/BaseExponentialBackoffRetryFailureDetector.java: ## @@ -0,0 +1,174 @@ +/** + * Licensed to the Apach

[GitHub] [pinot] mayankshriv commented on issue #8363: Auth Token should not go into Logs

2022-04-12 Thread GitBox
mayankshriv commented on issue #8363: URL: https://github.com/apache/pinot/issues/8363#issuecomment-1097432018 @KKcorps can this issue be closed, seems like the fixes were already merged? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8491: Add connection based FailureDetector

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8491: URL: https://github.com/apache/pinot/pull/8491#discussion_r848974745 ## pinot-broker/src/main/java/org/apache/pinot/broker/failuredetector/FailureDetectorFactory.java: ## @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Software Founda

[GitHub] [pinot] npawar commented on a diff in pull request #8502: Add zk metadata to segment's metadata.properties file

2022-04-12 Thread GitBox
npawar commented on code in PR #8502: URL: https://github.com/apache/pinot/pull/8502#discussion_r848972965 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java: ## @@ -53,11 +55,13 @@ public class RealtimeSegmentCon

[pinot] branch minimize-instance-movement updated (b8df8d1046 -> 2fe95b6461)

2022-04-12 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch minimize-instance-movement in repository https://gitbox.apache.org/repos/asf/pinot.git from b8df8d1046 Minimize data movement between instances in pools add 2fe95b6461 Reuse InstanceReplic

[pinot] branch master updated: Manual tracing (#8485)

2022-04-12 Thread siddteotia
This is an automated email from the ASF dual-hosted git repository. siddteotia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 4f30ab2b89 Manual tracing (#8485) 4f30ab2b89

[GitHub] [pinot] siddharthteotia merged pull request #8485: Manual tracing

2022-04-12 Thread GitBox
siddharthteotia merged PR #8485: URL: https://github.com/apache/pinot/pull/8485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pino

[GitHub] [pinot] mayankshriv closed issue #8500: Bad table config may result in Zombie tables

2022-04-12 Thread GitBox
mayankshriv closed issue #8500: Bad table config may result in Zombie tables URL: https://github.com/apache/pinot/issues/8500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [pinot] mayankshriv commented on issue #8500: Bad table config may result in Zombie tables

2022-04-12 Thread GitBox
mayankshriv commented on issue #8500: URL: https://github.com/apache/pinot/issues/8500#issuecomment-1097427420 Fixed by #8509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8491: Add connection based FailureDetector

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8491: URL: https://github.com/apache/pinot/pull/8491#discussion_r848972208 ## pinot-broker/src/main/java/org/apache/pinot/broker/failuredetector/BaseExponentialBackoffRetryFailureDetector.java: ## @@ -0,0 +1,174 @@ +/** + * Licensed to the A

[GitHub] [pinot] mcvsubbu commented on pull request #8525: Set 'shutdownInProgress' in server config when server starts before the startup check

2022-04-12 Thread GitBox
mcvsubbu commented on PR #8525: URL: https://github.com/apache/pinot/pull/8525#issuecomment-1097426193 But we already do this here: https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java#L491 -- This is an aut

[pinot] branch master updated (e4b1d050a2 -> ad6ae117b5)

2022-04-12 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from e4b1d050a2 upgrade to hadoop 2.10.1 due to cves (#8478) add ad6ae117b5 add shouldNotMerge field in segment metadat

[GitHub] [pinot] Jackie-Jiang merged pull request #8508: add shouldNotMerge field in segment metadata to indicate if it is safe to merge

2022-04-12 Thread GitBox
Jackie-Jiang merged PR #8508: URL: https://github.com/apache/pinot/pull/8508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

[GitHub] [pinot] codecov-commenter commented on pull request #8525: Set 'shutdownInProgress' in server config when server starts before the startup check

2022-04-12 Thread GitBox
codecov-commenter commented on PR #8525: URL: https://github.com/apache/pinot/pull/8525#issuecomment-1097416597 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8525?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8523: URL: https://github.com/apache/pinot/pull/8523#discussion_r848961214 ## pinot-core/src/main/java/org/apache/pinot/core/operator/BaseOperator.java: ## @@ -35,7 +35,7 @@ public final T nextBlock() { if (Thread.interrupted()) {

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8523: URL: https://github.com/apache/pinot/pull/8523#discussion_r848960177 ## pinot-core/src/main/java/org/apache/pinot/core/plan/CombinePlanNode.java: ## @@ -73,87 +76,31 @@ public CombinePlanNode(List planNodes, QueryContext queryContext,

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848959813 ## pinot-core/src/main/java/org/apache/pinot/core/operator/filter/JsonMatchFilterOperator.java: ## @@ -87,4 +94,20 @@ public String toExplainString() { stringBui

[GitHub] [pinot] codecov-commenter commented on pull request #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
codecov-commenter commented on PR #8523: URL: https://github.com/apache/pinot/pull/8523#issuecomment-1097390331 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8523?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #8525: Set 'shutdownInProgress' in server config when server starts before the startup check

2022-04-12 Thread GitBox
Jackie-Jiang opened a new pull request, #8525: URL: https://github.com/apache/pinot/pull/8525 Fix #8519 For new joined server, or server not shut down normal previously, the `shutdownInProgress` won't be set properly when it starts. This can cause brokers routing queries to it befor

[GitHub] [pinot] codecov-commenter commented on pull request #8522: Make IngestionConfigUtils.getConfigMapWithPrefix method auto append trailing dot if missing

2022-04-12 Thread GitBox
codecov-commenter commented on PR #8522: URL: https://github.com/apache/pinot/pull/8522#issuecomment-1097346321 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8522?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] pjfanning opened a new pull request, #8524: Upgrade helix jackson

2022-04-12 Thread GitBox
pjfanning opened a new pull request, #8524: URL: https://github.com/apache/pinot/pull/8524 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat` 6.

[GitHub] [pinot] walterddr commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
walterddr commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848930788 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

[GitHub] [pinot] walterddr commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
walterddr commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848930488 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

[GitHub] [pinot] codecov-commenter commented on pull request #8520: Bump hadoop-common from 2.10.1 to 3.2.3 in /pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark

2022-04-12 Thread GitBox
codecov-commenter commented on PR #8520: URL: https://github.com/apache/pinot/pull/8520#issuecomment-1097308002 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8520?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] richardstartin commented on a diff in pull request #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8523: URL: https://github.com/apache/pinot/pull/8523#discussion_r848929522 ## pinot-core/src/main/java/org/apache/pinot/core/operator/BaseOperator.java: ## @@ -35,7 +35,7 @@ public final T nextBlock() { if (Thread.interrupted()) {

[GitHub] [pinot] richardstartin commented on a diff in pull request #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8523: URL: https://github.com/apache/pinot/pull/8523#discussion_r848928130 ## pinot-core/src/main/java/org/apache/pinot/core/plan/CombinePlanNode.java: ## @@ -73,87 +76,31 @@ public CombinePlanNode(List planNodes, QueryContext queryContex

[GitHub] [pinot] richardstartin commented on a diff in pull request #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8523: URL: https://github.com/apache/pinot/pull/8523#discussion_r848927822 ## pinot-core/src/main/java/org/apache/pinot/core/plan/CombinePlanNode.java: ## @@ -73,87 +76,31 @@ public CombinePlanNode(List planNodes, QueryContext queryContex

[GitHub] [pinot] walterddr commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
walterddr commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848926826 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

[GitHub] [pinot] walterddr commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
walterddr commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848926826 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #8523: Add tracing for operator planning

2022-04-12 Thread GitBox
Jackie-Jiang opened a new pull request, #8523: URL: https://github.com/apache/pinot/pull/8523 Currently operator planning time is not traced (even though the thread is registered within the tracing). This PR adds the tracing for the `CombinePlanNode`. It does not add tracing for individu

[GitHub] [pinot] richardstartin commented on a diff in pull request #8518: [WIP] Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8518: URL: https://github.com/apache/pinot/pull/8518#discussion_r848924792 ## pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/FilterKind.java: ## @@ -48,4 +48,13 @@ public boolean isRange() { return this == GREATER_TH

[GitHub] [pinot] richardstartin commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848921246 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/nodes/SerDeUtils.java: ## @@ -0,0 +1,90 @@ +/** + * Licensed to the Apache Software Foundation (

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848919778 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

[GitHub] [pinot] mapshen commented on issue #7849: Inconsistent Row Counts from Upsert Tables

2022-04-12 Thread GitBox
mapshen commented on issue #7849: URL: https://github.com/apache/pinot/issues/7849#issuecomment-1097289988 > @tuor713 I see the routing issue now. On a single server, it can have up to one consuming segment per partition; but on multiple servers serving the same segment, they might not fini

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848917350 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/LuceneFSTIndexReader.java: ## @@ -61,7 +63,7 @@ public MutableRoaringBitma

[GitHub] [pinot] xiangfu0 opened a new pull request, #8522: Make IngestionConfigUtils.getConfigMapWithPrefix method auto append trailing dot if missing

2022-04-12 Thread GitBox
xiangfu0 opened a new pull request, #8522: URL: https://github.com/apache/pinot/pull/8522 Improve the usability of IngestionConfigUtils.getConfigMapWithPrefix method to auto append trailing dot when missing. -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848911057 ## pinot-core/src/main/java/org/apache/pinot/core/operator/filter/JsonMatchFilterOperator.java: ## @@ -87,4 +94,20 @@ public String toExplainString() { stringB

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848910457 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java: ## @@ -291,6 +291,7 @@ private RoaringBitmap getMa

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848910329 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java: ## @@ -150,6 +155,14 @@ public MutableRoa

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848908825 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/LuceneFSTIndexReader.java: ## @@ -61,7 +63,7 @@ public MutableRoaringBitma

[pinot] branch dependabot/maven/org.apache.hadoop-hadoop-common-3.2.3 created (now f72d7d9fd4)

2022-04-12 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.hadoop-hadoop-common-3.2.3 in repository https://gitbox.apache.org/repos/asf/pinot.git at f72d7d9fd4 Bump hadoop-common from 2.10.1 to 3.2.3 No new revisio

[GitHub] [pinot] dependabot[bot] opened a new pull request, #8521: Bump hadoop-common from 2.10.1 to 3.2.3

2022-04-12 Thread GitBox
dependabot[bot] opened a new pull request, #8521: URL: https://github.com/apache/pinot/pull/8521 Bumps hadoop-common from 2.10.1 to 3.2.3. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.hadoop:hadoop

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848907859 ## pinot-core/src/main/java/org/apache/pinot/core/operator/filter/JsonMatchFilterOperator.java: ## @@ -87,4 +94,20 @@ public String toExplainString() { stringB

[pinot] branch dependabot/maven/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/org.apache.hadoop-hadoop-common-3.2.3 created (now 55e5dbd1e5)

2022-04-12 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/org.apache.hadoop-hadoop-common-3.2.3 in repository https://gitbox.apache.org/repos/asf/pinot.git at 55e5

[GitHub] [pinot] dependabot[bot] opened a new pull request, #8520: Bump hadoop-common from 2.10.1 to 3.2.3 in /pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark

2022-04-12 Thread GitBox
dependabot[bot] opened a new pull request, #8520: URL: https://github.com/apache/pinot/pull/8520 Bumps hadoop-common from 2.10.1 to 3.2.3. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.hadoop:hadoop

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848906719 ## pinot-common/src/main/proto/worker.proto: ## @@ -59,7 +61,7 @@ message QueryResponse { message StagePlan { int32 stageId = 1; Review Comment: This can b

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848902515 ## pinot-core/src/main/java/org/apache/pinot/core/operator/filter/JsonMatchFilterOperator.java: ## @@ -87,4 +94,20 @@ public String toExplainString() { stringBui

[GitHub] [pinot] codecov-commenter commented on pull request #8518: [WIP] Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
codecov-commenter commented on PR #8518: URL: https://github.com/apache/pinot/pull/8518#issuecomment-1097265336 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848902465 ## pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java: ## @@ -415,10 +421,12 @@ private ForwardIndexReaderContext getReaderContext() { }

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848901562 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848900486 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848900155 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848899692 ## pinot-common/src/main/proto/plan.proto: ## @@ -0,0 +1,74 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848898781 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java: ## @@ -291,6 +297,19 @@ private RoaringBitmap getM

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #8484: Add customizable parser module

2022-04-12 Thread GitBox
xiangfu0 commented on code in PR #8484: URL: https://github.com/apache/pinot/pull/8484#discussion_r848760309 ## contrib/pinot-fmpp-maven-plugin/src/main/java/org/apache/pinot/fmpp/FMPPMojo.java: ## @@ -0,0 +1,270 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on code in PR #8479: URL: https://github.com/apache/pinot/pull/8479#discussion_r848898024 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/nodes/SerDeUtils.java: ## @@ -0,0 +1,90 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] [pinot] chenboat opened a new issue, #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

2022-04-12 Thread GitBox
chenboat opened a new issue, #8519: URL: https://github.com/apache/pinot/issues/8519 Today if an empty box Pinot server joins the cluster to replace an existing server (i.e., the two servers share the same id), it can take 10s of minutes for Helix to instruct the new server to download all

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848893952 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java: ## @@ -291,6 +297,19 @@ private RoaringBitmap getM

[GitHub] [pinot] siddharthteotia commented on pull request #8479: Use proto for query plan serialization

2022-04-12 Thread GitBox
siddharthteotia commented on PR #8479: URL: https://github.com/apache/pinot/pull/8479#issuecomment-1097240111 Can we check if this is something can be used for plan serialization ? Probably offers more support for generic relational algebra support in serialization https://substrait.

[GitHub] [pinot] vvivekiyer opened a new pull request, #8518: Support filtering on bool/scalar fields without evaluator

2022-04-12 Thread GitBox
vvivekiyer opened a new pull request, #8518: URL: https://github.com/apache/pinot/pull/8518 Fixes the bugs in #8444 and #8487. Added unit tests. Testing is in-progress. Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature`

[GitHub] [pinot] walterddr commented on issue #8072: Support instant deletion

2022-04-12 Thread GitBox
walterddr commented on issue #8072: URL: https://github.com/apache/pinot/issues/8072#issuecomment-1097233026 this should be closed by #8122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [pinot] walterddr closed issue #8072: Support instant deletion

2022-04-12 Thread GitBox
walterddr closed issue #8072: Support instant deletion URL: https://github.com/apache/pinot/issues/8072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848878731 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java: ## @@ -291,6 +297,19 @@ private RoaringBitmap getMat

[GitHub] [pinot] zhtaoxiang commented on issue #8517: Improve GcsPinotFs to Read Credentials Directly from User Configurations

2022-04-12 Thread GitBox
zhtaoxiang commented on issue #8517: URL: https://github.com/apache/pinot/issues/8517#issuecomment-1097223654 I will pick up and fix this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [pinot] zhtaoxiang opened a new issue, #8517: Improve GcsPinotFs to Read Credentials Directly from User Configurations

2022-04-12 Thread GitBox
zhtaoxiang opened a new issue, #8517: URL: https://github.com/apache/pinot/issues/8517 Now, the [GcsPinotFs](https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-gcs/src/main/java/org/apache/pinot/plugin/filesystem/GcsPinotFS.java) can only [read credentials fro

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848871661 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/LuceneFSTIndexReader.java: ## @@ -61,19 +65,27 @@ public MutableRoaringBit

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848870940 ## pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java: ## @@ -131,6 +131,11 @@ public DataTable processQuery(ServerQueryRe

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
Jackie-Jiang commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848865538 ## pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java: ## @@ -415,10 +421,12 @@ private ForwardIndexReaderContext getReaderContext() { }

[GitHub] [pinot] zhtaoxiang commented on issue #8516: GcsPinotFs cannot list files in the root directory of buckets

2022-04-12 Thread GitBox
zhtaoxiang commented on issue #8516: URL: https://github.com/apache/pinot/issues/8516#issuecomment-1097209548 I will pick up and fix this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848861991 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/LuceneFSTIndexReader.java: ## @@ -61,19 +65,27 @@ public MutableRoaringBit

[GitHub] [pinot] zhtaoxiang opened a new issue, #8516: GcsPinotFs cannot list files in the root directory of buckets

2022-04-12 Thread GitBox
zhtaoxiang opened a new issue, #8516: URL: https://github.com/apache/pinot/issues/8516 The [GcsPinotFs](https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-gcs/src/main/java/org/apache/pinot/plugin/filesystem/GcsPinotFS.java) cannot list files in the root direc

[GitHub] [pinot] Jackie-Jiang commented on issue #8515: Pinot minion doesn't have a healthcheck endpoint.

2022-04-12 Thread GitBox
Jackie-Jiang commented on issue #8515: URL: https://github.com/apache/pinot/issues/8515#issuecomment-1097208226 @lfernandez93 Good catch! Do you want to help contribute this? We can add a `PinotMinionHealthCheck` similar to `PinotBrokerHealthCheck` and add `LifecycleServiceStatusCallback` t

[GitHub] [pinot] richardstartin commented on a diff in pull request #8485: Manual tracing

2022-04-12 Thread GitBox
richardstartin commented on code in PR #8485: URL: https://github.com/apache/pinot/pull/8485#discussion_r848860429 ## pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java: ## @@ -415,10 +421,12 @@ private ForwardIndexReaderContext getReaderContext() { }

  1   2   >