[GitHub] [pinot] abhioncbr commented on a diff in pull request #11052: [multistage] support physical plan in Explain queries

2023-07-19 Thread via GitHub
abhioncbr commented on code in PR #11052: URL: https://github.com/apache/pinot/pull/11052#discussion_r1268875829 ## pinot-query-planner/src/test/java/org/apache/pinot/query/QueryEnvironmentTestBase.java: ## @@ -202,4 +205,10 @@ public String toString() { } } } + +

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11110: Add stat to track number of segments that have valid doc id snapshots

2023-07-19 Thread via GitHub
Jackie-Jiang commented on code in PR #0: URL: https://github.com/apache/pinot/pull/0#discussion_r1268831445 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -540,6 +540,8 @@ protected void doTakeSna

[GitHub] [pinot] Jackie-Jiang commented on pull request #10815: Periodically delete Tmp Segment file brought by the SplitCommit End phase

2023-07-19 Thread via GitHub
Jackie-Jiang commented on PR #10815: URL: https://github.com/apache/pinot/pull/10815#issuecomment-1642964875 What I meant is to always enable this periodic task and remove the sync cleanup logic. Adding a dedicated periodic task per specific job is more modular, but we also need to consi

[pinot] branch master updated: Allow each test to specify the query engine (#11138)

2023-07-19 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 961695798b Allow each test to specify the query

[GitHub] [pinot] xiangfu0 merged pull request #11138: Allow each integration test to specify use v1 or v2 or both query engines

2023-07-19 Thread via GitHub
xiangfu0 merged PR #11138: URL: https://github.com/apache/pinot/pull/11138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

[GitHub] [pinot] egalpin opened a new issue, #11139: CAST + alias + IS_NULL/IS_NOT_NULL filter expression does not mix

2023-07-19 Thread via GitHub
egalpin opened a new issue, #11139: URL: https://github.com/apache/pinot/issues/11139 ```sql SELECT CAST(myCol AS LONG) AS myCol WHERE myCol IS NOT NULL ``` The above will fail with ``` Caused by: java.lang.UnsupportedOperationException: Unsupported predicate type: IS

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11136: WIP: Returning 403 status code in case of authorization failures

2023-07-19 Thread via GitHub
soumitra-st commented on code in PR #11136: URL: https://github.com/apache/pinot/pull/11136#discussion_r1268613693 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java: ## @@ -121,6 +121,8 @@ public void processSqlQueryGet(@ApiParam(value =

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11136: WIP: Returning 403 status code in case of authorization failures

2023-07-19 Thread via GitHub
soumitra-st commented on code in PR #11136: URL: https://github.com/apache/pinot/pull/11136#discussion_r1268608762 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java: ## @@ -390,7 +396,9 @@ public String sendPostRaw(String urlStr,

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11136: WIP: Returning 403 status code in case of authorization failures

2023-07-19 Thread via GitHub
soumitra-st commented on code in PR #11136: URL: https://github.com/apache/pinot/pull/11136#discussion_r1268606733 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -385,8 +387,7 @@ private BrokerResponseNative handleReques

[GitHub] [pinot] codecov-commenter commented on pull request #11138: Allow each integration test to specify use v1 or v2 or both query engines

2023-07-19 Thread via GitHub
codecov-commenter commented on PR #11138: URL: https://github.com/apache/pinot/pull/11138#issuecomment-1642614280 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11138?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] summerhasama-stripe commented on a diff in pull request #11073: For table rebalance, check if instances are equal for NO_OP

2023-07-19 Thread via GitHub
summerhasama-stripe commented on code in PR #11073: URL: https://github.com/apache/pinot/pull/11073#discussion_r1268165609 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -569,7 +585,7 @@ private InstancePartitions

[pinot] branch master updated: Fix reload bug (#11131)

2023-07-19 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 9cb876422c Fix reload bug (#11131) 9cb876422c is de

[GitHub] [pinot] jackjlli merged pull request #11131: Fix bug in reload segment when one or more indexes are removed.

2023-07-19 Thread via GitHub
jackjlli merged PR #11131: URL: https://github.com/apache/pinot/pull/11131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

[GitHub] [pinot] wirybeaver commented on pull request #10758: Add a new controller endpoint for segment deletion with a time window

2023-07-19 Thread via GitHub
wirybeaver commented on PR #10758: URL: https://github.com/apache/pinot/pull/10758#issuecomment-1642565646 @Jackie-Jiang Do you have an suggestion on which section shall I update? This one https://docs.pinot.apache.org/users/api/controller-api-reference ? -- This is an automated me

[GitHub] [pinot] xiangfu0 opened a new pull request, #11138: Allow each test to specify the query engine

2023-07-19 Thread via GitHub
xiangfu0 opened a new pull request, #11138: URL: https://github.com/apache/pinot/pull/11138 Allow each test to specify the query engine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [pinot] codecov-commenter commented on pull request #11137: Separate BitmapBasedFilterOperator into two filter operators.

2023-07-19 Thread via GitHub
codecov-commenter commented on PR #11137: URL: https://github.com/apache/pinot/pull/11137#issuecomment-1642540581 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11137?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] walterddr closed issue #11003: pinot-client unable to dynamically choose broker for multi-table query

2023-07-19 Thread via GitHub
walterddr closed issue #11003: pinot-client unable to dynamically choose broker for multi-table query URL: https://github.com/apache/pinot/issues/11003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [pinot] walterddr commented on issue #11003: pinot-client unable to dynamically choose broker for multi-table query

2023-07-19 Thread via GitHub
walterddr commented on issue #11003: URL: https://github.com/apache/pinot/issues/11003#issuecomment-1642540085 this is fixed by #11069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #11131: Fix bug in reload segment when one or more indexes are removed.

2023-07-19 Thread via GitHub
vvivekiyer commented on code in PR #11131: URL: https://github.com/apache/pinot/pull/11131#discussion_r1268451707 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java: ## @@ -157,9 +157,12 @@ void generat

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #11131: Fix bug in reload segment when one or more indexes are removed.

2023-07-19 Thread via GitHub
vvivekiyer commented on code in PR #11131: URL: https://github.com/apache/pinot/pull/11131#discussion_r1268451707 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java: ## @@ -157,9 +157,12 @@ void generat

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11136: WIP: Returning 403 status code in case of authorization failures

2023-07-19 Thread via GitHub
Jackie-Jiang commented on code in PR #11136: URL: https://github.com/apache/pinot/pull/11136#discussion_r1268436162 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java: ## @@ -390,7 +396,9 @@ public String sendPostRaw(String urlStr

[GitHub] [pinot] xiangfu0 closed pull request #11106: Upgrade grpc version in pinot-pulsar module

2023-07-19 Thread via GitHub
xiangfu0 closed pull request #11106: Upgrade grpc version in pinot-pulsar module URL: https://github.com/apache/pinot/pull/11106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [pinot] shenyu0127 opened a new pull request, #11137: Separate BitmapBasedFilterOperator into two filter operators.

2023-07-19 Thread via GitHub
shenyu0127 opened a new pull request, #11137: URL: https://github.com/apache/pinot/pull/11137 This is a refactoring PR that separate `BitmapBasedFilterOperator` into `BitmapBasedFilterOperator` and `InvertedIndexFilterOperator`. -- This is an automated message from the Apache Git Service

[GitHub] [pinot] codecov-commenter commented on pull request #11136: WIP: Returning 403 status code in case of authorization failures

2023-07-19 Thread via GitHub
codecov-commenter commented on PR #11136: URL: https://github.com/apache/pinot/pull/11136#issuecomment-1642461675 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11136?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] soumitra-st opened a new pull request, #11136: WIP: Returning 403 status code in case of authorization failures

2023-07-19 Thread via GitHub
soumitra-st opened a new pull request, #11136: URL: https://github.com/apache/pinot/pull/11136 Currently, if SQL query execution has authorization issues, the return code is 200 and the errorCode 180 is set in the response JSON. This happens for both controller and broker endpoints.

[pinot] branch master updated (9824f603f3 -> cfc3abcf71)

2023-07-19 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 9824f603f3 [multistage] add hint option for table node (#4) add cfc3abcf71 [multistage][agg] support agg with l

[GitHub] [pinot] walterddr merged pull request #11105: [multistage][agg] support agg with literal arguments

2023-07-19 Thread via GitHub
walterddr merged PR #11105: URL: https://github.com/apache/pinot/pull/11105 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

[GitHub] [pinot] walterddr opened a new issue, #11135: [flakytest] timeout in fork most likely on RealtimeKinesisIntegrationTest

2023-07-19 Thread via GitHub
walterddr opened a new issue, #11135: URL: https://github.com/apache/pinot/issues/11135 see https://github.com/apache/pinot/actions/runs/5595643067/jobs/10231738127?pr=11132 https://github.com/apache/pinot/actions/runs/5595738571/jobs/10231935149?pr=11133 https://github.com/apa

[GitHub] [pinot] robertzych commented on pull request #10463: Segment compaction for upsert real-time tables

2023-07-19 Thread via GitHub
robertzych commented on PR #10463: URL: https://github.com/apache/pinot/pull/10463#issuecomment-1642355508 @Jackie-Jiang Re: [your question about segments that end in the future](https://github.com/apache/pinot/pull/10463#discussion_r1267601214), I added this condition to support the integr

[GitHub] [pinot] walterddr commented on issue #11072: [multistage] QueryRunnerTest tear down takes 10sec (default timeout)

2023-07-19 Thread via GitHub
walterddr commented on issue #11072: URL: https://github.com/apache/pinot/issues/11072#issuecomment-1642337396 yes we should do a framegraph dump and see what's the time spent during the tearDown() process (e.g. is it on server executor or mailbox service or the scheduler async wait. once w

[GitHub] [pinot] somandal commented on a diff in pull request #11131: Fix bug in reload segment when one or more indexes are removed.

2023-07-19 Thread via GitHub
somandal commented on code in PR #11131: URL: https://github.com/apache/pinot/pull/11131#discussion_r1268273117 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java: ## @@ -157,9 +157,12 @@ void generateI

[pinot] branch master updated: [multistage] add hint option for table node (#11114)

2023-07-19 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 9824f603f3 [multistage] add hint option for table

[GitHub] [pinot] walterddr merged pull request #11114: [multistage] add option for table hints

2023-07-19 Thread via GitHub
walterddr merged PR #4: URL: https://github.com/apache/pinot/pull/4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

[GitHub] [pinot] walterddr commented on a diff in pull request #11052: [multistage] support physical plan in Explain queries

2023-07-19 Thread via GitHub
walterddr commented on code in PR #11052: URL: https://github.com/apache/pinot/pull/11052#discussion_r1268239486 ## pinot-query-planner/src/test/java/org/apache/pinot/query/QueryEnvironmentTestBase.java: ## @@ -202,4 +205,10 @@ public String toString() { } } } + +

[GitHub] [pinot] walterddr commented on a diff in pull request #11052: [multistage] support physical plan in Explain queries

2023-07-19 Thread via GitHub
walterddr commented on code in PR #11052: URL: https://github.com/apache/pinot/pull/11052#discussion_r1268199694 ## pinot-query-planner/src/test/java/org/apache/pinot/query/QueryEnvironmentTestBase.java: ## @@ -202,4 +205,10 @@ public String toString() { } } } + +

[GitHub] [pinot] summerhasama-stripe commented on a diff in pull request #11073: For table rebalance, check if instances are equal for NO_OP

2023-07-19 Thread via GitHub
summerhasama-stripe commented on code in PR #11073: URL: https://github.com/apache/pinot/pull/11073#discussion_r1268165609 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -569,7 +585,7 @@ private InstancePartitions

[GitHub] [pinot] summerhasama-stripe commented on a diff in pull request #11073: For table rebalance, check if instances are equal for NO_OP

2023-07-19 Thread via GitHub
summerhasama-stripe commented on code in PR #11073: URL: https://github.com/apache/pinot/pull/11073#discussion_r1268165609 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -569,7 +585,7 @@ private InstancePartitions

[GitHub] [pinot] walterddr commented on a diff in pull request #11105: [multistage][agg] support agg with literal arguments

2023-07-19 Thread via GitHub
walterddr commented on code in PR #11105: URL: https://github.com/apache/pinot/pull/11105#discussion_r1268129739 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/LiteralHintUtils.java: ## @@ -0,0 +1,109 @@ +/** + * Licensed to the Apache Software Found

[GitHub] [pinot] Aravind-Suresh commented on issue #11072: [multistage] QueryRunnerTest tear down takes 10sec (default timeout)

2023-07-19 Thread via GitHub
Aravind-Suresh commented on issue #11072: URL: https://github.com/apache/pinot/issues/11072#issuecomment-1642107199 I tried running this test locally and added a few debugger breakpoints here and there - looks like the cleanup / shutdown is taking a long time. ``` at org.ap

[GitHub] [pinot] abhioncbr commented on a diff in pull request #11052: [multistage] support physical plan in Explain queries

2023-07-19 Thread via GitHub
abhioncbr commented on code in PR #11052: URL: https://github.com/apache/pinot/pull/11052#discussion_r1268052970 ## pinot-query-planner/src/test/java/org/apache/pinot/query/QueryEnvironmentTestBase.java: ## @@ -202,4 +205,10 @@ public String toString() { } } } + +

[GitHub] [pinot] codecov-commenter commented on pull request #11134: Bump azure-storage-file-datalake from 12.14.1 to 12.16.0

2023-07-19 Thread via GitHub
codecov-commenter commented on PR #11134: URL: https://github.com/apache/pinot/pull/11134#issuecomment-1641931238 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11134?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[pinot] branch dependabot/maven/com.azure-azure-storage-file-datalake-12.16.0 created (now daed8ee9c6)

2023-07-19 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.azure-azure-storage-file-datalake-12.16.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at daed8ee9c6 Bump azure-storage-file-datalake from 12.14.1 t

[GitHub] [pinot] dependabot[bot] opened a new pull request, #11134: Bump azure-storage-file-datalake from 12.14.1 to 12.16.0

2023-07-19 Thread via GitHub
dependabot[bot] opened a new pull request, #11134: URL: https://github.com/apache/pinot/pull/11134 Bumps [azure-storage-file-datalake](https://github.com/Azure/azure-sdk-for-java) from 12.14.1 to 12.16.0. Release notes Sourced from https://github.com/Azure/azure-sdk-for-java/relea

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11105: [multistage][agg] support agg with literal arguments

2023-07-19 Thread via GitHub
xiangfu0 commented on code in PR #11105: URL: https://github.com/apache/pinot/pull/11105#discussion_r1267771638 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/LiteralHintUtils.java: ## @@ -0,0 +1,109 @@ +/** + * Licensed to the Apache Software Founda

[GitHub] [pinot] Jackie-Jiang commented on pull request #11109: Use PUT request to enable/disable table/instance

2023-07-19 Thread via GitHub
Jackie-Jiang commented on PR #11109: URL: https://github.com/apache/pinot/pull/11109#issuecomment-1641595767 @jayeshchoudhary Yes, let's always use PUT to enable/disable table/instance in the UI -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [pinot] Jackie-Jiang commented on issue #7993: Support for `any_value` aggregation function

2023-07-19 Thread via GitHub
Jackie-Jiang commented on issue #7993: URL: https://github.com/apache/pinot/issues/7993#issuecomment-1641588824 Semantic: https://docs.snowflake.com/en/sql-reference/functions/any_value -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [pinot] Jackie-Jiang closed issue #9103: Support ANY_VALUE aggregation function

2023-07-19 Thread via GitHub
Jackie-Jiang closed issue #9103: Support ANY_VALUE aggregation function URL: https://github.com/apache/pinot/issues/9103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-07-19 Thread via GitHub
Jackie-Jiang commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1267660646 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -274,6 +309,13 @@ protected void addSegm

[GitHub] [pinot] Jackie-Jiang commented on issue #10960: Large off-heap memory usage?

2023-07-19 Thread via GitHub
Jackie-Jiang commented on issue #10960: URL: https://github.com/apache/pinot/issues/10960#issuecomment-1641553188 @geekboood My suggestion would be setting a smaller heap size. If the pod has 32GB memory limit, then 12GB heap might be a good starting point. It will use up to 12GB heap + 12G

[GitHub] [pinot] snleee commented on issue #11025: Unable to Run Ingestion Job when using ADLS Gen2 as a source and destination

2023-07-19 Thread via GitHub
snleee commented on issue #11025: URL: https://github.com/apache/pinot/issues/11025#issuecomment-1641553136 @Akash-Nair We will need a bit more information (full logs and your ingestion spec configuration). Can you join to Pinot community slack if possible? -- This is an automated messa

[pinot] branch dependabot/maven/pinot-plugins/pinot-stream-ingestion/pinot-pulsar/com.squareup.okio-okio-3.4.0 deleted (was 7c4ba9aecc)

2023-07-19 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/pinot-plugins/pinot-stream-ingestion/pinot-pulsar/com.squareup.okio-okio-3.4.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 7c4ba9aecc Bump okio in /pin

[GitHub] [pinot] dependabot[bot] commented on pull request #11100: Bump okio from 1.6.0 to 3.4.0 in /pinot-plugins/pinot-stream-ingestion/pinot-pulsar

2023-07-19 Thread via GitHub
dependabot[bot] commented on PR #11100: URL: https://github.com/apache/pinot/pull/11100#issuecomment-1641546685 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

[GitHub] [pinot] Jackie-Jiang closed pull request #11100: Bump okio from 1.6.0 to 3.4.0 in /pinot-plugins/pinot-stream-ingestion/pinot-pulsar

2023-07-19 Thread via GitHub
Jackie-Jiang closed pull request #11100: Bump okio from 1.6.0 to 3.4.0 in /pinot-plugins/pinot-stream-ingestion/pinot-pulsar URL: https://github.com/apache/pinot/pull/11100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [pinot] Jackie-Jiang commented on pull request #10758: Add a new controller endpoint for segment deletion with a time window

2023-07-19 Thread via GitHub
Jackie-Jiang commented on PR #10758: URL: https://github.com/apache/pinot/pull/10758#issuecomment-1641546168 Thanks for the contribution! Can you help also update the pinot documentation for this new API? -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10463: Segment compaction for upsert real-time tables

2023-07-19 Thread via GitHub
Jackie-Jiang commented on code in PR #10463: URL: https://github.com/apache/pinot/pull/10463#discussion_r1267594751 ## pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskGenerator.java: