[pinot] branch master updated (876e58dcc5 -> 1b7cb166de)

2023-10-11 Thread saurabhd336
This is an automated email from the ASF dual-hosted git repository. saurabhd336 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 876e58dcc5 [multistage] framework to back-propagate metadata across opChains (#11746) add 1b7cb166de Add sup

Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

2023-10-11 Thread via GitHub
saurabhd336 merged PR #11729: URL: https://github.com/apache/pinot/pull/11729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.

Re: [PR] [draft][multistage] add exchange removal rule [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11794: URL: https://github.com/apache/pinot/pull/11794#issuecomment-1759013638 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11794?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-11 Thread via GitHub
itschrispeck commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1356140349 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int g

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-11 Thread via GitHub
itschrispeck commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1356139329 ## pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractIndexTransformFunction.java: ## @@ -0,0 +1,256 @@ +/** + * Licensed to the

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-11 Thread via GitHub
xiangfu0 commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1758982444 > @xiangfu0 Let me know if I can pick this up. cc: @mayankshriv Thanks Anand for your interest! Since Dao took this up, I will find more stuffs for you :p -- This is an aut

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
CA1087 commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758981718 What should be the correct value? I have the env variable named PINOT_AUTH set from the secret in kubernetes. -- This is an automated message from the Apache Git Service. To respond

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-11 Thread via GitHub
xiangfu0 commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1758981228 > Hi @xiangfu0, I'm coming from kafka world and new to pinot , hence, I would like to take on this task to understand more about pinot internal. Would that be ok for the team?

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758979549 > @KKcorps - One more thing that i notice that in our config we are using something like this `pinot.server.segment.uploader.auth.token=${env:PINOT_AUTH}` and PINOT_AUTH was not ge

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
CA1087 commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758974542 @KKcorps - One more thing that i notice that in our config we are using something like this `pinot.server.segment.uploader.auth.token=${env:PINOT_AUTH}` and PINOT_AUTH was not getti

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-11 Thread via GitHub
anandheritage commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1758960276 @xiangfu0 Let me know if I can pick this up. cc: @mayankshriv -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[PR] [draft] add exchange removal rule [pinot]

2023-10-11 Thread via GitHub
walterddr opened a new pull request, #11794: URL: https://github.com/apache/pinot/pull/11794 no need for exchange/agg/exchange sequence if agg is a leaf --> this could happen when shuffling rules move around exchanges and deem it unnecessary. -- This is an automated message from the Apac

Re: [PR] Add pending segments to reload job status [pinot]

2023-10-11 Thread via GitHub
saurabhd336 commented on PR #11785: URL: https://github.com/apache/pinot/pull/11785#issuecomment-1758950479 @walterddr 1) With replication > 1, the response may include the same segment name multiple times in the list. Do you think that should be okay? 2) Feel like in the very beginin

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758926757 > yes @KKcorps I verified the same and it was right correctly in the server config. The server is able to consume message for sometime and it stops after a point. Yes that I

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
CA1087 commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758924479 yes @KKcorps I verified the same and it was right correctly in the server config. The server is able to consume message for sometime and it stops after a point. -- This is an aut

[pinot] branch master updated: [multistage] framework to back-propagate metadata across opChains (#11746)

2023-10-11 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 876e58dcc5 [multistage] framework to back-propagat

Re: [PR] [multistage] framework to back-propagate metadata across opChains [pinot]

2023-10-11 Thread via GitHub
walterddr merged PR #11746: URL: https://github.com/apache/pinot/pull/11746 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758919766 > @KKcorps - All the calls are getting failed. Thanks, that means that there is auth token mismatch. Between server and controller. I tried reproducing it in local with 0.12.1

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
CA1087 commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758917488 @KKcorps - All the calls are getting failed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

2023-10-11 Thread via GitHub
saurabhd336 commented on code in PR #11729: URL: https://github.com/apache/pinot/pull/11729#discussion_r1356055209 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java: ## @@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId,

Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

2023-10-11 Thread via GitHub
saurabhd336 commented on code in PR #11729: URL: https://github.com/apache/pinot/pull/11729#discussion_r1356044894 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java: ## @@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId,

Re: [PR] [UI] use indexes API instead of fetching all segment metadata [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11793: URL: https://github.com/apache/pinot/pull/11793#issuecomment-1758895219 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11793?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

2023-10-11 Thread via GitHub
saurabhd336 commented on code in PR #11729: URL: https://github.com/apache/pinot/pull/11729#discussion_r1356044470 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java: ## @@ -91,4 +92,41 @@ pub

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758890416 > @KKcorps - in my case ACL config was enabled since the first deployment and it was working for a while until it stops working and if i restart the components it works for sometime

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
CA1087 commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758883774 @KKcorps - in my case ACL config was enabled since the first deployment and it was working for a while until it stops working and if i restart the components it works for sometime til

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758880223 Also, has the ACL configs been enabled since the first deployment or were they enabled later on? If later on, were all the pinot components restarted or only controller or server?

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758876009 @rodolphogarrido does this happen on the first segment commit or do you see some segments getting committed correctly and then all of a sudden it starts failing? You can chec

Re: [PR] 11085: Added commons-configuration2 dependency. [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11792: URL: https://github.com/apache/pinot/pull/11792#issuecomment-1758858563 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11792?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] [UI] use indexes API instead of fetching all segment metadata [pinot]

2023-10-11 Thread via GitHub
walterddr opened a new pull request, #11793: URL: https://github.com/apache/pinot/pull/11793 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[PR] 11085: Added commons-configuration2 dependency. [pinot]

2023-10-11 Thread via GitHub
abhioncbr opened a new pull request, #11792: URL: https://github.com/apache/pinot/pull/11792 As per the [issue](https://github.com/apache/pinot/issues/11085). - This is related to the first task i.e. adding the `commons-configuration2` dependency in the root pom file. - Also added an

Re: [PR] Add compression configuration for functionColumnPairs to StartreeIndexConfigs [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11744: URL: https://github.com/apache/pinot/pull/11744#discussion_r1355966332 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/StarTreeIndexConfig.java: ## @@ -42,15 +45,17 @@ public StarTreeIndexConfig( @JsonProperty(valu

Re: [I] Create DATETIMECONVERTWINDOWHOP function [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on issue #11775: URL: https://github.com/apache/pinot/issues/11775#issuecomment-1758789100 Is this supported in other DBs? Or is this more like a workaround for window function? Trying to see if there is a standard SQL equivalence to this function to better underst

Re: [I] pinot-controller pod clash over and over again in kubernetes [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on issue #11764: URL: https://github.com/apache/pinot/issues/11764#issuecomment-1758782674 cc @xiangfu0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] Then query runner tool doesn't accept header as a parameter which make the tool unusable for cluster with access control. [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on issue #11772: URL: https://github.com/apache/pinot/issues/11772#issuecomment-1758782378 cc @xiangfu0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] [multistage] framework to back-propagate metadata across opChains [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11746: URL: https://github.com/apache/pinot/pull/11746#discussion_r1355940162 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/ReceivingMailbox.java: ## @@ -49,6 +50,8 @@ public class ReceivingMailbox { // TODO: Revisit

Re: [I] flaky test org.apache.pinot.controller.api.TableViewsTest.setUp [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on issue #11783: URL: https://github.com/apache/pinot/issues/11783#issuecomment-1758770326 Duplicate of #11760, fixed with #11770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] flaky test org.apache.pinot.controller.api.TableViewsTest.setUp [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang closed issue #11783: flaky test org.apache.pinot.controller.api.TableViewsTest.setUp URL: https://github.com/apache/pinot/issues/11783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Enhance Broker reducer to handle expression format change [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11762: URL: https://github.com/apache/pinot/pull/11762#discussion_r1355936238 ## pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java: ## @@ -99,8 +119,23 @@ public BrokerResponseNative reduceOnDataTable(Broke

Re: [PR] Enhance Broker reducer to handle expression format change [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang merged PR #11762: URL: https://github.com/apache/pinot/pull/11762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[pinot] branch master updated: Enhance Broker reducer to handle expression format change (#11762)

2023-10-11 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ad0d2b173a Enhance Broker reducer to handle expre

Re: [PR] Enhance Broker reducer to handle expression format change [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11762: URL: https://github.com/apache/pinot/pull/11762#discussion_r1355935410 ## pinot-core/src/main/java/org/apache/pinot/core/query/reduce/AggregationDataTableReducer.java: ## @@ -47,9 +45,12 @@ @SuppressWarnings({"rawtypes", "unchecked"}

Re: [PR] [multistage] framework to back-propagate metadata across opChains [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11746: URL: https://github.com/apache/pinot/pull/11746#discussion_r1355932764 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MailboxSendOperator.java: ## @@ -147,12 +150,12 @@ protected TransferableBlock getNext

Re: [PR] Support Upsert deletion for TTL: construct queryableDocIds when adding segments out of TTL [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11791: URL: https://github.com/apache/pinot/pull/11791#issuecomment-1758750086 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11791?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] move jobId out of rebalanceConfig as it's more like a runtime context [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11790: URL: https://github.com/apache/pinot/pull/11790#issuecomment-1758749256 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11790?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
mayankshriv commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758742120 @KKcorps assigning to you, please take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11729: URL: https://github.com/apache/pinot/pull/11729#discussion_r1355906002 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/ForwardIndexReader.java: ## @@ -916,4 +920,72 @@ default byte[][] getBytesMV(int docId

[PR] Support Upsert deletion for TTL: construct queryableDocIds when adding segments out of TTL [pinot]

2023-10-11 Thread via GitHub
deemoliu opened a new pull request, #11791: URL: https://github.com/apache/pinot/pull/11791 `bugfix`, `upsert`: construct queryableDocIds when adding segments out of TTL Previously we don't handle cases that a upsert table with both upsert deletion and upsert ttl. In this PR, fixe

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355895061 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceRetryConfig.java: ## @@ -0,0 +1,74 @@ +/** + * Licensed to the Apache S

[PR] move jobId out of rebalanceConfig as it's more like a runtime context [pinot]

2023-10-11 Thread via GitHub
klsince opened a new pull request, #11790: URL: https://github.com/apache/pinot/pull/11790 Move jobId out of rebalanceConfig as it's more like a runtime context. And as in the table rebalance restful API, jobId is not provided by caller but auto-gen'ed and returned to caller. -- This is

Re: [PR] Use expected version in api for table config read modify write change [pinot]

2023-10-11 Thread via GitHub
xiangfu0 commented on code in PR #11782: URL: https://github.com/apache/pinot/pull/11782#discussion_r1355893388 ## pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java: ## @@ -73,12 +74,33 @@ public static void setUserConfig(ZkHelixPropertyStore p

Re: [I] Add partition, validDocId as the derived column [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on issue #11580: URL: https://github.com/apache/pinot/issues/11580#issuecomment-1758706213 @snleee > 1. `$partitionValues` are the ones partition id of the segment. This information is already getting populated to segment metadata as `partitionValues` when the

Re: [PR] custom partial upsert row merger [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on PR #11584: URL: https://github.com/apache/pinot/pull/11584#issuecomment-1758692248 My suggestion is to not add a new interface, but modify the existing `PartialUpsertMerger` interface -- This is an automated message from the Apache Git Service. To respond to the

[pinot] branch master updated (884dbdbb6e -> 314cc03de9)

2023-10-11 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 884dbdbb6e Fix flaky TableViewsTest (#11770) add 314cc03de9 Adding support for generating data in JSON format. (#1

Re: [PR] Adding support for generating data in JSON format. [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang merged PR #11778: URL: https://github.com/apache/pinot/pull/11778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-11 Thread via GitHub
xiangfu0 commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1758663286 Looking into this, hypnosis is that server calls to controller may not carry the credential. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355861888 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -137,6 +139,23 @@ public static String createUniqueRebala

Re: [PR] Update Segment builder to use column major tables [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11776: URL: https://github.com/apache/pinot/pull/11776#discussion_r1355825121 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java: ## @@ -154,4 +179,12 @@ public static Schema

Re: [PR] [Do not review] add support for retention on deleted upsert keys [pinot]

2023-10-11 Thread via GitHub
deemoliu commented on code in PR #11779: URL: https://github.com/apache/pinot/pull/11779#discussion_r1355858602 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -129,20 +133,23 @@ public void addSegment(Imm

[I] Allow schema to be created with JSON config in UI [pinot]

2023-10-11 Thread via GitHub
ksnijjer opened a new issue, #11789: URL: https://github.com/apache/pinot/issues/11789 Currently if you need to add a schema OSS Pinot Console requires you to input each column manually one by one in the UI. In case user already has a JSON config it doesn't allow you to simply paste the con

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355848314 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -98,10 +117,9 @@ private void updateOnStart

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355856100 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -126,30 +143,73 @@ public int getNumUpdates

Re: [I] Retention support for upsert deleted key [pinot]

2023-10-11 Thread via GitHub
tibrewalpratik17 commented on issue #11736: URL: https://github.com/apache/pinot/issues/11736#issuecomment-1758644725 @Jackie-Jiang @deemoliu referring to #11779, we do not intend to leverage the watermark feature of upsertTTL to address this use case. We would require loading all segments'

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355848314 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java: ## @@ -98,10 +117,9 @@ private void updateOnStart

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355843815 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceRetryConfig.java: ## @@ -0,0 +1,74 @@ +/** + * Licensed to the Apache S

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355843815 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceRetryConfig.java: ## @@ -0,0 +1,74 @@ +/** + * Licensed to the Apache S

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355840502 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceChecker.java: ## @@ -0,0 +1,279 @@ +/** + * Licensed to the Apache Software

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355832863 ## pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java: ## @@ -199,6 +199,9 @@ public static class ControllerPeriodicTasksConf { // Real

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11740: URL: https://github.com/apache/pinot/pull/11740#discussion_r1355830231 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceChecker.java: ## @@ -0,0 +1,279 @@ +/** + * Licensed to the Apache Software

Re: [I] Upgrade commons-configuration to commons-configuration2 [pinot]

2023-10-11 Thread via GitHub
abhioncbr commented on issue #11085: URL: https://github.com/apache/pinot/issues/11085#issuecomment-1758579927 I am creating the tasks to track the progress based on the above doc - [ ] Add `commons-configurations2` dependency in parent pom file - [ ] Upgrade PinotConfiguration to u

Re: [PR] Use expected version in api for table config read modify write change [pinot]

2023-10-11 Thread via GitHub
Jackie-Jiang commented on code in PR #11782: URL: https://github.com/apache/pinot/pull/11782#discussion_r1355807619 ## pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java: ## @@ -73,12 +74,33 @@ public static void setUserConfig(ZkHelixPropertySto

Re: [PR] reloadStatus should only count online segments [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11787: URL: https://github.com/apache/pinot/pull/11787#issuecomment-1758574711 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11787?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Capture build scans on ge.apache.org to benefit from deep build insights [pinot]

2023-10-11 Thread via GitHub
vvivekiyer commented on PR #11767: URL: https://github.com/apache/pinot/pull/11767#issuecomment-1758561341 Thanks @clayburn for creating this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Add support to read exact buffer byte ranges corresponding to a given forward index doc id [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11729: URL: https://github.com/apache/pinot/pull/11729#discussion_r1355756437 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java: ## @@ -91,4 +92,41 @@ public

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-11 Thread via GitHub
dttung2905 commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1758389808 Hi @xiangfu0, I'm coming from kafka world and new to pinot , hence, I would like to take on this task to understand more about pinot internal. Would that be ok for the team? --

Re: [I] [flaky test] Controller TableViewsTest [pinot]

2023-10-11 Thread via GitHub
xiangfu0 closed issue #11760: [flaky test] Controller TableViewsTest URL: https://github.com/apache/pinot/issues/11760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Fix flaky TableViewsTest [pinot]

2023-10-11 Thread via GitHub
xiangfu0 merged PR #11770: URL: https://github.com/apache/pinot/pull/11770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

[pinot] branch master updated: Fix flaky TableViewsTest (#11770)

2023-10-11 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 884dbdbb6e Fix flaky TableViewsTest (#11770) 884

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-11 Thread via GitHub
xiangfu0 commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1758324152 To implement this, we just need to add the corresponding function in TransformFunction and ScalarFunction, as well as register the function in TransformFunctionType. In `org.

Re: [I] Create an API for cloning a table [pinot]

2023-10-11 Thread via GitHub
deemoliu commented on issue #11781: URL: https://github.com/apache/pinot/issues/11781#issuecomment-1758224103 do you mean the cloned dataset and the new dataset will be sharing the same copy of segments in the deepstore? there might be a race condition if one of the table is deleting the

Re: [I] segmentReloadStatus API never completes [pinot]

2023-10-11 Thread via GitHub
jadami10 commented on issue #11603: URL: https://github.com/apache/pinot/issues/11603#issuecomment-1758216417 i've replicated this locally. it happens when you have several thousands segments and reload several times back to back, so it's not pervasive. But it does seem to be to clock skew.

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-11 Thread via GitHub
chenboat commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1355496520 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int getDo

[I] Add GENERATE_ARRAY function [pinot]

2023-10-11 Thread via GitHub
xiangfu0 opened a new issue, #11786: URL: https://github.com/apache/pinot/issues/11786 Pinot supports `ARRAY[x, y, z]` syntax to manually craft an array literal. This is good for small scale, and in real prod, we need to add an ARRAY function to generate an ARRAY sequence for a given range.

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-11 Thread via GitHub
chenboat commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1355488928 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int getDo

Re: [PR] track rebalance status when instance assignment already correct [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11768: URL: https://github.com/apache/pinot/pull/11768#discussion_r1355436444 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -241,13 +241,19 @@ public RebalanceResult rebalance(Table

Re: [PR] track rebalance status when instance assignment already correct [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11768: URL: https://github.com/apache/pinot/pull/11768#discussion_r1355431764 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -241,13 +241,19 @@ public RebalanceResult rebalance(Table

Re: [PR] track rebalance status when instance assignment already correct [pinot]

2023-10-11 Thread via GitHub
klsince commented on code in PR #11768: URL: https://github.com/apache/pinot/pull/11768#discussion_r1355431764 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java: ## @@ -241,13 +241,19 @@ public RebalanceResult rebalance(Table

Re: [PR] Add UI to track segment reload progress [pinot]

2023-10-11 Thread via GitHub
walterddr commented on code in PR #9521: URL: https://github.com/apache/pinot/pull/9521#discussion_r1355399536 ## pinot-controller/src/main/resources/app/pages/TenantDetails.tsx: ## @@ -319,19 +325,62 @@ const TenantPageDetails = ({ match }: RouteComponentProps) => { const

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-11 Thread via GitHub
chenboat commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1355372242 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int getDo

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-11 Thread via GitHub
chenboat commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1355363772 ## pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractIndexTransformFunction.java: ## @@ -0,0 +1,256 @@ +/** + * Licensed to the Apa

Re: [PR] Add pending segments to reload job status [pinot]

2023-10-11 Thread via GitHub
codecov-commenter commented on PR #11785: URL: https://github.com/apache/pinot/pull/11785#issuecomment-1758067822 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11785?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Enhance Broker reducer to handle expression format change [pinot]

2023-10-11 Thread via GitHub
walterddr commented on code in PR #11762: URL: https://github.com/apache/pinot/pull/11762#discussion_r1355312354 ## pinot-core/src/main/java/org/apache/pinot/core/query/reduce/AggregationDataTableReducer.java: ## @@ -47,9 +45,12 @@ @SuppressWarnings({"rawtypes", "unchecked"})

[PR] Add pending segments to reload job status [pinot]

2023-10-11 Thread via GitHub
walterddr opened a new pull request, #11785: URL: https://github.com/apache/pinot/pull/11785 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Issue-6420 Add support for Apache Datasketches CPC sketch [pinot]

2023-10-11 Thread via GitHub
davecromberge commented on code in PR #11774: URL: https://github.com/apache/pinot/pull/11774#discussion_r1355129147 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawCPCAggregationFunction.java: ## @@ -0,0 +1,53 @@ +/** + * Licensed to

Re: [PR] Issue-6420 Add support for Apache Datasketches CPC sketch [pinot]

2023-10-11 Thread via GitHub
davecromberge commented on code in PR #11774: URL: https://github.com/apache/pinot/pull/11774#discussion_r1355091896 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCAggregationFunction.java: ## @@ -0,0 +1,358 @@ +/** + * Licensed to t

Re: [PR] Issue-6420 Add support for Apache Datasketches CPC sketch [pinot]

2023-10-11 Thread via GitHub
davecromberge commented on code in PR #11774: URL: https://github.com/apache/pinot/pull/11774#discussion_r1355090385 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawCPCAggregationFunction.java: ## @@ -0,0 +1,53 @@ +/** + * Licensed to

Re: [PR] Issue-6420 Add support for Apache Datasketches CPC sketch [pinot]

2023-10-11 Thread via GitHub
davecromberge commented on code in PR #11774: URL: https://github.com/apache/pinot/pull/11774#discussion_r1355088458 ## pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java: ## @@ -515,23 +518,23 @@ public VarianceTuple deserialize(ByteBuffer byteBuffer) {

Re: [I] Upgrade inconsistency: Confusing behaviour with noDictionaryConfig and fieldConfig for enabling compression on raw forward index. [pinot]

2023-10-11 Thread via GitHub
gortiz commented on issue #11745: URL: https://github.com/apache/pinot/issues/11745#issuecomment-1757656853 Sorry for the delay. I was able to reproduce the issue. It should be fixed in #11784, which also adds some tests to verify the correct behavior -- This is an automated message from

[PR] Fix regression in ForwardIndexType detected in https://github.com/apache/pinot/issues/11745 [pinot]

2023-10-11 Thread via GitHub
gortiz opened a new pull request, #11784: URL: https://github.com/apache/pinot/pull/11784 Since Index SPI forward index config was ignoring `noDictionaryConfig` and `noDictionaryColumns`. This PR fixes that issue and add a couple of tests to detect similar regressions in the future.

[pinot] branch master updated: Fix a flaky test (#11771)

2023-10-11 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7124a035e0 Fix a flaky test (#11771) 7124a035e0

Re: [PR] Fix a flaky test [pinot]

2023-10-11 Thread via GitHub
xiangfu0 merged PR #11771: URL: https://github.com/apache/pinot/pull/11771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

  1   2   >