Re: [PR] Support helm chart server separate liveness and readiness probe endpoints [pinot]

2023-10-12 Thread via GitHub
zhtaoxiang commented on code in PR #11800: URL: https://github.com/apache/pinot/pull/11800#discussion_r1357801716 ## helm/pinot/templates/server/statefulset.yaml: ## @@ -88,15 +88,23 @@ spec: initialDelaySeconds: {{ .Values.probes.initialDelaySeconds }} per

Re: [I] Configure QueryServer and GrpcMailboxServer for TLS [pinot]

2023-10-12 Thread via GitHub
anandheritage commented on issue #11777: URL: https://github.com/apache/pinot/issues/11777#issuecomment-1760701931 @xiangfu0 Let me know if this should be fine as a beginner task. I can start with this. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] pinot-controller pod clash over and over again in kubernetes [pinot]

2023-10-12 Thread via GitHub
haowanv commented on issue #11764: URL: https://github.com/apache/pinot/issues/11764#issuecomment-1760692623 This issue has been fixed. Actually in the past six months, pinot keeps restarting, so there have accumulated so many tmp files in our s3 table dir. I cleaned them up(which cost a

Re: [PR] 11085: Added commons-configuration2 dependency. [pinot]

2023-10-12 Thread via GitHub
abhioncbr commented on PR #11792: URL: https://github.com/apache/pinot/pull/11792#issuecomment-1760681501 @Jackie-Jiang, can you please retrigger the failing tests, as failure is irrelevant to this change? And also, can you please review it? Thanks -- This is an automated message f

Re: [PR] [multistage][hotfix] init execution stats map to avoid NPE [pinot]

2023-10-12 Thread via GitHub
walterddr merged PR #11801: URL: https://github.com/apache/pinot/pull/11801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

[pinot] branch master updated: [multistage][hotfix] init execution stats map to avoid NPE (#11801)

2023-10-12 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new bfa03fe387 [multistage][hotfix] init execution sta

Re: [PR] [multistage][hotfix] init execution stats map to avoid NPE [pinot]

2023-10-12 Thread via GitHub
codecov-commenter commented on PR #11801: URL: https://github.com/apache/pinot/pull/11801#issuecomment-1760646049 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11801?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] [multistage][hotfix] init execution stats map to avoid NPE [pinot]

2023-10-12 Thread via GitHub
walterddr opened a new pull request, #11801: URL: https://github.com/apache/pinot/pull/11801 regression #11746. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Add pending segments to reload job status [pinot]

2023-10-12 Thread via GitHub
walterddr commented on PR #11785: URL: https://github.com/apache/pinot/pull/11785#issuecomment-1760594606 yes #11787 addresses the OFFLINE issue. this one was used more for debugging purpose to find out which segment was slow reloading. -- This is an automated message from the Apache Git

Re: [PR] Add support to retry failed table rebalance [pinot]

2023-10-12 Thread via GitHub
klsince commented on PR #11740: URL: https://github.com/apache/pinot/pull/11740#issuecomment-1760593159 > Can we provide an rest API to abort the current rebalance and stop the following retries? Good idea. This would need a new ABORTED status, different from FAILED status. Aborted

Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]

2023-10-12 Thread via GitHub
abhioncbr commented on code in PR #11799: URL: https://github.com/apache/pinot/pull/11799#discussion_r1357586948 ## pinot-plugins/pinot-file-system/pinot-s3/pom.xml: ## @@ -39,6 +39,7 @@ 4.4.13 2.12.2 3.1.0 +5.3.0 Review Comment: We have the woodstox-core

Re: [I] Create DATETIMECONVERTWINDOWHOP function [pinot]

2023-10-12 Thread via GitHub
Jackie-Jiang commented on issue #11775: URL: https://github.com/apache/pinot/issues/11775#issuecomment-1760562714 @dario-liberman Thanks for providing the context! The docs you referenced are actually window functions, and here are some discussions around it #7213. It is not really possible

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-12 Thread via GitHub
itschrispeck commented on PR #11739: URL: https://github.com/apache/pinot/pull/11739#issuecomment-1760561296 Updated perf results after adding caching, please take another look when convenient. Also added some short info about json_data blob size, and another query pattern w/ `json_match` f

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-12 Thread via GitHub
itschrispeck commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1357580261 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int g

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-12 Thread via GitHub
itschrispeck commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1357579947 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int g

Re: [I] UI does not show full stacktrace for minion tasks [pinot]

2023-10-12 Thread via GitHub
Jackie-Jiang commented on issue #11795: URL: https://github.com/apache/pinot/issues/11795#issuecomment-1760550228 @jayeshchoudhary Can you help take a look? Is the stack trace trimmed by UI or the response already have it trimmed? -- This is an automated message from the Apache Git Servic

Re: [PR] Support Upsert deletion for TTL: construct queryableDocIds when adding segments out of TTL [pinot]

2023-10-12 Thread via GitHub
Jackie-Jiang commented on code in PR #11791: URL: https://github.com/apache/pinot/pull/11791#discussion_r1357567627 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -138,15 +138,17 @@ public void addSegment

Re: [PR] Support helm chart server separate liveness and readiness probe endpoints [pinot]

2023-10-12 Thread via GitHub
codecov-commenter commented on PR #11800: URL: https://github.com/apache/pinot/pull/11800#issuecomment-1760547086 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11800?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] API to validate two tables are co-located [pinot]

2023-10-12 Thread via GitHub
ankitsultana commented on issue #11788: URL: https://github.com/apache/pinot/issues/11788#issuecomment-1760543269 @xiangfu0 : can you also help define what colocation mean here exactly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Use expected version in api for table config read modify write change [pinot]

2023-10-12 Thread via GitHub
Jackie-Jiang commented on code in PR #11782: URL: https://github.com/apache/pinot/pull/11782#discussion_r1357530963 ## pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java: ## @@ -73,12 +74,61 @@ public static void setUserConfig(ZkHelixPropertySto

Re: [PR] Support helm chart server separate liveness and readiness probe endpoints [pinot]

2023-10-12 Thread via GitHub
alvinzhangdd commented on PR #11800: URL: https://github.com/apache/pinot/pull/11800#issuecomment-1760503106 @walterddr I don't quite follow, do you mean I should change the values.yaml? Can you tell me where should I change? I thought I should keep values.yaml unchanged, it is th

Re: [I] Allow schema to be created with JSON config in UI [pinot]

2023-10-12 Thread via GitHub
Jackie-Jiang commented on issue #11789: URL: https://github.com/apache/pinot/issues/11789#issuecomment-1760490055 @jayeshchoudhary Can we make the schema config part editable, which can be reflected to the per-column config? -- This is an automated message from the Apache Git Service. To

[PR] Support helm chart server separate liveness and readiness probe endpoints [pinot]

2023-10-12 Thread via GitHub
alvinzhangdd opened a new pull request, #11800: URL: https://github.com/apache/pinot/pull/11800 ### Change Support helm chart server separate liveness and readiness probe endpoints ### Issue https://github.com/apache/pinot/issues/11798 ### Problem Latest pinot helm char

Re: [PR] Issue-6420 Add support for Apache Datasketches CPC sketch [pinot]

2023-10-12 Thread via GitHub
davecromberge commented on code in PR #11774: URL: https://github.com/apache/pinot/pull/11774#discussion_r1357492853 ## pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java: ## @@ -515,23 +518,23 @@ public VarianceTuple deserialize(ByteBuffer byteBuffer) {

Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]

2023-10-12 Thread via GitHub
codecov-commenter commented on PR #11799: URL: https://github.com/apache/pinot/pull/11799#issuecomment-1760412460 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11799?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] move jobId out of rebalanceConfig as it's more like a runtime context [pinot]

2023-10-12 Thread via GitHub
klsince merged PR #11790: URL: https://github.com/apache/pinot/pull/11790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[pinot] branch master updated (1b7cb166de -> 0174c71535)

2023-10-12 Thread xbli
This is an automated email from the ASF dual-hosted git repository. xbli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 1b7cb166de Add support to read exact buffer byte ranges corresponding to a given forward index doc id (#11729) add

[PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]

2023-10-12 Thread via GitHub
dang-stripe opened a new pull request, #11799: URL: https://github.com/apache/pinot/pull/11799 cc @Jackie-Jiang @xiangfu0 We ran into an issue where our custom plugin jars were deleted from disk and our realtime tables started failing to upload their segments. This occurred without

Re: [PR] This commit reverts the PR https://github.com/apache/pinot/pull/8852 [pinot]

2023-10-12 Thread via GitHub
dongxiaoman commented on PR #11797: URL: https://github.com/apache/pinot/pull/11797#issuecomment-1760357983 For context, in my previous PR I was only trying to do refactor in hopes that we may create an application-layer Proxy easily. It is not needed anymore, so this PR is cool and should

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-12 Thread via GitHub
dttung2905 commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1760350716 Yes @xiangfu0 . I will definitely ping you on community slack channel if I encounter any problem -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] This commit reverts the PR https://github.com/apache/pinot/pull/8852 [pinot]

2023-10-12 Thread via GitHub
codecov-commenter commented on PR #11797: URL: https://github.com/apache/pinot/pull/11797#issuecomment-1760350412 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11797?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Upgrade spark from 3.2 to 3.5 [pinot]

2023-10-12 Thread via GitHub
xiangfu0 commented on PR #11702: URL: https://github.com/apache/pinot/pull/11702#issuecomment-1760344429 can you rebase the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[I] Support helm chart server separate liveness and readiness probe endpoints [pinot]

2023-10-12 Thread via GitHub
alvinzhangdd opened a new issue, #11798: URL: https://github.com/apache/pinot/issues/11798 ### Issue Latest pinot helm chart doesn't support separate liveness and readiness probe endpoint for server ([link](https://github.com/apache/pinot/blob/master/helm/pinot/values.yaml#L311-L314)) b

[PR] This commit reverts the PR https://github.com/apache/pinot/pull/8852 [pinot]

2023-10-12 Thread via GitHub
soumitra-st opened a new pull request, #11797: URL: https://github.com/apache/pinot/pull/11797 This commit reverts the PR https://github.com/apache/pinot/pull/8852 because the annotations defined on the interface methods are not accessible using https://docs.oracle.com/javase/8/docs/api/jav

Re: [I] Create an API for cloning a table [pinot]

2023-10-12 Thread via GitHub
Jackie-Jiang commented on issue #11781: URL: https://github.com/apache/pinot/issues/11781#issuecomment-1760260316 @deemoliu This is a good question! IMO when cloning the table, we should also copy the segments in the deep store, so that after cloning the table the segments are isolated in t

Re: [PR] Issue-6420 Add support for Apache Datasketches CPC sketch [pinot]

2023-10-12 Thread via GitHub
cbalci commented on code in PR #11774: URL: https://github.com/apache/pinot/pull/11774#discussion_r1357254877 ## pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java: ## @@ -515,23 +518,23 @@ public VarianceTuple deserialize(ByteBuffer byteBuffer) { }

Re: [PR] [Do not review] add support for retention on deleted upsert keys [pinot]

2023-10-12 Thread via GitHub
deemoliu commented on code in PR #11779: URL: https://github.com/apache/pinot/pull/11779#discussion_r1357214183 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -129,20 +133,23 @@ public void addSegment(Imm

[I] Dropping Out-of-order events for upserts [pinot]

2023-10-12 Thread via GitHub
tibrewalpratik17 opened a new issue, #11796: URL: https://github.com/apache/pinot/issues/11796 In one of our use-cases, there are lot of out-of-order events being sent in Kafka. Though we don't consider out-of-order events for updating [Metadata HashMap](https://github.com/apache/pinot/blob

Re: [PR] reloadStatus should only count online segments [pinot]

2023-10-12 Thread via GitHub
walterddr commented on code in PR #11787: URL: https://github.com/apache/pinot/pull/11787#discussion_r1357079898 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -665,7 +665,7 @@ public ServerReloadControllerJobS

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-12 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1759900835 > What should be the correct value? I have the env variable named PINOT_AUTH set from the secret in kubernetes. I meant the `${env:PINOT_AUTH}` should resolve to the base64 aut

Re: [PR] add json_index_extract transform function to leverage json index for json value extraction [pinot]

2023-10-12 Thread via GitHub
chenboat commented on code in PR #11739: URL: https://github.com/apache/pinot/pull/11739#discussion_r1357016262 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java: ## @@ -300,6 +305,92 @@ private int getDo

Re: [PR] [UI] use indexes API instead of fetching all segment metadata [pinot]

2023-10-12 Thread via GitHub
walterddr commented on code in PR #11793: URL: https://github.com/apache/pinot/pull/11793#discussion_r1356981755 ## pinot-controller/src/main/resources/app/utils/PinotMethodUtils.ts: ## @@ -898,6 +898,29 @@ const reloadStatusOp = (tableName, tableType) => { return reloadStatu

Re: [I] Create DATETIMECONVERTWINDOWHOP function [pinot]

2023-10-12 Thread via GitHub
dario-liberman commented on issue #11775: URL: https://github.com/apache/pinot/issues/11775#issuecomment-1759789164 There are also moving time window SQL extensions in timeseries DBs, for example: https://docs.tdengine.com/taos-sql/distinguished/#time-window -- This is an automated messag

Re: [PR] Add compression configuration for functionColumnPairs to StartreeIndexConfigs [pinot]

2023-10-12 Thread via GitHub
t0mpere commented on code in PR #11744: URL: https://github.com/apache/pinot/pull/11744#discussion_r1356817966 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/StarTreeIndexConfig.java: ## @@ -42,15 +45,17 @@ public StarTreeIndexConfig( @JsonProperty(value = "

Re: [PR] track rebalance status when instance assignment already correct [pinot]

2023-10-12 Thread via GitHub
jadami10 closed pull request #11768: track rebalance status when instance assignment already correct URL: https://github.com/apache/pinot/pull/11768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] track rebalance status when instance assignment already correct [pinot]

2023-10-12 Thread via GitHub
jadami10 commented on PR #11768: URL: https://github.com/apache/pinot/pull/11768#issuecomment-1759519754 it seems we fixed this in #11073 and never merged it internally. tested with #11073, and rebalances are working as expected. thank you folks for looking. -- This is an automated messag

Re: [I] Create DATETIMECONVERTWINDOWHOP function [pinot]

2023-10-12 Thread via GitHub
dario-liberman commented on issue #11775: URL: https://github.com/apache/pinot/issues/11775#issuecomment-1759412122 A traditional SQL query with an example distinct count as aggregation could look something along these lines: ` SELECT DATETIMECONVERT(t1.timestamp, <>) AS window_st

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-12 Thread via GitHub
KKcorps commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1759368369 > Hi, @KKcorps how are you? Thanks for looking into this issue! > > > > Answering your questions: > > > > 1. The first segment commit is getting commited,

Re: [I] Servers can't consume messages due to permission denied error in the endpoint '/segmentConsumed' when ACL is enabled [pinot]

2023-10-12 Thread via GitHub
rodolphogarrido commented on issue #11407: URL: https://github.com/apache/pinot/issues/11407#issuecomment-1759360624 Hi, @KKcorps how are you? Thanks for looking into this issue! Answering your questions: 1. The first segment commit is getting commited, but subsequent ones are n

Re: [PR] [Do not review] add support for retention on deleted upsert keys [pinot]

2023-10-12 Thread via GitHub
tibrewalpratik17 commented on code in PR #11779: URL: https://github.com/apache/pinot/pull/11779#discussion_r1356593308 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -129,20 +133,23 @@ public void addSeg

Re: [I] Add GENERATE_ARRAY function [pinot]

2023-10-12 Thread via GitHub
anandheritage commented on issue #11786: URL: https://github.com/apache/pinot/issues/11786#issuecomment-1759204841 Okay @xiangfu0 , Kindly let me know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[I] UI does not show full stacktrace for minion tasks [pinot]

2023-10-12 Thread via GitHub
piby180 opened a new issue, #11795: URL: https://github.com/apache/pinot/issues/11795 It would be nice if the stacktrace is not trimmed and full stacktrace can be read from the UI instead of going to the logs. ![image](https://github.com/apache/pinot/assets/8469664/8782e4bc-f441-481f

Re: [I] Create DATETIMECONVERTWINDOWHOP function [pinot]

2023-10-12 Thread via GitHub
dario-liberman commented on issue #11775: URL: https://github.com/apache/pinot/issues/11775#issuecomment-1759101836 In streaming SQL there are also similar workarounds, for example see flink construct here: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/wind

Re: [PR] [UI] use indexes API instead of fetching all segment metadata [pinot]

2023-10-12 Thread via GitHub
saurabhd336 commented on PR #11793: URL: https://github.com/apache/pinot/pull/11793#issuecomment-1759067174 @jayeshchoudhary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add pending segments to reload job status [pinot]

2023-10-12 Thread via GitHub
walterddr commented on PR #11785: URL: https://github.com/apache/pinot/pull/11785#issuecomment-1759062112 > @walterddr > > 1. With replication > 1, the response may include the same segment name multiple times in the list. Do you think that should be okay? Good question. I actu