[pinot] branch master updated: [multistage][feature] add pipeline breaker stats collector (#10958)

2023-06-23 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 6a47311c03 [multistage][feature] add pipeline brea

[GitHub] [pinot] walterddr merged pull request #10958: [multistage][feature] add pipeline breaker stats collector

2023-06-23 Thread via GitHub
walterddr merged PR #10958: URL: https://github.com/apache/pinot/pull/10958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

[GitHub] [pinot] chenboat commented on pull request #10937: New API to get tenant tags

2023-06-23 Thread via GitHub
chenboat commented on PR #10937: URL: https://github.com/apache/pinot/pull/10937#issuecomment-1605171907 @npuppala111 @ankitsultana the instance tag can be retrieved from Pinot controller using the controller's instance api. The tag is stored in Zookeeper and the Pinot controller is usually

[GitHub] [pinot] chenboat commented on a diff in pull request #10937: New API to get tenant tags

2023-06-23 Thread via GitHub
chenboat commented on code in PR #10937: URL: https://github.com/apache/pinot/pull/10937#discussion_r1240527020 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/InstanceResource.java: ## @@ -0,0 +1,69 @@ +/** + * Licensed to the Apache Software Foundation (ASF

[GitHub] [pinot] Jackie-Jiang commented on issue #10934: TEXT_CONTAINS fails with NullPointerException

2023-06-23 Thread via GitHub
Jackie-Jiang commented on issue #10934: URL: https://github.com/apache/pinot/issues/10934#issuecomment-1605124817 Per the stack trace, the exception is thrown from the native text index. @atris Can you help take a look? Seems `ImmutableFST` is using the `OffHeapMutableBytesStore` which se

[GitHub] [pinot] codecov-commenter commented on pull request #10971: [multistage][feature] pipeline breaker on broker reduce

2023-06-23 Thread via GitHub
codecov-commenter commented on PR #10971: URL: https://github.com/apache/pinot/pull/10971#issuecomment-1605118330 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10971?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] walterddr commented on a diff in pull request #10971: [multistage][feature] pipeline breaker on broker reduce

2023-06-23 Thread via GitHub
walterddr commented on code in PR #10971: URL: https://github.com/apache/pinot/pull/10971#discussion_r1240492928 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java: ## @@ -85,15 +87,15 @@ public QueryDispatcher() { } public

[pinot] branch master updated: Simplify the handling for partial-upsert record update (#10970)

2023-06-23 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new e8841316fa Simplify the handling for partial-upse

[GitHub] [pinot] Jackie-Jiang merged pull request #10970: Simplify the handling for partial-upsert record update

2023-06-23 Thread via GitHub
Jackie-Jiang merged PR #10970: URL: https://github.com/apache/pinot/pull/10970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] walterddr opened a new pull request, #10971: [multistage][feature] pipeline breaker on broker reduce

2023-06-23 Thread via GitHub
walterddr opened a new pull request, #10971: URL: https://github.com/apache/pinot/pull/10971 this fixes #10131 - improved upon #10794 - depends on #10958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [pinot] chenboat commented on pull request #10867: FUNNEL_COUNT Aggregation Function

2023-06-23 Thread via GitHub
chenboat commented on PR #10867: URL: https://github.com/apache/pinot/pull/10867#issuecomment-1605060642 > In order for this aggregation to work, does it require all the data to be partitioned by segments (i.e. all users show up in the same segment, and no user can be shared across segments

[GitHub] [pinot] walterddr commented on a diff in pull request #10963: Update dockerfile with recent jdk distro changes

2023-06-23 Thread via GitHub
walterddr commented on code in PR #10963: URL: https://github.com/apache/pinot/pull/10963#discussion_r1240455111 ## docker/images/pinot-base/pinot-base-build/openjdk.dockerfile: ## @@ -16,33 +16,37 @@ # specific language governing permissions and limitations # under the Licens

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #10963: Update dockerfile with recent jdk distro changes

2023-06-23 Thread via GitHub
xiangfu0 commented on code in PR #10963: URL: https://github.com/apache/pinot/pull/10963#discussion_r1240449412 ## docker/images/pinot-base/pinot-base-build/openjdk.dockerfile: ## @@ -16,33 +16,37 @@ # specific language governing permissions and limitations # under the License

[GitHub] [pinot] codecov-commenter commented on pull request #10970: Simplify the handling for partial-upsert record update

2023-06-23 Thread via GitHub
codecov-commenter commented on PR #10970: URL: https://github.com/apache/pinot/pull/10970#issuecomment-1604944654 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10970?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] Jackie-Jiang commented on pull request #10867: FUNNEL_COUNT Aggregation Function

2023-06-23 Thread via GitHub
Jackie-Jiang commented on PR #10867: URL: https://github.com/apache/pinot/pull/10867#issuecomment-1604934620 In order for this aggregation to work, does it require all the data to be partitioned by segments (i.e. all users show up in the same segment, and no user can be shared across segmen

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #10970: Simplify the handling for partial-upsert record update

2023-06-23 Thread via GitHub
Jackie-Jiang opened a new pull request, #10970: URL: https://github.com/apache/pinot/pull/10970 Since we moved the out-of-order event handling into the `doAddRecord()` in #10944, `doUpdateRecord()` can be simplified -- This is an automated message from the Apache Git Service. To respond t

[pinot] branch master updated: Remove handle out-of-order from updateRecord method (#10968)

2023-06-23 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new f8e783899f Remove handle out-of-order from update

[GitHub] [pinot] Jackie-Jiang merged pull request #10968: Remove handle out-of-order-events call from updateRecord method

2023-06-23 Thread via GitHub
Jackie-Jiang merged PR #10968: URL: https://github.com/apache/pinot/pull/10968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] Jackie-Jiang merged pull request #10703: Add delete support to upsert tables

2023-06-23 Thread via GitHub
Jackie-Jiang merged PR #10703: URL: https://github.com/apache/pinot/pull/10703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[pinot] branch master updated: Add delete support to upsert tables (#10703)

2023-06-23 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 608cb8de2b Add delete support to upsert tables (#

[pinot] branch master updated: Use _hasNull in the object single column distinct executors. (#10914)

2023-06-23 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7e76cac586 Use _hasNull in the object single colu

[GitHub] [pinot] Jackie-Jiang merged pull request #10914: Use _hasNull in the object single column distinct executors.

2023-06-23 Thread via GitHub
Jackie-Jiang merged PR #10914: URL: https://github.com/apache/pinot/pull/10914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] jbewing commented on issue #10960: Large off-heap memory usage?

2023-06-23 Thread via GitHub
jbewing commented on issue #10960: URL: https://github.com/apache/pinot/issues/10960#issuecomment-1604721257 One thing to look out for is that some of the 0.12.0/1 runtime images for pinot are based on an older, deprecated [openjdk:11-jdk-slim docker image](https://hub.docker.com/layers/lib

[GitHub] [pinot] klsince commented on a diff in pull request #10953: Add a PrioritizedFilterOperator

2023-06-23 Thread via GitHub
klsince commented on code in PR #10953: URL: https://github.com/apache/pinot/pull/10953#discussion_r1240063859 ## pinot-core/src/main/java/org/apache/pinot/core/operator/filter/PrioritizedFilterOperator.java: ## @@ -0,0 +1,51 @@ +/** + * Licensed to the Apache Software Foundatio

[GitHub] [pinot] kirkrodrigues commented on issue #10945: `mvn clean install` fail on Mac with M1 chip

2023-06-23 Thread via GitHub
kirkrodrigues commented on issue #10945: URL: https://github.com/apache/pinot/issues/10945#issuecomment-1604558647 Hi @mingmxu, clp-ffi doesn't have aarch64 support yet. I will try and add it this weekend and submit a PR. -- This is an automated message from the Apache Git Service. To res

[GitHub] [pinot] mingmxu commented on pull request #10966: use MultiStageRequestIdGenerator in v1 engine

2023-06-23 Thread via GitHub
mingmxu commented on PR #10966: URL: https://github.com/apache/pinot/pull/10966#issuecomment-1604544437 cc @Jackie-Jiang @walterddr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [pinot] walterddr commented on a diff in pull request #10963: Update dockerfile with recent jdk distro changes

2023-06-23 Thread via GitHub
walterddr commented on code in PR #10963: URL: https://github.com/apache/pinot/pull/10963#discussion_r1239991746 ## docker/images/pinot-base/pinot-base-build/openjdk.dockerfile: ## @@ -16,33 +16,37 @@ # specific language governing permissions and limitations # under the Licens

[GitHub] [pinot] tibrewalpratik17 commented on pull request #10968: Remove handle out-of-order-events call from updateRecord method

2023-06-23 Thread via GitHub
tibrewalpratik17 commented on PR #10968: URL: https://github.com/apache/pinot/pull/10968#issuecomment-1604276432 cc @Jackie-Jiang @atris -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [pinot] abhioncbr commented on a diff in pull request #10891: Ensure min/max value generation in the segment metadata.

2023-06-23 Thread via GitHub
abhioncbr commented on code in PR #10891: URL: https://github.com/apache/pinot/pull/10891#discussion_r1239791153 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java: ## @@ -570,12 +570,14 @@ public static void

[GitHub] [pinot] abhioncbr commented on a diff in pull request #10891: Ensure min/max value generation in the segment metadata.

2023-06-23 Thread via GitHub
abhioncbr commented on code in PR #10891: URL: https://github.com/apache/pinot/pull/10891#discussion_r1239788013 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/columnminmaxvalue/ColumnMinMaxValueGenerator.java: ## @@ -119,65 +121,126 @@

[GitHub] [pinot] codecov-commenter commented on pull request #10969: Instance drop operation validation check API

2023-06-23 Thread via GitHub
codecov-commenter commented on PR #10969: URL: https://github.com/apache/pinot/pull/10969#issuecomment-1604210200 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10969?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] shounakmk219 opened a new pull request, #10969: Instance drop check api

2023-06-23 Thread via GitHub
shounakmk219 opened a new pull request, #10969: URL: https://github.com/apache/pinot/pull/10969 Expose an endpoint to perform safety check validation of the instance drop operation. Right now in order to drop an instance we can use - `/instances/{instanceName}` with `DELETE` method t

[GitHub] [pinot] codecov-commenter commented on pull request #10968: Remove handle out-of-order-events call from updateRecord method

2023-06-23 Thread via GitHub
codecov-commenter commented on PR #10968: URL: https://github.com/apache/pinot/pull/10968#issuecomment-1604053339 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10968?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #10944: Push out-of-order events metrics for full upsert

2023-06-23 Thread via GitHub
tibrewalpratik17 commented on code in PR #10944: URL: https://github.com/apache/pinot/pull/10944#discussion_r1239596214 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -371,13 +371,15 @@ public GenericRow

[GitHub] [pinot] tibrewalpratik17 opened a new pull request, #10968: Remove handle out-of-order-events call from updateRecord method

2023-06-23 Thread via GitHub
tibrewalpratik17 opened a new pull request, #10968: URL: https://github.com/apache/pinot/pull/10968 Follow-up to a comment in #10944 to remove double accounting of out-of-order events for partial upsert table. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [pinot] codecov-commenter commented on pull request #10967: Make ProjectionBlockValSet::getNullBitmap return a clone of the cache.

2023-06-23 Thread via GitHub
codecov-commenter commented on PR #10967: URL: https://github.com/apache/pinot/pull/10967#issuecomment-1603908474 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10967?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] codecov-commenter commented on pull request #10966: use MultiStageRequestIdGenerator in v1 engine

2023-06-23 Thread via GitHub
codecov-commenter commented on PR #10966: URL: https://github.com/apache/pinot/pull/10966#issuecomment-1603902669 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10966?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] eaugene commented on pull request #10961: Backfill segments into real-time table

2023-06-23 Thread via GitHub
eaugene commented on PR #10961: URL: https://github.com/apache/pinot/pull/10961#issuecomment-1603875645 @mneedham Can you share about the following : 1. When real-time segments are uploaded, There's a chance we may end up with two segments having overlapping timestamps, how would this be

[GitHub] [pinot] shenyu0127 opened a new pull request, #10967: Make ProjectionBlockValSet::getNullBitmap return a clone of the cache.

2023-06-23 Thread via GitHub
shenyu0127 opened a new pull request, #10967: URL: https://github.com/apache/pinot/pull/10967 This is to avoid the cache being modified. Tested in unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [pinot] mingmxu opened a new pull request, #10966: use MultiStageRequestIdGenerator in v1 engine

2023-06-23 Thread via GitHub
mingmxu opened a new pull request, #10966: URL: https://github.com/apache/pinot/pull/10966 As mentioned in #10943 , broker requestId in v1 engine is not unique since it's generated as below: ``` protected final AtomicLong _requestIdGenerator = new AtomicLong(); long requestId = _req

[GitHub] [pinot] shenyu0127 commented on pull request #10914: Use _hasNull in the object single column distinct executors.

2023-06-23 Thread via GitHub
shenyu0127 commented on PR #10914: URL: https://github.com/apache/pinot/pull/10914#issuecomment-1603859189 > LGTM! Can you resolve the conflict in the test? Thank you for the review and approval! Rebased and force-pushed. -- This is an automated message from the Apache Git Service.

[GitHub] [pinot] gortiz commented on issue #10945: `mvn clean install` fail on Mac with M1 chip

2023-06-23 Thread via GitHub
gortiz commented on issue #10945: URL: https://github.com/apache/pinot/issues/10945#issuecomment-1603801121 I usually use sdkman to install different JDKs. In case you are using it, you need to enable Rosetta in sdkman as well in order to install the amd64 JDK. This article explains how to

[GitHub] [pinot] Jackie-Jiang commented on pull request #10905: Startree index build enhancement

2023-06-23 Thread via GitHub
Jackie-Jiang commented on PR #10905: URL: https://github.com/apache/pinot/pull/10905#issuecomment-1603769836 Can you please take a look at the failed integration test? I think it is related to the changes -- This is an automated message from the Apache Git Service. To respond to the messa