(pinot) branch master updated (eeaf1f0811 -> 35faeb6712)

2024-01-22 Thread saurabhd336
This is an automated email from the ASF dual-hosted git repository. saurabhd336 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from eeaf1f0811 [feature] allow dim table config to detect/disallow duplicate PK (#12290) add 35faeb6712 Revert

Re: [PR] Revert allowing tag override with upserts [pinot]

2024-01-22 Thread via GitHub
saurabhd336 merged PR #12311: URL: https://github.com/apache/pinot/pull/12311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.

[PR] Revert allowing tag override with upserts [pinot]

2024-01-22 Thread via GitHub
saurabhd336 opened a new pull request, #12311: URL: https://github.com/apache/pinot/pull/12311 Reverts https://github.com/apache/pinot/pull/12233. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
siddharthteotia commented on PR #12297: URL: https://github.com/apache/pinot/pull/12297#issuecomment-1905460977 Also, it will be good to update the PR description with a bit more details on how we noticed the problem. -- This is an automated message from the Apache Git Service. To respond

[PR] Support table suffix in ZkBasicAuthAccessControlFactory [pinot]

2024-01-22 Thread via GitHub
xiangfu0 opened a new pull request, #12310: URL: https://github.com/apache/pinot/pull/12310 This is to support `_OFFLINE` and `_REALTIME` suffix for querying table. https://github.com/apache/pinot/assets/1202120/74ee2f68-f361-426f-855c-ceef12e1fbda";> -- This is an automated mes

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
siddharthteotia commented on PR #12297: URL: https://github.com/apache/pinot/pull/12297#issuecomment-1905459050 LGTM overall. @mcvsubbu / @vvivekiyer / @somandal - in case one of you wants to take a look -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
siddharthteotia commented on PR #12297: URL: https://github.com/apache/pinot/pull/12297#issuecomment-1905457451 Another qq - Will the same release method be called if the table is dropped / deleted ? I ask because we start the executor service with threads = numSegments When th

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
siddharthteotia commented on code in PR #12297: URL: https://github.com/apache/pinot/pull/12297#discussion_r1462833181 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -223,8 +224,12 @@ protected void releaseAndRemoveAllSegments() {

Re: [I] Support uptime API in all pinot components [pinot]

2024-01-22 Thread via GitHub
xiangfu0 commented on issue #12298: URL: https://github.com/apache/pinot/issues/12298#issuecomment-1905344428 Thanks @suyashpatel98 ! Assigned the ticket to you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
jackjlli commented on code in PR #12297: URL: https://github.com/apache/pinot/pull/12297#discussion_r1462699835 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -223,8 +224,12 @@ protected void releaseAndRemoveAllSegments() {

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (3ed9d68147 -> 2abd892fe1)

2024-01-22 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments in repository https://gitbox.apache.org/repos/asf/pinot.git discard 3ed9d68147 Release all segments of a table in releaseAndRemoveAllSegments meth

Re: [I] Parsing http-headers in RequestContext - broker query event listener [pinot]

2024-01-22 Thread via GitHub
walterddr commented on issue #12273: URL: https://github.com/apache/pinot/issues/12273#issuecomment-1905231612 some example i had in mind, - should we allow cluster config modification (e.g. not as a static broker conf file but modifiable during runtime) - where do we allow this confi

Re: [PR] Add HttpHeaders in broker event listener requestContext [pinot]

2024-01-22 Thread via GitHub
walterddr commented on code in PR #12258: URL: https://github.com/apache/pinot/pull/12258#discussion_r1462683329 ## pinot-spi/src/main/java/org/apache/pinot/spi/eventlistener/query/PinotBrokerQueryEventListenerFactory.java: ## @@ -44,6 +50,8 @@ private PinotBrokerQueryEventListe

Re: [PR] Add HttpHeaders in broker event listener requestContext [pinot]

2024-01-22 Thread via GitHub
walterddr commented on code in PR #12258: URL: https://github.com/apache/pinot/pull/12258#discussion_r1462680033 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -44,6 +44,7 @@ private CommonConstants() { public static final String UNKNOWN = "

Re: [I] Parsing http-headers in RequestContext - broker query event listener [pinot]

2024-01-22 Thread via GitHub
walterddr commented on issue #12273: URL: https://github.com/apache/pinot/issues/12273#issuecomment-1905217828 i think an allowed list would be desirable. but it is a bit hard to make it configurable/extensible without overriding it. do we have any idea on how one can extend this to support

Re: [PR] add ScalingThreadPoolExecutor and use for realtime Lucene thread pool [pinot]

2024-01-22 Thread via GitHub
walterddr commented on PR #12274: URL: https://github.com/apache/pinot/pull/12274#issuecomment-1905216084 CC @xiangfu0 to take a loo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Added ARRAY_GENERATE function [pinot]

2024-01-22 Thread via GitHub
walterddr commented on code in PR #12293: URL: https://github.com/apache/pinot/pull/12293#discussion_r1462673614 ## pinot-common/src/test/java/org/apache/pinot/common/function/FunctionDefinitionRegistryTest.java: ## @@ -43,7 +43,7 @@ public class FunctionDefinitionRegistryTest {

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
walterddr commented on code in PR #12297: URL: https://github.com/apache/pinot/pull/12297#discussion_r1462672871 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -223,8 +224,12 @@ protected void releaseAndRemoveAllSegments() {

Re: [I] Support uptime API in all pinot components [pinot]

2024-01-22 Thread via GitHub
suyashpatel98 commented on issue #12298: URL: https://github.com/apache/pinot/issues/12298#issuecomment-1905188785 Hello, @xiangfu0! I am very new to Pinot and I am interested in contributing. I think this issue will be a great starting point. Can you please assign this ticket to me? Thanks

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
jackjlli commented on PR #12297: URL: https://github.com/apache/pinot/pull/12297#issuecomment-1905150851 > Are we going to start an ExecutorService every time? The `releaseAndRemoveAllSegments` method is only invoked when shutting down the whole pinot-server. So within the lifecycle o

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-01-22 Thread via GitHub
siddharthteotia commented on PR #12297: URL: https://github.com/apache/pinot/pull/12297#issuecomment-1905117278 - Are we going to start an ExecutorService every time ? - Did we take a thread dump to understand why the segment release was blocked / what was it waiting on ? -- This is an

Re: [I] Multistage engine and multivalue column semantics incompatibilities [pinot]

2024-01-22 Thread via GitHub
walterddr commented on issue #12306: URL: https://github.com/apache/pinot/issues/12306#issuecomment-1905073303 could you attach the explain plan result for this query. This is actually a problem b/c the mv column is not exactly semantically "relational algebra" it is a short-handed r

Re: [I] Error while installing using mvn clean install -DskipTests -Pbin-dist [pinot]

2024-01-22 Thread via GitHub
xiangfu0 commented on issue #11836: URL: https://github.com/apache/pinot/issues/11836#issuecomment-1905062689 Hi @nakul777 have you made it work on windows? The previous logs seem to be correct, just need more time to compile it -- This is an automated message from the Apache Git Servi

[I] [multistage] prevent OOM from broker/server QueryRunner that received too many data [pinot]

2024-01-22 Thread via GitHub
walterddr opened a new issue, #12309: URL: https://github.com/apache/pinot/issues/12309 there are several execution path that could cause a legitimate OOM issue when executing a super large result set query such as `select * FROM a JOIN b on a.boolCol = b.boolCol`, this could result in m

Re: [I] Able to add duplicate rows with same primary key in dimension table [pinot]

2024-01-22 Thread via GitHub
walterddr commented on issue #12284: URL: https://github.com/apache/pinot/issues/12284#issuecomment-1905041985 feature has been merged. closing this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Able to add duplicate rows with same primary key in dimension table [pinot]

2024-01-22 Thread via GitHub
walterddr closed issue #12284: Able to add duplicate rows with same primary key in dimension table URL: https://github.com/apache/pinot/issues/12284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [feature] allow dim table config to detect/disallow duplicate PK [pinot]

2024-01-22 Thread via GitHub
walterddr merged PR #12290: URL: https://github.com/apache/pinot/pull/12290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

Re: [PR] [multistage][feature] function registry and lookup method consolidation [pinot]

2024-01-22 Thread via GitHub
walterddr closed pull request #12229: [multistage][feature] function registry and lookup method consolidation URL: https://github.com/apache/pinot/pull/12229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

(pinot) branch master updated (f1fec060a6 -> eeaf1f0811)

2024-01-22 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from f1fec060a6 Support runtime reload for TLS resources (#12277) add eeaf1f0811 [feature] allow dim table config to det

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (cb5a82080f -> 3ed9d68147)

2024-01-22 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments in repository https://gitbox.apache.org/repos/asf/pinot.git discard cb5a82080f Release all segments of a table in releaseAndRemoveAllSegments meth

Re: [PR] [feature] allow dim table config to detect/disallow duplicate PK [pinot]

2024-01-22 Thread via GitHub
walterddr commented on code in PR #12290: URL: https://github.com/apache/pinot/pull/12290#discussion_r1462459714 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java: ## @@ -87,6 +87,7 @@ public static DimensionTableDataManager g

Re: [PR] [feature] allow dim table config to detect/disallow duplicate PK [pinot]

2024-01-22 Thread via GitHub
Jackie-Jiang commented on code in PR #12290: URL: https://github.com/apache/pinot/pull/12290#discussion_r1462413055 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java: ## @@ -87,6 +87,7 @@ public static DimensionTableDataManager

[I] PartitionAware Routing does not work on Realtime Upsert Table using kinesis [pinot]

2024-01-22 Thread via GitHub
abhijeetkushe opened a new issue, #12308: URL: https://github.com/apache/pinot/issues/12308 I have a realtime upsert table which reads data from kinesis.There are 2 partitions or 2 kinesis shards.I updated config but after I restarted broker controller and server I still see 2 servers being

(pinot) branch master updated: Support runtime reload for TLS resources (#12277)

2024-01-22 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new f1fec060a6 Support runtime reload for TLS resour

Re: [PR] Support runtime reload for TLS resources [pinot]

2024-01-22 Thread via GitHub
xiangfu0 merged PR #12277: URL: https://github.com/apache/pinot/pull/12277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] [feature] allow dim table config to detect/disallow duplicate PK [pinot]

2024-01-22 Thread via GitHub
walterddr commented on code in PR #12290: URL: https://github.com/apache/pinot/pull/12290#discussion_r1462343378 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java: ## @@ -87,6 +87,7 @@ public static DimensionTableDataManager g

Re: [PR] Adding support for configuration through environment variables [pinot]

2024-01-22 Thread via GitHub
xiangfu0 commented on code in PR #12307: URL: https://github.com/apache/pinot/pull/12307#discussion_r1462291215 ## pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java: ## @@ -201,16 +200,16 @@ private static Configuration loadProperties(String configPath) {

Re: [PR] Adding support for configuration through environment variables [pinot]

2024-01-22 Thread via GitHub
xiangfu0 commented on code in PR #12307: URL: https://github.com/apache/pinot/pull/12307#discussion_r1462287971 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -331,7 +331,6 @@ public static class Broker { // Broker config indicating the ma

Re: [PR] Adding support for configuration through environment variables [pinot]

2024-01-22 Thread via GitHub
codecov-commenter commented on PR #12307: URL: https://github.com/apache/pinot/pull/12307#issuecomment-1904631042 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12307?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Enhance SSL Context Handling in Client Transports for Dynamic Certificate Rotation [pinot]

2024-01-22 Thread via GitHub
kay-owolabi commented on PR #12194: URL: https://github.com/apache/pinot/pull/12194#issuecomment-1904618233 > @kay-owolabi Thanks for the fix. I am working on a PR to make the KeyManager and TrustManager swappable (as mentioned in #12107). Maybe we can wait a while to see if my fix can fix

Re: [PR] Implement IN and NOT_IN filters in FilterOperand class (#12285 fix) [pinot]

2024-01-22 Thread via GitHub
codecov-commenter commented on PR #12305: URL: https://github.com/apache/pinot/pull/12305#issuecomment-1904599666 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12305?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[I] Multistage engine and multivalue column semantics incompatibilities [pinot]

2024-01-22 Thread via GitHub
gyorfimi opened a new issue, #12306: URL: https://github.com/apache/pinot/issues/12306 Multistage engine and multivalue column semantics incompatibilities In Pinot filtering on a multivalue column with `=` operator actually a *contains* operator, aggregation to a multivalue column is

Re: [PR] Bump com.jayway.jsonpath:json-path from 2.8.0 to 2.9.0 [pinot]

2024-01-22 Thread via GitHub
codecov-commenter commented on PR #12303: URL: https://github.com/apache/pinot/pull/12303#issuecomment-1904527617 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12303?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] Multistage query failure when select a CASE/IN expression [pinot]

2024-01-22 Thread via GitHub
gyorfimi commented on issue #12285: URL: https://github.com/apache/pinot/issues/12285#issuecomment-1904493186 #12305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Implement IN and NOT_IN filters in FilterOperand class (#12285 fix) [pinot]

2024-01-22 Thread via GitHub
gyorfimi opened a new pull request, #12305: URL: https://github.com/apache/pinot/pull/12305 [bugfix] Added IN and NOT_IN cases to the FilterOperand class and TransformOperandFactory classes. The PR is related to the 12285 bug: using of the IN expression in the CASE statement within

[I] [multistage] attach default limit 10 to queries that doesn't have a limit [pinot]

2024-01-22 Thread via GitHub
walterddr opened a new issue, #12304: URL: https://github.com/apache/pinot/issues/12304 - to prevent expensive queries from potentially killing the broker if we do not apply this by default. - we also need a way to explicitly warn the user the query result is default limited as a warnin

(pinot) branch dependabot/maven/com.jayway.jsonpath-json-path-2.9.0 created (now b29ff5a7d6)

2024-01-22 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.jayway.jsonpath-json-path-2.9.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at b29ff5a7d6 Bump com.jayway.jsonpath:json-path from 2.8.0 to 2.9.0 N

[PR] Bump com.jayway.jsonpath:json-path from 2.8.0 to 2.9.0 [pinot]

2024-01-22 Thread via GitHub
dependabot[bot] opened a new pull request, #12303: URL: https://github.com/apache/pinot/pull/12303 Bumps [com.jayway.jsonpath:json-path](https://github.com/jayway/JsonPath) from 2.8.0 to 2.9.0. Release notes Sourced from https://github.com/jayway/JsonPath/releases";>com.jayway.json

(pinot) branch master updated: Use higher fetch timeout for Kinesis (#12214)

2024-01-22 Thread kharekartik
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ced6bc282e Use higher fetch timeout for Kine

Re: [PR] Use higher fetch timeout for Kinesis [pinot]

2024-01-22 Thread via GitHub
KKcorps merged PR #12214: URL: https://github.com/apache/pinot/pull/12214 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

Re: [PR] [multistage][feature] FunctionRegistry unification attempt2 [pinot]

2024-01-22 Thread via GitHub
codecov-commenter commented on PR #12302: URL: https://github.com/apache/pinot/pull/12302#issuecomment-1904390800 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12302?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] [multistage][feature] FunctionRegistry unification attempt2 [pinot]

2024-01-22 Thread via GitHub
walterddr opened a new pull request, #12302: URL: https://github.com/apache/pinot/pull/12302 this is #12229 improvement to address the recent modification to #12278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[I] IndexOutOfBoundsException is thrown when loading previously compacted segment with BYTES dimension [pinot]

2024-01-22 Thread via GitHub
robertzych opened a new issue, #12301: URL: https://github.com/apache/pinot/issues/12301 When the schema includes a `BYTES` dimension, and the table config contains an empty `noDictionaryColumns` list, an `IndexOutOfBoundsException` is raised during compaction when gathering stats for a pre

Re: [PR] Mark distribution as multi-release [pinot]

2024-01-22 Thread via GitHub
codecov-commenter commented on PR #12300: URL: https://github.com/apache/pinot/pull/12300#issuecomment-1903540535 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12300?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

(pinot) branch master updated: Refactoring the upsert compaction related code (#12275)

2024-01-22 Thread snlee
This is an automated email from the ASF dual-hosted git repository. snlee pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7b69d094be Refactoring the upsert compaction relat

Re: [PR] Refactoring the upsert compaction related code [pinot]

2024-01-22 Thread via GitHub
snleee merged PR #12275: URL: https://github.com/apache/pinot/pull/12275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[PR] Mark distribution as multi-release [pinot]

2024-01-22 Thread via GitHub
gortiz opened a new pull request, #12300: URL: https://github.com/apache/pinot/pull/12300 As explained in https://github.com/apache/pinot/pull/12131, we need to distribute Pinot as a multi-release Jar. The reason is that we distribute Pinot as a uber-jar with all dependencies in the same ja