[GitHub] [pinot] Lvnszn closed issue #7818: Pinot will skip the transform if the column already exists

2021-11-23 Thread GitBox
Lvnszn closed issue #7818: URL: https://github.com/apache/pinot/issues/7818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pi

[GitHub] [pinot] Lvnszn commented on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
Lvnszn commented on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977606846 Thanks for your reply. IMO, the tableConfig is base on schemaConfig, so give me a hint than the rule in tableConfig higher weight than schema. -- This is an automated message from th

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7364: support large payload in zk put API

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7364: URL: https://github.com/apache/pinot/pull/7364#issuecomment-945155571 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7364?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] siddharthteotia commented on a change in pull request #7823: return exception when unavailable segments on empty broker response

2021-11-23 Thread GitBox
siddharthteotia commented on a change in pull request #7823: URL: https://github.com/apache/pinot/pull/7823#discussion_r755762444 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java ## @@ -459,7 +459,13 @@ private Broke

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7364: support large payload in zk put API

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7364: URL: https://github.com/apache/pinot/pull/7364#issuecomment-945155571 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7364?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7364: support large payload in zk put API

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7364: URL: https://github.com/apache/pinot/pull/7364#issuecomment-945155571 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7364?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7364: support large payload in zk put API

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7364: URL: https://github.com/apache/pinot/pull/7364#issuecomment-945155571 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7364?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7825: Remove pinot-ingestion-job script

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7825: URL: https://github.com/apache/pinot/pull/7825#issuecomment-977547948 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7825: Remove pinot-ingestion-job script

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7825: URL: https://github.com/apache/pinot/pull/7825#issuecomment-977547948 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7825: Remove pinot-ingestion-job script

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7825: URL: https://github.com/apache/pinot/pull/7825#issuecomment-977547948 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter commented on pull request #7825: Remove pinot-ingestion-job script

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7825: URL: https://github.com/apache/pinot/pull/7825#issuecomment-977547948 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] xiangfu0 opened a new pull request #7825: Remove pinot-ingestion-job script

2021-11-23 Thread GitBox
xiangfu0 opened a new pull request #7825: URL: https://github.com/apache/pinot/pull/7825 ## Description Replaced with ``` bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml ``` ## Upgrade Notes Does this PR prevent

[GitHub] [pinot] kautshukla opened a new pull request #7824: Pass in Headers when creating PinotConnection. Fix #7773

2021-11-23 Thread GitBox
kautshukla opened a new pull request #7824: URL: https://github.com/apache/pinot/pull/7824 Prerequisite: pinot/pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotDriver.java headers map is not empty {JDBC with headers for Auth enabled client} As of n

[GitHub] [pinot] kautshukla closed pull request #7774: Pass in Headers when creating PinotConnection. Fix #7773

2021-11-23 Thread GitBox
kautshukla closed pull request #7774: URL: https://github.com/apache/pinot/pull/7774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsub

[GitHub] [pinot] kautshukla commented on pull request #7774: Pass in Headers when creating PinotConnection. Fix #7773

2021-11-23 Thread GitBox
kautshukla commented on pull request #7774: URL: https://github.com/apache/pinot/pull/7774#issuecomment-977530469 @Jackie-Jiang : Will be closing and re creating pull request. Mistakenly pushed to down stream. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7823: return exception when unavailable segments on empty broker response

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7823: URL: https://github.com/apache/pinot/pull/7823#issuecomment-977499036 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7823?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter commented on pull request #7823: return exception when unavailable segments on empty broker response

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7823: URL: https://github.com/apache/pinot/pull/7823#issuecomment-977499036 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7823?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7568: EXPLAIN PLAN

2021-11-23 Thread GitBox
Jackie-Jiang commented on a change in pull request #7568: URL: https://github.com/apache/pinot/pull/7568#discussion_r755656841 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java ## @@ -519,6 +543,18 @@ private BrokerRe

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7568: EXPLAIN PLAN

2021-11-23 Thread GitBox
Jackie-Jiang commented on a change in pull request #7568: URL: https://github.com/apache/pinot/pull/7568#discussion_r755656841 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java ## @@ -519,6 +543,18 @@ private BrokerRe

[pinot] branch master updated (819cba0 -> 90f5881)

2021-11-23 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from 819cba0 Fix the broker routing when segment is deleted (#7817) add 90f5881 Pass Pinot Connection properties fr

[GitHub] [pinot] xiangfu0 merged pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
xiangfu0 merged pull request #7822: URL: https://github.com/apache/pinot/pull/7822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubsc

[GitHub] [pinot] jadami10 opened a new pull request #7823: return exception when unavailable segments on empty broker response

2021-11-23 Thread GitBox
jadami10 opened a new pull request #7823: URL: https://github.com/apache/pinot/pull/7823 ## Description This fixes #7811. In #7264 we added exceptions when not all segments are available, but this was one case that was missed. Specifically if all servers are unavailable, both realti

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7822: URL: https://github.com/apache/pinot/pull/7822#issuecomment-977308961 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7822: URL: https://github.com/apache/pinot/pull/7822#issuecomment-977308961 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7822: URL: https://github.com/apache/pinot/pull/7822#issuecomment-977308961 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7774: Pass in Headers when creating PinotConnection. Fix #7773

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7774: URL: https://github.com/apache/pinot/pull/7774#issuecomment-977346867 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7774?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter commented on pull request #7774: Pass in Headers when creating PinotConnection. Fix #7773

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7774: URL: https://github.com/apache/pinot/pull/7774#issuecomment-977346867 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7774?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7822: URL: https://github.com/apache/pinot/pull/7822#issuecomment-977308961 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
Jackie-Jiang commented on a change in pull request #7822: URL: https://github.com/apache/pinot/pull/7822#discussion_r755606608 ## File path: pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotConnection.java ## @@ -40,11 +41,20 @@ private PinotContro

[GitHub] [pinot] Jackie-Jiang merged pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
Jackie-Jiang merged pull request #7817: URL: https://github.com/apache/pinot/pull/7817 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-uns

[pinot] branch master updated (a74d944 -> 819cba0)

2021-11-23 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from a74d944 Adding pinot client connection config to allow skip fail on broker response exception (#7816) add 819cb

[GitHub] [pinot] Jackie-Jiang commented on issue #7791: Only download required files from segment for metadata push

2021-11-23 Thread GitBox
Jackie-Jiang commented on issue #7791: URL: https://github.com/apache/pinot/issues/7791#issuecomment-977312406 Just went over the code, and we are actually using the proper metadata push API (the other way is letting the controller download the segment and extract the metadata). The ineffi

[GitHub] [pinot] codecov-commenter commented on pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7822: URL: https://github.com/apache/pinot/pull/7822#issuecomment-977308961 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7817: URL: https://github.com/apache/pinot/pull/7817#issuecomment-977204199 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7817?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] Jackie-Jiang commented on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
Jackie-Jiang commented on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977303001 This is expected behavior. Pinot will skip the transform if the column already exists -- This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7817: URL: https://github.com/apache/pinot/pull/7817#issuecomment-977204199 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7817?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] xiangfu0 edited a comment on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
xiangfu0 edited a comment on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977289127 I think by default pinot extract all the top level fields from JSON into a PinotRecord for consuming, then evaluate each transform function. So filed `sku` is already extract

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7817: URL: https://github.com/apache/pinot/pull/7817#issuecomment-977204199 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7817?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] xiangfu0 commented on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
xiangfu0 commented on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977289127 I think by default pinot extract all the top level fields into a record for consuming, then evaluate each transform function. So filed `sku` is already extracted then the transform

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] kkrugler edited a comment on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
kkrugler edited a comment on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977282279 Hmm, others should weigh in here, but since the JSON record ALSO contains an `sku` field at the top level, this looks like undefined behavior to me, not a bug. -- This is an

[GitHub] [pinot] kkrugler commented on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
kkrugler commented on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977282279 Hmm, others should weigh in here, but since your record ALSO contains an `sku` field at the top level, this looks like undefined behavior to me, not a bug. Why not call your column so

[GitHub] [pinot] xiangfu0 opened a new pull request #7822: Pass Pinot Connection properties from JDBC driver

2021-11-23 Thread GitBox
xiangfu0 opened a new pull request #7822: URL: https://github.com/apache/pinot/pull/7822 ## Description Passing info properties also to Pinot Connection. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[GitHub] [pinot] richardstartin commented on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
richardstartin commented on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977264186 The try/catches can be removed if https://github.com/json-path/JsonPath/pull/767 is merged and released. -- This is an automated message from the Apache Git Service. To r

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7781: URL: https://github.com/apache/pinot/pull/7781#issuecomment-971204560 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7781?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter commented on pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7820: URL: https://github.com/apache/pinot/pull/7820#issuecomment-977259274 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7781: URL: https://github.com/apache/pinot/pull/7781#issuecomment-971204560 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7781?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] richardstartin opened a new issue #7821: Discuss: Forking the JsonPath library

2021-11-23 Thread GitBox
richardstartin opened a new issue #7821: URL: https://github.com/apache/pinot/issues/7821 I have made several improvements and bug fixes to the [JsonPath library](https://github.com/json-path/JsonPath) since September, motivated by improving Pinot, and these changes have been merged but no

[GitHub] [pinot] kkrugler commented on issue #7791: Only download required files from segment for metadata push

2021-11-23 Thread GitBox
kkrugler commented on issue #7791: URL: https://github.com/apache/pinot/issues/7791#issuecomment-977233280 Hi @Jackie-Jiang - yes, re using a metadata push so that the controller doesn't have to download the segment. But I'm curious what the 2 versions of metadata push were. I only know of

[GitHub] [pinot] mqliang edited a comment on issue #6720: Benchmark setColumn() in DataTableBuilder and write all values one by one instead of using rowId and colId to position if need be

2021-11-23 Thread GitBox
mqliang edited a comment on issue #6720: URL: https://github.com/apache/pinot/issues/6720#issuecomment-808974129 I write a bench mark here: https://github.com/mqliang/pinot/commit/a32a61aad5dfa6b6c4a09064c75926b00495cd3a The benchmark compares three ways to build a data table: * `

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
richardstartin commented on a change in pull request #7820: URL: https://github.com/apache/pinot/pull/7820#discussion_r72508 ## File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java ## @@ -120,6 +121,17 @@ public int getNumDocs() { retur

[GitHub] [pinot] richardstartin commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
richardstartin commented on a change in pull request #7820: URL: https://github.com/apache/pinot/pull/7820#discussion_r71967 ## File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java ## @@ -120,6 +121,17 @@ public int getNumDocs() { retur

[GitHub] [pinot] xiangfu0 commented on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
xiangfu0 commented on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977228027 > Hi @Lvnszn - are you saying that because the "sku" field in the incoming JSON record IS NOT in the `custom_props` sub-record, you don't think it should wind up in the row? ye

[GitHub] [pinot] mayankshriv commented on a change in pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
mayankshriv commented on a change in pull request #7820: URL: https://github.com/apache/pinot/pull/7820#discussion_r70432 ## File path: pinot-core/src/main/java/org/apache/pinot/core/common/DataBlockCache.java ## @@ -120,6 +121,17 @@ public int getNumDocs() { return i

[GitHub] [pinot] richardstartin opened a new pull request #7820: push JSON Path evaluation down to storage layer

2021-11-23 Thread GitBox
richardstartin opened a new pull request #7820: URL: https://github.com/apache/pinot/pull/7820 Pushes JSON path evaluation down to the storage layer (giving direct access to dictionaries and forward index) which avoids various intermediate materialisations of strings, byte arrays and so on

[GitHub] [pinot] codecov-commenter commented on pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7817: URL: https://github.com/apache/pinot/pull/7817#issuecomment-977204199 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7817?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] lakshmanan-v commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
lakshmanan-v commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755529666 ## File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GapFillGroupByDataTableReducer.java ## @@ -0,0 +1,490 @@ +/** + * Licensed t

[GitHub] [pinot] weixiangsun commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
weixiangsun commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755528039 ## File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GapFillGroupByDataTableReducer.java ## @@ -0,0 +1,490 @@ +/** + * Licensed to

[GitHub] [pinot] kkrugler commented on issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
kkrugler commented on issue #7818: URL: https://github.com/apache/pinot/issues/7818#issuecomment-977187482 Hi @Lvnszn - are you saying that because the "sku" field in the incoming JSON record IS NOT in the `custom_props` sub-record, you don't think it should wind up in the row? -- This

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
Jackie-Jiang commented on a change in pull request #7817: URL: https://github.com/apache/pinot/pull/7817#discussion_r755511591 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java ## @@ -110,94 +110,99 @@ public void init(HelixManager he

[pinot] branch master updated (4aef0ba -> a74d944)

2021-11-23 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from 4aef0ba change default range index version to v2 (#7815) add a74d944 Adding pinot client connection config to

[GitHub] [pinot] xiangfu0 closed issue #7770: Make broker segment unavailable exception configurable on Pinot Java client

2021-11-23 Thread GitBox
xiangfu0 closed issue #7770: URL: https://github.com/apache/pinot/issues/7770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@

[GitHub] [pinot] xiangfu0 merged pull request #7816: Adding pinot client connection config to allow skip fail on broker response exception

2021-11-23 Thread GitBox
xiangfu0 merged pull request #7816: URL: https://github.com/apache/pinot/pull/7816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubsc

[GitHub] [pinot] xiangfu0 commented on a change in pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
xiangfu0 commented on a change in pull request #7817: URL: https://github.com/apache/pinot/pull/7817#discussion_r754805996 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java ## @@ -110,94 +110,99 @@ public void init(HelixManager helixM

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
Jackie-Jiang commented on a change in pull request #7817: URL: https://github.com/apache/pinot/pull/7817#discussion_r755459241 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java ## @@ -110,94 +110,99 @@ public void init(HelixManager he

[pinot] branch master updated (767aa8a -> 4aef0ba)

2021-11-23 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from 767aa8a Proceed to next time window when no segment match for RealtimeToOfflineTask (#7814) add 4aef0ba change

[GitHub] [pinot] Jackie-Jiang merged pull request #7815: change default range index version to v2

2021-11-23 Thread GitBox
Jackie-Jiang merged pull request #7815: URL: https://github.com/apache/pinot/pull/7815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-uns

[GitHub] [pinot] Jackie-Jiang edited a comment on pull request #7815: change default range index version to v2

2021-11-23 Thread GitBox
Jackie-Jiang edited a comment on pull request #7815: URL: https://github.com/apache/pinot/pull/7815#issuecomment-977083516 @siddharthteotia Yes, reader can read all different versions, and the reader support is included in `0.9.0` release. Existing segments won't be modified -- This is a

[GitHub] [pinot] Jackie-Jiang commented on pull request #7815: change default range index version to v2

2021-11-23 Thread GitBox
Jackie-Jiang commented on pull request #7815: URL: https://github.com/apache/pinot/pull/7815#issuecomment-977083516 @siddharthteotia Yes, reader can read all different versions, and the reader support is included in `0.9.0` release -- This is an automated message from the Apache Git Serv

[GitHub] [pinot] lfernandez93 commented on issue #5246: Pagination support for groupBy queries

2021-11-23 Thread GitBox
lfernandez93 commented on issue #5246: URL: https://github.com/apache/pinot/issues/5246#issuecomment-977078950 +1 is there any update on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [pinot] siddharthteotia commented on pull request #7815: change default range index version to v2

2021-11-23 Thread GitBox
siddharthteotia commented on pull request #7815: URL: https://github.com/apache/pinot/pull/7815#issuecomment-977068519 To confirm, reader should still be able to read existing segments with old version right ? Are we planning to convert existing segments to V3 during segment preprocessor /

[GitHub] [pinot] lfernandez93 commented on issue #5246: Pagination support for groupBy queries

2021-11-23 Thread GitBox
lfernandez93 commented on issue #5246: URL: https://github.com/apache/pinot/issues/5246#issuecomment-977027075 @kishoreg how do we up vote this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [pinot] jackjlli commented on a change in pull request #7817: Fix the broker routing when segment is deleted

2021-11-23 Thread GitBox
jackjlli commented on a change in pull request #7817: URL: https://github.com/apache/pinot/pull/7817#discussion_r755425417 ## File path: pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java ## @@ -110,94 +110,99 @@ public void init(HelixManager helixM

[GitHub] [pinot] weixiangsun commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
weixiangsun commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755366473 ## File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GapFillGroupByDataTableReducer.java ## @@ -0,0 +1,490 @@ +/** + * Licensed to

[GitHub] [pinot] weixiangsun commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
weixiangsun commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755367238 ## File path: pinot-core/src/test/java/org/apache/pinot/queries/PostAggregationGapfillQueriesTest.java ## @@ -0,0 +1,550 @@ +/** + * Licensed to the Ap

[GitHub] [pinot] weixiangsun commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
weixiangsun commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755366473 ## File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GapFillGroupByDataTableReducer.java ## @@ -0,0 +1,490 @@ +/** + * Licensed to

[GitHub] [pinot] weixiangsun commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
weixiangsun commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755365577 ## File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GapFillGroupByDataTableReducer.java ## @@ -0,0 +1,490 @@ +/** + * Licensed to

[GitHub] [pinot] weixiangsun commented on a change in pull request #7781: Add Post-Aggregation Gapfilling functionality.

2021-11-23 Thread GitBox
weixiangsun commented on a change in pull request #7781: URL: https://github.com/apache/pinot/pull/7781#discussion_r755364417 ## File path: pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GapFillGroupByDataTableReducer.java ## @@ -0,0 +1,490 @@ +/** + * Licensed to

[pinot] branch master updated: Proceed to next time window when no segment match for RealtimeToOfflineTask (#7814)

2021-11-23 Thread nehapawar
This is an automated email from the ASF dual-hosted git repository. nehapawar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 767aa8a Proceed to next time window when no s

[GitHub] [pinot] npawar closed issue #6988: Realtime to Offline segment conversion task fails to progress with data gaps

2021-11-23 Thread GitBox
npawar closed issue #6988: URL: https://github.com/apache/pinot/issues/6988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pi

[GitHub] [pinot] npawar merged pull request #7814: Proceed to next time window when no segment match for RealtimeToOfflineTask

2021-11-23 Thread GitBox
npawar merged pull request #7814: URL: https://github.com/apache/pinot/pull/7814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr.

[GitHub] [pinot] mneedham commented on issue #7789: [Docs] IN_ID_SET

2021-11-23 Thread GitBox
mneedham commented on issue #7789: URL: https://github.com/apache/pinot/issues/7789#issuecomment-976614503 This is where we've got to so far - https://docs.pinot.apache.org/users/user-guide-query/filtering-with-idset -- This is an automated message from the Apache Git Service. To respond

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7819: URL: https://github.com/apache/pinot/pull/7819#issuecomment-976515875 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7819?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] codecov-commenter commented on pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7819: URL: https://github.com/apache/pinot/pull/7819#issuecomment-976515875 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7819?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] richardstartin commented on pull request #7802: avoid binary searches in distinct count

2021-11-23 Thread GitBox
richardstartin commented on pull request #7802: URL: https://github.com/apache/pinot/pull/7802#issuecomment-976509437 @Jackie-Jiang it depends. Constructing bitmaps from unordered values is inefficient, and in general there's no point in using `RoaringBitmapWriter`, so if the dictionary i

[GitHub] [pinot] richardstartin edited a comment on pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
richardstartin edited a comment on pull request #7819: URL: https://github.com/apache/pinot/pull/7819#issuecomment-976445015 It turns out there's a bug in the library which prevents this configuration from being respected in all cases, which I have fixed in a [PR to the library](https://gi

[GitHub] [pinot] richardstartin commented on a change in pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
richardstartin commented on a change in pull request #7819: URL: https://github.com/apache/pinot/pull/7819#discussion_r755082884 ## File path: pinot-common/src/test/java/org/apache/pinot/common/function/scalar/ArrayAwareJacksonJsonProviderTest.java ## @@ -48,15 +49,11 @@ publi

[GitHub] [pinot] richardstartin commented on a change in pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
richardstartin commented on a change in pull request #7819: URL: https://github.com/apache/pinot/pull/7819#discussion_r755082361 ## File path: pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java ## @@ -149,50 +159,45 @@ public static String js

[GitHub] [pinot] richardstartin commented on pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
richardstartin commented on pull request #7819: URL: https://github.com/apache/pinot/pull/7819#issuecomment-976445015 It turns out there's a bug in the library which prevents this configuration from being respected in all cases, which I have fixed in a PR to the library. Converting to draf

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7815: change default range index version to v2

2021-11-23 Thread GitBox
codecov-commenter edited a comment on pull request #7815: URL: https://github.com/apache/pinot/pull/7815#issuecomment-976097602 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=T

[GitHub] [pinot] richardstartin opened a new pull request #7819: prevent exceptions from being thrown when jsonpaths don't match inputs

2021-11-23 Thread GitBox
richardstartin opened a new pull request #7819: URL: https://github.com/apache/pinot/pull/7819 By default, when a JSON path doesn't match a document, the JSONPath library throws an exception, but this can be disabled with configuration. These exceptions can be quite costly during ingestion

[GitHub] [pinot] codecov-commenter commented on pull request #7816: Adding pinot client connection config to allow skip fail on broker response exception

2021-11-23 Thread GitBox
codecov-commenter commented on pull request #7816: URL: https://github.com/apache/pinot/pull/7816#issuecomment-976405029 # [Codecov](https://codecov.io/gh/apache/pinot/pull/7816?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apac

[GitHub] [pinot] Lvnszn opened a new issue #7818: 【BUG】Field not in json will be persist.

2021-11-23 Thread GitBox
Lvnszn opened a new issue #7818: URL: https://github.com/apache/pinot/issues/7818 ## TableConfig ``` { "REALTIME": { "tableName": "tracker_analyze_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "tracker_analyze", "timeC

[GitHub] [pinot] richardstartin commented on pull request #7815: change default range index version to v2

2021-11-23 Thread GitBox
richardstartin commented on pull request #7815: URL: https://github.com/apache/pinot/pull/7815#issuecomment-976393298 @Jackie-Jiang I see, that's a different error, it didn't get that far before I force pushed the first time. -- This is an automated message from the Apache Git Service. T