[GitHub] [pinot] Ali-Armstrong opened a new issue, #9111: Nodejs Client

2022-07-26 Thread GitBox
Ali-Armstrong opened a new issue, #9111: URL: https://github.com/apache/pinot/issues/9111 Is Nodejs client support in progress? or is there a reason why pinot doesn't support nodejs ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [pinot] sajjad-moradi opened a new pull request, #9110: Fix authentication issue when auth annotation is not required

2022-07-26 Thread GitBox
sajjad-moradi opened a new pull request, #9110: URL: https://github.com/apache/pinot/pull/9110 AccessControl on pinot controller can enable authentication on the endpoints that don't even have `@Authentication` annotation (the access type is inferred based `@PUT`, `@GET`, `@POST`, and `@DEL

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930674996 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -1984,6 +1987,82 @@ private Set getAllInstancesForTable(

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930666999 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -557,6 +557,23 @@ public static class Minion { public static final String CONFIG

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930665635 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -557,6 +557,23 @@ public static class Minion { public static final String CONFIG

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930665449 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -557,6 +557,23 @@ public static class Minion { public static final String CONFIG

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930664772 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/ControllerJobStatusResource.java: ## @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software Foun

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930663113 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -1984,6 +1987,82 @@ private Set getAllInstancesForTable(

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930662579 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -1984,6 +1987,82 @@ private Set getAllInstancesForTable(

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930659917 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -1984,6 +1987,82 @@ private Set getAllInstancesForTable(

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930647944 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -577,6 +598,98 @@ public SuccessResponse reloadSegm

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930643047 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -577,6 +598,98 @@ public SuccessResponse reloadSegm

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930641884 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -577,6 +598,98 @@ public SuccessResponse reloadSegm

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
saurabhd336 commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r930640033 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -459,9 +464,25 @@ public SuccessResponse reloadSegm

[GitHub] [pinot] klsince commented on issue #8843: Allow a server to host segments on multi storage tiers

2022-07-26 Thread GitBox
klsince commented on issue #8843: URL: https://github.com/apache/pinot/issues/8843#issuecomment-1196277533 found another related issue: https://github.com/apache/pinot/issues/6343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [pinot] taobupt opened a new issue, #9109: roaringbitmap aggregation and deserialization

2022-07-26 Thread GitBox
taobupt opened a new issue, #9109: URL: https://github.com/apache/pinot/issues/9109 ### Background For use case like the dashboard, we have the requirements to precompute metrics for several years so we can enable the year_over_year comparison. One of the biggest challenges for the metri

[pinot] branch master updated: Add consuming metadata to broker response (#9092)

2022-07-26 Thread snlee
This is an automated email from the ASF dual-hosted git repository. snlee pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 9cf7d81dd0 Add consuming metadata to broker respon

[GitHub] [pinot] snleee closed issue #7144: Add `numConsumingSegmentsProcessed` and `numConsumingSegmentsMatched` into the query response metadata

2022-07-26 Thread GitBox
snleee closed issue #7144: Add `numConsumingSegmentsProcessed` and `numConsumingSegmentsMatched` into the query response metadata URL: https://github.com/apache/pinot/issues/7144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[GitHub] [pinot] snleee merged pull request #9092: Add consuming metadata to broker response

2022-07-26 Thread GitBox
snleee merged PR #9092: URL: https://github.com/apache/pinot/pull/9092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.

[pinot] branch message-header updated (f89cc3b53c -> b52d0659b6)

2022-07-26 Thread kishoreg
This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a change to branch message-header in repository https://gitbox.apache.org/repos/asf/pinot.git from f89cc3b53c Adding support to extract values from message header.. initial support for kafka headers add

[pinot] branch master updated: Fail spark application in case of no input files (#9105)

2022-07-26 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 3c799cef75 Fail spark application in case of no inp

[GitHub] [pinot] jackjlli merged pull request #9105: Fail spark application in case of no input files

2022-07-26 Thread GitBox
jackjlli merged PR #9105: URL: https://github.com/apache/pinot/pull/9105 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[GitHub] [pinot] amol-mathur opened a new issue, #9108: Ability to change logging levels at a component level dynamically

2022-07-26 Thread GitBox
amol-mathur opened a new issue, #9108: URL: https://github.com/apache/pinot/issues/9108 Hi, i want the ability to change log levels at a pinot component level (controller, broker etc) dynamically without requiring a restarts this is a pretty useful feature for anyone running pinot

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9093: Handle case when segment time threshold is crossed and 0 batch messages are processed

2022-07-26 Thread GitBox
Jackie-Jiang commented on code in PR #9093: URL: https://github.com/apache/pinot/pull/9093#discussion_r930415617 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java: ## @@ -440,6 +443,14 @@ protected boolean consumeLoop()

[GitHub] [pinot] deemoliu commented on pull request #9062: [Upsert] persist validDocsIndex snapshot for Pinot upsert optimization

2022-07-26 Thread GitBox
deemoliu commented on PR #9062: URL: https://github.com/apache/pinot/pull/9062#issuecomment-1196014305 @yupeng9 @Jackie-Jiang Do you have any suggestion on when to persist snapshot? I looked into partitionUpsertMetaDataManager and summarized the [scenarios for persisting snapshot](https://

[GitHub] [pinot] navina commented on pull request #9071: Add support for IAM role based credentials in Kinesis Plugin

2022-07-26 Thread GitBox
navina commented on PR #9071: URL: https://github.com/apache/pinot/pull/9071#issuecomment-1195971777 Please tag the PR with `feature` and `release-notes` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9086: Proper null handling in Aggregation functions for SV data types

2022-07-26 Thread GitBox
Jackie-Jiang commented on code in PR #9086: URL: https://github.com/apache/pinot/pull/9086#discussion_r930400432 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/MaxAggregationFunction.java: ## @@ -28,13 +28,19 @@ import org.apache.pinot.core.query.

[pinot] branch master updated: [UI] Add controller UI to use multi-stage engine (#9072)

2022-07-26 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 0abd8f87ab [UI] Add controller UI to use multi-sta

[GitHub] [pinot] walterddr merged pull request #9072: [UI] Add controller UI to use multi-stage engine

2022-07-26 Thread GitBox
walterddr merged PR #9072: URL: https://github.com/apache/pinot/pull/9072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[GitHub] [pinot] codecov-commenter commented on pull request #9100: [multistage] carry partition scheme for optimization

2022-07-26 Thread GitBox
codecov-commenter commented on PR #9100: URL: https://github.com/apache/pinot/pull/9100#issuecomment-1195879142 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9100?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8828: Add controller API for reload segment task status

2022-07-26 Thread GitBox
Jackie-Jiang commented on code in PR #8828: URL: https://github.com/apache/pinot/pull/8828#discussion_r929429595 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java: ## @@ -459,9 +464,25 @@ public SuccessResponse reloadSeg

[GitHub] [pinot] mcvsubbu commented on pull request #9058: Task genrator debug api

2022-07-26 Thread GitBox
mcvsubbu commented on PR #9058: URL: https://github.com/apache/pinot/pull/9058#issuecomment-1195866632 I had asked some questions about the use case you are tackling. Can you please answer those? How frequently are you generating tasks (in your use case)? Is a metric bump not enough for you

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9058: Task genrator debug api

2022-07-26 Thread GitBox
mcvsubbu commented on code in PR #9058: URL: https://github.com/apache/pinot/pull/9058#discussion_r930314460 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java: ## @@ -231,6 +260,55 @@ public Map getTasksDebugInf return

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9105: Fail spark application in case of no input files

2022-07-26 Thread GitBox
Jackie-Jiang commented on code in PR #9105: URL: https://github.com/apache/pinot/pull/9105#discussion_r930239005 ## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java: #

[GitHub] [pinot] klsince commented on issue #9084: Batch Ingestion from Delta Table

2022-07-26 Thread GitBox
klsince commented on issue #9084: URL: https://github.com/apache/pinot/issues/9084#issuecomment-1195761958 I just read a bit about the delta lib. A simple flow may look like below, where we can open the delta table with the lib and loop through all the records. The lib also supports data fi

[GitHub] [pinot] jugomezv commented on issue #9107: Issue with documentation for M1 Mac Support

2022-07-26 Thread GitBox
jugomezv commented on issue #9107: URL: https://github.com/apache/pinot/issues/9107#issuecomment-1195757933 Sure, let me try that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [pinot] snleee commented on issue #9107: Issue with documentation for M1 Mac Support

2022-07-26 Thread GitBox
snleee commented on issue #9107: URL: https://github.com/apache/pinot/issues/9107#issuecomment-1195752134 @jugomezv https://github.com/pinot-contrib/pinot-docs/ The documentation can be updated in the above repository. Can you update the change and file the PR? -- This is an automa

[GitHub] [pinot] snleee commented on issue #5089: Enhance Data Ingestion Engine

2022-07-26 Thread GitBox
snleee commented on issue #5089: URL: https://github.com/apache/pinot/issues/5089#issuecomment-1195734895 To add here, I think that we should introduce the column-based interface (maybe it's the same idea as `Design an interface (close to the idea of the stats collector) to store all the co

[GitHub] [pinot] jugomezv opened a new issue, #9107: Issue with documentation for M1 Mac Support

2022-07-26 Thread GitBox
jugomezv opened a new issue, #9107: URL: https://github.com/apache/pinot/issues/9107 I was going through the set up for M1 and found that [this](https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally#m1-mac-support)section seems to have an issue: The provided sett

[GitHub] [pinot] KKcorps commented on issue #9102: In the instance config, entries with same key are treated as array which can cause unexpected behavior

2022-07-26 Thread GitBox
KKcorps commented on issue #9102: URL: https://github.com/apache/pinot/issues/9102#issuecomment-1195651316 Yeah, I have been a victim of this as well. It is because of apache CommonsConfiguration class we are using. -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [pinot] saurabhd336 closed pull request #8961: Allow bloom filter for no dictionary SV columns

2022-07-26 Thread GitBox
saurabhd336 closed pull request #8961: Allow bloom filter for no dictionary SV columns URL: https://github.com/apache/pinot/pull/8961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [pinot] saurabhd336 commented on pull request #8961: Allow bloom filter for no dictionary SV columns

2022-07-26 Thread GitBox
saurabhd336 commented on PR #8961: URL: https://github.com/apache/pinot/pull/8961#issuecomment-1195533920 After further testing and discussions, came to realise that people generally move to no dictionary columns if they see high dictionary sizes. Introducing a hasSet to store unique values

[pinot] branch master updated: Replace deprecated methods in ParquetNativeRecordReader (#9106)

2022-07-26 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 6e0adf6c41 Replace deprecated methods in Parquet

[GitHub] [pinot] xiangfu0 merged pull request #9106: Replace deprecated methods in ParquetNativeRecordReader

2022-07-26 Thread GitBox
xiangfu0 merged PR #9106: URL: https://github.com/apache/pinot/pull/9106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[GitHub] [pinot] ddcprg commented on pull request #9093: Unit test on LLRealtimeSegmentDataManager that attests issue #9091

2022-07-26 Thread GitBox
ddcprg commented on PR #9093: URL: https://github.com/apache/pinot/pull/9093#issuecomment-1195269349 I've now pushed a fix using boolean instead, I'll convert this to a proper PR when the build is all green -- This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [pinot] codecov-commenter commented on pull request #9106: Replace deprecated methods in ParquetNativeRecordReader

2022-07-26 Thread GitBox
codecov-commenter commented on PR #9106: URL: https://github.com/apache/pinot/pull/9106#issuecomment-1195249990 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9106?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] ddcprg commented on pull request #9093: Unit test on LLRealtimeSegmentDataManager that attests issue #9091

2022-07-26 Thread GitBox
ddcprg commented on PR #9093: URL: https://github.com/apache/pinot/pull/9093#issuecomment-1195191865 @Jackie-Jiang thanks for feed back. The reasons why I've decided to return a counter instead are: - a boolean tells the end criteria has reached but doesn't tell anything about progress

[GitHub] [pinot] KKcorps opened a new pull request, #9106: Replace deprecated methods in ParquetNativeRecordReader

2022-07-26 Thread GitBox
KKcorps opened a new pull request, #9106: URL: https://github.com/apache/pinot/pull/9106 * Replace deprecated read footer method in Parquet record reader * Also, provide localFS hadoop configuration to Native reader (already provided to avro record reader) -- This is an automated messa

[GitHub] [pinot] nizarhejazi commented on a diff in pull request #9086: Proper null handling in Aggregation functions for SV data types

2022-07-26 Thread GitBox
nizarhejazi commented on code in PR #9086: URL: https://github.com/apache/pinot/pull/9086#discussion_r929689219 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/MaxAggregationFunction.java: ## @@ -28,13 +28,19 @@ import org.apache.pinot.core.query.a

[GitHub] [pinot] codecov-commenter commented on pull request #9105: Fail spark application in case of no input files

2022-07-26 Thread GitBox
codecov-commenter commented on PR #9105: URL: https://github.com/apache/pinot/pull/9105#issuecomment-1195171287 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9105?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] KKcorps opened a new pull request, #9105: Fail spark application in case of no input files

2022-07-26 Thread GitBox
KKcorps opened a new pull request, #9105: URL: https://github.com/apache/pinot/pull/9105 Currently spark application fails silently with `a positive number of partitions is required` error instead of throwing a proper exception when no input files are present. We should throw a runtime exce