[GitHub] [pinot] saurabhd336 commented on a diff in pull request #11576: Add table indexes API

2023-09-12 Thread via GitHub
saurabhd336 commented on code in PR #11576: URL: https://github.com/apache/pinot/pull/11576#discussion_r1323998034 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java: ## @@ -964,6 +965,77 @@ public String getTableAggregateM

[GitHub] [pinot] Jackie-Jiang commented on issue #11562: Add protection to Pinot upon reload of table

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11562: URL: https://github.com/apache/pinot/issues/11562#issuecomment-1716959378 Aborting a submitted reload request is a good feature to have, but even if we abort it, server will load the segments when it starts the next time, which will trigger the star-t

[GitHub] [pinot] rajagopr commented on a diff in pull request #11581: Enabled the method 'next(GenericRow row)' in the CSVRecordReader while using the line iterator

2023-09-12 Thread via GitHub
rajagopr commented on code in PR #11581: URL: https://github.com/apache/pinot/pull/11581#discussion_r1323908431 ## pinot-plugins/pinot-input-format/pinot-csv/src/test/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderTest.java: ## @@ -538,8 +538,9 @@ private List readC

[GitHub] [pinot] snleee commented on a diff in pull request #11581: Enabled the method 'next(GenericRow row)' in the CSVRecordReader while using the line iterator

2023-09-12 Thread via GitHub
snleee commented on code in PR #11581: URL: https://github.com/apache/pinot/pull/11581#discussion_r1323886100 ## pinot-plugins/pinot-input-format/pinot-csv/src/test/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderTest.java: ## @@ -538,8 +538,9 @@ private List readCSV

[GitHub] [pinot] snleee commented on a diff in pull request #11581: Enabled the method 'next(GenericRow row)' in the CSVRecordReader while using the line iterator

2023-09-12 Thread via GitHub
snleee commented on code in PR #11581: URL: https://github.com/apache/pinot/pull/11581#discussion_r1323886100 ## pinot-plugins/pinot-input-format/pinot-csv/src/test/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderTest.java: ## @@ -538,8 +538,9 @@ private List readCSV

[pinot] branch master updated: default enable metadata manager for colocated join quickstart (#11579)

2023-09-12 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 5253c37088 default enable metadata manager for col

[GitHub] [pinot] walterddr merged pull request #11579: [multistage] default enable metadata manager for colocated join quickstart

2023-09-12 Thread via GitHub
walterddr merged PR #11579: URL: https://github.com/apache/pinot/pull/11579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

[GitHub] [pinot] codecov-commenter commented on pull request #11581: Enabled the method 'next(GenericRow row)' in the CSVRecordReader while using the line iterator

2023-09-12 Thread via GitHub
codecov-commenter commented on PR #11581: URL: https://github.com/apache/pinot/pull/11581#issuecomment-1716872289 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11581?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] wirybeaver commented on a diff in pull request #10815: Periodically delete Tmp Segment file brought by the SplitCommit End phase

2023-09-12 Thread via GitHub
wirybeaver commented on code in PR #10815: URL: https://github.com/apache/pinot/pull/10815#discussion_r1323877297 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -239,6 +239,14 @@ public void cleanupSegmentCountGauge(final String reso

[GitHub] [pinot] wirybeaver commented on a diff in pull request #10815: Periodically delete Tmp Segment file brought by the SplitCommit End phase

2023-09-12 Thread via GitHub
wirybeaver commented on code in PR #10815: URL: https://github.com/apache/pinot/pull/10815#discussion_r1323785121 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -239,6 +239,14 @@ public void cleanupSegmentCountGauge(final String reso

[GitHub] [pinot] rajagopr opened a new pull request, #11581: Enabled the method 'next(GenericRow row)' in the CSVRecordReader while using the line iterator

2023-09-12 Thread via GitHub
rajagopr opened a new pull request, #11581: URL: https://github.com/apache/pinot/pull/11581 Changes are to initialize the incoming record with the record fetched by the line iterator as opposed to throwing an exception. -- This is an automated message from the Apache Git Service. To respo

[GitHub] [pinot] swaaap-nil commented on issue #9125: Allow using system properties to fill up the template value in configs.

2023-09-12 Thread via GitHub
swaaap-nil commented on issue #9125: URL: https://github.com/apache/pinot/issues/9125#issuecomment-1716791724 Hi i am new to the repo and would like to work on this. Can you please give more details? -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323331770 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneDocIdCollector.java: ## @@ -60,9 +61,16 @@ public voi

[GitHub] [pinot] wirybeaver commented on a diff in pull request #10815: Periodically delete Tmp Segment file brought by the SplitCommit End phase

2023-09-12 Thread via GitHub
wirybeaver commented on code in PR #10815: URL: https://github.com/apache/pinot/pull/10815#discussion_r1323793165 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -239,6 +239,14 @@ public void cleanupSegmentCountGauge(final String reso

[GitHub] [pinot] shenyu0127 closed pull request #11449: Register single value columns as nullable.

2023-09-12 Thread via GitHub
shenyu0127 closed pull request #11449: Register single value columns as nullable. URL: https://github.com/apache/pinot/pull/11449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [pinot] shenyu0127 commented on pull request #11449: Register single value columns as nullable.

2023-09-12 Thread via GitHub
shenyu0127 commented on PR #11449: URL: https://github.com/apache/pinot/pull/11449#issuecomment-1716761045 Had discussions with @Jackie-Jiang and @walterddr , and we decided not to merge this PR. This is because if we do table level nullability registration, queries on non-null colum

[GitHub] [pinot] shenyu0127 closed pull request #11474: Remove the outer istrue and isfalse functions.

2023-09-12 Thread via GitHub
shenyu0127 closed pull request #11474: Remove the outer istrue and isfalse functions. URL: https://github.com/apache/pinot/pull/11474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [pinot] shenyu0127 commented on pull request #11474: Remove the outer istrue and isfalse functions.

2023-09-12 Thread via GitHub
shenyu0127 commented on PR #11474: URL: https://github.com/apache/pinot/pull/11474#issuecomment-1716759087 Had a discussion with @Jackie-Jiang and decided we don't want to merge this PR. - This PR is to optimize queries from V2. Now we are not merging https://github.com/apache/pinot/

[GitHub] [pinot] wirybeaver commented on a diff in pull request #10815: Periodically delete Tmp Segment file brought by the SplitCommit End phase

2023-09-12 Thread via GitHub
wirybeaver commented on code in PR #10815: URL: https://github.com/apache/pinot/pull/10815#discussion_r1323785121 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -239,6 +239,14 @@ public void cleanupSegmentCountGauge(final String reso

[GitHub] [pinot] snleee opened a new issue, #11580: Add partition, validDocId as the derived column

2023-09-12 Thread via GitHub
snleee opened a new issue, #11580: URL: https://github.com/apache/pinot/issues/11580 We currently support some derived columns(e.g. $segmentName , $docId ). It will be useful to add $partitionValues, $isValidDocId -- This is an automated message from the Apache Git Service. To respo

[pinot] branch master updated: Disallow peer download when replication is < 2 (#11469)

2023-09-12 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 0929b4cdd1 Disallow peer download when replicatio

[GitHub] [pinot] Jackie-Jiang merged pull request #11469: Disallow peer download when replication is < 2

2023-09-12 Thread via GitHub
Jackie-Jiang merged PR #11469: URL: https://github.com/apache/pinot/pull/11469 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10815: Periodically delete Tmp Segment file brought by the SplitCommit End phase

2023-09-12 Thread via GitHub
Jackie-Jiang commented on code in PR #10815: URL: https://github.com/apache/pinot/pull/10815#discussion_r1323752263 ## pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java: ## @@ -215,9 +215,15 @@ public static class ControllerPeriodicTasksConf {

[GitHub] [pinot] Jackie-Jiang commented on issue #11494: Leverage Json index in the group-by clause

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11494: URL: https://github.com/apache/pinot/issues/11494#issuecomment-1716653409 Think of json index as an inverted index. It is extremely expensive to use inverted index as forward index because each value read requires scanning the whole inverted index.

[GitHub] [pinot] codecov-commenter commented on pull request #11579: [multistage] default enable metadata manager for colocated join quickstart

2023-09-12 Thread via GitHub
codecov-commenter commented on PR #11579: URL: https://github.com/apache/pinot/pull/11579#issuecomment-1716644245 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11579?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] codecov-commenter commented on pull request #11578: pre-configuration based assignment

2023-09-12 Thread via GitHub
codecov-commenter commented on PR #11578: URL: https://github.com/apache/pinot/pull/11578#issuecomment-1716641667 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11578?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] Jackie-Jiang commented on issue #11549: Does Pinot hybrid table allowed to be created with different schema

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11549: URL: https://github.com/apache/pinot/issues/11549#issuecomment-1716631325 No. I think we didn't enforce that before, but no longer allow such configuration going forward -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323686919 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndexSearcherPool.java: ## @@ -0,0 +1,56 @@ +/**

[GitHub] [pinot] walterddr opened a new pull request, #11579: [multistage] default enable metadata manager for colocated join quickstart

2023-09-12 Thread via GitHub
walterddr opened a new pull request, #11579: URL: https://github.com/apache/pinot/pull/11579 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[GitHub] [pinot] zhtaoxiang closed pull request #11545: [multistage] (improve #11518) disable calcite implicit float to double cast

2023-09-12 Thread via GitHub
zhtaoxiang closed pull request #11545: [multistage] (improve #11518) disable calcite implicit float to double cast URL: https://github.com/apache/pinot/pull/11545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [pinot] jasperjiaguo opened a new pull request, #11578: pre-configuration based assignment

2023-09-12 Thread via GitHub
jasperjiaguo opened a new pull request, #11578: URL: https://github.com/apache/pinot/pull/11578 Assign instances based on a pre-configured instance assignment map -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11545: [multistage] (improve #11518) disable calcite implicit float to double cast

2023-09-12 Thread via GitHub
Jackie-Jiang commented on code in PR #11545: URL: https://github.com/apache/pinot/pull/11545#discussion_r1323618552 ## pinot-query-planner/src/main/java/org/apache/calcite/rex/PinotRexBuilder.java: ## @@ -0,0 +1,64 @@ +/** + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #11437: Add broker query event listener

2023-09-12 Thread via GitHub
tibrewalpratik17 commented on code in PR #11437: URL: https://github.com/apache/pinot/pull/11437#discussion_r1323605056 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -242,19 +246,23 @@ public BrokerResponse handleReques

[pinot] branch master updated: Fix direct memory OOM on broker (#11496)

2023-09-12 Thread siddteotia
This is an automated email from the ASF dual-hosted git repository. siddteotia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new e62a271221 Fix direct memory OOM on broker (

[GitHub] [pinot] siddharthteotia merged pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia merged PR #11496: URL: https://github.com/apache/pinot/pull/11496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pi

[GitHub] [pinot] ege-st commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
ege-st commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716467737 > > My concern is that we are trying to prove that the fix is working using tests/heap dump, etc. vs the restart will just work. > > Let me elaborate a bit on the nature of problem w

[GitHub] [pinot] soumitra-st commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
soumitra-st commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716461492 > @soumitra-st @ege-st - I hope this gives some insight into where we are coming from. We can also chat offline and align if need be I agree with the operational toil and thanks

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323586218 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java: ## @@ -109,27 +112,43 @@ public Immut

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323585652 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndexSearcherPool.java: ## @@ -0,0 +1,56 @@ +/**

[GitHub] [pinot] Jackie-Jiang closed issue #11543: Queries that involve string timestamps do not work properly unless the string is singled quoted

2023-09-12 Thread via GitHub
Jackie-Jiang closed issue #11543: Queries that involve string timestamps do not work properly unless the string is singled quoted URL: https://github.com/apache/pinot/issues/11543 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [pinot] Jackie-Jiang commented on issue #11543: Queries that involve string timestamps do not work properly unless the string is singled quoted

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11543: URL: https://github.com/apache/pinot/issues/11543#issuecomment-1716448923 This is incorrect. `2023-08-31-23` will be parsed as `2023 - 8 - 31 - 23` (minus) ==> `1961`. The query itself parses but is semantically wrong. String literal must be quoted

[GitHub] [pinot] Jackie-Jiang closed issue #11541: Pulsar Stream Metadata not being written to Pinot

2023-09-12 Thread via GitHub
Jackie-Jiang closed issue #11541: Pulsar Stream Metadata not being written to Pinot URL: https://github.com/apache/pinot/issues/11541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [pinot] Jackie-Jiang merged pull request #11534: do not emit lag once tabledatamanager shutdown

2023-09-12 Thread via GitHub
Jackie-Jiang merged PR #11534: URL: https://github.com/apache/pinot/pull/11534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[pinot] branch master updated: do not emit lag once tabledatamanager shutdown (#11534)

2023-09-12 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 30e29abe60 do not emit lag once tabledatamanager

[GitHub] [pinot] siddharthteotia commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716427968 > My concern is that we are trying to prove that the fix is working using tests/heap dump, etc. vs the restart will just work. Let me elaborate a bit on the nature of probl

[GitHub] [pinot] ege-st commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
ege-st commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323558463 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under o

[GitHub] [pinot] deemoliu commented on pull request #11346: Add hyperLogLogPlus aggregation function for distinct count

2023-09-12 Thread via GitHub
deemoliu commented on PR #11346: URL: https://github.com/apache/pinot/pull/11346#issuecomment-1716413409 Test Query ``` SELECT DISTINCTCOUNTHLLPLUS(some_id, 12) FROM test_pinot_table where time_ms >= 169438000 time_ms <= 169438200 and filter1 > 0

[GitHub] [pinot] siddharthteotia commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716388980 > Thanks @jasperjiaguo for you comments! > > > Could you eleborate your concern here? I think the tests/heap dump/graphs show that we recover deterministically and the direc

[pinot] branch master updated (bc0d9fb659 -> c254e393ef)

2023-09-12 Thread snlee
This is an automated email from the ASF dual-hosted git repository. snlee pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from bc0d9fb659 Fail reload if derived columns can't be created (#11559) add c254e393ef Keep header parsing logic the sa

[GitHub] [pinot] snleee merged pull request #11540: Keep header parsing logic the same in default CSVRecordReader and the line iterator

2023-09-12 Thread via GitHub
snleee merged PR #11540: URL: https://github.com/apache/pinot/pull/11540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[GitHub] [pinot] soumitra-st commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
soumitra-st commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716370562 Thanks @jasperjiaguo for you comments! > Could you eleborate your concern here? I think the tests/heap dump/graphs show that we recover deterministically and the direct buffers

[GitHub] [pinot] ege-st commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
ege-st commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716354398 > > has less unknowns > > Could you eleborate your concern here? I think the tests/heap dump/graphs show that we recover deterministically and the direct buffers are deallocated.

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
soumitra-st commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323505609 ## pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java: ## @@ -321,6 +322,10 @@ private void sendResponse(ChannelHandlerContext ctx

[GitHub] [pinot] ege-st commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
ege-st commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323504740 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under o

[GitHub] [pinot] jasperjiaguo commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
jasperjiaguo commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716347496 > Sorry for starting a new thread here, but while we are trying to "handle" OOM by resetting the netty channels, queries may still fail (I think). So, System.exit can be called inste

[GitHub] [pinot] ege-st commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
ege-st commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323502750 ## pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java: ## @@ -321,6 +322,10 @@ private void sendResponse(ChannelHandlerContext ctx, St

[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
jasperjiaguo commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323501260 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11437: Add broker query event listener

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11437: URL: https://github.com/apache/pinot/pull/11437#discussion_r1323500172 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -242,19 +246,23 @@ public BrokerResponse handleRequest(Js

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11437: Add broker query event listener

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11437: URL: https://github.com/apache/pinot/pull/11437#discussion_r1323500172 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -242,19 +246,23 @@ public BrokerResponse handleRequest(Js

[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
jasperjiaguo commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323497068 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,95 @@ +/** + * Licensed to the Apache Software Foundation (ASF) un

[GitHub] [pinot] shwin commented on issue #11519: Log when creating all index types

2023-09-12 Thread via GitHub
shwin commented on issue #11519: URL: https://github.com/apache/pinot/issues/11519#issuecomment-1716295505 @Jackie-Jiang yeah I'd be happy to contribute to this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [pinot] Jackie-Jiang commented on pull request #11566: Support up to 15 boolean expressions for casewhen scalar function

2023-09-12 Thread via GitHub
Jackie-Jiang commented on PR #11566: URL: https://github.com/apache/pinot/pull/11566#issuecomment-1716293076 @snleee When else is not configured, we should just return `null` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [pinot] codecov-commenter commented on pull request #11577: [multistage] Add ROWS support for aggregation window functions with ORDER BY within OVER

2023-09-12 Thread via GitHub
codecov-commenter commented on PR #11577: URL: https://github.com/apache/pinot/pull/11577#issuecomment-1716291672 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11577?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] somandal opened a new pull request, #11577: [multistage] Add ROWS support for aggregation window functions with ORDER BY within OVER

2023-09-12 Thread via GitHub
somandal opened a new pull request, #11577: URL: https://github.com/apache/pinot/pull/11577 This PR adds support for `ROWS` type window frames for aggregation window functions with an `ORDER BY` clause within the `OVER()`. The default frame type for `ORDER BY` within `OVER()` is `RANGE BETW

[GitHub] [pinot] rajagopr commented on a diff in pull request #11540: Keep header parsing logic the same in default CSVRecordReader and the line iterator

2023-09-12 Thread via GitHub
rajagopr commented on code in PR #11540: URL: https://github.com/apache/pinot/pull/11540#discussion_r1323399278 ## pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReader.java: ## @@ -104,50 +103,42 @@ public void init(Fil

[GitHub] [pinot] Jackie-Jiang closed issue #11531: flaky SimpleMinionClusterIntegrationTest

2023-09-12 Thread via GitHub
Jackie-Jiang closed issue #11531: flaky SimpleMinionClusterIntegrationTest URL: https://github.com/apache/pinot/issues/11531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [pinot] Jackie-Jiang commented on issue #11531: flaky SimpleMinionClusterIntegrationTest

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11531: URL: https://github.com/apache/pinot/issues/11531#issuecomment-1716173056 This is caused by a Helix bug. Reverted Helix version in #11560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [pinot] Jackie-Jiang commented on issue #11529: Reloading segments does not update LUCENE vs NATIVE text indices

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11529: URL: https://github.com/apache/pinot/issues/11529#issuecomment-1716167677 Might be related to #11520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [pinot] soumitra-st commented on pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
soumitra-st commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1716166853 Sorry for starting a new thread here, but while we are trying to "handle" OOM by resetting the netty channels, queries may still fail (I think). So, System.exit can be called instead,

[GitHub] [pinot] Jackie-Jiang commented on issue #11520: Pinot minions not building NATIVE indices

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11520: URL: https://github.com/apache/pinot/issues/11520#issuecomment-1716165848 @gortiz Can you help take a look? I took a quick look and seems `TextIndexType.getFileExtension()` always returns lucene index extension. Not sure if that is related -- This

[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
jasperjiaguo commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323362776 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,95 @@ +/** + * Licensed to the Apache Software Foundation (ASF) un

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11556: Add argument check for aggregation functions with single argument

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11556: URL: https://github.com/apache/pinot/pull/11556#discussion_r1323359088 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/BooleanOrAggregationFunction.java: ## @@ -19,18 +19,21 @@ package org.apache.pin

[GitHub] [pinot] Jackie-Jiang commented on issue #11519: Log when creating all index types

2023-09-12 Thread via GitHub
Jackie-Jiang commented on issue #11519: URL: https://github.com/apache/pinot/issues/11519#issuecomment-1716139742 @shwin Good suggestion. Would you like to contribute this? We can go over all the index creators and log some useful stats when an index is created -- This is an automated mes

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323331770 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneDocIdCollector.java: ## @@ -60,9 +61,16 @@ public voi

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11575: Add metrics to export netty direct memory used and max

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11575: URL: https://github.com/apache/pinot/pull/11575#discussion_r1323344023 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java: ## @@ -46,6 +47,26 @@ public enum ServerGauge implements AbstractMetrics.Gauge {

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11575: Add metrics to export netty direct memory used and max

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11575: URL: https://github.com/apache/pinot/pull/11575#discussion_r1323343206 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java: ## @@ -46,6 +47,26 @@ public enum ServerGauge implements AbstractMetrics.Gauge {

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323340895 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,95 @@ +/** + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323331770 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneDocIdCollector.java: ## @@ -60,9 +61,16 @@ public voi

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11575: Add metrics to export netty direct memory used and max

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11575: URL: https://github.com/apache/pinot/pull/11575#discussion_r1323334725 ## pinot-core/src/main/java/org/apache/pinot/core/transport/ServerChannels.java: ## @@ -151,7 +153,24 @@ private class ServerChannel { ServerChannel(Serv

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323332979 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java: ## @@ -109,27 +112,43 @@ public Immut

[GitHub] [pinot] itschrispeck commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
itschrispeck commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323331770 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneDocIdCollector.java: ## @@ -60,9 +61,16 @@ public voi

[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #11437: Add broker query event listener

2023-09-12 Thread via GitHub
tibrewalpratik17 commented on code in PR #11437: URL: https://github.com/apache/pinot/pull/11437#discussion_r1323318749 ## pinot-spi/src/main/java/org/apache/pinot/spi/eventlistener/query/PinotBrokerQueryEventListenerUtils.java: ## @@ -0,0 +1,105 @@ +/** + * Licensed to the Apac

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323316058 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323315675 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323314926 ## pinot-core/src/main/java/org/apache/pinot/core/transport/InstanceRequestHandler.java: ## @@ -321,6 +322,10 @@ private void sendResponse(ChannelHandlerContext

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323309261 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
siddharthteotia commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323308080 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11501: Change the Forbidden error to Unauthorized

2023-09-12 Thread via GitHub
soumitra-st commented on code in PR #11501: URL: https://github.com/apache/pinot/pull/11501#discussion_r1323288750 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotBrokerDebug.java: ## @@ -158,7 +158,7 @@ public Map> getRoutingTableForQuery( if (!

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11501: Change the Forbidden error to Unauthorized

2023-09-12 Thread via GitHub
soumitra-st commented on code in PR #11501: URL: https://github.com/apache/pinot/pull/11501#discussion_r1323288750 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotBrokerDebug.java: ## @@ -158,7 +158,7 @@ public Map> getRoutingTableForQuery( if (!

[GitHub] [pinot] soumitra-st commented on a diff in pull request #11501: Change the Forbidden error to Unauthorized

2023-09-12 Thread via GitHub
soumitra-st commented on code in PR #11501: URL: https://github.com/apache/pinot/pull/11501#discussion_r1323288750 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotBrokerDebug.java: ## @@ -158,7 +158,7 @@ public Map> getRoutingTableForQuery( if (!

[GitHub] [pinot] vvivekiyer commented on a diff in pull request #11496: Fix direct memory OOM on broker

2023-09-12 Thread via GitHub
vvivekiyer commented on code in PR #11496: URL: https://github.com/apache/pinot/pull/11496#discussion_r1323279081 ## pinot-core/src/main/java/org/apache/pinot/core/transport/DirectOOMHandler.java: ## @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [pinot] soumitra-st commented on issue #11483: HTTP 1.1 authentication challenge

2023-09-12 Thread via GitHub
soumitra-st commented on issue #11483: URL: https://github.com/apache/pinot/issues/11483#issuecomment-1716030378 @jvenant , Great find! What is the AccessControlFactory set to in the test environment? [Broker request filter](https://github.com/apache/pinot/blob/61dcea6b71a99746805bc7f322dae

[GitHub] [pinot] ege-st commented on a diff in pull request #11576: Add table indexes API

2023-09-12 Thread via GitHub
ege-st commented on code in PR #11576: URL: https://github.com/apache/pinot/pull/11576#discussion_r1323257805 ## pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java: ## @@ -285,15 +289,60 @@ public String getSegmentMetadata( return ResourceU

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11561: Decouple metrics Key , client ID in stream consumer and add identifiers for kafka steam clientID to prevent mBean already exis

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11561: URL: https://github.com/apache/pinot/pull/11561#discussion_r1323172892 ## pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaConsumerFactory.java: ## @@ -25,18 +25,22 @@ p

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11561: Decouple metrics Key , client ID in stream consumer and add identifiers for kafka steam clientID to prevent mBean already exis

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11561: URL: https://github.com/apache/pinot/pull/11561#discussion_r1323170830 ## pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionGroupMetadataFetcher.java: ## @@ -63,8 +63,9 @@ public Exception getException() { @Override p

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11437: Add broker query event listener

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11437: URL: https://github.com/apache/pinot/pull/11437#discussion_r1323167542 ## pinot-spi/src/main/java/org/apache/pinot/spi/eventlistener/query/PinotBrokerQueryEventListenerUtils.java: ## @@ -0,0 +1,105 @@ +/** + * Licensed to the Apache S

[GitHub] [pinot] ankitsultana commented on a diff in pull request #11558: Handle interrupt gracefully for Lucene mutable index

2023-09-12 Thread via GitHub
ankitsultana commented on code in PR #11558: URL: https://github.com/apache/pinot/pull/11558#discussion_r1323163212 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java: ## @@ -109,27 +112,46 @@ public Immut

[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #11330: [multistage] Fixing pinot client resolve table name

2023-09-12 Thread via GitHub
tibrewalpratik17 commented on code in PR #11330: URL: https://github.com/apache/pinot/pull/11330#discussion_r1322951203 ## pinot-common/src/main/java/org/apache/pinot/common/utils/request/RequestUtils.java: ## @@ -290,4 +302,52 @@ public static Map getOptionsFromJson(JsonNode r

[GitHub] [pinot] codecov-commenter commented on pull request #11575: Add metrics to export netty direct memory used and max

2023-09-12 Thread via GitHub
codecov-commenter commented on PR #11575: URL: https://github.com/apache/pinot/pull/11575#issuecomment-1715599971 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11575?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] codecov-commenter commented on pull request #11576: Add table indexes API

2023-09-12 Thread via GitHub
codecov-commenter commented on PR #11576: URL: https://github.com/apache/pinot/pull/11576#issuecomment-1715599000 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11576?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

  1   2   >