[GitHub] [pinot] codecov-commenter commented on pull request #11717: [Flaky-test] Fix TableCacheTest

codecov-commenter commented on PR #11717: URL: https://github.com/apache/pinot/pull/11717#issuecomment-1741690131 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11717?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11715: Reduce cpu usage by avoiding throwing an exception during query execution

Jackie-Jiang commented on code in PR #11715: URL: https://github.com/apache/pinot/pull/11715#discussion_r1341915681 ## pinot-core/src/main/java/org/apache/pinot/core/operator/combine/CombineOperatorUtils.java: ## @@ -48,15 +48,12 @@ public static void setExecutionStatistics(Base

[GitHub] [pinot] Jackie-Jiang commented on pull request #11711: Post build index creation

Jackie-Jiang commented on PR #11711: URL: https://github.com/apache/pinot/pull/11711#issuecomment-1741687994 cc @gortiz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [pinot] Jackie-Jiang opened a new issue, #11718: [Flaky-test] StarTreeClusterIntegrationTest.testGeneratedQueries

Jackie-Jiang opened a new issue, #11718: URL: https://github.com/apache/pinot/issues/11718 ``` Error: Failures: Error: StarTreeClusterIntegrationTest.testGeneratedQueries:163->testStarQuery:185 Query comparison failed for: Star Query: SELECT avg(ArrDel15) FROM mytable WHERE D

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

Jackie-Jiang commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341914079 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -362,6 +365,11 @@ public static class QueryOptionKey { public static fin

[pinot] branch master updated: Add more test for broker jersey bounded thread pool (#11705)

This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ae1681273c Add more test for broker jersey bounde

[GitHub] [pinot] Jackie-Jiang merged pull request #11705: Add more test for broker jersey bounded thread pool

Jackie-Jiang merged PR #11705: URL: https://github.com/apache/pinot/pull/11705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11717: [Flaky-test] Fix TableCacheTest

Jackie-Jiang opened a new pull request, #11717: URL: https://github.com/apache/pinot/pull/11717 Fix #11435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [pinot] codecov-commenter commented on pull request #11716: Use string to represent BigDecimal response

codecov-commenter commented on PR #11716: URL: https://github.com/apache/pinot/pull/11716#issuecomment-1741607104 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11716?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[pinot] branch master updated (82140f1177 -> dcffc6dea1)

This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 82140f1177 JSON index: Add support for ignoring values longer than a given length. (#11604) add dcffc6dea1 Gapfil

[GitHub] [pinot] Jackie-Jiang merged pull request #11714: Gapfill: Fix bug with SumAvgGapfillProcessor.

Jackie-Jiang merged PR #11714: URL: https://github.com/apache/pinot/pull/11714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] abhioncbr commented on a diff in pull request #11653: ScalarTransformFunctionWrapper NULL support

abhioncbr commented on code in PR #11653: URL: https://github.com/apache/pinot/pull/11653#discussion_r1341865991 ## pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java: ## @@ -297,6 +297,7 @@ public DataType toDataType() { case TIMESTAMP: c

[GitHub] [pinot] Jackie-Jiang commented on pull request #11716: Use string to represent BigDecimal response

Jackie-Jiang commented on PR #11716: URL: https://github.com/apache/pinot/pull/11716#issuecomment-1741594165 This issue was introduced in #8503 where responses from aggregation are stored as string, responses from other queries are stored as `BigDecimal`; #11453 changed it to always store a

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11716: Use string to represent BigDecimal response

Jackie-Jiang opened a new pull request, #11716: URL: https://github.com/apache/pinot/pull/11716 It is best practice to store `BigDecimal` as string in JSON to prevent losing precision. Currently we directly serialize `BigDecimal` into number (without quote) in the query response, but `Js

[GitHub] [pinot] abhioncbr commented on a diff in pull request #11653: ScalarTransformFunctionWrapper NULL support

abhioncbr commented on code in PR #11653: URL: https://github.com/apache/pinot/pull/11653#discussion_r1341862027 ## pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapperTest.java: ## @@ -262,6 +262,27 @@ public void testIsNullO

[GitHub] [pinot] walterddr closed pull request #11686: [multistage] support legacy query function "lookup" in multi-stage

walterddr closed pull request #11686: [multistage] support legacy query function "lookup" in multi-stage URL: https://github.com/apache/pinot/pull/11686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [pinot] walterddr commented on pull request #11686: [multistage] support legacy query function "lookup" in multi-stage

walterddr commented on PR #11686: URL: https://github.com/apache/pinot/pull/11686#issuecomment-1741579551 closing as it is not a full support fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [pinot] codecov-commenter commented on pull request #11715: Reduce cpu usage by avoiding throwing an exception during query execution

codecov-commenter commented on PR #11715: URL: https://github.com/apache/pinot/pull/11715#issuecomment-1741574245 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11715?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] vvivekiyer opened a new pull request, #11715: Reduce cpu usage by avoiding throwing an exception during query execution

vvivekiyer opened a new pull request, #11715: URL: https://github.com/apache/pinot/pull/11715 In #9092, we added query execution stats to fetch setNumConsumingSegmentsProcessed and setNumConsumingSegmentsMatched In one of our production clusters, we observed that `Operator.getIndexSe

[GitHub] [pinot] codecov-commenter commented on pull request #11714: Gapfill: Fix bug with SumAvgGapfillProcessor.

codecov-commenter commented on PR #11714: URL: https://github.com/apache/pinot/pull/11714#issuecomment-1741493424 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11714?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] ege-st commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

ege-st commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341787880 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -170,15 +171,18 @@ public BaseBrokerRequestHandler(PinotConfigu

[GitHub] [pinot] ege-st commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

ege-st commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341781521 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[] processQueryAndSerialize(ServerQueryRequ

[GitHub] [pinot] ege-st commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

ege-st commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341779996 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -318,6 +318,9 @@ public static class Broker { "pinot.broker.enable.partition.m

[GitHub] [pinot] weixiangsun opened a new pull request, #11714: Gapfill: Fix bug with SumAvgGapfillProcessor.

weixiangsun opened a new pull request, #11714: URL: https://github.com/apache/pinot/pull/11714 This is bugfix for Gapfill SumAvgGapfillProcessor. The bug is that SumAvgGapfillProcessor is not using the correct previous value before doing the gapfill. The value after the gapfill range should

[GitHub] [pinot] zhtaoxiang commented on pull request #11711: Post build index creation

zhtaoxiang commented on PR #11711: URL: https://github.com/apache/pinot/pull/11711#issuecomment-1741368196 I approved the PR by accident 😛 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [pinot] zhtaoxiang commented on a diff in pull request #11711: Post build index creation

zhtaoxiang commented on code in PR #11711: URL: https://github.com/apache/pinot/pull/11711#discussion_r1341714287 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java: ## @@ -313,6 +325,36 @@ private void ha

[GitHub] [pinot] Jackie-Jiang closed issue #11445: [multistage] infer query runner host from broker or server hostname

Jackie-Jiang closed issue #11445: [multistage] infer query runner host from broker or server hostname URL: https://github.com/apache/pinot/issues/11445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341681228 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[] processQueryAndSerialize(Server

[GitHub] [pinot] siddharthteotia commented on pull request #11575: Add metrics to export netty direct memory used and max

siddharthteotia commented on PR #11575: URL: https://github.com/apache/pinot/pull/11575#issuecomment-1741307162 I think @Jackie-Jiang has a comment. I am good with merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [pinot] siddharthteotia commented on pull request #11575: Add metrics to export netty direct memory used and max

siddharthteotia commented on PR #11575: URL: https://github.com/apache/pinot/pull/11575#issuecomment-1741305937 > But maybe what do you mean is that in some place in this PR I'm not calling the version with the supplier and therefore the value never change. It wouldn't be strange given that

[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

jasperjiaguo commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341666747 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[] processQueryAndSerialize(ServerQue

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341664954 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -1654,6 +1669,24 @@ private long setQueryTimeout(Strin

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341663123 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -362,6 +365,11 @@ public static class QueryOptionKey { public static

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341663319 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -318,6 +318,9 @@ public static class Broker { "pinot.broker.enable.pa

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341663319 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -318,6 +318,9 @@ public static class Broker { "pinot.broker.enable.pa

[GitHub] [pinot] vvivekiyer commented on pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

vvivekiyer commented on PR #11710: URL: https://github.com/apache/pinot/pull/11710#issuecomment-1741230828 > How will a user determine what the threshold should be? @ege-st Good question. We did think about this. The ideal answer is that the value is use case dependent. Howeve

[GitHub] [pinot] chenboat merged pull request #11604: JSON index: Add support for ignoring values longer than a given length.

chenboat merged PR #11604: URL: https://github.com/apache/pinot/pull/11604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

[pinot] branch master updated: JSON index: Add support for ignoring values longer than a given length. (#11604)

This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 82140f1177 JSON index: Add support for ignoring

[GitHub] [pinot] ege-st commented on pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

ege-st commented on PR #11710: URL: https://github.com/apache/pinot/pull/11710#issuecomment-1740980503 Cool, thanks! I think reducing the likelyhood of DM OOMs is the logical next step. How will a user determine what the threshold should be? A critical requirement for a feature like

[GitHub] [pinot] codecov-commenter commented on pull request #11713: Adding a metric to measure total thread cpu time for a table

codecov-commenter commented on PR #11713: URL: https://github.com/apache/pinot/pull/11713#issuecomment-1740964085 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11713?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] swaminathanmanish commented on a diff in pull request #11712: [DO NOT MERGE] Add a test to prove how to use protobuf nullability

swaminathanmanish commented on code in PR #11712: URL: https://github.com/apache/pinot/pull/11712#discussion_r1341419664 ## pinot-plugins/pinot-input-format/pinot-protobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordExtractor.java: ## @@ -62,7 +62,7

[GitHub] [pinot] mcvsubbu opened a new pull request, #11713: Adding a metric to measure total thread cpu time for a table

mcvsubbu opened a new pull request, #11713: URL: https://github.com/apache/pinot/pull/11713 Allows us to compute the total thread cpu time each table uses for queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[GitHub] [pinot] codecov-commenter commented on pull request #11712: [DO NOT MERGE] Add a test to prove how to use protobuf nullability

codecov-commenter commented on PR #11712: URL: https://github.com/apache/pinot/pull/11712#issuecomment-1740862007 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11712?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] gortiz opened a new pull request, #11712: [DO NOT MERGE] Add a test to prove how to use protobuf nullability

gortiz opened a new pull request, #11712: URL: https://github.com/apache/pinot/pull/11712 This PR adds a test to verify the correct nullability read on `ProtoBufRecordExtractor`. The PR is not yet ready to merge because for some reason the descriptor is using v2 instead of 3.

[GitHub] [pinot] codecov-commenter commented on pull request #11711: Post build index creation

codecov-commenter commented on PR #11711: URL: https://github.com/apache/pinot/pull/11711#issuecomment-1740701803 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11711?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] saurabhd336 opened a new pull request, #11711: Post build index creation

saurabhd336 opened a new pull request, #11711: URL: https://github.com/apache/pinot/pull/11711 It may be desirable for some index types to be build post segment file has been created. This PR allows for that. -- This is an automated message from the Apache Git Service. To respond to the m