[GitHub] [pinot] eaugene commented on issue #10816: Segment MetaData mismatch between API's

2023-06-01 Thread via GitHub
eaugene commented on issue #10816: URL: https://github.com/apache/pinot/issues/10816#issuecomment-1573182830 Thanks @Jackie-Jiang for the clarification. Should we enhance the first API to have a different naming convention for this property, say "segmentCommitedTime" to make it more explic

[GitHub] [pinot] eaugene opened a new issue, #10832: option(SkipUpsert=true) is not case inSensitive

2023-06-01 Thread via GitHub
eaugene opened a new issue, #10832: URL: https://github.com/apache/pinot/issues/10832 Using `option(SkipUpsert=true)` in Query works by including upserted records, while `option(skipUpsert=true)` doesn't work as intended, also is silent and doesn't throw any error when processing query.

[pinot] branch master updated: Add required JSON annotation in H3IndexResolution (#10792)

2023-06-01 Thread yupeng
This is an automated email from the ASF dual-hosted git repository. yupeng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 60ab652e32 Add required JSON annotation in H3Inde

[GitHub] [pinot] yupeng9 merged pull request #10792: Add required JSON annotation in H3IndexResolution

2023-06-01 Thread via GitHub
yupeng9 merged PR #10792: URL: https://github.com/apache/pinot/pull/10792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[GitHub] [pinot] yupeng9 commented on a diff in pull request #10792: Add required JSON annotation in H3IndexResolution

2023-06-01 Thread via GitHub
yupeng9 commented on code in PR #10792: URL: https://github.com/apache/pinot/pull/10792#discussion_r1213866750 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/H3IndexResolution.java: ## @@ -91,4 +94,11 @@ public boolean equals(Object o) { public i

[GitHub] [pinot] abhioncbr commented on pull request #10814: float-double-comparison: updated for avoiding right cast transformation incase of float…

2023-06-01 Thread via GitHub
abhioncbr commented on PR #10814: URL: https://github.com/apache/pinot/pull/10814#issuecomment-1573065703 Thanks for the help @Jackie-Jiang for the right pointers. I figured out how we can handle the `CAST` function addition based on the data type precision -- This is an automated mess

[GitHub] [pinot] codecov-commenter commented on pull request #10831: [multisage][perf] do not canonicalize row when schema exactly matches

2023-06-01 Thread via GitHub
codecov-commenter commented on PR #10831: URL: https://github.com/apache/pinot/pull/10831#issuecomment-1573008954 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10831?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] codecov-commenter commented on pull request #10830: (WIP..NOT READY) Adding LazyReader for lazy initialization of record readers.

2023-06-01 Thread via GitHub
codecov-commenter commented on PR #10830: URL: https://github.com/apache/pinot/pull/10830#issuecomment-1572966576 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10830?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] walterddr commented on a diff in pull request #10790: [multistage]: Changes for supporting Json scalar functions.

2023-06-01 Thread via GitHub
walterddr commented on code in PR #10790: URL: https://github.com/apache/pinot/pull/10790#discussion_r1213791510 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java: ## @@ -308,13 +309,16 @@ private static boolean

[GitHub] [pinot] walterddr closed pull request #10756: Refactor usage of WorkerMetadata and StageMetadata

2023-06-01 Thread via GitHub
walterddr closed pull request #10756: Refactor usage of WorkerMetadata and StageMetadata URL: https://github.com/apache/pinot/pull/10756 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [pinot] walterddr commented on pull request #10756: Refactor usage of WorkerMetadata and StageMetadata

2023-06-01 Thread via GitHub
walterddr commented on PR #10756: URL: https://github.com/apache/pinot/pull/10756#issuecomment-1572946323 this can be closed as covered by https://github.com/apache/pinot/pull/10791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [pinot] walterddr opened a new pull request, #10831: [multisage][perf] do not canonicalize row when schema exactly matches

2023-06-01 Thread via GitHub
walterddr opened a new pull request, #10831: URL: https://github.com/apache/pinot/pull/10831 we do not need to canonicalize row-by-row when v1 and v2 schema exactly matches (and requires no conversion) -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [pinot] swaminathanmanish opened a new pull request, #10830: (WIP..NOT READY) Adding LazyReader for lazy initialization of record readers.

2023-06-01 Thread via GitHub
swaminathanmanish opened a new pull request, #10830: URL: https://github.com/apache/pinot/pull/10830 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[GitHub] [pinot] Jackie-Jiang commented on issue #10816: Segment MetaData mismatch between API's

2023-06-01 Thread via GitHub
Jackie-Jiang commented on issue #10816: URL: https://github.com/apache/pinot/issues/10816#issuecomment-1572809481 The first metadata is returned from the server local metadata; the second metadata is returned from the ZK. The creation time for the server local metadata is when the segment

[GitHub] [pinot] Jackie-Jiang commented on issue #10753: Wrong error message for incorrect column name during lookup

2023-06-01 Thread via GitHub
Jackie-Jiang commented on issue #10753: URL: https://github.com/apache/pinot/issues/10753#issuecomment-1572801823 This warning is appended because these segments do not contain the column asked, thus being pruned before query execution. `QueryExecutionError` is reported when an error occurr

[GitHub] [pinot] Jackie-Jiang commented on pull request #10814: float-double-comparison: updated for avoiding right cast transformation incase of float…

2023-06-01 Thread via GitHub
Jackie-Jiang commented on PR #10814: URL: https://github.com/apache/pinot/pull/10814#issuecomment-1572787445 cc @walterddr ^^ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [pinot] walterddr opened a new issue, #10829: [flaky test] query with pipeline breaker seems unstable

2023-06-01 Thread via GitHub
walterddr opened a new issue, #10829: URL: https://github.com/apache/pinot/issues/10829 see: https://github.com/apache/pinot/actions/runs/5147527036/jobs/9268030409 ``` [flaky test] 2023-06-01T18:30:47.9165106Z [ERROR] org.apache.pinot.query.runtime.queries.ResourceBasedQueriesTest

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10463: Segment compaction for upsert real-time tables

2023-06-01 Thread via GitHub
Jackie-Jiang commented on code in PR #10463: URL: https://github.com/apache/pinot/pull/10463#discussion_r1213629542 ## pinot-common/src/main/java/org/apache/pinot/common/utils/SegmentUtils.java: ## @@ -39,14 +39,30 @@ private SegmentUtils() { @Nullable public static Intege

[GitHub] [pinot] gortiz commented on a diff in pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
gortiz commented on code in PR #10528: URL: https://github.com/apache/pinot/pull/10528#discussion_r1213531610 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/MmapMemory.java: ## @@ -0,0 +1,348 @@ +/** + * Licensed to the Apache Software Foundation (

[GitHub] [pinot] npawar commented on a diff in pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
npawar commented on code in PR #10528: URL: https://github.com/apache/pinot/pull/10528#discussion_r1213524082 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/MmapMemory.java: ## @@ -0,0 +1,348 @@ +/** + * Licensed to the Apache Software Foundation (

[pinot] branch master updated: Keeps nullness attributes of merged in comparison column values (#10704)

2023-06-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 6dbd9e23fe Keeps nullness attributes of merged in

[GitHub] [pinot] Jackie-Jiang merged pull request #10704: Keeps nullness attributes of merged in comparison column values

2023-06-01 Thread via GitHub
Jackie-Jiang merged PR #10704: URL: https://github.com/apache/pinot/pull/10704 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] eaugene commented on pull request #10812: Enhancing Table Size API to return table size per replica

2023-06-01 Thread via GitHub
eaugene commented on PR #10812: URL: https://github.com/apache/pinot/pull/10812#issuecomment-1572223388 > Not required in the PR, but I feel, enhancing the api to also report the compressed size in deepstore would be quite useful. Sure @mayankshriv . Yes , this would be useful. Crea

[GitHub] [pinot] eaugene opened a new issue, #10828: Enhance TableSize API to return compressed size in deep store

2023-06-01 Thread via GitHub
eaugene opened a new issue, #10828: URL: https://github.com/apache/pinot/issues/10828 Carrying out this from: https://github.com/apache/pinot/pull/10812#pullrequestreview-1451921617 . It looks like a nice feature to have to see disk usage in the deep stores as well. Creating th

[GitHub] [pinot] eaugene commented on a diff in pull request #10812: Enhancing Table Size API to return table size per replica

2023-06-01 Thread via GitHub
eaugene commented on code in PR #10812: URL: https://github.com/apache/pinot/pull/10812#discussion_r1213287408 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/TableSizeReader.java: ## @@ -132,12 +144,19 @@ public TableSizeDetails getTableSizeDetails(@Nonnull

[GitHub] [pinot] eaugene commented on a diff in pull request #10812: Enhancing Table Size API to return table size per replica

2023-06-01 Thread via GitHub
eaugene commented on code in PR #10812: URL: https://github.com/apache/pinot/pull/10812#discussion_r1213281580 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/TableSizeReader.java: ## @@ -132,12 +144,19 @@ public TableSizeDetails getTableSizeDetails(@Nonnull

[GitHub] [pinot] gortiz commented on pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
gortiz commented on PR #10528: URL: https://github.com/apache/pinot/pull/10528#issuecomment-1572200315 I've decided to remove the `finalize` from `PinotDataBuffer` because it doesn't seem trivial to keep track on whether we actually need to release the memory or not, given that we may have

[GitHub] [pinot] eaugene commented on a diff in pull request #10812: Enhancing Table Size API to return table size per replica

2023-06-01 Thread via GitHub
eaugene commented on code in PR #10812: URL: https://github.com/apache/pinot/pull/10812#discussion_r1213188242 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/TableSizeReader.java: ## @@ -248,7 +279,7 @@ public TableSubTypeSizeDetails getTableSubtypeSize(Strin

[GitHub] [pinot] gortiz commented on issue #8837: `order by DESC limit N` queries are not optimised at segment level

2023-06-01 Thread via GitHub
gortiz commented on issue #8837: URL: https://github.com/apache/pinot/issues/8837#issuecomment-1571593283 That is the PR that originally had a partial implementation that optimized the order by desc, but we decided to drop these changes and only optimize the order by asc case because we wer

[GitHub] [pinot] gortiz commented on pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
gortiz commented on PR #10528: URL: https://github.com/apache/pinot/pull/10528#issuecomment-1571552536 > This question is not answered yet: With Java ByteBuffer and LBuffer, when the buffer is not explicitly closed, GC will eventually release the memory. Will the new unsafe buffer be releas

[GitHub] [pinot] gortiz commented on a diff in pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
gortiz commented on code in PR #10528: URL: https://github.com/apache/pinot/pull/10528#discussion_r1212713174 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/MmapMemory.java: ## @@ -0,0 +1,348 @@ +/** + * Licensed to the Apache Software Foundation (

[GitHub] [pinot] gortiz commented on a diff in pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
gortiz commented on code in PR #10528: URL: https://github.com/apache/pinot/pull/10528#discussion_r1212702603 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/PinotBufferFactory.java: ## @@ -0,0 +1,40 @@ +/** + * Licensed to the Apache Software Foundation (

[GitHub] [pinot] gortiz commented on a diff in pull request #10528: PinotBufferFactory and a buffer implementation that uses Unsafe

2023-06-01 Thread via GitHub
gortiz commented on code in PR #10528: URL: https://github.com/apache/pinot/pull/10528#discussion_r1212700823 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/unsafe/MmapMemory.java: ## @@ -0,0 +1,348 @@ +/** + * Licensed to the Apache Software Foundation (