[GitHub] [pinot] codecov-commenter commented on pull request #10930: [multistage] Do Not Log Entire Plan in QueryServer

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10930: URL: https://github.com/apache/pinot/pull/10930#issuecomment-1594177962 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] ankitsultana opened a new pull request, #10930: [multistage] Do Not Log Entire Plan in QueryServer

2023-06-15 Thread via GitHub
ankitsultana opened a new pull request, #10930: URL: https://github.com/apache/pinot/pull/10930 Saw this when fixing tpch tests. Logging entire plan can be dangerous in production scenarios (plans are usually large) cc: @walterddr -- This is an automated message from the Apache Gi

[GitHub] [pinot] eaugene commented on issue #10923: Is it possible to update the `retentionTimeValue` of a table after it's been created?

2023-06-15 Thread via GitHub
eaugene commented on issue #10923: URL: https://github.com/apache/pinot/issues/10923#issuecomment-1594138286 Yes, @travis-cook-sfdc. That is possible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[pinot] branch master updated: Change list.of to immutableList.of to solve java 8 compilation (#10929)

2023-06-15 Thread mcvsubbu
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 85bb1fab54 Change list.of to immutableList.of t

[GitHub] [pinot] mcvsubbu merged pull request #10929: Change list.of to immutableList.of to fix compilation in Java 8

2023-06-15 Thread via GitHub
mcvsubbu merged PR #10929: URL: https://github.com/apache/pinot/pull/10929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

[GitHub] [pinot] codecov-commenter commented on pull request #10929: Change list.of to immutableList.of to fix compilation in Java 8

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10929: URL: https://github.com/apache/pinot/pull/10929#issuecomment-1594007095 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10929?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] vvivekiyer opened a new pull request, #10929: Change list.of to immutableList.of to fix compilation in Java 8

2023-06-15 Thread via GitHub
vvivekiyer opened a new pull request, #10929: URL: https://github.com/apache/pinot/pull/10929 Fix test to make it java 8 compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [pinot] chenboat commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-06-15 Thread via GitHub
chenboat commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1231686445 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -395,6 +402,18 @@ protected void finishOperat

[GitHub] [pinot] chenboat commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-06-15 Thread via GitHub
chenboat commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1231681071 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java: ## @@ -55,12 +64,17 @@ public class Concur

[GitHub] [pinot] chenboat commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-06-15 Thread via GitHub
chenboat commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1231680171 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java: ## @@ -243,6 +295,47 @@ protected GenericR

[GitHub] [pinot] chenboat commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-06-15 Thread via GitHub
chenboat commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1231677527 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java: ## @@ -181,6 +210,29 @@ protected void rem

[GitHub] [pinot] chenboat commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-06-15 Thread via GitHub
chenboat commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1231674612 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapPartitionUpsertMetadataManager.java: ## @@ -243,6 +295,47 @@ protected GenericR

[GitHub] [pinot] chenboat commented on a diff in pull request #10915: [Upsert TTL] Add Watermark for each partitions for Primary key cleanup

2023-06-15 Thread via GitHub
chenboat commented on code in PR #10915: URL: https://github.com/apache/pinot/pull/10915#discussion_r1231673523 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertTTLConfig.java: ## @@ -0,0 +1,76 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [pinot] codecov-commenter commented on pull request #10928: Take upsert snapshot when creating new consuming segment

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10928: URL: https://github.com/apache/pinot/pull/10928#issuecomment-1593894851 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10928?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] codecov-commenter commented on pull request #10927: [POC] improve disk read for partial upsert handler

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10927: URL: https://github.com/apache/pinot/pull/10927#issuecomment-1593889477 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10927?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] dario-liberman commented on pull request #10867: FUNNEL_COUNT Aggregation Function

2023-06-15 Thread via GitHub
dario-liberman commented on PR #10867: URL: https://github.com/apache/pinot/pull/10867#issuecomment-1593886945 Please find proposed documentation here: https://github.com/pinot-contrib/pinot-docs/pull/187 -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [pinot] mcvsubbu commented on issue #10452: Support delete record with upsert table

2023-06-15 Thread via GitHub
mcvsubbu commented on issue #10452: URL: https://github.com/apache/pinot/issues/10452#issuecomment-1593878626 > @Jackie-Jiang @npawar @yupeng9 @mcvsubbu Here is the updated design proposal for this issue - [design](https://docs.google.com/document/d/19s1AHCRjmqeVa0z_djYBTAt_meg6qWcUxwRO3y-G

[GitHub] [pinot] Jackie-Jiang commented on pull request #10928: Take upsert snapshot when creating new consuming segment

2023-06-15 Thread via GitHub
Jackie-Jiang commented on PR #10928: URL: https://github.com/apache/pinot/pull/10928#issuecomment-1593875061 cc @deemoliu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #10928: Take upsert snapshot when creating new consuming segment

2023-06-15 Thread via GitHub
Jackie-Jiang opened a new pull request, #10928: URL: https://github.com/apache/pinot/pull/10928 Fix #10800 - Track all the segments managed by the `BasePartitionUpsertMetadataManager` - Remove the tracking for replaced segments because that can be replaced by tracking all the segme

[GitHub] [pinot] deemoliu opened a new pull request, #10927: [POC] improve disk read for partial upsert handler

2023-06-15 Thread via GitHub
deemoliu opened a new pull request, #10927: URL: https://github.com/apache/pinot/pull/10927 `refactor`: POC for improving disk read for partial upsert handler - The current Partial upsert handler read all columns of prev and new value. - To reduce num of column read, refactored the

[GitHub] [pinot] xiangfu0 commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
xiangfu0 commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593813274 Here are some takes from my side: High level principals: - CPU solution - KNN search has to be a distributed solution - The minimal search space is considered within one s

[GitHub] [pinot] xiangfu0 commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
xiangfu0 commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593802400 cc: @kkrugler -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [pinot] codecov-commenter commented on pull request #10926: [wip] Realtime pre-aggregation for Distinct Count HLL & Big Decimal

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10926: URL: https://github.com/apache/pinot/pull/10926#issuecomment-1593781079 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10926?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] codecov-commenter commented on pull request #10925: UI: Updated AuthProvider to extract user identity

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10925: URL: https://github.com/apache/pinot/pull/10925#issuecomment-1593716324 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10925?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] priyen opened a new pull request, #10926: [wip] Realtime pre-aggregation for Distinct Count HLL & Big Decimal

2023-06-15 Thread via GitHub
priyen opened a new pull request, #10926: URL: https://github.com/apache/pinot/pull/10926 [wip] Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incomp

[GitHub] [pinot] priyen closed pull request #9883: Distinct Count HLL pre-aggregation in realtime segments

2023-06-15 Thread via GitHub
priyen closed pull request #9883: Distinct Count HLL pre-aggregation in realtime segments URL: https://github.com/apache/pinot/pull/9883 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [pinot] joshigaurava opened a new pull request, #10925: UI: Updated AuthProvider to extract user identity

2023-06-15 Thread via GitHub
joshigaurava opened a new pull request, #10925: URL: https://github.com/apache/pinot/pull/10925 `ui` `feature` This change updates the AuthProvider logic to decode the access token and extract user name and email. This information will now be available in the app for features to cons

[pinot] branch master updated: [Multi-stage] Support partition based colocated join (#10886)

2023-06-15 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new f1966d9fa0 [Multi-stage] Support partition based

[GitHub] [pinot] Jackie-Jiang merged pull request #10886: [Multi-stage] Support partition based colocated join

2023-06-15 Thread via GitHub
Jackie-Jiang merged PR #10886: URL: https://github.com/apache/pinot/pull/10886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

[GitHub] [pinot] lucifer4j opened a new issue, #10924: Implementing PinotClientTransport using Vert.x

2023-06-15 Thread via GitHub
lucifer4j opened a new issue, #10924: URL: https://github.com/apache/pinot/issues/10924 Hi! I are trying to implement the PinotClientTransport interface as a part of creating a Vert.x Client adapter for the Pinot client. There are two changes we have identified so far that can help move in

[GitHub] [pinot] jasperjiaguo commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
jasperjiaguo commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593539838 @Aravind-Suresh Exactly. I've also been using [llama_index](https://github.com/jerryjliu/llama_index) and langchain with chatgpt apis. I think one usability addition to this fea

[GitHub] [pinot] kishoreg commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
kishoreg commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593493167 cc @KKcorps who is also thinking about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [pinot] Aravind-Suresh commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
Aravind-Suresh commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593475379 Thanks for the inputs @siddharthteotia @jasperjiaguo - yes, given the high dimensionality of the embeddings (OpenAI-davinci embeddings are >12k in dimensions), it's practical

[GitHub] [pinot] jasperjiaguo commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
jasperjiaguo commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593447601 Recommendation systems and Language Model (LLM) applications often utilize high-dimensional vector spaces to represent complex data like user profiles or linguistic patterns. Si

[GitHub] [pinot] codecov-commenter commented on pull request #10922: Bump snappy-java from 1.1.8.2 to 1.1.10.1

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10922: URL: https://github.com/apache/pinot/pull/10922#issuecomment-1593436893 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10922?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] travis-cook-sfdc opened a new issue, #10923: Is it possible to update the `retentionTimeValue` of a table after it's been created?

2023-06-15 Thread via GitHub
travis-cook-sfdc opened a new issue, #10923: URL: https://github.com/apache/pinot/issues/10923 As far as I know, the best way to update the `retentionTimeValue` would be to drop the table, recreate it with a different time value and then rerun a metadataPush job, but that seems like a lot o

[GitHub] [pinot] dependabot[bot] opened a new pull request, #10922: Bump snappy-java from 1.1.8.2 to 1.1.10.1

2023-06-15 Thread via GitHub
dependabot[bot] opened a new pull request, #10922: URL: https://github.com/apache/pinot/pull/10922 Bumps [snappy-java](https://github.com/xerial/snappy-java) from 1.1.8.2 to 1.1.10.1. Release notes Sourced from https://github.com/xerial/snappy-java/releases";>snappy-java's release

[pinot] branch dependabot/maven/org.xerial.snappy-snappy-java-1.1.10.1 created (now 0d96cec1ac)

2023-06-15 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.xerial.snappy-snappy-java-1.1.10.1 in repository https://gitbox.apache.org/repos/asf/pinot.git at 0d96cec1ac Bump snappy-java from 1.1.8.2 to 1.1.10.1 No new revi

[GitHub] [pinot] ege-st opened a new issue, #10921: Improve Syntax Error Messages

2023-06-15 Thread via GitHub
ege-st opened a new issue, #10921: URL: https://github.com/apache/pinot/issues/10921 Syntax error messages are very hard to read. Usually, they are just stack traces from Calcite along with an HTTP Code 200. This creates a very confusing message for users: a stack trace tells the user that

[GitHub] [pinot] ege-st commented on issue #10907: Enabling Null Support after table creation results in incorrect results for `IS [NOT] NULL` expressions

2023-06-15 Thread via GitHub
ege-st commented on issue #10907: URL: https://github.com/apache/pinot/issues/10907#issuecomment-1593143788 > Ideally, we want a fail fast behavior for a NULL support enabled query over NULL support disabled segments. > > In reality, we don't fail fast: if a query is mis-configured, t

[GitHub] [pinot] abhioncbr commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
abhioncbr commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1593068427 This is interesting. +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [pinot] codecov-commenter commented on pull request #10920: Bump guava from 30.1.1-jre to 32.0.1-jre

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10920: URL: https://github.com/apache/pinot/pull/10920#issuecomment-1592996461 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10920?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] siddharthteotia commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
siddharthteotia commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1592973429 Would love to collaborate on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [pinot] siddharthteotia commented on issue #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
siddharthteotia commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-1592971805 Glad to see there are others thinking about this as well. I had recently created a short internal proposal on why a case can be made for vector storage and indexing in

[GitHub] [pinot] dependabot[bot] closed pull request #10913: Bump guava from 30.1.1-jre to 32.0.0-jre

2023-06-15 Thread via GitHub
dependabot[bot] closed pull request #10913: Bump guava from 30.1.1-jre to 32.0.0-jre URL: https://github.com/apache/pinot/pull/10913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[pinot] branch dependabot/maven/com.google.guava-guava-32.0.0-jre deleted (was b1fc1540e0)

2023-06-15 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.guava-guava-32.0.0-jre in repository https://gitbox.apache.org/repos/asf/pinot.git was b1fc1540e0 Bump guava from 30.1.1-jre to 32.0.0-jre The revisions tha

[GitHub] [pinot] dependabot[bot] commented on pull request #10913: Bump guava from 30.1.1-jre to 32.0.0-jre

2023-06-15 Thread via GitHub
dependabot[bot] commented on PR #10913: URL: https://github.com/apache/pinot/pull/10913#issuecomment-1592931585 Superseded by #10920. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[pinot] branch dependabot/maven/com.google.guava-guava-32.0.1-jre created (now 15d01ee44a)

2023-06-15 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.guava-guava-32.0.1-jre in repository https://gitbox.apache.org/repos/asf/pinot.git at 15d01ee44a Bump guava from 30.1.1-jre to 32.0.1-jre No new revisions

[GitHub] [pinot] dependabot[bot] opened a new pull request, #10920: Bump guava from 30.1.1-jre to 32.0.1-jre

2023-06-15 Thread via GitHub
dependabot[bot] opened a new pull request, #10920: URL: https://github.com/apache/pinot/pull/10920 Bumps [guava](https://github.com/google/guava) from 30.1.1-jre to 32.0.1-jre. Release notes Sourced from https://github.com/google/guava/releases";>guava's releases. 32.0.1 M

[GitHub] [pinot] Aravind-Suresh opened a new issue, #10919: Vector embeddings support in Pinot

2023-06-15 Thread via GitHub
Aravind-Suresh opened a new issue, #10919: URL: https://github.com/apache/pinot/issues/10919 Creating this issue to initiate discussions about supporting vector embeddings in Pinot. This [write-up](https://docs.google.com/document/d/1aiXPbwK4rU_YdfMPt3K752SuCMy8KQehqM4ltPg9juE/edit)

[GitHub] [pinot] codecov-commenter commented on pull request #10918: [WIP] [multistage] Add TPC-H Test for Multistage Engine

2023-06-15 Thread via GitHub
codecov-commenter commented on PR #10918: URL: https://github.com/apache/pinot/pull/10918#issuecomment-1592732040 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10918?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[GitHub] [pinot] ankitsultana opened a new pull request, #10918: [WIP] [multistage] Add TPC-H Test for Multistage Engine

2023-06-15 Thread via GitHub
ankitsultana opened a new pull request, #10918: URL: https://github.com/apache/pinot/pull/10918 - [x] Setup tables with very little data - [ ] Stabilize the test - [ ] Add tpch queries - [ ] Add final data cc: @walterddr @kishoreg -- This is an automated message from the A