Re: [I] Multiple upstream source ingestion support on Pinot [pinot]

2024-09-17 Thread via GitHub
hpvd commented on issue #13780: URL: https://github.com/apache/pinot/issues/13780#issuecomment-2357613319 as described: > The implementation strategy should consider decoupling the partition concept between stream and Pinot. Theoretically, stream and OLAP db are two independent infra

[PR] Replace fmpp-maven-plugin with fmpp Ant Task to avoid pulling the vul… [pinot]

2024-09-17 Thread via GitHub
chrajeshbabu opened a new pull request, #14015: URL: https://github.com/apache/pinot/pull/14015 …nerable log4j jars -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Make Custom Config Optional [pinot]

2024-09-17 Thread via GitHub
ashishjayamohan opened a new pull request, #14014: URL: https://github.com/apache/pinot/pull/14014 - Made custom config in tableConfig optional - Changed key for custom config from "metadata" to "customConfig" - Added nullity check for getter -- This is an automated message from the

Re: [PR] Part-2: Add Combine and Segment Level Operators for Time Series [pinot]

2024-09-17 Thread via GitHub
ankitsultana commented on code in PR #13999: URL: https://github.com/apache/pinot/pull/13999#discussion_r1764341724 ## pinot-core/src/main/java/org/apache/pinot/core/query/request/ServerQueryRequest.java: ## @@ -107,6 +107,28 @@ public ServerQueryRequest(Server.ServerRequest se

Re: [I] Pagination Support [pinot]

2024-09-17 Thread via GitHub
monicaluodialpad commented on issue #5246: URL: https://github.com/apache/pinot/issues/5246#issuecomment-2357403566 Any update here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Aggregation filter is being pushed down in a way it breaks SQL semantics [pinot]

2024-09-17 Thread via GitHub
kmozaid commented on issue #12647: URL: https://github.com/apache/pinot/issues/12647#issuecomment-2357393174 @gortiz Hi, any update on this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

(pinot) branch master updated: TLS Configuration Support for QueryServer and Dispatch Client (#13645)

2024-09-17 Thread ankitsultana
This is an automated email from the ASF dual-hosted git repository. ankitsultana pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 21ff6bf354 TLS Configuration Support for Qu

Re: [PR] TLS Configuration Support for QueryServer and Dispatch Client [pinot]

2024-09-17 Thread via GitHub
ankitsultana merged PR #13645: URL: https://github.com/apache/pinot/pull/13645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Add multi stream ingestion support [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #13790: URL: https://github.com/apache/pinot/pull/13790#discussion_r1764232316 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -774,15 +778,32 @@ Set getPartitionIds(Str

Re: [PR] Add multi stream ingestion support [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #13790: URL: https://github.com/apache/pinot/pull/13790#discussion_r1764232316 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java: ## @@ -774,15 +778,32 @@ Set getPartitionIds(Str

Re: [PR] Add multi stream ingestion support [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #13790: URL: https://github.com/apache/pinot/pull/13790#discussion_r1764227690 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java: ## @@ -173,15 +173,18 @@ public static void validate(TableConfig table

Re: [PR] Add multi stream ingestion support [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #13790: URL: https://github.com/apache/pinot/pull/13790#discussion_r1764220715 ## pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java: ## @@ -686,9 +686,8 @@ public void ingestionStreamConfigsTest() {

Re: [PR] [WIP] [Feature] Enable the capability to specify zstd and lz4 segment compression via config [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #14008: URL: https://github.com/apache/pinot/pull/14008#discussion_r1764161399 ## pinot-common/src/main/java/org/apache/pinot/common/utils/TarCompressionUtils.java: ## @@ -94,14 +107,27 @@ public static void createCompressedTarFile(File inputFile

Re: [PR] [WIP] [Feature] Enable the capability to specify zstd and lz4 segment compression via config [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #14008: URL: https://github.com/apache/pinot/pull/14008#discussion_r1764143375 ## pinot-common/src/main/java/org/apache/pinot/common/utils/TarCompressionUtils.java: ## @@ -94,14 +107,27 @@ public static void createCompressedTarFile(File inputFile

Re: [PR] [WIP] [Feature] Enable the capability to specify zstd and lz4 segment compression via config [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #14008: URL: https://github.com/apache/pinot/pull/14008#discussion_r1764132139 ## pinot-common/src/main/java/org/apache/pinot/common/utils/TarCompressionUtils.java: ## @@ -94,14 +107,27 @@ public static void createCompressedTarFile(File inputFile

Re: [PR] [WIP] [Feature] Enable the capability to specify zstd and lz4 segment compression via config [pinot]

2024-09-17 Thread via GitHub
deemoliu commented on code in PR #14008: URL: https://github.com/apache/pinot/pull/14008#discussion_r1764132139 ## pinot-common/src/main/java/org/apache/pinot/common/utils/TarCompressionUtils.java: ## @@ -94,14 +107,27 @@ public static void createCompressedTarFile(File inputFile

Re: [PR] track newly added segments for upsert tables for a more complete upsert data view [pinot]

2024-09-17 Thread via GitHub
klsince commented on code in PR #13992: URL: https://github.com/apache/pinot/pull/13992#discussion_r1764087728 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -1200,12 +1219,45 @@ public void trackSegmentF

Re: [PR] Increase unit test coverage for AbstractMetrics [pinot]

2024-09-17 Thread via GitHub
spanasch commented on PR #13991: URL: https://github.com/apache/pinot/pull/13991#issuecomment-2357015959 Thanks @tibrewalpratik17 ! I've made the changes you suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Increase unit test coverage for AbstractMetrics [pinot]

2024-09-17 Thread via GitHub
spanasch commented on code in PR #13991: URL: https://github.com/apache/pinot/pull/13991#discussion_r1764072035 ## pinot-common/src/test/java/org/apache/pinot/common/metrics/AbstractMetricsTest.java: ## @@ -108,4 +112,154 @@ public void testMultipleGauges() { controllerMetr

Re: [PR] Increase unit test coverage for AbstractMetrics [pinot]

2024-09-17 Thread via GitHub
spanasch commented on code in PR #13991: URL: https://github.com/apache/pinot/pull/13991#discussion_r1764071915 ## pinot-common/src/test/java/org/apache/pinot/common/metrics/AbstractMetricsTest.java: ## @@ -108,4 +112,154 @@ public void testMultipleGauges() { controllerMetr

Re: [PR] Increase unit test coverage for AbstractMetrics [pinot]

2024-09-17 Thread via GitHub
spanasch commented on code in PR #13991: URL: https://github.com/apache/pinot/pull/13991#discussion_r1764071627 ## pinot-common/src/test/java/org/apache/pinot/common/metrics/AbstractMetricsTest.java: ## @@ -108,4 +112,154 @@ public void testMultipleGauges() { controllerMetr

Re: [PR] Make Custom Config Optional [pinot]

2024-09-17 Thread via GitHub
ashishjayamohan closed pull request #14013: Make Custom Config Optional URL: https://github.com/apache/pinot/pull/14013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] support for expansion of * in the project part of a query [pinot]

2024-09-17 Thread via GitHub
shauryachats commented on issue #6470: URL: https://github.com/apache/pinot/issues/6470#issuecomment-2356898120 @kant777 you could try using ``` select *, 'phone' device from pageviews ``` which should do the same thing effectively. -- This is an automated message from the Apac

[PR] Make Custom Config Optional [pinot]

2024-09-17 Thread via GitHub
ashishjayamohan opened a new pull request, #14013: URL: https://github.com/apache/pinot/pull/14013 - Make custom config in tableConfig optional - Change custom config key from "metadata" to "customConfig" -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Make custom config optional and change key [pinot]

2024-09-17 Thread via GitHub
ashishjayamohan closed pull request #14012: Make custom config optional and change key URL: https://github.com/apache/pinot/pull/14012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[PR] Make custom config optional and change key [pinot]

2024-09-17 Thread via GitHub
ashishjayamohan opened a new pull request, #14012: URL: https://github.com/apache/pinot/pull/14012 - Made custom config in tableConfig optional - Changed custom config key from "metadata" to "customConfig" -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] track newly added segments for upsert tables for a more complete upsert data view [pinot]

2024-09-17 Thread via GitHub
Jackie-Jiang commented on code in PR #13992: URL: https://github.com/apache/pinot/pull/13992#discussion_r1763761214 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -1200,12 +1219,45 @@ public void trackSeg

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-09-17 Thread via GitHub
Jackie-Jiang commented on PR #13632: URL: https://github.com/apache/pinot/pull/13632#issuecomment-2356680555 @jadami10 So unfortunate that you also run into this 1 in 1 chance bug.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Part-2: Add Combine and Segment Level Operators for Time Series [pinot]

2024-09-17 Thread via GitHub
raghavyadav01 commented on code in PR #13999: URL: https://github.com/apache/pinot/pull/13999#discussion_r1763618914 ## pinot-core/src/main/java/org/apache/pinot/core/query/request/ServerQueryRequest.java: ## @@ -107,6 +107,28 @@ public ServerQueryRequest(Server.ServerRequest s

Re: [PR] Part-2: Add Combine and Segment Level Operators for Time Series [pinot]

2024-09-17 Thread via GitHub
raghavyadav01 commented on PR #13999: URL: https://github.com/apache/pinot/pull/13999#issuecomment-2356500159 @ankitsultana There are couple of tests failing https://github.com/apache/pinot/actions/runs/10860622852/job/30141445761?pr=13999 Are these related or unrelated? -- This is an a

Re: [PR] Part-2: Add Combine and Segment Level Operators for Time Series [pinot]

2024-09-17 Thread via GitHub
raghavyadav01 commented on code in PR #13999: URL: https://github.com/apache/pinot/pull/13999#discussion_r1763614274 ## pinot-core/src/main/java/org/apache/pinot/core/operator/timeseries/TimeSeriesAggregationOperator.java: ## @@ -0,0 +1,251 @@ +/** + * Licensed to the Apache Sof

Re: [PR] Storage Quota imposition on Realtime tables [pinot]

2024-09-17 Thread via GitHub
shounakmk219 commented on code in PR #13584: URL: https://github.com/apache/pinot/pull/13584#discussion_r1763607852 ## pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java: ## @@ -108,8 +112,45 @@ protected void processTable

Re: [PR] Storage Quota imposition on Realtime tables [pinot]

2024-09-17 Thread via GitHub
mcvsubbu commented on code in PR #13584: URL: https://github.com/apache/pinot/pull/13584#discussion_r1763445783 ## pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java: ## @@ -108,8 +112,45 @@ protected void processTable(Str

[I] Query execution error when filtering constant columns from subquery [pinot]

2024-09-17 Thread via GitHub
richard-misiak-az opened a new issue, #14010: URL: https://github.com/apache/pinot/issues/14010 Pinot 1.2.0 The following query ``` SELECT * FROM (SELECT CASE WHEN m.userId = 'abc123' THEN 'Test1' ELSE 'Test2'

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-09-17 Thread via GitHub
jadami10 commented on PR #13632: URL: https://github.com/apache/pinot/pull/13632#issuecomment-2355707225 We just ran into this in 1.2, and I see `parking to wait for ` from 2 threads that have been waiting 9+ hours. I traced back the segmentLock to the same code you have here. Thanks for ro

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
hpvd commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2355464977 as added in comment above, there are at least 3-4 issues talking about we want/need to go smaller: maybe we should open a final issue on this as summarization or parent issue? -- Thi

(pinot) branch dependabot/maven/software.amazon.awssdk-bom-2.28.2 created (now 5a4a1aa9d3)

2024-09-17 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/software.amazon.awssdk-bom-2.28.2 in repository https://gitbox.apache.org/repos/asf/pinot.git at 5a4a1aa9d3 Bump software.amazon.awssdk:bom from 2.28.1 to 2.28.2 No n

[PR] Bump software.amazon.awssdk:bom from 2.28.1 to 2.28.2 [pinot]

2024-09-17 Thread via GitHub
dependabot[bot] opened a new pull request, #14009: URL: https://github.com/apache/pinot/pull/14009 Bumps software.amazon.awssdk:bom from 2.28.1 to 2.28.2. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=software

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
hpvd commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2355447230 > we can build a new Pinot image, say `Pinot-slim`, with the minimum required packages installed. +1 on building a smaller image. There is also an issue about this: https://

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
abhioncbr commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2355431107 Thanks @xiangfu0. I think part of the problem can be resolved by building the pinot-base-runtime images once daily. If I am interpreting the [data correctly](https://github.

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
xiangfu0 commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2355008916 @abhioncbr Please let me know if there is anything more you need. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
xiangfu0 commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2354988948 For MS OpenJDK, the one CVE is [CVE-2024-26800](https://scout.docker.com/vulnerabilities/id/CVE-2024-26800/org/apachepinot?s=ubuntu&n=linux&ns=ubuntu&t=deb&osn=ubuntu&osv=22.04&vr=%3

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
xiangfu0 commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2354984508 And here is the 2 CVEs for corretto: For amazon [jdk](https://hub.docker.com/layers/apachepinot/pinot-base-runtime/17-amazoncorretto/images/sha256-1336fb03a4edfe4ccff927fcbd618a7

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
xiangfu0 commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2354977398 The sample run for base runtime image build is: https://github.com/apachepinot/pinot-fork/actions/workflows/build-pinot-base-runtime-docker-image.yml -- This is an automated m

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
xiangfu0 commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2354976255 I think openjdk 21 base image is not updated for a year, and mentioned deprecated (https://hub.docker.com/_/openjdk) I changed the build to use both Amazon Corretto and MS OpenJDK

Re: [I] Apache Pinot vulnerabilities issues [pinot]

2024-09-17 Thread via GitHub
xiangfu0 commented on issue #13461: URL: https://github.com/apache/pinot/issues/13461#issuecomment-2354973101 We are using this script to build docker runtime image: https://github.com/apache/pinot/blob/master/docker/images/pinot-base/pinot-base-runtime/amazoncorretto.dockerfile

(pinot) branch master updated: Enforce removeSegment flow with _enableDeletedKeysCompactionConsistency (#13914)

2024-09-17 Thread pratik
This is an automated email from the ASF dual-hosted git repository. pratik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c565a83df1 Enforce removeSegment flow with _enab

Re: [PR] Enforce removeSegment flow with _enableDeletedKeysCompactionConsistency [pinot]

2024-09-17 Thread via GitHub
tibrewalpratik17 merged PR #13914: URL: https://github.com/apache/pinot/pull/13914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@p