Re: [PR] Make groupBy trim size configurable at Broker [pinot]

2023-11-06 Thread via GitHub
codecov-commenter commented on PR #11958: URL: https://github.com/apache/pinot/pull/11958#issuecomment-1797973258 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11958?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] Make groupBy trim size configurable at Broker [pinot]

2023-11-06 Thread via GitHub
vvivekiyer opened a new pull request, #11958: URL: https://github.com/apache/pinot/pull/11958 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
jhyao commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797885674 Tested select count with "skipUpsert"=true, no this issue. ![image](https://github.com/apache/pinot/assets/17529008/66ead424-19f3-4ff9-86ea-9d226e3cbd03) -- This is an automat

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
jhyao commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797882360 That's all table configs. I rely on kafka partition so I don't need segmentPartitionConfig, and in my testing the kafka topic only have one partition. And only one server. -- This i

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
klsince commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797828108 is the table config posted above the complete config? There is no `segmentPartitionConfig`, which defines table partitioning required by upsert tables to place the segments and route

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
klsince commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797811501 cool! How about using this query `set "skipUpsert"=true; select count(*) ...` to run test with the upsert table again, to see if any critical upsert states were not visible to the qu

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
jhyao commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797769758 After publishing 1M ids, producer continued to send 2M upsert data with same ids as first 1M ids. Producer code like this: ```python def generate_record(id): record = {

[I] flaky test on org.apache.pinot.core.operator.combine.SelectionCombineOperatorTest [pinot]

2023-11-06 Thread via GitHub
xiangfu0 opened a new issue, #11957: URL: https://github.com/apache/pinot/issues/11957 ``` Error: Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.336 s <<< FAILURE! -- in org.apache.pinot.core.operator.combine.SelectionCombineOperatorTest Error: org.apache.pin

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
klsince commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797391108 Nice test setup. If you can customize it, could you test with a table without using upsert, just to see if this issue is specific to upsert table or not. Thanks! -- This is an aut

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
KKcorps commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797272697 Is the consumption and producer still running even after publishing 1M ids? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Enhance Force-Commit Endpoint to Stagger Segment Commit for Different Partitions [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang commented on issue #11950: URL: https://github.com/apache/pinot/issues/11950#issuecomment-1797129836 When force commit is triggered, I believe server side should have a semaphore controlling the concurrent segment commit allowed. We do detect a hotspot on controller side, whi

Re: [PR] Upgrade lucene to 9.8.0 and upgrade text index version [pinot]

2023-11-06 Thread via GitHub
xiangfu0 commented on code in PR #11857: URL: https://github.com/apache/pinot/pull/11857#discussion_r1384234864 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java: ## @@ -44,14 +44,17 @@ public static class Indexes { public static final String

Re: [I] Count number of upsert table is incorrect when a new segment is created [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang commented on issue #11948: URL: https://github.com/apache/pinot/issues/11948#issuecomment-1797121368 Thanks for reporting the issue. We will take a look cc @klsince @KKcorps -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Support constant filter in QueryContext, and make server able to handle it [pinot]

2023-11-06 Thread via GitHub
codecov-commenter commented on PR #11956: URL: https://github.com/apache/pinot/pull/11956#issuecomment-1797120527 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11956?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] fix flakyness by replacing HashSet and HashMap with LinkedHashSet and LinkedHashMap [pinot]

2023-11-06 Thread via GitHub
codecov-commenter commented on PR #11941: URL: https://github.com/apache/pinot/pull/11941#issuecomment-1797119813 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11941?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

(pinot) branch dependabot/maven/com.google.http-client-google-http-client-1.43.3 deleted (was e32f652cf7)

2023-11-06 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.http-client-google-http-client-1.43.3 in repository https://gitbox.apache.org/repos/asf/pinot.git was e32f652cf7 Bump com.google.http-client:google-http-clie

Re: [PR] Bump com.google.http-client:google-http-client from 1.36.0 to 1.43.3 [pinot]

2023-11-06 Thread via GitHub
dependabot[bot] commented on PR #11947: URL: https://github.com/apache/pinot/pull/11947#issuecomment-1797115972 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Bump com.google.http-client:google-http-client from 1.36.0 to 1.43.3 [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang closed pull request #11947: Bump com.google.http-client:google-http-client from 1.36.0 to 1.43.3 URL: https://github.com/apache/pinot/pull/11947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Upgrade lucene to 9.8.0 and upgrade text index version [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang commented on code in PR #11857: URL: https://github.com/apache/pinot/pull/11857#discussion_r1384225641 ## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java: ## @@ -44,14 +44,17 @@ public static class Indexes { public static final Str

Re: [PR] Adds support for leveraging StarTree index in conjunction with filtered aggregations [pinot]

2023-11-06 Thread via GitHub
egalpin commented on PR #11886: URL: https://github.com/apache/pinot/pull/11886#issuecomment-1797102630 Contrived example to show the working of FILTER (only has a result for the one row that matches `AirlineID`) https://github.com/apache/pinot/assets/7582748/180999f0-72cb-47e9-afcf-e

Re: [PR] Adds support for leveraging StarTree index in conjunction with filtered aggregations [pinot]

2023-11-06 Thread via GitHub
egalpin commented on PR #11886: URL: https://github.com/apache/pinot/pull/11886#issuecomment-1797101125 @xiangfu0 @Jackie-Jiang This can now be considered ready for review, as it's operationally functional. I'll add tests, but wanted to flag that this is ready for review in case that helps

[PR] Support constant filter in QueryContext, and make server able to handle it [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang opened a new pull request, #11956: URL: https://github.com/apache/pinot/pull/11956 Currently we rely on broker to remove the constant (true/false) filter, and server doesn't support processing constant filter. This PR adds the server side support to handle constant filter which

Re: [PR] fix flakyness by replacing HashSet and HashMap with LinkedHashSet and LinkedHashMap [pinot]

2023-11-06 Thread via GitHub
Nemesis123925 commented on PR #11941: URL: https://github.com/apache/pinot/pull/11941#issuecomment-1797049371 I try the way that Jackie Jiang proposed to use `toString()` of `HashSet` in the test to generate the `PROJECT` clause. It doesn't seems to work because the NonDex tool, which we us

(pinot) branch master updated: [bugfix][multistage] explicit warning flags set on each stage stats (#11936)

2023-11-06 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new d177866527 [bugfix][multistage] explicit warning f

Re: [I] [multistage][bug] warning exceptions not populate to broker [pinot]

2023-11-06 Thread via GitHub
walterddr closed issue #11927: [multistage][bug] warning exceptions not populate to broker URL: https://github.com/apache/pinot/issues/11927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [bugfix][multistage] explicit warning flags set on each stage stats [pinot]

2023-11-06 Thread via GitHub
walterddr merged PR #11936: URL: https://github.com/apache/pinot/pull/11936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

Re: [PR] [bugfix] Add check for illegal character '/' in taskName [pinot]

2023-11-06 Thread via GitHub
aishikbh commented on code in PR #11955: URL: https://github.com/apache/pinot/pull/11955#discussion_r1383825809 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/task/AdhocTaskConfig.java: ## @@ -57,6 +57,7 @@ public AdhocTaskConfig(@JsonProperty(value = "taskType", requir

Re: [PR] [bugfix] Add check for illegal character '/' in taskName [pinot]

2023-11-06 Thread via GitHub
swaminathanmanish commented on code in PR #11955: URL: https://github.com/apache/pinot/pull/11955#discussion_r1383822916 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/task/AdhocTaskConfig.java: ## @@ -57,6 +57,7 @@ public AdhocTaskConfig(@JsonProperty(value = "taskType"

Re: [PR] Optimize segment commit to not read partition group metadata [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang commented on PR #11943: URL: https://github.com/apache/pinot/pull/11943#issuecomment-1795755694 @mcvsubbu Thanks for taking time writing this program! > According to this, it takes slightly more number of iterations to stabilize to the right segment size if we apply the a

Re: [PR] [bugfix] Add check for illegal character '/' in taskName [pinot]

2023-11-06 Thread via GitHub
snleee commented on code in PR #11955: URL: https://github.com/apache/pinot/pull/11955#discussion_r1383746812 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/task/AdhocTaskConfig.java: ## @@ -57,6 +57,7 @@ public AdhocTaskConfig(@JsonProperty(value = "taskType", required

Re: [PR] [bugfix] Add check for illegal character '/' in taskName [pinot]

2023-11-06 Thread via GitHub
codecov-commenter commented on PR #11955: URL: https://github.com/apache/pinot/pull/11955#issuecomment-1795740305 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11955?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] [bugfix][multistage] explicit warning flags set on each stage stats [pinot]

2023-11-06 Thread via GitHub
Jackie-Jiang commented on code in PR #11936: URL: https://github.com/apache/pinot/pull/11936#discussion_r1383722267 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/HashJoinOperator.java: ## @@ -225,6 +226,8 @@ private void buildBroadcastHashTable()

Re: [PR] Optimize segment commit to not read partition group metadata [pinot]

2023-11-06 Thread via GitHub
mcvsubbu commented on PR #11943: URL: https://github.com/apache/pinot/pull/11943#issuecomment-1795513639 I wrote this program to test out some hypothesis. According to this, it takes slightly more number of iterations to stabilize to the right segment size if we apply the algorithm for all

Re: [I] Add real-time table support for data ingestion rest API [pinot]

2023-11-06 Thread via GitHub
rohityadav1993 commented on issue #11914: URL: https://github.com/apache/pinot/issues/11914#issuecomment-1794560148 Is the scope of this to also handle upsert tables? We have some requirements where we need to backfill upsert table -- This is an automated message from the Apache Git Servi

Re: [PR] Add DATETIMECONVERTWINDOWHOP function [pinot]

2023-11-06 Thread via GitHub
alexch2000 commented on PR #11773: URL: https://github.com/apache/pinot/pull/11773#issuecomment-1794270853 > Looks great! Can you help add documentation for this new feature following instructions [here](https://docs.pinot.apache.org/contributing/contributing)? Sure! Will do this week