Re: [I] SIGSEGV errors while ingesting realtime data [pinot]

2024-02-08 Thread via GitHub
gortiz commented on issue #12376: URL: https://github.com/apache/pinot/issues/12376#issuecomment-1935442118 May be related to #12286. I would suggest the same thing I suggested there. Could you try again but running Pinot with Java 17 or 21? Alternatively, could you change `pinot.offheap.bu

Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

2024-02-08 Thread via GitHub
gortiz commented on issue #12286: URL: https://github.com/apache/pinot/issues/12286#issuecomment-1935438524 This kind of problems should not produce a SIGSEV. I think this may be related to using LArray buffers when the index is larger than 2GBs. One of the issues of LArray is that it

Re: [PR] Add uptime and start-time endpoints for server, broker, minion and controller [pinot]

2024-02-08 Thread via GitHub
suyashpatel98 commented on code in PR #12389: URL: https://github.com/apache/pinot/pull/12389#discussion_r1483917499 ## pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java: ## @@ -109,6 +112,8 @@ protected void configure() { bind(br

[PR] Add uptime and start-time endpoints for server, broker, minion and controller [pinot]

2024-02-08 Thread via GitHub
suyashpatel98 opened a new pull request, #12389: URL: https://github.com/apache/pinot/pull/12389 Add uptime and start-time endpoints for server, broker, minion and controller. Verified all endpoints work as expected. Following is sample output obtained while testing: ``` [ 7:47PM

Re: [PR] Enhancing metadata API to return upsert partition to primary key count map for both controller and server APIs [pinot]

2024-02-08 Thread via GitHub
KKcorps merged PR #12334: URL: https://github.com/apache/pinot/pull/12334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

(pinot) branch master updated: Enhancing metadata API to return upsert partition to primary key count map for both controller and server APIs (#12334)

2024-02-08 Thread kharekartik
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 947b47e3f4 Enhancing metadata API to return

Re: [I] PinotFSSegmentUploader uploads an unbounded number of files when AWS SDK has issues [pinot]

2024-02-08 Thread via GitHub
jadami10 commented on issue #12264: URL: https://github.com/apache/pinot/issues/12264#issuecomment-1935256955 > One caveat is that if the segment is not committed, the temp files won't be cleaned up. that's part of the issue in this case. uploading a temp file and moving seems like a

Re: [PR] Adding bytes/string data type integration tests [pinot]

2024-02-08 Thread via GitHub
codecov-commenter commented on PR #12387: URL: https://github.com/apache/pinot/pull/12387#issuecomment-1935201961 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12387?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] [Multi-stage] Stage stats are very misleading [pinot]

2024-02-08 Thread via GitHub
KKcorps commented on issue #12388: URL: https://github.com/apache/pinot/issues/12388#issuecomment-1935178397 That is weird. What's causing the doubling of leaf data? iirc, the stats are only sent when EOS block is received -- This is an automated message from the Apache Git Service. To r

Re: [I] [Multi-stage] Stage stats are very misleading [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang commented on issue #12388: URL: https://github.com/apache/pinot/issues/12388#issuecomment-1935173927 cc @gortiz @KKcorps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[I] [Multi-stage] Stage stats are very misleading [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang opened a new issue, #12388: URL: https://github.com/apache/pinot/issues/12388 Currently the stage stats reported in the query response is the aggregated stats over all the operators, which can be quite confusing. When tracing is disabled, we collect stats for leaf and send op

[PR] Adding bytes/string data type integration tests [pinot]

2024-02-08 Thread via GitHub
xiangfu0 opened a new pull request, #12387: URL: https://github.com/apache/pinot/pull/12387 For string bytes conversion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] [Flaky test] MultiStageEngineCustomTenantIntegrationTest.testGeneratedQueries() [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang commented on issue #12368: URL: https://github.com/apache/pinot/issues/12368#issuecomment-1935108446 Might be related to #11200. @jasperjiaguo Can you help take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

(pinot) branch master updated: [Multi-stage] Optimize mailbox info in query plan (#12382)

2024-02-08 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2d41b3806e [Multi-stage] Optimize mailbox info in

Re: [PR] [Multi-stage] Optimize mailbox info in query plan [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang merged PR #12382: URL: https://github.com/apache/pinot/pull/12382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] [Multi-stage] Optimize mailbox info in query plan [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang commented on PR #12382: URL: https://github.com/apache/pinot/pull/12382#issuecomment-1935093799 @kishoreg Yes, but probably not worth it. We can add new fields and let servers handle both old and new fields. It will take 2 releases to roll out any change. For multi-stage eng

Re: [PR] skip invalid json string rather than throwing error during json indexing [pinot]

2024-02-08 Thread via GitHub
ankitsultana commented on PR #12238: URL: https://github.com/apache/pinot/pull/12238#issuecomment-1935085148 Merged since very minor changes since Rong's last review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

(pinot) branch master updated: Skip invalid json string rather than throwing error during json indexing (#12238)

2024-02-08 Thread ankitsultana
This is an automated email from the ASF dual-hosted git repository. ankitsultana pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 8684e046c0 Skip invalid json string rather

Re: [PR] skip invalid json string rather than throwing error during json indexing [pinot]

2024-02-08 Thread via GitHub
ankitsultana merged PR #12238: URL: https://github.com/apache/pinot/pull/12238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] [Multi-stage] Optimize mailbox info in query plan [pinot]

2024-02-08 Thread via GitHub
kishoreg commented on PR #12382: URL: https://github.com/apache/pinot/pull/12382#issuecomment-1935026210 Is there a way to maintain backwards compatibility? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Add upsert-snapshot timer metric [pinot]

2024-02-08 Thread via GitHub
tibrewalpratik17 commented on code in PR #12383: URL: https://github.com/apache/pinot/pull/12383#discussion_r1483644800 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -660,8 +666,9 @@ protected void doTak

Re: [PR] Add upsert-snapshot timer metric [pinot]

2024-02-08 Thread via GitHub
deemoliu commented on code in PR #12383: URL: https://github.com/apache/pinot/pull/12383#discussion_r1483638578 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -660,8 +666,9 @@ protected void doTakeSnapsho

Re: [PR] Update superset docker build script [pinot]

2024-02-08 Thread via GitHub
codecov-commenter commented on PR #12385: URL: https://github.com/apache/pinot/pull/12385#issuecomment-1934961693 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12385?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-08 Thread via GitHub
ankitsultana commented on PR #12353: URL: https://github.com/apache/pinot/pull/12353#issuecomment-1934941633 @Jackie-Jiang : can you also take a look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] Update superset docker build script [pinot]

2024-02-08 Thread via GitHub
xiangfu0 opened a new pull request, #12385: URL: https://github.com/apache/pinot/pull/12385 Split superset docker build script to single arch and multi arch scripts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

(pinot) branch master updated: Fixing the multiple files concurrent write issue when reloading SSLFactory (#12384)

2024-02-08 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2b69d6afe2 Fixing the multiple files concurrent

Re: [PR] Fixing the multiple files concurrent write issue when reloading SSLFactory [pinot]

2024-02-08 Thread via GitHub
xiangfu0 merged PR #12384: URL: https://github.com/apache/pinot/pull/12384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] add retry to SSLFactory reloading [pinot]

2024-02-08 Thread via GitHub
xiangfu0 commented on code in PR #12384: URL: https://github.com/apache/pinot/pull/12384#discussion_r1483555919 ## pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java: ## @@ -447,12 +450,32 @@ static void reloadSslFactoryWhenFileStoreChanges(SSLFactory baseSs

Re: [PR] add retry to SSLFactory reloading [pinot]

2024-02-08 Thread via GitHub
codecov-commenter commented on PR #12384: URL: https://github.com/apache/pinot/pull/12384#issuecomment-1934879992 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12384?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] [discussion] On Upsert Hybrid Tables [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang commented on issue #12261: URL: https://github.com/apache/pinot/issues/12261#issuecomment-1934827413 This is not officially supported because of the known caveats, and it is quite hard to manage. User will need to create the post-upsert segments for offline table on their own.

Re: [I] PinotFSSegmentUploader uploads an unbounded number of files when AWS SDK has issues [pinot]

2024-02-08 Thread via GitHub
Jackie-Jiang commented on issue #12264: URL: https://github.com/apache/pinot/issues/12264#issuecomment-1934820014 I believe the intention of uploading a temp file first, then moving it to the final location is to avoid race condition or failed file upload. Thinking through the problem, I gu

[PR] add retry to SSLFactory reloading [pinot]

2024-02-08 Thread via GitHub
zhtaoxiang opened a new pull request, #12384: URL: https://github.com/apache/pinot/pull/12384 In our tests, we found that when reloading SSLFactory, need to retry a few times because when one file (key store or trust store) is updated, the other file (trust store or key store) may not have

Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

2024-02-08 Thread via GitHub
ksnijjer commented on issue #12286: URL: https://github.com/apache/pinot/issues/12286#issuecomment-1934779531 @snleee ^ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] SIGSEGV errors while ingesting realtime data [pinot]

2024-02-08 Thread via GitHub
snleee commented on issue #12376: URL: https://github.com/apache/pinot/issues/12376#issuecomment-1934763716 @donatelloOo This looks to be some low level bug where we write & read the data from the segment. Is it possible for us to get some extra information on how to reproduce issue? --

(pinot) branch master updated: Adding support for configuration through environment variables (#12307)

2024-02-08 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7c9bf8cf49 Adding support for configuration thro

Re: [PR] Adding support for configuration through environment variables [pinot]

2024-02-08 Thread via GitHub
xiangfu0 merged PR #12307: URL: https://github.com/apache/pinot/pull/12307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

(pinot) branch master updated: Make segment download from Peer servers more robust by retrying both peer discovery and download. (#12317)

2024-02-08 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 3fe25ba2e2 Make segment download from Peer serv

Re: [PR] Make segment download from Peer servers more robust by retrying both peer discovery and download. [pinot]

2024-02-08 Thread via GitHub
chenboat merged PR #12317: URL: https://github.com/apache/pinot/pull/12317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

(pinot) branch master updated: Make thread local variable static to avoid the memory leaking issue (#12242)

2024-02-08 Thread ankitsultana
This is an automated email from the ASF dual-hosted git repository. ankitsultana pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 506892d953 Make thread local variable stati

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

2024-02-08 Thread via GitHub
wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452592342 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

2024-02-08 Thread via GitHub
ankitsultana merged PR #12242: URL: https://github.com/apache/pinot/pull/12242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

2024-02-08 Thread via GitHub
ankitsultana commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1934520257 Thanks @gortiz for your inputs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-08 Thread via GitHub
tibrewalpratik17 commented on code in PR #12353: URL: https://github.com/apache/pinot/pull/12353#discussion_r1483220623 ## pinot-common/src/main/java/org/apache/pinot/common/utils/RoundRobinURIProvider.java: ## @@ -23,44 +23,66 @@ import java.net.URI; import java.net.URISyntax

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

2024-02-08 Thread via GitHub
wirybeaver commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1934436316 > > is set to null in the close() function > > Oh... try to do not do that. It is usually worthless. If the context is closed it should be no references to it, so it should be co

Re: [PR] Enhancing metadata API to return upsert partition to primary key count map for both controller and server APIs [pinot]

2024-02-08 Thread via GitHub
ege-st commented on code in PR #12334: URL: https://github.com/apache/pinot/pull/12334#discussion_r1483042388 ## pinot-controller/src/main/java/org/apache/pinot/controller/util/ServerSegmentMetadataReader.java: ## @@ -140,16 +143,17 @@ public TableMetadataInfo getAggregatedTabl

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-08 Thread via GitHub
ankitsultana commented on code in PR #12353: URL: https://github.com/apache/pinot/pull/12353#discussion_r1483039473 ## pinot-common/src/main/java/org/apache/pinot/common/utils/RoundRobinURIProvider.java: ## @@ -23,44 +23,66 @@ import java.net.URI; import java.net.URISyntaxExce

Re: [I] Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer [pinot]

2024-02-08 Thread via GitHub
ImAbhishekTomar commented on issue #10225: URL: https://github.com/apache/pinot/issues/10225#issuecomment-1934202266 After one year, I restarted this proof of concept and got stuck at the same point. Can someone help me with this? If possible, can we talk on a call? -- This is an auto

Re: [PR] Add a test that verifies how mode works right now with null [pinot]

2024-02-08 Thread via GitHub
gortiz closed pull request #12226: Add a test that verifies how mode works right now with null URL: https://github.com/apache/pinot/pull/12226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

2024-02-08 Thread via GitHub
gortiz commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1933835538 > is set to null in the close() function Oh... try to do not do that. It is usually worthless. If the context is closed it should be no references to it, so it should be collected. T

Re: [PR] Add upsert-snapshot timer metric [pinot]

2024-02-08 Thread via GitHub
codecov-commenter commented on PR #12383: URL: https://github.com/apache/pinot/pull/12383#issuecomment-1933797833 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12383?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] Add upsert-snapshot timer metric [pinot]

2024-02-08 Thread via GitHub
tibrewalpratik17 opened a new pull request, #12383: URL: https://github.com/apache/pinot/pull/12383 label: - `observability` This patch adds metrics to report the time taken by the upsert-snapshot workflow. Also improves logging by adding number of consuming segments. At pre

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-08 Thread via GitHub
tibrewalpratik17 commented on code in PR #12353: URL: https://github.com/apache/pinot/pull/12353#discussion_r1482663033 ## pinot-common/src/main/java/org/apache/pinot/common/utils/RoundRobinURIProvider.java: ## @@ -63,4 +78,33 @@ public URI next() { _index = (_index + 1) %

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-08 Thread via GitHub
tibrewalpratik17 commented on code in PR #12353: URL: https://github.com/apache/pinot/pull/12353#discussion_r1482662603 ## pinot-common/src/main/java/org/apache/pinot/common/utils/RoundRobinURIProvider.java: ## @@ -63,4 +78,33 @@ public URI next() { _index = (_index + 1) %