Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452592342 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

ankitsultana merged PR #12242: URL: https://github.com/apache/pinot/pull/12242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

ankitsultana commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1934520257 Thanks @gortiz for your inputs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1934436316 > > is set to null in the close() function > > Oh... try to do not do that. It is usually worthless. If the context is closed it should be no references to it, so it should be co

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

gortiz commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1933835538 > is set to null in the close() function Oh... try to do not do that. It is usually worthless. If the context is closed it should be no references to it, so it should be collected. T

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1932714219 @gortiz Could you review the code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1932695513 I made an A/B testing to compare the performance option 1(static thread local) vs option 2(move bytes array to context) in two clusters with same size and configuration. It turns out t

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452592342 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1456837479 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

gortiz commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1454925120 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ private St

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1453781995 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

gortiz commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1453022438 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ private St

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

gortiz commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1453021005 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ private St

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1892981565 I added a screenshot which proved that the max heap memory is reduced from 220GB to 60GB and the heap memory is never ramping up. -- This is an automated message from the Apache Git

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452592342 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452592342 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452578998 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -103,11 +103,20 @@ pri

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

gortiz commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452127117 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -103,11 +103,20 @@ private

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

gortiz commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1452125608 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ private St

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

walterddr commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1889835222 CC @gortiz @Jackie-Jiang to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1446742933 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1446737468 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ privat

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

ankitsultana commented on code in PR #12242: URL: https://github.com/apache/pinot/pull/12242#discussion_r1446400925 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java: ## @@ -85,6 +85,10 @@ priv

Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]

codecov-commenter commented on PR #12242: URL: https://github.com/apache/pinot/pull/12242#issuecomment-1882253382 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12242?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] make thread local variable static to avoid the memory leaking issue [pinot]

wirybeaver opened a new pull request, #12242: URL: https://github.com/apache/pinot/pull/12242 For easer explanation, let's assume the server holds exactly one table and the table has only one column. The column is a playload column and the larget byte size of the entry is 2MB. The segment h