yupeng9 commented on issue #6121: URL: https://github.com/apache/incubator-pinot/issues/6121#issuecomment-706577224
``` RealtimeProvisioningHelper -tableConfigFile table -numPartitions 4 -pushFrequency null -numHosts 16,32,64 -numHours 2,3,4,5,6,7,8,9,10,11,12 -sampleCompletedSegmentDir rta_eats_canonical_rt_order_states__1__4457__20201008T2039Z -ingestionRate 1000 -maxUsableHostMemory 192G -retentionHours 336 Note: * Table retention and push frequency ignored for determining retentionHours since it is specified in command * See https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime Memory used per host (Active/Mapped) numHosts --> 16 |32 |64 | numHours 2 --------> 106G/106G |106G/106G |106G/106G | 3 --------> 106.22G/106.22G |106.22G/106.22G |106.22G/106.22G | 4 --------> 106.45G/106.45G |106.45G/106.45G |106.45G/106.45G | 5 --------> 107.92G/107.92G |107.92G/107.92G |107.92G/107.92G | 6 --------> 106.89G/106.89G |106.89G/106.89G |106.89G/106.89G | 7 --------> 107.11G/107.11G |107.11G/107.11G |107.11G/107.11G | 8 --------> 107.33G/107.33G |107.33G/107.33G |107.33G/107.33G | 9 --------> 109.43G/109.43G |109.43G/109.43G |109.43G/109.43G | 10 --------> 109.03G/109.03G |109.03G/109.03G |109.03G/109.03G | 11 --------> 109.56G/109.56G |109.56G/109.56G |109.56G/109.56G | 12 --------> 108.21G/108.21G |108.21G/108.21G |108.21G/108.21G | Optimal segment size numHosts --> 16 |32 |64 | numHours 2 --------> 643.23M |643.23M |643.23M | 3 --------> 964.85M |964.85M |964.85M | 4 --------> 1.26G |1.26G |1.26G | 5 --------> 1.57G |1.57G |1.57G | 6 --------> 1.88G |1.88G |1.88G | 7 --------> 2.2G |2.2G |2.2G | 8 --------> 2.51G |2.51G |2.51G | 9 --------> 2.83G |2.83G |2.83G | 10 --------> 3.14G |3.14G |3.14G | 11 --------> 3.45G |3.45G |3.45G | 12 --------> 3.77G |3.77G |3.77G | Consuming memory numHosts --> 16 |32 |64 | numHours 2 --------> 1.1G |1.1G |1.1G | 3 --------> 1.64G |1.64G |1.64G | 4 --------> 2.17G |2.17G |2.17G | 5 --------> 2.71G |2.71G |2.71G | 6 --------> 3.24G |3.24G |3.24G | 7 --------> 3.78G |3.78G |3.78G | 8 --------> 4.31G |4.31G |4.31G | 9 --------> 4.85G |4.85G |4.85G | 10 --------> 5.38G |5.38G |5.38G | 11 --------> 5.92G |5.92G |5.92G | 12 --------> 6.45G |6.45G |6.45G | Total number of segments queried per host (for all partitions) numHosts --> 16 |32 |64 | numHours 2 --------> 168 |168 |168 | 3 --------> 112 |112 |112 | 4 --------> 84 |84 |84 | 5 --------> 68 |68 |68 | 6 --------> 56 |56 |56 | 7 --------> 48 |48 |48 | 8 --------> 42 |42 |42 | 9 --------> 38 |38 |38 | 10 --------> 34 |34 |34 | 11 --------> 31 |31 |31 | 12 --------> 28 |28 |28 | ``` Curious, how much memory do you allocate for Pinot server (i.e. -xmx) at Linkedin? And whats the memory size of your machine. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
