Are you building with -tags stringlabels? This tends to reduce memory used by Labels. Also I hope to reduce memory much more, though work stalled for a bit. (PR <https://github.com/prometheus/prometheus/pull/12304>)
Since Prometheus has to store all the output series in the head, I wouldn't take Cortex ruler impact as a guide to Prometheus impact. But if you measure that please do update us. I expect we would accept a PR adding an interface, unless it impacted performance noticeably. Can you share a pointer to your Cortex branch? Bryan (Prometheus maintainer, ex-Cortex maintainer) On Monday, 4 September 2023 at 16:14:40 UTC+1 Noam Dishon wrote: > Hello, > > I’m from Microsoft managed Prometheus team. We are running multiple cortex > ruler instances. > I’ve noticed that a large percentage of the memory of the ruler is > invested in keeping track of stale marks for series that no longer appear > in the result set. > > The complete set of labels of each iteration (of each rule, of each rule > group) are stored in memory, in seriesInPreviousEval > <https://github.com/prometheus/prometheus/blob/95e705612c1d557f1681bd081a841b78f93ee158/rules/manager.go#L325C8-L325C8>, > > until the next iteration. They are then compared > <https://github.com/prometheus/prometheus/blob/95e705612c1d557f1681bd081a841b78f93ee158/rules/manager.go#L755C13-L755C13> > > to the current iteration to see if any series is no longer returns from the > query. If so, that series is sent to the appender to be marked as stale. > > > I did a POC where the seriesInPreviousEval of each rule in stored in a > disk temp file. The results shows a reduction of about 75% in memory. > There is also a reduction of around 30% in CPU. No apparent negative affect > on evaluation duration. > > [image: Picture1.png] > [image: Picture2.png] > [image: Picture3.png] > [image: Picture4.png] > [image: Picture5.png] > > (These results are for cortex ruler, which does only rule processing, so > the effect is dramatic. But I’m positive that a standalone Prometheus > installation which handles a large number of rules will also benefit from > different handling of seriesInPreviousEval.) > > > > I would like to propose a couple of changes in this area: > > 2 1. Create an interface “PreviousTimeseriesStore” (or any better > name…) > > 2 2. Create a default implementation > MemoryPreviousTimeseriesStore. It would be similar to the current one. > However, I do think we can improve the memory implementation. It currently > is based on map[string]labels.Labels > <https://github.com/prometheus/prometheus/blob/95e705612c1d557f1681bd081a841b78f93ee158/rules/manager.go#L739> > > . The string part is made from the Bytes > <https://github.com/prometheus/prometheus/blob/b6f903b5f92b5458ad2244d9f442f7f859c01eb3/model/labels/labels.go#L72C33-L72C33> > > of the labels. > > I think we can use instead a map[string]bool (effectively a hash-set). > > Only if the labels are needed (which is just when a series doesn’t appear > in the current result set) then we need the labels set, and it can then be > reconstructed from the string. It requires FromBytes function in Labels > pkg, which isn’t complicated. > > C 3. Create another implementation which stores the series into the > disk. The same string of each series will be stored in a file as lines. > Switching to this implementation will be done via configuration. > > > 4. Custom implementation of the interface could be passed in > ManagerOptions > > > > Thanks, > Noam > > > > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6666c947-352f-4f0e-9f19-ff9b94c149ben%40googlegroups.com.
