Thanks Jay Kataria for the comments.
> 1. Can you give an example of the scenarios where this can be useful. > Adding rate limiters to logs seems like an interesting idea, but just > wondering what is the business motivation. > 3. I am interested in what you talked about - dimensions and allow > thresholds to be shared across these dimensions or metrics. Could you give > an example of this particularly, I just want to know about the real world > applications of this. Please let me have a try on clarifing it. Generally speaking, the logging rate of each logger varies. In some scenarios or under the influence of existing filters, if a particular logger generates logs at an especially high rate, the log output of other loggers might be affected. In short, all loggers compete for the same type of rate-limited resources without any proactive intervention logic. For example, suppose there are logger1 and logger2, and the user is interested in the log output of logger2. A filter is configured to limit the log rate to 100 records/min. If logger1 produces logs at a rate of 200 records/min, it is highly likely that logger2 will be unable to output any logs because logger1 has already reached the rate-limiting threshold. The user expects that while ensuring the rate-limiting of logs, the target logger should still be able to output the necessary logs. At the least, no logger should be completely blocked from outputting logs due to rate-limiting. The best solution in this case is to set a shared rate-limiting condition for each logger. For example, allow each logger to output 100 records/min. This way, every logger is guaranteed a certain log output rate under rate-limiting. When the number of loggers is small, or when the log generation rate of the process is relatively low, even if each logger has reached the rate-limiting threshold, some output can still be allowed. This refers to the shared rate-limited resources or thresholds among all loggers. In this rate limiter, this corresponds to a process-level rate-limiting threshold. I drafted an example to illustrate how loggers can isolate rate-limited resources and compete for shared rate-limited resources. - The filter limiter statistics window is 1min. - Filter configs: - process level: 1000 records/min - logger level: 500 records/min - All loggers in the system: logger1, logger2 - Statistics of already generated log records - Process Stats: 998 records - logger1 Stats: 499 records - logger2 Stats: 499 records - The new log records sequences NO_n: record n of logger1 Result: Here's remaining 1 record in the current threshold of logger1 (499 to 500), so the record n is allowed to print. Stats change: - Process Stats: 999 records - logger1 Stats: 500 records NO_n+1. record n+1 of logger1 Result: Here's remaining 0 record in the current threshold of logger1 (500 to 500). but here's remaining 1 record in the process level threshold (999 to 1000). So the record n+1 is allowed to print. Stats change: - Process Stats: 1000 records - logger1 Stats: 501 records NO_n+2. record n+2 of logger1 Result: Here are no remaining records in threshold of logger1 level and process level. So the record n+2 is not allowed to print. Stats change: N.A NO_n+3. record n+3 of logger2 Result: Here's remaining 0 record in the current threshold of process level (1000 to 1000.) but here's remaining 1 record in the logger2 level threshold (499 to 500). So the record n+3 is allowed to print. Stats change: - Process Stats: 1001 records - logger2 Stats: 500 records NO_...: Subsequent logs will no longer be output as both dedicated rate-limited resources and shared rate-limited resources have been exhausted. > 2. Number of customers requesting this feature? Maintenance as @Piotr > Karwasz <pkarw...@apache.org> , mentioned is going to be a 5 - 10 year > period, if we do not have enough customers requesting this, then > maintenance of this feature + efforts might not be worth it. Thanks for the response. Sorry, I was not aware of this rule before. I'm not aware of the actual size of the user group with such needs. If necessary, perhaps we could conduct a survey in the user mailing list. This email is merely a discussion. If it is prohibited based on this rule before the discussion even begins, it might not be a bad thing, as it could help everyone avoid unnecessary discussions. Best, Yuepeng At 2025-01-27 03:32:28, "Jay Kataria" <jaykataria1...@gmail.com> wrote: >Hi Yuepeng, > >This seems interesting there are a few comments that I have based on the >doc and the feature request: > >1. Can you give an example of the scenarios where this can be useful. >Adding rate limiters to logs seems like an interesting idea, but just >wondering what is the business motivation. >2. Number of customers requesting this feature? Maintenance as @Piotr >Karwasz <pkarw...@apache.org> , mentioned is going to be a 5 - 10 year >period, if we do not have enough customers requesting this, then >maintenance of this feature + efforts might not be worth it. >3. I am interested in what you talked about - dimensions and allow >thresholds to be shared across these dimensions or metrics. Could you give >an example of this particularly, I just want to know about the real world >applications of this. > > >Regards, >Jay Katariya > > > >On Sun, Jan 26, 2025 at 2:57 AM Yuepeng Pan <panyuep...@apache.org> wrote: > >> Hi, community, >> >> >> >> >> In some business scenarios, users expect the log rate limit thresholds to >> be influenced >> >> by different dimensions and allow thresholds to be shared across these >> dimensions or metrics. >> >> This enables the system to flexibly output as many logs as possible within >> the safe constraints of the thresholds. >> >> Therefore, it is meaningful to introduce rate limiters based on process >> granularity and logger granularity, >> >> targeting both the number of log entries and the size of the logs. >> >> >> >> >> So, I'd like to start a discussion about 'Support a cross-rate Filter >> based on process and logger granularity'.[1] >> >> >> >> >> Looking forward to your attention and comments. >> >> >> >> >> Thank you. >> >> >> >> >> [1] >> https://docs.google.com/document/d/1kVa0V_RrPpT5aa5rfxEaH-QxyXTplQr65xUMZMmDoFA/edit?tab=t.0#heading=h.jfuayzme0ome >> >> >> >> >> Best, >> >> Yuepeng Pan
