Interesting, thanks for the data. It does seem like the process is spending a lot of time doing GC like I thought.
One trick you could try is to increase the memory allocated to the prober, which would reduce the time spent on GC. The default setting is is GOGC=100. You could try increasing this by setting the environment variable, GOGC. Try something like GOGC=200 or GOGC=300. This will make the process use more memory, but it should reduce the CPU time spent. On Sun, Feb 25, 2024 at 11:18 PM Alexander Wilke <[email protected]> wrote: > Hello, > > I attached a few screenshots showing the results and graphs for 1h and 6h. > In addition I added a screenshot from node_exporter metrics to give you an > overview of the system itself. > On the same system there is prometheus, grafana, snmp_exporter (200-800% > CPU), smokeping prober, node_exporter, blackbox_exporter. > The main CPU consumers are snmp_exporter and smokeping. > > Ben Kochie schrieb am Sonntag, 25. Februar 2024 um 19:22:35 UTC+1: > >> Looking at the CPU profile, I'm seeing almost all the time spent in the >> Go runtime. Mostly the ICMP packet receiving code and garbage collection. >> I'm not sure there's a lot we can optimize here as it's core Go code for >> ICMP packet handling. >> >> Can you also post me a graph of a few metrics queries? >> >> rate(process_cpu_seconds_total{job="smokeping_prober"}[30s]) >> rate(go_gc_duration_seconds_count{job="smokeping_prober"}[5m]) >> rate(go_gc_duration_seconds_sum{job="smokeping_prober"}[5m]) >> >> >> On Sun, Feb 25, 2024 at 7:08 PM Alexander Wilke <[email protected]> >> wrote: >> >>> Hello, >>> any Chance to investigate the Reports and any suggestions? >>> >>> Alexander Wilke schrieb am Donnerstag, 22. Februar 2024 um 12:40:09 >>> UTC+1: >>> >>>> Hello, >>>> >>>> sorry for the delay. here are the results. to be honest - I do not >>>> understand anything of it. >>>> >>>> Smokeping_Prober Heap: >>>> >>>> >>>> https://pprof.me/a1e7400d32859dbc217e2182398485df/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes&dashboard_items=icicle >>>> >>>> >>>> >>>> smokeping_prober profile30s >>>> >>>> >>>> https://pprof.me/340674b335e114e4b0df6b4582f0644e/?profileType=profile%3Asamples%3Acount%3Acpu%3Ananoseconds%3Adelta >>>> >>>> Ben Kochie schrieb am Dienstag, 20. Februar 2024 um 10:27:10 UTC+1: >>>> >>>>> Best thing you can do is capture some pprof data. That will show you >>>>> what it's spending the time on. >>>>> >>>>> :9374/debug/pprof/heap >>>>> :9374/debug/pprof/profile?seconds=30 >>>>> >>>>> You can post the results to https://pprof.me/ for sharing. >>>>> >>>>> On Tue, Feb 20, 2024 at 6:22 AM Alexander Wilke <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> I am running smokeping_prober from one VM to Monitor around 500 >>>>>> destinations. >>>>>> Around 30 devices are monitored with 0.2s Intervall and Others with >>>>>> 1.65s Intervall. >>>>>> >>>>>> Prometheus scrapes every 5s. >>>>>> >>>>>> So there are roughly 600 icmp ipv4 24byte pings per Seconds. >>>>>> CPU usage jumps between 700-1200% using "top" >>>>>> >>>>>> What Else except reducing Interval or Host Count could Help to reduce >>>>>> CPU usage? >>>>>> Is the UDP Socket "better" or any other optimization which could be >>>>>> relevant for that Type of Traffic? Running on RHEL8 >>>>>> >>>>>> Someone with similar CPU usage and this amount of pings per Seconds? >>>>>> Maybe Others Ping 6.000 Destination every 10s? >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/d803c1a2-64ee-48d1-8513-b864856f53c8n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/prometheus-users/d803c1a2-64ee-48d1-8513-b864856f53c8n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/62aeaacb-9fd6-4d64-8bec-d7171f592766n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/62aeaacb-9fd6-4d64-8bec-d7171f592766n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/045a6fb0-b82c-4425-8bae-e2c99403b2adn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/045a6fb0-b82c-4425-8bae-e2c99403b2adn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrDGh4cJWHCtmD_-SvJDEiTtA-kOnafmu0oe57F4rWz_w%40mail.gmail.com.
