We are having a weird case where an atomic number was increasing and not
decreasing, despite calls to decrease it. Hopefully, someone on the list
can point out something I'm doing wrong with the atomic package.
The function with the issue is:
func UnhealthyOverConcurrent(prefix string, limit int64) Middleware {
var concur int64
lhealth.Register(lflag.Prefixed("lhttp-concurrent", prefix), func() error {
c := atomic.LoadInt64(&concur)
if c <= limit {
return nil
}
return fmt.Errorf("requests at %d and limit is %d", c, limit)
})
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt64(&concur, 1)
h.ServeHTTP(w, r)
atomic.AddInt64(&concur, -1)
})
}
}
We started to observe that the error was being returned from the register
callback because c was over 500 (limit value in this case) but there was
definitely not 500 concurrent connections. There actually was only around
50. The lhealth.Register callback is called roughly every second and until
the service was restarted it had stayed constant over 500. This happened to
multiple instances of the service across multiple servers. The weird thing
is that we don't use atomic in another spot that also happens to track
concurrent requests and it didn't have the issue.
That spot roughly does:
func Metrics() Middleware {
var l sync.Mutex
var val float64
incr := func(by float64) {
l.Lock()
defer l.Unlock()
val += by
}
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
incr(1)
h.ServeHTTP(w, r)
incr(-1)
})
}
}
This second instance reported the correct number of concurrent requests
while the other function was off by a factor of 10.
Is there anything I'm doing wrong with the atomic package? Is it better to
just switch to a locking version like the second case? Since an http
handler can panic and cause the counts to be off, we are switching the
decrements to be deferred but we don't see any panics in the logs during
this time period.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
