I have an application with more than 50k goroutines. I pass a context to
each of these goroutines which contains a logger and that logger writes to
a file. At some point, the number of goroutines trying to write a log
peaked and my application crashed with - runtime: program exceeds
10000-thread limitfatal error: thread exhaustion.
To provide a quick fix, I added a writer in between the logger and the file
writer which would buffer all logs in a channel and a separate goroutine
can then read from the channel and write to the file, thus, avoiding having
to create so many OS threads. Something like:
type writer struct {
wg sync.WaitGroup
buffer chan []byte
out io.WriteCloser
pool sync.Pool
}
func (w *writer) Write(p []byte) (int, error) {
tmp := w.pool.Get().([]byte)
tmp = append(tmp[:0], p...)
w.buffer <- tmp
return len(tmp), nil
}
go func() {
for p := range w.buffer {
if _, err := w.out.Write(p); err != nil {
log.Printf("error writing to log file: %s\n%s\n", err.Error(), string(p))
}
w.pool.Put(p)
}
w.wg.Done()
}()
I want to have suggestions if it can be fixed in a better way.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
