You can use a semaphore-like pattern described
in https://tip.golang.org/ref/mem to limit the number of goroutines trying
to concurrently write to the log file. This remove the need for the
buffering scheme.
On Sunday, August 5, 2018 at 9:32:06 AM UTC-7, Anuj Agrawal wrote:
>
> I have an application with more than 50k goroutines. I pass a context to
> each of these goroutines which contains a logger and that logger writes to
> a file. At some point, the number of goroutines trying to write a log
> peaked and my application crashed with - runtime: program exceeds
> 10000-thread limitfatal error: thread exhaustion.
>
> To provide a quick fix, I added a writer in between the logger and the
> file writer which would buffer all logs in a channel and a separate
> goroutine can then read from the channel and write to the file, thus,
> avoiding having to create so many OS threads. Something like:
>
> type writer struct {
> wg sync.WaitGroup
> buffer chan []byte
> out io.WriteCloser
> pool sync.Pool
> }
>
> func (w *writer) Write(p []byte) (int, error) {
> tmp := w.pool.Get().([]byte)
> tmp = append(tmp[:0], p...)
> w.buffer <- tmp
> return len(tmp), nil
> }
>
> go func() {
> for p := range w.buffer {
> if _, err := w.out.Write(p); err != nil {
> log.Printf("error writing to log file: %s\n%s\n", err.Error(), string(p))
> }
> w.pool.Put(p)
> }
> w.wg.Done()
> }()
>
>
>
> I want to have suggestions if it can be fixed in a better way.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
