Channels are very useful and necessary to really get your program using all
the resources at your disposal. Doing that with locks or callbacks is error
prone and makes solving some problems all but impossible. Let me give you
an example of a pattern we used a few days ago:
We were processing message from a queue, writing some data to Cassandra,
and then committing offsets when the work was completed. It was really
important for the commits to come out in the same order they came in, so
that we never commit an offset when we haven't actually written all the
data. (on crash or restart its ok to process the same messages again
because the writing is idempotent)
Modeled as a pipeline we had a few stages:
intake -> decode -> encode -> write to cassandra -> commit
The naive approach would be to spin up a bunch of writer goroutines, but
then the order of messages to commit would be unpredictable. Instead we
created two slices of channels:
ins := make([]chan intakeMessage, options.workers)
outs := make([]chan commitMessage, options.workers)
And spun up goroutines that read from one of the input channels and wrote
to the corresponding out channel:
for i := 0; i < options.workers; i++ {
ins[i] = make(chan intakeMessage, 1)
outs[i] := make(chan commitMessage, 1)
go worker(ins[i], outs[i])
}
We then use these slices as a circular buffer and keep track of two
pointers, one for the next available in and another for the next remaining
out. There are 3 cases:
1. The slice is empty, in which case we insert at 0 and increment the in
counter
2. The slice is full (all workers are busy), in which case we wait until
a result is pushed into the channel at the out counter
3. We're somewhere in between, in which case we use a select on either
the next available in channel or the next remaining out counter
So all the commits come out in the order they came in, we get nice
parallelism and there's no blocking or polling.
There is overhead to channels but if you send messages of adequate size
you'll barely notice it in real programs. Batch and send slices.
The thing I find remarkable about channels and goroutines is how often the
solutions for improving the performance of a Go program are almost exactly
the same as the solutions for distributing that program across multiple
machines. The patterns really map well to these kinds of problems.
On Tuesday, August 8, 2017 at 2:01:12 AM UTC-4, snmed wrote:
>
> Hi Gophers
>
> I stumbled over a nice and very interesting Blog entry "Go channels are
> bad and you should feel bad
> <http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/>"
>
> , I would like to hear some opinions about that article
> from seasoned go developers. Because I just used go for a couple of months
> for my private web projects and rarely get in touch with channels.
>
> By the way, that article is not a rant and the author likes go very much
> as far as I can conclude from the article.
>
> Cheers snmed
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
