Hi Caleb
Your approach seems to me as a reasonable use case for channels and I'm
sure it would be a lot complexer if you use a bunch of mutexes.
One part in the article looks a little bit weird to me:
> // ...
> s.mtx.Lock()
> // ...
> s.ch <- val // might block!
> s.mtx.Unlock()
> // ...
>
>
My experience with go is limited but until now I have never seen that kind
of code, I don't think it is something advanced go users would do or I'm
wrong?
Cheers snmed
Am Mittwoch, 9. August 2017 14:30:38 UTC+2 schrieb Caleb Doxsey:
>
> Channels are very useful and necessary to really get your program using
> all the resources at your disposal. Doing that with locks or callbacks is
> error prone and makes solving some problems all but impossible. Let me give
> you an example of a pattern we used a few days ago:
>
> We were processing message from a queue, writing some data to Cassandra,
> and then committing offsets when the work was completed. It was really
> important for the commits to come out in the same order they came in, so
> that we never commit an offset when we haven't actually written all the
> data. (on crash or restart its ok to process the same messages again
> because the writing is idempotent)
>
> Modeled as a pipeline we had a few stages:
>
> intake -> decode -> encode -> write to cassandra -> commit
>
> The naive approach would be to spin up a bunch of writer goroutines, but
> then the order of messages to commit would be unpredictable. Instead we
> created two slices of channels:
>
> ins := make([]chan intakeMessage, options.workers)
> outs := make([]chan commitMessage, options.workers)
>
> And spun up goroutines that read from one of the input channels and wrote
> to the corresponding out channel:
>
> for i := 0; i < options.workers; i++ {
> ins[i] = make(chan intakeMessage, 1)
> outs[i] := make(chan commitMessage, 1)
> go worker(ins[i], outs[i])
> }
>
> We then use these slices as a circular buffer and keep track of two
> pointers, one for the next available in and another for the next remaining
> out. There are 3 cases:
>
> 1. The slice is empty, in which case we insert at 0 and increment the
> in counter
> 2. The slice is full (all workers are busy), in which case we wait
> until a result is pushed into the channel at the out counter
> 3. We're somewhere in between, in which case we use a select on either
> the next available in channel or the next remaining out counter
>
> So all the commits come out in the order they came in, we get nice
> parallelism and there's no blocking or polling.
>
> There is overhead to channels but if you send messages of adequate size
> you'll barely notice it in real programs. Batch and send slices.
>
> The thing I find remarkable about channels and goroutines is how often the
> solutions for improving the performance of a Go program are almost exactly
> the same as the solutions for distributing that program across multiple
> machines. The patterns really map well to these kinds of problems.
>
> On Tuesday, August 8, 2017 at 2:01:12 AM UTC-4, snmed wrote:
>>
>> Hi Gophers
>>
>> I stumbled over a nice and very interesting Blog entry "Go channels are
>> bad and you should feel bad
>> <http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/>"
>>
>> , I would like to hear some opinions about that article
>> from seasoned go developers. Because I just used go for a couple of
>> months for my private web projects and rarely get in touch with channels.
>>
>> By the way, that article is not a rant and the author likes go very much
>> as far as I can conclude from the article.
>>
>> Cheers snmed
>>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
