Do you want something like this?
(ns dda.test)
(def test-infinite-lazy-seq (repeatedly
(fn [] {:id (rand-int 2)
:val (rand-int 10)})))
(def test-finite-seq [{:id 1 :val 1}
{:id 1 :val 2}
{:id 3 :val 1}])
(defn parallel-per
[k seqf ls]
(pmap #(map seqf %) (partition-by #(k %) ls)))
(take 10 (parallel-per :id
(fn [m] (update m :val inc))
test-infinite-lazy-seq))
(parallel-per :id
(fn [m] (update m :val inc))
test-finite-seq)
It handles your simple example, and can also handle infinite sequences
lazily, since I assumed your Kinesis stream would be infinite and you want
to process things as they come through.
Now this only parallelize groups that come through back to back. It is not
possible to do a group by ":id" on an infinite sequence, so the only thing
you could do better then this would be to chunk. So you could take in batch
of 100 from the stream, then group-by on it, and parallelize each groups. I
can try to write a solution for that too if you want.
On Tuesday, 20 June 2017 11:57:59 UTC-7, Tom Connors wrote:
>
> Great, I'll watch that video. Thanks again.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
