Hello, folks!
I'm a relative noob in Clojure especially when it comes to concurrency, so
please forgive my ignorance. I have a processing stage (producer) that
feeds to another one (consumer). The producer has a bunch of items to
process and it's I/O blocking which takes random time, but the order of the
items is insignificant, so ideally they would materialize on the consumer
side on the first come first serve basis.
I would like to create a blocking lazy sequence I could just give to the
consumer. I know how to create a lazy sequence (lazy-seq), or how to make
it run in background and block on results (seque), but what I can't wrap my
head around is how parallelize the processing the Clojure way. I was
considering kicking off multiple agents, but how can I wait for *any one *of
them to finish, not all of them (as await does)? I'm not sure but I think
the same goes for futures/promises. I could have multiple agents putting
the results into some shared sequence, but then how do I block on the
sequence itself?
What I'm trying to do can be described in the following way in a silly
imperative pseudo-code:
workers = new Worker[10] ; initially w.got_data ==
nil
for each x in source_data:
w = wait_for_any_worker_ready(workers) ; initially all of them are
ready
if (w.got_data)
output.enqueue(w.data) ; the consumer will read
output in a blocking way
w.process(x) ; non-blocking, kicks off in
the background
Or, another way to describe it, given a seq of integers:
[ 1, 2, 3, 4 ... ]
and a simple function with a variable delay:
(defn process [x]
(Thread/sleep (* 10000 (rand)))
(* 2 x))
How can I write a function which would return a blocking lazy sequence of
processed integers, in arbitrary order, parallelizing the processing in up
to 10 threads?
Thank you!
Artem.
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
