I tend to think of pull operations in these situations. So perhaps wrap a lazy sequence around a chunked form of the read + transform operation, perhaps parallelizing the transform, and do the insert in a doseq on the lazy sequence?
On Tue, Oct 22, 2013 at 8:03 PM, Brian Craft <[email protected]> wrote: > I'm doing a load to db, which looks roughly like "read, transform, insert, > repeat". > > Blocking on the inserts leaves the cores sitting cold while they could be > doing the next read/transform. It's tempting to try an agent for the > inserts. If I understand them correctly, that would queue the inserts as > they are ready, allowing the read/transform to continue. I'm concerned that > this would just fill memory, as the inserts can't keep up with the reads. > > Is there some other obvious way to structure this? The only relevant hits > I've found on this list are about the async lib, which I gather can handle > this by keeping a bounded queue that can block the producer. I was hoping > to put off learning the async stuff until later. ;) > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to [email protected] > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
