On Sat, Apr 27, 2013 at 7:35 PM, Stanislav Yurin <[email protected]> wrote:
> Actually, what I was trying to do, is to prototype multithreaded i/o > operation via reducers. And then use fold to regulate number of concurrent > operations. > But now something tells me I am doing not very clever thing. > I'm not entirely sure what you mean above, but I've been very happy using reducers with I/O so far. Reducers just have a few tricks you need to be aware of first; 1. You wont get parallel processing unless the input is a vector. 2. Each thread gets ~512 elements each, so reducing a vector of 800 elements will only use two cores. 3. How you aggregate your final result can greatly impact performance. (Ex. (r/fold +) is fast, fold-into-vec is slower, etc) I wrote a library to use reducers over text files (input) and have found it to be invaluable for working with giant TSV files. Link: https://github.com/thebusby/iota/ I often then use fold-into-lazy-seq to write the output back to a file. Link: https://gist.github.com/thebusby/5472980 Hope this helps! ;) -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
