On Mon, Mar 16, 2009 at 4:21 PM, Stu Hood <[email protected]> wrote:
> If you write your CSV -> XML processing as a function, you could pmap (
> http://clojure.org/api#pmap) that function across the list of input
> files. pmap will transparently create the threads as needed, and it will
> probably be enough to saturate your disk.
>
> Thanks,
> Stu
>
Stu, I just want to make sure I understand what you are saying here. Would
it look something like this:
(pmap (fn [input-file]
; open input for reading
; open output for writing
; read line
; process line
; write line
)
["input-file1", "input-file2", "input-file3", ....])
Thanks.
>
>
> On Mon, Mar 16, 2009 at 5:56 PM, Brian Doyle <[email protected]>wrote:
>
>> I've been using Clojure for about 6 months now and really like it. I am
>> somewhat new to multi-threading
>> and using any of the parallel features in Clojure though. I have a
>> situation where I need to convert
>> 7 files from CSV to XML. Each one of these files is about 180MB apiece in
>> size. I have dual core machine
>> with 2GB of RAM and would like some advice on the best strategy for
>> processing these files in a way that
>> really utilizes both cores and my memory to really speed up the
>> processing. I'm sure this isn't the best
>> way, but I've only come up with starting up two threads at first, having
>> each thread open up a file,
>> call line-seq on that file, write out the XML for each line and then go to
>> the next file when it's complete. Any
>> advice would be great. Thanks.
>>
>>
>>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
