Hi,
I was experimenting with some code and I had an largish sequence I
did a doseq over. The memory hit the ceiling, which you expect since
even though the head isn't retained GC doesn't happen until you hit
your memory limit (is there a way to change that). Once it hit the
memory limit, 1.2 gigs in this case, the doseq slows to a halt.
On the other hand if rather than make a long lazy seq, I do it
implicitly in the doseq itself (see FAST snippet below), the
performance is great. Is there anyway to get good performance using a
single lazy seq? Relevant snippets below.
Thanks, Aria
(defn lines
"get lines from gz file"
[#^String path]
(-> path
java.io.FileInputStream.
java.util.zip.GZIPInputStream.
java.io.InputStreamReader.
java.io.BufferedReader.
line-seq))
; the tree-from-str does some processing, but doesn't have state
(def trees (for [l (lines "/usr/local/corpora//NANC/003.gz")
:when (not (empty? l))
:let [[t _] (tree/tree-from-str l)]]
t))
; SLOW: hits memory limit and becomes slow b/c of constant
; GC hits
(doseq [t trees]
(println (str t)))
; FAST: low memory
(doseq [l (lines "/usr/local/corpora//NANC/003.gz")
:when (not (empty? l))
:let [[t _] (tree/tree-from-str l)]]
(println (str t)))
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
To unsubscribe from this group, send email to
clojure+unsubscribegooglegroups.com or reply to this email with the words
"REMOVE ME" as the subject.
