One of the objectives of Clojure is to reduce incidental complexity. And one of the biggest sources of incidental complexity in Clojure was the retention of the head of a lazy sequence due to its being referenced by some local (argument or local (let) binding). One might expect that, if no subsequent code in the body of a function uses that arg/local, it would be subject to GC. Unfortunately, on the JVM, that is, in many cases, not true - the local is considered a live reference and is thus not GCed. This yields the infamous 'holding onto the head' problem, and subsequent Out Of Memory errors on large data sets.
I had put in place a workaround, which was the 'clearing/nulling-out' of locals on the tail call of the function. This helps in many, but not all, cases. Not all logic flows are amenable to rearrangement to leverage this cleanup. And there are many cases where the local is not visible - e.g. when destructuring. The full solution is to track, during compilation, the lifetime of all locals on all branches of the execution path and to emit code that clears them at the point of last use in any particular branch. I'm happy to announce I have implemented this fine-grained locals clearing in the compiler, in the 'new' branch. It should automatically cover all cases in which the code doesn't explicitly reuse the head - including non-tail usage, destructuring etc. In short, such cases should 'just work' from now on. N.B. that this is strictly a lifetime management issue and does not change the nature of lazy sequences - they are real, linked data structures, the tail of which might not yet have been created. They are most emphatically *not* ephemeral streams of values. However, with fine-grained locals clearing, they are subject to GC 'as you go', delivering the benefits of both. If you've got a pet case of incidental head-retention, please try out the 'new' branch and let me know how it goes. Thanks, Rich -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en
