Hey Brian, The fact that a reducible (as I have implemented it in the reducible-stream library) will fully realize itself if it is used as a sequence is a drawback that I'm not happy with. In the back of my mind I believe there may be a way around it, but I'm not sure, and it is still a topic of further thought/research for me. However, I think it can easily be avoided.
The point of using a reducible instead of a sequence is to invert the control so that the collection is in charge of processing itself. When you use a reducible as a sequence you are processing the collection externally, making the collection responsible only for doling itself out one item at a time, which it cannot do if it must also manage the scope for some resource (like a connection) that must be closed. I would suggest that you have more processing that needs to be pushed into the reduction process. So, instead of getting a database result set, processing it in some way, and then trying to pass it as a sequence into Ring (I'm not sure if this is exactly your problem, but I think it is representative?), where Ring will want to consume that sequence, which will trigger it to fully realize in memory---instead push the work that Ring would be doing into the reduction process. So, reduce over the result set with an output stream (or one end of a piped output stream), and in the reduction process write to the stream. Ring takes the other end of the pipe and consumes the stream to produce its result. A reducible is a delayed computation, and with transducers or by other means you can layer more delayed computation onto the reducible. As soon as you fire the reducible it will produce its entire result, whether the firing is because of a call to `reduce` or `seq`. A reducible is like a spring with potential energy built in, and when it is sprung it springs. A lazy sequence is like a wheel, if you apply external force to it, it will turn, but otherwise it is inert. Probably a terrible analogy, but that's the best I can come up with. :) I hope that's helpful. Paul http://www.realworldclojure.com/ http://paul.stadig.name/ @pjstadig On Thursday, May 4, 2017 at 3:04:40 PM UTC-4, Brian Craft wrote: > > It's definitely the same problem, but I don't think it helps me. This > part, in particular: > > "If you treat this object like a sequence, it will fully consume the input > stream and fully realize the decoded data in memory." > > I'm specifically trying to avoid realizing the full collection in memory, > because it won't fit. > > On Thursday, May 4, 2017 at 11:22:36 AM UTC-7, Josh Tilles wrote: >> >> I think the “reducible streams” approach described by Paul Stadig here >> <http://paul.stadig.name/2016/08/reducible-streams.html> has potential. >> It might not cover all of the scenarios you’re thinking of, though. >> >> On Thursday, May 4, 2017 at 1:35:48 PM UTC-4, Brian Craft wrote: >>> >>> The with-open style is used a lot in the jdbc lib, and elsewhere. It's >>> pretty simple when data is very small, as you can just evaluate the entire >>> result with doall, etc. >>> >>> How do you deal with larger data, where you need to evaluate >>> iteratively? If there's only one with-open it can be reasonably simple to >>> pass the consumer into that context (though from ring it's pretty >>> convoluted, due to needing to pass control to ring first). But if there are >>> multiple with-open you have to nest them, with a continuation passing >>> style, or middleware pattern, or something, which quickly becomes onerous >>> as it affects all the code surrounding the with-open. >>> >>> Is there some simpler pattern? >>> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
