Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-29 Thread Luis Pedro Coelho
Hello all, On Monday 29 December 2008 17:40:07 Gael Varoquaux wrote: > It is interesting to see that you take a slightly different approach than > the others already discussed. This probably stems from the fact that you > are mostly interested by parallelism, whereas there are other adjacent > pro

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-29 Thread Gael Varoquaux
Hi Luis, On Mon, Dec 29, 2008 at 02:51:48PM -0500, Luis Pedro Coelho wrote: > I coincidently started my own implementation of a system to manage > intermediate results last week, which I called jug. I wasn't planning > to make such an alpha version public just now, but it seems to be on > topic.

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-29 Thread Zachary Pincus
This looks really cool -- thanks Luis. Definitely keep us posted as this progresses, too. Zach On Dec 29, 2008, at 4:41 PM, Luis Pedro Coelho wrote: > On Monday 29 December 2008 14:51:48 Luis Pedro Coelho wrote: >> I will make the git repository publicly available once I figure out >> how to

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-29 Thread Luis Pedro Coelho
On Monday 29 December 2008 14:51:48 Luis Pedro Coelho wrote: > I will make the git repository publicly available once I figure out how to > do that. You can get my code with: git clone http://coupland.cbi.cmu.edu/jug As I said, I consider this alpha code and am only making it publicly available

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-29 Thread Luis Pedro Coelho
Hello, I coincidently started my own implementation of a system to manage intermediate results last week, which I called jug. I wasn't planning to make such an alpha version public just now, but it seems to be on topic. The main idea is to use hashes to map function arguments to paths on the f

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-27 Thread Gael Varoquaux
On Sat, Dec 27, 2008 at 04:59:25PM +0100, Bradford Cross wrote: >I prototyped an approach last year that worked out well. I don't really >know what to call it - maybe something like "property based persistence." >It is kind of strange and I am not sure how broadly applicable it is - I >

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-27 Thread Bradford Cross
I prototyped an approach last year that worked out well. I don't really know what to call it - maybe something like "property based persistence." It is kind of strange and I am not sure how broadly applicable it is - I have only used it for financial time series data. I'll try to explain how the

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-24 Thread Gael Varoquaux
On Tue, Dec 23, 2008 at 02:10:50AM +0100, Olivier Grisel wrote: >Interesting topic indeed. I think I have been hit with similar problems on >toy experimental scripts. So far the solution was always adhoc FS caches >of numpy arrays with manual filename management. Maybe the first step fo

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-22 Thread Olivier Grisel
Interesting topic indeed. I think I have been hit with similar problems on toy experimental scripts. So far the solution was always adhoc FS caches of numpy arrays with manual filename management. Maybe the first step for designing a generic solution would be to list some representative yet simple

[Numpy-discussion] Thoughts on persistence/object tracking in scientific code

2008-12-22 Thread Gael Varoquaux
Hi, This mailing list is full of people spending their time writing non-trivial numerical code. This is why I would like to share my interrogations on a code smell that I notice a lot in my numerical code that revolves around persisting to disk often, and the mess that results. It is a bit hard to