On Jan 23, 2006, at 3:33 AM, Bulat Ziganshin wrote:
what you mean? afaik, there is no standard FastPackedString
implementation, but there is some library that with minimal
modifications used in darcs, jhc and many other apps
I considered the version at Don Stewart's web site to be the
standard, perhaps that was silly of me.
JD> still reports large amounts of memory used. Does ghc use mmap to
JD> allocate memory instead of malloc?
afaik, the answer is yes. look at ghc-6.4.1-src\ghc\rts\MBlock.c
I've been doing the bulk of my testing in linux on i386, and if I'm
reading the ifdefs correctly that is mmap'd anonymously. So at least
I would expect Ian's memory tool to report the memory allocations
correctly on that platform.
JD> I'm looking for advice or help in optimizing darcs in this
case. I
JD> guess this could be viewed as a challenge for people that felt
like
JD> the micro benchmarks of the shootout were unfair to Haskell.
Can we
JD> demonstrate that Haskell provides good performance in the real-
world
JD> when working with large files? Ideally, darcs could easily
work with
JD> a patch that is 10GB in size using only a few megs of ram if
need be
JD> and doing so in about the time it takes read the file once or
twice
JD> and gzip it.
it seems that Ian just used this as memory/time-efficient alternative
for hGetContents. reading from memory-mapped file may be done as pure
computation if the whole file is mapped. is this used in darcs?
I'm not sure, I have looked at the code but I can't tell. I think
that was the point with the mmap'd files. There are several layers
of abstraction at work here. Slurpies, PackedStrings, (custom) Lazy
Reader monad, and maybe some others.
if not, then we can map small block at the time or even just use
hGetBuf - at least, for first version. what you will say about this?
I'd expect it to be horribly slow, but if it gives guarantees about
memory usage then I bet we could tune it up and make it tolerable.
i'll be happy to help, but i'm not ready to study the darcs code. i'm
ready to solve more concrete problems - such as providing new
input mechanism with given external interface or optimizing something
rather small
That reminds me, you wrote FastIO right? I haven't looked at FastIO,
do you think it would apply here?
Thanks,
Jason
_______________________________________________
Haskell-Cafe mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/haskell-cafe
