Hello Ben, thanks for your input! I'm attaching a series of patches to wrap up what we've discussed so far, more details are in the commit messages quoted below.
I've tested the patches by running blockdev-wipe, they are looking good. I
haven't tried to build the installer with the new block-dev wipe, though,
and therefore would appreciate further testing and/or code review.
Even with the latest version of blockdev-wipe, I'm still seeing ~20%
improvement by setting min_speed to zero. Yet, I'd suggest to hold back the
speedlimit patch because I'd expect similar gains for the package
installation phase. Thus I believe that it makes more sense to set
speed_limit_min to zero during the startup of the debian installer. Could
someone please advise me as to where that would fit in best?
Cheers,
Thiemo
#5 blockdev-wipe: Set blocksize to 512k
This should be large enough to avoid excessive system call
overhead and small enough to prevent problems on systems with
very little RAM.
#4 blockdev-wipe: Sync at most once per second
Don't open output devices with O_SYNC, instead sync manually
every time the progress indicator is updated, but not more
often than once per second. This yields performance gains
of up to factor 10 in setups with dm-crypt on dm-raid.
Note: Seven years ago, O_SYNC was added to fix OOM issues
(cf. bug #381135), however it is believed that this problem
has been addressed in the kernel by now.
#3 blockdev-wipe: Allocate buffer from heap instead of stack
This allows to use buffers larger than 8M, also it fails more
gracefully in case the memory can't be allocated.
#2 blockdev-wipe: Reduce progress indicator granularity to 1/1000
On Sun, Sep 15, 2013 at 12:31 AM, Ben Hutchings <b...@decadent.org.uk> wrote:
> On Sat, 2013-09-14 at 23:33 +0200, Thiemo Nagel wrote:
> [...]
> > What I take away from this: For optimal performance, the frequency of
> > syncs should be kept low, probably well below 50 Hz, ideally as low as
> > possible. I'd be in favour of removing them altogether, but there
> > were some OOM issues seven years ago that were fixed by adding them.
> > Does anybody know whether bug 381135 still applies with today's
> > kernels?
>
> I believe Linux will throttle writing processes so that the size of
> buffered writes doesn't keep growing. And I have never seen problems
> when wiping disks with dd (I disposed of a whole bunch of old disks
> recently) without periodic sync'ing.
>
> However, the size of buffered writes can still grow quite large, which
> will limit the accuracy of a progress display. So I would suggest you
> sync whenever you're about to indicate progress, but also increase the
> granularity of progress (65536 steps is ridiculous) so that doesn't
> happen too often.
>
> > In any case, I'd suggest to go at least for 1M block size, even if
> > just to reduce the amount of system calls.
> [...]
>
> It looks like 1 MB is just about enough to maximise throughput, and this
> matches my memory.
>
> Ben.
>
> --
> Ben Hutchings
> If at first you don't succeed, you're doing about average.
>
2-blockdev-wipe-Reduce-progress-indicator-granularity-.patch
Description: Binary data
3-blockdev-wipe-Allocate-buffer-from-heap-instead-of-s.patch
Description: Binary data
4-blockdev-wipe-Sync-at-most-once-per-second.patch
Description: Binary data
5-blockdev-wipe-Set-blocksize-to-512k.patch
Description: Binary data
