Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-15 Thread Benoît Canet
> > Kevin what do you think of this ? > > I could strip down the dedupe journal code to specialize it. > > If you think it turns out easier than using the journalling > infrastructure that we're going to implement anyway, then why not. This question need to be though. The good thing about the cur

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-06 Thread Fam Zheng
On Fri, 09/06 11:57, Kevin Wolf wrote: > Am 06.09.2013 um 11:20 hat Fam Zheng geschrieben: > > On Wed, 09/04 11:39, Kevin Wolf wrote: > > > First of all, excuse any inconsistencies in the following mail. I wrote > > > it from top to bottom, and there was some thought process involved in > > > almos

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-06 Thread Fam Zheng
On Tue, 09/03 15:45, Kevin Wolf wrote: > This contains an extension of the qcow2 spec that introduces journalling > to the image format, plus some preliminary type definitions and > function prototypes in the qcow2 code. > > Journalling functionality is a crucial feature for the design of data > d

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-06 Thread Kevin Wolf
Am 06.09.2013 um 11:20 hat Fam Zheng geschrieben: > On Wed, 09/04 11:39, Kevin Wolf wrote: > > First of all, excuse any inconsistencies in the following mail. I wrote > > it from top to bottom, and there was some thought process involved in > > almost every paragraph... > > > > Am 04.09.2013 um 10

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-06 Thread Fam Zheng
On Wed, 09/04 11:39, Kevin Wolf wrote: > First of all, excuse any inconsistencies in the following mail. I wrote > it from top to bottom, and there was some thought process involved in > almost every paragraph... > > Am 04.09.2013 um 10:03 hat Stefan Hajnoczi geschrieben: > > On Tue, Sep 03, 2013

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-06 Thread Kevin Wolf
Am 05.09.2013 um 17:26 hat Benoît Canet geschrieben: > Le Thursday 05 Sep 2013 à 11:24:40 (+0200), Stefan Hajnoczi a écrit : > > On Wed, Sep 04, 2013 at 11:55:23AM +0200, Benoît Canet wrote: > > > > > I'm not sure if multiple journals will work in practice. Doesn't this > > > > > re-introduce the

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Eric Blake
On 09/05/2013 09:20 AM, Kevin Wolf wrote: > > One additional thought that might make the thing a bit more interesting: > Sequence numbers can wrap around as well. On the other hand, if sequence numbers are 64-bit, the number of operations required to cause a wrap far exceeds the expected lifetime

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Kevin Wolf
Am 05.09.2013 um 16:55 hat Stefan Hajnoczi geschrieben: > On Thu, Sep 5, 2013 at 1:18 PM, Kevin Wolf wrote: > > Am 05.09.2013 um 11:21 hat Stefan Hajnoczi geschrieben: > >> On Wed, Sep 04, 2013 at 11:39:51AM +0200, Kevin Wolf wrote: > >> > > > +A journal is organised in journal blocks, all of whic

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Stefan Hajnoczi
On Thu, Sep 5, 2013 at 1:18 PM, Kevin Wolf wrote: > Am 05.09.2013 um 11:21 hat Stefan Hajnoczi geschrieben: >> On Wed, Sep 04, 2013 at 11:39:51AM +0200, Kevin Wolf wrote: >> > However, what if we run 'qemu-img check -r leaks' with an old qemu-img >> > version? It will reclaim the clusters used by

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Benoît Canet
Le Thursday 05 Sep 2013 à 11:24:40 (+0200), Stefan Hajnoczi a écrit : > On Wed, Sep 04, 2013 at 11:55:23AM +0200, Benoît Canet wrote: > > > > I'm not sure if multiple journals will work in practice. Doesn't this > > > > re-introduce the need to order update steps and flush between them? > > > > >

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Benoît Canet
> Then you get very quickly alternating sequences of "L2 depends on > refcount update" (for allocation) and "refcount update depends on L2 > update" (for freeing), which means that Qcow2Cache starts flushing all > the time without accumulating many requests. These are cases that would > benefit as

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Kevin Wolf
Am 05.09.2013 um 11:35 hat Stefan Hajnoczi geschrieben: > Although we are still discussing details of the on-disk layout, the > general design is clear enough to discuss how the journal will be used. > > Today qcow2 uses Qcow2Cache to do lazy, ordered metadata updates. The > performance is pretty

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Kevin Wolf
Am 05.09.2013 um 11:21 hat Stefan Hajnoczi geschrieben: > On Wed, Sep 04, 2013 at 11:39:51AM +0200, Kevin Wolf wrote: > > However, what if we run 'qemu-img check -r leaks' with an old qemu-img > > version? It will reclaim the clusters used by the journal, and if we > > continue using the journal we

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Stefan Hajnoczi
On Tue, Sep 03, 2013 at 03:45:52PM +0200, Kevin Wolf wrote: > This contains an extension of the qcow2 spec that introduces journalling > to the image format, plus some preliminary type definitions and > function prototypes in the qcow2 code. > > Journalling functionality is a crucial feature for t

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Stefan Hajnoczi
On Wed, Sep 04, 2013 at 11:55:23AM +0200, Benoît Canet wrote: > > > I'm not sure if multiple journals will work in practice. Doesn't this > > > re-introduce the need to order update steps and flush between them? > > > > This is a question for Benoît, who made this requirement. I asked him > > the

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-05 Thread Stefan Hajnoczi
On Wed, Sep 04, 2013 at 11:39:51AM +0200, Kevin Wolf wrote: > First of all, excuse any inconsistencies in the following mail. I wrote > it from top to bottom, and there was some thought process involved in > almost every paragraph... I should add this disclaimer to all my emails ;-). > Am 04.09.2

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-04 Thread Benoît Canet
> > I'm not sure if multiple journals will work in practice. Doesn't this > > re-introduce the need to order update steps and flush between them? > > This is a question for Benoît, who made this requirement. I asked him > the same a while ago and apparently his explanation made some sense to > me

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-04 Thread Kevin Wolf
Am 04.09.2013 um 10:32 hat Max Reitz geschrieben: > On 2013-09-03 15:45, Kevin Wolf wrote: > >This contains an extension of the qcow2 spec that introduces journalling > >to the image format, plus some preliminary type definitions and > >function prototypes in the qcow2 code. > > > >Journalling func

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-04 Thread Kevin Wolf
First of all, excuse any inconsistencies in the following mail. I wrote it from top to bottom, and there was some thought process involved in almost every paragraph... Am 04.09.2013 um 10:03 hat Stefan Hajnoczi geschrieben: > On Tue, Sep 03, 2013 at 03:45:52PM +0200, Kevin Wolf wrote: > > @@ -103,

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-04 Thread Benoît Canet
> > +They consist of transactions, which in turn contain operations that > > +are effectively executed atomically. A qcow2 image can have a main image > > +journal that deals with cluster management operations, and additional > > specific > > +journals can be used by other features like data dedup

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-04 Thread Max Reitz
On 2013-09-03 15:45, Kevin Wolf wrote: This contains an extension of the qcow2 spec that introduces journalling to the image format, plus some preliminary type definitions and function prototypes in the qcow2 code. Journalling functionality is a crucial feature for the design of data deduplicati

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-04 Thread Stefan Hajnoczi
On Tue, Sep 03, 2013 at 03:45:52PM +0200, Kevin Wolf wrote: > @@ -103,7 +107,11 @@ in the description of a field. > write to an image with unknown auto-clear features if it > clears the respective bits from this field first. > > -Bits

Re: [Qemu-devel] [RFC] qcow2 journalling draft

2013-09-03 Thread Benoît Canet
Le Tuesday 03 Sep 2013 à 15:45:52 (+0200), Kevin Wolf a écrit : > This contains an extension of the qcow2 spec that introduces journalling > to the image format, plus some preliminary type definitions and > function prototypes in the qcow2 code. > > Journalling functionality is a crucial feature f

[Qemu-devel] [RFC] qcow2 journalling draft

2013-09-03 Thread Kevin Wolf
This contains an extension of the qcow2 spec that introduces journalling to the image format, plus some preliminary type definitions and function prototypes in the qcow2 code. Journalling functionality is a crucial feature for the design of data deduplication, and it will improve the core part of