Re: [Qemu-devel] drive-mirroring to nbd is failing with multiple parallel jobs (qemu 2.9 -> 2.11)

2018-02-16 Thread Alex Bligh
t's > something you can configure on your system if you need it to be lower. +1 for just using SO_KEEPALIVE. I think I even submitted some (untested and thus unmerged) patches for this. -- Alex Bligh

Re: [Qemu-devel] [PATCH 1/1] nbd: increase maximum size of the PWRITE_ZERO request

2018-02-10 Thread Alex Bligh
> On 10 Feb 2018, at 18:43, Alex Bligh wrote: > > So I think a reasonable logic for Qemu would be to try NBD_CMD_INFO and find > the maximum write size, and if that's unsupported use 0x (capping at > export size, or export size minus write offset). Ur actually

Re: [Qemu-devel] [PATCH 1/1] nbd: increase maximum size of the PWRITE_ZERO request

2018-02-10 Thread Alex Bligh
BD_CMD_WRITE_ZEROES because it doesn't have a payload, I think the rebuttal is that a server which supports NBD_CMD_WRITE of a given length must also support NBD_CMD_WRITE_ZEROES of that length. So I think a reasonable logic for Qemu would be to try NBD_CMD_INFO and find the maximum write size, and if that's unsupported use 0x (capping at export size, or export size minus write offset). -- Alex Bligh

Re: [Qemu-devel] [RFH PATCH 00/10] i386: hvf: miscellaneous cleanups

2017-10-03 Thread Alex Bligh
> On 3 Oct 2017, at 15:45, Paolo Bonzini wrote: > > Alex (both of them), Sergio, anyone else who can help? Very interested in this (and thanks!) but it will be a while before I have a sensible number of cycles available to play with this one again. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [Qemu-block] How to online resize qemu disk with nbd protocol?

2017-01-23 Thread Alex Bligh
citly say people should know the implications of using extensions in shipping code - specifically that the specification may change! -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2017-01-20 Thread Alex Bligh
avily. I'm not sure sending 4,096 items for an empty 16TB disk is any great hardship to be honest. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [Qemu-block] How to online resize qemu disk with nbd protocol?

2017-01-14 Thread Alex Bligh
> On 14 Jan 2017, at 14:48, Wouter Verhelst wrote: > > On Thu, Jan 12, 2017 at 06:56:42PM +0000, Alex Bligh wrote: >> My preferred way to do this would be essentially to allow NBD_OPT_INFO >> to be sent (wrapped appropriately) during the main transmission phase. >>

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2017-01-13 Thread Alex Bligh
> On 13 Jan 2017, at 09:48, Vladimir Sementsov-Ogievskiy > wrote: > > 12.01.2017 16:11, Alex Bligh wrote: >>> On 12 Jan 2017, at 07:05, Vladimir Sementsov-Ogievskiy >>> wrote: >>> >>> Yes this is better. But is it actually needed to force

Re: [Qemu-devel] [Qemu-block] [Nbd] How to online resize qemu disk with nbd protocol?

2017-01-12 Thread Alex Bligh
NFO stuff to be reread. If it's just this (rather than a resize command) I guess it could go into the NBD_OPT_INFO extension branch. If it's going to do more than _INFO_, then I guess it needs its own extension. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] How to online resize qemu disk with nbd protocol?

2017-01-12 Thread Alex Bligh
mand, let alone a resize command. The size of an NBD disk is (currently) entirely in the hands of the server. What I think we'd really need would be a 'reread size' command, and have the server capable of supporting resizing. That would then work for readonly images too. -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2017-01-12 Thread Alex Bligh
>> replies followed by NBD_REP_ACK.: > NBD_REP_ERR_UNSUP for the case in braces? Should it be normal option reply > packet? Thanks, fixed. -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2017-01-12 Thread Alex Bligh
#x27;ve changed this to: of the file), a server MAY reply with a single block status descriptor with *length* matching the requested length, rather than reporting the error; in this case the context MAY mandate the status returned. -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2017-01-11 Thread Alex Bligh
is Wouter's wording, but I think 'a status appropriate to the context' would be better. Each context then needs to define what that is. Either that or 'the context's default status' and that should be in the definition of the context. -- Alex Bligh

Re: [Qemu-devel] implementing architectural timers using QEMU timers

2017-01-09 Thread Alex Bligh
ing timerlistgroup_deadline_ns rather than qemu_clock_deadline_ns_all. The comment at the top of qemu_clock_deadline_ns_all says: /* Calculate the soonest deadline across all timerlists attached * to the clock. This is used for the icount timeout so we * ignore whether or not the clock should be used in deadline * calculations. */ I'm wondering whether there is something up with that logic. -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2016-12-29 Thread Alex Bligh
eady have a read loop. If we don't then the 'one chunk' method is perfectly acceptable. If I implemented structured replies at all, I'd be sharing the code between multiple users in any case. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-29 Thread Alex Bligh
ore metadata > contexts with the NBD_OPT_SET_META_CONTEXT command. If needed, the > client can use NBD_OPT_LIST_META_CONTEXT to list contexts that the > server supports. That still seems correct. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-29 Thread Alex Bligh
sn't help much to add a flag here. I agree. Vladimir: if this isn't clear from the text, please suggest a change. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-17 Thread Alex Bligh
arguments, but > they'd have to be very good ones ;-) But we were at cross purposes. I was talking about the return from queries, and you've adjusted namespaces (in toto). I've fixed up your edit slightly, and brought the return for queries in line with it. I hope this nit is now dead! -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-16 Thread Alex Bligh
> On 16 Dec 2016, at 15:52, Wouter Verhelst wrote: > > On Thu, Dec 15, 2016 at 05:34:48PM +0000, Alex Bligh wrote: >> >>> On 15 Dec 2016, at 16:49, Wouter Verhelst wrote: >>> >>>> Because the namespaces and leaf-names are already restricted to >

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-15 Thread Alex Bligh
idea. So to be clear do you want to include all whitespace everywhere? Or just to the right of the colon in queries? -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-15 Thread Alex Bligh
t; > Why restrict to non-whitespace characters? (printable makes sense...) Because the namespaces and leaf-names are already restricted to non-whitespace characters. I thought having tabs, line feeds, returns, em-space, en-space etc. was not particularly useful. I could be persuaded to relent re spaces. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-15 Thread Alex Bligh
255 bytes). This may be helpful where a client can construct algorithmic queries. For instance, a client might reply simply with the namespace with no leaf-name (e.g. 'X-FooBar:') or with a range of values (e.g. 'X-ModifiedDate:20160310-20161214'). The semantics of such a reply are a matter for the definition of the namespace. Hope that works. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
the spec for that and know that it has X-Backup:modified if so'. So I've suggested it return 'X-Backup:' only in that case, in which case from that (*and the spec*) you know how to build any query. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
> On 14 Dec 2016, at 18:18, Alex Bligh wrote: > > Let me have a go at a change. OK I've pushed a change. I reordered a few bits (to put the base:allocation next to the stuff that talks about metadata queries at the start), but the main change is below. -- Alex Bligh @@ -9

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
good. Let me have a go at a change. However, note that Vladimir was answering a slightly different question. I was asking about listing ALL contexts (previously an empty query string, now a zero length list of queries), not a 'backup:*' type thing. -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
ro length list of queries) does. It's currently defined to return every context from every namespace. Options would include * ETOOBIG * Listing some subset of available contexts (arguably 'none' is a subset) * Allowing abbreviation of algorithmically specified contexts (e.g. in this case just returning 'backup:' to represent all available backup contexts) -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
a "REQ_ONE" flag and a length field of zero could be an > interesting way to enumerate a whole export, too. Essentially, we could > define that as a client saying "I'm interested in what the size of the > extent at offset X is, and what its properties are". Also +1 -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
#x27; or similar. So that's less of a problem. And 'set all' doesn't exist (unlike 'list all'). I think in the LIST case we should allow the server to omit contexts under certain circumstances, and allow SET to return ETOOBIG. -- Alex Bligh

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
> On 14 Dec 2016, at 16:58, Eric Blake wrote: > > s/botht he/both the/ Thanks - fixed -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [Qemu-block] [PATCH] doc: Propose NBD_FLAG_INIT_ZEROES extension

2016-12-14 Thread Alex Bligh
sal) is actually fine. #1 seems too heavy #2 also seems pretty heavyweight - adding a whole new info command for one bit #3 is pretty simple, but carries the disadvantage that you won't be able to provide a reference implementation without also putting NBD_OPT_GO support into the reference implementation. Oh hang on, perhaps that's an advantage :-) So I'd either go with #0 or #3. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
ich might (in theory) return a list of blocks which are newer than the given timestamp. It would clearly be impossible to return all such contexts. I wonder if we should carve out an exception here. -- Alex Bligh

[Qemu-devel] [PATCH] Further tidy-up on block status

2016-12-14 Thread Alex Bligh
ed to zero length query) list all contexts, as absence of any query is now simple. * Move definition of namespaces in the document to somewhere more appopriate. * Various other minor changes as discussed on the mailing list Signed-off-by: Alex Bligh --- doc/proto.md

Re: [Qemu-devel] [Nbd] [PATCH v5] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-13 Thread Alex Bligh
till not be an > error for the server to return ENOSPC) All of this suggests 'SHOULD NOT' would be more appropriate than 'MUST NOT'. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH v5] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-13 Thread Alex Bligh
t it does not affect that? How about: "If a server supports the `base:allocation` metadata context, then writing to an extent which has `NBD_STATE_HOLE` clear MUST NOT fail with ENOSPC unless for reasons specified in the definition of another context." >>> +For the `base:allocation` context, the remainder of the flags field is >>> +reserved. Servers SHOULD set it to all-zero; >> >> Surely if we want to reserve them for extension, we need "Servers >> MUST set it to all-zero" > > No, SHOULD, otherwise a future extension which adds meaning to those > bits will suddenly become incompatible with this spec. Think about it > ;-) I did! If there is a future extension, it will change the spec to incorporate those bits, so they won't be included within 'the remainder' any more. > (feel free to update the branch with those suggestions I've not NAK'd, > as I think they make sense...) OK. May take a look later. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH v5] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-13 Thread Alex Bligh
think the easy way to do this would be to have context IDs as a payload to the command, like they are with NBD_CMD_WRITE, but the payload is currently always defined to be of the length specified within the length section. The question really is whether we should fix this silly protocol limitation. I don't think it's as bad as the structured reply fix, and could conceivably go in there. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH v5] doc: Add NBD_CMD_BLOCK_STATUS extension (part 2)

2016-12-13 Thread Alex Bligh
l > + namespaces by simple request to the mailinglist. Surely also we need to specify multiple queries? -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH v5] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-13 Thread Alex Bligh
> +status of 0 for any block, although the server SHOULD return > +additional status values when they can be easily detected. > + > +If an error occurs, the server SHOULD set the appropriate error > +code in the error field of an error chunk. However, if the error > +does not involve invalid usage (such as a request beyond the bounds > +of the file), a server MAY reply with a single block status > +descriptor with *length* matching the requested length, and *status* > +of 0 rather than reporting the error. > + > +A client MAY initiate a hard disconnect if it detects that the > +server has sent an invalid chunk. The server SHOULD return `EINVAL` > +if it receives a `NBD_CMD_BLOCK_STATUS` request including one or > +more sectors beyond the size of the device. > + > * Other requests > > Some third-party implementations may require additional protocol > > -- > < ron> I mean, the main *practical* problem with C++, is there's like a dozen > people in the world who think they really understand all of its rules, > and pretty much all of them are just lying to themselves too. > -- #debian-devel, OFTC, 2016-02-12 > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! > http://sdm.link/slashdot___ > Nbd-general mailing list > nbd-gene...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nbd-general -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [Qemu-block] [PATCH] doc: Propose NBD_FLAG_INIT_ZEROES extension

2016-12-13 Thread Alex Bligh
d, where space is > at a premium. I did suggest a few non-Qemu uses (see below). I think it might be an idea if the reference implementation supported it before merging (which per below should be trivial). -- Alex Bligh > Begin forwarded message: > > From: Alex Bligh &g

Re: [Qemu-devel] [Nbd] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-08 Thread Alex Bligh
t when TRIM cannot guarantee reads-as-zero. Yes. It was actually exactly that discussion I was trying to remember. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-08 Thread Alex Bligh
uld read as > all-zeroes) is not an invalid thing for a server to set. The spec here > clarifies what a client should do with that information if it gets it > (i.e., "don't read it, it doesn't contain anything interesting"). That's fair enough until the last bit in brackets. Rather than saying a client SHOULD NOT read it, it should simply say that a read on such areas will succeed but the data read is undefined (and may not be stable). > My WIP patch moves this out from the (older) "BLOCK_STATUS extension" > section and into the main body of the spec. It also makes a few changes > in wording as per what Vladimir suggested, and I was working on an > NBD_OPT_LIST_META_CONTEXT rather than an NBD_OPT_META_CONTEXT > negotiation option, with the idea that I'd add an OPT_ADD_META_CONTEXT > and an OPT_DEL_META_CONTEXT later. Your idea of using a SET has merit > though, so I'll update it to that effect. > > It already removed the two bits that BASE:allocation doesn't use, and > makes a few other changes as well. I haven't had the time to finish it > and send it out for review though, but I'll definitely include your > comments now. Thanks. -- Alex Bligh

Re: [Qemu-devel] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-08 Thread Alex Bligh
too and a lot of uppercase will come to the code =( I agree -- Alex Bligh

Re: [Qemu-devel] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-07 Thread Alex Bligh
> On 2 Dec 2016, at 18:45, Alex Bligh wrote: > > Thanks. That makes sense - or enough sense for me to carry on commenting! I finally had some time to go through this extension in detail. Rather than comment on all the individual patches, I squashed them into a single commit, did a &#

Re: [Qemu-devel] [Nbd] [PATCH] doc: Propose NBD_FLAG_INIT_ZEROES extension

2016-12-06 Thread Alex Bligh
> On 6 Dec 2016, at 08:46, Alex Bligh wrote: > > I would support this. > > In fact the patch is sufficiently simple I think I'd merge this > into extension-write-zeroes then merge that into master. Hence: Reviewed-By: Alex Bligh -- Alex Bligh

Re: [Qemu-devel] [Nbd] [Qemu-block] [PATCH] doc: Propose NBD_FLAG_INIT_ZEROES extension

2016-12-06 Thread Alex Bligh
ssue passing complete zero status back to the client if it's so obvious from a stat(). -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] doc: Propose NBD_FLAG_INIT_ZEROES extension

2016-12-06 Thread Alex Bligh
s Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today.http://sdm.link/xeonphi > ___ > Nbd-general mailing list > nbd-gene...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nbd-general > -- Alex Bligh

Re: [Qemu-devel] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-03 Thread Alex Bligh
I tried to pick some QEMU-like ones, but I am sure there are examples that would work outside of QEMU. -- Alex Bligh

Re: [Qemu-devel] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-12-02 Thread Alex Bligh
at all (well, that's my excuse anyway). > Anyway, I hope I am being useful and just not more confounding. It seems > to me that we're having difficulty conveying precisely what it is we're > trying to accomplish, so I hope that I am making a good effort in > elaborating on our goals/requirements. Yes absolutely. I think part of the challenge is that you are quite reasonably coming at it from the point of view of qemu's particular need, and I'm coming at it from 'what should the nbd protocol look like in general' position, having done lots of work on the protocol docs (though I'm an occasional qemu contributor). So there's necessarily a gap of approach to be bridged. I'm overdue on a review of Wouter's latest patch (partly because I need to re-diff it against the version with no NBD_CMD_BLOCK_STATUS in), but I think it's a bridge worth building. -- Alex Bligh

Re: [Qemu-devel] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-11-29 Thread Alex Bligh
16TB disk, as it might be very large! Even if the client worked at e.g. a 64MB level (where they'd get a maximum of 1024 extents per reply), this isn't going to noticeably increase the round trip timing. One issue here is that to determine a 'reasonable' size, the client needs to know the minimum length of any extent. I think the answer is probably a 'maximum number of extents' in the request packet. Of course with statuses in extent, the final extent could be represented as 'I don't know, break this bit into a separate request' status. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-11-29 Thread Alex Bligh
ow a hole) might need backing up). So don't we need multiple independent lists of extents? Of course a server might *implement* them under the hood with separate bitmaps or one big bitmap, or no bitmap at all (for instance using file extents on POSIX). -- Alex Bligh

Re: [Qemu-devel] [PATCH v3] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-11-29 Thread Alex Bligh
er from the server refusing to provide information? We don't permit short reads etc. > . The server SHOULD use different > +*status* values between consecutive descriptors, and SHOULD use > +descriptor lengths that are an integer multiple of 512 bytes where > +possible (the first and last descriptor of an unaligned query being > +the most obvious places for an exception). Surely better would be an an integer multiple of the minimum block size. Being able to offer bitmap support at finer granularity than the absolute minimum block size helps no one, and if it were possible to support a 256 byte block size (I think some floppy disks had that) I see no reason not to support that as a granularity. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] proto: add 'shift' extension.

2016-09-27 Thread Alex Bligh
> On 27 Sep 2016, at 00:41, Wouter Verhelst wrote: > > Thoughts? Honestly? This whole thing seems like complication for little gain. One command per 2GB wiped doesn't seem like a bad thing. -- Alex Bligh

Re: [Qemu-devel] [PATCH] proto: add 'shift' extension.

2016-09-26 Thread Alex Bligh
do by mistake! This whole 'clear the whole disk in one command' seems like a dire case of premature optimisation. -- Alex Bligh

Re: [Qemu-devel] [PATCH] proto: add 'shift' extension.

2016-09-26 Thread Alex Bligh
s > + not specified. If after shift `(offset + length)` exceeds disk size, length > + should be reduced to `( - offset)`. However, `(offset + length)` > + must not exceed disk size by more than `(1 << N) - 1`. Is there a good reason the shift size can't be either the minimum block size or otherwise negotiated using NBD_OPT_INFO? -- Alex Bligh

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
ou not to use more than 2^31). And in any case, you probably want to parallelise reads and writes and have more than one write in flight in any case, all of which suggests you are going to be breaking up requests anyway. -- Alex Bligh

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
> On 24 Sep 2016, at 18:13, Vladimir Sementsov-Ogievskiy > wrote: > > On 24.09.2016 19:49, Alex Bligh wrote: >>> On 24 Sep 2016, at 17:42, Vladimir Sementsov-Ogievskiy >>> wrote: >>> >>> On 24.09.2016 19:31, Alex Bligh wrote: >>>

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
> On 24 Sep 2016, at 17:52, Alex Bligh wrote: > > In *your* use-case holes may be desirable. However in the general case, you > cannot assume a server supports holes. Optional support for holes isn't even > in the mainline spec yet (AFAIR). You should also be aw

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
gy, you cannot guarantee that the server will not implement them by ignoring NBD_CMD_TRIM (this is perfectly acceptable as NBD_CMD_TRIM is advisory only) and making NBD_CMD_WRITE_ZEREOS do a long write of zeroes. IE there is no way to detect whether the server actually supports holes. -- Alex Bligh

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
> On 24 Sep 2016, at 17:42, Vladimir Sementsov-Ogievskiy > wrote: > > On 24.09.2016 19:31, Alex Bligh wrote: >>> On 24 Sep 2016, at 13:06, Vladimir Sementsov-Ogievskiy >>> wrote: >>> >>> Note: if disk size is not aligned to X we will have t

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
ou can pretty much assume that a server that supports holes will write holes. A server that does not support holes will write zeroes. If you don't care whether the resultant data is zero, just use NBD_CMD_TRIM. But as you do care (see above) you must be prepared for a 'thick' write of zeroes on servers that don't support it. -- Alex Bligh

Re: [Qemu-devel] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
multiple of the minimum block size. So that would work. -- Alex Bligh

Re: [Qemu-devel] [Nbd] write_zeroes/trim on the whole disk

2016-09-24 Thread Alex Bligh
ole disk" commands. We should probably make it an illegal > flag for any command that actually sends data over the wire, though. I'd prefer an approach like this. Perhaps X could be negotiated with the block size extension (or simply be defined as the 'preferred block size'. This could then be defined to work with all commands. -- Alex Bligh

Re: [Qemu-devel] kvm_arch_put_registers and xsave

2016-06-19 Thread Alex Bligh
rs and FPState (for instance) appear to be in an opaque format: https://developer.apple.com/library/mac/documentation/Hypervisor/Reference/Hypervisor_Reference/index.html#//apple_ref/c/func/hv_vcpu_read_fpstate -- Alex Bligh

[Qemu-devel] kvm_arch_put_registers and xsave

2016-06-19 Thread Alex Bligh
XRSTOR (by writing the XSAVE region with the appropriate hv_ call) and ignore all the other register futzing? Or is it more complicated than that because (presumably) something sets up register states in the cpu->regs, cpu->sregs areas? I'm a bit confused as to how all this works to be honest. Any ideas / docs to point to? -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH v4 04/11] nbd: Improve server handling of bogus commands

2016-06-15 Thread Alex Bligh
, then not sending any payload, which is a nonsense. Perhaps this should read "If an error occurs, the server MUST either initiate a hard disconnect before the entire payload has been sent or set the appropriate code in the error field and send the response header without any payload." if we want to go down this route. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH v4 04/11] nbd: Improve server handling of bogus commands

2016-06-15 Thread Alex Bligh
yload. > > In other words, the current behaviour of qemu is correct, is now > documented to be correct, and should not be changed. So what should those servers do (like 2 of mine) which don't buffer the entire read, if they get an error having already sent some data? -- Alex Bligh

Re: [Qemu-devel] [PATCH v4 04/11] nbd: Improve server handling of bogus commands

2016-06-14 Thread Alex Bligh
been reasonably well known (I wrote about it at least 3 years ago), that the current implementation (reference + kernel) does not cope well with errors on reads, so I'm guessing one is just trading one set of brokenness for another. So I'm pretty relaxed about what goes in qemu. -- Alex Bligh

Re: [Qemu-devel] [PATCH v4 04/11] nbd: Improve server handling of bogus commands

2016-06-14 Thread Alex Bligh
On 14 Jun 2016, at 14:32, Paolo Bonzini wrote: > > On 13/06/2016 23:41, Alex Bligh wrote: >> That's one of the reasons that there is a proposal to add >> STRUCTURED_READ to the spec (although I still haven't had time to >> implement that for qemu), so that we

Re: [Qemu-devel] [PATCH v4 04/11] nbd: Improve server handling of bogus commands

2016-06-13 Thread Alex Bligh
t broken way to do things (IIRC). This is actually what nbd-server.c does in the threaded handler: https://github.com/yoe/nbd/blob/master/nbd-server.c#L1468 For amusement value, the non-threaded handler (which is not used any more) does not send any payload on an error: https://github.com/yoe

Re: [Qemu-devel] OS-X hypervisor.framework support

2016-06-12 Thread Alex Bligh
On 6 Jun 2016, at 13:29, Alex Bligh wrote: > Is anyone working on support for hypervisor.framework (OS-X's equivalent to > kvm as far as I can see)? > > If not, I might have a go in my copious spare time (cough) and if anyone > would be interesting in helping, or giving

Re: [Qemu-devel] OS-X hypervisor.framework support

2016-06-06 Thread Alex Bligh
Peter, On 6 Jun 2016, at 13:43, Peter Maydell wrote: > Not that I am aware of. > > Previous discussion: > https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg03086.html Thanks. Missed that due to "Framwork" spelling! -- Alex Bligh

[Qemu-devel] OS-X hypervisor.framework support

2016-06-06 Thread Alex Bligh
xhyve has support for this, and its license is described as "BSD" (that's it). Is that Qemu GPL compatible? Though I suspect a clean implementation may be easier. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PULL 23/28] nbd: always query export list in fixed new style protocol

2016-05-17 Thread Alex Bligh
we need a specific number, but I think a one sentence statement that servers are in general permitted to disconnect clients which are behaving in a denial of service attack type manner would be useful. Given nbdkit only ever has one export, it seems eminently reasonable for it to use a lower num

Re: [Qemu-devel] [Nbd] [PULL 23/28] nbd: always query export list in fixed new style protocol

2016-05-17 Thread Alex Bligh
fix it, so I think we should fix the software to confirm to the standard, not the standard to conform to the software. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PULL 23/28] nbd: always query export list in fixed new style protocol

2016-05-17 Thread Alex Bligh
On 17 May 2016, at 16:54, Richard W.M. Jones wrote: > On Tue, May 17, 2016 at 04:22:06PM +0100, Alex Bligh wrote: >> nbdkit is non-compliant in that case. Support of NBD_OPT_LIST is >> compulsory, even if you support it by returning a nameless export >> (or default). Moreo

Re: [Qemu-devel] [Nbd] [PULL 23/28] nbd: always query export list in fixed new style protocol

2016-05-17 Thread Alex Bligh
On 17 May 2016, at 16:52, Eric Blake wrote: > On 05/17/2016 09:22 AM, Alex Bligh wrote: > >>>> nbdkit's implemention of NBD_OPT_LIST returns an error, because there >>>> is no such thing as a list of export names supported (in effect nbdkit >>>>

Re: [Qemu-devel] [Nbd] [PULL 23/28] nbd: always query export list in fixed new style protocol

2016-05-17 Thread Alex Bligh
failing would be 'this server doesn't have anything it can export'. >> Naturally the protocol document >> (https://github.com/yoe/nbd/blob/master/doc/proto.md) isn't clear on >> this case. > > You're right that we may also want to tweak the NBD protocol to make > this interoperability point obvious. I can't actually see the issue here. It explains what needs to be implemented by the server, and that includes NBD_OPT_LIST. Very happy to add some clarity, but I'm not sure where it's needed. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-12 Thread Alex Bligh
On 11 May 2016, at 22:06, Wouter Verhelst wrote: > On Tue, May 10, 2016 at 04:08:50PM +0100, Alex Bligh wrote: >> What surprises me is that a release kernel is using experimental >> NBD extensions; there's no guarantee these won't change. Or does >> fstrim work

Re: [Qemu-devel] [PATCH v4 00/11] nbd: tighter protocol compliance

2016-05-12 Thread Alex Bligh
On 11 May 2016, at 23:39, Eric Blake wrote: > Fix several corner-case bugs in our implementation of the NBD > protocol, both as client and as server. I thought I'd added a Reviewed-By: line to more of these before. On a very very quick look, they all look good to me. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-12 Thread Alex Bligh
On 11 May 2016, at 22:12, Wouter Verhelst wrote: > On Tue, May 10, 2016 at 04:38:29PM +0100, Alex Bligh wrote: >> On 10 May 2016, at 16:29, Eric Blake wrote: >>> So the kernel is currently one of the clients that does NOT honor block >>> sizes, and as such, servers

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-11 Thread Alex Bligh
nal). ... at this rate you'd be better with a list of command / maximum size tuples! -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-10 Thread Alex Bligh
irrelevant to your current problem, which is how this would work /with/ the block size extensions, as Eric is in the process of implementing them). > I do agree with your point number 3, obviously if the lenght > field type doesn't allow something bigger than a u32, then the kernel has > to do some breaking up in that case. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-10 Thread Alex Bligh
man7.org/linux/man-pages/man1/fallocate.1.html As it takes length and offset in TB, PB, EB, ZB and YB, it seems to be 64 bit aware :-) -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-10 Thread Alex Bligh
st some breaking up. -- Alex Bligh

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-10 Thread Alex Bligh
eed to do *some* breaking up of requests. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-10 Thread Alex Bligh
Or does >> fstrim work some other way? > > No extension in play. The kernel is obeying NBD_FLAG_SEND_TRIM, which > is in the normative standard, and unrelated to the INFO/BLOCK_SIZE > extensions. My mistake. I was confusing 'WRITE_ZEROES' with 'TRIM'. -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [Nbd] [PATCH] nbd: fix trim/discard commands with a length bigger than NBD_MAX_BUFFER_SIZE

2016-05-10 Thread Alex Bligh
R_SIZE); >> rc = -EINVAL; >> > > -- > Eric Blake eblake redhat com+1-919-301-3266 > Libvirt virtualization library http://libvirt.org > > -- > Mobile security can be enabling,

Re: [Qemu-devel] [Nbd] question on ioctl NBD_SET_FLAGS

2016-05-04 Thread Alex Bligh
3.9 and later servers. I'm going to find a minimal patch to nbd-client and offer that to Ubuntu / Debian. This message is here in part so I have something to point them at on the mailing list :-) -- Alex Bligh

Re: [Qemu-devel] [PATCH v3 36/44] nbd: Improve handling of shutdown requests

2016-04-25 Thread Alex Bligh
on't play nicely with all the other stuff you've been doing). -- Alex Bligh signature.asc Description: Message signed with OpenPGP using GPGMail

Re: [Qemu-devel] [PATCH v3 44/44] nbd: Implement NBD_OPT_BLOCK_SIZE on client

2016-04-25 Thread Alex Bligh
LOG("Export size %" PRId64 " too large for 32-bit kernel", > info->size); > return -E2BIG; > } > @@ -724,18 +784,18 @@ int nbd_init(int fd, QIOChannelSocket *sioc, > NbdExportInfo *info) > return -serrno; > } > > -TRACE("Setting block size to %lu", (unsigned long)BDRV_SECTOR_SIZE); > +TRACE("Setting block size to %lu", sector_size); > > -if (ioctl(fd, NBD_SET_BLKSIZE, (unsigned long)BDRV_SECTOR_SIZE) < 0) { > +if (ioctl(fd, NBD_SET_BLKSIZE, sector_size) < 0) { > int serrno = errno; > LOG("Failed setting NBD block size"); > return -serrno; > } > > TRACE("Setting size to %lu block(s)", sectors); > -if (size % BDRV_SECTOR_SIZE) { > -TRACE("Ignoring trailing %d bytes of export", > - (int) (size % BDRV_SECTOR_SIZE)); > +if (info->size % sector_size) { > +TRACE("Ignoring trailing %" PRId64 " bytes of export", > + info->size % sector_size); > } > > if (ioctl(fd, NBD_SET_SIZE_BLOCKS, sectors) < 0) { > -- > 2.5.5 > > -- Alex Bligh

Re: [Qemu-devel] [PATCH v3 43/44] nbd: Implement NBD_OPT_BLOCK_SIZE on server

2016-04-25 Thread Alex Bligh
Eric, See my message on nbd-general today re the necessity (or not) of getting NBD_OPT_BLOCK_SIZE first; it may be just that you can assume 512 is OK. Otherwise Reviewed-by: Alex Bligh Alex On 23 Apr 2016, at 00:40, Eric Blake wrote: > The upstream NBD Protocol has defined a new extens

Re: [Qemu-devel] [PATCH v3 42/44] nbd: Implement NBD_CMD_WRITE_ZEROES on client

2016-04-25 Thread Alex Bligh
falling back to the obvious > write of lots of zeroes if we return -ENOTSUP because the server > does not have WRITE_ZEROES. > > Signed-off-by: Eric Blake Reviewed-by: Alex Bligh > > --- > v3: rebase, tell block layer about our support > --- > block/nbd-

Re: [Qemu-devel] [PATCH v3 41/44] nbd: Implement NBD_CMD_WRITE_ZEROES on server

2016-04-25 Thread Alex Bligh
On 23 Apr 2016, at 00:40, Eric Blake wrote: > Upstream NBD protocol recently added the ability to efficiently > write zeroes without having to send the zeroes over the wire, > along with a flag to control whether the client wants a hole. > > Signed-off-by: Eric Blake Reviewed

Re: [Qemu-devel] [PATCH v3 40/44] nbd: Implement NBD_OPT_GO on client

2016-04-25 Thread Alex Bligh
ing when a server > requires TLS but does not have NBD_OPT_GO!), and on success > it provides at least as much info as NBD_OPT_EXPORT_NAME sends. > > Signed-off-by: Eric Blake > Reviewed-by: Alex Bligh > --- > v3: revamp to match latest version of NBD protocol > --- >

Re: [Qemu-devel] [PATCH v3 36/44] nbd: Improve handling of shutdown requests

2016-04-25 Thread Alex Bligh
return -EINVAL; > + /* NBD spec says we must reply before disconnecting, > + * but that we must also tolerate guests that don't > + * wait for our reply. */ > +ret = nbd_negotiate_send_rep(client->ioc, NBD_REP_ACK, > + clientflags); > +if (!ret) { > +ret = -EINVAL; > +} > +return ret; > > case NBD_OPT_EXPORT_NAME: > return nbd_negotiate_handle_export_name(client, length); > -- > 2.5.5 > > -- Alex Bligh

Re: [Qemu-devel] [PATCH v3 32/44] nbd: Share common option-sending code in client

2016-04-25 Thread Alex Bligh
ic Blake > Reviewed-by: Alex Bligh > > --- > v3: rebase, tweak a debug message > --- > include/block/nbd.h | 29 +- > nbd/nbd-internal.h | 2 +- > nbd/client.c| 250 ++-- > 3 files changed, 129 insertions(+)

Re: [Qemu-devel] [PATCH v3 31/44] nbd: Share common reply-sending code in server

2016-04-25 Thread Alex Bligh
On 23 Apr 2016, at 00:40, Eric Blake wrote: > Rather than open-coding NBD_REP_SERVER, reuse the code we > already have by adding a length parameter. Additionally, > the refactoring will make adding NBD_OPT_GO in a later patch > easier. > > Signed-off-by: Eric Blake Revie

Re: [Qemu-devel] [PATCH v3 30/44] nbd: Treat flags vs. command type as separate fields

2016-04-25 Thread Alex Bligh
ent upstream > NBD protocol (https://github.com/yoe/nbd/blob/master/doc/proto.md), > and touch some nearby code to keep checkpatch.pl happy. > > Signed-off-by: Eric Blake Reviewed-by: Alex Bligh > > --- > v3: rebase to other changes earlier in series > --- > include/bl

Re: [Qemu-devel] [PATCH v3 29/44] nbd: Avoid magic number for NBD max name size

2016-04-25 Thread Alex Bligh
stick to the required minimum. > > Signed-off-by: Eric Blake Reviewed-by: Alex Bligh > > --- > v3: enlarge the limit, and document choice of new value > --- > include/block/nbd.h | 6 ++ > nbd/client.c| 2 +- > nbd/server.c| 4 ++-- > 3 files changed, 9 i

Re: [Qemu-devel] [PATCH v3 03/44] nbd: Improve server handling of bogus commands

2016-04-25 Thread Alex Bligh
goto out; > +/* unreachable, thanks to special case in nbd_co_receive_request() */ > +abort(); > + > case NBD_CMD_FLUSH: > TRACE("Request type is FLUSH"); > > @@ -1182,10 +1203,14 @@ static void nbd_trip(void *opaque) > break; > default: > LOG("invalid request type (%" PRIu32 ") received", request.type); > -invalid_request: > reply.error = EINVAL; > error_reply: > -if (nbd_co_send_reply(req, &reply, 0) < 0) { > +/* We must disconnect after replying with an error to > + * NBD_CMD_READ, since we choose not to send bogus filler > + * data; likewise after NBD_CMD_WRITE if we did not read the > + * payload. */ > +if (nbd_co_send_reply(req, &reply, 0) < 0 || command == NBD_CMD_READ > || > +(command == NBD_CMD_WRITE && !req->complete)) { > goto out; > } > break; > -- > 2.5.5 > > -- Alex Bligh

Re: [Qemu-devel] [PATCH v3 08/44] nbd: Add qemu-nbd -D for human-readable description

2016-04-25 Thread Alex Bligh
asier to test commit 200650d4, which > is the client counterpart of receiving the description. > > Signed-off-by: Eric Blake Reviewed-by: Alex Bligh > --- > include/block/nbd.h | 1 + > nbd/nbd-internal.h | 5 +++-- > nbd/server.c| 34

  1   2   3   4   5   6   7   8   9   10   >