* Andrew Jones ([email protected]) wrote: > On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote: > > On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote: > > > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote: > > > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote: > > > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote: > > > > > > Hi all, > > > > > > > > > > > > First some background: > > > > > > > > > > > > For the userspace side of AArch64 guest SVE support we need to > > > > > > expose KVM's allowed vector lengths bitmap to the user and allow > > > > > > the user to choose a subset of that bitmap. Since bitmaps are a > > > > > > bit awkward to work with then we'll likely want to expose it as > > > > > > an array of vector lengths instead. Also, assuming we want to > > > > > > expose the lengths as number-of-quadwords (quadword == 128 bits > > > > > > for AArch64 and vector lengths must be multiples of quadwords) > > > > > > rather than number-of-bits, then an example array (which will > > > > > > always be a sequence) might be > > > > > > > > > > > > [ 8, 16, 32 ] > > > > > > > > > > > > The user may choose a subsequence, but only through truncation, > > > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is. > > > > > > > > > > > > Furthermore, different hosts may support different sequences > > > > > > which have the same maximum. For example, if the above sequence > > > > > > is for Host_A, then Host_B could be > > > > > > > > > > > > [ 8, 16, 24, 32 ] > > > > > > > > > > It doesn't really matter for this discussion, but I just realized > > > > > that I picked bogus numbers for these examples. 32 would be too > > > > > big. The largest supported is 16. I probably should have just used > > > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to > > > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512. > > > > > > > > I'd argue differently from the ABI perspective. > > > > > > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask > > > > for [ 1,2,4 ]. KVM will fail the KVM_REG_ARM64_SVE_VLS write with > > > > EINVAL, but I don't distinguish between a set that is not satisfiable on > > > > this hardware and a set that is not valid at all. The kernel basically > > > > doesn't check for the latter: an architecturally invalid set will always > > > > not be satisfiable on the hardware. > > > > > > I agree that 1,2,4 should be a valid set, but how do we avoid the user > > > attempting to select it on a platform that supports 1,2,3,4 now? Until > > > the hardware and KVM will allow it, then I don't think we should try > > > to "support" it in any way. Maybe we should name the property such that > > > it's clear we need to use truncation when constructing a subset now. > > > This would allow us to add another, more generally named, property later > > > when we can select nearly arbitrary subsets. > > > > > > > > > > > Ideally, userspace should not preempt this decision, in case the > > > > architecture becomes more flexible someday in what it can virtualise. > > > > > > > > > > The host must support all lengths in the sequence, which means > > > > > > that while Host_A supports 32, since it doesn't support 24 and > > > > > > we can only truncate sequences, we must use either [ 8 ] or > > > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate > > > > > > between the hosts. > > > > > > > > > > > > Now to the $SUBJECT question: > > > > > > > > > > > > My feeling is that we should require the sequence to be > > > > > > provided on the command line as a cpu property. Something > > > > > > like > > > > > > > > > > > > -cpu host,sve-vl-list=8:16 > > > > > > > > I can't decide whether it's more or less user-friendly to denote VL in > > > > terms of bytes in this kind of context. > > > > > > I'm not sure either. Switching to my second take at quadwords this should > > > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'. > > > > > > > > > > > Usually when I say "VL" in the kernel ABI I try to mean this. > > > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily > > > > encoded rather than being passed as integers, so the precise unit used > > > > is a more abstract concept here ... or anyway, that's my excuse. > > > > _SVE_VQS felt odd here.) > > > > > > My thinking is that using the number of quadwords keeps the numbers > > > smaller. But, if people are mostly going to want to think about vector > > > lengths in terms of bits, like the specification does, then maybe we > > > should just let the numbers be bigger: [ 128, 256, 512 ] > > > > > > > > > > > For kvmtool I went with quadwords on the command line, but only as a > > > > quick hack to simplify the parser slightly. > > > > > > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and > > > > error-prone than 16,32,64. > > > > > > > > Naming the option with "vq" instead of "vl" is another option, though > > > > "vq" is Linux terminology not endorsed by the architecture. > > > > > > I think VL is OK as long we document the unit. > > > > As you wish (and that's basically where I ended up for now with kvmtool). > > > > > > > > > > For kvmtool I've considered a range / filtering syntax, but I didn't > > > > implement it yet. > > > > > > I think we want each VL to be explicitly listed on the command line > > > for the compatibility issue described above. On one host the range > > > [1,4] can be different than the range [1,4] on anther host, depending > > > on whether '3' is in there. And syntax like 1-4:-3 would be overly > > > complicated, IMO. So I'm inclined to just require all of them to be > > > listed, but of course order wouldn't matter. > > > > Probably reasonable. For now (and quite likely forever) the length of > > the list is at most 16 elements. To avoid overengineering I stuck with > > a flat comma-separated list for now. > > > > > > It's worth nothing that there are two problems to be solved here: one is > > to specify an exact set unambiguously, which is important for migration > > scenarios. > > > > The other is to be able to clamp the vector length for user convenience, > > but without particularly considering migration. There's no direct way > > for the user to know the set of vector lengths supported by KVM today, > > so cooking up the right magic to go on the command line is non-trivial: > > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find > > the host's supported set. Or you just have to know. > > Right, we'll have to query first to know what's available. libvirt will > do that and tools built on libvirt should help provide the user with > sane defaults or a relatively painless selection process. The QEMU command > line is primarily for developers, so they'll likely just know what they > want. > > > > > A separate max-vl option (or whatever) might be a useful alternative. > > Yeah, we could offer this to be nicer to QEMU command line users that > don't intend to migrate, or don't care if a migration can fail. This > would be analogous to '-cpu max' which enables all available cpu > features. Bad for migration, but a nice shorthand if you want your > guest to have everything.
One that printed out what was available on that host might be nice (and/or via a qmp query- command) for when you're trying to figure out what's going. Dave > Thanks, > drew -- Dr. David Alan Gilbert / [email protected] / Manchester, UK
