Daniel P. Berrangé <[email protected]> writes: > On Fri, Oct 20, 2023 at 02:37:16PM +0200, Markus Armbruster wrote: >> Fabiano Rosas <[email protected]> writes: >> >> > Markus Armbruster <[email protected]> writes: >> > >> >> Fabiano Rosas <[email protected]> writes: >> >> >> >>> Signed-off-by: Fabiano Rosas <[email protected]> >> > >> > Side question: are we using valid JSON at all? I threw this in a random >> > online linter and it complains about the single quotes. We could have a >> > proper tool doing the validation in CI. >> >> You've come a sad, sad place. >> >> docs/devel/qapi-code-gen.rst: >> >> Schema syntax >> ------------- >> >> Syntax is loosely based on `JSON <http://www.ietf.org/rfc/rfc8259.txt>`_. >> Differences: >> >> * Comments: start with a hash character (``#``) that is not part of a >> string, and extend to the end of the line. >> >> * Strings are enclosed in ``'single quotes'``, not ``"double quotes"``. >> >> * Strings are restricted to printable ASCII, and escape sequences to >> just ``\\``. >> >> * Numbers and ``null`` are not supported. >> >> If your reaction to item 2 is "this is stupid", you'd be exactly right. >> >> Here's the conclusion of a discussion on possible improvements we had in >> 2020: >> https://lore.kernel.org/qemu-devel/[email protected]/ > > Looking at those options again I so strongly want to be able to > say "none of the above". > > We have a need to describe data structures, and generate code for > serializing/deserializing them on the wire. We created a language > for this and wrote our own C code generator, our own docs generator, > own our json parser & formatter, and now are also writing our own > Go code generator (Python, Rust too ?). > > IMHO this is way too much NiH for my likely, and creates a maint > burden for ourselves that we could do without. > > <open-can-of-worms> > At the point in time we invented QAPI this was perhaps justifiable, > though even back then I thought we should have done a binary format > and used XDR to describe it, as a standard pre-existing language and > toolset.
Path dependence... I wasn't involved in the decisions that led to QAPI, nor its initial design and implementation. Its design was substantially constrained by QMP, which predates QAPI by almost two years. I was involved in QMP's design for a bit. We argued back and forth, and I eventually stepped aside to let the guys doing the actual work make the decisions. QAPI was further constrained by the desire to use it with QOM. The two are joined at the hip by visitors. Nevertheless, QOM and QAPI are integrated poorly. My general attitude towards generating source code is "don't; use a more powerful programming language". Instead of creating yet another pillar supporting Greenspun's Tenth Rule, I feel we should have embedded a sufficiently powerful programming language, and used it for the control plane. My choice would've been Lisp. The maintenance burden is modest, but real. The QAPI generator has gotten ~1.6 patches per week for the last five years, trending down. In the last year, it's been 68 patches, 437 insertions, 338 deletions total, much of it in docs/devel/qapi-code-gen.rst. Meanwhile, the entire project has had 130 times as many patches, 620 times as many insertions, and 400 times as many deletions. QAPI infrastructure maintenance is dwarved several times over by QAPI schema maintenance. Chiefly patch review. To be fair, adding language bindings will take a non-trivial one time investment plus ongoing maintenance for each language. > Today I wouldn't suggest XDR, but would be inclined to suggest > Protobuf. This already has code & docs generators for every > programming language we would ever need. > > Protobuf has a defined serialization format too though which is > binary based IIUC, so distinct from our JSON wire format. > > The interesting question though is whether it would be feasible to > transition to this today, despite our historical baggage ? With > recent interest in accessing QAPI for many more languages, it Go, Rust, and what else? > is timely to consider if we can adopt a standardized toolset to > reduce this burden of talking to QEMU from other languages. > > A transition would need several big ticket items > > * A QAPI visitor impl that can spit out the protobuf document. > ie to translate all our existing QAPI formats into protobuf, > as doing this manually would be madness. This is probably > the easy bit. This is about machine-assisted translation of the QAPI schema to protobuf, isn't it? > * A custom protobuf visitor impl that can spit out C code > that has the same API as our QAPI C generator. ie so we can > avoid a big bang to convert all of QEMU internals. I suspect > this is quite alot of work. The alternative is a big bang to convert, which could be less work or more. > * A custom protobuf serializer compatible with our current > JSON format. ie so that QEMU can continue to expose the > current QMP protocol to apps for legacy compat for some > period of time, while also exposing a new native binary > protobuf protocol for future usage. Also probably quite > alot of work. No alternative. > That's all certainly quite alot of work and probably things I have > forgotten. QOM's use of QAPI might add problems. > Above all it does assume it is possible to define a > loss-less mapping from QAPI -> Protobuf language. I've not proved > this is possible, but my inclination is that it probably should be, > and if not, we could likely enable it by strategically deprecated > and then deleting troublesome bits prior to a conversion. > </open-can-of-worms> While I share your distaste for the massive NIH QAPI has become, I'm not sure replacing it now is economical. It requires a massive one-time investment, offset by saving us one-time investments into QAPI bindings for other languages of interest. Whether it can actually reduce ongoing maintenance appreciably after the replacement is unclear. Pretty much the same for QOM.
