On Tue, Jul 19, 2016 at 11:39:03AM +0200, Lukáš Doktor wrote: > Dne 18.7.2016 v 22:41 Ademar Reis napsal(a): > > On Mon, Jul 18, 2016 at 07:33:43PM +0200, Lukáš Doktor wrote: > > > Dne 18.7.2016 v 12:46 Cleber Rosa napsal(a): > > > > > > > > On 07/07/2016 10:44 AM, Lukáš Doktor wrote: > > > > > 0Dne 5.7.2016 v 16:10 Ademar Reis napsal(a): > > > > > > On Fri, Jul 01, 2016 at 03:57:31PM +0200, Lukáš Doktor wrote: > > > > > > > Dne 30.6.2016 v 22:57 Ademar Reis napsal(a): > > > > > > > > On Thu, Jun 30, 2016 at 06:59:39PM +0200, Lukáš Doktor wrote: > > > > > > > > > Hello guys, > > > > > > > > > > > > > > > > > > the purpose of this RFC is to properly split and define the > > > > > > > > > way test > > > > > > > > > parameters are processed. There are several ways to split them > > > > > > > > > apart, each > > > > > > > > > with some benefits and drawbacks. > > > > > > > > > > > > > > > > > > > > > > > > > > > Current params process > > > > > > > > > ====================== > > > > > > > > > > > > > > > > > > `tree.TreeNode` -> Object allowing to store things > > > > > > > > > (environment, > > > > > > > > > filters, > > > > > > > > > ...) in tree-structure > > > > > > > > > `multiplexer.Mux` -> Interface between job and multiplexer. > > > > > > > > > Reports > > > > > > > > > number > > > > > > > > > of tests and yields modified test_suite templates > > > > > > > > There has been ideas and suggestions about using the multiplexer for > > > > purposes other than multiplexing tests. While those other use cases are > > > > only ideas, `Mux` and its `itertests` method are a good enough name, but > > > > there may be the opportunity here to provide a common and more standard > > > > interface for different "multiplexations". Example: > > > > > > > > 1) For tests, a class TestMux, with an standard Python iterable > > > > interface[1]. > > > > 2) For test execution location (say different hosts), a HostMux class, > > > > with the same standard Python interface. > > > > > > > > Even if we find no real use for other Mux* classes, having a default > > > > iterable implementation is a good idea. So moving the `tests` from the > > > > `itertests` method to the `Mux` class name, from `Mux` to `TestsMux`, > > > > feels right to me. > > > > > > > > UPDATE: then I looked at PR #1293, and noticed that it intends to take > > > > the custom variant responsibility out of the `multiplexer.Mux()` class. > > > > It feels right because it moves the variant processing into its own > > > > domain, kind of what I had in mind by naming it `TestsMux()`. Still, > > > > having a standard iterable interface instead of `itertests()` feels > > > > right to me. > > > > > > > Well without that PR it's impossible to use the python standard __iter__ > > > method as it requires argument. With that cleanup sure, `__iter__` method > > > is > > > better. > > > > > > As for the multiple classes I don't see a reason for it. Multiplexer (as > > > the > > > variants generator) is independent on anything, it simply produces all > > > possible variants. So let me just turn the `iter_tests` into `__iter__` > > > and > > > we're done. > > > > > > > > > > > > `multiplexer.MuxTree` -> Object representing part of the tree > > > > > > > > > from > > > > > > > > > the root > > > > > > > > > to leaves or another multiplex domain. Recursively it creates > > > > > > > > > multiplexed > > > > > > > > > variants of the full tree. > > > > > > > > > `multiplexer.AvocadoParams` -> Params object used to retrieve > > > > > > > > > params from > > > > > > > > > given path, allows defining several domains for relative paths > > > > > > > > > matching > > > > > > > > > defined as `mux_path`s. > > > > > > > > > `multiplexer.AvocadoParam` -> Slice of the `AvocadoParams` > > > > > > > > > which > > > > > > > > > handles > > > > > > > > > given `mux_path`. > > > > > > > > > `test.Test.default_params` -> Dictionary which can define > > > > > > > > > test's > > > > > > > > > default > > > > > > > > > values, it's intended for removal for some time. > > > > > > > > > > > > > > > > > > > > > > > > > > > Creating variants > > > > > > > > > ----------------- > > > > > > > > > > > > > > > > > > 1. app > > > > > > > > > 2. parser -> creates the root tree > > > > > > > > > `args.default_avocado_params = > > > > > > > > > TreeNode()` > > > > > > > > This shows the broken link between other forms of configuration (other > > > > than command line parameters) and the parameter mechanism. I hope to > > > > address some of it in the Configuration Management RFC. > > > > > > > > > > > > > 3. plugins.* -> inject key/value into > > > > > > > > > `args.default_avocado_params` > > > > > > > > > (or it's > > > > > > > > > children). One example is `plugins.run`'s --mux-inject, the > > > > > > > > > other is > > > > > > > > > `avocado_virt`'s default values. > > > > > > > > > 4. job -> creates multiplexer.Mux() object > > > > > > > > > a. If "-m" specified, parses and filters the yaml file(s), > > > > > > > > > otherwise > > > > > > > > > creates an empty TreeNode() called `mux_tree` > > > > > > > > > b. If `args.default_avocado_params` exists, it merges it > > > > > > > > > into the > > > > > > > > > `mux_tree` (no filtering of the default params) > > > > > > > > > c. Initializes `multiplexer.MuxTree` object using the > > > > > > > > > `mux_tree` > > > > > > > > `mux_tree` is an internal variable name and not part of any interface, > > > > right? > > > It's defined in step `4b`, it's basically the combination of all trees. > > > > > > > Also, the checking for multiplex files (and other arguments) > > > > happen in the context of the `multiplexer.Mux()` class and the job is > > > > only supposed to use the outcome of `multiplexer.Mux()`, say via > > > > `get_number_of_tests()` or `itertests()`, right? > > > Yes, this was a prep work for the future to be able to tweak the > > > parameters > > > by each possible plugin. > > > > I'm not familiar with the implementation details and I'm probably > > missing something here... I focused on the higher-level > > architecture and APIs, and to me, the multiplexer and the Avocado > > Params API(s) should be at the test level. Please see my comments > > further down. > > > > > > > > > > > > > Finally, is `multiplexer.Mux().variants` something that should be > > > > considered public in your opinion? > > > > > > > It's only public because the replay plugin accesses it and logs the list > > > of > > > variants. When we introduce the > > > https://trello.com/c/fEnQFYRE/601-record-the-mux-tree-in-yaml-format-instead-of-pickle > > > we can make it private. > > > > > > > > > > > > 5. job -> asks the Mux() object for number of tests > > > > > > > > > a. Mux iterates all MuxTree variants and reports > > > > > > > > > `no_variants * > > > > > > > > > no_tests` > > > > > > > > > 6. runner -> iterates through test_suite > > > > > > > > > a. runner -> iterates through Mux: > > > > > > > > > i. multiplexer.Mux -> iterates through MuxTree > > > > > > > > > * multiplexer.MuxTree -> yields list of leaves of > > > > > > > > > the > > > > > > > > > `mux_tree` > > > > > > > > > ii, yields the modified test template > > > > > > > > > b. runs the test template: > > > > > > > > > i. Test.__init__: | > > > > > > > > > if isinstance(params, dict): > > > > > > > > > # update test's default params > > > > > > > > > elif params is None: > > > > > > > > > # New empty multiplexer.AvocadoParams are > > > > > > > > > created > > > > > > > > > elif isinstance(params, tuple): > > > > > > > > > # multiplexer.AvocadoParams are created from > > > > > > > > > params > > > > > > > > > 7. exit > > > > > > > > > > > > > > > > > This overview is pretty nice. I really feel a lot less ignorant about > > > > this subject now! :) > > > > > > > > > > > > > AvocadoParams initialization > > > > > > > > > ---------------------------- > > > > > > > > > > > > > > > > > > def __init__(self, leaves, test_id, tag, mux_path, > > > > > > > > > default_params): > > > > > > > > > """ > > > > > > > > > :param leaves: List of TreeNode leaves defining > > > > > > > > > current > > > > > > > > > variant > > > > > > > > > :param test_id: test id > > > > > > > > > :param tag: test tag > > > > > > > > > :param mux_path: list of entry points > > > > > > > > > :param default_params: dict of params used when no > > > > > > > > > matches > > > > > > > > > found > > > > > > > > > """ > > > > > > > > > > > > > > > > > > 1. Iterates through `mux_path` and creates `AvocadoParam` > > > > > > > > > slices > > > > > > > > > containing > > > > > > > > > params from only matching nodes, storing them in > > > > > > > > > `self._rel_paths` > > > > > > > > > 2. Creates `AvocadoParam` slice containing the remaining > > > > > > > > > params, > > > > > > > > > storing > > > > > > > > > them in `self._abs_path` > > > > > > > > > > > > > > > > > > Test params > > > > > > > > > ----------- > > > > > > > > > > > > > > > > > > def get(self, key, path=None, default=None): > > > > > > > > > """ > > > > > > > > > Retrieve value associated with key from params > > > > > > > > > :param key: Key you're looking for > > > > > > > > > :param path: namespace ['*'] > > > > > > > > > :param default: default value when not found > > > > > > > > > :raise KeyError: In case of multiple different values > > > > > > > > > (params clash) > > > > > > > > > """ > > > > > > > > > > > > > > > > > > 1. Tries to obtain the (key, path, default) from cache > > > > > > > > > 2. Tries to get the (key, path) from `self._rel_paths` > > > > > > > > > 3. When the `path` is abs_path it tries to get the param from > > > > > > > > > `self._abs_path` > > > > > > > > > 4. Looks into the Test's default params dictionary > > > > > > > > > 5. Returns `default` > > > > > > > > > > > > > > > > > > Overview > > > > > > > > > -------- > > > > > > > > > > > > > > > > > > Basically now there are 3 components: > > > > > > > > > > > > > > > > > > 1. params parser (yaml => tree) > > > > > > > > > 2. variants generator (tree => iterator) > > > > > > > > > 3. test parameters (key => value from variant values) > > > > > > > > > > > > > > > > > > and we need to decide how do we want to split it and what > > > > > > > > > should be > > > > > > > > > part of > > > > > > > > > the plugin API and what core API. > > > > > > > > > > > > > > > > > > > > > > > > > > > Plugin parser->tree > > > > > > > > > =================== > > > > > > > > > > > > > > > > > > We can say the `tree` is the actual params API and we could > > > > > > > > > only allow > > > > > > > > > producing custom parsers to construct the tree. This would be > > > > > > > > > the > > > > > > > > > simplest > > > > > > > > > method as it only requires us to move the yaml parsing into > > > > > > > > > the > > > > > > > > > module and > > > > > > > > > the `AvocadoParams` would stay as they are. The cons is that > > > > > > > > > the > > > > > > > > > plugin > > > > > > > > > writers would only be able to produce params compatible with > > > > > > > > > the tree > > > > > > > > > structure (flat, or tree-like). > > > > > > > > > > > > > > > > > > If we decide to chose this method, we can keep the current > > > > > > > > > avocado > > > > > > > > > arguments > > > > > > > > > and only allow replacing the parser plugin, eg. by > > > > > > > > > `--multiplex-plugin > > > > > > > > > NAME`. Alternatively we might even detect the suitable plugin > > > > > > > > > based > > > > > > > > > on the > > > > > > > > > multiplex file and even allow combining them (`.cfg vs. > > > > > > > > > .yaml, ...) > > > > > > > > > > > > > > > > > > The plugin would have to support: > > > > > > > > > > > > > > > > > > * parse_file(FILE) => tree_node > > > > > > > > > * check_file(FILE) => BOOL // in case we want automatic > > > > > > > > > detection of > > > > > > > > > file->plugin > > > > > > > > > > > > > > > > > I really think that, at this point, this is sufficient. This delivers > > > > what most users would like to have, that is a flat structure of > > > > parameters. Having good documentation on the tree-like capabilities can > > > > allow users to write, say, database backed implementations that map > > > > record keys to tree nodes. > > > > > > > Glad to hear that :-) > > > > > > > > > > > > Plugin parser->variant > > > > > > > > > ====================== > > > > > > > > > > > > > > > > > > This would require deeper changes, but allow greater > > > > > > > > > flexibility. > > > > > > > > > We'd also > > > > > > > > > have to chose whether we want to allow combinations, or > > > > > > > > > whether the > > > > > > > > > plugin > > > > > > > > > should handle the whole workflow. I don't think we should > > > > > > > > > allow > > > > > > > > > combinations > > > > > > > > > as that would imply another convention for storing the parsed > > > > > > > > > results. > > > > > > > > > > > > > > > > > If I understand this correctly, you mean that the plugin would have a > > > > broader scope, and not only stop at the tree creation, but continue all > > > > the way to the production of variants, right? > > > > > > > > If that's the case, I'd avoid this at this moment. IMHO, feeding the > > > > tree from different sources sounds more like what users may need than > > > > tweaking with the variants generation itself. > > > > > > > > In the future, if it is really needed, we may introduce a pluggable > > > > `multiplexer.TestsMux()` interface, so that users may stick in their own > > > > variant creation code. > > > > > > > Yep, that was the intention and I also prefer to allow just the "feeding" > > > and not the variatns generation itself. And if needed we should be > > > prepared > > > as it already lives inside a partially defined structure > > > `multiplexer.Mux`. > > > > > > > > > > > > The user would configure in config or on cmdline which plugin > > > > > > > > > he > > > > > > > > > wants to > > > > > > > > > use and the arguments would stay the same (optionally > > > > > > > > > extended by the > > > > > > > > > plugin's arguments) > > > > > > > > > > > > > > > > > > The plugin would have to support: > > > > > > > > > > > > > > > > > > * configure(parser) // optional, add extended options like > > > > > > > > > --mux-version > > > > > > > > > * parse_file(FILE) // does not return as it's up to plugin > > > > > > > > > to > > > > > > > > > store the > > > > > > > > > results > > > > > > > > > * inject_value(key, value, path) // used by avocado-virt to > > > > > > > > > inject default > > > > > > > > > values > > > > > > > > > * __len__() // Return number of variants (we might > > > > > > > > > want > > > > > > > > > to extend this to > > > > > > > > > accept TEMPLATE and allow different number of variants per > > > > > > > > > TEMPLATE. That is > > > > > > > > > currently not supported, but it might be handy > > > > > > > > > * itervariants(TEMPLATE) // yields modified TEMPLATE with > > > > > > > > > params > > > > > > > > > set in > > > > > > > > > AvocadoParams understandable format > > > > > > > > > > > > > > > > > > > > > > > > > > > Plugin AvocadoParams > > > > > > > > > ==================== > > > > > > > > > > > > > > > > > > I don't think we should make the AvocadoParams replaceable, > > > > > > > > > but if > > > > > > > > > we want > > > > > > > > > to we should strictly require `params.get` compatibility so > > > > > > > > > all > > > > > > > > > tests can > > > > > > > > > run seamlessly with all params. Anyway if we decided to make > > > > > > > > > AvocadoParams > > > > > > > > > replaceable, then we can create a proxy between the params > > > > > > > > > and the > > > > > > > > > plugin. > > > > > > > > > > > > > > > > > > > > > > > > > > I also don't think it's needed *at this point*. But our code should be > > > > clear enough that, given requests, we can add this interface for > > > > pluggable parameter handling later with minimal changes. > > > > > > > yep > > > > > > > > > > > > Conclusion > > > > > > > > > ========== > > > > > > > > > > > > > > > > > > I'm looking forward to cleaner multiplexer API. I don't think > > > > > > > > > people would > > > > > > > > > like to invest much time in developing fancy multiplexer > > > > > > > > > plugins so > > > > > > > > > I'd go > > > > > > > > > with the `parser->tree` variant, which allows easy > > > > > > > > > extensibility > > > > > > > > > with some > > > > > > > > > level of flexibility. The flexibility is for example > > > > > > > > > sufficient to > > > > > > > > > implement > > > > > > > > > cartesian_config parser. > > > > > > > > +1. > > > > > > > > > > > > > > > > > > > > > > As for the automatic detection, I donẗ like the idea as people > > > > > > > > > might want to > > > > > > > > > use the same format with different custom tags. > > > > > > > > From the Zen of Python: "explicit is better than implicit". It's so > > > > important it's the 2nd line. > > > > > > > > > > > > > > > > > > > > Hi Lukáš. > > > > > > > > > > > > > > > > I believe we're in sync, but I miss the high level overview, or > > > > > > > > at least review, of how params, variants and the multiplexer or > > > > > > > > other plugins are all related to each other. > > > > > > > > > > > > > > > > Please check the definitions/examples below to see if we're in > > > > > > > > sync: > > > > > > > > > > > > > > > > Params > > > > > > > > ------ > > > > > > > > > > > > > > > > A dictionary of key/values, with an optional path (we could > > > > > > > > simply call it prefix), which is used to identify the key > > > > > > > > when there are multiple versions of it. The path is > > > > > > > > interpreted from right to left to find a match. > > > > > > > > > > > > > > > > The Params data structure can be populated by multiple > > > > > > > > sources. > > > > > > > > > > > > > > > > Example: > > > > > > > > (implementation and API details are not discussed here) > > > > > > > > > > > > > > > > key: var1=a > > > > > > > > path: /foo/bar/baz > > > > > > > > > > > > > > > > key: var1=b > > > > > > > > path: /foo/bar > > > > > > > > > > > > > > > > key: var2=c > > > > > > > > path: NULL (empty) > > > > > > > > > > > > > > > > get(key=var1, path=/foo/) ==> error ("/foo/var1" not found) > > > > > > > > get(key=var1, path=/foo/*) ==> error (multiple var1) > > > > > > > > get(key=var1, path=/foo/bar/baz/weeee/) ==> error > > > > > > > > get(key=var1, path=/foo/bar/weeee/) ==> error > > > > > > > > > > > > > > > > get(key=var2) ==> c > > > > > > > > get(key=var2, path=foobar) ==> error ("foobar/var2" not > > > > > > > > found) > > > > > > > > > > > > > > > > get(key=var1, path=/foo/bar/baz/) ==> a > > > > > > > > (unique match for "/foo/bar/baz/var1") > > > > > > > > > > > > > > > > get(key=var1, path=/foo/bar/) ==> b > > > > > > > > (unique match for "/foo/bar/var1/") > > > > > > > > > > > > > > > > get(key=var1, path=baz) ==> a > > > > > > > > (unique match for "baz/var1") > > > > > > > > > > > > > > > > get(key=var1, path=bar) ==> b > > > > > > > > (unique match for "bar/var1") > > > > > > > > > > > > > > > > This kind of "get" API is exposed in the Test API. > > > > > > > > > > > > > > > > > > > > > > > > Variants > > > > > > > > -------- > > > > > > > > > > > > > > > > Multiple sets of params, all with the same set of keys and > > > > > > > > paths, but potentially different values. Each variant is > > > > > > > > identified by a "Variant ID" (see the "Test ID RFC"). > > > > > > > > > > > > > > > > The test runner is responsible for the association of tests > > > > > > > > and variants. That is, the component creating the > > > > > > > > variants has absolutely no visibility on which tests are > > > > > > > > going to be associated with variants. > > > > > > > > > > > > > > > > This is also completely abstract to tests: they don't have > > > > > > > > any visibility about which variant they're using, or which > > > > > > > > variants exist. > > > > > > > > > > > > > > > Hello Ademar, > > > > > > > > > > > > > > Thank you for the overview, I probably should have included it. I > > > > > > > omitted it > > > > > > > as it's described in the documentation, so I only mentioned in the > > > > > > > `Plugin > > > > > > > AvocadoParams` that I don't think we should turn that part into a > > > > > > > plugin. > > > > > > > > > > > > > > The variant, as described earlier, is the method which modifies > > > > > > > the > > > > > > > `test_template` and as you pointed out it compounds of `Variant > > > > > > > ID` and > > > > > > > `Params`. The way it works now it can go even further and alter > > > > > > > all the > > > > > > > test's arguments (name, methodName, params, base_logdir, tag, job, > > > > > > > runner_queue) but it's not documented and might change in the > > > > > > > future. > > > > > > > > > > > > OK, so I think we should change this. The layers should have > > > > > > clear responsibilities and abstractions, with variants being > > > > > > restricted to params only, as defined above. > > > > > > > > > > > > I don't think the component responsible for creating variants > > > > > > needs any visibility or knowledge about tests. > > > > > > > > > > > Yes, there is no need for that, it was only simplification: > > > > > > > > > > https://github.com/avocado-framework/avocado/pull/1293 > > > > > > > > > > > > > BTW, why was this PR closed? Inteded to be sent again with other work? > > > > > > > I was also wondering. Probably just an accidental mouse click... Let me > > > reopen it... (I thought it was avocado-vt, but it works well with it...) > > > > > > > > > > > > > > > > > > Given the above, the multiplexer (or any other component, like a > > > > > > > > "cartesian config" implementation from Autotest) would be bound > > > > > > > > to these APIs. > > > > > > > The cartesian config is not related to params at all. Avocado-vt > > > > > > > uses a > > > > > > > hybrid mode and it replaces the params with their custom object, > > > > > > > while > > > > > > > keeping the `avocado` params in `test.avocado_params`. So in > > > > > > > `avocado_vt` > > > > > > > tests you don't have `self.params`, but you have `test.params` and > > > > > > > `test.avocado_params`, where `test.params` is a dictionary and > > > > > > > `test.avocado_params` the avocado params interface with > > > > > > > path/key/value. > > > > > > > Cartesian config produces variants not by creating test variants, > > > > > > > but by > > > > > > > adding multiple tests with different parameters to the test_suite. > > > > > > > > > > > > What I mean is that we probably could, in theory at least, > > > > > > implement a plugin that parses a "cartesian config" and provides > > > > > > the data as needed to fill the variants and param APIs I > > > > > > described above. I'm not saying we should do that, much less that > > > > > > it would be useful as a replacement for the current cartesian > > > > > > config implementation in avocado-vt. I'm simply stating that once > > > > > > we have a clear plugin API for Params and Variants, we should be > > > > > > able to replace the multiplexer with other mechanisms that > > > > > > provide a similar functionality. > > > > > > > > > > > > Thanks. > > > > > > - Ademar > > > > > > > > > > > > > > > > In that case yes. You can see it in the conclusion that even the > > > > > simpler > > > > > version (parser->tree) is capable of using cartesian_config as source > > > > > of > > > > > params. > > > > > > > > > > > > > So, to make sure we're on the same page: we intend to allow users to > > > > write and choose their own tree producers (it's pluggable). With a > > > > given tree producer active, the multiplex mechanism is going to be, at > > > > this point, a single, non-pluggable one. > > > > > > > > Right? > > > Yes and the switching is going to be done explicitly by > > > `--multiplex-plugin` > > > cmdline argument with the default. Also the second (a bit hidden) change > > > is, > > > that `multiplexer` is going to be only the variants generation code. Maybe > > > we should go agead and change the options as we'd have following parts: > > > > > > 1. ?? tree-producer ?? - to parse user input to tree (currently unnamed) > > > > To me, tree is an implementation detail, a characteristic of the > > multiplexer. I would even rename the "path" to "prefix", as I > > suggested in my comment above (or maybe "namespace"). > > > > The API should be about Avocado Params. Plugins (multiplexer or > > whatever else) should provide sets of Avocado Params, identified > > by IDs (by default, only one set). > > > > Params could be defined as special dictionaries where the key > > contains a prefix (namespace) that gets resolved from right to > > left. How the prefix is implemented (or if it even exists) is an > > implementation detail. > > > > That's what I meant by my comment above, with two sections > > (Params and Variants). Please review them if the above is not > > clear, as we should not be talking about trees outside of the > > multiplexer component. > > > > So, again: > > > > - Avocado runner: knows about tests (each identified by a Test > > Name) and sets of Avocado Params, each identified by a > > Variant ID. This is done at test creation time, after the job > > is configured. > > > > The runner is responsible for matching Test Names to Variant > > IDs, creating Test IDs. > > > > - The multiplexer: parses a yaml file and creates multiple sets > > of Avocado Params, each identified by a Variant ID. > > > > - The Variant API is the abstraction layer between both. The > > API is provided by the test runner and fulfilled by the > > Multiplexer (or other component). > > > > This means the multiplexer (anything implementing the API) can't > > be used for things which are outside of the test layer, such as > > job runtime configuration. > > > > If we want to do that, we would have at least two options: > > > > 1. We extend the multiplexer, from a provider of Avocado (Test) > > Params, to a provider of other things, such as "Avocado Job > > Params" and/or "Avocado Configuration Params". This would > > require new APIs (Job APIs, Configuration APIs) and changes in > > the multiplexer to be aware of these APIs. The multiplexer would > > then be invoked and used in multiple places in Avocado (avocado > > configuration, job creation time, test creation time). > > > > In other words, here we would be using the .yaml file from the > > multiplexer as a general configuration file for jobs and the > > avocado runtime (probably using special namespaces). We've been > > doing some of this already, IIRC. > > > > Hopefully those of you who know me understand by now that I > > don't like this approach as stated, as it's inelegant and > > hackish. But it can be done properly, which brings us to option > > 2: > > > > 2. We introduce the new APIs (same as above: avocado > > configuration, job configuration) but also a different mechanism > > or component for avocado and job level configuration. Maybe even > > using yaml as well, maybe also with variants, but a different > > component, with new semantics and syntax. > > > > It could be, for example, a yaml file that contains multiple > > sections. Multiplexation would optionally occur inside each > > section to create multiple variants, but never across: > > > > section 1: avocado configuration params, used with the > > Configuration API. A hypothetical test runner could use this > > to run Avocado multiple times, each under a different > > configuration via multiplexation. > > > > section 2: job configuration params. Used with the Job > > Configuration API, again a hypothetical test runner could use > > this to run multiple jobs in Avocado, each configured in a > > different way via multiplexation of these params. > > > > section 3: test params. Used with the Test > > Configuration API (currently "Avocado Params"). This is our > > current multiplexer. > > > > Again, this is one possible implementation. The APIs should be > > abstract enough to introduce proper concepts that could be > > implemented in different ways. Words such as Mux, Multiplexer and > > Multiplexation should never be present inside Avocado core or > > these APIs. > > > > But I guess I'm getting ahead of myself here... I think all we > > need right now is a proper and clean Test Params API with an > > abstract multiplexer. > > > I think we are more or less on the same page. I understood from Cleber, that > he wants to use the Mux API to produce different set of params and create > one Mux object for test params, other object for configuration and so on. > The multiplexer is universal so it's possible to use it as a "database" with > variants. Cleber please correct me if I got it wrong... > > To summarize it here (once again): > > 1. The tree is a "database" which allows injecting values under certain keys > with given path (namespace would be also acceptable by me, I don't like the > tag, though). So far we agreed that we should turn the yaml=>tree > pseudoplugin to real plugin and we are looking for a name (I'd call it > yaml2params). > 2. Mux is the variant generation mechanism, which is responsible for walking > the database and producing variants (which are lists of path along with > their values) > 3. AvocadoParams is a per-variant database used inside tests to query for > values > > So (1) is actually a core API (independent on yaml parser) which feeds the > Mux, the (2) is used by the runner (or job API?) to produce multiple > variants of the same test and the (3) is the API available inside test to > query for the current test parameters. > > The (1), (2) and (3) are essential parts of avocado, independent on yaml and > currently should stay in `core`, even though that in the future we might > want to allow replacing the (1) and maybe even (2) with plugins. Currently > this is not important and we are not aware of anyone demanding it. We take > (3) as definite for now, so it should not change apart from optimizations.
I think we have different views of what the abstraction layer should be. But I don't think this is going to be used anytime soon by 3rd parties and we can always make more changes in the future, so it's OK to disagree here. > > The workflow is: > > 1. params-feeder (eg. `yaml2params`, or `--mux-inject`) -> inject records > into `tree` (in the future it could be any arbitrary database, currently > hardcoded as tree) I think we need a better vocabulary. In particular, we need to define what params are. Are params branches/leaves of the tree? Or are params the result of the processing of a tree? > 2. `tree` merges all records and applies filters Filters are part of the tree structure? So are the multiplex flags? Again, if this is exposed at the avocado-core API level, we're exposing too many implementation details and using the (current) multiplex as the API, instead of something more generic and abstract. That's why I think the core API should be about params (as defined by me previously), not about trees to be processed. > 3. `Mux` object is created on top of the tree > 4. test runner uses `Mux` to produce variants and runs tests+variant > combinations > 5. each test creates AvocadoParams from the variant > > Currently it works a bit differently (inherited from the old days) because > the `yaml2param`, currently `core.tree.create_from_yaml`, actually creates > the tree database directly. This is not a problem now, it would be no > problem even after turning it into a proper plugin, but it'd become a > problem if we chose to replace `tree` with any arbitrary database, therefor > I think it's time to extract the `create_from_yaml` and instead of creating > the tree we should only ask the `tree` (or database) to inject/update/remove > variables/filters/flags. Then we have a clear interface. > > So the multiplexer is core and IMO should stay there. Only the > params-feeders should be extracted, which means the PyYAML would be > optional. Also multiplexer is just a database with combinations so it could > be used for anything, also we want to keep using one database for test > variants and not mixed settings/config_files/test_params. > > I hope everything is clear, because if not I'd have to draw a > chart :-D I think I get what you're proposing. I would do things differently, but this is all going to be internal for a while more and we can improve it incrementally. Thanks. - Ademar > > Regards, > Lukáš > > > Thanks. > > - Ademar > > > > > 2. multiplexer - to produce variants > > > 3. avocado params - to obtain params from variant > > > > > > We should probably find a suitable name to ambiguously identify each part > > > as > > > of today we only have multiplexer and avocado params, which can lead to > > > confusions. I can't come up with any good name so unless you have a good > > > name we maybe end up with the same name. The situation should be better, > > > though, because those parts will be really separated. > > > > > > Thank you for the feedback, let's get my hands dirty. :-) > > > > > > Regards, > > > Lukáš > > > > > > > > > > > > Regards, > > > > > Lukáš > > > > > > > > > > > > > [1] - https://docs.python.org/2.6/glossary.html#term-iterable > > > > > > > > > > > > > > > > -- Ademar Reis Red Hat ^[:wq! _______________________________________________ Avocado-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/avocado-devel
