On 2014-23-06 21:48, John Bollinger wrote:
On Monday, June 23, 2014 7:45:05 AM UTC-5, henrik lindberg wrote:
I see this as an essential feature to support maintainable modules.
And by "this", I mean a mechanism that decouples the wanted Element and
its Container (where the Element is the thing we want/need/require, and
the Container is the named thing it is in, typically a Module).
Agreed.
This can be achieved by modules describing their provided and required
capabilities. These are described as name spaced names and they are
versioned.
That's so easy to say, yet so fraught with difficulties. In particular,
the biggest issues I see revolve around the meanings of capability
names. Capability names defined and maintained by the modules that
require them present a potential problem because capability provider
modules can't be expected to know and declare all the capabilities that
they in fact provide.
Well, they do provide everything in the module. The publishing tool
would extract those as puppet capabilities i.e. class x, resource type
y, function f, etc. (each in the module's namespace).
A module that accepts "plugins" declares a required capability
(typically an optional requirement) - i.e. the module supports
additional things, special things if the optional requirement is
fulfilled. It may also have hard requirements.
This works well in practice in the Eclipse ecosystem with many thousands
of components and requirements spread across a large number of projects
(several hundred at Eclipse itself, and thousands elsewhere).
Running with the authentication example, if module 'ssh' requires
capability ssh::authentication::service, then why should it be the
responsibility of nsssd and nslcd to know about that in order to declare
that they provide it? And then, what if I switch out puppetlabs-ssh for
example42-ssh, which happens also to require a capability of the same
name? What system or procedure can make sure that puppetlabs and
example42 agree on the meaning of that capability?
Someone ones the capability name space. They should be named after the
publisher. It is the publisher of the capability name that defines what
it is. This is no different than the definition of any other API. If an
API is something that is a shared concern the involved parties should
collaborate on its definition (again not different from any other API).
Ideally they write a test that validates that something has the
capability it states it has.
I think the capability consumer must declare what capabilities it
requires, yes, but it must be up to the user to bind capability
providers to the names. At least, it must be within the user's power to
do so. Indeed, I'm having trouble how the desired decoupling is
achieved any other way.
Picking one implementation over another typically means including one
"container of functionality" (in puppet's case a "module") instead of
another in the system's configuration (in puppet's case, include it on
the module path). The difficulty comes when there are two modules that
happens to both provide the same capability, and the system wants to use
other non overlapping parts. In this case, the binding of the capability
that clashes is a problem the user would need to handle.
(In practice from Eclipse, I have not seen this happen. If I had this
problem I would break out the clashing part into a separate container if
I had no other means to control this). I can imagine several ways to
filter out capabilities from modules so only the wanted one is exposed.
The desire here seems to be to say that I need an x, and there is one
called your::x, and another called mine::x (two different capabilities
because the publishers did not map the :x to be instances of the same
capability. Instead, a consumer figures out that "hey I can use either
one of those two 'x' because I only set the parameter "bar" (there are
lots of others, but I only care about what it does when I create an
x {name: bar => 10 }
In this case we bascially just need to be able to bind 'x' to your::x,
or mine::x. This is something the binding system can do (i.e. map one
name to another). The configuration just needs to contain the target of
the binding.
A fancier approach is to define the fact that your::x and mine::x share
a capability and publish this in a module "our:x", make it bind our:x to
either your::x or mine:.x and we can use our::x everywhere in the logic
(except in the our module where we do the binding).
If our::x is a truly common shared concern then maybe it should become
an API in its own right, and the publishers of the two 'x's would agree
to declare that they do indeed publish this capability.
(long story on how it could work...)
This creates an open ended system that can describe
dependencies on either modules (like now) by using a name in the Module
namespace, a gem in a gem namespace, a puppet class in a puppet name
space, new types of dependencies can be added etc.
As a very nice side effect the ability to describe provided
capabilities
makes it possible for a described entity to list the same capability
multiple times with different versions, or version ranges, thus making
it possible to describe components as "backwards compatible", if not in
full, for a smaller portion of the services it contains.
So far, this sounds a lot like the dependency features of software
packaging formats such as RPM and DEB. But look at what's happened in
that space, especially with RPMs: despite the commonality of the format,
RPMs are for the most part partitioned by family (RedHat vs. SUSE vs.
...) and distro version, with comparatively poor compatibility across
those lines. It think the fact that they have even that much coherency
is related to the distro maintainer serving as central heavyweight with
strong influence on which capability names are used and what they mean.
Yes, I am aware of such issues. It is a complex domain to start with.
The difficulty with this (any other similar solution) is the resolution
of the dependencies as it requires something like a SAT solver.
I think you're saying that given a set of available modules, one needs
the equivalent of a SAT solver to find a self-consistent subset that
provides a given collection of capabilities. But that's a problem for
the user to solve, so in addition to computing an answer (perhaps with
the help of a tool), he has the alternative of redefining the problem to
make it easy by creating new modules or modifying existing ones.
yes.
I guess that could be a problem that you want an enhanced module tool to
be able to handle, but that seems a side issue to me. The problem that
the catalog builder has to solve is much simpler: whether a given
collection of classes and resources (i.e. the contents of one catalog)
has any unsatisfied requirements.
yes (as I also pointed out later) - there are only cases of either
fulfilled (ok), missing, or ambiguously resolved to consider at runtime.
(with a mechanism to block out unwanted things to handle the
ambiguities, and manual "go find something that satisfies the
requirement for what is missing).
The finding of missing things is however a problem esp. if the
granularity of modules increases (which I think will happen, just like
in the Eclipse / OSGi ecosystem) where modules act as fragments,
options, extra conditional features on certain platforms etc. Basically
because it is both easier and cleaner to handle them this way when there
is a solver / provisioning system that handles these cases (rather than
complex conditional logic inside of larger modules that carry
implementations of all the 'what-ifs', and 'also does this on..' features.
Moreover, I'm inclined to think that even though SAT is in general a
hard problem (NP-complete, in fact), the instances likely to arise in a
Puppet context are all fairly easily computable. There will be few
components -- usually just one -- providing any given capability, and
few exclusion constraints. I don't think you actually need a very
clever SAT solver there.
That is what the Eclipse community thought at first and the so called
"update manager" then plagued thousands of users for 10 years until it
was replaced.
The fact that SAT solving is typically simple means that it works in
practice at reasonable speed. It is however invaluable for the more
complex cases.
Also, all that is moot if, as I suggested, it is the user's
responsibility to bind provider components to capability names.
I think you need both - not initially, but I think the Puppet ecosystem
will evolve just like other component based software systems I have seen
(I happen to know the Eclipse one fairly well having been part of it
almost from the start).
I worked on the implementation of such a system for Eclipse (Eclipse
p2)
which has been in use for a couple of years now as *the* software
update/configuration mechanism for the Ecipse ecosystem. (If you are a
Puppet Labs Geppetto user you have already used it, as it is what
updates Geppetto with new releases).
While such a system (as p2) is very flexible and powerful, the main
problem is to explain why something is not installable (complete /
updateable) - i.e. when there are parts missing, or when there are
ambiguities. Even though p2 has such capabilities (thanks to the sat
solver in use) it is often still a bit of a puzzle when facing non
regular configurations (or tracking down the metadata that has bad
consequences).
If we do not attempt to solve the resolution, and simply validate the
constraints, the problem is much much simpler, but you also do not get
any help configuring a solution (except being slapped when the
configuration is wrong/incomplete).
At least as a first go, I think it would be fine to stop at validating.
That's a pretty natural extension of the current system, and it seems
like it would present a fairly low barrier to entry.
I think so to. And it has to be done at runtime even if a SAT solver (or
similar) was used to calculate the configuration.
It would be very interesting to conduct an experiment using p2 to
describe configurations in the puppet domain. The p2 system can be used
for other things than Java/OSGi/Eclipe and it is supported in the Nexus
repository manager.
It's nice to have a variety of interesting problems from which to
choose. :-)
Yes for sure :-)
Basically, I want to separate the two problems; describing "who can
do/provide what" and resolving that, and "what does the name x mean,
what is it bound to".
I will come back with some ideas for the binding of names - which I
think (for classes and resource types) is basically a operation for the
new Type system (and something that we discussed in the form of creating
a name as an alias for a type. It then follows naturally that this could
be done between resource types and classes as well. i.e.
something like:
type Resource[our::x] = type Resource[your::x]
type Class[our::y] = type Class[awesome::y]
Which is a general mechanism - e.g.
type MyStruct = Struct[
{ name => String[1,3], shape => Enum[big, small, green]}
]
(+ more to define types from scratch)
After that point, the aliased type is simply used instead of the
original type.
Resource[our::x] { title:
bar => 10
}
or indeed
our::x { title:
bar => 10
}
which means the same thing, since our::x can only be a resource type at
that point and would resolve to Resource[our::x], which is an alias for
whatever was defined.
Regards
- henrik
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/loanvi%24fv8%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.
