My personal expectation is that Jini is entrenched on the Java serialization
quagmire that you probably have to work around a lot more problems than the
myriad of discovery and transport protocols that are currently available in
Java?
It is a pity because Jini and OSGi started in the same time and we had
discussions together.
Kind regards,
Peter Kriens
> On 21 Feb 2017, at 07:25, Christian Schneider <[email protected]> wrote:
>
> Have you thought about implementing the jini support as a discovery and
> transport for Remote Service Admin?
> I am not sure if it fully matches as jini is special in some aspects but I
> see many similarities.
>
> Christian
>
> 2017-02-21 0:56 GMT+01:00 Peter <[email protected] <mailto:[email protected]>>:
>
> Sounds like you've been burnt.
>
> Modular or die, no hacked compromises, won't support unworkable complex
> non-modular cases, only support what works.
>
> For updates:
> 1. Server module advises proxy's of graceful shutdown, proxy's unregister
> services with OSGi registrar (or just stop working in hard shut or
> disconnection, in which case they are discarded).
> 2. Server unregisters from Jini service registrar and unexports.
> 3. Server performs upgrades
> 4. Server exports and reregisters with jini service registrar
> 5. Service is dynamically discovered by ServiceDiscoveryManager in clients,
> smart proxy's are provisioned and registered with local OSGi registrar.
> 6. Clients notified of service availability.
>
> The smart proxy bundle must be of identical version at both endpoints,
> connection must be refused if this is not the case.
>
> Service api bundles are separate from smart proxy bundles.
>
> Clients don't import packages from smart proxy.
>
> The smart proxy contains the api for communication between proxy and remote
> service only. Only service implementation imports packages from smart proxy.
> Communication method and communication api is a private implementation
> concern.
>
> The services smart proxy imports the service api packages, as does the
> client. Service api is compatibility layer between smart proxy and client.
> The smart proxy is a provider of the service api, the client is the consumer.
>
> Service api update is client responsibility, not likely to update often.
> Prefer to add mixin interfaces or replace api (new service type) over
> breaking change. Smart proxy needs to deregister when service api is
> updated (should listen for event notifications).
>
> Service will only be rediscovered if it is compatible.
>
> As the smart proxy needs to be resourced and provisioned it is this stage of
> the process where it may not be found api compatible with the client, in
> which case it will not be discovered by the client...
>
> The service can advertise its requirements (incl packages and versions) via
> the Jini service registrar, services that are not api compatible will be
> filtered out.
>
> The server / service remains responsible for maintaining compatibility.
>
> Discovery is dynamic.
>
> We've had some discussion on river-dev, one user wanted much wider support
> scope including remote objects with code download that are not services, also
> wanted to use codebase annotations and non osgi URLClassLoaders with bundle
> ClassLoaders as parents, using Context ClassLoader to find bundle
> ClassLoader. Agree that's an unworkable scenario due to complexity.
>
> I want to keep this as simple as possible for obvious reasons. Force
> communication method to be an implementation concern, allowing it to be
> replaced in future.
>
> Regards,
>
> Peter.
>
> Sent from my Samsung device.
>
> ---- Original message ----
> From: Peter Kriens <[email protected] <mailto:[email protected]>>
> Sent: 21/02/2017 12:24:39 am
> To: OSGi Developer Mail List <[email protected]
> <mailto:[email protected]>>
> Subject: Re: [osgi-dev] OSGifying an existing application
>
> The problem is that in all distributed I’ve been involved in there were
> rolling updates. This implies that you cannot guarantee that each server runs
> the same bundle. And clusters that do not do rolling updates seem kind of
> useless because they introduce a huge failure point. So I have a hard time
> understanding how you can guarantee that invariant? Yes, it works most of the
> time but when I learned to work with computers that tended not be good enough.
>
> Now I do understand ( and sympathise) that you cannot change the app. I’ve
> been there.
>
> You indicate you have the same bundle on both sides. So the class graph on
> both sides is equal. This implies that if you ensure that the API interface
> is in the same bundle as the implementation classes then you’re safe as long
> as you use the interface’s class loader as the root loader for classes you
> find. This is of course highly not modular but then you’re not anyway under
> the covers :-) You could also make the API bundle import the implementation
> bundles, that should work as well.
>
> As long as you make sure OSGi has the proper dependency information things
> tend to work out of the box.
>
> Kind regards,
>
> Peter Kriens
>
>
>
>> On 20 Feb 2017, at 11:47, Peter <[email protected] <mailto:[email protected]>>
>> wrote:
>>
>> Thanks Pete, good to hear from you again, I must admit it's been too long.
>> We last spoke when I was refactoring a class dependency tool to use ASM
>> instead of the jdk's tools.jar. You once asked, how do you find a
>> dependency for calls to Class.forName?
>>
>> The reasons you've stated are also why I've chosen to support a very narrow
>> use case, in which you may have already noted that the serialized connection
>> is between two identical bundles, in separate jvm's with compatible package
>> imports.
>>
>> There’s no intent to support transferring any classes outside of the Service
>> Now API and that includes overriding classes. What you describe about data
>> hiding reminds me of Entry's, which have public fields.
>>
>> I'll be the first to admit there are significant issues with the design of
>> Java's Serialization's extralinguistic api. Ironically though the wire
>> protocol is reasonably well thought out, with regards to evolution.
>>
>> As an exercise to fix security issues, I have reimplemented Java
>> serialization with input validation using a public api, it has backward
>> compatible serial form, but only supports a subset of Java serialization, it
>> doesn't support circular object graphs for example as this would compromise
>> security. It performs input validation, sets resource limits and expects
>> periodic stream resets to avoid DOS and gadget attacks. The problem is
>> there's a lot of existing software that utilises java Serialization, that's
>> going to need support for some time. Things like Serialization and Remote
>> method invocation are damaged by attempts to implement too much
>> functionality, when a more rigid subset would avoid a number of issues. But
>> I guess no one was thinking of modularity and versioning when they created
>> these frameworks either.
>>
>> James Gosling said something once about why Generics weren't included in
>> Java from the outset, which was because at the time they didn't know how to
>> do it properly, it's better leaving it out until you do.
>>
>> JBoss has a nice web page with some graphics that illustrate some major
>> issues with implementation hiding you've mentioned with Serialization and
>> modular frameworks here:
>> https://developer.jboss.org/wiki/ModularSerialization
>> <https://developer.jboss.org/wiki/ModularSerialization>
>>
>> It's worth noting Service API of the smart proxy bundle doesn't need to be
>> Serializable, instead it's relegated to a communication means between two
>> identical bundles in different JVM's. It's also important to recognise
>> that it doesn't need to be the communcation mechanism either. These
>> bundles have an identical class namespace, although there may be variances
>> in package import versions.
>>
>> Yes we are also looking at moving away from java serialization.
>>
>> Also over time, because this is a service interface, at some point down the
>> track, serialization can be replaced, without impacting the public api. So
>> yes the underlying protocols can be stripped out to data and message passing
>> if that's more satisfactory.
>>
>> So yes java serialization is an existing part of our application and it has
>> it's warts.
>>
>> But we've also had a number of users over the years who have requested
>> support for OSGi.
>>
>> This is not a greenfields project, I'm hoping that I'm not going to be told
>> that no, the chasm is too wide you can't cross over to OSGi, rewrite or
>> start again, there's just too many LOC.
>>
>> So you have raised some important questions. Some of our users have had a
>> lot of success with Maven (recognising there a pro's and cons's with module
>> based versioning and transitive dependencies), where versioning on a module
>> level allows codebase annotations to be utilised in remote invocation,
>> avoiding class visibility issues by mapping module ClassLoader's directly to
>> a URI based identity. However with OSGi there's a mismatch between
>> different jvm's and how bundles and packages imports will be resolved will
>> end up being wired, so we can't rely on codebase annotations for OSGi.
>>
>> Jvm's using OSGi frameworks are quite likely to have different dependency
>> graphs (wires) between bundles and their package imports.
>>
>> While I don't expect to solve the worlds problems or boil the ocean, I'm
>> looking for the most workable compromise, one that doesn't promise the world
>> and is easier to explain what users can and can't expect to do. I'm
>> relatively pragmatic. To me it would seem logical that a subset where two
>> identical bundles (that should have resolved similar package import
>> versions) should be a good place to start.
>>
>> Hence my post on this list, as I realise many of you have already spent a
>> lot of time bumping into these issues.
>>
>> Cheers,
>>
>> Peter.
>>
>> On 20/02/2017 6:38 PM, Peter Kriens wrote:
>>> After working in this area for too many years I’ve come to the conclusion
>>> that objects cannot be really transferred to other systems in a reliable
>>> way, only self typed data can. JPA, RMI, and many other systems promise
>>> heaven to the programmer that they can use their objects local and remote
>>> transparently. The consequence of this dream is a huge amount of complexity
>>> that far outweighs any gains in programmer friendliness. Few things have
>>> caused so much trauma in the software world as ORM. (Persistence is
>>> communications to a future process.)
>>>
>>> The reason objects are so complex to use in communications is that it is in
>>> direct violation of the goal of OO to hide your data. However, once you
>>> expose the internal data on the wire you have effectively made it public
>>> but too many people they can still have the advantages of abstract data
>>> types. OSGi is a bitch in this case because it tells you that you’re trying
>>> to do something wrong by refusing to cooperate. In this case, it balks at
>>> you because you create an invisible dependency between the sender and the
>>> receiver. Though this is a good thing too often the receivers of this
>>> message blame the messenger.
>>>
>>> You can handle this dependency but you’ll find out is that it is a hugely
>>> complex task that introduces a lot of frailty in the overall system. Having
>>> tried this several times I can assure you that any gains in programmer
>>> friendliness are dwarfed by the complexity of creating this facade.
>>>
>>> The best solution I found is to give up on data hiding. The fact your
>>> objects is on the wire means that that wire format is public. I therefore
>>> use Data Transfer Objects, in my case objects with public fields. On both
>>> sides I have my own objects to provide behavior to this data with methods
>>> and classes but this data record is at the core of my code. Since this data
>>> is public because it goes over the wire it is better to wrap you code
>>> around that ‘standardized public’ object than to try you internal object
>>> data.
>>>
>>> If you look at the OSGi specifications of the past 5 year then you will
>>> notice that all applicable APIs have been designed to be useful with
>>> Distributed OSGi. Calls do not pass objects but they pass DTOs back and
>>> forth. They do not rely that the receiver and sender have exactly the same
>>> type and version. In this model it is easy to replace an endpoint using
>>> another language, which is a really good sign.
>>>
>>> For Java developers this is often an unpleasant message, and quite often
>>> OSGi get the blame. However, the fact OSGi gives you these problems means
>>> that you’re trying to do something that has hidden dependencies.
>>>
>>> Distributed computing has 7 well known fallacies[1] but I strongly believe
>>> that there is an eighth: ’One can communicate objects over a network’.
>>>
>>> Now your question. Yes, you could run a resolve and load the proper bundles
>>> but you introduce a huge amount of error cases and a large amount of
>>> complexity and you won’t solve the fundamental problem
>>>
>>> Kind regards,
>>>
>>> Peter Kriens
>>>
>>> [1]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
>>> <https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing>
>>>
>>>
>>>> On 20 Feb 2017, at 05:13, Peter <[email protected]
>>>> <mailto:[email protected]> <mailto:[email protected]
>>>> <mailto:[email protected]>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I'm currently working on converting an existing application to OSGi.
>>>>
>>>> This application has a network service architecture based on java
>>>> interfaces. I've broken the application into modules, using a Maven
>>>> build, which uses bnd and bndtools to create bundle manifests. Some of
>>>> these modules are ServiceLoader provider's, so I've used annotations to
>>>> ensure these are loaded into the OSGi service registry using the Service
>>>> Loader Mediator.
>>>>
>>>> The main issue that I face is this application is a networked application
>>>> and has it's own Remote Invocation protocols (which currently utilise Java
>>>> Serialization, but not Java RMI). As you'll appreciate, class visiblity
>>>> is a little different in OSGi. :)
>>>>
>>>> The services mentioned above are remote services, these remote services
>>>> have a proxy which implements the service interface, these services are
>>>> discovered and installed at the client. There are two types of proxy's,
>>>> one, called a smart proxy, requires a codebase from which to retrieve a
>>>> jar or jar files that are downloaded and installed at the cleint
>>>> (traditionally during deserialization), the other type of proxy is called
>>>> a dynamic proxy (it's basically just an instance of
>>>> java.lang.reflect.Proxy), which is dynamically generated at the client.
>>>>
>>>> The Service implementation is broken up into three components:
>>>>
>>>> 1. The service api
>>>> 2. The smart proxy (resolved and provisioned into in client jvm).
>>>> 3. The server
>>>>
>>>> The server bundle imports packages from the smart proxy bundle, while the
>>>> smart proxy imports packages from the service api as well as exporting
>>>> it's own packages, as required by the server bundle.
>>>>
>>>> The server that provides the remote service has three bundles loaded;
>>>> server-impl, smart-proxy & service-api.
>>>>
>>>> The client only has the service api bundle installed at deployment and the
>>>> smart proxy is resolved and provisioned before the service is made
>>>> available via the local OSGi service registry, where the client will learn
>>>> of it's existence using ServiceTracker.
>>>>
>>>> At first glance only the smart proxy bundle needs to be provisioned at the
>>>> client, however for cases where a dynamic proxy is required to implement
>>>> interfaces from different packages, where class visibility issues may
>>>> exist, it may be beneficial in these cases to utilise and provision a
>>>> proxy bundle that imports all these interfaces, one might do that by
>>>> taking advantage of java's interface multiple inheritance; create a bundle
>>>> that contains one interface (annotated with @ProviderType) which extends
>>>> all interfaces, which the bundle doesn't export, so we ensure that the
>>>> dynamic proxy has a proper bundle manifest with all package imports and
>>>> version ranges correctly defined.
>>>>
>>>> The inbuilt remote invocation protocol has server and client endpoints,
>>>> the protocol is extensible and has a number of implementations (for
>>>> example https, http, tls, kerberos, tcp). Each endpoint is assigned a
>>>> ClassLoader when it's created.
>>>>
>>>> For classes installed at the client, these are typically installed in a
>>>> URLClassLoader, typically with the Application loader as parent loader.
>>>> In an OSGi environment however, the smart proxy bundle will be installed
>>>> at the client, it's ClassLoader utilised by the client endpoint, the smart
>>>> proxy bundle will also be installed at the server and it's ClassLoader
>>>> utilised by the server endpoint. In this case the visibility of the
>>>> bundles at each endpoint will be utilised to resolve serializable classes.
>>>> Private smart proxy serializable classes will be resolvable at each
>>>> end, but only public classes from imported packages will be
>>>> deserializable, since the client interacts using the Service API, all
>>>> serializable classes in the Service API packages will need to be exported
>>>> and public and imported by the client and smart proxy.
>>>>
>>>> Once a bundle has been provisioned its ClassLoader will be given to the
>>>> client endpoint and the marshalled state of the proxy unmarshalled into
>>>> it. At this point the service that the proxy provides would be registered
>>>> with the OSGi service registry for the client to discover and consume.
>>>> The smart proxy communicates with it's server via an internal dynamic
>>>> proxy (java.lang.reflect.Proxy), it's used to invoke methods on the server.
>>>>
>>>> While the existing protocol uses Java serialization, it doesn't use Java
>>>> serialization's method of resolving classes Java Serialization walks the
>>>> stack and finds the first non system classloader (looking for the
>>>> application ClassLoader). The existing class resolution method isn't
>>>> suitable for OSGi, however the mechanism is extensible, so can be replaced
>>>> with something suitable.
>>>>
>>>>
>>>>
>>>> Does anyone have any advise or experience utilising the OSGi Enterprise
>>>> Resolver Service Specification (chapter 136) and the OSGi Enterprise
>>>> Repository Service Specification (chapter 132) to resolve and provision a
>>>> bundle for the smart proxy at the client?
>>>>
>>>>
>>>>
>>>> The intent here is the bundle manifests at each endpoint will be used to
>>>> determine class visiblity, so the resolution and provisioning process will
>>>> be of critical importance.
>>>>
>>>> For anyone curios, the application is a fork of Apache River / Jini and
>>>> I'm experimenting with support for OSGi. I'm also a committer and PMC
>>>> member of Apache River. This isn't the old Jini we all know and love
>>>> however, there are some additional features that allow provisioning to
>>>> occur using a feature called delayed unmarshalling, so we can avoid the
>>>> need for codebase annotations and URLClassLoaders.
>>>>
>>>> The work in progress can be found here, for anyone who's curious:
>>>>
>>>> https://github.com/pfirmstone/JGDMS/tree/Maven_build/modularize/JGDMS
>>>> <https://github.com/pfirmstone/JGDMS/tree/Maven_build/modularize/JGDMS>
>>>>
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>> _______________________________________________
>>>> OSGi Developer Mail List
>>>> [email protected] <mailto:[email protected]>
>>>> https://mail.osgi.org/mailman/listinfo/osgi-dev
>>>> <https://mail.osgi.org/mailman/listinfo/osgi-dev>
>>>
>>>
>>> _______________________________________________
>>> OSGi Developer Mail List
>>> [email protected] <mailto:[email protected]>
>>> https://mail.osgi.org/mailman/listinfo/osgi-dev
>>> <https://mail.osgi.org/mailman/listinfo/osgi-dev>
>>
>> _______________________________________________
>> OSGi Developer Mail List
>> [email protected] <mailto:[email protected]>
>> https://mail.osgi.org/mailman/listinfo/osgi-dev
>> <https://mail.osgi.org/mailman/listinfo/osgi-dev>
>
>
>
> _______________________________________________
> OSGi Developer Mail List
> [email protected] <mailto:[email protected]>
> https://mail.osgi.org/mailman/listinfo/osgi-dev
> <https://mail.osgi.org/mailman/listinfo/osgi-dev>
>
>
>
> --
> --
> Christian Schneider
> http://www.liquid-reality.de
> <https://owa.talend.com/owa/redir.aspx?C=3aa4083e0c744ae1ba52bd062c5a7e46&URL=http%3a%2f%2fwww.liquid-reality.de>
>
> Open Source Architect
> http://www.talend.com
> <https://owa.talend.com/owa/redir.aspx?C=3aa4083e0c744ae1ba52bd062c5a7e46&URL=http%3a%2f%2fwww.talend.com>_______________________________________________
> OSGi Developer Mail List
> [email protected]
> https://mail.osgi.org/mailman/listinfo/osgi-dev
_______________________________________________
OSGi Developer Mail List
[email protected]
https://mail.osgi.org/mailman/listinfo/osgi-dev
