[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935157#comment-16935157
]
Noble Paul edited comment on SOLR-13661 at 9/21/19 9:35 PM:
------------------------------------------------------------
bq.How would e.g. SolrCell / Tika contrib be defined as a package? It has some
42 dependency jars. Guess through a blobs array?
Yes.
bq.If you DELETE a package a package through /api/cluster/packages,
No, Package Loading system does not delete blobs. This has to be a separate
operation and we will offer a separate API to clean unused blobs
bq.Why do you need a package="pkg_name"...
This is important for backward compatibility. Otherwise how do you know that
our plugin should use a class from a package or the per core classloader? This
will break existing deployments
Isolation is important. IRL people will get their packages from different
repositories. May be there is a less invasive way. But, we will solve it
another time
bq.u say that the version is not used by the system....
The Package manager CLI ([~ichattopadhyaya] will add more details soon) .It
maintains more data on this
This document is about the internal Package Loader/classloader management and
the APIs for the power users. This document has the bare minimum info required
to make it work
was (Author: noble.paul):
bq.How would e.g. SolrCell / Tika contrib be defined as a package? It has some
42 dependency jars. Guess through a blobs array?
Yes.
bq.If you DELETE a package a package through /api/cluster/packages,
No, Package system does not delete blobs. This has to be a separate operation
and we will offer a separate API to clean unused blobs
bq.hy do you need a package="pkg_name"...
This is important for backward compatibility. Otherwise how do you know that
our plugin should use a class from a package.
Isolation is important. IRL people will get their packages from different
repositories. May bne there is a less inbvasive way. But, we will solve it
another time
bq.u say that the version is not used by the system....
The Package manager CLI ([~ichattopadhyaya] will add more details soon) .It
maintains more data on this
This document is about the internal classloader management and the APIs for the
power users. This documentjust has the bare minimum info required to make it
work
> A package management system for Solr
> ------------------------------------
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Noble Paul
> Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
>
> Solr needs a unified cohesive package management system so that users can
> deploy/redeploy plugins in a safe manner. This is an umbrella issue to
> eventually build that solution
> h2. What is a package?
> A package has 2 parts
> * The {{jar}} binaries which may contain standard plugins (or even other
> libraries). It does not expect anything special. Basically , any jar file can
> be added into a package
> * The configuration in ZK. The structure is as follows
> {code:json}
> {
> "name": "package-name",
> "version": "some-version-string",
> "blob": {
> "sha256": "the-sha256-of-the-blob",
> "sig": "signature-signed-with-your-private-key",
> "name": "some-identifier"
> }
> }
> // use the following format for your multi jar package
> {
> "name": "package-name",
> "version": "some-version-string",
> "blobs": [
> {
> "sha256": "the-sha256-of-the-blob",
> "sig": "signature-signed-with-your-private-key",
> "name": "some-identifier"
> }
> ]
> }
> {code}
> * {{*name*}} : this is identifier used to access the package
> * {{*version*}}: This is not consumed by the system. But, it is required so
> that the user can make out which version is being used
> * {{*blob/sha256*}}: The sha256 of the blob. You should have uploaded this
> blob into the new FS blob store
> * {{*blob/sig*}} : signature. ensure that your public keys are already there
> in {{/keys/exe}} in ZK
> * {{blob/name}} : a friendly identifier (not required)
> Voila! this is all you need to have a package in Solr
> Every package gets its own isolated unique classloader in a Solr node. The
> classloader is a child of the {{CoreContainer}} classloader. Basically, your
> classes have to be either in the root classloader or in the blobs in your
> package
> h2. What are the components of this system?
> h3. {{PackageResourceLoader}} class
> This is a standard {{SolrResourceLoader}} . It has all the blobs of a package
> in its classpath.
> h3. {{PackageManager}} class
> This is a registry of all Package classloaders . essentially a {{Map<String,
> PackageResourceLoader>}}
> This class is responsible for listening to the state changes in ZK to the
> packages ADD/UPDATE/DELETE operations.
> * ADD : create a new {{PackageResourceLoader}} and add it to the registry
> * UPDATE: create a new {{PackageResourceLoader}}, update the registry and
> notify every listener
> * DELETE: remove a {{PackageResourceLoader}}
> h4. The Public APIs
> * {{GET /api/cluster/packages}} : Get a list of all registered packages
> * {{POST /api/cluster/packages}}
> ** {{add}} command: create a new package
> ** {{add}} command: update an existing package
> ** {{delete}} command: delete an existing package
> h4. configuring your plugins to use packages
> * in your {{solrconfig.xml}}
> {code:xml}
> <requestHandler name="/my_handler" class="my.full.ClassName"
> package="pkg_name">
> </requestHandler>
> {code}
> * using config API
> {code:xml}
> {
> "add-requesthandler": {"name":""/my_handler", "class":"my.full.ClassName",
> "package":"pkg_name"
> }
> }
> {code}
> h4. Example hot loading schema from package (TBD)
> {code:java}
> <schemaFactory class="ClassicIndexSchemaFactory" package= "pkg_name"/>
> {code}
> h4. verifying your components
> {code:java}
> GET /solr/<collection-name>/config/<component-type>/component-name?meta=true
> {code}
> This API would return the component configuration and curerent package being
> used as follows
> {code:json}
> {
> "config": {
> "requestHandler": {
> "/my_handler": {
> "name": "/my_handler",
> "class": "org.apache.solr.core.RuntimeLibReqHandler",
> "package": "global",
> "_packageinfo_": {
> "name": "global",
> "version": "1",
> "blob": {
> "sha256":
> "e1f9e23988c19619402f1040c9251556dcd6e02b9d3e3b966a129ea1be5c70fc",
> "sig":
> "L3q/qIGs4NaF6JiO0ZkMUFa88j0OmYc+I6O7BOdNuMct/xoZ4h73aZHZGc0+nmI1f/U3bOlMPINlSOM6LK3JpQ=="
> }
> }
> }
> }}}
> {code}
> h3. The Listeners
> Listeners listen to changes to packages. They get notified and they can take
> appropriate action
> h3. {{PackagePluginHolder}}
> This is a special {{PluginHolder}} that is a {{PackageListener}} . Whenever
> it gets a notification of a package update , it reinitializes the plugin it's
> holding and throw away the old one
> h2. {{FsBlobStore}} a.k.a FIle System Blob Store
> This is a fully replicated file system blob store.
> * It lives at {{<solr.home>/blobs}} on each Solr node
> * Every blob is a file. The file name is the sha256 of the blob
> When the {{FsBlobStore}} is requested a blob, it
> * looks up in the local file store for the file
> * If not found, it asks the blob store in every other node in the cluster
> one by one
> ** After it finds a node, the blob is downloaded and persisted to the local
> file system
> h3. public APIs
> * {{POST /api/cluster/blob}} : create a new blob in the system. The
> {{sha256}} is computed and it is saved as a file.
> * * This API asks other nodes to download the blob.
> * {{GET /api/node/blob}} : List of all the locally available blobs
> * {{GET /api/node/blob/<sha256>}} : Download a blob
> h2. The CLI PackageManager
> <TBD>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
