I noticed a bug in Ambari while playing around with these settings. Ambari ignores the livy.server.recovery.state-store value and attempts to create a file bases state store even if state-store is set to zookeeper. The attempt fails because zkhost1:port,zkhost2:port,zkhost3:port,... is not a valid path on HDFS. It looks like it is related to this issue: https://issues.apache.org/jira/browse/AMBARI-25250
Thanks, Meisam On Mon, Apr 15, 2019 at 8:39 PM Marco Gaido <[email protected]> wrote: > I think in general our website documentation is not complete and I think it > would be great if we could improve it significantly. For instance, there is > no place where all configurations are described properly (other than > reading the code). I think that PRs and help in improving the doc would be > highly appreciated. > > Thanks, > Marco > > On Tue, 16 Apr 2019, 05:13 Meisam Fathi, <[email protected]> wrote: > > > Hi Peter, > > > > Are you using ZooKeeper for recovery store? > > If yes, in conf/livy.conf, is > > livy.server.recovery.zk-state-store.key-prefix > > set to different values in different Livy instances? If not, all of Livy > > instances will read/write the recovery data from/to the same path, which > is > > default is /livy/v1 by default. > > > > @dev mailing list: > > This behavior is not documented in livy.conf nor on the website. It might > > be a good idea to document it somewhere. > > > > Thanks, > > Meisam > > > > On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi <[email protected]> > > wrote: > > > > > Hi Peter, > > > > > > Livy 0.6 has a new feature to give each session a name: > > > https://github.com/apache/incubator-livy/pull/48 > > > > > > Would this feature be useful in your usecase? > > > > > > Thanks, > > > Meisam > > > > > > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) <[email protected]> > > > wrote: > > > > > >> Greetings, > > >> > > >> > > >> > > >> I have a custom service that connects to Livy, v0.4 soon to be v0.5 > once > > >> we go to HDP3. If sessions already exist it logs the session ID’s and > > >> starts using them, if sessions don’t exist it creates new ones. The > > problem > > >> is the account used to launch the Livy sessions is not unique to this > > >> service, nor is the kind of session. So sometimes it grabs other > > people’s > > >> sessions and absconds off with them. Also, there are multiple > instances > > of > > >> the service, running under the same account, and they are not supposed > > to > > >> use each other’s sessions… that’s not working out so well. > > >> > > >> > > >> > > >> The service names the sessions, but I can’t find any way to retrieve > > >> detailed session data so that I can update the service to check if the > > Livy > > >> Session belongs to the service or not. > > >> > > >> > > >> > > >> I found some older comments 2016/2017 about retrieving Livy sessions > by > > >> name. I don’t really need that, I just want to be able to read the > name > > >> through the regular sessions REST call. > > >> > > >> > > >> > > >> Any REST calls I missed, or undocumented calls… that can help? > > >> > > >> > > >> > > >> Thanks, > > >> > > >> Peter > > >> > > >> > > >> > > >> Ref: > > >> > > > https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name > > , > > >> https://issues.cloudera.org/browse/LIVY-41 > > >> > > >> > > >> > > >> > > >> > > > > > >
