Network fencing is working correctly therefore credentials should be fine. Maybe there are differences between Azure Cloud in Europe and US because I get these errors only with our clusters in US Central and US East 2 but not in Europe West and North. Any hint?
Thomas -----Original Message----- From: Users [mailto:[email protected]] On Behalf Of Oyvind Albrigtsen Sent: Dienstag, 12. Februar 2019 09:51 To: Cluster Labs - All topics related to open-source clustering welcomed <[email protected]> Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog At least that limits what the issue could be. I'd check if the credentials allows getting and setting network-settings: https://github.com/ClusterLabs/fence-agents/blob/master/agents/azure_arm/fen ce_azure_arm.py#L204 On 11/02/19 16:46 +0100, Thomas Berreis wrote: >Hi Oyvind, > >Debug mode doesn't help me. The error occurs after successfully getting >Bearer token. No other issues are shown: > >DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks. >2019-02-11 15:27:12,687 DEBUG: Invoking 0 token response hooks. >DEBUG:requests_oauthlib.oauth2_session:Obtained token {u'resource': >u'https://management.core.windows.net/', u'access_token': ***', >u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at': >1549902431.687557, u'token_type': u'Bearer', u'not_before': >u'1549898532', >u'expires_on': u'1549902432'}. >2019-02-11 15:27:12,687 DEBUG: Obtained token {u'resource': >u'https://management.core.windows.net/', u'access_token': u'***', >u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at': >1549902431.687557, u'token_type': u'Bearer', u'not_before': >u'1549898532', >u'expires_on': u'1549902432'}. >WARNING:msrestazure.azure_active_directory:Keyring cache token has failed: >No recommended backend was available. Install the keyrings.alt package >if you want to use the non-recommended backends. See README.rst for details. >2019-02-11 15:27:12,687 WARNING: Keyring cache token has failed: No >recommended backend was available. Install the keyrings.alt package if >you want to use the non-recommended backends. See README.rst for details. >DEBUG:requests_oauthlib.oauth2_session:Encoding `client_id` "***" with >`client_secret` as Basic auth credentials. >2019-02-11 15:27:12,727 DEBUG: Encoding `client_id` "***" with >`client_secret` as Basic auth credentials. >DEBUG:requests_oauthlib.oauth2_session:Requesting url >https://login.microsoftonline.com/***/oauth2/token using method POST. >2019-02-11 15:27:12,727 DEBUG: Requesting url >https://login.microsoftonline.com/***/oauth2/token using method POST. > >2019-02-11 15:27:13,072 DEBUG: Obtained token {u'resource': >u'https://management.core.windows.net/', u'access_token': u'e***w', >u'ext_expires_in': u'3600', u'expires_in': u'3600', u'expires_at': >1549902433.072768, u'token_type': u'Bearer', u'not_before': >u'1549898533', >u'expires_on': u'1549902433'}. >WARNING:msrestazure.azure_active_directory:Keyring cache token has failed: >No recommended backend was available. Install the keyrings.alt package >if you want to use the non-recommended backends. See README.rst for details. >2019-02-11 15:27:13,073 WARNING: Keyring cache token has failed: No >recommended backend was available. Install the keyrings.alt package if >you want to use the non-recommended backends. See README.rst for details. >INFO:root:getting power status for VM node01 > >Thomas > >-----Original Message----- >From: Users [mailto:[email protected]] On Behalf Of Oyvind >Albrigtsen >Sent: Montag, 11. Februar 2019 14:58 >To: Cluster Labs - All topics related to open-source clustering >welcomed <[email protected]> >Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog > >You can try adding verbose=1 and see if that gives you some more >details on where it's failing. > >On 11/02/19 14:47 +0100, Thomas Berreis wrote: >>We need this feature because shutdown / reboot takes too much time ( > >>5 >>min) and network fencing fences the virtual machine much faster ( < 5 sec). >>We finished all the required steps and network fencing works as >>expected but I'm still confused about these errors in the log and the >>failure counts showed by pcs ... >> >>fence_azure_arm: Keyring cache token has failed: No recommended >>backend was available. Install the keyrings.alt package if you want to >>use the non-recommended backends. See README.rst for details. >> >>stonith-ng[7789]: warning: fence_azure_arm[7896] stderr: [ 2019-02-11 >>13:41:19,178 WARNING: Keyring cache token has failed: No recommended >>backend was available. Install the keyrings.alt package if you want to >>use the non-recommended backends. See README.rst for details. ] >> >>stonith-net_monitor_60000 on node02 'unknown error' (1): call=369, >>status=Timed Out, exitreason='', last-rc-change='Mon Feb 11 10:38:02 >>2019', queued=0ms, exec=20007ms >> >>Thomas >> >>-----Original Message----- >>From: Users [mailto:[email protected]] On Behalf Of Oyvind >>Albrigtsen >>Sent: Montag, 11. Februar 2019 13:58 >>To: Cluster Labs - All topics related to open-source clustering >>welcomed <[email protected]> >>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog >> >>Oh. Thanks for clarifying. >> >>Network-fencing requires extra setup as explained if you run "pcs >>stonith resource describe fence_azure_arm". >> >>You probably dont need network-fencing, so you can just skip that part. >> >>On 11/02/19 13:50 +0100, Thomas Berreis wrote: >>>> It seems like the issue might be the space in "pcmk_host_list= node01". >>>Sorry, this typo was only in my mail because I anonymized in- and output. >>> >>>The issue only occurs when I add the parameter "network-fencing=on". >>> >>>WARNING:msrestazure.azure_active_directory:Keyring cache token has failed: >>>No recommended backend was available. Install the keyrings.alt >>>package if you want to use the non-recommended backends. See >>>README.rst for >>details. >>>2019-02-11 12:29:37,079 WARNING: Keyring cache token has failed: No >>>recommended backend was available. Install the keyrings.alt package >>>if you want to use the non-recommended backends. See README.rst for details. >>> >>> >>>Thomas >>> >>>-----Original Message----- >>>From: Users [mailto:[email protected]] On Behalf Of >>>Oyvind Albrigtsen >>>Sent: Montag, 11. Februar 2019 10:38 >>>To: Cluster Labs - All topics related to open-source clustering >>>welcomed <[email protected]> >>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog >>> >>>pcmk_host_list is used by Pacemaker to limit which hosts can be >>>fenced by the STONITH device, so you'll have to use the plug >>>parameter for >>testing. >>>Pacemaker uses it internally when it fences a host, so you wouldnt >>>use plug= as part of the pcs stonith create command. >>> >>>It seems like the issue might be the space in "pcmk_host_list= node01". >>> >>>Oyvind >>> >>>On 10/02/19 14:43 +0100, Thomas Berreis wrote: >>>>Hi Oyvind, >>>> >>>>Seems, option "pcmk_host_list" is not supported by fence_azure_arm. >>>>Instead fence_azure_arm requires "plug" and however this is not >>>>supported >>>by pcs. >>>>What's the correct syntax to add such a stonith device with >>>fence_azure_arm? >>>> >>>>[root ~]# fence_azure_arm >>>>login=*** >>>>network-fencing=on >>>>passwd=*** >>>>pcmk_host_list= node01 >>>>resourceGroup=*** >>>>retry_on=0 >>>>subscriptionId=*** >>>>tenantId=*** >>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list= >node01' >>>> >>>>ERROR:root:Failed: You have to enter plug number or machine >>>>identification >>>> >>>>2019-02-10 12:54:51,862 ERROR: Failed: You have to enter plug number >>>>or machine identification >>>> >>>>------------------------------ >>>> >>>>[root ~]# fence_azure_arm >>>>login=*** >>>>network-fencing=on >>>>passwd=*** >>>>pcmk_host_list= node01 >>>>resourceGroup=*** >>>>retry_on=0 >>>>subscriptionId=*** >>>>tenantId=*** >>>>plug=node01 >>>>WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list= >node01' >>>> >>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has >failed: >>>>No recommended backend was available. >>>>2019-02-10 12:56:30,377 WARNING: Keyring cache token has failed: No >>>>recommended backend was available. Install th >>>>WARNING:msrestazure.azure_active_directory:Keyring cache token has >failed: >>>>No recommended backend was available. >>>>2019-02-10 12:56:30,729 WARNING: Keyring cache token has failed: No >>>>recommended backend was available. Install th >>>>Success: Powered OFF >>>> >>>>------------------------------ >>>> >>>>[root ~]# fence_azure_arm >>>>login=*** >>>>network-fencing=on >>>>passwd=*** >>>>plug=node01 >>>>resourceGroup=*** >>>>retry_on=0 >>>>subscriptionId=*** >>>>tenantId=*** >>>> >>>>Success: Powered OFF >>>> >>>>------------------------------ >>>> >>>>[root ~]# pcs stonith create stonith-net fence_azure_arm [...] >>>>plug=node01 network-fencing=on >>>>Error: invalid stonith option 'plug' >>>> >>>>[root ~]# fence_azure_arm -h >>>>Usage: >>>> fence_azure_arm [options] >>>>Options: >>>>[...] >>>> -n, --plug=[id] Physical plug number on device, UUID or >>>>identification of machine >>>> >>>> >>>>Thanks! >>>> >>>>Thomas >>>> >>>>-----Original Message----- >>>>From: Users [mailto:[email protected]] On Behalf Of >>>>Oyvind Albrigtsen >>>>Sent: Freitag, 8. Februar 2019 14:16 >>>>To: Cluster Labs - All topics related to open-source clustering >>>>welcomed <[email protected]> >>>>Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog >>>> >>>>Hi Thomas, >>>> >>>>It looks like it's trying to use credentials from keyring, which >>>>generates these error messages. >>>> >>>>Maybe there's a typo in one of the authentitcation parameters in >>>>your config. Can you post the output of "pcs stonith show --full"? >>>> >>>>You can try running fence_azure_arm without parameters on the >>>>command line and copy the parameters (one per line) to it, and add e.g. >>>>action=status and end with Ctrl+D to see . >>>> >>>> >>>>Oyvind >>>> >>>>On 08/02/19 14:02 +0100, Thomas Berreis wrote: >>>>>Hi, >>>>> >>>>> >>>>> >>>>>I'm having a two-node cluster in Azure with fencing enabled via >>>>>module fence_azure_arm. >>>>> >>>>>The cluster management is flooding my syslog on both nodes with >>>>>these >>>>>messages: >>>>> >>>>> >>>>> >>>>>** fence_azure_arm: Keyring cache token has failed: No recommended >>>>>backend was available. Install the keyrings.alt package if you want >>>>>to use the non-recommended backends. See README.rst for details. >>>>> >>>>> >>>>> >>>>>** stonith-ng[5220]: warning: fence_azure_arm[14847] stderr: [ >>>>>2019-02-08 >>>>>12:52:02,703 WARNING: Keyring cache token has failed: No >>>>>recommended backend was available. Install the keyrings.alt package >>>>>if you want to use the non-recommended backends. See README.rst for >>>>>details. ] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>Can anyone give me a hint how to fix these issue? >>>>> >>>>> >>>>> >>>>>pcs version: 0.9.165 >>>>> >>>>>pacemaker version: 1.1.19-8.el7_6.1 >>>>> >>>>>fence_azure_arm version: 4.2.1 >>>>> >>>>>corosync 2.4.3 >>>>> >>>>> >>>>> >>>>>Thanks! >>>>> >>>>> >>>>> >>>>>Thomas >>>>> >>>> >>>>>_______________________________________________ >>>>>Users mailing list: [email protected] >>>>>https://lists.clusterlabs.org/mailman/listinfo/users >>>>> >>>>>Project Home: http://www.clusterlabs.org Getting started: >>>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>Bugs: http://bugs.clusterlabs.org >>>> >>>>_______________________________________________ >>>>Users mailing list: [email protected] >>>>https://lists.clusterlabs.org/mailman/listinfo/users >>>> >>>>Project Home: http://www.clusterlabs.org Getting started: >>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>Bugs: http://bugs.clusterlabs.org >>>> >>>>_______________________________________________ >>>>Users mailing list: [email protected] >>>>https://lists.clusterlabs.org/mailman/listinfo/users >>>> >>>>Project Home: http://www.clusterlabs.org Getting started: >>>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>Bugs: http://bugs.clusterlabs.org >>>_______________________________________________ >>>Users mailing list: [email protected] >>>https://lists.clusterlabs.org/mailman/listinfo/users >>> >>>Project Home: http://www.clusterlabs.org Getting started: >>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>Bugs: http://bugs.clusterlabs.org >>> >>>_______________________________________________ >>>Users mailing list: [email protected] >>>https://lists.clusterlabs.org/mailman/listinfo/users >>> >>>Project Home: http://www.clusterlabs.org Getting started: >>>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>Bugs: http://bugs.clusterlabs.org >>_______________________________________________ >>Users mailing list: [email protected] >>https://lists.clusterlabs.org/mailman/listinfo/users >> >>Project Home: http://www.clusterlabs.org Getting started: >>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>Bugs: http://bugs.clusterlabs.org >> >>_______________________________________________ >>Users mailing list: [email protected] >>https://lists.clusterlabs.org/mailman/listinfo/users >> >>Project Home: http://www.clusterlabs.org Getting started: >>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>Bugs: http://bugs.clusterlabs.org >_______________________________________________ >Users mailing list: [email protected] >https://lists.clusterlabs.org/mailman/listinfo/users > >Project Home: http://www.clusterlabs.org Getting started: >http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >Bugs: http://bugs.clusterlabs.org > >_______________________________________________ >Users mailing list: [email protected] >https://lists.clusterlabs.org/mailman/listinfo/users > >Project Home: http://www.clusterlabs.org Getting started: >http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >Bugs: http://bugs.clusterlabs.org _______________________________________________ Users mailing list: [email protected] https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org _______________________________________________ Users mailing list: [email protected] https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
