Hi Dmitri
On Wed, Apr 1, 2015 at 3:06 PM, Dmitri Pal <[email protected]> wrote: > On 04/01/2015 07:52 AM, Traiano Welcome wrote: >> >> Hi Dmitri >> >> >> On Wed, Apr 1, 2015 at 2:23 PM, Dmitri Pal <[email protected]> wrote: >>> >>> On 04/01/2015 04:14 AM, Traiano Welcome wrote: >>>> >>>> Hi Martin >>>> >>>> Thanks for the response. Check results inline: >>>> >>>> >>>> On Wed, Apr 1, 2015 at 10:37 AM, Martin Babinsky <[email protected]> >>>> wrote: >>>>> >>>>> On 04/01/2015 09:20 AM, Traiano Welcome wrote: >>>>>> >>>>>> Some information from the dirsrv error log (sanitized: XYZ = realm): >>>>>> >>>>>> [01/Apr/2015:11:01:49 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 >>>>>> starting up >>>>>> [01/Apr/2015:11:01:49 +0300] schema-compat-plugin - warning: no >>>>>> entries set up under cn=computers, cn=compat,dc=idm,dc=local >>>>>> [01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password >>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>>>> should be added before the CoS Definition. >>>>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> cleanAllRUV task found, resuming the cleaning of rid(6)... >>>>>> [01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password >>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>>>> should be added before the CoS Definition. >>>>>> [01/Apr/2015:11:01:49 +0300] - slapd started. Listening on All >>>>>> Interfaces port 389 for LDAP requests >>>>>> [01/Apr/2015:11:01:49 +0300] - Listening on All Interfaces port 636 >>>>>> for LDAPS requests >>>>>> [01/Apr/2015:11:01:49 +0300] - Listening on >>>>>> /var/run/slapd-IDM-LOCAL.socket for LDAPI requests >>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>>>> Minor code may provide more information (No Kerberos credentials >>>>>> available)) >>>>>> [01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>>>> Minor code may provide more information (No Kerberos credentials >>>>>> available)) >>>>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - signaling >>>>>> operation >>>>>> threads >>>>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - waiting for 27 >>>>>> threads to terminate >>>>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - closing down >>>>>> internal subsystems and plugins >>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Cleaning rid (6)... >>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Waiting to process all the updates from the deleted replica... >>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Waiting for all the replicas to be online... >>>>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Server shutting down. Process will resume at server startup >>>>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed >>>>>> out) >>>>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -1 (Can't contact LDAP server) >>>>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact >>>>>> LDAP server) () >>>>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>>>> Minor code may provide more information (No Kerberos credentials >>>>>> available)) >>>>>> errors >>>>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication >>>>>> bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): >>>>>> generic failure: GSSAPI Error: Unspecified GSS failure. Minor code >>>>>> may provide more information (No Kerberos credentials available)) >>>>>> [01/Apr/2015:11:02:09 +0300] - Waiting for 4 database threads to stop >>>>>> [01/Apr/2015:11:02:10 +0300] - All database threads now stopped >>>>>> [01/Apr/2015:11:02:10 +0300] - slapd stopped. >>>>>> [01/Apr/2015:10:15:39 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 >>>>>> starting up >>>>>> [01/Apr/2015:10:15:39 +0300] schema-compat-plugin - warning: no >>>>>> entries set up under cn=computers, cn=compat,dc=idm,dc=local >>>>>> [01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password >>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>>>> should be added before the CoS Definition. >>>>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> cleanAllRUV task found, resuming the cleaning of rid(6)... >>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password >>>>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>>>> should be added before the CoS Definition. >>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 2 (No such file or directory) >>>>>> [01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time >>>>>> skew (-2771 secs). Current seqnum=3 >>>>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>>>> Minor code may provide more information (No Kerberos credentials >>>>>> available)) >>>>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>>>> [01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time >>>>>> skew (-2770 secs). Current seqnum=1 >>>>>> [01/Apr/2015:10:15:39 +0300] - slapd started. Listening on All >>>>>> Interfaces port 389 for LDAP requests >>>>>> [01/Apr/2015:10:15:39 +0300] - Listening on All Interfaces port 636 >>>>>> for LDAPS requests >>>>>> [01/Apr/2015:10:15:39 +0300] - Listening on >>>>>> /var/run/slapd-IDM-LOCAL.socket for LDAPI requests >>>>>> [01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>>>> Minor code may provide more information (No Kerberos credentials >>>>>> available)) >>>>>> [01/Apr/2015:10:15:40 +0300] csngen_new_csn - Warning: too much time >>>>>> skew (-2771 secs). Current seqnum=1 >>>>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - signaling >>>>>> operation >>>>>> threads >>>>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - waiting for 28 >>>>>> threads to terminate >>>>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - closing down >>>>>> internal subsystems and plugins >>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Cleaning rid (6)... >>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Waiting to process all the updates from the deleted replica... >>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Waiting for all the replicas to be online... >>>>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>>>> Server shutting down. Process will resume at server startup >>>>>> [01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed >>>>>> out) >>>>>> [01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -1 (Can't contact LDAP server) >>>>>> [01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact >>>>>> LDAP server) () >>>>>> [01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389): >>>>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>>>> Minor code may provide more information (No Kerberos credentials >>>>>> available)) >>>>>> [01/Apr/2015:10:15:59 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>>>> GSS failure. Minor code may provide more information (No Kerberos >>>>>> credentials available)) errno 0 (Success) >>>>>> [01/Apr/2015:10:15:59 +0300] slapi_ldap_bind - Error: could not >>>>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>>>> error -2 (Local error) >>>>>> [01/Apr/2015:10:15:59 +0300] NSMMReplicationPlugin - >>>>>> agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication >>>>>> bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): >>>>>> generic failure: GSSAPI Error: Unspecified GSS failure. Minor code >>>>>> may provide more information (No Kerberos credentials available)) >>>>>> [01/Apr/2015:10:15:59 +0300] - Waiting for 4 database threads to stop >>>>>> [01/Apr/2015:10:16:00 +0300] - All database threads now stopped >>>>>> [01/Apr/2015:10:16:00 +0300] - slapd stopped. >>>>>> >>>>>> On Wed, Apr 1, 2015 at 9:56 AM, Traiano Welcome <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hi List >>>>>>> >>>>>>> I've just tried to restart my IPA services after recently adding a >>>>>>> new >>>>>>> replica (0 configuration changes on the IPA server otherwise!), but >>>>>>> ipactl fails when starting up named: >>>>>>> >>>>>>> --- >>>>>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# ipactl start >>>>>>> Starting Directory Service >>>>>>> Starting krb5kdc Service >>>>>>> Starting kadmin Service >>>>>>> Starting named Service >>>>>>> Job for named.service failed. See 'systemctl status named.service' >>>>>>> and >>>>>>> 'journalctl -xn' for details. >>>>>>> Failed to start named Service >>>>>>> Shutting down >>>>>>> Aborting ipactl >>>>>>> --- >>>>>>> >>>>>>> I then manual start named service and try again, but then smb service >>>>>>> fails: >>>>>>> >>>>>>> --- >>>>>>> [root@lolpr-xyz-mstr ~]# ipactl start >>>>>>> Existing service file detected! >>>>>>> Assuming stale, cleaning and proceeding >>>>>>> Starting Directory Service >>>>>>> Starting krb5kdc Service >>>>>>> Starting kadmin Service >>>>>>> Starting named Service >>>>>>> Starting ipa_memcached Service >>>>>>> Starting httpd Service >>>>>>> Starting pki-tomcatd Service >>>>>>> Starting smb Service >>>>>>> Job for smb.service failed. See 'systemctl status smb.service' and >>>>>>> 'journalctl -xn' for details. >>>>>>> Failed to start smb Service >>>>>>> Shutting down >>>>>>> Aborting ipactl >>>>>>> --- >>>>>>> >>>>>>> systemctl status shows the following output for smb.service: >>>>>>> >>>>>>> --- >>>>>>> [root@lolpr-xyz-mstr ~]# systemctl -l status smb.service >>>>>>> smb.service - Samba SMB Daemon >>>>>>> Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled) >>>>>>> Active: failed (Result: exit-code) since Wed 2015-04-01 >>>>>>> 09:21:10 >>>>>>> AST; 1min 14s ago >>>>>>> Process: 4662 ExecStart=/usr/sbin/smbd $SMBDOPTIONS >>>>>>> (code=exited, >>>>>>> status=1/FAILURE) >>>>>>> Main PID: 4662 (code=exited, status=1/FAILURE) >>>>>>> Status: "Starting process..." >>>>>>> CGroup: /system.slice/smb.service >>>>>>> >>>>>>> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI client >>>>>>> step >>>>>>> 1 >>>>>>> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI Error: >>>>>>> Unspecified GSS failure. Minor code may provide more information >>>>>>> (Server ldap/[email protected] not found in Kerberos database) >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01 >>>>>>> 09:21:10.211028, 0] ipa_sam.c:4440(pdb_init_ipasam) >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: Failed to get >>>>>>> base >>>>>>> DN. >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01 >>>>>>> 09:21:10.211210, 0] >>>>>>> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name) >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: pdb backend >>>>>>> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not >>>>>>> correctly >>>>>>> init (error was NT_STATUS_UNSUCCESSFUL) >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: >>>>>>> main >>>>>>> process exited, code=exited, status=1/FAILURE >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start >>>>>>> Samba SMB Daemon. >>>>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service >>>>>>> entered failed state. >>>>>>> Apr 01 09:21:12 lolpr-xyz-mstr.xyz.local systemd[1]: Stopped Samba >>>>>>> SMB >>>>>>> Daemon. >>>>>>> --- >>>>>>> >>>>>>> >>>>>>> I manually try to start the smb service as follows, but can't (Of >>>>>>> course the directory service is not up, so there's a little catch22 >>>>>>> there and this many not mean much): >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# systemctl status smb.service >>>>>>> smb.service - Samba SMB Daemon >>>>>>> Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled) >>>>>>> Active: failed (Result: exit-code) since Wed 2015-04-01 >>>>>>> 09:50:38 >>>>>>> AST; >>>>>>> 57s ago >>>>>>> Process: 8089 ExecStart=/usr/sbin/smbd $SMBDOPTIONS >>>>>>> (code=exited, >>>>>>> status=1/FAILURE) >>>>>>> Main PID: 8089 (code=exited, status=1/FAILURE) >>>>>>> Status: "Starting process..." >>>>>>> >>>>>>> Apr 01 09:50:36 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error: >>>>>>> code=-1765328228, message=Cannot contact any KDC for realm >>>>>>> 'XYZ.LOCAL' >>>>>>> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01 >>>>>>> 09:50:37.573772, 0] ipa_sam.c:4128(bind_callback_cleanup) >>>>>>> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error: >>>>>>> code=-1765328228, message=Cannot contact any KDC for realm >>>>>>> 'XYZ.LOCAL' >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01 >>>>>>> 09:50:38.574722, 0] ipa_sam.c:4440(pdb_init_ipasam) >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: Failed to get >>>>>>> base >>>>>>> DN. >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01 >>>>>>> 09:50:38.574903, 0] >>>>>>> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name) >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: pdb backend >>>>>>> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not >>>>>>> correctly >>>>>>> init (error was NT_STATUS_UNSUCCESSFUL) >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: >>>>>>> main >>>>>>> process exited, code=exited, status=1/FAILURE >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start >>>>>>> Samba SMB Daemon. >>>>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service >>>>>>> entered failed state. >>>>>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> Please could someone advise me on how to drill deeper into debugging >>>>>>> this issue to get ipactl to start ? >>>>>>> >>>>>>> NOTES: >>>>>>> >>>>>>> - This server is successfully in a Trust relationship with >>>>>>> ActiveDirectory. >>>>>>> - There are a number of replicas established which have been working >>>>>>> fine til this morning >>>>>>> - Another replica was added around the time of the failure using the >>>>>>> same steps as usual (not sure how this could be related) >>>>>>> >>>>>>> >>>>>>> Many thanks in advance, >>>>>>> Traiano >>>>>> >>>>>> >>>>> Hi Traiano, >>>>> >>>>> it seems like there is some problem with Kerberos keytab for DS >>>>> service. >>>>> >>>>> Take a look at this guide: >>>>> >>>>> http://www.freeipa.org/page/Troubleshooting#Service_does_not_start >>>>> >>>>> and check whether there is something wrong with DS keytab and that the >>>>> service principal is set up correctly. >>>>> >>>> >>>> Walking through this pedantically: >>>> >>>> Service does not start: >>>> >>>> 1) See service log of the respective service for the exact error text. >>>> For example, the Directory Server stores the log in >>>> /var/log/dirsrv/slapd-REALM-NAME/errors >>>> >>>> check >>>> >>>> 2) Make sure that the server the service is running on has a fully >>>> qualified domain name >>>> >>>> --- >>>> [root@lolpr-xyz-mstr ~]# hostname >>>> lolpr-xyz-mstr.xyz.local >>>> [root@lolpr-xyz-mstr ~]# host `hostname` >>>> lolpr-xyz-mstr.xyz.local has address 172.16.100.68 >>>> [root@lolpr-xyz-mstr ~]# host 172.16.100.68 >>>> 68.100.16.172.in-addr.arpa domain name pointer lolpr-xyz-mstr.xyz.local. >>>> [root@lolpr-xyz-mstr ~]# >>>> --- >>>> >>>> 3) See what keys are in the keytab used for authentication of the >>>> service, >>>> e.g.: >>>> # klist -kt /etc/dirsrv/ds.keytab >>>> >>>> >>>> --- >>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# klist -kt /etc/dirsrv/ds.keytab >>>> Keytab name: FILE:/etc/dirsrv/ds.keytab >>>> KVNO Timestamp Principal >>>> ---- ------------------- >>>> ------------------------------------------------------ >>>> 2 11/06/2014 13:13:06 ldap/[email protected] >>>> 2 11/06/2014 13:13:06 ldap/[email protected] >>>> 2 11/06/2014 13:13:06 ldap/[email protected] >>>> 2 11/06/2014 13:13:06 ldap/[email protected] >>>> --- >>>> >>>> 4) Make sure that the stored principals match the system FQDN system >>>> name >>>> >>>> check: >>>> >>>> --- >>>> [root@lolpr-xyz-mstr ~]# host lolpr-xyz-mstr.xyz.local >>>> lolpr-xyz-mstr.xyz.local has address 172.16.100.68 >>>> [root@lolpr-xyz-mstr ~]# >>>> --- >>>> >>>> 5) Make sure that the version of the keys (KVNO) stored in the keytab >>>> and in the FreeIPA server match: >>>> $ kvno ldap/[email protected] >>>> >>>> >>>> check ... This is unusual: >>>> >>>> --- >>>> [root@lolpr-xyz-mstr ~]# kvno ldap/[email protected] >>>> kvno: Credentials cache keyring 'persistent:0:0' not found while >>>> getting client principal name >>>> --- >>>> >>>> Now, when I look at my krb5.conf, I see the file has had a recent >>>> change ... yet, I'm sure this file was never edited: Does the >>>> krb5.conf below look correct for a standard IPA primary server?: >>>> >>>> --- >>>> [root@lolpr-xyz-mstr ~]# ls -l /etc/krb5.conf >>>> -rw-r--r-- 1 root root 811 Apr 1 11:01 /etc/krb5.conf >>>> --- >>>> >>>> >>>> --- >>>> [root@lolpr-xyz-mstr ~]# cat /etc/krb5.conf >>>> includedir /var/lib/sss/pubconf/krb5.include.d/ >>>> >>>> [logging] >>>> default = FILE:/var/log/krb5libs.log >>>> kdc = FILE:/var/log/krb5kdc.log >>>> admin_server = FILE:/var/log/kadmind.log >>>> >>>> [libdefaults] >>>> default_realm = XYZ.LOCAL >>>> dns_lookup_realm = false >>>> dns_lookup_kdc = true >>>> rdns = false >>>> ticket_lifetime = 24h >>>> forwardable = yes >>>> default_ccache_name = KEYRING:persistent:%{uid} >>>> >>>> [realms] >>>> XYZ.LOCAL = { >>>> kdc = lolpr-xyz-mstr.xyz.local:88 >>>> master_kdc = lolpr-xyz-mstr.xyz.local:88 >>>> admin_server = lolpr-xyz-mstr.xyz.local:749 >>>> default_domain = xyz.local >>>> pkinit_anchors = FILE:/etc/ipa/ca.crt >>>> auth_to_local = >>>> RULE:[1:$1@$0](^.*@WINDOM.LOCAL$)s/@WINDOM.LOCAL/@windom.local/ >>>> auth_to_local = DEFAULT >>>> } >>>> >>>> [domain_realm] >>>> .xyz.local = XYZ.LOCAL >>>> xyz.local = XYZ.LOCAL >>>> >>>> [dbmodules] >>>> XYZ.LOCAL = { >>>> db_library = ipadb.so >>>> } >>>> --- >>> >>> >>> >>> I do not see any glaring problems in this file. >>> This seems to be 4.1 bits. >> >> >> IPA 3.3 on CentOS release 7.0.1406 (Core) >> >> >>> There is definitely something wrong with the Kerberos part though. >>> And the fact that you can't access credential cache is pointing to a >>> problem. >> >> Yes. Trying to start the krb5kdc service manually: >> >> >> --- >> job for krb5kdc.service failed. See 'systemctl status krb5kdc.service' >> and 'journalctl -xn' for details. >> --- >> >> Checking the krb5kdc.service status: >> >> --- >> [root@lolpr-xyz-mstr log]# systemctl status krb5kdc.service >> krb5kdc.service - Kerberos 5 KDC >> Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled) >> Active: failed (Result: exit-code) since Wed 2015-04-01 14:42:15 AST; >> 7s ago >> Process: 3884 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid >> $KRB5KDC_ARGS (code=exited, status=1/FAILURE) >> >> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Starting Kerberos 5 >> KDC... >> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local krb5kdc[3884]: krb5kdc: >> cannot initialize realm XYZ.LOCAL - see log file for details >> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: krb5kdc.service: >> control process exited, code=exited status=1 >> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start >> Kerberos 5 KDC. >> Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Unit >> krb5kdc.service entered failed state. >> --- >> >> >> Checking the logs: >> >> --- >> [root@lolpr-xyz-mstr log]# cat krb5kdc.log >> krb5kdc: Server error - while fetching master key K/M for realm XYZ.LOCAL >> --- >> >> >> >>> Do you see any selinux denials? >> >> Selinux has been disabled for months. I see this is still so in >> selinux conf: SELINUX=disabled >> >> >> >>> If the file was touched may be it was touched by recent update or >>> installation of some other package on the system. >>> The update/install might have set wrong context on the cred cache causing >>> problems like this. >> >> I've been careful to disable all external repos on the system since >> installation, so I'm only using packages on the original installation >> iso. It's a hermetically sealed system from the package point of view: >> >> [root@lolpr-xyz-mstr yum.repos.d]# ls -l >> total 4 >> -rw-r--r--. 1 root root 133 Nov 5 19:06 CentOS-Local.repo >> [root@lolpr-xyz-mstr yum.repos.d]# >> [root@lolpr-xyz-mstr yum.repos.d]# >> [root@lolpr-xyz-mstr yum.repos.d]# cat CentOS-Local.repo >> [LocalRepo] >> name=Local Repository >> baseurl=file:///repo >> enabled=1 >> gpgcheck=1 >> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 >> [root@lolpr-xyz-mstr yum.repos.d]# >> >> >>> Anything interesting in the KDC log? >>> >> >> This looks like a clue: >> >> krb5kdc: Server error - while fetching master key K/M for realm XYZ.LOCAL >> >> ... But I'm not sure how to interpret this usefully ... > > > This means that DS has not started as master key is in DS. > Can you check the DS server logs? > > I do see this entry in the dirsrv error loog (full log is below): set_krb5_creds - Could not get initial credentials for principal [ldap/lolpr-xyz-mstr@] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) But looking at it with ktutil, I can't see what the issue is: --- [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# ls -l /etc/dirsrv/ds.keytab -rw-------. 1 dirsrv dirsrv 338 Nov 6 13:13 /etc/dirsrv/ds.keytab [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# ktutil ktutil: read_kt /etc/dirsrv/ds.keytab ktutil: list slot KVNO Principal ---- ---- --------------------------------------------------------------------- 1 2 ldap/[email protected] 2 2 ldap/[email protected] 3 2 ldap/[email protected] 4 2 ldap/[email protected] ktutil: ktutil: --- This is a freshly generated DS log (sanitized: XYZ = realm): 389-Directory/1.3.1.6 B2014.160.2139 lolpr-xyz-mstr.xyz.local:636 (/etc/dirsrv/slapd-XYZ-LOCAL) [01/Apr/2015:15:19:01 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 starting up [01/Apr/2015:15:19:01 +0300] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=xyz,dc=local [01/Apr/2015:15:19:02 +0300] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=xyz,dc=local--no CoS Templates found, which should be added before the CoS Definition. [01/Apr/2015:15:19:02 +0300] NSMMReplicationPlugin - CleanAllRUV Task: cleanAllRUV task found, resuming the cleaning of rid(6)... [01/Apr/2015:15:19:02 +0300] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 0 (Success) [01/Apr/2015:15:19:02 +0300] NSMMReplicationPlugin - agmt="cn=masterAgreement1-lolospr-xyz-slve.xyz.local-pki-tomcat" (lolospr-xyz-slve:389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) () [01/Apr/2015:15:19:02 +0300] set_krb5_creds - Could not get initial credentials for principal [ldap/lolpr-xyz-mstr@] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) [01/Apr/2015:15:19:02 +0300] set_krb5_creds - Could not get initial credentials for principal [ldap/lolpr-xyz-mstr@] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) [01/Apr/2015:15:19:02 +0300] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=xyz,dc=local--no CoS Templates found, which should be added before the CoS Definition. [01/Apr/2015:15:19:02 +0300] set_krb5_creds - Could not get initial credentials for principal [ldap/lolpr-xyz-mstr@] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) [01/Apr/2015:15:19:02 +0300] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 2 (No such file or directory) [01/Apr/2015:15:19:02 +0300] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [01/Apr/2015:15:19:02 +0300] NSMMReplicationPlugin - agmt="cn=meTololard-xyz-slve.xyz.local" (lolard-xyz-slve:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [01/Apr/2015:15:19:02 +0300] set_krb5_creds - Could not get initial credentials for principal [ldap/lolpr-xyz-mstr@] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) [01/Apr/2015:15:19:02 +0300] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 0 (Success) [01/Apr/2015:15:19:02 +0300] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server) [01/Apr/2015:15:19:02 +0300] NSMMReplicationPlugin - agmt="cn=meTololospr-xyz-slve.xyz.local" (lolospr-xyz-slve:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () [01/Apr/2015:15:19:02 +0300] - slapd started. Listening on All Interfaces port 389 for LDAP requests [01/Apr/2015:15:19:02 +0300] - Listening on All Interfaces port 636 for LDAPS requests [01/Apr/2015:15:19:02 +0300] - Listening on /var/run/slapd-XYZ-LOCAL.socket for LDAPI requests [01/Apr/2015:15:19:02 +0300] set_krb5_creds - Could not get initial credentials for principal [ldap/lolpr-xyz-mstr@] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) [01/Apr/2015:15:19:02 +0300] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [01/Apr/2015:15:19:02 +0300] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [01/Apr/2015:15:19:02 +0300] NSMMReplicationPlugin - agmt="cn=meTololpr-xyz-slve.xyz.local" (lolpr-xyz-slve:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [01/Apr/2015:15:19:02 +0300] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [01/Apr/2015:15:19:02 +0300] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [01/Apr/2015:15:19:02 +0300] NSMMReplicationPlugin - agmt="cn=meToukpr-xyz-slve.xyz.local" (ukpr-xyz-slve:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [01/Apr/2015:15:19:02 +0300] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [01/Apr/2015:15:19:04 +0300] - slapd shutting down - signaling operation threads [01/Apr/2015:15:19:04 +0300] - slapd shutting down - closing down internal subsystems and plugins [01/Apr/2015:15:19:05 +0300] NSMMReplicationPlugin - CleanAllRUV Task: Cleaning rid (6)... [01/Apr/2015:15:19:05 +0300] NSMMReplicationPlugin - CleanAllRUV Task: Waiting to process all the updates from the deleted replica... [01/Apr/2015:15:19:05 +0300] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to be online... [01/Apr/2015:15:19:05 +0300] NSMMReplicationPlugin - CleanAllRUV Task: Server shutting down. Process will resume at server startup [01/Apr/2015:15:19:05 +0300] - Waiting for 4 database threads to stop [01/Apr/2015:15:19:05 +0300] - All database threads now stopped [01/Apr/2015:15:19:05 +0300] - slapd stopped. >> >> >> >>>> 6) Make sure that there are no DNS Issues and both forward and reverse >>>> DNS records of the are OK and match the system name and the stored >>>> principal keys >>>> >>>> check. DNS works. >>>> >>>> 7) Make sure that the system time difference on the host and FreeIPA >>>> server is not greater than 5 minutes >>>> >>>> They're one and the same in this case. >>>> >>>>> -- >>>>> Martin^3 Babinsky >>>> >>>> Thanks, >>>> Traiano >>>> >>> >>> -- >>> Thank you, >>> Dmitri Pal >>> >>> Sr. Engineering Manager IdM portfolio >>> Red Hat, Inc. >>> >>> -- >>> Manage your subscription for the Freeipa-users mailing list: >>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> Go to http://freeipa.org for more info on the project > > > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager IdM portfolio > Red Hat, Inc. > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
