On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote: > On Tue, July 31, 2012 10:20, Petr Spacek wrote: > > On 07/30/2012 10:37 PM, Sigbjorn Lie wrote: > > > >> Hi, > >> > >> > >> I've been having performance issues after I upgraded to RHEL 6.3 / IPA > >> 2.2. I > >> still have a LDAP server having unusual high cpu usage even after it's > >> been removed from the SRV > >> records and is serving almost no clients anymore, but it would seem as my > >> main issues is with > >> the kerberos server. > >> > >> All kerberos services are performing very slowly, and the IPA servers has > >> much > >> higher CPU load now then what they had with IPA 2.1. Some services are > >> timing out, like > >> kerberized web servers, other kerberized services perform authentication > >> very slowly. I had to > >> switch our automounter away from kerberos authentication as it is no > >> longer usable. > >> > >> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes > >> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA > >> 2.2. > >> > >> The IPA web admin interface is definitely not faster than in IPA 2.1. > >> > >> > >> For a comparison, listing out all the folders in an automount map, causing > >> them to be looked up from LDAP and mounted takes over 5 minutes with IPA > >> 2.2 when using kerberos > >> authentication for the automounter. There are approx 130 folders in that > >> automount map. > >> > >> After unmounting all the mounted folders, and changing to using a username > >> and > >> password authentication with a TLS connection, attempting the same > >> operating again, and it now > >> finishes in about 14 seconds for both the lookup from LDAP and the mount > >> operation. > >> > >> After unmounting all the mounted folders again, changing to username and > >> password authentication with a simple unencrypted bind, and then > >> attempting the same operation > >> and it now finishes both lookup and mount in just over 5 seconds! > >> > >> I don't have any timing for kerberized automount pre IPA-2.2, but we we're > >> not > >> talking about several minutes to mount all the folders in this automount > >> map. Unfortunately > >> mounting all the folders is what happens when the users use konqueror to > >> browse the automount > >> maps, so this is a very noticable issue. > >> > >> Even loading a new gnome-terminal or konsole terminal which causes an > >> automount folder to be mounted takes anything between 5 - 15 seconds after > >> the upgrade. There > >> we're no notiable delay when opening a new terminal window pre IPA-2.2. > >> > >> > >> I am not using SSSD for the automounter. > >> > >> > >> I do notice that the dbmodule for the kerberos server has changed from > >> "kldap" > >> to "ipadb.so" Perhaps there is some issues with the new library? > >> > >> > >> > >> > >> Regards, > >> Siggi > >> > > > > > > Hello, > > > > > > I'm not a Kerberos guy, so I can give only general advice: > > "Overloaded-CPU-problems" can be troubleshooted with OProfile. > > > > > > Oprofile is lightweight statistic profiler (AFAIK it was designed for > > production environment). > > > > Step-by-step documentation for RHEL 6 is available from: > > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht > > ml#ch-OProfile > > > > As you can see in section 22.5.1., it allows to break whole CPU usage > > between > > processes, libraries and even individual symbols (if proper debuginfos are > > installed). > > > > I recommend to run OProfile on problematic system - results from opreport > > can > > provide missing clue to us. > > > > OProfile gives best results on bare-metal machines. On virtual machines you > > has to use timer mode in place of hardware performance counters, please see > > the documentation. > > > > > > Short getting started guide: > > http://oprofile.sourceforge.net/doc/overview.html#getting-started > > > > > > Nice article with theory && examples: > > http://people.redhat.com/wcohen/Oprofile.pdf > > > > > > Homepage with a lot of useful information: > > http://oprofile.sourceforge.net/ > > > > > > > > Thank you. > > All 3 IPA servers are close to idle now after switching from kerberos to > user/pwd bind for the > Linux automounter. > > Still there is an issue with kerberos failing to issue a ticket every now and > then and it's > responding very slowly. > > There seem to be low activity on this list just now. Is the kerberos people > away on vacation?
Hi Siggi, some people are on vacation, some are busy covering others :-) Would you be able to take a wireshark trace of an automount going on ? I would like to see precise timing of packets on the wire to make a first assesment of where is the bottleneck. We did change from ldap.so to ipadb.so, but the structure of the drivers is not much different, so I am surprised it would be much slower, however it is possible, I would like to find out what is going on with your help. Simo. -- Simo Sorce * Red Hat, Inc * New York _______________________________________________ Freeipa-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/freeipa-users
