On Tue, 2012-07-31 at 21:08 +0200, Sigbjorn Lie wrote: > On 07/31/2012 01:50 PM, Simo Sorce wrote: > > On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote: > >> On Tue, July 31, 2012 10:20, Petr Spacek wrote: > >>> On 07/30/2012 10:37 PM, Sigbjorn Lie wrote: > >>> > >>>> Hi, > >>>> > >>>> > >>>> I've been having performance issues after I upgraded to RHEL 6.3 / IPA > >>>> 2.2. I > >>>> still have a LDAP server having unusual high cpu usage even after it's > >>>> been removed from the SRV > >>>> records and is serving almost no clients anymore, but it would seem as > >>>> my main issues is with > >>>> the kerberos server. > >>>> > >>>> All kerberos services are performing very slowly, and the IPA servers > >>>> has much > >>>> higher CPU load now then what they had with IPA 2.1. Some services are > >>>> timing out, like > >>>> kerberized web servers, other kerberized services perform authentication > >>>> very slowly. I had to > >>>> switch our automounter away from kerberos authentication as it is no > >>>> longer usable. > >>>> > >>>> Using SSH to log on to SSSD enabled hosts are also very slow, a login > >>>> takes > >>>> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA > >>>> 2.2. > >>>> > >>>> The IPA web admin interface is definitely not faster than in IPA 2.1. > >>>> > >>>> > >>>> For a comparison, listing out all the folders in an automount map, > >>>> causing > >>>> them to be looked up from LDAP and mounted takes over 5 minutes with IPA > >>>> 2.2 when using kerberos > >>>> authentication for the automounter. There are approx 130 folders in that > >>>> automount map. > >>>> > >>>> After unmounting all the mounted folders, and changing to using a > >>>> username and > >>>> password authentication with a TLS connection, attempting the same > >>>> operating again, and it now > >>>> finishes in about 14 seconds for both the lookup from LDAP and the mount > >>>> operation. > >>>> > >>>> After unmounting all the mounted folders again, changing to username and > >>>> password authentication with a simple unencrypted bind, and then > >>>> attempting the same operation > >>>> and it now finishes both lookup and mount in just over 5 seconds! > >>>> > >>>> I don't have any timing for kerberized automount pre IPA-2.2, but we > >>>> we're not > >>>> talking about several minutes to mount all the folders in this automount > >>>> map. Unfortunately > >>>> mounting all the folders is what happens when the users use konqueror to > >>>> browse the automount > >>>> maps, so this is a very noticable issue. > >>>> > >>>> Even loading a new gnome-terminal or konsole terminal which causes an > >>>> automount folder to be mounted takes anything between 5 - 15 seconds > >>>> after the upgrade. There > >>>> we're no notiable delay when opening a new terminal window pre IPA-2.2. > >>>> > >>>> > >>>> I am not using SSSD for the automounter. > >>>> > >>>> > >>>> I do notice that the dbmodule for the kerberos server has changed from > >>>> "kldap" > >>>> to "ipadb.so" Perhaps there is some issues with the new library? > >>>> > >>>> > >>>> > >>>> > >>>> Regards, > >>>> Siggi > >>>> > >>> > >>> Hello, > >>> > >>> > >>> I'm not a Kerberos guy, so I can give only general advice: > >>> "Overloaded-CPU-problems" can be troubleshooted with OProfile. > >>> > >>> > >>> Oprofile is lightweight statistic profiler (AFAIK it was designed for > >>> production environment). > >>> > >>> Step-by-step documentation for RHEL 6 is available from: > >>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht > >>> ml#ch-OProfile > >>> > >>> As you can see in section 22.5.1., it allows to break whole CPU usage > >>> between > >>> processes, libraries and even individual symbols (if proper debuginfos > >>> are installed). > >>> > >>> I recommend to run OProfile on problematic system - results from opreport > >>> can > >>> provide missing clue to us. > >>> > >>> OProfile gives best results on bare-metal machines. On virtual machines > >>> you > >>> has to use timer mode in place of hardware performance counters, please > >>> see the documentation. > >>> > >>> > >>> Short getting started guide: > >>> http://oprofile.sourceforge.net/doc/overview.html#getting-started > >>> > >>> > >>> Nice article with theory&& examples: > >>> http://people.redhat.com/wcohen/Oprofile.pdf > >>> > >>> > >>> Homepage with a lot of useful information: > >>> http://oprofile.sourceforge.net/ > >>> > >>> > >>> > >> Thank you. > >> > >> All 3 IPA servers are close to idle now after switching from kerberos to > >> user/pwd bind for the > >> Linux automounter. > >> > >> Still there is an issue with kerberos failing to issue a ticket every now > >> and then and it's > >> responding very slowly. > >> > >> There seem to be low activity on this list just now. Is the kerberos > >> people away on vacation? > > Hi Siggi, > > some people are on vacation, some are busy covering others :-) > > > > Would you be able to take a wireshark trace of an automount going on ? > > I would like to see precise timing of packets on the wire to make a > > first assesment of where is the bottleneck. > > > > We did change from ldap.so to ipadb.so, but the structure of the drivers > > is not much different, so I am surprised it would be much slower, > > however it is possible, I would like to find out what is going on with > > your help. > > > > OK, I will get that done when I'm back in the office tomorrow. I suspect > it will be somewhat better than my first results as the load on the IPA > servers are now much lower when the linux automounters are no longer > using kerberos for authentication. > > It seem like there is a race condition going on as the shit didn't hit > the fan until the week after the upgrade to IPA 2.2 when people returned > to work. The slowness issues then gradually became worse and worse. > > I will send you the captures in a private email. Do you need anything > besides TCP 389, 636 and TCP/UDP 88 ?
no need for TCP 636, but it may be intresting to see DNS queries, do you use the IPA integrated DNS or do you use your own infra ? Simo. -- Simo Sorce * Red Hat, Inc * New York _______________________________________________ Freeipa-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/freeipa-users
