Sure, just let me know what needs to be run/applied. I've already rolled back to LDAP, so if the fix looks like it works I can then roll it out again.
Thanks, _____________________________________________________ John Moyer Director, IT Operations On Sep 4, 2013, at 9:12 AM, Dmitri Pal <[email protected]> wrote: > On 09/04/2013 08:53 AM, John Moyer wrote: >> >> That summary is correct. The only thing I would add is that other >> applications could easily bring the IPA server to it's knees as well. > > Yes this is what I meant. It is not only JIRA. Any client that creates a lot > of connections can cause problems. > >> Our artifact server also did many connections per sec when used, and one >> person doing a build could bring IPA to it's knees as well. Also, not only >> would IPA be maxed at 100%, but users would complain that their builds were >> taking longer than normal (with or without the JIRA sync going, however, it >> was obviously worse when JIRA was running). >> >> Also, my IPA server was a larger/faster server than my LDAP server. So my >> LDAP server would run circles around IPA even though it was on a smaller >> machine. LDAP would run at about 10% maybe 15% CPU when the JIRA sync ran. >> >> IF you need any other information let me know. > > No this seems to be enough. > Thank you. > > Would you be willing to test a fix if one is provided? > > Thanks > Dmitri > >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> >> >> On Sep 4, 2013, at 8:32 AM, Dmitri Pal <[email protected]> wrote: >> >>> On 09/04/2013 08:01 AM, John Moyer wrote: >>>> >>>> Martin, >>>> >>>> I apologize there was a large offline conversation between Rich and >>>> myself. Rich was kind enough to help me through some of my issues. We >>>> did a lot more tests and poking and prodding. We discovered that IPA is >>>> not as efficient when dealing with large number of connections. Most of >>>> my load inefficiently reconnect to IPA over and over and over and though >>>> LDAP can deal with this fairly efficiently, IPA apparently drops to it's >>>> knees. >>>> >>>> A ticket was opened to addressed this issue. >>>> >>>> https://fedorahosted.org/freeipa/ticket/3892 >>>> >>>> >>> >>> Thank you for reporting this ticket. >>> Martin is investigating it and trying to see what is the cause. The >>> information mentioned above is missing from the the ticket, thus the >>> question. >>> >>> So to summarize: you identified that the cause of the performance issue is >>> that JIRA makes a lot of parallel connections to LDAP server and IPA is >>> slow processing bind operations thus clients that do a lot of connections >>> can experience a low performance. >>> >>> Martin, I wonder if we can have a test that would just do a lot of binds. >>> There are a lot of plugins and one of the recent ones is the OTP one. I >>> wonder if we do too much during bind when OTP is not enabled (by default). >>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> Digital Reasoning Systems, Inc. >>>> [email protected] >>>> Office: 703.678.2311 >>>> Mobile: 240.460.0023 >>>> Fax: 703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Sep 4, 2013, at 3:44 AM, Martin Kosek <[email protected]> wrote: >>>> >>>>> On 08/30/2013 11:08 PM, John Moyer wrote: >>>>>> Well IPA has machine entries on some test clusters that I'm rolling IPA >>>>>> out on (20 machines maybe) but the user base is the same (about 80 ~ 100) >>>>>> accounts with maybe 40 to 50 groups? >>>>>> >>>>>> I've stood up a clone of the jira server along with IPA. I cleared my >>>>>> logs and then did the sync and ran the log analyzer on it. These stats >>>>>> are pretty much ONLY for that jira sync I don't have any other >>>>>> connections >>>>>> pointed to it. >>>>>> >>>>>> >>>>>> Start of Log: 30/Aug/2013:15:57:13 End of Log: >>>>>> 30/Aug/2013:16:01:14 >>>>>> >>>>>> Processed Log Time: Hours, 4 Minutes, 1 Seconds >>>>>> >>>>>> Restarts: 1 Total Connections: 824 SSL >>>>>> Connections: 824 Peak Concurrent Connections: 6 Total >>>>>> Operations: 1806 Total Results: 1805 Overall >>>>>> Performance: 99.9% >>>>>> >>>>>> Searches: 968 (4.02/sec) (241.00/min) >>>>>> Modifications: 5 (0.02/sec) (1.24/min) Adds: >>>>>> 0 (0.00/sec) (0.00/min) Deletes: 0 >>>>>> (0.00/sec) (0.00/min) Mod RDNs: 0 >>>>>> (0.00/sec) >>>>>> (0.00/min) Compares: 0 (0.00/sec) >>>>>> (0.00/min) Binds: 833 (3.46/sec) >>>>>> (207.39/min) >>>>>> >>>>>> Proxied Auth Operations: 0 Persistent Searches: 1 Internal >>>>>> Operations: 0 Entry Operations: 0 Extended >>>>>> Operations: 0 Abandoned Requests: 0 Smart Referrals >>>>>> Received: 0 >>>>>> >>>>>> VLV Operations: 0 VLV Unindexed Searches: 0 SORT >>>>>> Operations: 0 >>>>>> >>>>>> Entire Search Base Queries: 0 Unindexed Searches: 1 >>>>>> >>>>> >>>>> This looks like a promising way to find out the reason, thanks John. >>>>> However, >>>>> I see just one unindexed search. Is the access log complete? Previously I >>>>> see >>>>> that the sync takes 900 seconds/15 minutes, but there is only 4 minutes >>>>> the >>>>> access log. Note that it it may take some time until the log is dumped. >>>>> >>>>> I think it would be also useful to run the analyzer with "-ula" flags as >>>>> Rob >>>>> suggested earlier to find out the unindexed searches (if any). >>>>> >>>>> What I find interesting is that JIRA does a lot of LDAP BINDs. Can the >>>>> problem be in longer BINDs than with than expected (compared to for >>>>> example >>>>> plain LDAP servers)? Performance-wise, it would be I think better if JIRA >>>>> does just one BIND and run all the LDAP searches the established >>>>> connection. >>>>> But I do not know if it can be configured this way. >>>>> >>>>> Rich, Rob, I am wondering if the slow up is not really caused by the >>>>> binds, >>>>> we have several DS plugins tied to the BIND operation, it may be useful to >>>>> analyze if they do not take too long. >>>>> >>>>> Martin >>>> >>>> >>>> >>>> _______________________________________________ >>>> Freeipa-users mailing list >>>> [email protected] >>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>> >>> >>> -- >>> Thank you, >>> Dmitri Pal >>> >>> Sr. Engineering Manager for IdM portfolio >>> Red Hat Inc. >>> >>> >>> ------------------------------- >>> Looking to carve out IT costs? >>> www.redhat.com/carveoutcosts/ >>> >>> >>> _______________________________________________ >>> Freeipa-users mailing list >>> [email protected] >>> https://www.redhat.com/mailman/listinfo/freeipa-users >> > > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager for IdM portfolio > Red Hat Inc. > > > ------------------------------- > Looking to carve out IT costs? > www.redhat.com/carveoutcosts/ > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Freeipa-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/freeipa-users
