Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output?
Start of Log: 27/Aug/2013:02:36:08 End of Log: 27/Aug/2013:12:17:15 Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds Restarts: 2 Total Connections: 45224 SSL Connections: 44735 Peak Concurrent Connections: 76 Total Operations: 132568 Total Results: 132737 Overall Performance: 100.0% Searches: 61318 (1.76/sec) (105.52/min) Modifications: 277 (0.01/sec) (0.48/min) Adds: 10 (0.00/sec) (0.02/min) Deletes: 12 (0.00/sec) (0.02/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds: 62143 (1.78/sec) (106.94/min) Proxied Auth Operations: 0 Persistent Searches: 3 Internal Operations: 0 Entry Operations: 0 Extended Operations: 8808 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 353 Entire Search Base Queries: 106 Unindexed Searches: 319 FDs Taken: 45262 FDs Returned: 45210 Highest FD Taken: 139 Broken Pipes: 0 Connections Reset By Peer: 0 Resource Unavailable: 0 Binds: 62143 Unbinds: 44539 LDAP v2 Binds: 2 LDAP v3 Binds: 62141 SSL Client Binds: 0 Failed SSL Client Binds: 0 SASL Binds: 1466 1458 GSSAPI 8 EXTERNAL Directory Manager Binds: 10 Anonymous Binds: 1476 Other Binds: 60657 Thanks, _____________________________________________________ John Moyer Director, IT Operations On Aug 27, 2013, at 1:13 PM, Rob Crittenden <[email protected]> wrote: > John Moyer wrote: >> Is there any way to see what fields are index'ed? > > $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b > 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' > > Your best bet is to use the logconv.pl tool to examine your logs. > > rob > >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> Digital Reasoning Systems, Inc. >> [email protected] >> Office: 703.678.2311 >> Mobile: 240.460.0023 >> Fax: 703.678.2312 >> www.digitalreasoning.com >> >> On Aug 27, 2013, at 10:36 AM, John Moyer <[email protected]> >> wrote: >> >>> That looks like the output I just got shown below: >>> >>> >>> dn: cn=mapping tree,cn=config >>> >>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>> >>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>> >>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof >>> idnssoaserial >>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn >>> krblasts >>> uccessfulauth krblastfailedauth krbloginfailedcount >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> >>> >>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden <[email protected]> wrote: >>> >>>> John Moyer wrote: >>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>> server that authenticates heavily the IPA came to it's knees again. >>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>> more, and also know exactly what server caused it (Mercurial with help >>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>> user is logging in very quickly and each time it logs in I can see in >>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>> called) is updated over and over and over in the changelog >>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>> quickly and then disappear fairly quickly as well. >>>>> >>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>> everything down wait times were around 25%+ >>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>> adding to the mess >>>>> >>>>> >>>>> My question is, why does the IPA server fail to keep up with the load >>>>> when the openLDAP server didn't have an issue. Indexes? >>>>> >>>>> >>>>> I'm running the following: >>>>> >>>>> CentOS release 6.4 (Final) >>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>> >>>>> >>>>> So I've implemented this server anyway (against my better judgement with >>>>> these issues and just made the user that logs into mercurial a local >>>>> user instead of IPA). >>>>> >>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>> trying to keep up with the load. I have also killed the replication as >>>>> well. >>>>> >>>>> Any help would be appreciated. >>>>> >>>> >>>> krblastsuccessfulauth should be excluded from replication, though I guess >>>> that doesn't prevent it from ending up in the changelog. >>>> >>>> You can confirm that they are excluded by searching the agreements: >>>> >>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory >>>> manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>> >>>> They should look like: >>>> >>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof >>>> idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth >>>> krbloginfailedcount >>>> >>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn >>>> krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>> >>>> rob >>> >> >
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Freeipa-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/freeipa-users
