Rob or anyone else, So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up.
So just to double check what I'm thinking. I need to create three indexes
1. objectclass pres
2. objecclass eq
3. uid pres
Please let me know if I'm reading this correctly or if I'm way off?
7337 (objectclass=inetorgperson)
4597 (objectclass=*)
4560 (&(objectclass=inetorgperson)(uid=senior.developer.login))
307 (objectclass=krbticketpolicyaux)
292 (uid=*)
Thanks,
_____________________________________________________
John Moyer
Director, IT Operations
Digital Reasoning Systems, Inc.
[email protected]
Office: 703.678.2311
Mobile: 240.460.0023
Fax: 703.678.2312
www.digitalreasoning.com
On Aug 28, 2013, at 11:40 AM, Rob Crittenden <[email protected]> wrote:
> John Moyer wrote:
>> So this method of search logs is great, and it shows some indexes that would
>> likely highly increase efficiency with my usage. So, are there
>> instructions how to do that? or do you know off hand how to do that?
>
> I'd start with
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes
>
> Note that you'll want to create the same index on all hosts. This
> configuration is not replicated.
>
> You can see the ones we create in /usr/share/ipa/indices.ldif and
> /usr/share/ipa/updates/20-indices.update
>
> rob
>
>>
>>
>> Thanks,
>> _____________________________________________________
>> John Moyer
>> Director, IT Operations
>> Digital Reasoning Systems, Inc.
>> [email protected]
>> Office: 703.678.2311
>> Mobile: 240.460.0023
>> Fax: 703.678.2312
>> www.digitalreasoning.com
>>
>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden <[email protected]> wrote:
>>
>>> John Moyer wrote:
>>>> Wow, this is quite insightful, this is the output from that, it looks like
>>>> there aren't many unindexed searches (319 doesn't seem like a lot to me at
>>>> least). Do you have any suggestions from this output?
>>>
>>> There are a slew of options you can provide to logconv.pl. I typically use
>>> logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search
>>> analysis.
>>>
>>> rob
>>>
>>>>
>>>>
>>>>
>>>> Start of Log: 27/Aug/2013:02:36:08
>>>> End of Log: 27/Aug/2013:12:17:15
>>>>
>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds
>>>>
>>>> Restarts: 2
>>>> Total Connections: 45224
>>>> SSL Connections: 44735
>>>> Peak Concurrent Connections: 76
>>>> Total Operations: 132568
>>>> Total Results: 132737
>>>> Overall Performance: 100.0%
>>>>
>>>> Searches: 61318 (1.76/sec) (105.52/min)
>>>> Modifications: 277 (0.01/sec) (0.48/min)
>>>> Adds: 10 (0.00/sec) (0.02/min)
>>>> Deletes: 12 (0.00/sec) (0.02/min)
>>>> Mod RDNs: 0 (0.00/sec) (0.00/min)
>>>> Compares: 0 (0.00/sec) (0.00/min)
>>>> Binds: 62143 (1.78/sec) (106.94/min)
>>>>
>>>> Proxied Auth Operations: 0
>>>> Persistent Searches: 3
>>>> Internal Operations: 0
>>>> Entry Operations: 0
>>>> Extended Operations: 8808
>>>> Abandoned Requests: 0
>>>> Smart Referrals Received: 0
>>>>
>>>> VLV Operations: 0
>>>> VLV Unindexed Searches: 0
>>>> SORT Operations: 353
>>>>
>>>> Entire Search Base Queries: 106
>>>> Unindexed Searches: 319
>>>>
>>>> FDs Taken: 45262
>>>> FDs Returned: 45210
>>>> Highest FD Taken: 139
>>>>
>>>> Broken Pipes: 0
>>>> Connections Reset By Peer: 0
>>>> Resource Unavailable: 0
>>>>
>>>> Binds: 62143
>>>> Unbinds: 44539
>>>>
>>>> LDAP v2 Binds: 2
>>>> LDAP v3 Binds: 62141
>>>> SSL Client Binds: 0
>>>> Failed SSL Client Binds: 0
>>>> SASL Binds: 1466
>>>> 1458 GSSAPI
>>>> 8 EXTERNAL
>>>>
>>>> Directory Manager Binds: 10
>>>> Anonymous Binds: 1476
>>>> Other Binds: 60657
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> _____________________________________________________
>>>> John Moyer
>>>> Director, IT Operations
>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden <[email protected]> wrote:
>>>>
>>>>> John Moyer wrote:
>>>>>> Is there any way to see what fields are index'ed?
>>>>>
>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b
>>>>> 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config'
>>>>>
>>>>> Your best bet is to use the logconv.pl tool to examine your logs.
>>>>>
>>>>> rob
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> _____________________________________________________
>>>>>> John Moyer
>>>>>> Director, IT Operations
>>>>>> Digital Reasoning Systems, Inc.
>>>>>> [email protected]
>>>>>> Office: 703.678.2311
>>>>>> Mobile: 240.460.0023
>>>>>> Fax: 703.678.2312
>>>>>> www.digitalreasoning.com
>>>>>>
>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer
>>>>>> <[email protected]> wrote:
>>>>>>
>>>>>>> That looks like the output I just got shown below:
>>>>>>>
>>>>>>>
>>>>>>> dn: cn=mapping tree,cn=config
>>>>>>>
>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
>>>>>>>
>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
>>>>>>>
>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\
>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config
>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
>>>>>>> idnssoaserial
>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
>>>>>>> krblasts
>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> _____________________________________________________
>>>>>>> John Moyer
>>>>>>> Director, IT Operations
>>>>>>>
>>>>>>>
>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> John Moyer wrote:
>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a
>>>>>>>>> server that authenticates heavily the IPA came to it's knees again.
>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot
>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help
>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The
>>>>>>>>> user is logging in very quickly and each time it logs in I can see in
>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is
>>>>>>>>> called) is updated over and over and over in the changelog
>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY
>>>>>>>>> quickly and then disappear fairly quickly as well.
>>>>>>>>>
>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows
>>>>>>>>> everything down wait times were around 25%+
>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus
>>>>>>>>> adding to the mess
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My question is, why does the IPA server fail to keep up with the load
>>>>>>>>> when the openLDAP server didn't have an issue. Indexes?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm running the following:
>>>>>>>>>
>>>>>>>>> CentOS release 6.4 (Final)
>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64
>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64
>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64
>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64
>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64
>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64
>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64
>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64
>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So I've implemented this server anyway (against my better judgement
>>>>>>>>> with
>>>>>>>>> these issues and just made the user that logs into mercurial a local
>>>>>>>>> user instead of IPA).
>>>>>>>>>
>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put
>>>>>>>>> the
>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts
>>>>>>>>> where
>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to
>>>>>>>>> 100%
>>>>>>>>> trying to keep up with the load. I have also killed the replication
>>>>>>>>> as
>>>>>>>>> well.
>>>>>>>>>
>>>>>>>>> Any help would be appreciated.
>>>>>>>>>
>>>>>>>>
>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I
>>>>>>>> guess that doesn't prevent it from ending up in the changelog.
>>>>>>>>
>>>>>>>> You can confirm that they are excluded by searching the agreements:
>>>>>>>>
>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory
>>>>>>>> manager' -W nsDS5ReplicatedAttributeList
>>>>>>>> nsDS5ReplicatedAttributeListTotal
>>>>>>>>
>>>>>>>> They should look like:
>>>>>>>>
>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
>>>>>>>> idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
>>>>>>>> krbloginfailedcount
>>>>>>>>
>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
>>>>>>>> krblastsuccessfulauth krblastfailedauth krbloginfailedcount
>>>>>>>>
>>>>>>>> rob
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Freeipa-users mailing list [email protected] https://www.redhat.com/mailman/listinfo/freeipa-users
