On 27/05/16 07:37, "Howard Chu" <[email protected]> wrote:
>Jürg Bircher wrote:
>> Hello
>>
>> Improved handling for large number of databases
>> ===============================================
>
>> If interested let me know how to contribute.
>
>Looks interesting, yes. I assume you have profiled the code before and after
>the suggested changes, please provide your profiling results.
>
>Please read the Developer Guidelines.
> http://www.openldap.org/devel/contributing.html
>
> > Access newly opened database from another transaction
> > =======================================================
>
>Sounds like an oddball case. Applications should open all their DBIs from a
>single thread and not start any other threads/transactions until all setup is
>completed.
Yes it is a simpler way to open all the databases at startup. However if the
environment contains many databases which are not necessarily opened during the
life time of the application it is an advantage to open them lazily. Any
additional open database generates more overhead especially when using begin
and abort.
lmdb improved (renew, reset)
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 10] open databases in [0.31975] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [10 of 10] open databases in [0.20350] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 100] open databases in [0.25845] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [100 of 100] open databases in [0.29663] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 1000] open databases in [0.28590] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1000 of 1000] open databases in [0.42897] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 10000] open databases in [0.30004] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [10000 of 10000] open databases in [1.68870] seconds
lmdb improved (begin, commit)
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 10] open databases in [0.36538] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [10 of 10] open databases in [0.35923] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 100] open databases in [0.34858] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [100 of 100] open databases in [0.39294] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 1000] open databases in [0.40222] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1000 of 1000] open databases in [0.54752] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 10000] open databases in [0.78595] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [10000 of 10000] open databases in [2.32414] seconds
lmdb original (renew, reset)
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 10] open databases in [0.18597] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [10 of 10] open databases in [0.21572] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 100] open databases in [0.24173] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [100 of 100] open databases in [0.46497] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 1000] open databases in [0.27127] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1000 of 1000] open databases in [4.18579] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [1 of 10000] open databases in [0.30128] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database
with [10000 of 10000] open databases in [49.35048] seconds
lmdb original (begin, commit)
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 10] open databases in [0.45135] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [10 of 10] open databases in [0.38990] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 100] open databases in [0.47890] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [100 of 100] open databases in [0.69917] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 1000] open databases in [1.84908] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1000 of 1000] open databases in [5.88098] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [1 of 10000] open databases in [22.12491] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database
with [10000 of 10000] open databases in [74.53854] seconds
>
>> Hope it is useful!
>
>Thanks.
>
>--
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
