Hi, as far as my scenario goes, I shortly afterward detected serious errors in my setup that were responsible to the buildup. There is an article on the symas website (https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/) that adresses the question/issue of database growth, I assume you have read that. Crashes sould of course not happen, but what exactly do you mean by "crash" anyway?
regards, Christian > salesman1 IIS Advertising <[email protected]> hat am 26. Juni 2016 um > 00:59 geschrieben: > > > Re: I HAVE THE SAME PROBLEMS, TOO ! > > > from your comments > > http://www.openldap.org/lists/openldap-technical/201511/msg00201.html > > > > I noticed the following using lmdb. Tested version is up-to-date 0.9.18, from > 2016. > > > Database grows much larger than expected in many use cases, and eventually > crashes, also was > > not a well-behaved database when simple, missing or misbehaved client code > executes it. > > . > > Do you think the same ? > > > Maybe lbmd is was fine-tuned for other especific needs, or is a toy database, > or a complicated > > case of application in our scenario. Unfortunatelly is not like Redis yet. > > We can not throw anything special on it without some effort. > > > > We really want do use lbmd, it's fast, but can't get it to work as expected. > Lmbd may be incomplete, > > buggy for (the common coder) or || (the experienced coder) who tries do use > it for other needs > > than designed. So it's now a dream but not easily realistic. Needs an expert. > > > > > 1. Database size grows 3x the data we put in, than stops growing - when > records (key+data) from 410 to 4010 bytes are added. > > -> and consumed much more space than expected for 5000*400 bytes = 2.000.000 > bytes > > > 2. Second, when we put key+data > 4096 bytes, database grows much larger and > crashes a lot of times, with a number > > of page overflows shown on 'mdb_dump ./testdb' > > > We made keys ("k:%d", i%5000) for (i=0; i < 1000000; i++) // so there are > 5000 keys, else is overwriting; > > Data is random ascii or fixed memset( 'x',400 ) > > > Program name: sample2 > > We are overwriting old values AND NOT appending, but it grows anyway. Larger > data than 4096 bytes per key-value > > seems impossible or non-sense at this time. Are we using wrong flags or > program structure ? > > > > trials - first run goes fine: #./sample2 > > > [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l > Start > Init done > written 100000 pairs of len=400 bytes > -rw-r--r-- 1 root root 2584576 Jun 25 21:43 ./testdb/data.mdb > -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb > [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l > Start > Init done > written 100000 pairs of len=400 bytes > -rw-r--r-- 1 root root 5173248 Jun 25 21:43 ./testdb/data.mdb > -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb > [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l > Start > Init done > written 100000 pairs of len=400 bytes > -rw-r--r-- 1 root root 7761920 Jun 25 21:43 ./testdb/data.mdb > -rw-r--r-- 1 root root 8192 Jun 25 21:43 ./testdb/lock.mdb > [root@ip-172-31-13-74 liblmdb]# > > > > Looks like it's not replacing old keys and have a page/data size problem, > > > Any suggestions ? > > > REGARDS !! > > Fabio Martinez - Buenos Aires and São Paulo > > gcc developer > > Brazil > > > -- > > > > > Hello, > > I am trying to use LMDB to store large (huge) amounts of binary data which, > for the reason of limiting memory footprint, are split into chunks. Each chunk > ist stored under a separate key, made up of [collectionId, chunkId], so that I > can later iterate the chunks using a LMDB cursor. Chunk size is configurable. > > During my tests, I encountered a strange scenario where, after inserting some > 2000 chunks consisting of 512KB each, the database size had grown to a value > that was roughly 135 times the calculated size of the data. I ran stat over > the db and saw that there were > 12000 overflow pages vs. approx. 2000 data > pages. When I reduced the chunk size to 4060 bytes, the number of overflow > pages went down to 1000, and the database size went down to the expected > number (I experimented with different sizes, this was the best result). I did > not find any documentation that would explain this behavior, or how to deal > with it. Of course it makes me worry about database bloat and the > consequences. Can anyone shed light on this? > > thanks, > Christian >
