mark jayson wrote: > Hello, > > I've created a program that writes data to an lmdb database using python > bindings. I've written about 300gb worth of data but when I was re-reading > the database > by running my script again, it crashes immediately. Did a backtrace in gdb > and this is what I got: > > Really have no clue what's going on here. I suspect that the server has > rebooted (or was rebooted by someone while I was running the script but I > thought lmdb > is crash-proof.
It is. Extensively tested so, but storage devices can still lie about whether they successfully wrote data. > Do you have any idea how to best troubleshoot this? The database is worth > 300gb+ so it's not easy to share it, not to mention it contains sensitive > information. Try using mdb_dump to see if it will backup the contents without crashing. Also, when asking for help you should specify exactly what version of liblmdb you used. The backtrace is a bit odd because normally just opening an environment shouldn't require committing any write txns. > > Starting program: /usr/bin/python3 read_events.py -d jul012022-23/ > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > build/lib/mdb.c:3274: Assertion 'len >= 0 && id <= env->me_pglast' failed in > mdb_freelist_save() > > Program received signal SIGABRT, Aborted. > __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350283264) > at ./nptl/pthread_kill.c:44 > 44./nptl/pthread_kill.c: No such file or directory. > (gdb) backtrace > #0 __pthread_kill_implementation (no_tid=0, signo=6, > threadid=140737350283264) at ./nptl/pthread_kill.c:44 > #1 __pthread_kill_internal (signo=6, threadid=140737350283264) at > ./nptl/pthread_kill.c:78 > #2 __GI___pthread_kill (threadid=140737350283264, signo=signo@entry=6) at > ./nptl/pthread_kill.c:89 > #3 0x00007ffff7c96476 in __GI_raise (sig=sig@entry=6) at > ../sysdeps/posix/raise.c:26 > #4 0x00007ffff7c7c7f3 in __GI_abort () at ./stdlib/abort.c:79 > #5 0x00007ffff62fb3e2 in mdb_assert_fail (env=0x5555565b49f0, > expr_txt=expr_txt@entry=0x7ffff62fe308 "len >= 0 && id <= env->me_pglast", > func=func@entry=0x7ffff62fe930 <__func__.13> "mdb_freelist_save", > line=line@entry=3274, file=0x7ffff62fe010 "build/lib/mdb.c") at > build/lib/mdb.c:1545 > #6 0x00007ffff62f10df in mdb_freelist_save (txn=0x555556752950) at > build/lib/mdb.c:3274 > #7 mdb_txn_commit (txn=0x555556752950) at build/lib/mdb.c:3646 > #8 0x00007ffff62f369b in txn_db_from_name (env=env@entry=0x7ffff61c8090, > name=<optimized out>, flags=262144) at lmdb/cpython.c:1017 > #9 0x00007ffff62f6e1d in env_open_db (self=0x7ffff61c8090, args=<optimized > out>, kwds=<optimized out>) at lmdb/cpython.c:1665 > #10 0x00005555556b23a9 in ?? () > #11 0x0000555555699c14 in _PyEval_EvalFrameDefault () > #12 0x0000555555696176 in ?? () > #13 0x000055555578bc56 in PyEval_EvalCode () > #14 0x00005555557b8b18 in ?? () > #15 0x00005555557b196b in ?? () > #16 0x00005555557b8865 in ?? () > #17 0x00005555557b7d48 in _PyRun_SimpleFileObject () > #18 0x00005555557b7a43 in _PyRun_AnyFileObject () > #19 0x00005555557a8c3e in Py_RunMain () > #20 0x000055555577ebcd in Py_BytesMain () > #21 0x00007ffff7c7dd90 in __libc_start_call_main > (main=main@entry=0x55555577eb90, argc=argc@entry=14, > argv=argv@entry=0x7fffffffe378) at > ../sysdeps/nptl/libc_start_call_main.h:58 > #22 0x00007ffff7c7de40 in __libc_start_main_impl (main=0x55555577eb90, > argc=14, argv=0x7fffffffe378, init=<optimized out>, fini=<optimized out>, > rtld_fini=<optimized out>, stack_end=0x7fffffffe368) > at ../csu/libc-start.c:392 > #23 0x000055555577eac5 in _start () > (gdb) client_loop: send disconnect: Broken pipe > > > For what it's worth, this is how I've opened the database environment: > > dbenv = lmdb.open(db_dir, map_size=1099511627776, max_dbs=11, > readahead=False) > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
