https://bugs.kde.org/show_bug.cgi?id=434926
--- Comment #11 from nyanpasu64 <nyanpas...@tuta.io> --- (In reply to tagwerk19 from comment #10) > My understanding is when something is searching it opens the index > "read-only" (and many things can open the index for reading concurrently). > When updating the index, baloo_file opens "read-write" (and just the one > thing should open the index "read write" at one time). If baloo_file sees > the index opened for reading, it appends its writes to the index. That's plausible. > Baloo_file crashes when it is wanting to update the index and sees it is (or > has been) opened read-only? I think baloo_file crashes when it asks LMDB to read existing data from disk, it's malformed, and LMDB negative-overflows an unsigned int and (I think) performs an out-of-bounds pointer access. The pointer actually points within the 256 gigabytes that baloo_file/LMDB mmaps, but past the end of the underlying file, so the thread (and process) dies with SIGBUS. ## Do other processes write to the database? (unknown) I don't know who wrote the corrupted file. Is baloo_file the only process to write to the database? I don't know, and I'll probably find out from reading compiler errors, once I try ripping LMDB out of Baloo and replacing with SQLite to evaluate the performance differences. It's probably going to be more reliable; SQLite is known for robustness, whereas LMDB and other mmap-based databases (and Baloo's usage of it) are known to corrupt easily. ## Does baloo_file open LMDB in an unsafe mode? (no) It appears that for writable databases, Baloo calls mdb_env_open() with MDB_NOSUBDIR | MDB_NOMEMINIT (https://invent.kde.org/frameworks/baloo/-/blob/master/src/engine/database.cpp#L123-131). Reading https://lmdb.readthedocs.io/en/release/#lmdb.Environment and http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5d579e71f22e9340, MDB_NOSYNC is known to cause lost or corrupted data but we do not pass it, and MDB_WRITEMAP also has some tradeoffs but we don't pass it either. So baloo_file is not operating LMDB in a mode where it's known to corrupt data. (I don't know if any other processes *are*.) ## Does baloo_file make threading errors? (unknown) Reading http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5d579e71f22e9340, if you don't pass MDB_NOTLS, each thread is only allowed to create one read transaction at a time, and you're not allowed to pass transactions between threads. Reading https://lmdb.readthedocs.io/en/release/#threads, even if you do pass it, a write transaction cannot be passed between threads, and Cursor is scoped within a transaction or something. baloo_file's main thread initializes a lazy-init static Baloo::Database through Baloo::globalDatabaseInstance() (which never gets freed, so no use-after-free), then opens the database (main -> Baloo::Database::open -> mdb_env_open()), and calls ...qt -> FileIndexScheduler::scheduleIndexing(), which creates UnindexedFileIndexer and schedules UnindexedFileIndexer::run() on a Qt worker thread. The crash happens in UnindexedFileIndexer::run(). I don't know if there's thread-unsafety going on, whether anything on the main thread invoked by Qt's event loop accesses Baloo::Database or MDB_env concurrently. Perhaps I could try wrapping UnindexedFileIndexer or Baloo::Database in a mutex, and see who accesses it. -- You are receiving this mail because: You are watching all bug changes.
