For the record,
On Mon 27 Oct 2025 22:51:06 +0100, Chris Hofstädtler wrote:
> Source: apt
> Version: 3.1.11
> Severity: important
> Control: block 1117120 by -1
>
> Your package depends or otherwise uses db5.3 ("Berkeley DB").
> The upstream of db5.3 has ceased maintaining it, and it is also orphaned in
> Debian.
> See #987013 for discussion.
> Please update your package to remove any uses of db5.3 for forky.
> If necessary, provide an upgrade path in forky and remove usage in duke.
I am a nosy bystander (who uses apt-ftparchive).
AFAICT the only part of apt using bdb is apt-ftparchive/cachedb.{h,cc} (in the
apt-utils .deb).
i.e. people just doing apt update/install/full-upgrade are NOT IMPACTED.
apt-ftparchive reads pool/ and generates dists/.
apt-ftparchive can use a bdb cache to avoid re-processing unchanged files in
pool/.
apt-ftparchive can run without a cache -- it's just annoyingly slow (3 seconds
vs 351 seconds for an 8GB 3Kinode pool/).
In theory we could just comment out the db references in debian/control,
CMake*, and apt-ftparchive/cache*, and most users will not even notice.
(I would, but I am weird.)
If apt-ftparchive is patched to use e.g. liblmdb0 instead of libdb5.3,
we don't need to migrate the data in the cache,
we can just say "delete it and let it be re-created".
(This already happens in ftparchive/cachedb.cc for HASH->BTREE transition
within bdb.)
I didn't find a general guid "how to port your C app from bdb to lmdb".
All the search results were Fedora 389 DS users migrating their LDAP database
from bdb to lmdb.
(lmdb is originally from OpenLDAP.)
I had a quick crack at porting cachedb.cc from bdb to lmdb, despite not knowing
any of the components involved.
I think I got the cmake part working OK.
I didn't get much further because
file:///usr/share/doc/lmdb-doc/html/starting.html seems to mandate a longer
chain than bdb:
mdb_env_open() // path (as char*) to environment
mdb_txn_begin() // environment to transaction
mdb_dbi_open() // transaction to database (just use NULL as database name)
# no cursor mode, seen in cachedb.h:CacheDB:Put/Get
mdb_get() // transaction+database+key to value
mdb_put() // transaction+database+key+value to nothing
# with cursor mode, needed by cachedb.cc:CacheDB::Clean to remove obsolete
key/value pairs
mdb_cursor() // transaction+database to cursor
mdb_cursor_get() // cursor+key to value
mdb_cursor_put() // cursor+key+value to nothing
mdb_cursor_close() // iff a read-only cursor
mdb_dbi_close() // not actually needed
mdb_txn_commit()
mdb_env_close()
I also noticed that:
* You need a txn even for MDB_RDONLY.
* You need a dbi even if your only db name is NULL.
* MDB_val is a direct equivalent of DBT (for a key or value).
It's (usually) zero-copy, so you have to make your own copy if you
want to refer to it after closing lmdb.
I'm not good enough at C++ to tell at a glance if ftparchive/cachedb.cc is
doing so.
* I don't think apt-ftparchive is doing threading.
If it is, you have to care about thread-locality for parts of lmdb.
* There's no in-place Dbp->compact().
I think you have to mdb_env_copy2() with MDB_COMPACT flag, then
I guess manually rotate the copies using renameat2() or whatever.
* There's no stat_print().
I think you have to mdb_env_stat() then manually printf() the 6 numbers.
diff --git c/CMake/FindBerkeley.cmake i/CMake/FindBerkeley.cmake
deleted file mode 100644
index a6215d7..0000000
--- c/CMake/FindBerkeley.cmake
+++ /dev/null
@@ -1,59 +0,0 @@
-# - Try to find Berkeley DB
-# Once done this will define
-#
-# BERKELEY_FOUND - system has Berkeley DB
-# BERKELEY_INCLUDE_DIRS - the Berkeley DB include directory
-# BERKELEY_LIBRARIES - Link these to use Berkeley DB
-# BERKELEY_DEFINITIONS - Compiler switches required for using Berkeley DB
-
-# Copyright (c) 2006, Alexander Dymo, <[email protected]>
-# Copyright (c) 2016, Julian Andres Klode <[email protected]>
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions
-# are met:
-#
-# 1. Redistributions of source code must retain the copyright
-# notice, this list of conditions and the following disclaimer.
-# 2. Redistributions in binary form must reproduce the copyright
-# notice, this list of conditions and the following disclaimer in the
-# documentation and/or other materials provided with the distribution.
-# 3. The name of the author may not be used to endorse or promote products
-# derived from this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
-# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
-# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
-# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
-# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
-# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
-# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-
-# We need NO_DEFAULT_PATH here, otherwise CMake helpfully picks up the wrong
-# db.h on BSD systems instead of the Berkeley DB one.
-find_path(BERKELEY_INCLUDE_DIRS db.h
- ${CMAKE_INSTALL_FULL_INCLUDEDIR}/db5
- /usr/local/include/db5
- /usr/include/db5
-
- ${CMAKE_INSTALL_FULL_INCLUDEDIR}/db4
- /usr/local/include/db4
- /usr/include/db4
-
- ${CMAKE_INSTALL_FULL_INCLUDEDIR}
- /usr/local/include
- /usr/include
-
- NO_DEFAULT_PATH
-)
-
-find_library(BERKELEY_LIBRARIES NAMES db db-5)
-
-include(FindPackageHandleStandardArgs)
-find_package_handle_standard_args(Berkeley "Could not find Berkeley DB >= 4.1" BERKELEY_INCLUDE_DIRS BERKELEY_LIBRARIES)
-# show the BERKELEY_INCLUDE_DIRS and BERKELEY_LIBRARIES variables only in the advanced view
-mark_as_advanced(BERKELEY_INCLUDE_DIRS BERKELEY_LIBRARIES)
diff --git c/CMake/FindLMDB.cmake i/CMake/FindLMDB.cmake
new file mode 100644
index 0000000..40226fc
--- /dev/null
+++ i/CMake/FindLMDB.cmake
@@ -0,0 +1,25 @@
+# - Try to find LMDB
+# Once done, this will define
+#
+# LMDB_FOUND - system has LMDB
+# LMDB_INCLUDE_DIRS - the LMDB include directories
+# LMDB_LIBRARIES - the LMDB library
+find_package(PkgConfig)
+
+pkg_check_modules(LMDB_PKGCONF lmdb)
+
+find_path(LMDB_INCLUDE_DIRS
+ NAMES lmdb.h
+ PATHS ${LMDB_PKGCONF_INCLUDE_DIRS}
+)
+
+
+find_library(LMDB_LIBRARIES
+ NAMES lmdb
+ PATHS ${LMDB_PKGCONF_LIBRARY_DIRS}
+)
+
+include(FindPackageHandleStandardArgs)
+find_package_handle_standard_args(LMDB DEFAULT_MSG LMDB_INCLUDE_DIRS LMDB_LIBRARIES)
+
+mark_as_advanced(LMDB_INCLUDE_DIRS LMDB_LIBRARIES)
diff --git c/CMakeLists.txt i/CMakeLists.txt
index f979962..6f70c29 100644
--- c/CMakeLists.txt
+++ i/CMakeLists.txt
@@ -88,9 +88,9 @@ add_optional_compile_options(Werror=suggest-override)
add_optional_compile_options(Werror=return-type)
add_optional_compile_options(Wp,-D_GLIBCXX_ASSERTIONS)
# apt-ftparchive dependencies
-find_package(Berkeley REQUIRED)
-if (BERKELEY_FOUND)
- set(HAVE_BDB 1)
+find_package(LMDB REQUIRED)
+if (LMDB_FOUND)
+ set(HAVE_LMDB 1)
endif()
find_package(OpenSSL REQUIRED)
diff --git c/COPYING i/COPYING
index 0f0e3f6..3584863 100644
--- c/COPYING
+++ i/COPYING
@@ -53,33 +53,6 @@ Files: methods/connect.c
Copyright: Copyright (c) 1996 - 2023, Daniel Stenberg, <[email protected]>, and many contributors
License: GPL-2+ and curl
-Files: CMake/FindBerkeley.cmake
-Copyright: 2006, Alexander Dymo, <[email protected]>
- 2016, Julian Andres Klode <[email protected]>
-License: BSD-3-clause
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
- .
- 1. Redistributions of source code must retain the copyright
- notice, this list of conditions and the following disclaimer.
- 2. Redistributions in binary form must reproduce the copyright
- notice, this list of conditions and the following disclaimer in the
- documentation and/or other materials provided with the distribution.
- 3. The name of the author may not be used to endorse or promote products
- derived from this software without specific prior written permission.
- .
- THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
- IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
- IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
- INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
- NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
- THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
Files: CMake/Documentation.cmake
CMake/FindLFS.cmake
Copyright: 2016 Julian Andres Klode <[email protected]>
diff --git c/debian/control i/debian/control
index a3e4c45..d0686cb 100644
--- c/debian/control
+++ i/debian/control
@@ -16,7 +16,7 @@ Build-Depends: dpkg-dev (>= 1.22.5) <!pkg.apt.ci>,
gettext (>= 0.12),
googletest <!nocheck> | libgtest-dev <!nocheck>,
libbz2-dev,
- libdb-dev,
+ libmdb-dev,
libssl-dev,
liblz4-dev (>= 0.0~r126),
liblzma-dev,
diff --git c/ftparchive/CMakeLists.txt i/ftparchive/CMakeLists.txt
index 4af2504..53753a8 100644
--- c/ftparchive/CMakeLists.txt
+++ i/ftparchive/CMakeLists.txt
@@ -1,4 +1,4 @@
-include_directories(${BERKELEY_INCLUDE_DIRS})
+include_directories(${LMDB_INCLUDE_DIRS})
# Definition of the C++ files used to build the program - note that this
# is expanded at CMake time, so you have to rerun cmake if you add or remove
# a file (you can just run cmake . in the build directory)
@@ -6,8 +6,8 @@ file(GLOB_RECURSE source "*.cc")
add_executable(apt-ftparchive ${source})
# Link the executables against the libraries
-target_include_directories(apt-ftparchive PRIVATE ${BERKELEY_INCLUDE_DIRS})
-target_link_libraries(apt-ftparchive apt-pkg apt-private ${BERKELEY_LIBRARIES})
+target_include_directories(apt-ftparchive PRIVATE ${LMDB_INCLUDE_DIRS})
+target_link_libraries(apt-ftparchive apt-pkg apt-private ${LMDB_LIBRARIES})
# Install the executables
install(TARGETS apt-ftparchive RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
diff --git c/ftparchive/cachedb.cc i/ftparchive/cachedb.cc
index 7ad2873..f43de30 100644
--- c/ftparchive/cachedb.cc
+++ i/ftparchive/cachedb.cc
@@ -55,7 +55,7 @@ bool CacheDB::ReadyDB(std::string const &DB)
// Close the old DB
if (Dbp != 0)
- Dbp->close(Dbp,0);
+ mdb_env_close(Dbp,0);
/* Check if the DB was disabled while running and deal with a
corrupted DB */
@@ -72,30 +72,23 @@ bool CacheDB::ReadyDB(std::string const &DB)
if (DB.empty())
return true;
- db_create(&Dbp, NULL, 0);
- if ((err = Dbp->open(Dbp, NULL, DB.c_str(), NULL, DB_BTREE,
- (ReadOnly?DB_RDONLY:DB_CREATE),
- 0644)) != 0)
+ err = mdb_env_create(&Dbp);
+ if (err)
+ {
+ Dbp = 0;
+ return _error->Error(_("Unable to open DB file %s: %s"),DB.c_str(), mdb_strerror(err));
+ }
+ if ((err = mdb_env_open(Dbp, DB.c_str(),
+ (ReadOnly?MDB_RDONLY:NULL))) != 0)
{
- if (err == DB_OLD_VERSION)
- {
- _error->Warning(_("DB is old, attempting to upgrade %s"),DBFile.c_str());
- err = Dbp->upgrade(Dbp, DB.c_str(), 0);
- if (!err)
- err = Dbp->open(Dbp, NULL, DB.c_str(), NULL, DB_HASH,
- (ReadOnly?DB_RDONLY:DB_CREATE), 0644);
-
- }
- // the database format has changed from DB_HASH to DB_BTREE in
- // apt 0.6.44
- if (err == EINVAL)
+ if (err == EINVAL) // FIXME: probably the wrong errno now
{
_error->Error(_("DB format is invalid. If you upgraded from an older version of apt, please remove and re-create the database."));
}
if (err)
{
Dbp = 0;
- return _error->Error(_("Unable to open DB file %s: %s"),DB.c_str(), db_strerror(err));
+ return _error->Error(_("Unable to open DB file %s: %s"),DB.c_str(), mdb_strerror(err));
}
}
@@ -193,7 +186,7 @@ bool CacheDB::GetFileStat(bool const &doStat)
bool CacheDB::GetCurStatCompatOldFormat()
{
InitQueryStats();
- Data.data = &CurStatOldFormat;
+ Data.mv_data = &CurStatOldFormat;
Data.flags = DB_DBT_USERMEM;
Data.ulen = sizeof(CurStatOldFormat);
if (Get() == false)
@@ -216,7 +209,7 @@ bool CacheDB::GetCurStatCompatOldFormat()
bool CacheDB::GetCurStatCompatNewFormat()
{
InitQueryStats();
- Data.data = &CurStat;
+ Data.mv_data = &CurStat;
Data.flags = DB_DBT_USERMEM;
Data.ulen = sizeof(CurStat);
if (Get() == false)
@@ -238,25 +231,25 @@ bool CacheDB::GetCurStat()
{
// do a first query to just get the size of the data on disk
InitQueryStats();
- Data.data = &CurStat;
+ Data.mv_data = &CurStat;
Data.flags = DB_DBT_USERMEM;
Data.ulen = 0;
Get();
- if (Data.size == 0)
+ if (Data.mv_size == 0)
{
// nothing needs to be done, we just have not data for this deb
}
// check if the record is written in the old format (32bit filesize)
- else if(Data.size == sizeof(CurStatOldFormat))
+ else if(Data.mv_size == sizeof(CurStatOldFormat))
{
GetCurStatCompatOldFormat();
}
- else if(Data.size == sizeof(CurStat))
+ else if(Data.mv_size == sizeof(CurStat))
{
GetCurStatCompatNewFormat();
} else {
- return _error->Error("Cache record size mismatch (%ul)", Data.size);
+ return _error->Error("Cache record size mismatch (%ul)", Data.mv_size);
}
CurStat.Flags = ntohl(CurStat.Flags);
@@ -306,7 +299,7 @@ bool CacheDB::LoadSource() /*{{{*/
{
// Lookup the control information
InitQuerySource();
- if (Get() == true && Dsc.TakeDsc(Data.data, Data.size) == true)
+ if (Get() == true && Dsc.TakeDsc(Data.mv_data, Data.mv_size) == true)
{
return true;
}
@@ -324,7 +317,7 @@ bool CacheDB::LoadSource() /*{{{*/
// Write back the control information
InitQuerySource();
- if (Put(Dsc.Data.c_str(), Dsc.Length) == true)
+ if (Put(Dsc.mv_data.c_str(), Dsc.Length) == true)
CurStat.Flags |= FlSource;
return true;
@@ -340,7 +333,7 @@ bool CacheDB::LoadControl()
{
// Lookup the control information
InitQueryControl();
- if (Get() == true && Control.TakeControl(Data.data,Data.size) == true)
+ if (Get() == true && Control.TakeControl(Data.mv_data,Data.mv_size) == true)
return true;
CurStat.Flags &= ~FlControl;
}
@@ -377,7 +370,7 @@ bool CacheDB::LoadContents(bool const &GenOnly)
InitQueryContent();
if (Get() == true)
{
- if (Contents.TakeContents(Data.data,Data.size) == true)
+ if (Contents.TakeContents(Data.mv_data,Data.mv_size) == true)
return true;
}
@@ -393,7 +386,7 @@ bool CacheDB::LoadContents(bool const &GenOnly)
// Write back the control information
InitQueryContent();
- if (Put(Contents.Data,Contents.CurSize) == true)
+ if (Put(Contents.mv_data,Contents.CurSize) == true)
CurStat.Flags |= FlContents;
return true;
}
@@ -541,24 +534,24 @@ bool CacheDB::Clean()
/* I'm not sure what VERSION_MINOR should be here.. 2.4.14 certainly
needs the lower one and 2.7.7 needs the upper.. */
DBC *Cursor;
- if ((errno = Dbp->cursor(Dbp, NULL, &Cursor, 0)) != 0)
+ if ((errno = mdb_cursor_open(Dbp, NULL, &Cursor, 0)) != 0)
return _error->Error(_("Unable to get a cursor"));
- DBT Key;
- DBT Data;
+ MDB_val Key;
+ MDB_val Data;
memset(&Key,0,sizeof(Key));
memset(&Data,0,sizeof(Data));
while ((errno = Cursor->c_get(Cursor,&Key,&Data,DB_NEXT)) == 0)
{
- const char *Colon = (char*)memrchr(Key.data, ':', Key.size);
+ const char *Colon = (char*)memrchr(Key.mv_data, ':', Key.mv_size);
if (Colon)
{
- if (stringcmp(Colon + 1, (char *)Key.data+Key.size,"st") == 0 ||
- stringcmp(Colon + 1, (char *)Key.data+Key.size,"cl") == 0 ||
- stringcmp(Colon + 1, (char *)Key.data+Key.size,"cs") == 0 ||
- stringcmp(Colon + 1, (char *)Key.data+Key.size,"cn") == 0)
+ if (stringcmp(Colon + 1, (char *)Key.mv_data+Key.mv_size,"st") == 0 ||
+ stringcmp(Colon + 1, (char *)Key.mv_data+Key.mv_size,"cl") == 0 ||
+ stringcmp(Colon + 1, (char *)Key.mv_data+Key.mv_size,"cs") == 0 ||
+ stringcmp(Colon + 1, (char *)Key.mv_data+Key.mv_size,"cn") == 0)
{
- std::string FileName = std::string((const char *)Key.data,Colon);
+ std::string FileName = std::string((const char *)Key.mv_data,Colon);
if (FileExists(FileName) == true) {
continue;
}
@@ -566,12 +559,17 @@ bool CacheDB::Clean()
}
Cursor->c_del(Cursor,0);
}
- int res = Dbp->compact(Dbp, NULL, NULL, NULL, NULL, DB_FREE_SPACE, NULL);
- if (res < 0)
- _error->Warning("compact failed with result %i", res);
+ // FIXME: mdb_env_copy2() with MDB_COMPACT flag?
+ // int res = Dbp->compact(Dbp, NULL, NULL, NULL, NULL, DB_FREE_SPACE, NULL);
+ // if (res < 0)
+ // _error->Warning("compact failed with result %i", res);
- if(_config->FindB("Debug::APT::FTPArchive::Clean", false) == true)
- Dbp->stat_print(Dbp, 0);
+ if(_config->FindB("Debug::APT::FTPArchive::Clean", false) == true) {
+ // FIXME: lmdb has stat but not stat_print -- print this info somehow
+ MDB_stat stat;
+ mdb_env_stat(Dbp, &stat);
+ printf("%d\t%d\t%d\t%d\t%d\t%d\n", stat.ms_psize, stat.ms_depth, stat.ms_branching_pages, stat.ms_leaf_pages, stat.ms_overflow_pages, stat.ms_entries)
+ }
return true;
diff --git c/ftparchive/cachedb.h i/ftparchive/cachedb.h
index 0002c60..f5fdf07 100644
--- c/ftparchive/cachedb.h
+++ i/ftparchive/cachedb.h
@@ -14,7 +14,7 @@
#include <apt-pkg/debfile.h>
#include <apt-pkg/hashes.h>
-#include <db.h>
+#include <lmdb.h>
#include <cerrno>
#include <cstdint>
#include <cstdio>
@@ -32,10 +32,10 @@ class CacheDB
protected:
// Database state/access
- DBT Key;
- DBT Data;
+ MDB_val Key;
+ MDB_val Data;
std::string TmpKey;
- DB *Dbp;
+ MDB_env *Dbp;
bool DBLoaded;
bool ReadOnly;
std::string DBFile;
@@ -46,8 +46,8 @@ class CacheDB
memset(&Key,0,sizeof(Key));
memset(&Data,0,sizeof(Data));
TmpKey.assign(FileName).append(":").append(Type);
- Key.data = TmpKey.data();
- Key.size = TmpKey.size();
+ Key.mv_data = TmpKey.data();
+ Key.mv_size = TmpKey.size();
}
void InitQueryStats() {
@@ -65,15 +65,23 @@ class CacheDB
inline bool Get()
{
- return Dbp->get(Dbp,0,&Key,&Data,0) == 0;
+ int err;
+ MDB_txn txn;
+ MDB_dbi dbi;
+ err = mdb_txn_begin(Dbp, NULL, MDB_RDONLY, &txn);
+ assert(0 == err);
+ err = mdb_dbi_open(txn, NULL, NULL, &dbi);
+ assert(0 == err);
+
+ return mdb_get(txn,0,&Key,&Data,0) == 0;
};
inline bool Put(const void *In,unsigned long const &Length)
{
if (ReadOnly == true)
return true;
- Data.size = Length;
- Data.data = (void *)In;
- if (DBLoaded == true && (errno = Dbp->put(Dbp,0,&Key,&Data,0)) != 0)
+ Data.mv_size = Length;
+ Data.mv_data = (void *)In;
+ if (DBLoaded == true && (errno = mdb_put(Dbp,0,&Key,&Data,0)) != 0)
{
DBLoaded = false;
return false;
