FTBFS with parallel make

[Philipp Hahn]

Hello fellows,

we (Univention GmbH) rebuild packages (from Debian-Jessie or newer)
using "-j8". Several builds failed, but work when I use "-j1":

* gcc-4.7
> mv: cannot stat 'stamps/07-install-stamp': No such file or directory
> debian/rules.d/binary-libgcc.mk:362: recipe for target 
> 'stamps/08-binary-stamp-libgcc-dev' failed
> make[1]: *** [stamps/08-binary-stamp-libgcc-dev] Error 1
> make[1]: *** Waiting for unfinished jobs
> mv: cannot stat 'stamps/07-install-stamp': No such file or directory

* perl-5.20.2-3+deb8u9
* systemd-215-17+deb8u7
* exim-4.84.2-2+deb8u4
> cp: cannot stat './EDITME.eximon.oOCpeNf': No such file or directory
> debian/rules:116: recipe for target 'b-exim4-daemon-heavy/Makefile' failed
> make: *** [b-exim4-daemon-heavy/Makefile] Error 123

* freeradius-2.2.5+dfsg-0.2+deb8u1
* mysql-5.5.58-0+deb8u1: Bug#857059
* libpaper: Bug#857058
* isc-dhcp: Bug#651922
* e2fsprogs: (attachment)

IMHO with todays multi-core CPUs -jX is a must:
- If I rebuild the full distribution, building multiple packages each
with -j1 is okay.
- But if I need a single package not being able to use -jX is a NO-GO:
For Spectre I re-build gcc-4.9 and it took 13h15m with -j1, because -j8
failed with the above error.

What do you thing: If parallel build a worthy goal?

With all the reproducible build stuff going on, I think it would be nice
if someone™ can also donate CPU time to check that -j`nproc` works.

Philipp
--- Begin Message ---
Hello Theodore,

Am 10.02.2017 um 22:02 schrieb Theodore Ts'o:
> On Thu, Feb 09, 2017 at 05:03:30PM +0100, Philipp Hahn wrote:
>>  install-std: DH_OPTIONS=
>> -install-std: build
>> +install-std: build cleanup
> 
> I don't think this works.  The clean target *must* be done before the
> build target.  In fact, it must be completed before the build target
> starts --- and the way the depedencies are specified above, dh_prep's
> deletion of the build directory would be racing with the build
> target's trying to populate them.
> 
> No?

I think "No" ;-)
Notice the difference between "clean" and "clean*up*":

> $ sed -ne '/^clean/,/^$/p' debian/rules
> clean:
> dh_testdir
> rm -rf ${STAMPSDIR}
> [ ! -f ${stdbuilddir}/Makefile ] || $(MAKE) -C ${stdbuilddir} V=1 
> distclean
> [ ! -f ${bfbuilddir}/Makefile ] || $(MAKE) -C ${bfbuilddir} V=1 
> distclean
> [ ! -f ${staticbuilddir}/Makefile ] || $(MAKE) -C ${staticbuilddir} 
> V=1 distclean
> rm -rf ${stdbuilddir} ${bfbuilddir} ${staticbuilddir} ${mipsbuilddir} 
> ${mipsbuilddir64}
> rm -f debian/*.substvars
> dh_clean
> 
> cleanup:
> dh_testdir
> dh_testroot
> dh_prep

When "dh_prep" is called, it only removed the files belonging to Debian
packages, e.g. the debhelper fragments, package specific directories,
substvars and tmp/. - all the things that must be removed before
"binary" builds the Debian packages:

> $ ls -1 debian > ../before
> $ dh_prep
> $ ls -1 debian > ../after
> $ diff ../before ../after |grep ^\<
> < comerr-dev
> < comerr-dev.substvars
> < e2fsck-static
> < e2fsck-static.substvars
> < e2fslibs
> < e2fslibs-dbg
> < e2fslibs-dbg.substvars
> < e2fslibs-dev
> < e2fslibs-dev.substvars
> < e2fslibs.postinst.debhelper
> < e2fslibs.postrm.debhelper
> < e2fslibs.substvars
> < e2fsprogs
> < e2fsprogs-dbg
> < e2fsprogs-dbg.substvars
> < e2fsprogs-udeb
> < e2fsprogs-udeb.substvars
> < e2fsprogs.substvars
> < libcomerr2
> < libcomerr2-dbg
> < libcomerr2-dbg.substvars
> < libcomerr2.postinst.debhelper
> < libcomerr2.postrm.debhelper
> < libcomerr2.substvars
> < libss2
> < libss2-dbg
> < libss2-dbg.substvars
> < libss2.postinst.debhelper
> < libss2.postrm.debhelper
> < libss2.substvars
> < ss-dev
> < ss-dev.substvars
> < tmp

So "cleanup" must be called fore "binary" or "binary-arch" or
"binary-indep".
FYI: First calling "make binary-arch" and then "make binary-indep" is
not equivalent to "make binary", as the state would be purged in between
then.

I've tested the build multiple times on a 4 CPU KVM: without the patch
most builds fail with the quoted error or with some other race-condition
error; since the patch every built (~6) went fine.

Philipp "pmh...@debian.org" Hahn
--- End Message ---

Re: FTBFS with parallel make

[Joerg Jaspert]

On 14929 March 1977, Philipp Hahn wrote:
> we (Univention GmbH) rebuild packages (from Debian-Jessie or newer)
> using "-j8". Several builds failed, but work when I use "-j1":

> With all the reproducible build stuff going on, I think it would be nice
> if someone™ can also donate CPU time to check that -j`nproc` works.

Very well volunteered, you are running the builds already, so go the
next step and submit bugs and patches based on them. Not very useful for
others to waste time in doing the thing you already run.

-- 
bye, Joerg

Re: FTBFS with parallel make

[Andrey Rahmatullin]

On Fri, Jan 26, 2018 at 09:42:05AM +0100, Philipp Hahn wrote:
> we (Univention GmbH) rebuild packages (from Debian-Jessie or newer)
> using "-j8".
Is that a dpkg-buildpackage option? It's documented to fail on certain
packages, you need to use -J instead, and maintainers need to certify that
a package can be built in parallel by bumping the debhelper compat level
or passing appropriate flags to debhelper tools.

> What do you thing: If parallel build a worthy goal?
Yes. Please test packages and submit patches.

> With all the reproducible build stuff going on, I think it would be nice
> if someone™ can also donate CPU time to check that -j`nproc` works.
No, building the package with -j`nproc` is not enough, you also need to be
sure the result works correctly (and repro is not very useful for this
until 100% coverage).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Bug#888489: ITP: ngspice-dfsg -- Spice circuit simulator - DFSG compatible parts

[Carsten Schoenert]

Package: wnpp
Severity: wishlist
Owner: Carsten Schoenert 

* Package name: ngspice-dfsg
  Version : 27
  Upstream Author : various (mainly Holger Vogt, Paolo Nenzi, Robert
Larice)
* URL : http://ngspice.sourceforge.net/
* License : BSD-3-clause, LGPL-2(+), GPL-2(+)
  Programming Lang: C, TclTk, plain text
  Description : Spice circuit simulator - DFSG compatible parts

Packages for NGspice are available due license incompatibilities in old
versions to the DFSG only in non-free.
With version 27 (released in September 2017) most of the used non DFSG
compatible licensed files/folders have been moved over to BSD 3-clause
(upstream uses here the name 'New BSD') and by this the build-able files
and binary stuff can now be considered as free enough by the DFSG.
There are still parts that uses licenses that aren't compatible with the
DFSG. All those parts can be excluded and the binaries are built without
support of that software parts. NGspice upstream has updated their
online available FAQ [2] about the used licenses, please look at point 1.5.

I prepared a wiki site to collect and track the status for files and
folders in question [3].

My working tree can be found on GitHub [4] for now. I'd really happy if
someone can have a look at this especially for the license issues which
the new version shouldn't have any longer. I had some nice communication
with upstream which gave me some guidance for the needed removal of
still not free enough licensed files and folders, which would bring in
some packages could still only served by non-free. I will probably also
maintain the src:ngspice package later as there are otherwise some
overlapping files between both source packages. I'm in contact to the
current Uploader Gudjon I. Gudjonsson about this.

I will maintain this package in the pkg-electronics-team, co-maintainers are
welcome! It's a build dependency for KiCad 5 which come with schematic
simulation based on the libngspice library which isn't available in the
non-free packages.

[1] https://tracker.debian.org/pkg/ngspice
[2] http://ngspice.sourceforge.net/faq.html
[3] https://wiki.debian.org/KiCad/ngspice
[4] https://github.com/tijuca/ngspice-dfsg

Re: Reducing the attack surface caused by Berkeley DB...

[David Kalnischkies]

On Thu, Jan 25, 2018 at 11:59:06PM +0100, Lionel Debroux wrote:
> In practice, Berkeley DB is a core component of most *nix distros.
> Debian popcon indicates that libdb5.3 is installed on ~80% of the
> computers which report to popcon.

I wonder how many of this ~80% is only due to having installed apt-utils
(99.83%) for apt-extracttemplates (which is responsible for having many
debconf questions before the installation process starts).

Anyway, the only util in apt-utils making use of libdb is apt-ftparchive
which a) isn't used much in Debian – but by some derivatives¹ and b) can
operate without the backing of a db, but you don't want to run a large
archive without it.

Famous last words, but I doubt there is anything libdb does for
ftparchive which couldn't be done by any other database, so switching
shouldn't be too hard database-wise…

Finding someone performing the daunting task of actually switching code,
documentation and existing databases over on the other hand… I at least
don't see me enthusiastically raising my arm crying "let me, let me, …".


Best regards

David Kalnischkies

¹ The Census has a field for "Archive tool", but that isn't filled by
everyone in the census. The biggest fish might be launchpad/Ubuntu.


signature.asc
Description: PGP signature

Re: Reducing the attack surface caused by Berkeley DB...

[Colin Watson]

On Fri, Jan 26, 2018 at 11:39:29AM +0100, David Kalnischkies wrote:
> On Thu, Jan 25, 2018 at 11:59:06PM +0100, Lionel Debroux wrote:
> > In practice, Berkeley DB is a core component of most *nix distros.
> > Debian popcon indicates that libdb5.3 is installed on ~80% of the
> > computers which report to popcon.
> 
> I wonder how many of this ~80% is only due to having installed apt-utils
> (99.83%) for apt-extracttemplates (which is responsible for having many
> debconf questions before the installation process starts).
> 
> Anyway, the only util in apt-utils making use of libdb is apt-ftparchive
> which a) isn't used much in Debian – but by some derivatives¹ and b) can
> operate without the backing of a db, but you don't want to run a large
> archive without it.

I agree that the database cache makes a substantial difference at this
scale.  On the occasions when we've suffered corruption and had to
rebuild it, it's taken on the order of 12 hours for a single suite,
while normal runs are more like 15-20 minutes.

> Famous last words, but I doubt there is anything libdb does for
> ftparchive which couldn't be done by any other database, so switching
> shouldn't be too hard database-wise…
> 
> Finding someone performing the daunting task of actually switching code,
> documentation and existing databases over on the other hand… I at least
> don't see me enthusiastically raising my arm crying "let me, let me, …".

I don't blame you!

> ¹ The Census has a field for "Archive tool", but that isn't filled by
> everyone in the census. The biggest fish might be launchpad/Ubuntu.

It's true that Launchpad uses apt-ftparchive for the Ubuntu archive.  We
publish PPAs directly from our database rather than going via
apt-ftparchive, but we've never quite managed to get that to perform
acceptably at the scale of the Ubuntu archive.  I suppose it might be
worth another go.

-- 
Colin Watson   [cjwat...@debian.org]

Re: Reducing the attack surface caused by Berkeley DB...

[Miriam Ruiz]

2018-01-26 12:02 GMT+01:00 Colin Watson :
>> Finding someone performing the daunting task of actually switching code,
>> documentation and existing databases over on the other hand… I at least
>> don't see me enthusiastically raising my arm crying "let me, let me, …".
>
> I don't blame you!

Might that be a candidate project for GSOC?

Greetings,
Miry

Re: Reducing the attack surface caused by Berkeley DB...

[Timo Aaltonen]

On 26.01.2018 00:59, Lionel Debroux wrote:
> ---
> Do you think we should start the journey of getting rid of libdb5.3 at a
> wide scale ? And if so, how to optimize resource usage in general ? :)
> ---

I asked 389-ds-base upstream about their plans, and got this draft plan
of getting rid of bdb from Fedora:

https://fedoraproject.org/wiki/User:Pkubat/Draft_-_Removing_BerkeleyDB_from_Fedora?rd=User:Jstanek/Draft_-_Removing_BerkeleyDB_from_Fedora

as for 389-ds-base, there is https://pagure.io/389-ds-base/issue/49476
for tracking the progress (not really started yet).


-- 
t

Re: FTBFS with parallel make

[Roberto C . Sánchez]

On Fri, Jan 26, 2018 at 02:37:08PM +0500, Andrey Rahmatullin wrote:
> On Fri, Jan 26, 2018 at 09:42:05AM +0100, Philipp Hahn wrote:
> > we (Univention GmbH) rebuild packages (from Debian-Jessie or newer)
> > using "-j8".
> Is that a dpkg-buildpackage option? It's documented to fail on certain
> packages, you need to use -J instead, and maintainers need to certify that
> a package can be built in parallel by bumping the debhelper compat level
> or passing appropriate flags to debhelper tools.
> 
That is interesting.  I build using gbp/cowbuilder and so I set these
environment variables:

DEB_BUILD_OPTIONS="parallel=`nproc`" DH_VERBOSE=1

I was not previously aware of the distinction between -j and -J for
dpkg-buildpackage.  However, looking at the dpkg-buildpackage man page
there does not appear to be a DEB_BUILD_OPTIONS setting to trigger the
use of -J in place of -j.  At least, that is the case on stretch.  Is
there an easy way (preferrably via environment variables) to achieve
that?

Regards,

-Roberto

-- 
Roberto C. Sánchez

Re: FTBFS with parallel make

[Andrey Rahmatullin]

On Fri, Jan 26, 2018 at 07:02:52AM -0500, Roberto C. Sánchez wrote:
> That is interesting.  I build using gbp/cowbuilder and so I set these
> environment variables:
> 
> DEB_BUILD_OPTIONS="parallel=`nproc`" DH_VERBOSE=1
> 
> I was not previously aware of the distinction between -j and -J for
> dpkg-buildpackage.  However, looking at the dpkg-buildpackage man page
> there does not appear to be a DEB_BUILD_OPTIONS setting to trigger the
> use of -J in place of -j.  At least, that is the case on stretch.  Is
> there an easy way (preferrably via environment variables) to achieve
> that?
AFAIK -J *is* DEB_BUILD_OPTIONS=parallel, as opposed to -j setting
MAKEFLAGS which directly affects upstream makefiles.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: FTBFS with parallel make

[Adam Borowski]

On Fri, Jan 26, 2018 at 09:42:05AM +0100, Philipp Hahn wrote:
> we (Univention GmbH) rebuild packages (from Debian-Jessie or newer)
> using "-j8". Several builds failed, but work when I use "-j1":

> IMHO with todays multi-core CPUs -jX is a must:
> - If I rebuild the full distribution, building multiple packages each
> with -j1 is okay.
> - But if I need a single package not being able to use -jX is a NO-GO:
> For Spectre I re-build gcc-4.9 and it took 13h15m with -j1, because -j8
> failed with the above error.
> 
> What do you thing: If parallel build a worthy goal?

Well, it is worthy, but let's take a look at buildd options:
* amd64:  parallel=4
* arm64:  parallel=6
* armel:  parallel=4
* armhf:  parallel=4
* i386:   parallel=4
* mips:   parallel=4
* mips64el:   parallel=4
* mipsel: parallel=4
* ppc64el:parallel=4
* s390x:  parallel=2

* alpha:  parallel=2
* hppa:   parallel=1
* hurd-i386:  parallel=1
* ia64:   parallel=2
* kfreebsd-amd64: (buildd down)
* kfreebsd-i386:  (buildd down)
* m68k:   nobench nocheck
* powerpc:parallel=4
* powerpcspe: nobench nocheck
* ppc64:  parallel=32
* sh4:nobench nocheck
* sparc64:parallel=32
* x32:parallel=2
(May differ per-buildd, I listed only one per arch.)

So this is _already_ done; at most individual packages might not use this
option, either via an explicit opt-out, or by having old crufty packaging.

Most of those packages are too small to bother; if you find one that fails
despite having a lengthy build, it'd be worth reporting.


However:
> * perl-5.20.2-3+deb8u9
> * systemd-215-17+deb8u7
> * exim-4.84.2-2+deb8u4
> * freeradius-2.2.5+dfsg-0.2+deb8u1
> * mysql-5.5.58-0+deb8u1: Bug#857059
> * libpaper: Bug#857058
> * isc-dhcp: Bug#651922
> * e2fsprogs: (attachment)

You're not expecting wishlist bugs to be fixed in oldstable, are you?


> if someone™ can also donate CPU time to check that -j`nproc` works.

As others in the thread have already mentioned, -j is an old broken option
that probably should be removed, -J is the one to use; the preferred way is
to set DEB_BUILD_OPTIONS=parallel=auto, although I think this is already the
default these days.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ The bill with 3 years prison for mentioning Polish concentration
⣾⠁⢰⠒⠀⣿⡁ camps is back.  What about KL Warschau (operating until 1956)?
⢿⡄⠘⠷⠚⠋⠀ Zgoda?  Łambinowice?  Most ex-German KLs?  If those were "soviet
⠈⠳⣄ puppets", Bereza Kartuska?  Sikorski's camps in UK (thanks Brits!)?

Re: FTBFS with parallel make

[Andrey Rahmatullin]

On Fri, Jan 26, 2018 at 03:07:25PM +0100, Adam Borowski wrote:
> So this is _already_ done; at most individual packages might not use this
> option, either via an explicit opt-out, or by having old crufty packaging.
I wonder what percent of the packages has compat < 10.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: FTBFS with parallel make

[Jeremy Bicha]

On Fri, Jan 26, 2018 at 9:38 AM, Andrey Rahmatullin  wrote:
> I wonder what percent of the packages has compat < 10.

Well start with
https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html

Thanks,
Jeremy Bicha

Re: FTBFS with parallel make

[Adam Borowski]

On Fri, Jan 26, 2018 at 07:38:21PM +0500, Andrey Rahmatullin wrote:
> On Fri, Jan 26, 2018 at 03:07:25PM +0100, Adam Borowski wrote:
> > So this is _already_ done; at most individual packages might not use this
> > option, either via an explicit opt-out, or by having old crufty packaging.
> I wonder what percent of the packages has compat < 10.

For packages in jessie (as in the original post), I believe the percent is
quite high.


-- 
⢀⣴⠾⠻⢶⣦⠀ The bill with 3 years prison for mentioning Polish concentration
⣾⠁⢰⠒⠀⣿⡁ camps is back.  What about KL Warschau (operating until 1956)?
⢿⡄⠘⠷⠚⠋⠀ Zgoda?  Łambinowice?  Most ex-German KLs?  If those were "soviet
⠈⠳⣄ puppets", Bereza Kartuska?  Sikorski's camps in UK (thanks Brits!)?

I sent wrong description in BUG TRACKING SYSTEM

[Manas Kashyap]

Is there a way to rectify the wrong description i sent to the bug tracking
system . Bug#888475

Re: I sent wrong description in BUG TRACKING SYSTEM

[Adam Borowski]

On Fri, Jan 26, 2018 at 08:47:45PM +0530, Manas Kashyap wrote:
> Is there a way to rectify the wrong description i sent to the bug tracking
> system . Bug#888475

Specify as the first line[s] of your mail to 888...@bugs.debian.org :
Control: retitle -1 ITP: my-doodad: new swanky short desc

The rest of the bug is immutable, but is not read by machine, just humans --
so instead of replacing, you post the new version.

This said, you don't need to make the posted description be identical to
what will be uploaded, it's just a starting point so people can comment upon
it.

Keeping the short desc (bug title) is more important -- not strictly
required, but it makes reading WNPP a lot more accurate.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ The bill with 3 years prison for mentioning Polish concentration
⣾⠁⢰⠒⠀⣿⡁ camps is back.  What about KL Warschau (operating until 1956)?
⢿⡄⠘⠷⠚⠋⠀ Zgoda?  Łambinowice?  Most ex-German KLs?  If those were "soviet
⠈⠳⣄ puppets", Bereza Kartuska?  Sikorski's camps in UK (thanks Brits!)?

Re: FTBFS with parallel make

[Niels Thykier]

Jeremy Bicha:
> On Fri, Jan 26, 2018 at 9:38 AM, Andrey Rahmatullin  wrote:
>> I wonder what percent of the packages has compat < 10.
> 
> Well start with
> https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html
> 
> Thanks,
> Jeremy Bicha
> 

For the curious:

 * Deprecated compat levels: ~5300 packages
 * Compat 9: ~14800 packages
 * Compat 10: ~6700 packages
 * Compat 11: Unknown (but at most 1132)

Please keep in mind that compat 9 and earlier could use --parallel (and
compat 10+ can still use --no-parallel or --max-parallel), so these
numbers are at best rough guesstimates for the number of packages with
parallel enabled/disabled.

Thanks,
~Niels

Data sources:
 *
https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html
 *
https://lintian.debian.org/tags/package-uses-old-debhelper-compat-version.html

Re: Reducing the attack surface caused by Berkeley DB...

[Lionel Debroux]

Hi Ryan,

On 1/26/18 1:02 AM, Ryan Tandy wrote:
> On Thu, Jan 25, 2018 at 11:59:06PM +0100, Lionel Debroux wrote:
> > the vast majority of the ~170 reverse dependencies of libdb5.3
> > listed by `apt-cache rdepends libdb5.3` on sid will require (much)
> > more work to get rid of that dependency, with impact on backwards
> > compatibility...
> > Among those packages are:
> > [...] slapd
>
> The BDB-based backends are already deprecated upstream in favour of
> LMDB, so when the time comes I'm happy to forcibly migrate the
> remaining users on upgrade. At that point we can simply stop building
> the module, possibly even in the same version where we force the
> migration.
>
> There are still a few cases where LMDB may not be entirely
> satisfactory: databases containing a lot of aliases, or workloads that
> happen to cause a lot of fragmentation in LMDB. LMDB 1.0/OpenLDAP 2.5
> will have some improvements in these areas. (No ETA from upstream at
> this time.)
>
> Hope this helps, or at least marks slapd as "minor" on your list.
Good to know :)
I don't use slapd myself, but I noticed it in the list of daemons which
directly depend on libdb5.3.


Thanks,
Lionel.

Re: Reducing the attack surface caused by Berkeley DB...

[Lionel Debroux]

Hi Marco,

On 1/26/18 1:46 AM, Marco d'Itri wrote:
> On Jan 25, Lionel Debroux  wrote:
> > Several days ago, jmm from the security team suggested that I start
> > a discussion on debian-devel about Berkeley DB, which has known
> > security issues, because doing so may enable finding a consensus on
> > how to move
> Can you clarify the threat model?
I'll try to; admittedly, the threat model is less clear to me than the
number of publicly known weaknesses in a source package unmaintained
upstream (at least publicly), the ease of hanging or crashing the code
with corrupted input, and to some extent, the mild ease of creating such
corrupted input in non-fuzzing conditions (though it would probably have
been somewhat fixed earlier if it were frequent enough) :)

> E.g. is libdb attackable by user-supplied data from the program using
> it or do attacks require write access to the db files?
Most attacks I can imagine require write access to the DB files, but I'm
not convinced that most of the programs / libraries which use BDBs do it
in a read-only fashion, if that is possible.

As shown by the original reporter of #652036, BDB can (or could, at some
point) be programmatically caused to corrupt a DB file in such a way
that later opens, by e.g. a repair tool such as db_dump, causes an
infinite loop. I'm obviously unqualified to tell whether any of the many
callers of libdb can be locally, or remotely, caused to trigger such
data corruption.
Later, what got me interested in BDB as my first fuzzing target was DoS
following external data corruption caused by forced power off on a
laptop (<=> power outage for desktops, in theory), when launching `mocp`
from the package of the same name.
External data corruption could also infrequently occur on aging mass
storage (non-backup transfers of BDBs are probably infrequent), and bite
when restoring data, e.g. archived SVN repositories from the era before
FSFS became recommended. I got that thought because having a peek at
data from old backup tapes is relevant, once in every while, to some
people I know. I'll grant that the data sets these persons are
interested in are likely too old to be in a SVN in the first place, but
you get the idea.


Bye,
Lionel.

Re: Reducing the attack surface caused by Berkeley DB...

[Lionel Debroux]

Hi Timo,

On 1/26/18 12:21 PM, Timo Aaltonen wrote:
> On 26.01.2018 00:59, Lionel Debroux wrote:
> > ---
> > Do you think we should start the journey of getting rid of libdb5.3
> > at a wide scale ? And if so, how to optimize resource usage in
> > general ? :)
> > ---
>
> I asked 389-ds-base upstream about their plans, and got this draft
> plan of getting rid of bdb from Fedora:
>
>
https://fedoraproject.org/wiki/User:Pkubat/Draft_-_Removing_BerkeleyDB_from_Fedora?rd=User:Jstanek/Draft_-_Removing_BerkeleyDB_from_Fedora
>
> as for 389-ds-base, there is https://pagure.io/389-ds-base/issue/49476
> for tracking the progress (not really started yet).
Interesting, thanks a lot for the pointers :)


Bye,
Lionel.

Re: Reducing the attack surface caused by Berkeley DB...

[Lionel Debroux]

Hi Scott,

On 1/26/18 7:05 AM, Scott Kitterman wrote:
> On Thursday, January 25, 2018 11:59:06 PM Lionel Debroux wrote:
> >
> > [...]
> > ---
> > Do you think we should start the journey of getting rid of libdb5.3
> > at a wide scale ? And if so, how to optimize resource usage in
> > general ? :)
> > ---
>
> Ultimately BDB is a dead end for non-AGPL projects.  So my answer to
> your first question is a definite yes.
>
> I'd like to know what the preferred replacement is.  I maintain a few
> less heavily used packages that use libdb5.3 and I need to know what
> to tell upstream they should port to.  I don't know enough to have a
> real technical opinion.  Is lmdb the general solution?
In some (most ?) cases, it would seem so, indeed.
But there's now a wider variety of key/value databases (even those which
don't fork and communicate through *nix or network sockets) than, say, a
decade ago. I remember that several years ago, bitcoind/bitcoin-qt
switched from BDB to LevelDB.

> As far as postfix goes (which I also co-maintain) that is a two
> release cycle project (it's complicated, but upgrades don't work
> otherwise - if anyone cares see what we did for postfix-sqlite.  It's
> no problem to switch to a difference default map type, but it'd be
> nice if we could switch it once to something that was otherwise
> already likely to be installed.
liblmdb* or libleveldb* are much less popular in popcon by_inst than
libdb, yeah...


Do we know whether LMDB, and other candidate databases for replacing
BDB, have received suitable hardening against data corruption and fuzzing ?


Bye,
Lionel.

Re: Reducing the attack surface caused by Berkeley DB...

[Lionel Debroux]

Hi,

On 1/26/18 11:39 AM, David Kalnischkies wrote:
> On Thu, Jan 25, 2018 at 11:59:06PM +0100, Lionel Debroux wrote:
> > In practice, Berkeley DB is a core component of most *nix distros.
> > Debian popcon indicates that libdb5.3 is installed on ~80% of the
> > computers which report to popcon.
> I wonder how many of this ~80% is only due to having installed
> apt-utils (99.83%) for apt-extracttemplates (which is responsible for
> having many debconf questions before the installation process starts).
Indeed. The weight of apt-utils (#65 popcon by_inst), and even more so
of libpam-modules (#5 !), in libdb5.3's popularity is probably pretty
high. Even if both of those dependencies went away, libdb5.3 would
remain quite popular anyway due to Perl, Python (2 and 3), and others.

> Anyway, the only util in apt-utils making use of libdb is
> apt-ftparchive which a) isn't used much in Debian – but by some
> derivatives¹ and b) can operate without the backing of a db, but you
> don't want to run a large archive without it.
Could that program conceivably be split to another package ?
This also goes for libpam-modules, where pam_userdb.so is the only user
of libdb5.3.
Of course, I probably don't foresee all of the consequences of such
changes - maybe splits are harder to perform on essential packages, to
begin with ?

> Famous last words, but I doubt there is anything libdb does for
> ftparchive which couldn't be done by any other database, so switching
> shouldn't be too hard database-wise…
>
> Finding someone performing the daunting task of actually switching
> code, documentation and existing databases over on the other hand… I
> at least don't see me enthusiastically raising my arm crying "let me,
> let me, …".
Heh. Neither do I see myself for such a task :)
Miriam's suggestion (BTW: thanks Miriam for some Debian packaging
example I used years ago at work) makes sense.


Thanks,
Lionel.

Re: Reducing the attack surface caused by Berkeley DB...

[Guillem Jover]

Hi!

On Thu, 2018-01-25 at 23:59:06 +0100, Lionel Debroux wrote:
> Several days ago, jmm from the security team suggested that I start a
> discussion on debian-devel about Berkeley DB, which has known security
> issues, because doing so may enable finding a consensus on how to move
> away from it in Debian (which is hard). So here's a post :)

> ---
> Do you think we should start the journey of getting rid of libdb5.3 at a
> wide scale ? And if so, how to optimize resource usage in general ? :)
> ---

As with many things in Debian, this was already discussed some years
ago. :) The maintainers are supposedly even on board, see the thread
starting at:

  

Thanks,
Guillem

Bug#888554: ITP: wxpython4.0 -- Python interface to the wxWidgets Cross-platform C++ GUI toolkit, Phoenix version

[Scott Talbert]

Package: wnpp
Severity: wishlist
Owner: Scott Talbert 

* Package name: wxpython4.0
  Version : 4.0.0
  Upstream Author : Robin Dunn 
* URL : https://www.wxpython.org/
* License : wxWindows Library License
  Programming Lang: Python
  Description : Python interface to the wxWidgets Cross-platform C++ GUI 
toolkit, Phoenix version

wxWidgets (formerly known as wxWindows) is a class library for C++ providing
GUI components and other facilities on several popular platforms (and some
unpopular ones as well).
.
This package provides a Python interface to the wxGTK library and the 
wxPython runtime support libraries.  This is the "Phoenix" 
implementation which now supports Python 3.

Re: Reducing the attack surface caused by Berkeley DB...

[Adrian Bunk]

On Fri, Jan 26, 2018 at 11:49:41PM +0100, Lionel Debroux wrote:
>...
> On 1/26/18 11:39 AM, David Kalnischkies wrote:
>...
> > Finding someone performing the daunting task of actually switching
> > code, documentation and existing databases over on the other hand… I
> > at least don't see me enthusiastically raising my arm crying "let me,
> > let me, …".
> Heh. Neither do I see myself for such a task :)
>...

Open source development does not work by random people on the internet 
telling volunteers in a project what they should do.

Guillem already pointed at a mail thread with the same suggestion
on this same mailing list 4 years ago.

There doesn't seem to be any disagreement on the general idea,
the only thing missing is a person doing the work on getting
all Debian packages ported away from Berkeley DB while ensuring
that users don't lose data due to that.[1]

Since this is something that is important for you,
this is a great opportunity for you to get involved
in Debian development.

If it is not important enough for you to spend your time on it,
then 2022 is the estimated date when the next person will start
a "remove Berkeley DB" discussion on this mailing list...  ;-)

> Thanks,
> Lionel.

cu
Adrian

[1] it is not clear whether there will have to be one more
Debian release with Berkeley DB just for upgrades

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed