Byte swapped /usr/share?
Hi all, again a question for large data files (and still the same packages [1] :-) ) I need to package some large (~80MB) data files with a machine dependent byte order. How should I do this? * Creating machine dependent data files (arch=ANY) is obviously wasting Debian server space. Also, all big endian archs would share the same files. * Putting files for all endianesses into the same package would also waste Debian server, but also local disk space. * Swapping them during install would destroy the checksum of the files, so they are no longer verifyable. * The program itself (astrometry.net) would be quite difficult to change so that it can handle other than natural byte orders, and this also would be a perfomance problem. However, I opened an issue for that [2]. Best regards Ole [1] http://bugs.debian.org/795933 [2] https://github.com/dstndstn/astrometry.net/issues/66
Re: Byte swapped /usr/share?
* Ole Streicher , 2015-09-26, 10:28: How should I do this? * Creating machine dependent data files (arch=ANY) is obviously wasting Debian server space. Also, all big endian archs would share the same files. * Putting files for all endianesses into the same package would also waste Debian server, but also local disk space. * Build two arch:all packages, one little-endian and one big-endian. * Swapping them during install would destroy the checksum of the files, so they are no longer verifyable. Yeah, don't do this. -- Jakub Wilk
Re: service failures should not fail dpkg installation
Hallo, * Marvin Renich [Fri, Sep 25 2015, 08:26:34AM]: > > > Still, that would need some proper design work, and a reasonable amount of > > > code to be written and tested. Some of it will hook into the package > > > system, some of it needs to interface to the services subsystem (systemd, > > > sysvinit, others). > > I agree, but I don't think we should wait for this feature to appear > before fixing packages to _not_ fail upgrades when the service fails to > start. The current situation does more harm than good. I am wondering why this topic doesn't get more attention. For me, it feels like being one of the top causes of breaking an upgrade process somewhere inbetween, leaving the system in some intermediate state... with modern APT, it has become easier to continue from this messy situation but it's still a situation I would like to avoid. > I agree completely. The decision on whether or not to fail the dpkg > installation should depend on what action needs to be taken to correct > the situation (and this is true whether we are talking about a service > failing to start or a database upgrade failure or something else). > > However, most existing cases of service restart failures require > something other than re-running the dpkg installation to fix them, and > the default, without careful thought by the maintainer about the > possible failure modes, should be to allow the dpkg run to succeed. The basic idea might be that a package should be able to handle startup failures in different categories (and resolution strategies), defined by maintainers. However, it's not so easy because of subsequent errors that might happen in other services far way in the dependency chain, and it's hard to predict them all. We need some compromise here. Something I imagine is: a) packages that participate in the "error-tolerant" scheme get some attribute set. They also run delicate commands through a wrapper command that collects the failure/success state and records TODO tickets in some global configuration file. b) apt might add additional hints to the package installation, letting maintainer scripts know whether there are dependent packages somewhere in the chain. c) for failed tasks, dpkg and apt frontends show the user messages "there are things to fix that require your attention: ", and when the admin solved the problem, he can close the ticket with the imaginary tool. > Should I open a wishlist bug against the developers reference pointing > to this discussion? Feel free. Best regards, Eduard.
Re: Byte swapped /usr/share?
Hello, On Sat, 2015-09-26 at 10:28 +0200, Ole Streicher wrote: > Hi all, > > I need to package some large (~80MB) data files with a machine > dependent byte order. > > How should I do this? > How about creating one package -bigendian and one -lowendian and a virtual package that depends on one or the other based on the architecture? Best, Gert
Re: Byte swapped /usr/share?
Gert Wollny writes: > On Sat, 2015-09-26 at 10:28 +0200, Ole Streicher wrote: >> Hi all, >> >> I need to package some large (~80MB) data files with a machine >> dependent byte order. >> >> How should I do this? >> > How about creating one package -bigendian and one -lowendian and a > virtual package that depends on one or the other based on the > architecture? Good idea. One drawback: I have not just one binary package, but five; this makes a total of 15 packages then. Best regards Ole
Re: Byte swapped /usr/share?
Gert Wollny writes:
> How about creating one package -bigendian and one -lowendian and a
> virtual package that depends on one or the other based on the
> architecture?
Just to make it sure: for the architectures that currently built on
Debian (officially or inofficially), I should then use someting like the
following in d/control:
Depends: astrometry-data-tycho2-07-bigendian [any-amd64 | any-i386 | armel |
mipsel | ppc64el | mips64el | x32 | alpha],
astrometry-data-tycho2-07-bigendian [mips | arm64 | armhf | powerpc |
powerpcspe | ppc64 | hppa | m68k | sparc64 | s390x | sh4],
${misc:Depends}
Does this look correctly (especiall what arch is little and whatis big
endian)? BTW, wrap-and-sort doesn't break this nicely :-(
Best regards
Ole
Re: Byte swapped /usr/share?
Hi,
On Sat, Sep 26, 2015 at 02:05:41PM +0200, Ole Streicher wrote:
> Just to make it sure: for the architectures that currently built on
> Debian (officially or inofficially), I should then use someting like the
> following in d/control:
>
> Depends: astrometry-data-tycho2-07-bigendian [any-amd64 | any-i386 | armel |
> mipsel | ppc64el | mips64el | x32 | alpha],
> astrometry-data-tycho2-07-bigendian [mips | arm64 | armhf | powerpc
> | powerpcspe | ppc64 | hppa | m68k | sparc64 | s390x | sh4],
> ${misc:Depends}
>
> Does this look correctly (especiall what arch is little and whatis big
> endian)? BTW, wrap-and-sort doesn't break this nicely :-(
ITYM -littleendian in the first line ;)
And you got some archs wrong, see
https://wiki.debian.org/ArchitectureSpecificsMemo
e.g. arm64/armhf is LE
Regards,
Rene
Re: binNMU or reproducible builds (choose only one)
On Thu, Sep 24, 2015 at 11:56:56AM +0100, Colin Watson wrote: > On Thu, Sep 24, 2015 at 12:29:59PM +0200, Santiago Vila wrote: > > But once we are able to trigger a rebuild with sourceful NMUs, as > > Ubuntu does, binNMUs will hopefully be a thing of the past. > > Amusingly, the way we do it in Ubuntu is a huge hassle in some cases, > and at least some of us would rather have binNMUs. (That's partly > because it's a manual process; if it were automated it would be better, > but it still wouldn't solve the problem that in some cases you really do > want to do single-architecture rebuilds without having to rebuild a > stack of packages on slower architectures entirely unnecessarily. Hi, > Haskell.) So. binNMU's are special in that they're linked (in the dak db) to a source with a different version than the binary package. My point was, really, that we could change the "special" feature of a binNMU so that it isn't linked to a souce with a different version anymore, but instead so that the source version of architecture X doesn't necessarily need to correspond to the source version of architecture Y. In effect, that would result in sourceful architecture-specific uploads rather than binNMU's. Of course, that doesn't fix the multiarch problem, but it does fix the reproducible build issue. -- It is easy to love a country that is famous for chocolate and beer -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26
Bug#800074: ITP: miller -- name-indexed data processing tool
Package: wnpp Severity: wishlist Owner: Stephen Kitt * Package name: miller Version : 2.2.1 Upstream Author : John Kerl * URL : https://github.com/johnkerl/miller * License : BSD-2-Clause Programming Lang: C Description : name-indexed data processing tool Miller allows name-indexed data such as CSV files to be processed with functions equivalent to sed, awk, cut, join, sort etc. It can convert between formats, preserves headers when sorting or reversing, and streams data where possible so its memory requirements stay small. It works well with pipes and can feed "tail -f".
Re: Byte swapped /usr/share?
* Rene Engelhard , 2015-09-26, 14:30:
Just to make it sure: for the architectures that currently built on
Debian (officially or inofficially), I should then use someting like
the following in d/control:
Depends: astrometry-data-tycho2-07-bigendian [any-amd64 | any-i386 | armel |
mipsel | ppc64el | mips64el | x32 | alpha],
astrometry-data-tycho2-07-bigendian [mips | arm64 | armhf | powerpc |
powerpcspe | ppc64 | hppa | m68k | sparc64 | s390x | sh4],
${misc:Depends}
Does this look correctly (especiall what arch is little and whatis big
endian)? BTW, wrap-and-sort doesn't break this nicely :-(
ITYM -littleendian in the first line ;)
And you got some archs wrong, see
https://wiki.debian.org/ArchitectureSpecificsMemo e.g. arm64/armhf is
LE
Also, the syntax is wrong. Architectures between square brackets should
be separated by spaces.
But more importantly, every time you hardcode architecture list in
debian/control, you make architecture bootstrappers cry.
Instead, you should use something like this:
Depends: ${astrometry-data}, ${misc:Depends}
and generate substitution for ${astrometry-data} appropriate for the
host architecture in debian/rules.
--
Jakub Wilk
Re: Byte swapped /usr/share?
On Sat, Sep 26, 2015 at 03:04:47PM +0200, Jakub Wilk wrote:
> * Rene Engelhard , 2015-09-26, 14:30:
> >>Just to make it sure: for the architectures that currently built
> >>on Debian (officially or inofficially), I should then use
> >>someting like the following in d/control:
> >>
> >>Depends: astrometry-data-tycho2-07-bigendian [any-amd64 | any-i386 | armel
> >>| mipsel | ppc64el | mips64el | x32 | alpha],
> >>astrometry-data-tycho2-07-bigendian [mips | arm64 | armhf | powerpc
> >> | powerpcspe | ppc64 | hppa | m68k | sparc64 | s390x | sh4],
> >>${misc:Depends}
> >>
> >>Does this look correctly (especiall what arch is little and
> >>whatis big endian)? BTW, wrap-and-sort doesn't break this nicely
> >>:-(
> >
> >ITYM -littleendian in the first line ;)
> >
> >And you got some archs wrong, see
> >https://wiki.debian.org/ArchitectureSpecificsMemo e.g. arm64/armhf
> >is LE
>
> Also, the syntax is wrong. Architectures between square brackets
> should be separated by spaces.
>
> But more importantly, every time you hardcode architecture list in
> debian/control, you make architecture bootstrappers cry.
>
> Instead, you should use something like this:
>
> Depends: ${astrometry-data}, ${misc:Depends}
>
> and generate substitution for ${astrometry-data} appropriate for the
> host architecture in debian/rules.
Wouldn't it be better to just add Depends on an empty
"astrometry-data-tycho2-07" in the programs, and have that one Depend on
either astrometry-data-tycho2-07-bigendian or
astrometry-data-tycho2-07-lowendian, so this gets decoupled.
Unless everything is in one source package anyway and not used
elsewhere, that is.
Michael
Re: Byte swapped /usr/share?
On Sat, Sep 26, 2015 at 03:04:47PM +0200, Jakub Wilk wrote:
> But more importantly, every time you hardcode architecture list in
> debian/control, you make architecture bootstrappers cry.
>
> Instead, you should use something like this:
>
> Depends: ${astrometry-data}, ${misc:Depends}
>
> and generate substitution for ${astrometry-data} appropriate for the host
> architecture in debian/rules.
... especially if you use DEB_HOST_ARCH_ENDIAN to do it. See
dpkg-architecture(1).
--
Colin Watson [cjwat...@debian.org]
Re: service failures should not fail dpkg installation [was: Re: promoting virtualbox-dkms to virtualbox pre-depends]
At Thu, 24 Sep 2015 08:11:48 -0400, Marvin Renich wrote: > > * Jeroen Dekkers [150924 07:23]: > > At Wed, 23 Sep 2015 13:53:11 -0400, > > Marvin Renich wrote: > > > I think it should be documented in the developers reference that if you > > > attempt to start or restart a service in postinst, you should guard it > > > so that a failure in the service does not propagate to a failure of the > > > postinst. > > > > But then when something goes wrong when upgrading and the service > > doesn't (re)start apt/dpkg will report success but the service isn't > > running anymore. That also sounds wrong to me. Letting postinst fail > > might not be the best way to signal this, but to change that we need > > something else to let the user know that something went wrong. Just > > printing an error message isn't enough, because the user might not see > > that (for example when multiple packages are installed/upgraded and a > > later package asks some questions using dialog or when using > > unattended-upgrades). > > How does failing the upgrade solve anything? The upgrade should only > fail if the failure of the service to start was because something in the > upgrade itself was broken; this is rarely the case. I think it solves the problem of notifying the user that something went wrong quite clearly. Not in the correct way, I agree with that, but the solution to that should be to notify the user in a better way, not to stop notifying the user. Failing silently is worse than failing in the wrong way. > What makes this even worse is that when installing or upgrading a large > number of packages, this kind of incorrect failure sometimes affects > many completely unrelated packages. For an unattended upgrade, this is > so much worse than having one service that (for a correct reason) > refused to restart after the upgrade. Unattended-upgrades has the MinimalSteps option that splits upgrades in the smallest possible chunks so that isn't really a problem. > What you are looking for is a more prominent notification that a service > did not restart. But the current situation is like the "check engine" > light flashing when you are low on fuel; yes, it gets your attention, > but it is telling you the wrong thing. Yes, but the way to solve that is to flash a "low on fuel" light, not to stop notifying you and leaving you alone in the desert without fuel. And if a "low on fuel" light isn't possible, it's better to keep flashing the "check engine" light like it has been doing for the past 15 years. Kind regards, Jeroen Dekkers
Re: Byte swapped /usr/share?
Jakub Wilk writes:
> Also, the syntax is wrong. Architectures between square brackets
> should be separated by spaces.
OK. I diddn't find this in the docs.
> But more importantly, every time you hardcode architecture list in
> debian/control, you make architecture bootstrappers cry.
>
> Instead, you should use something like this:
>
> Depends: ${astrometry-data}, ${misc:Depends}
>
> and generate substitution for ${astrometry-data} appropriate for the
> host architecture in debian/rules.
The virtual package should then not be Architecture: all, but "any"?
Best regards
Ole
Re: binNMU or reproducible builds (choose only one)
On Sat, 2015-09-26 at 14:35 +0200, Wouter Verhelst wrote: > On Thu, Sep 24, 2015 at 11:56:56AM +0100, Colin Watson wrote: > > On Thu, Sep 24, 2015 at 12:29:59PM +0200, Santiago Vila wrote: > > > But once we are able to trigger a rebuild with sourceful NMUs, as > > > Ubuntu does, binNMUs will hopefully be a thing of the past. > > > > Amusingly, the way we do it in Ubuntu is a huge hassle in some cases, > > and at least some of us would rather have binNMUs. (That's partly > > because it's a manual process; if it were automated it would be better, > > but it still wouldn't solve the problem that in some cases you really do > > want to do single-architecture rebuilds without having to rebuild a > > stack of packages on slower architectures entirely unnecessarily. Hi, > > Haskell.) > > So. > > binNMU's are special in that they're linked (in the dak db) to a source > with a different version than the binary package. [...] That's not specific to binNMUs. E.g. linux-latest always builds binaries with different version numbers from the source. Ben. -- Ben Hutchings Any smoothly functioning technology is indistinguishable from a rigged demo. signature.asc Description: This is a digitally signed message part
Re: Byte swapped /usr/share?
On Sat, Sep 26, 2015 at 03:30:39PM +0200, Ole Streicher wrote:
> Jakub Wilk writes:
> > Instead, you should use something like this:
> >
> > Depends: ${astrometry-data}, ${misc:Depends}
> >
> > and generate substitution for ${astrometry-data} appropriate for the
> > host architecture in debian/rules.
>
> The virtual package should then not be Architecture: all, but "any"?
That would be required for architecture-specific dependencies in any
case; [...] architecture specifiers are handled at build-time by
dpkg-gencontrol, not at install-time.
--
Colin Watson [cjwat...@debian.org]
Re: tyr-quake both quake1 & quakewolrd close to conservative engine stable-based
On Wed, Sep 23, 2015 at 12:20 PM, Mattia Rizzolo wrote: > On Wed, Sep 23, 2015 at 11:30:55AM -0430, PICCORO McKAY Lenz wrote: >> There's any consensus around quakeworld packagin? > > I don't hold a clue. Also you removed debian-devel@ from CC, so it's > quite improbable somebody will reply to you. re-added here, maybe wait > and se... I packaged and uploaded ezquake, a different quakeworld client. It still has upstream activity unlike tyr-quake. It's been waiting for ftp master review for about a month now [0]. Best wishes, Mike [0] https://ftp-master.debian.org/new/ezquake_2.2+git20150324-1.html
Bug#800093: ITP: libdpkg-parse-perl -- module to parse various dpkg files into Perl Objects
Package: wnpp Owner: Andy Beverley Severity: wishlist X-Debbugs-CC: debian-devel@lists.debian.org, debian-p...@lists.debian.org * Package name: libdpkg-parse-perl Version : 0.03 Upstream Author : Adam Jacob * URL : https://metacpan.org/release/DPKG-Parse * License : Artistic or GPL-1+ Programming Lang: Perl Description : module to parse various dpkg files into Perl Objects DPKG::Parse contains utilities to parse the various files created by dpkg and turn them into helpful Perl objects. Current files understood by various DPKG::Parse modules: /var/lib/dpkg/status - DPKG::Parse::Status /var/lib/dpkg/available - DPKG::Parse::Available Packages.gz - DPKG::Parse::Packages See each module's documentation for particulars - You should not be calling DPKG::Parse directly. The package will be maintained under the umbrella of the Debian Perl Group.
Re: DAK Commands for Bikesheds
On 09/21/2015 06:12 PM, Raphael Hertzog wrote:
> Hi,
>
> On Sun, 20 Sep 2015, Joerg Jaspert wrote:
>>> (It seems they are implicitly considered a master when updating the
>>> bikeshed's ACLs) I would have thought "Owner:" would make more sense
>>> than "Master:" fwiw.
>>
>> Then you end up having multiple owners. Master IMO shows better what the
>> intended value of it is - the ones with access to change around its
>> settings. (Got to that from the MASTER role in irc.debian.orgs chanserv).
>
> FWIW, I also find Owner clearer than Master. Or Admin. But master is
> rather unusual.
>
>> Your -pull-from is basically my -add, just with an added suite. My way
>> of thinking is coming from a way more limited usage, so I adjusted add
>> to optionally take Suite as parameter, now one can select from where the
>> package comes.
>
> BTW bikeshed-add and bikeshed-rm should be named "bikeshed-add-package"
> and "bikeshed-rm-package" otherwise they end up at the same level
> than "bikeshed-modify" and you would expect the current commands to
> add/remove a bikeshed instead of actually adding/removing a package
> in it.
>
> Cheers,
In which case maybe we want bikeshed-pkg-{add,rm} (and probably more
later, like maybe update or sync (from another bikeshed / suite)?).
Just my 2 cents...
Thomas
Bug#800114: RFA: libtext-header-perl -- RFC 822/2068 header and unheader functions
Package: wnpp Severity: normal Description-en: RFC 822/2068 header and unheader functions This module provides two new functions, header and unheader, which provide general-purpose RFC 822 header construction and parsing. They do not provide any intelligent defaults of HTTP-specific methods. They are simply aimed at providing an easy means to address the mechanics of header parsing. . The output style is designed to mimic CGI.pm and HTTP::Headers, so that users familiar with these interfaces will feel at home with these functions. If somebody wants to pick up libtext-header-perl, that'd be great. It's not particularly much used (popcon has 23 installations), but I don't use it any more and don't particularly care for it. Its packaging surely needs updating, since it hasn't received any attention for a long time. If nobody picks this up in a reasonable time frame, I'll just ask for it to be removed. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are
