Re: tag2upload service architecture and risk assessment - draft v2

[Raphael Hertzog]

Hi,

I reviewed the whole thread and the point of friction is the requirement
to sign the .dsc file to make sure that the source package matches what
the maintainer intended to upload. Ian doesn't want the maintainer to have
to deal with the .dsc and the ftpmasters wants to have a signature within
the archive to verify that the code that we build is the code that the
maintainer wanted to ship.

I have a proposal to get out of this impasse.

1/ When the maintainer generates the git tag, he will sign the tag
   but he will also sign the output of "git ls-tree -r HEAD"
   and that signature is stored in the tag long description.

   This signature will be the basis of the trust that ftpmaster are
   looking for.

2/ tag2upload constructs a .dsc that embeds a new field where the
   output of "git ls-tree -r HEAD" is provided (let's call it "Content")
   and a second field with the signature of the maintainer (let's call
   it "Content-Signature"). tag2upload signs the .dsc as a whole with
   its own key.

3/ "dpkg-source -x" is modified to remove any file that does not appear
   in the "Content" field. It could warn about missing files too (and maybe
   silently ignore for a few common files that are usually not part of the
   "make dist"). It could also verify the checksums and complain if they
   don't match.

4/ dak is modified to also verify the signature in "Content-Signature"

5/ profit, everybody should be happy.


Comments are welcome, obviously.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: Git Packaging: Native source formats

[Raphael Hertzog]

Hi,

On Wed, 28 Aug 2019, Sam Hartman wrote:
> Internet is faster and disks are cheaper.

I'm not sure that DSA (which is maintaining snapshot.debian.org) will
be happy with this assertion.

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.
> You don't need to synthesize a separate upstream tarball.

Do we really care about this if we assume that this work can be
transparent to the maintainer and entirely handled by tag2upload
(or some other equivalent service)?

It has all the information required to be able to build something
reusing an upstream tarball. It can:
- pick a pre-existing tarball if already in the archive
- pick the upstream tarball via the watch file
- pick the output of "git archive"

If there are still differences due to the use of an external tarball, it
can generate a supplementary patch to bring the content in line with the
content in the git repository.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#936033: ITP: pyprof2calltree -- visualise Python cProfile data with this kcachegrind converter

[Nicholas D Steeves]

Package: wnpp
Severity: wishlist
Owner: Nicholas D Steeves 

Package name: pyprof2calltree
Version : 1.4.4
Upstream Author : Peter Waller 
URL : https://github.com/pwaller/pyprof2calltree
License : Expat, MIT, or custom-permissive (needs verification)
Programming Lang: Python
Description : visualise Python cProfile data with this kcachegrind converter

 Pyprof2calltree converts cProfile data into a format that is
 consumable by kcachegrind and qcachegrind for graphical calltree
 analysis.  This combination provides similar capabilities to Snakeviz
 or RunSnakeRun.
 .
 Pyprof2calltree is an adaptation of lsprofcalltree.py, by David
 Allouche, Jp Calderone, Itamar Shtull-Trauring, and Johan Dahlin.  It
 has been adapted to behave more like scripts in the
 kcachegrind-converters package.  One of the authors' objectives is
 for pyprof2calltree to become part of the official upstream kdesdk
 package.
 .
 This package installs the library for Python 3.

I am packaging this because of the cProfile visualisers Elpy (Emacs
IDE for Python) supports: one displays in a browser, RunSnakeRune is
Python 2 only, which leaves this package.  IMHO it's the most
desirable, but I have a KDE bias.

As I'm already on the QT/KDE Team and upstream intents to eventually
merge it into kdesdk, I believe the KDE Extras project is probably the
most appropriate place for it.  I will need a sponsor for the initial
upload.


Regards,
Nicholas

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Joerg Jaspert writes ("Re: tag2upload service architecture and risk assessment 
- draft v2"):
> First off: I, for personal reasons, am a bit detached right now with
> anything Debian (though that should change soon). For that reason, I
> haven't read most of the mail threads, though i skimmed over this
> one a bit.

Hi.  Thanks for your attention.

> I currently do not have too deep a thought on how good their
> implementation is. Just one thing I've seen picked at multiple times,
> and in different places: The current implementation appears to move away
> the final integrity check linking an upload to a person away from the
> archive software to some other.

I don't think that's really true, although I can see why you might
think it from reading the thead.  I will try to explain.

> Thats a no-go.
> 
> Note: I do not say it must be "a dsc" "a git commit" or "a something"
> that is used for this check. That is an implementation detail. But the
> final check/link of an upload with a maintainer(s key) has to be "in"
> the archive. Systems before it can *additionally* do any number of them,
> but the final one is in dak.

In my proposal, as soon as dak supports it, each upload from the
tag2upload service would include a copy of the original uploader's
signed git tag object.

This would allow dak to verify the identity of the original uploader,
and the uploader's intended source package, version, suite, and
distribution; and see that they intended to use the tag2upload
service.  All of that is covered by the original uploader's signature
on the git tag object and can be checked very simply without further
data.  Based on what's been said in the thread I imagine that dak
would want to check all of these things.

The difficulty is that it is complicated to verify the *contents* of
the source package.  (It wasn't clear to me, from your mail, whether
that is something you think is critical for dak to do.)

It would be *possible* for dak to verify the contents, since the .dscs
generated by the tag2upload service are reproducible.  But it's not
easy.

Firstly, it depends on additional data: the referenced git objects
have to be obtained.  (There is at least a designated place to get
them, so finding them is not a problem.)

Secondly, it is a complicated process, because it is reproducing the
complicated mapping between maintainer git branches (which may be in a
variety of forms - see my git packaging practices survey [1]) and
uploaded .dscs (which must be in a standard form).

So to do that reproduction dak would have to effectively have another
instance of the tag2upload infrastructure within it.  Overall I don't
think this would make sense, even though it's possible.

I hope that clarifies things.

Thanks,
Ian.

[1] https://wiki.debian.org/GitPackagingSurvey
  Not all of these are supported by tag2upload yet, but the most
  common ones are.  The biggest one which is missing is the monorepos.
  They present additional difficulties because the maintainer tag name
  must include the package name, so can't conform to standard gbp or
  DEP-14 practice.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: tag2upload service architecture and risk assessment - draft v2

[Holger Levsen]

On Wed, Aug 28, 2019 at 05:07:00PM +0100, Ian Jackson wrote:
> In my proposal the source package is reproducible (in the
> "reproducible builds" sense) from the uploader's signed git tag.  
 
i'm confused. 'reproducible builds' is about creating bit by bit
identical binaries from a given source.

if you are talking about re-creating bit by bit identical source
packages, that's fine, but nothing within the scope of reproducible
builds.

also, as a side note, we have tried to reproduce bit by bit identical
source packages, failed and moved on. it didnt seem trival when we
tried.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Holger Levsen writes ("Re: tag2upload service architecture and risk assessment 
- draft v2"):
> On Wed, Aug 28, 2019 at 05:07:00PM +0100, Ian Jackson wrote:
> > In my proposal the source package is reproducible (in the
> > "reproducible builds" sense) from the uploader's signed git tag.  
>  
> i'm confused. 'reproducible builds' is about creating bit by bit
> identical binaries from a given source.
> 
> if you are talking about re-creating bit by bit identical source
> packages, that's fine, but nothing within the scope of reproducible
> builds.

Sorry for the confusion.  When I wrote

  reproducible (in the "reproducible builds" sense)

I wasn't saying that this is somehow part of, or within the scope of,
the reproducible builds project.  I was just clarifying what the word
"reproducible" meant in my sentence: I am using the word
"reproducible" the same way that the reproducible builds project uses
it - ie I am borrowing that definition of reproducible.  (That's what
"X (in the Y sense)" means.)

I was indeed clarifying that I do mean bit-by-bit identical.  In this
case, bit-by-bit identical dsc (apart from the signature of course),
from (i) git tag (ii) _source.buildinfo containing tools versions etc.

> also, as a side note, we have tried to reproduce bit by bit identical
> source packages, failed and moved on. it didnt seem trival when we
> tried.

I remember some of those discussions.  I'm pretty sure it's possible
in my context, although there are as you say some difficulties with it
in the wider reproducible builds context.

I hope that helps.

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Theodore Y. Ts'o writes ("Re: Git Packaging: Native source formats"):
> Or if we end up moving to dgit for everything, and we don't want to
> use pristine-tar (which I like, but I realize that's not an opinion
> shared by everyone; some people seem to hate it), and upstream uses a
> non-git repo (say, bzr, or hg) and still uses signed tar.gz files, I'd
> argue we need to have a good way to reserve the cryptographic
> signature of upstream's foo.tar.gz and foo.tar.gz.asc in a dgit-only
> world.

I think dgit ought to be compatible with the idea of shipping
upstream's .asc's about, but maybe there are bugs.  I don't ever do
this so I don't know if it works and I doubt there are tests for it.

So, if you have a package where you want to use dgit push and you find
the upstream .asc is not being included, please file bug(s).

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Sam Hartman writes ("Git Packaging: Native source formats"):
> Internet is faster and disks are cheaper.
> I assert this is much less of a concern than it used to be.

We have some extremely large packages.  Also we have users in places
where internet is slow and/or expensive.

Even for medium-sized packages, it is not desirable to constantly ship
copies of the source code about.  I want to be able to upload from a
train wifi, or when roaming abroad on hideously expensive mobile data.

Having said that, I do have some packages which have an upstream where
I don't bother with orig tarballs and just use a native format.

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.
> You don't need to synthesize a separate upstream tarball.

If you already don't care about bit-identical upstream tarballs, then
dealing with these tarballs is a reasonably well-solved problem.
git-deborig etc. FTW.

The real cost of non-native packages is not the few commands you have
to type for your own packages.  It's the cognitive and codebase cost
of knowing how to deal with them, particularly when you're not the
maintainer.  Especially, `3.0 (quilt)' is very complex.  So we would
gain something if we could get rid of non-native packages *entirely*.
But we can't.

Note that many maintainers use a bare debian ("packaging only")
packaging workflow which doesn't work (at least with current tooling)
without orig tarballs.  You'll have seen messages where some of those
people say they think that is the only tolerable approach.

> I'd like to understand the downsides to doing this:

You have been a bit vague about what exactly you are proposing.  If
you are just proposing that native packages should be a strong
recommendation then the benefits are IMO too weak.

OTOH if people feel they "shouldn't" use a native source format for a
small package for some reason, I think that would be worth fixing.

One thing I would like to see changed in this area: some of our tools
issue warnings about packages with a non-native version, but a native
source format.  I think there is nothing wrong with this and it should
not be discouraged and doesn't merit a warning.  (There is an
awkardness here in that you can sometimes unintentionally generate a
native format source package if your origs are missing...)

HTH.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#936043: ITP: gitbatch -- Manage git repositories in one place

[Dawid Dziurla]

Package: wnpp
Severity: wishlist
Owner: Dawid Dziurla 

* Package name: gitbatch
  Version : 0.5.0-1
  Upstream Author : Ibrahim Serdar Acikgoz
* URL : https://github.com/isacikgoz/gitbatch
* License : Expat
  Programming Lang: Go
  Description : Manage git repositories in one place

 Managing multiple git repositories is easier than ever. Often one would end
 up working on many directories and manually pulling updates etc. To make
 this routine faster, gitbatch was created, a simple tool to handle this job.
 Although the focus is batch jobs, one can still do de facto micro management of
 git repositories (e.g add/reset, stash, commit etc.)

Useful tool for managing multiple git repositiories at once.
Does not need additional dependencies, only those already in archive.

Re: Git Packaging: Native source formats

[Simon Richter]

Hi,

On Wed, Aug 28, 2019 at 04:00:10PM -0400, Sam Hartman wrote:

> Back in the day, one of the big reasons for separating .orig.tar.gz from
> .diff.gz was to reuse upstream tarballs for space reasons, both in terms
> of space on mirrors when the pool had two Debian revisions with the same
> upstream, as well as to reduce upload time.

Saving space and bandwidth when only changing the packaging was a nice
bonus, but the main reason we use upstream tarballs verbatim is
verifiability (which you also mention later on).

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.
> You don't need to synthesize a separate upstream tarball.

FWIW, I don't think that this is much of an impediment. "git archive"
exists and is deterministic, so if we can also compress the archive in a
reproducible way, then the "upstream" tarball will be identical.

> It seems particularly attractive when upstream doesn't produce tarballs
> and instead does their development in git.

Lots of upstream projects use git and still produce tarballs for releases.

What we don't have a good workflow for is upstream software that does not
have distinct releases, and frankly, the technical aspect of that is the
smallest part of the problem.

We already have lots of software in the archive that upstream will not
support, essentially telling users to update to the most recent version.

Unless the package maintainer is willing to support the users themselves in
that case, I would say that including that software in a stable release is
not beneficial for users, and making it easier for Debian contributors to
package this software is not going to improve things.

Not everything needs to be packaged, and even less needs to be included in
releases. As a maintainer, it is a completely valid technical decision to
file an RC bug on your own package "this should not go into the release",
and the stable release managers will not ask you to close the bug so the
package can be included.

We need to work together with upstream to identify a state of the software
that we feel comfortable supporting (also together with upstream) for the
lifetime of a Debian release.

No change in the packaging workflow will make this part any easier.

>   It's clear that we value integrity of upstream source.  That is we
>   want to make it easy for people to start from some upstream source
>   that is trusted because upstream has produced it and audit just our
>   changes.
>   One way to do this is with an upstream tarball and a diff (or set of
>   diffs or a debian directory).

>   But if we're thinking that people will be working in Git, another way
>   to do this is to merge in a signed upstream git tag.  Then you can
>   perform a diff against that git tag.

That requires speaking a complex protocol in which a program that is not
expecting malicious input repeatedly parses and decompresses untrusted data
before it is able to verify any signatures. It also assumes availability of
unfiltered Internet services over which cryptographic signatures can be
exchanged.

   Simon

Re: dput problem: Ancient sha256sum?

[Peter Pentchev]

On Thu, Aug 29, 2019 at 03:49:57AM +0100, Ben Hutchings wrote:
> On Thu, 2019-08-29 at 00:33 +0200, dettus wrote:
> > So, I am trying to make a package out of my awesome project dMagnetic.
> > I applied some patches, but unfortunately, now I am getting some errors.
> > After each attempt to dput something, I get the following Email:
> > 
> > 
> > What happened here? How can I fix it?
> 
> I would guess that you previously uploaded dmagnetic version 0.17-1 and
> now you are uploading (say) 0.17-2, with a repacked orig tarball.
> 
> Since the upstream version is the same, the archive will use the same
> orig tarball for them - you cannot replace it.  So, you need to build
> your source package using the same orig tarball as before.
> 
> Ben

...or, since this is mentors.d.n, you (dettus) could try to log into
your account there, go to "Your packages", find the upload there and
remove it. After that, try the dput again.

Actually, no, don't do that. Since dmagnetic is a package that is not in
Debian yet, the first version to be uploaded must have a -1 revision;
rebuild your package as 0.17-1 and try to upload it again.  Then
mentors.d.n will recognize that you are trying to upload a new version
of the same package and (unlike the actual Debian archive) will allow
you to replace the original upstream tarball.

G'luck,
Peter

> > Hello,
> > 
> > Unfortunately your package "dmagnetic" was rejected because of the 
> > following
> > reason:
> > 
> > dmagnetic dsc reference dmagnetic_0.17.orig.tar.gz, but the file differs:
> > in dsc: 82c115920b5570e1e33c613b008736086db210bc8f9b2c2e75f970e9696d8ec5
> > found: 64ca6eeb1d646e9992b134b4c89c7b0da5d2f9e141d03ffe29ff76729c4a4975
> > 
> > Please, rebuild your package against the correct file.
> > 
> > Please try to fix it and re-upload. Thanks,
> > 
> > -- mentors.debian.net
> > (The sha256sum in the .dsc file is correct, btw.)

-- 
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: dput problem: Ancient sha256sum?

[Mattia Rizzolo]

That said, the fact that your .orig.tar.gz changed is an indicator that you
are doing something fishy and you should double check your workflow.  You
should not be able to accidentally end up with a different .orig.tar.gz.

On Thu, 29 Aug 2019, 3:18 pm Peter Pentchev,  wrote:

> On Thu, Aug 29, 2019 at 03:49:57AM +0100, Ben Hutchings wrote:
> > On Thu, 2019-08-29 at 00:33 +0200, dettus wrote:
> > > So, I am trying to make a package out of my awesome project dMagnetic.
> > > I applied some patches, but unfortunately, now I am getting some
> errors.
> > > After each attempt to dput something, I get the following Email:
> > >
> > >
> > > What happened here? How can I fix it?
> >
> > I would guess that you previously uploaded dmagnetic version 0.17-1 and
> > now you are uploading (say) 0.17-2, with a repacked orig tarball.
> >
> > Since the upstream version is the same, the archive will use the same
> > orig tarball for them - you cannot replace it.  So, you need to build
> > your source package using the same orig tarball as before.
> >
> > Ben
>
> ...or, since this is mentors.d.n, you (dettus) could try to log into
> your account there, go to "Your packages", find the upload there and
> remove it. After that, try the dput again.
>
> Actually, no, don't do that. Since dmagnetic is a package that is not in
> Debian yet, the first version to be uploaded must have a -1 revision;
> rebuild your package as 0.17-1 and try to upload it again.  Then
> mentors.d.n will recognize that you are trying to upload a new version
> of the same package and (unlike the actual Debian archive) will allow
> you to replace the original upstream tarball.
>
> G'luck,
> Peter
>
> > > Hello,
> > >
> > > Unfortunately your package "dmagnetic" was rejected because of the
> > > following
> > > reason:
> > >
> > > dmagnetic dsc reference dmagnetic_0.17.orig.tar.gz, but the file
> differs:
> > > in dsc:
> 82c115920b5570e1e33c613b008736086db210bc8f9b2c2e75f970e9696d8ec5
> > > found: 64ca6eeb1d646e9992b134b4c89c7b0da5d2f9e141d03ffe29ff76729c4a4975
> > >
> > > Please, rebuild your package against the correct file.
> > >
> > > Please try to fix it and re-upload. Thanks,
> > >
> > > -- mentors.debian.net
> > > (The sha256sum in the .dsc file is correct, btw.)
>
> --
> Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} p...@storpool.com
> PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
> Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13
>

Re: Git Packaging: Native source formats

[Simon McVittie]

On Thu, 29 Aug 2019 at 11:56:55 +0100, Ian Jackson wrote:
> If you already don't care about bit-identical upstream tarballs, then
> dealing with these tarballs is a reasonably well-solved problem.
> git-deborig etc. FTW.

I think it's important to distinguish between the two things that you
might mean by "bit-identical upstream tarballs": you might not care
whether the orig tarball is bit-identical to upstream's official release
artifact (assuming they have one), but you still have to care about
the orig tarball being bit-identical to any orig tarball of the same
name that might previously have been uploaded to the Debian archive,
because otherwise the archive will reject your uploads.

Does/can git-deborig guarantee that a given git commit will always
produce the same tarball?

> One thing I would like to see changed in this area: some of our tools
> issue warnings about packages with a non-native version, but a native
> source format.  I think there is nothing wrong with this and it should
> not be discouraged and doesn't merit a warning.  (There is an
> awkardness here in that you can sometimes unintentionally generate a
> native format source package if your origs are missing...)

I think that last point is a significant reason to want the nativeness
of the source package to match the nativeness of the version number: for
version 1.0 source packages, detecting a non-native version with a native
source format is the only way to generate any sort of warning about this.

Unlike 1.0 (non-native) vs. 3.0 (quilt), where some people prefer one and
some people prefer the other, I am not aware of any advantages of 1.0
(native) over 3.0 (native). If 3.0 (native) is indeed strictly better
than 1.0 (native), perhaps it would be reasonable to say that packages
that intentionally have a non-native version number but a native
source format should declare this explicitly, by using "3.0 (native)"
in d/source/format? That way, if a version 1.0 source package has a
non-native version number, tools can assume that it was meant to have
a .orig, and issue warnings; conversely, if a source package with a
non-native version number explicitly has "3.0 (native)" format, tools
could assume that the maintainer wants what they asked for.

smcv

Re: Git Packaging: Native source formats

[Milan Kupcevic]

On 8/28/19 4:00 PM, Sam Hartman wrote:
> 
> Back in the day, one of the big reasons for separating .orig.tar.gz from
> .diff.gz was to reuse upstream tarballs for space reasons, both in terms
> of space on mirrors when the pool had two Debian revisions with the same
> upstream, as well as to reduce upload time.
> 

Besides the clear separation of upstream content and downstream tweaks
an important quality of this concept is immutability of the upstream
code that has already been obtained, inspected, signed by a DD/DM and
subsequently deposited into Debian archive.

I've seen cases where obtaining an upstream tarball from an official
upstream website at two different points in time results in slightly
different content. I've also seen developers deleting a git tag and then
creating a new git tag using exactly the same name/release number
pointing to different commit. And finally downloading a dynamically
created tarball based on a git tag does not mean you'll get exactly the
same content every time either because the currently running
archiving/compressing tools have changed or because of already mentioned
human factors.

Git commit hash is somewhat more reliable than a git tag but reliance on
SHA-1 hash has also been mentioned as a step backward as Debian has
moved on to a more advanced hash algorithm. These and other possibly
unforeseen problems make the concept of immutable upstream code
deposited in the Debian archive very appealing and surely more reliable
compared to the Git tag method of sourcing.

Milan



signature.asc
Description: OpenPGP digital signature

Re: dput problem: Ancient sha256sum? [and 1 more messages]

[Ian Jackson]

Mattia Rizzolo writes ("Re: dput problem: Ancient sha256sum?"):
> That said, the fact that your .orig.tar.gz changed is an indicator
> that you are doing something fishy and you should double check your
> workflow.  You should not be able to accidentally end up with a
> different .orig.tar.gz.

I agree with this.  That is, Thomas, I would advise you to try to
understand what you did differently to generate the two different orig
tarballs.

When you know why they are different you will know which one you want.
If it's the old one, download it from mentors and arrange for your
re-upload to use it.  If it's the new one, delete your upload as Peter
Pentchev suggests and then you can upload your new .orig.

I do have to disagree with something Peter writes:

Peter Pentchev writes ("Re: dput problem: Ancient sha256sum?"):
> [...]   Since dmagnetic is a package that is not in
> Debian yet, the first version to be uploaded must have a -1 revision;
> rebuild your package as 0.17-1 and try to upload it again.  Then
> mentors.d.n will recognize that you are trying to upload a new version
> of the same package and (unlike the actual Debian archive) will allow
> you to replace the original upstream tarball.

As far as I'm aware there is no requirement for the first version of a
package uploaded to Debian to have a -1 revision.  There is certainly
no general requirement that first upload of any particular upstream
version has to have a -1 revision, or even that it must have a -1 or
-0.1 revision.

I think the practice of reusing version numbers for different packages
is confusing.  It leads to mistakes, and should be deprecated.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Simon McVittie writes ("Re: Git Packaging: Native source formats"):
> On Thu, 29 Aug 2019 at 11:56:55 +0100, Ian Jackson wrote:
> > If you already don't care about bit-identical upstream tarballs, then
> > dealing with these tarballs is a reasonably well-solved problem.
> > git-deborig etc. FTW.
> 
> I think it's important to distinguish between the two things that you
> might mean by "bit-identical upstream tarballs": you might not care
> whether the orig tarball is bit-identical to upstream's official release
> artifact (assuming they have one), but you still have to care about
> the orig tarball being bit-identical to any orig tarball of the same
> name that might previously have been uploaded to the Debian archive,
> because otherwise the archive will reject your uploads.

Good point.  I meant "bit-identical to upstream".

> Does/can git-deborig guarantee that a given git commit will always
> produce the same tarball?

No, it doesn't.  But it uses git-archive and in practice that nearly
always gives you the same tarball - even though that's not guaranteed.
I haven't seen any trouble (but I normally use `dgit clone' or `dgit
fetch' and that arranges for any existing origs in the archive to be
in the right place on my laptop, so perhaps that means I avoid trouble
that others see).

> I think that last point is a significant reason to want the nativeness
> of the source package to match the nativeness of the version number: for
> version 1.0 source packages, detecting a non-native version with a native
> source format is the only way to generate any sort of warning about this.

This is a good point.  Of course it's only a warning and warnings are
often ignored when you're in a hurry.

> Unlike 1.0 (non-native) vs. 3.0 (quilt), where some people prefer one and
> some people prefer the other, I am not aware of any advantages of 1.0
> (native) over 3.0 (native). If 3.0 (native) is indeed strictly better
> than 1.0 (native), perhaps it would be reasonable to say that packages
> that intentionally have a non-native version number but a native
> source format should declare this explicitly, by using "3.0 (native)"
> in d/source/format? That way, if a version 1.0 source package has a
> non-native version number, tools can assume that it was meant to have
> a .orig, and issue warnings; conversely, if a source package with a
> non-native version number explicitly has "3.0 (native)" format, tools
> could assume that the maintainer wants what they asked for.

Perhaps.  I have a vague feeling that there might be (or have once
been?) some reason to prefer 1.0 (native) to 3.0 (native) but I can't
bring it to mind, and now that I try to think about it it's all just
fog.  Maybe I am remembering some years-old abundance of caution.

Sean and I faced an aspect of this problem with tag2upload and 1.0
source format.  (We didn't want to just say 1.0 wasn't supported.)
The information in git needs to specify whether the output is supposed
to be a native package.  Obviously git-debpush shouldn't look for
tarballs in your `..', like dpkg-source does.  So we ask the users of
format 1.0 source packages to add an appropriate setting to
debian/source/options.

Anyway, it seems to me that if debian/source/format says `3.0
(native)' then the warning is inappropriate.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Milan Kupcevic writes ("Re: Git Packaging: Native source formats"):
> I've also seen developers deleting a git tag and then creating a new
> git tag using exactly the same name/release number pointing to
> different commit.

It is possible to avoid some of these problems by using a git server
which does not permit tag rewriting and/or branch rewinding.

The dgit git server has these properties: once you have done `dgit
push' and the tag is on the dgit git server, it can't be deleted or
changed.  Likewise, the suite branches that you push to are
fast-forwarding.  Breaching these rules would involve administrator
intervention; as the administrator I would expect to do that only in
very rare situations, where we need to rewrite history eg to remove
legally dangerous objects.

(Because of the archive's policy about new packages, which permits
reuse of version numbers, there is a corresponding exception for
packages which are still entirely in NEW.)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Ian Jackson]

Ian Jackson writes ("Re: Git Packaging: Native source formats"):
> Simon McVittie writes ("Re: Git Packaging: Native source formats"):
> > Unlike 1.0 (non-native) vs. 3.0 (quilt), where some people prefer one and
> > some people prefer the other, I am not aware of any advantages of 1.0
> > (native) over 3.0 (native). If 3.0 (native) is indeed strictly better
> > than 1.0 (native), perhaps it would be reasonable to say that packages
> > that intentionally have a non-native version number but a native
> > source format should declare this explicitly, by using "3.0 (native)"
> > in d/source/format? That way, if a version 1.0 source package has a
> > non-native version number, tools can assume that it was meant to have
> > a .orig, and issue warnings; conversely, if a source package with a
> > non-native version number explicitly has "3.0 (native)" format, tools
> > could assume that the maintainer wants what they asked for.
> 
> Perhaps.  I have a vague feeling that there might be (or have once
> been?) some reason to prefer 1.0 (native) to 3.0 (native) but I can't
> bring it to mind, and now that I try to think about it it's all just
> fog.  Maybe I am remembering some years-old abundance of caution.

Private email prompted me to check something, and I was right to
remember that there was a difficulty.  dpkg-source refuses to create a
`3.0 (native)' package with a Debian revision number:

$ dpkg-source  -b .
dpkg-source: error: can't build with source format '3.0 (native)': native 
package version may not have a revision
$

It looks like the dpkg maintainers are adamant that dpkg-source is
correct.  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=737634

So `3.0 (native)' is not strictly better than 1.0.  dpkg-source
refuses to work in the situation where I am saying (and you seem to be
agreeing) that it shouldn't even print a warning ...

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Git Packaging: Native source formats

[Simon McVittie]

On Thu, 29 Aug 2019 at 16:25:35 +0100, Ian Jackson wrote:
> Simon McVittie writes ("Re: Git Packaging: Native source formats"):
> > for
> > version 1.0 source packages, detecting a non-native version with a native
> > source format is the only way to generate any sort of warning about this.
> 
> This is a good point.  Of course it's only a warning and warnings are
> often ignored when you're in a hurry.

Sure, but it's better than nothing.

I was half tempted to suggest making the tuple (version 1.0 source,
non-native version number, native source format) into a fatal error,
but obviously that's a compatibility break - but if there's no reason
to prefer version 1.0 over version 3.0 (native), then maybe it's a
compat break that would be worth it to get rid of one of the traps
inherent in format 1.0 implicitly choosing between native and non-native?

> So we ask the users of
> format 1.0 source packages to add an appropriate setting to
> debian/source/options.
>
> Anyway, it seems to me that if debian/source/format says `3.0
> (native)' then the warning is inappropriate.

Maybe the setting you say is appropriate should also silence such
warnings. Do you know which tools produce them? The Lintian maintainers
are generally receptive to patches for this sort of thing.

smcv

Bug#936075: ITP: fortran-language-server -- Fortran Language Server for the Language Server Protocol

[Denis Danilov]

Package: wnpp
Severity: wishlist
Owner: Denis Danilov 

* Package name: fortran-language-server
  Version : 1.10.2
  Upstream Author : Chris Hansen
* URL : https://github.com/hansec/fortran-language-server
* License : MIT
  Programming Lang: Python
  Description : Fortran Language Server for the Language Server Protocol

Fortran Language Server (fortls) is an implementation of the Language Server
Protocol.  It can be used with editors that supports the protocol to offer
support for code completion and documentation.

Supported LSP features include:
* Document symbols (textDocument/documentSymbol)
* Auto-complete (textDocument/completion)
* Signature help (textDocument/signatureHelp)
* GoTo/Peek definition (textDocument/definition)
* Hover (textDocument/hover)
* GoTo implementation (textDocument/implementation)
* Find/Peek references (textDocument/references)
* Project-wide symbol search (workspace/symbol)
* Symbol renaming (textDocument/rename)
* Documentation parsing (Doxygen and FORD styles)
* Diagnostics

Re: Git Packaging: Native source formats

[Philipp Kern]

On 8/29/2019 8:32 PM, Andrej Shadura wrote:
>> So `3.0 (native)' is not strictly better than 1.0.  dpkg-source
>> refuses to work in the situation where I am saying (and you seem to be
>> agreeing) that it shouldn't even print a warning ...
> 
> I have to disagree with you but I consider this strictly an
> improvement. Allowing native packages with non-native versions
> significantly increases complexity of code handling Debian source
> packages. Not even all Debian tools support this case; arguably it
> should not be supported at all as often leads to malformed packages
> being uploaded to the archive.

While this may be true on some level, it is also important to be able to
build packages from checked-out source trees (say, git repositories)
without an original source present.

For instance at work we check in whole Debian packages as-is (including
their non-native version) to fork and then modify them. Changing the
versioning scheme is pretty disruptive there. For people unfamiliar to
Debian the diff is already represented in the VCS and there is no
technical need to have this conveyed in the intermediate source package
representation that is only needed to feed the build to the build system.

Of course one workaround is to always build from the build tree and to
always specify -b/-B and never build a source package at all.
Unfortunately the various defaults in Debian's toolchain don't make that
as easy as it should be. Some can be addressed through wrapper scripts,
but then it's odd to anyone familiar with Debian.

Obviously I'm not bound to that format being "3.0 (native)" but some
"3.0 (dumb)" that just tars up the whole tree without caring about the
version scheme would then be nice to have as a replacement. ;-)

Kind regards
Philipp Kern

Re: Git Packaging: Native source formats

[Sune Vuorela]

On 2019-08-28, Sam Hartman  wrote:
> * I've heard at least one person claim that native format packages are
>   problematic for downstreams.
>   I'd like to better understand both the theoretical argument here and
>   to understand from downstreams whether it is an issue in practice.
>
>   For downstreams where it is a problem, are you using a git or a
>   non-git workflow?

I've recently (professionally) been involved with a .. sidestream?.
Where going into debian packages, rpm packages and others to fetch their
fixes (as separate files) into a different environment (yocto).

Having the ability to extract separate patches - with clear
documentation - was so much simpler than trying to extract it from
anywhere else.


/Sune

Re: Git Packaging: Native source formats

[Russ Allbery]

Sam Hartman  writes:

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream versions are the same.  You don't need to synthesize a separate
> upstream tarball.

> It seems particularly attractive when upstream doesn't produce tarballs
> and instead does their development in git.

> I'd like to understand the downsides to doing this:

While this is not an argument against *ever* using 3.0 (native) or some
equivalent when packaging upstream software, I have found it to help
relationships with upstream considerably in the past to represent the
package as their software as released plus a set of Debian patches.  For
at least some upstreams, this is a very natural workflow; they can pull
patches easily at their leisure without requiring the Debian package
maintainer to keep updating a rebased patch submission.

Even with upstreams that switch to a PR workflow, sometimes it takes a
while to convince upstreams to even look at patches, and having a
cleanly-separated patch in the packaging makes it very easy to submit a PR
when upstream is ready.

There are obviously ways to represent this with a pure Git repository, but
apart from using a patch system on top of 3.0 (native), at which point I
don't understand why one wouldn't just use 3.0 (quilt), they require
multiple branches and thus aren't available directly in the archive.

Extracting specific changes by comparing only two Git branches with a
complex merge history is certainly possible to do with native Git tools,
but I would classify it as an advanced Git skill.  I think there are a lot
of upstreams using Git for whom that operation would still be quite
challenging.

There's also a psychological value for some upstreams to representing all
changes to upstream as patches.  If they're worried Debian is doing some
weird nonsense (from their perspective) to their package, they can look at
debian/rules for build flags and then at debian/patches and see exactly
what we've done, and feel assured that the rest of the package is exactly
as they released it.  That's easy for them to do; downloading the full
package and then diffing against their tree is more awkward and can't be
done with some casual web browsing.

This is one of those cases where knowing your upstream is invaluable.  A
lot of upstreams won't care in the slightest, some upstreams are Git
experts and know exactly how to get the data they care about, and others
are still using Subversion or even CVS and will find anything related to
Git opaque and frustrating.  Some upstreams actively encourage downstream
modifications and will seek them out and incorporate them, some upstreams
are happily oblivious and only look at patches or PRs submitted to them,
and some upstreams will get actively upset at what they view as
unmotivated or incompatible changes and may require delicate handling
(part of which, in my experience, is effortless transparency so that they
don't feel like anything is being hidden from them).

-- 
Russ Allbery (r...@debian.org)   

Re: Git Packaging: Native source formats

[Thomas Goirand]

On 8/29/19 1:49 AM, Theodore Y. Ts'o wrote:
> Or if we end up moving to dgit for everything, and we don't want to
> use pristine-tar (which I like, but I realize that's not an opinion
> shared by everyone; some people seem to hate it

/me raises hand. I hate it.

Pristine-tar forces you to have multiple branches when you may only need
a single one. It's also not reliable and may easily generate different
tarballs for the same tag, which defeats its purpose (and no, the
workaround to remove all timestamps and order files isn't acceptable).

Not only that. If upstream uses git, then I just fetch from it, and use
the upstream tag as reference to run "git archive", which does the job
very well.

Now, you're talking about upstream using bzr or hg. These are the very
few minority (and counting...). We may as well get rid of hg and bzr in
Debian if it doesn't get fixed so it uses Python 3 only... (well, I
guess someone will wake up and do the work, so this argument doesn't
count...).

Cheers,

Thomas Goirand (zigo)

Work-needing packages report for Aug 30, 2019

[wnpp]

The following is a listing of packages for which help has been requested
through the WNPP (Work-Needing and Prospective Packages) system in the
last week.

Total number of orphaned packages: 1350 (new: 1)
Total number of packages offered up for adoption: 159 (new: 1)
Total number of packages requested help for: 53 (new: 0)

Please refer to https://www.debian.org/devel/wnpp/ for more information.



The following packages have been orphaned:

   viewvc (#935691), orphaned 4 days ago
 Description: web interface for CVS and/or Subversion repositories
 Reverse Depends: fusionforge-plugin-scmsvn viewvc-query
 Installations reported by Popcon: 122
 Bug Report URL: https://bugs.debian.org/935691

1349 older packages have been omitted from this listing, see
https://www.debian.org/devel/wnpp/orphaned for a complete list.



The following packages have been given up for adoption:

   numba (#935626), offered 5 days ago
 Installations reported by Popcon: 130
 Bug Report URL: https://bugs.debian.org/935626

158 older packages have been omitted from this listing, see
https://www.debian.org/devel/wnpp/rfa_bypackage for a complete list.



For the following packages help is requested:

   autopkgtest (#846328), requested 1002 days ago
 Description: automatic as-installed testing for Debian packages
 Reverse Depends: autodeb-worker debci-worker
 Installations reported by Popcon: 1187
 Bug Report URL: https://bugs.debian.org/846328

   balsa (#642906), requested 2895 days ago
 Description: An e-mail client for GNOME
 Installations reported by Popcon: 763
 Bug Report URL: https://bugs.debian.org/642906

   broadcom-sta (#886599), requested 598 days ago (non-free)
 Description: Broadcom STA Wireless driver (non-free)
 Installations reported by Popcon: 1757
 Bug Report URL: https://bugs.debian.org/886599

   cargo (#860116), requested 870 days ago
 Description: Rust package manager
 Reverse Depends: dh-cargo
 Installations reported by Popcon: 1162
 Bug Report URL: https://bugs.debian.org/860116

   cyrus-sasl2 (#799864), requested 1436 days ago
 Description: authentication abstraction library
 Reverse Depends: 389-ds-base 389-ds-base-legacy-tools
   389-ds-base-libs adcli autofs-ldap cairo-dock-mail-plug-in
   claws-mail claws-mail-acpi-notifier claws-mail-address-keeper
   claws-mail-archiver-plugin (109 more omitted)
 Installations reported by Popcon: 195547
 Bug Report URL: https://bugs.debian.org/799864

   dee (#831388), requested 1140 days ago
 Description: model to synchronize mutiple instances over DBus
 Reverse Depends: dee-tools gir1.2-dee-1.0 libdee-dev zeitgeist-core
 Installations reported by Popcon: 45191
 Bug Report URL: https://bugs.debian.org/831388

   developers-reference (#759995), requested 1825 days ago
 Description: guidelines and information for Debian developers
 Installations reported by Popcon: 7589
 Bug Report URL: https://bugs.debian.org/759995

   devscripts (#800413), requested 1430 days ago
 Description: scripts to make the life of a Debian Package maintainer
   easier
 Reverse Depends: apt-build apt-listdifferences aptfs arriero
   autodeb-worker brz-debian bzr-builddeb customdeb debci
   debian-builder (32 more omitted)
 Installations reported by Popcon: 12146
 Bug Report URL: https://bugs.debian.org/800413

   docker.io (#908868), requested 348 days ago
 Description: Linux container runtime
 Reverse Depends: golang-docker-dev
   golang-github-fsouza-go-dockerclient-dev
   golang-github-google-cadvisor-dev
   golang-github-samalba-dockerclient-dev kubernetes-node subuser
   whalebuilder
 Installations reported by Popcon: 2128
 Bug Report URL: https://bugs.debian.org/908868

   ed (#886643), requested 598 days ago
 Description: classic UNIX line editor
 Reverse Depends: apt-cacher libdebbugs-perl opensmtpd sn
 Installations reported by Popcon: 16409
 Bug Report URL: https://bugs.debian.org/886643

   ejabberd (#767874), requested 1760 days ago
 Description: distributed, fault-tolerant Jabber/XMPP server written
   in Erlang
 Reverse Depends: ejabberd-contrib ejabberd-mod-cron
   ejabberd-mod-log-chat ejabberd-mod-logsession ejabberd-mod-logxml
   ejabberd-mod-message-log ejabberd-mod-muc-log-http
   ejabberd-mod-post-log ejabberd-mod-pottymouth ejabberd-mod-rest (4
   more omitted)
 Installations reported by Popcon: 457
 Bug Report URL: https://bugs.debian.org/767874

   fbcat (#565156), requested 3515 days ago
 Description: framebuffer grabber
 Installations reported by Popcon: 181
 Bug Report URL: https://bugs.debian.org/565

Re: Git Packaging: Native source formats

[Theodore Y. Ts'o]

On Fri, Aug 30, 2019 at 12:29:45AM +0200, Thomas Goirand wrote:
> 
> Pristine-tar forces you to have multiple branches when you may only need
> a single one. It's also not reliable and may easily generate different
> tarballs for the same tag, which defeats its purpose (and no, the
> workaround to remove all timestamps and order files isn't acceptable).

Huh?  When does pristine-tar be unreliable?  At least in my
experience, I can store both the GPG signature and upstream tar file
using pristine tar, and it's never failed to reproduce the exact tar
file such that the GPG signature verifies.

I'm not even sure what you mean by "remove all timestamps and order
files".  The whole point of pristine tar is that you don't have to do
that, and you can preserve the exact binary that was uploaded, and
signed, by the upstream.

> Not only that. If upstream uses git, then I just fetch from it, and use
> the upstream tag as reference to run "git archive", which does the job
> very well.

But not all upstreams use git...

> Now, you're talking about upstream using bzr or hg. These are the very
> few minority (and counting...). We may as well get rid of hg and bzr in
> Debian if it doesn't get fixed so it uses Python 3 only... (well, I
> guess someone will wake up and do the work, so this argument doesn't
> count...).

And if even if they do use git, they may not use signed tags.  Some of
them also do post-processing to generate the tar.gz file which gets
distributed.  (Example: util-linux).  Hence, using "git archive" isn't
a substitute.

I'm arguing that we need to have a sane soluion for those upstreams
that don't use git, or who use git but who don't use signed tags, or
who do post-processing to generate the tar file.  Hopefully you're not
arguing that we should just summarily eject any packages from Debian
which don't use git, and refuse to package any package which isn't
compatible with our new git packaging philosophy?

- Ted

Re: Git Packaging: Native source formats

[Theodore Y. Ts'o]

On Thu, Aug 29, 2019 at 11:23:01AM +0100, Ian Jackson wrote:
> Theodore Y. Ts'o writes ("Re: Git Packaging: Native source formats"):
> > Or if we end up moving to dgit for everything, and we don't want to
> > use pristine-tar (which I like, but I realize that's not an opinion
> > shared by everyone; some people seem to hate it), and upstream uses a
> > non-git repo (say, bzr, or hg) and still uses signed tar.gz files, I'd
> > argue we need to have a good way to reserve the cryptographic
> > signature of upstream's foo.tar.gz and foo.tar.gz.asc in a dgit-only
> > world.
> 
> I think dgit ought to be compatible with the idea of shipping
> upstream's .asc's about, but maybe there are bugs.  I don't ever do
> this so I don't know if it works and I doubt there are tests for it.
> 
> So, if you have a package where you want to use dgit push and you find
> the upstream .asc is not being included, please file bug(s).

The problem I have is that "dgit gbp" doesn't extract the upstream
.asc.  Not a big deal, I use /tmp/gbp for my build directory, and I
manually checkout and populate it with the .asc file.  But building
from "dgit clone" won't generate same package as I do (which includes
the .asc file for the orig.tar.gz file.)

Cheers,

- Ted