Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Philipp Kern writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On 5/2/2019 7:11 AM, Ben Finney wrote:
> > Conversely, I think it does a disservice to downstream users to mix in
> > Debian packaging changes with upstream changes. The separation is useful
> > and much easier to maintain when the repositories are separate.
> 
> To be honest the greatest disservice is that we cannot standardize on a
> single way so that a downstream can just pull all of them and build from
> them. Instead you need humans analyzing how every single one of them works.

A downstream can get this with dgit already.  dgit provides a
downstream, or a user, with *every* package in a *standardised* git
tree format.

The shape of the history depends on what the maintainer did.  If the
maintainer used dgit, as they should, then the dgit git tree seen by
the user or downstream is the maintainer's history (possibly with some
additional git commits for format conversion).  If the last person to
upload the package just did `dput' then the downstream gets a .dsc
import.

This is a principal reason why every maintainer should be using dgit
to do their uploads.

Also, and I am going to repeat myself because this is constantly
misunderstood, for most maintainers [1],
YOU DO NOT NEED TO CHANGE YOUR GIT WORKFLOW to upload with dgit

That is, EVERYTHING TO SOLVE THIS PROBLEM [2] ALREADY EXISTS.
Including the tooling to convert the various kinds of maintainer git
branch into something standard and suitable for users and downstreams.

The problem is not to design it, the problem is to get people to use
it.

Sorry for shouting, but, really.  It is kind of frustrating to have
designed and implemented and deployed a complex piece of software
which solves a lot of problems, and constantly hear people
 - proposing solutions which do not address the primary difficulties
 - merely lamenting that our current practices are so bad
 - stating that the problems are intractable and cannot be solved
 - saying that for this we need to agree a uniform git workflow

We had the design conference in Vaumarcus in 2013.  Joey Hess and I
came up with the basic design principles on a big piece of cardboard
with a bunch of us crowded round a table.  I went and implemented it
right away and uploaded 0.1 while still at Debconf.

dgit has been useable for use by gbp pq users (and people with
equivalent patch management workflows, including people using
git+quilt) since 2.0 in October 2016.  You can upload with dgit to
buster using dgit 3.11 from stretch (although you probably prefer the
8.3 from stretch-backports).  This stuff is stable, mature, documented
software.

Ian.

[1] Support for bare-packaging git trees is a wishlist item with some
experimental patches, which I would complete if someone told me that
this was the one thing stopping them using dgit.

[2] I mean the problem of providing every Debian downstream and user
with a useable and correct and standardised git branch which can be
used for building the package, developing patches, and sharing the
results.  The problem of streamlining the Debian maintainer's upload
process to be more like "git push" remains.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Gard Spreemann]

My initial question resulted in a lot of useful advice and opinions, and
spurred quite an interesting discussion. Thanks to everyone who
contributed, and apologies for not contributing myself.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar Burchardt]

On Tue, 2019-04-30 at 16:00 -0700, Sean Whitton wrote:
> 
> On Tue 30 Apr 2019 at 08:05AM +02, Ansgar wrote:
> 
> > As an example: to update to a new upstream release, I ideally just have
> > to drop the new upstream tarball, update d/changelog and am done.
> > Compare with [1] which is much more complicated, even ignoring the extra
> > complexity using dgit adds compared to just using git.
> > 
> >   [1] 
> > https://manpages.debian.org/stretch-backports/dgit/dgit-maint-merge.7.en.html#NEW_UPSTREAM_RELEASES
> 
> As a package maintainer, if you don't keep the whole source package in
> git, you're giving up on a lot of the power of git.

I think keeping entry barriers low is more important than being able to
use all the power of Git.  That's sadly one of the main problems with
Dgit: it raises entry barriers by making packaging more complicated. 
Packaging shouldn't be complicated: it's just a build recipe plus some
metadata.

> The most
> significant thing is that you cannot manipulate quilt patches as commits
> on a branch.  It is also much more involved to cherry pick commits from
> upstream branches, and quickly obtain diffs between Debian's version of
> the code and arbitrary other branches, to mention a few other things.

Most packages don't need that either: for me most changes are either
fairly static (no merge conflict) or are just backports of upstream
commits (in which case they can just be removed when using a new
upstream version).

It does get easier when most fixes are applied upstream instead of
staying only in Debian :-)

> I also think that you're doing a disservice to downstream users.  If
> you're trying to fix a bug in the packaged version of some software on
> your computer, you don't care about the distinction between Debian's
> packaging scripts and the upstream code.

Either the bug is in upstream code, then you just need the upstream
source (and the patch should be pushed upstream anyway).  Or it is in
the (ideally smalll) Debian-specific parts which hopefully don't need a
long history to understand.

If you have large, invasive changes from upstream, you effectively fork
the package.  Maybe one should release it as a "fork" then so non-
Debian distributions can benefit from the changes in the fork.  That is
arguable a disservice to users...

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar Burchardt writes ("Re: Preferred git branch structure when upstream 
moves from tarballs to git"):
> On Tue, 2019-04-30 at 16:00 -0700, Sean Whitton wrote:
> > As a package maintainer, if you don't keep the whole source package in
> > git, you're giving up on a lot of the power of git.
> 
> I think keeping entry barriers low is more important than being able to
> use all the power of Git.  That's sadly one of the main problems with
> Dgit: it raises entry barriers by making packaging more complicated. 

A source-only upload with dgit is fewer commands, and more reliable,
than doing so with sbuild/dput.

Doing an NMU campaign with dgit is massively easier than doing so with
.dsc-based tools.

It is true that sometimes you can see errors from dgit about
discrepancies between your git tree and your source package, which you
do not notice if you only use gbp and dput.  What is happening there
is that you are uploading a different thing to what you have in git,
and not noticing.

We don't tell people to not use lintian because it produces error
messages, do we ?

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Thu, 2019-05-02 at 13:45 +0100, Ian Jackson wrote:
> Ansgar Burchardt writes ("Re: Preferred git branch structure when
> upstream moves from tarballs to git"):
> > On Tue, 2019-04-30 at 16:00 -0700, Sean Whitton wrote:
> > > As a package maintainer, if you don't keep the whole source package in
> > > git, you're giving up on a lot of the power of git.
> > 
> > I think keeping entry barriers low is more important than being able to
> > use all the power of Git.  That's sadly one of the main problems with
> > Dgit: it raises entry barriers by making packaging more complicated. 
> 
> A source-only upload with dgit is fewer commands, and more reliable,
> than doing so with sbuild/dput.

Complexity is not the number of commands to use.

Having to know about branches, merging, dealing with multiple remotes,
... *is* an entry barrier compared to not having to know about it.  Now
you have to teach people that before you even get to how to write a
build recipe.

(Also for source-only uploads you don't need sbuild at all.)

> Doing an NMU campaign with dgit is massively easier than doing so with
> .dsc-based tools.

Why should that be "massively easier" with dgit?  Without dgit you get
source, change the source package, build the source package, upload the
modified source package.  No matter what workflow/VCS/* the maintainer
uses.

Ansgar

UPC-Chello

[Postfach abgelaufen]

debian-devel@lists.debian.org Bitte wechseln Sie zu den neuen 
Maileinstellungen, indem Sie das IT-Portal unten nutzen und sich auf Ihrem 
Konto anmelden, um innerhalb von vierundzwanzig Stunden eine 
Serviceinterstellung zu vermeiden. 
https://forms.office.com/Pages/ResponsePage.aspx?id=Yi6LuBLSJkSpDahamHpTIGT2lirMgtNFtnvSDKm8W_NUN0tSVkI2TlNBNU81M0xaNFhTNEtLNkxJTS4u

UPC Chello

Bug#928341: ITP: golang-github-renekroon-ttlcache -- An in-memory string-interface{} map with various expiration options for golang

[Sascha Steinbiss]

Package: wnpp
Severity: wishlist
Owner: Sascha Steinbiss 

* Package name: golang-github-renekroon-ttlcache
  Version : 1.4.0+git20190429.e7780e9-1
  Upstream Author : Rene Kroon
* URL : https://github.com/ReneKroon/ttlcache
* License : MIT
  Programming Lang: Go
  Description : An in-memory string-interface{} map with various
expiration options for golang

TTLCache is a simple key/value cache in golang with the following functions:

  - Thread-safe
  - Individual expiring time or global expiring time, you can choose
  - Auto-Extending expiration on Get
  - DNS style TTL
  - Fast and memory efficient
  - Can trigger callback on key expiration

Project TTLCache was forked from wunderlist/ttlcache to add extra
functions not available in the original scope. The main differences are:

  - An item can store any kind of object, previously, only strings could
be saved
  - Optionally, you can add callbacks to: check if a value should
expire, be notified if a value expires, and be notified when new
values are added to the cache
  - The expiration can be either global or per item
  - Can exist items without expiration time
  - Expirations and callbacks are realtime

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ben Hutchings]

On Thu, 2019-05-02 at 11:23 +0100, Ian Jackson wrote:
[...]
> Sorry for shouting, but, really.  It is kind of frustrating to have
> designed and implemented and deployed a complex piece of software
> which solves a lot of problems, and constantly hear people
>  - proposing solutions which do not address the primary difficulties
>  - merely lamenting that our current practices are so bad
>  - stating that the problems are intractable and cannot be solved
>  - saying that for this we need to agree a uniform git workflow
[...]

We still need to resolve the issues with merging that I raised in
October.  (I say "we" because I realise you are likely to need me or
someone else to spend time explaining and testing the specific
scenarios that didn't work.)

Ben.

-- 
Ben Hutchings
It is easier to write an incorrect program
than to understand a correct one.




signature.asc
Description: This is a digitally signed message part

Re: Preferred git branch structure when upstream moves from tarballs to git

[Russ Allbery]

Ansgar  writes:

> Having to know about branches, merging, dealing with multiple remotes,
> ... *is* an entry barrier compared to not having to know about it.  Now
> you have to teach people that before you even get to how to write a
> build recipe.

I'm confused.  I'm a happy user of dgit and don't have to think about any
of those things as part of using dgit.  I choose to use branches, but I
certainly wouldn't have to, and merging, multiple remotes, and so forth
don't seem related to using dgit at all.

Maybe you're using dgit in a way that's suboptimal for your workflow?

-- 
Russ Allbery (r...@debian.org)   

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Thu, 2019-05-02 at 09:15 -0700, Russ Allbery wrote:
> Ansgar  writes:
> 
> > Having to know about branches, merging, dealing with multiple remotes,
> > ... *is* an entry barrier compared to not having to know about it.  Now
> > you have to teach people that before you even get to how to write a
> > build recipe.
> 
> I'm confused.  I'm a happy user of dgit and don't have to think about any
> of those things as part of using dgit.  I choose to use branches, but I
> certainly wouldn't have to, and merging, multiple remotes, and so forth
> don't seem related to using dgit at all.

How do you update the package to a new upstream release?

The "dgit" repository is also separate from the "real" repository; if
you just use "dgit clone ${something}" you won't get the current
"master" branch (unless it happens to be identical to the last
release), or totally different if the maintainer doesn't use dgit.

The history is also strange if you "dgit clone" a repository where the
maintainer used dgit in the past, but no longer does.  Now you have a
commit tree with multiple roots which is also confusing for people.

All of this is very uncommon outside the dgit world.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git

[Russ Allbery]

Ansgar  writes:
> On Thu, 2019-05-02 at 09:15 -0700, Russ Allbery wrote:

>> I'm confused.  I'm a happy user of dgit and don't have to think about
>> any of those things as part of using dgit.  I choose to use branches,
>> but I certainly wouldn't have to, and merging, multiple remotes, and so
>> forth don't seem related to using dgit at all.

> How do you update the package to a new upstream release?

I think I understand what I was missing.  Are you using a separate
repository for only the debian directory and merging that with an unpacked
upstream tarball without using a version control system to do so?  I don't
know how that works with dgit if that's the case.

For whatever it's worth, I'm in the camp that believes the combination of
the Debian packaging and the imported upstream releases should be merged
into a single coherent Git history, because the packaging changes are
heavily dependent on the upstream changes and have an important sequencing
between upstream imports that is a key component of the total history of
the package.  Having to manually stitch together two separate histories is
exactly the kind of tedious chore that I think computers are good at and
humans are bad at, so I'd rather let Git do it.  YMMV, of course.

> The "dgit" repository is also separate from the "real" repository; if
> you just use "dgit clone ${something}" you won't get the current
> "master" branch (unless it happens to be identical to the last release),
> or totally different if the maintainer doesn't use dgit.

Yes, that's true, and somewhat inherent in the model.  I don't know that
that's avoidable without standardizing the actual Git tree used by every
maintainer for in-progress work, which is a much harder lift.  dgit stays
out of the business of investigating how the maintainer does work, at the
cost of only having visibility to the commits that have been pushed to the
archive.

-- 
Russ Allbery (r...@debian.org)   

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ben Hutchings writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git"):
> On Thu, 2019-05-02 at 11:23 +0100, Ian Jackson wrote:
> > Sorry for shouting, but, really.  It is kind of frustrating to have
> > designed and implemented and deployed a complex piece of software
> > which solves a lot of problems, and constantly hear people
> >  - proposing solutions which do not address the primary difficulties
> >  - merely lamenting that our current practices are so bad
> >  - stating that the problems are intractable and cannot be solved
> >  - saying that for this we need to agree a uniform git workflow
> 
> We still need to resolve the issues with merging that I raised in
> October.  (I say "we" because I realise you are likely to need me or
> someone else to spend time explaining and testing the specific
> scenarios that didn't work.)

That is to do with git-debrebase, not to do with dgit.

AIUI the current kernel team workflow has a bare-packaging git tree so
*if* you were to want to adopt dgit now you would want #903392 in
dgit, "want support for packaging-only maintainer views", implemented.


Packaging-only maintainer views are in a surprisingly large minority
but that minority largely consists of people who are using and keen on
quite old tools and workflows, and do not very much want to change.

>From the pov of a user/downstream who does `dgit clone': The benefit a
tarball import plus packaging git history (made by `dgit push' during
conversion from a packaging-only maintainer view), compared to a .dsc
import (tarball import, with patch queue converted to git branch, made
by `dgit clone' itself), is rather minor.

So that's why support packaging-only maintainer views is fairly low on
my todo list.


As for the linux kernel:

The proper git history is regarded by most people as the PFM.  So you
would really want #903392 fixed in a way that let dgit reuse upstream
git tags (and dgit would have to check them against your orig
tarballs).  This doesn't seem like something that many other people
would want but maybe it is ?

But all of this seems a lot of work to do for a quite suboptimal
answer when git-debrebase looks like it might be better, even though
there is still work to be done on git-debrebase.  The work on
git-debrebase will have much wider applicability.


So yes, the Debian linux kernel team have a good reason for not using
dgit yet.  Sorry if I gave a contrary impression.  There will be some
other odd cases too no doubt, but what I said was true for most normal
packages maintained using tools like native git, gbp pq, git+quilt, or
git-dpm.

Regards,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Work-needing packages report for May 3, 2019

[wnpp]

The following is a listing of packages for which help has been requested
through the WNPP (Work-Needing and Prospective Packages) system in the
last week.

Total number of orphaned packages: 1400 (new: 0)
Total number of packages offered up for adoption: 162 (new: 1)
Total number of packages requested help for: 55 (new: 0)

Please refer to https://www.debian.org/devel/wnpp/ for more information.



No new packages have been orphaned, but a total of 1400 packages are
orphaned.  See https://www.debian.org/devel/wnpp/orphaned
for a complete list.



The following packages have been given up for adoption:

   terminaltables (#927989), offered 6 days ago
 Description: Python library for printing tables to the console
 Reverse Depends: mycli pgcli python3-envs undertime
 Installations reported by Popcon: 179
 Bug Report URL: https://bugs.debian.org/927989

161 older packages have been omitted from this listing, see
https://www.debian.org/devel/wnpp/rfa_bypackage for a complete list.



For the following packages help is requested:

   autopkgtest (#846328), requested 883 days ago
 Description: automatic as-installed testing for Debian packages
 Reverse Depends: autodeb-worker debci-worker
 Installations reported by Popcon: 1166
 Bug Report URL: https://bugs.debian.org/846328

   balsa (#642906), requested 2776 days ago
 Description: An e-mail client for GNOME
 Installations reported by Popcon: 710
 Bug Report URL: https://bugs.debian.org/642906

   broadcom-sta (#886599), requested 479 days ago (non-free)
 Description: Broadcom STA Wireless driver (non-free)
 Installations reported by Popcon: 1878
 Bug Report URL: https://bugs.debian.org/886599

   cargo (#860116), requested 751 days ago
 Description: Rust package manager
 Reverse Depends: cargo-vendor dh-cargo
 Installations reported by Popcon: 970
 Bug Report URL: https://bugs.debian.org/860116

   cyrus-sasl2 (#799864), requested 1317 days ago
 Description: authentication abstraction library
 Reverse Depends: 389-ds-base 389-ds-base-legacy-tools
   389-ds-base-libs adcli autofs-ldap cairo-dock-mail-plug-in
   claws-mail claws-mail-acpi-notifier claws-mail-address-keeper
   claws-mail-archiver-plugin (114 more omitted)
 Installations reported by Popcon: 198034
 Bug Report URL: https://bugs.debian.org/799864

   dee (#831388), requested 1021 days ago
 Description: model to synchronize mutiple instances over DBus
 Reverse Depends: dee-tools gir1.2-dee-1.0 libdee-dev zeitgeist-core
 Installations reported by Popcon: 58298
 Bug Report URL: https://bugs.debian.org/831388

   developers-reference (#759995), requested 1706 days ago
 Description: guidelines and information for Debian developers
 Installations reported by Popcon: 1
 Bug Report URL: https://bugs.debian.org/759995

   devscripts (#800413), requested 1311 days ago
 Description: scripts to make the life of a Debian Package maintainer
   easier
 Reverse Depends: apt-build apt-listdifferences aptfs arriero
   autodeb-worker brz-debian bzr-builddeb customdeb debci
   debian-builder (31 more omitted)
 Installations reported by Popcon: 12479
 Bug Report URL: https://bugs.debian.org/800413

   docker.io (#908868), requested 229 days ago
 Description: Linux container runtime
 Reverse Depends: golang-docker-dev
   golang-github-fsouza-go-dockerclient-dev
   golang-github-google-cadvisor-dev
   golang-github-samalba-dockerclient-dev kubernetes-node subuser
   whalebuilder
 Installations reported by Popcon: 1642
 Bug Report URL: https://bugs.debian.org/908868

   ed (#886643), requested 479 days ago
 Description: classic UNIX line editor
 Reverse Depends: apt-cacher libdebbugs-perl opensmtpd sn
 Installations reported by Popcon: 17766
 Bug Report URL: https://bugs.debian.org/886643

   ejabberd (#767874), requested 1641 days ago
 Description: distributed, fault-tolerant Jabber/XMPP server written
   in Erlang
 Reverse Depends: ejabberd-contrib ejabberd-mod-cron
   ejabberd-mod-log-chat ejabberd-mod-logsession ejabberd-mod-logxml
   ejabberd-mod-message-log ejabberd-mod-muc-log-http
   ejabberd-mod-post-log ejabberd-mod-pottymouth ejabberd-mod-rest (4
   more omitted)
 Installations reported by Popcon: 498
 Bug Report URL: https://bugs.debian.org/767874

   fbcat (#565156), requested 3396 days ago
 Description: framebuffer grabber
 Installations reported by Popcon: 184
 Bug Report URL: https://bugs.debian.org/565156

   fgetty (#823061), requested 1097 days ago
 Description: console-only getty & login (issue with nis)
 Installations reported by P

Bug#928372: ITP: golang-github-anacrolix-dms-dev -- a UPnP DLNA Digital Media Server that includes basic video transcoding

[Drew Parsons]

Package: wnpp
Severity: wishlist
Owner: Drew Parsons 

* Package name: golang-github-anacrolix-dms-dev
  Version : 1.0.0
  Upstream Author : Matt Joiner 
* URL : https://github.com/anacrolix/dms
* License : BSD
  Programming Lang: Go
  Description : a UPnP DLNA Digital Media Server that includes basic video 
transcoding

dms is a UPnP DLNA Digital Media Server. It runs from the terminal,
and serves content directly from the filesystem from the working
directory, or the path given. The SSDP component will broadcast and
respond to requests on all available network interfaces.

dms advertises and serves the raw files, in addition to alternate
transcoded streams when it's able, such as mpeg2 PAL-DVD and WebM for
the Chromecast. It will also provide thumbnails where possible.

dms uses ffprobe/avprobe to get media data such as bitrate and
duration, ffmpeg/avconv for video transoding, and ffmpegthumbnailer
for generating thumbnails when browsing.


This package is required by latest versions of rclone (providing a new
dlna server).

Packaging will be managed under the Debian Go Packaging Team.