On Mon, 20 Apr 2020 at 13:43:57 +0200, Marc Espie wrote:
> On Sun, Apr 19, 2020 at 10:06:34PM +0100, Stuart Henderson wrote:
> > On 2020/04/19 19:54, f.holop wrote:
> > > hello,
> > >
> > > i don't have experience with go ports and i cannot seem to have
> > > `make update-patches` work on this go project:
> >
> > It is not ideal at present. The directory layout isn't put into place
> > until "make build" starts running. Last diff I saw to fix this broke
> > some of the go ports.
> >
> >
> Woah, I just looked at this, this is crazy.
>
> Why does go have to move stuff around again ?...
So some history around the Go dependency stuff:
* Initially Go used GOPATH to know where dependencies were.
It would look for things in $GOPATH/src/{$modulename} If they weren't there,
it would download them from HEAD for what ever source repo the deps were
hosted with.
This had a few issues:
- Project layout was forced. Everything needed to exist relative to GOPATH.
- HEAD might have breaking changes for the API used in what ever application.
- You relied on the upstream repos exist, and to not be malicious..
How it impacted OpenBSD:
- Because of the strict layout requirements of GOPATH, we needed to move
thingss around.
- Early on, in an attempt to make use of patched / shared code, we introduced
a number of "go-XXXX" ports (go-sys for example). We would include the path
that these were installed to in the GOPATH we set in go.port.mk.
This was fine and dandy as GOPATH had no concept of versions.
* Then came tools like dep, glide.. and a number of others. They would
populate the "vendor" directory with versioned (based on tags in the source
repo). Now "go build" would look in $GOPATH and $WRKSRC/vendor for
dependencies.
This had a few issues as well:
- Upstreams hated the churn in the SCM log that was caused by an update to
the vendor directory - thus causing a number of projects to refuse to
vendor their deps.
- There were _many_ different tools to manage the vendor directory, each with
it's own config (yaml, toml, json.. etc).
- Upstreams could up and vanish - rendering the projects that had not
vendor'd unable to build.
How it impacted OpenBSD:
- This is when we ramped up hand rolling tarballs with the vendor directory
added.
* Enter "go modules". The aim of modules is to resolve most of the problems
with how Go handled dependencies. With modules we have versioned dependencies
stored externally to any one SCM (proxy.golang.org). These modules are also
independently checksum'd to prevent re-publishing of versions.
How it impacts OpenBSD:
- Now we can remove the hacks. We don't have to care about GOPATH, we don't
need to hand-roll tarballs. We can (should) remove the go-XXX ports (The
only way to use them going forward is to explicitly use GOPATH - which
takes us back to non-versioned, forced-directory-layout builds)!
- We also get a place to fetch dependencies (proxy.golang.org)!
- We can have reproducible builds! The binaries we produce from the ports
tree can be exactly the same as binaries produced by upstreams (obviously
this precludes patched stuff)!
The stuff I have been doing with portgen aims to bring us a bit closer to the
"new world" modules.
I hope this helps clear up some confusion, and maybe shed some light on the
direction we need to be headed with Go ports!
Cheers,
Aaron
--
PGP: 0x1F81112D62A9ADCE / 3586 3350 BFEA C101 DB1A 4AF0 1F81 112D 62A9 ADCE
