[gentoo-dev] Building a portage repo?
In my work on a sparc64 port of Gentoo i have found it easier to just fork the portage repo, patch things, and merge upstream changes / resolve conflicts as they happen. The problem is that this isn't really the best way to test things on any kind of scale. My plan is to build a script that does the following. 1. pull latest portage tree from git 2. apply patches 3. merge new portage tree to rsync server for clients. The issue is that pulling from git i seem to be missing a ton of data, I.E. manifests and metadata like news (i'm assuming news is metadata). What is the utility that is used to take the git tree and generate all of this metadata, egencache? Am i correct in assuming that after applying patches, running repoman and subsequently egencache should give me the same type of portage tree that is currently officially distributed to users?
Re: [gentoo-dev] Building a portage repo?
Dnia 14 września 2016 09:01:43 CEST, alexmcwhir...@triadic.us napisał(a): >In my work on a sparc64 port of Gentoo i have found it easier to just >fork the portage repo, patch things, and merge upstream changes / >resolve conflicts as they happen. The problem is that this isn't really Portage is the package manager. The repository is named gentoo. Please use correct names because till the last paragraph, I thought you were actually referring to the package manager (and wtf-ed why would you distribute its sources via rsync). > >the best way to test things on any kind of scale. My plan is to build a > >script that does the following. > >1. pull latest portage tree from git >2. apply patches >3. merge new portage tree to rsync server for clients. > >The issue is that pulling from git i seem to be missing a ton of data, >I.E. manifests and metadata like news (i'm assuming news is metadata). >What is the utility that is used to take the git tree and generate all >of this metadata, egencache? > >Am i correct in assuming that after applying patches, running repoman >and subsequently egencache should give me the same type of portage tree > >that is currently officially distributed to users? Partially. You probably need to update layout.conf, to get desired manifest format first. Then remanifest, egencache with all options. You will also need to include additional repositories for news, glsas and fetch extra files for projects.xml. Generally, it's easier to fetch https://github.com/gentoo-mirror/gentoo that has all the extra content included. However, if you want it 100% rsync compliant, you will still have to patch layout.conf and remanifest. Also note that the manifest format will be subject to change in the future, to support fully GPG-signed repository. -- Best regards, Michał Górny (by phone)
Re: [gentoo-dev] Building a portage repo?
On 2016-09-14 03:40, Michał Górny wrote: Portage is the package manager. The repository is named gentoo. Please use correct names because till the last paragraph, I thought you were actually referring to the package manager (and wtf-ed why would you distribute its sources via rsync). Sorry, it's been a long night. Generally, it's easier to fetch https://github.com/gentoo-mirror/gentoo that has all the extra content included. However, if you want it 100% rsync compliant, you will still have to patch layout.conf and remanifest. Also note that the manifest format will be subject to change in the future, to support fully GPG-signed repository. I was unaware of that repository, thanks for pointing it out. It makes much more sense to start there than trying to rebuild everything, at least while my changes stay small anyways.
Re: [gentoo-dev] questions about small fixes/cleanups
On 09/14/2016 01:54 AM, Ulrich Mueller wrote: > > We had a review of such files before the git conversion: > https://bugs.gentoo.org/550434 > > Especially, there's a list of "maintainer scripts" in comment #13. > At the time, we didn't do anything about them. There are very few of > such files (namely, 12 when that comment was written) so the space > taken by them is negligible. > I'll take the low hanging fruit: * dev-libs/collada-dom/files/take_snapshot.sh * sys-libs/libcxxrt/files/prepare_snapshot.sh Those two are completely trivial; clone a repo, rm -rf .git, tar it up. * sys-libs/libcxx/files/prepare_snapshot.sh This one's only marginally more complicated because it uses find/xargs to remove all ".svn" directories (why not use svn export?). It also appears unused, now that there are regular releases being made upstream. * sys-apps/superiotool/files/make-tarball.sh Another trivial three-liner; svn export and tar it up. * net-misc/ntp/files/man-pages/genmans.sh Generating man pages from HTML can be done in src_compile().
[gentoo-dev] Re: Building a portage repo?
Michał Górny wrote: >> >>and subsequently egencache should give me the same type of portage tree >>that is currently officially distributed to users? > > Then remanifest, egencache with all options. You will also need to > include additional repositories for news, glsas and fetch extra files > for projects.xml. Not to forget dtd and xml-schema. There is app-portage/portage-postsyncd-mv in the mv overlay which does all this (and some more) for you in various (separately configurable) hooks in /etc/portage/repo.postsync.d/* Well, all except changing the manifest format and remanifest, but you can easily add another hook if you really want that.
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 08:45:58 -0400 Michael Orlitzky wrote: > On 09/14/2016 01:54 AM, Ulrich Mueller wrote: > > > > We had a review of such files before the git conversion: > > https://bugs.gentoo.org/550434 > > > > Especially, there's a list of "maintainer scripts" in comment #13. > > At the time, we didn't do anything about them. There are very few of > > such files (namely, 12 when that comment was written) so the space > > taken by them is negligible. > > > > I'll take the low hanging fruit: > > * dev-libs/collada-dom/files/take_snapshot.sh > * sys-libs/libcxxrt/files/prepare_snapshot.sh > > Those two are completely trivial; clone a repo, rm -rf .git, tar it > up. Guess you never had to maintain packages whose releases are only snapshots. They're trivial but it's a waste of time to re-assemble all the bits every couple of months when you want to make a snapshot. Questions one needs to answer: - What SCM is used ? - What is the URI ? - How to fetch it ? - What should be cleaned up after fetching ? - What dirname to use so that it matches ebuilds' $S ? - What compression to use so that it matches SRC_URI ? Those scripts are made to automate this, and as you can see are closely related to ebuilds, and thus fit perfectly next to them.
Re: [gentoo-dev] questions about small fixes/cleanups
On 09/14/2016 09:21 AM, Alexis Ballier wrote: > > Guess you never had to maintain packages whose releases are only > snapshots. They're trivial but it's a waste of time to re-assemble all > the bits every couple of months when you want to make a snapshot. > > Questions one needs to answer: > > - What SCM is used ? > - What is the URI ? > - How to fetch it ? > - What should be cleaned up after fetching ? These are all the same question, which does take a second to google, but isn't all that time consuming. > - What dirname to use so that it matches ebuilds' $S ? > - What compression to use so that it matches SRC_URI ? cat Manifest But seriously, once you've googled the repo, you might notice that it's hosted at github, and that you don't need to create your own snapshot tarballs any more. Want a snapshot from 20150423? Try, https://github.com/pathscale/libcxxrt/archive/665b0223ee27547175dc327a386fcd88243c7677.tar.gz Put COMMIT=665b0223ee27547175dc327a386fcd88243c7677 in the ebuild, stick that in $SRC_URI and $S, and then future snapshots will work without modification.
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 15:50:28 +0200 Alexis Ballier wrote: > > But seriously, once you've googled the repo, you might notice that > > it's hosted at github, and that you don't need to create your own > > snapshot tarballs any more. Want a snapshot from 20150423? Try, > > > > https://github.com/pathscale/libcxxrt/archive/665b0223ee27547175dc327a386fcd88243c7677.tar.gz > > > > Put COMMIT=665b0223ee27547175dc327a386fcd88243c7677 in the ebuild, > > stick that in $SRC_URI and $S, and then future snapshots will work > > without modification. > > that might be better, but how do you map date / $PV to commit ? You can't. > remember, we want ebuilds that are as much as possible > version-agnostic, otherwise it breaks simple copy of ebuilds to bump a > version... Ideally yes but this still beats manually preparing a tarball. I use this approach a lot. -- James Le Cuirot (chewi) Gentoo Linux Developer
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 14:53:18 +0100 James Le Cuirot wrote: > On Wed, 14 Sep 2016 15:50:28 +0200 > Alexis Ballier wrote: > > > > But seriously, once you've googled the repo, you might notice that > > > it's hosted at github, and that you don't need to create your own > > > snapshot tarballs any more. Want a snapshot from 20150423? Try, > > > > > > https://github.com/pathscale/libcxxrt/archive/665b0223ee27547175dc327a386fcd88243c7677.tar.gz > > > > > > Put COMMIT=665b0223ee27547175dc327a386fcd88243c7677 in the ebuild, > > > stick that in $SRC_URI and $S, and then future snapshots will work > > > without modification. > > > > that might be better, but how do you map date / $PV to commit ? > > You can't. Sorry, let me correct that, you can't by date but you can by $PV if they have created versioned git tags. Just swap the commit reference for the tag name. -- James Le Cuirot (chewi) Gentoo Linux Developer
Re: [gentoo-dev] questions about small fixes/cleanups
On 09/14/2016 09:50 AM, Alexis Ballier wrote: > > that might be better, but how do you map date / $PV to commit ? > Well, for that last one, I just looked down the list of commits and found the last one that happened before the date of the snapshot. But, if you're creating a new snapshot, it's easy. What day is it? That's the date of the snapshot, and you want the newest commit. > remember, we want ebuilds that are as much as possible > version-agnostic, otherwise it breaks simple copy of ebuilds to bump a > version... > With the COMMIT=... approach, all you have to do is copy the ebuild to a new one with today's date, and update the COMMIT variable to the latest commit listed on github. There is the possibility that more than one commit can happen on the same day, but that's a problem with snapshots in general, and not the method of generating them.
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 09:38:45 -0400 Michael Orlitzky wrote: > On 09/14/2016 09:21 AM, Alexis Ballier wrote: > > > > Guess you never had to maintain packages whose releases are only > > snapshots. They're trivial but it's a waste of time to re-assemble > > all the bits every couple of months when you want to make a > > snapshot. > > > > Questions one needs to answer: > > > > - What SCM is used ? > > - What is the URI ? > > - How to fetch it ? > > - What should be cleaned up after fetching ? > > These are all the same question, which does take a second to google, > but isn't all that time consuming. Manual stuff isnt that time consuming until you realize it can be automated :) > > - What dirname to use so that it matches ebuilds' $S ? > > - What compression to use so that it matches SRC_URI ? > > cat Manifest not really > > But seriously, once you've googled the repo, you might notice that > it's hosted at github, and that you don't need to create your own > snapshot tarballs any more. Want a snapshot from 20150423? Try, > > https://github.com/pathscale/libcxxrt/archive/665b0223ee27547175dc327a386fcd88243c7677.tar.gz > > Put COMMIT=665b0223ee27547175dc327a386fcd88243c7677 in the ebuild, > stick that in $SRC_URI and $S, and then future snapshots will work > without modification. that might be better, but how do you map date / $PV to commit ? remember, we want ebuilds that are as much as possible version-agnostic, otherwise it breaks simple copy of ebuilds to bump a version...
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 14:55:06 +0100
James Le Cuirot wrote:
> On Wed, 14 Sep 2016 14:53:18 +0100
> James Le Cuirot wrote:
>
> > On Wed, 14 Sep 2016 15:50:28 +0200
> > Alexis Ballier wrote:
> >
> [...]
> [...]
> >
> > You can't.
>
> Sorry, let me correct that, you can't by date but you can by $PV if
> they have created versioned git tags. Just swap the commit reference
> for the tag name.
>
Also, you'll probably want one of
SRC_URI="/archive/${SHA1}.tar.gz -> ${PN}-${SHA1}.tar.gz"
SRC_URI="/archive/${TAG}.tar.gz -> ${PN}-${TAG}.tar.gz"
Because otherwise your tars are just going to be
distfiles/dcd4e6df145853de19af532b0f581a8f0ddfe5ae.tar.gz
or distfiles/0.1.0.tar.gz ( and the latter is destined to collide )
and you'll then also need one of
S="${WORKDIR}/${PN}-${SHA1}"
S="${WORKDIR}/${PN}-${TAG}"
But if $TAG contains a "v" prefix, the first will work, but the second will
need the "v" stripped.
Because github.
pgppXPnd6eJ_9.pgp
Description: OpenPGP digital signature
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 14:55:06 +0100 James Le Cuirot wrote: > On Wed, 14 Sep 2016 14:53:18 +0100 > James Le Cuirot wrote: > > > On Wed, 14 Sep 2016 15:50:28 +0200 > > Alexis Ballier wrote: > > > > > > But seriously, once you've googled the repo, you might notice > > > > that it's hosted at github, and that you don't need to create > > > > your own snapshot tarballs any more. Want a snapshot from > > > > 20150423? Try, > > > > > > > > https://github.com/pathscale/libcxxrt/archive/665b0223ee27547175dc327a386fcd88243c7677.tar.gz > > > > > > > > Put COMMIT=665b0223ee27547175dc327a386fcd88243c7677 in the > > > > ebuild, stick that in $SRC_URI and $S, and then future > > > > snapshots will work without modification. > > > > > > that might be better, but how do you map date / $PV to > > > commit ? > > > > You can't. > > Sorry, let me correct that, you can't by date but you can by $PV if > they have created versioned git tags. Just swap the commit reference > for the tag name. We're talking of repos without tags :)
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 10:04:20 -0400 Michael Orlitzky wrote: > On 09/14/2016 09:50 AM, Alexis Ballier wrote: > > > > that might be better, but how do you map date / $PV to commit ? > > > > Well, for that last one, I just looked down the list of commits and > found the last one that happened before the date of the snapshot. > > But, if you're creating a new snapshot, it's easy. What day is it? > That's the date of the snapshot, and you want the newest commit. Seems I missed some questions in the original reply: - What day is it ? - How to write that so that it is PMS-compliant and properly ordered and following ebuild's conventions ? Nothing is hard, it's just a complete waste of time to redo that every time. > > remember, we want ebuilds that are as much as possible > > version-agnostic, otherwise it breaks simple copy of ebuilds to > > bump a version... > > > > With the COMMIT=... approach, all you have to do is copy the ebuild > to a new one with today's date, and update the COMMIT variable to the > latest commit listed on github. So, to sum it up, I have to: - Open a browser, go to github (*) - Find out latest commit hash, copy it - (*) Copy the ebuild, setting a 8 digit version representing the date - Open an editor - Edit COMMIT='...' variable by pasting what was found on github. (*) Thanks, but I prefer by far: - Run a script - Copy ebuild, setting version printed by the script. (*) represents everytime i have to switch my hands between keyboard and mouse
Re: [gentoo-dev] questions about small fixes/cleanups
On Wed, 14 Sep 2016 16:41:54 +0200 Alexis Ballier wrote: > > So, to sum it up, I have to: > - Open a browser, go to github (*) > - Find out latest commit hash, copy it > - (*) Copy the ebuild, setting a 8 digit version representing the date > - Open an editor > - Edit COMMIT='...' variable by pasting what was found on github. (*) > > > Thanks, but I prefer by far: > - Run a script > - Copy ebuild, setting version printed by the script. > > > (*) represents everytime i have to switch my hands between keyboard and > mouse > You can also abuse `git ls-remote Path/To/Repo` -> list of SHA1s -> ebuild -> sed But you're back to writing a script =) Or you can do that manually, it reduces the number of *'s pgpDM0zfY1Fs3.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] questions about small fixes/cleanups
On 14/09/16 12:20 PM, Kent Fredric wrote:
> On Wed, 14 Sep 2016 16:41:54 +0200
> Alexis Ballier wrote:
>
>>
>> So, to sum it up, I have to:
>> - Open a browser, go to github (*)
>> - Find out latest commit hash, copy it
>> - (*) Copy the ebuild, setting a 8 digit version representing the date
>> - Open an editor
>> - Edit COMMIT='...' variable by pasting what was found on github. (*)
>>
>>
>> Thanks, but I prefer by far:
>> - Run a script
>> - Copy ebuild, setting version printed by the script.
>>
>>
>> (*) represents everytime i have to switch my hands between keyboard and
>> mouse
>>
>
>
> You can also abuse `git ls-remote Path/To/Repo` -> list of SHA1s -> ebuild ->
> sed
>
> But you're back to writing a script =)
>
> Or you can do that manually, it reduces the number of *'s
>
More to the point of the original query, such scripts don't
necessarily need to exist in the gentoo repo; they can be in local
repos or project overlays instead.
Note also that I'm perfectly fine with scripts like this staying in
${FILESDIR}.
signature.asc
Description: OpenPGP digital signature
