Re: [gentoo-dev] Openstack image availability
On 06/08/2015 10:30 AM, Matthew Thode wrote: Hi, I've just started generation of Gentoo Openstack images. Right now it is just a basic amd64 image, but I plan on adding nomultilib and hardened variants (for a total of at least 4 images). I plan on generating these images at least weekly. These images are not yet sanctioned by our infra team, but I plan on remedying that (being a member of said team should help). I am currently using the scripts at https://github.com/prometheanfire/gentoo-cloud-prep to generate the images (based on a heavily modified version of Matt Vandermeulen's scripts). If you have any issues please submit bugs there or contact me on irc (prometheanfire on freenode). Here's the link to the images, I'm currently gpg signing them with the same key I use to sign this email (offline master key smartcard setup for security minded folk). http://23.253.251.73/ Let me know if you have questions, OK, So to test the images, just burn a dvd/usbstick and run it like a liveDVD? I did not see instructions or brief guidance; I'm noob with openstack but curious to see how she performs.. Long term, will we get an openstack-meta (testing) out of this in portage, or should I just dedicated an old dual core amd64 box for image testing? I'm building up (from 100% sources) a mesos cluster offering with cephfs, Apache-spark, Apache-storm and tachyon. It'd be great to run some tests of codes on openstack (running on local hardware) and then test the same hardware running Apache-mesos. So any suggestions you have on that (so the comparisons are as similar as possible, including recommended test codes) would be of keen interest to me. James
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via "one big tarball" versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. hth, James
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 06:50 PM, Zac Medico wrote: On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via "one big tarball" versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? You mention "a wide variety of sys-cluster builds," but I'm not sure what packages you're talking about. For example, are you aware of any other packages that use hashicorp's raft library [1]? First of all, I'm not sure why my nntp interface to gentoo-dev is not following the thread (sorry, I'm still working out how to use nntp to gentoo-dev). I'm not up on raft, although it looks very interesting [FSM] and all. I've been working on apache-mesos a bit. Consul is used frequently with mesos; here is one example [A]. My experience is that current clusters/clouds are mostly a unique mix of different software, consul being but one of many common components. Perhaps I did not have a sufficiently deep understanding of raft, but my comment was meant to encourage a consul package for gentoo, I guess dependant on a raft package too. Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. You can see all of the components (including github.com/hashicorp/raft) in the SRC_URI variable of the ebuild [2]. Yea, I need to read up on raft; it does look promising as it took mesos a while to become popular. Is raft as a separate ebuild useful; I'm not sure, but it does look interesting from what I've seen. Many projects within the cluster/cloud space have morphed, so raft has just as good a chance to diversify it's appeal and usefulness. Surely the convenience of the dev that maintains the package(s) is also keenly important. [1] https://github.com/hashicorp/raft [2] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup [A] https://github.com/CiscoCloud/mesos-consul James
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
On 07/09/2015 10:45 AM, Alec Warner wrote: On Thu, Jul 9, 2015 at 4:47 AM, Rich Freeman mailto:ri...@gentoo.org>> wrote: On Thu, Jul 9, 2015 at 5:31 AM, hasufell mailto:hasuf...@gentoo.org>> wrote: > > The quality problem is that we have too many developers. If you make > community contributions easier, sane and more reliable (due to code > review) then you solve several problems at once, because you need _less_ > developers. Are you aware that we could potentially "pull" in hundreds > of contributors who chose to work on their personal overlay instead of > the gentoo tree? Are you aware that there are a lot of those people? Yes and no. I'll agree that with a review workflow we might only need one reviewer for every 10 contributors, or something like that. If we have 100 active devs today, in theory we could instead turn 20 of them into reviewers, and now we can support 2000 contributors. There are some big assumptions with this kind of argument, though: 1. We might find that we don't even have 20 devs interested in doing a substantial amount of review. 2. The main repository is very diverse. If 50% of our packages have only one person interested in maintaining them, then they get dropped since reviewers will ignore their contributions. Or, they'll just rubber-stamp them which is adding valueless work. So, a review system could make manpower either more of an issue or less of one. I suspect that trying to force it would basically end up putting the entire distro on hold until most of the current devs quit, and a new crop signs up who is interested in using the new workflow, and then they're starting with zero experience. I think a review model is best implemented by individual project teams. They could use it to track changes in an overlay or branch in the main tree, and then move those into the main tree using whatever quality system seems best. The team can figure out what is working best for it, and if over time a large number of devs feel that it is a good way to work we could then talk about doing it with the main tree. I still suspect we'll end up having problems with the 70% of packages that don't fall into a project though. I think everybody is getting a bit heated over CI/Quality/automation, when clearly there is no good reason to get so heated. Let me explain. YES industry, commercial, research, FOSS and everybody is moving to some sort of paradigm where as much automation of code examination is streamline. Surely these efforts are a work in progress and as such will evolved over time. It does not mean the existing, wonderfully manual processes are being replaced, it just means that the simple, routine, (scriptable if you like) filters are automated. In fact those manual processes still need to occur, because the next generation of coders have to learn how things work in order to extent the paradigm. Resources. Surely the major arches are at an advantage. But the current cluster offerings are soon going to render very powerful processing so even gargantuan, single threaded codes can be automated for CI, quality, security etc etc. Both a cross-compile and native compile strategy needs to experimented with. Those smaller arches will follow the bigger arches; so in that we should just let x86 and other such arches blaze the path for gentoo (in a non binding manner) while the various other arches (with more humble resources) figure out how to build clusters on these other arches. Lots of folks are clustering smalller arches to get big tasks back into the realm of viability. The big benefit for those other (more constrained arches) is that folks that build embedded products on those arches will be very, very interested in work on how to build a cluster, specific to those (sub) arches for CI/quality/security. In fact mesos-distcc does not limit the various arch types. So I would think all arches would get on-board and just follow this paradigm in an easy, non-stressful manner. But the Gentoo leadership does have a responsibility to include those other arches in just how this occurs, during some extended transition period, so that these other arch's still fell like they are part of this extended gentoo family. Loosing those arches would be a horrible damage to gentoo's reputation, imho. That is our diversity in arches is something gentoo is widely know for and sets us apart, imho. hth, James
Re: [gentoo-dev] [RFC] Rebooting the Installer Project
On 07/18/2015 11:30 PM, J.Rutkowski wrote: On Sat, Jul 18, 2015, at 07:06 PM, hasufell wrote: The handbook consists of different sections. Installation is only one of them. An installer would just "replace" that part, not the others. Agreed. I think that any proposed or actual installer shouldn't replace any part of the handbook (and most would probably say the same). My opinion is that an installer should follow the steps (in order) already outlined in the handbook and up front, leave the rest to automation. Nothing then changes in the installation and configuration process. The only difference is making all choices up front and let the installer do the rest by itself. For example, once I prepare my disks I tell the installer "I want x, y, or z stage3; I want openrc or systemd; I want x,y, or z profile; I want the following xxx in my kernel; etc." then set it and forget about it til morning. I agree with what has been stated so far. I would just like to add that eventual support for embedded platforms like PPC, arm (particular arm8v) as well as new files systems like btrfs would been keenly appreciated, even if they are not yet specifically mentioned in the handbook. I.E. I see no reason to limit the new installer to what is strictly included in the handbook. Maybe those more aggressive (wider support) hardware issues are better off being tested in an experimental version(s) of the installer first. The handbook moves at a glacial pace and that's OK, but experimental versions of the installer could be targeted at the rich variety of gentoo and gentoo-derived installs; whilst a stable version mirrors the handbook? Also, I'd like the new installer to use usb media and persistence as the primary media for installation. Many new devices do not readily connect to cd/dvd type devices; maybe this is another stable vs experimental issue for the installer, rather than trying to create one system for disparagingly differ installation semantics. In that venue maybe resurrecting 'netconsole' code as a way to stream the output remotely is another approach for installing on smaller devices? hth, James
Re: [gentoo-dev] .gitignore
On 08/12/2015 09:52 PM, Mike Frysinger wrote: On 12 Aug 2015 18:27, Brian Dolbec wrote: 2) There is another alternate location that you can define files to ignore locally without having to commit them to .gitignore. Consider .gitignore a global setting. There is another setting inside .git/info/exclude which is a local config file that will persist and not be affected by pulls. So please use that for local exclusions you want to add and not try to force them into a global .gitignore which is part of the repo. Something that seems to be hotly debated. ;) as i stated, there's no reason for these paths to ever be committed. conversely, some people (not unreasonably so) will place files in there that have historically had known meanings. adding these to the global gitignore is simply the right thing to do and negatively impacts no one. -mike As a gentoo user now coding again, I find the need to have things "logically organized" really helps in my efforts. Like most others, I get codes from a variety of places and try to follow the 'logical gentoo organization'. I use /usr/local/portage extensively for ebuilds that I develop. There is also /opt/ which seems to collect up a variety of ebuilds. Some, are there as part of their lineage that nobody bothered to change (net-analyzer/jffnms) others are there for license or 'fit' reasons Maybe a survey of codes placed into /opt/ should also be examined as part of these migration efforts? I also experiment around with a variety of sources and I think (old bsd habits) that these should also be under /usr/local/. My point is this:: please keep in mind those that use gentoo for software development outside the dev teams as they have needs for logical organization that mostly follows the 'gentoo way' as you devs finalize the 'gentoo way' for code(s) management. Combined with this is the need to organize where the gentoo community should push their ebuilds which will end up being pushed towards the portage tree. I.E. clearly define the pathways like Sunrise, user-overlays and the corresponding admin semantics. Otherwise we'll end up with lots of viable but unique pathways and that will just add to the support burden on lists like gentoo user and for those mentors as well as the 'preferred or consensus' configuration for git by user's coding needs. hth, James
Re: [gentoo-dev] Re: .gitignore
On 08/13/2015 05:28 PM, Duncan wrote: wireless posted on Thu, 13 Aug 2015 08:33:13 -0500 as excerpted: On 08/12/2015 09:52 PM, Mike Frysinger wrote: On 12 Aug 2015 18:27, Brian Dolbec wrote: 2) There is another alternate location that you can define files to ignore locally without having to commit them to .gitignore. Consider .gitignore a global setting. There is another setting inside .git/info/exclude which is a local config file that will persist and not be affected by pulls. as i stated, there's no reason for these paths to ever be committed. conversely, some people (not unreasonably so) will place files in there that have historically had known meanings. adding these to the global gitignore is simply the right thing to do and negatively impacts no one. As a gentoo user now coding again, I find the need to have things "logically organized" really helps in my efforts. Like most others, I get codes from a variety of places and try to follow the 'logical gentoo organization'. I use /usr/local/portage extensively for ebuilds that I develop. There is also /opt/ which seems to collect up a variety of ebuilds Confused... What do /opt and /usr/local/portage have to do with the gentoo repo's .gitignore, which should only need ignore settings for locations inside the main tree, /usr/portage by default? A .gitignore listing for /usr/portage/local and /usr/portage/distfiles, etc, makes sense, since that's inside the default tree location. But /opt and /usr/local/portage aren't inside that default location and are thus about as apropos to that as the price of tea on Mars, aren't they? /distfiles/ /local/ /packages/ Other postings in this thread mention /var/ and /usr/ My bad, I thought those official directories that were up for discussion on gitignore relevancy? > /usr/portage/local was the original location for the user's own > personal ebuild space - an "overlay" if you will. > /usr/portage/distfiles and /usr/portage/packages are there because > that's where ports has put them for decades, and no-one has gotten > around to changing it in portage yet. FreeBSD defines the use of /usr > very differently to what Linux users are used to. > > Those dirs really should be in /var/portage, and the user's overlay > has no business being under main tree itself Ok, so why is net-analyzer/jffnms found in /usr/portage/distfiles yet it is installed in /opt/ ? If jffnms was (properly) installed like other net-analyzer packages, would it not be in the portage tree? There is no reason for it to remain in /opt/, that I'm aware of. But jffnms could benefit form gitignore as other packages do, regardless of where it is installed. Looking at the documentation for gitignore it would seem that the benefits of using gitignore on those /opt/ packages could be useful; so would it not be up to the particular package maintainer to make that decision? Regardless of where non-devs develop packages for gentoo, using gitignore might be useful during the development of the packages, particularly if it is destine for the portage tree (eventually). Apologies in advance if I have missed some key points already established by the gentoo dev community I'm certainly no whiz with git*. James
Re: [gentoo-dev] Re: .gitignore
On 08/14/2015 04:21 AM, Daniel Campbell (zlg) wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 08/13/2015 06:29 PM, wireless wrote: On 08/13/2015 05:28 PM, Duncan wrote: wireless posted on Thu, 13 Aug 2015 08:33:13 -0500 as excerpted: On 08/12/2015 09:52 PM, Mike Frysinger wrote: On 12 Aug 2015 18:27, Brian Dolbec wrote: 2) There is another alternate location that you can define files to ignore locally without having to commit them to .gitignore. Consider .gitignore a global setting. There is another setting inside .git/info/exclude which is a local config file that will persist and not be affected by pulls. as i stated, there's no reason for these paths to ever be committed. conversely, some people (not unreasonably so) will place files in there that have historically had known meanings. adding these to the global gitignore is simply the right thing to do and negatively impacts no one. As a gentoo user now coding again, I find the need to have things "logically organized" really helps in my efforts. Like most others, I get codes from a variety of places and try to follow the 'logical gentoo organization'. I use /usr/local/portage extensively for ebuilds that I develop. There is also /opt/ which seems to collect up a variety of ebuilds Confused... What do /opt and /usr/local/portage have to do with the gentoo repo's .gitignore, which should only need ignore settings for locations inside the main tree, /usr/portage by default? A .gitignore listing for /usr/portage/local and /usr/portage/distfiles, etc, makes sense, since that's inside the default tree location. But /opt and /usr/local/portage aren't inside that default location and are thus about as apropos to that as the price of tea on Mars, aren't they? /distfiles/ /local/ /packages/ Other postings in this thread mention /var/ and /usr/ My bad, I thought those official directories that were up for discussion on gitignore relevancy? /usr/portage/local was the original location for the user's own personal ebuild space - an "overlay" if you will. /usr/portage/distfiles and /usr/portage/packages are there because that's where ports has put them for decades, and no-one has gotten around to changing it in portage yet. FreeBSD defines the use of /usr very differently to what Linux users are used to. Those dirs really should be in /var/portage, and the user's overlay has no business being under main tree itself Ok, so why is net-analyzer/jffnms found in /usr/portage/distfiles yet it is installed in /opt/ ? If jffnms was (properly) installed like other net-analyzer packages, would it not be in the portage tree? There is no reason for it to remain in /opt/, that I'm aware of. But jffnms could benefit form gitignore as other packages do, regardless of where it is installed. If I may take a stab at this: /usr/portage/distfiles is simply the files Portage needs to fetch in order to extract them, build, and install the package. If ./distfiles/ is in .gitignore, then nobody can accidentally commit a package's source tarball or other related file. Agreed. /opt/ is just the installation location for the live system that it's being installed on. The Portage tree (known forward as the Gentoo repository) is the collection of ebuilds that allow a Gentoo system to build the software. It's historically been in /usr/portage/, and installs to a variety of locations, be it /bin, /sbin, /usr/bin, /usr/sbin, /lib, and so on. As documented here ?: https://devmanual.gentoo.org/general-concepts/filesystem/ Without reading the ebuild, jffnms likely has special properties or is built with specific technologies that indicate its files belong in /opt. Given that the Gentoo repository by default resides in /usr/portage, a .gitignore file can only apply to paths *under* that particular path. So it would cover ./local and ./distfiles, but not /opt, as /opt is outside the /usr/portage location and not under Git control. Looking at jffnms would have shed some light on the arbitrariness of /opt/ policies. Perhaps looking more closely at /opt/ and the various packages, you will find deviation from this aforementioned explanation. Look, I know this whole "git" migration is a work in progress. And everybody knows there is quite a bit of work to do on the docs and that will take some time after the discussions have settled. What I am suggesting is that /opt/ is a mess with sources, binaries, code-license restriction murkyness and codes just placed there without valid (currently) reasons. Some attention would seem integral to the 'gitignore' policies of both the gentoo-repo (tree) and the way ebuild construction documents suggest folks use gitignore on other git repos. That's it. No confusion at all, just a coder wanting some sanity and robustness injected into the processes (outside of official gentoo repo) th
Re: [gentoo-dev] New eclass: ros-catkin.eclass, new USE_EXPAND: ROS_MESSAGES, new categories: dev-ros and ros-meta for review.
On 09/22/2015 03:26 AM, Alexis Ballier wrote:
On Thu, 17 Sep 2015 18:36:06 +0200
Alexis Ballier wrote:
Hi all,
As part of an effort of bringing ros (http://www.ros.org/) to Gentoo,
I've come up with the following design:
ros-catkin.eclass that is a template, merging all common code for
catkinized ros packages.
ROS_MESSAGES will be a new USE_EXPANDed variable: ROS is message
based, processes write and listen to some topic, and a topic is typed
with a message. Messages are defined by simple text files but are then
pre-processed for easy usage in C++, Python & co.
Several packages are usually bundled in a single repository, with a
meta-package pulling all the others. Hence, I came with two new
categories: dev-ros for single packages, ros-meta for those
meta-packages.
Some numbers:
/bin/ls ros-meta | wc -l
52
/bin/ls dev-ros | wc -l
350
I think maybe half of them are in a state that they can be added to
gentoo-x86.
ebuild example:
$ cat roslib-1.12.2.ebuild
# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
EAPI=5
ROS_REPO_URI="https://github.com/ros/ros";
KEYWORDS="~amd64 ~arm"
PYTHON_COMPAT=( python{2_7,3_3,3_4} )
ROS_SUBDIR=core/${PN}
inherit ros-catkin
DESCRIPTION="Base dependencies and support libraries for ROS"
LICENSE="BSD"
SLOT="0"
IUSE=""
RDEPEND="
dev-python/rospkg[${PYTHON_USEDEP}]
dev-libs/boost:=[threads]
dev-ros/rospack
"
DEPEND="${RDEPEND}
test? ( dev-ros/rosmake ros-meta/ros )"
Eclass inlined at the end.
Best regards,
Alexis.
Since there hasn't been comments in a few days, I'm proceeding with the
above plan. (with hasufel's eclass review taken into account of course)
Alexis.
Hi Alexis.
As a gentoo user, an ebuild hacker and engineer, I'd suggest you
include a wiki page for gentoo users, to document the gentoo specific
issues. I'm a big fan of the new gentoo wiki. Access to either a
'virtual robot', or where to purchase devices that use ROS, would at
least greatly facilitate folks to participate with ROS robotics codes.
Include links to the main ROS repos, and relevant other urls.
For example; one of the common things missing in such projects is
an example linux kernel, stripped to optimize only the size and
performance of the ROS devices and software. Showing this kernel .config
would be keen. At list discussion why some of the key kernel configs
are select is sorely missing in the world of robotic and UAVs, imho.
You may also what to include 'hooks' to relevant and commonly used
embedded boards with ROS hardware. I am working on clustering for gentoo
using mesos + zookeeper + apache-spark (we have 2/3 in overlays now).
Lots of robotic work now attempts to use the power of the cluster over
an Rf link and/or use many robots computational resources, as a mere
node in a cluster.
I for one have been looking for such a project, if It combines
robotics issues with (UAV) issues. After all, all UAVs are robots, with
additional constraints.
hth,
James
Re: [gentoo-dev] RFI: A better workflow for github pull requests
On 09/24/2015 09:59 AM, Todd Goodman wrote: Again, I'm not arguing for or against Gerrit, just relating my experience with it. Hello Todd, I'm not a dev, but I have over a dozen ebuilds I have hacked together. They need work, and there are devs who are helping me. Some of these ebuilds are becoming quite large, due to the numerous jars or the extensive configuration of the initial config file setup. I enjoy the learning and hacking part, but gentoo's tool collection presents itself as somewhat disjointed, imho. If a collection of overlays could be used to setup a prototype of Gerrit, then, I'd be willing to participate in fully exploring Gerrit and its use to 'unify' the development effort between user hacked ebuilds and the portage tree. There is currently no well defined path for hackers to become contributors and maybe eventually devs, the flux is even more partitioned with github now in the mixxx. If Gerrit could unify (like an integrated, experimental restructuring of the devmanual into)the workflow tools, I think Gerrit could become a wonderful addition to gentoo. Maybe a prototype effort will allow the Gerrit community within Gentoo to flush out issues before being formally proposed to the larger gentoo dev community? The devmanual, at least the version used by hackers and those learning 'the gentoo way' to create ebuilds, should become more open to enhancements, details and examples; and who is more qualified to make those mods than folks trying to use the devmanual to get codes stable and maybe eventually into the portage tree? The current devmanual is good, if you know what you are doing; a bit weak in the teaching by example category, particularly for current and recent issues. curiously, James
