Re: [gentoo-dev] Re: mbox -- looks sort of interesting
Hi, It was my project. The portage changed a lot since that time, I can try to renew it, if it's still used. 2014-02-12 17:45 GMT+06:00 Michael Palimaka : > On 02/12/2014 04:56 PM, Brian Dolbec wrote: > > On Wed, 12 Feb 2014 01:36:01 +1100 > > Michael Palimaka wrote: > > > >> On 02/12/2014 01:03 AM, Rich Freeman wrote: > >>> On Tue, Feb 11, 2014 at 7:39 AM, Michael Palimaka > >>> wrote: > On 02/11/2014 11:34 PM, Rich Freeman wrote: > > > One of those ideas I've always wanted to implement is to create a > > portage hook/patch that looks at the dependencies for the package > > being built and configures sandbox to block read-access to > > anything that wasn't explicitly declared. Sandbox works for > > read-access as well as write-access, though > > in /etc/sandbox.d/00default read-access is enabled everywhere by > > default. > > > > And, yes, it could be configured to allow access to @system... > That's pretty much what emerge_strict does. > >>> > >>> What is emerge_strict? The Google is failing me here... > >>> > >>> Rich > >>> > >>> > >> Sorry, I should have clarified. It's provided by autodep, extending > >> the dependency analysis by denying access to any files not part of the > >> specified dependencies and @system. > >> > >> > > > > There was a gentoo gsoc project a few years ago that did exactly this > > for doing dep checks on ebuilds. There was also one for determining > > deps automatically. > > > > Is this the project mentioned? ^^^ > > > > Should be, autodep was GSoC 2011. > > >
Re: [gentoo-dev] Re: mbox -- looks sort of interesting
Ok, I'll work on it. 2014-02-13 23:00 GMT+06:00 Rich Freeman : > On Thu, Feb 13, 2014 at 11:07 AM, Brian Dolbec wrote: > > Yes, if you can please work on updating it. Please contact us on the > > gentoo-portage-dev mail list or irc #gentoo-portage for changes to > > portage that will help keep it working in the future. I started > > development of a public_api branch long ago just for having a stable > > API for apps to use. Perhaps some of what you need can be integrated > > there. > > Seems like a valuable tool. It would be best if it could either use > portage APIs where available, or if effort could be directed at > incorporating the necessary features into portage or its APIs so that > you're not stuck maintaining a fork. I'm sure the portage team will > help where they can. > > Rich > >
Re: [gentoo-dev] Re: mbox -- looks sort of interesting
Aleksandr 2014-02-14 13:50 GMT+06:00 Greg Turner : > Holy crap, that looks awesome! How does one pronounce your name, > Александр? > > On Thu, Feb 13, 2014 at 11:28 PM, Александр Берсенев > wrote: > > Ok, I'll work on it. > > > > > > 2014-02-13 23:00 GMT+06:00 Rich Freeman : > > > >> On Thu, Feb 13, 2014 at 11:07 AM, Brian Dolbec > wrote: > >> > Yes, if you can please work on updating it. Please contact us on the > >> > gentoo-portage-dev mail list or irc #gentoo-portage for changes to > >> > portage that will help keep it working in the future. I started > >> > development of a public_api branch long ago just for having a stable > >> > API for apps to use. Perhaps some of what you need can be integrated > >> > there. > >> > >> Seems like a valuable tool. It would be best if it could either use > >> portage APIs where available, or if effort could be directed at > >> incorporating the necessary features into portage or its APIs so that > >> you're not stuck maintaining a fork. I'm sure the portage team will > >> help where they can. > >> > >> Rich > >> > > > >
Re: [gentoo-dev] Re: mbox -- looks sort of interesting
Hello, I've updated the autodep and testing it now. I've fixed a problem with the integration of autodep in the newest version of Portage, but I think I did it in a hackish way. I want to ask if it is possble to solve it better. First, some words about the architecture of autodep. Autodep consists of two parts: 1) A library, injected with LD_PRELOAD, which intercepting glibc functions like open, read, write, etc. The library connects using unix socket with a server part and asks this server about each file operation. The server can say to permit or to block an operation. 2) The server part, which logs requests from clients and answers them. The integration with the portage is made, using two hooks on build(EbuildBuild.py): build = EbuildExecuter(background=self.background, pkg=pkg, scheduler=scheduler, settings=settings) build.addStartListener(self._build_start) build.addExitListener(self._build_stop) self._start_task(build, self._build_exit) The _build_start creates the server thread, and passes a unix-socket path using LOG_SOCKET environment variable. self.logserver.start() # Copy socket path to LOG_SOCKET environment variable env=self.settings.configdict["pkg"] env['LOG_SOCKET'] = self.logserver.socket_name The _build_stop stops the server and removes LOG_SOCKET environment variable. env=self.settings.configdict["pkg"] if 'LOG_SOCKET' in env: del env['LOG_SOCKET'] events=self.logserver.stop() self.logserver=None If LOG_SOCKET env var is set then the new processes(like ebuild.sh ) are started with my library injected using LD_PRELOAD trick. The problem: on newer portage versions packages are merged in a subprocess to allow the Scheduler run in main thread while files are moved or copied asynchronously(file _MergeProcess.py). This subprocess is created before the _build_stop is called, so it still has a copy of settings with LOG_SOCKET set. When _build_stop is called the server stops and merging fails because the unix-socket is closed. To solve this I added this code before forking in _MergeProcess.py: # Delete LOG_SOCKET from environment env=self.settings.configdict["pkg"] if 'LOG_SOCKET' in env: del env['LOG_SOCKET'] But I think this can be done better. Do you know a better way? P.S. The old variant of integration patch is available here: http://git.overlays.gentoo.org/gitweb/?p=proj/autodep.git;a=blob;f=integration_with_portage.patch;h=ff8f487306e85cbc6ba9e69be6fe613e99b39b53;hb=HEAD Best, Alexander Bersenev 2014-02-14 14:08 GMT+06:00 Александр Берсенев : > Aleksandr > > > 2014-02-14 13:50 GMT+06:00 Greg Turner : > > Holy crap, that looks awesome! How does one pronounce your name, >> Александр? >> >> On Thu, Feb 13, 2014 at 11:28 PM, Александр Берсенев >> wrote: >> > Ok, I'll work on it. >> > >> > >> > 2014-02-13 23:00 GMT+06:00 Rich Freeman : >> > >> >> On Thu, Feb 13, 2014 at 11:07 AM, Brian Dolbec >> wrote: >> >> > Yes, if you can please work on updating it. Please contact us on the >> >> > gentoo-portage-dev mail list or irc #gentoo-portage for changes to >> >> > portage that will help keep it working in the future. I started >> >> > development of a public_api branch long ago just for having a stable >> >> > API for apps to use. Perhaps some of what you need can be integrated >> >> > there. >> >> >> >> Seems like a valuable tool. It would be best if it could either use >> >> portage APIs where available, or if effort could be directed at >> >> incorporating the necessary features into portage or its APIs so that >> >> you're not stuck maintaining a fork. I'm sure the portage team will >> >> help where they can. >> >> >> >> Rich >> >> >> > >> >> >
Re: [gentoo-dev] Re: mbox -- looks sort of interesting
You are welcome! While testing I found strange race-condition bug in portage, I've filed it here: https://bugs.gentoo.org/show_bug.cgi?id=502428 2014-02-26 0:28 GMT+06:00 Michael Palimaka : > On 02/22/2014 08:50 PM, Александр Берсенев wrote: > > Hello, > > > > I've updated the autodep and testing it now. > > Thanks for taking the time to do this! > > > >
[gentoo-dev] Autodep project
Hi! I am Alexander Bersenev, I was participating in GSoC this year. I want to present you the tool I've developed during this program. I hope that you'll find it useful. The purpose of my project is to help ebuild developers to compose accurate dependency list for a package. The tool has many features in order to do it, most of them listed on the documentation site: http://soc.dev.gentoo.org/~bay/autodep/ The killer-feature is an emulating the file system without non-dependency packages installed. I call it dependency checking or strict emerging. If program builds successfully in this environment then check is passed. If no - check is failed and user likely will be having a bad experience while trying to build this package if he hasn't some packages installed. It works fine, I've reported a few dozens of bugs about missing dependencies here: https://bugs.gentoo.org/show_bug.cgi?id=autodep. How to install and use it: 1) add neurogeek overlay in your overlay list 2) emerge autodep 3) use autodep and emerge_strict commands. I want to tell you more about emerge_strict command. This is an emerge command but with strict dependency checking. I've modified a portage and add this feature into it. Actually, after "emerge autodep" you will have two versions of portage: one is from your system(emerge) and one is from modified portage(emerge_strict). !!!ATTENTION!!! I modified a last available portage version from git. It is about Portage 2.2.0_alpha50. The you running portage 2.1.x.x it theoretically can be unsafe to use them both. I used it together for about a month and not found any problems, but, anyway, be careful. !!!ATTENTION!!! Here is an example of emerge_strict dev-libs/nss output: https://381591.bugs.gentoo.org/attachment.cgi?id=285369 Missing dependency is founded here: sdb.c:58:21: fatal error: sqlite3.h: No such file or directory compilation terminated. [NOT IN DEPS] dev-db/sqlite-3.7.7.1 : [u'compile'] /usr/include/sqlite3.h blocked I've set up a tinderbox to catch missing dependencies. And it works right now. Also, I recently installed desktop gentoo linux on my new notebook using only emerge_strict. It is also works. Although GSoC is over, I want to support this tool in further. I will be appreciate for any feedback. Best, Alexander Bersenev
Re: [gentoo-dev] Autodep project
Thanks, I'll try to fix it. And English too. Best, Alexander Bersenev
Re: [gentoo-dev] Autodep project
Sorry for a long delay. New version of autodep is available. To install it do: 1) add neurogeek overlay in your overlay list 2) emerge autodep 3) use autodep and emerge_strict commands. This version was heavily on my few computers for a month. - Fixed a crash when file with non-unicode character was in the name of file. - Some packages redefined snprintf function and autodep wasn't expected it. Now it uses own snprintf function. I am testing 0.3 version now. I slightly rewrote an algo of allowed packages list composing: http://git.overlays.gentoo.org/gitweb/?p=proj/autodep.git;a=commitdiff;h=e541c423ce50003e05b9a154a0893cafe8bd7445 Best, Alexander Bersenev 2011/9/7 Alexander Bersenev : > I already have depcheck and depcheckstrict FEATURES. Emerge_strict script > launch modified version of emerge enabling depcheckstrict feature. > > I have a 0.2 version of this utility with some bugs fixed, I testing it now. > > I plan to move this package into main tree and I am going to talk about > integration of this patch into portage but I think some things must be > changed in it before. > > Best, > > Alexander Bersenev > > On 07.09.2011, at 15:08, Tomáš Chvátal wrote: > >> Hi, >> really cool thing you create :) >> >> Would it be possible to move that package to main tree, and merge >> or possibly add new FEATURES option to portage like "autobuildchecks" that >> would be set by -dev profile? >> >> Cheers >> Tom >> >
