Re: User-installable Debian packages?
Hi Steffen, Steffen Möller writes: > I just had a quick look and it seems really nice, indeed! Thank you tons for > pointing that out to us. Has anybody already tried that with Debian? I am one of a few guys behind that project. Gentoo Prefix with its own libc runs on Debian very well, the explicity tested distributions are listed at, https://wiki.gentoo.org/wiki/Prefix/libc#Tested_distributions The highlights are: 1. you can compile almost any package available in Gentoo. 2. you can run x86 Gentoo Prefix on a amd64 Debian, thus another form of multiarch. The downside compared to Debian and any binary distribution is that, everything need to be compiled from source. That's slow. A preliminary draft has been prepared to discuss its use on HPC environments: https://arxiv.org/abs/1610.02742 Alternative projects are _spack_ and _easybuild_, with slightly different motivations and implementations. Cheers, Benda
Bug#870130: ITP: jbuilder -- composable build system for OCaml projects
Package: wnpp Severity: wishlist Owner: Stéphane Glondu * Package name: jbuilder Version : 1.0~beta11 Upstream Author : Jane Street Group, LLC * URL : https://github.com/janestreet/jbuilder * License : Apache-2.0 Programming Lang: OCaml Description : composable build system for OCaml projects Jbuilder is a build system designed for OCaml/Reason projects only. It focuses on providing the user with a consistent experience and takes care of most of the low-level details of OCaml compilation. All you have to do is provide a description of your project and Jbuilder will do the rest. . The scheme it implements is inspired from the one used inside Jane Street and adapted to the open source world. It has matured over a long time and is used daily by hundred of developpers, which means that it is highly tested and productive. This is a new dependency of ppx-core, sexplib310...
Re: User-installable Debian packages?
Hi Benda, On 30.07.17 09:37, Benda Xu wrote: > Hi Steffen, > > Steffen Möller writes: > >> I just had a quick look and it seems really nice, indeed! Thank you tons for >> pointing that out to us. Has anybody already tried that with Debian? > I am one of a few guys behind that project. Gentoo Prefix with its own > libc runs on Debian very well, the explicity tested distributions are > listed at, > > https://wiki.gentoo.org/wiki/Prefix/libc#Tested_distributions > > The highlights are: > > 1. you can compile almost any package available in Gentoo. > 2. you can run x86 Gentoo Prefix on a amd64 Debian, thus another form > of multiarch. > > The downside compared to Debian and any binary distribution is that, > everything need to be compiled from source. That's slow. > > > A preliminary draft has been prepared to discuss its use on HPC > environments: > > https://arxiv.org/abs/1610.02742 > > > Alternative projects are _spack_ and _easybuild_, with slightly > different motivations and implementations. > This is good to hear. While I like all that I read about Gentoo, I do not know about how well equipped it is with packages in computational biology [edit: I had a look at https://packages.gentoo.org/categories/sci-biology and am impressed, well done!]. This is where BioConda (bioconda.github.io) is very strong now. And while the Conda community gets increasingly sophisticated with their packaging, we can decide to either just let them go for it or to find ways to compete. These folks start as low as libz, i.e. just above libc, really. I hence tend to think that it is something that we as a Linux distribution should care about. I have followed Johanness' link to https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap but frankly do not yet know how to transform that into something computational biologists would like to use and trust more than Conda. It would all start with cross-distribution packages of dpkg, right? And the testing infrastructure of Debian would need to care for such off-root-experiences, too. I do not see that just around the corner. Best, Steffen
Re: User-installable Debian packages?
Hi Steffen, Steffen Möller writes: > This is good to hear. While I like all that I read about Gentoo, I do > not know about how well equipped it is with packages in computational > biology [edit: I had a look at > https://packages.gentoo.org/categories/sci-biology and am impressed, > well done!]. As a physicist, I am not sure if bioconductors is closely related to computational biology. If so, a complete machine-generated Gentoo overlay for the repository of bioconductor[1] is available as part of R_Overlay[2]. > This is where BioConda (bioconda.github.io) is very strong now. And > while the Conda community gets increasingly sophisticated with their > packaging, we can decide to either just let them go for it or to find > ways to compete. These folks start as low as libz, i.e. just above > libc, really. I hence tend to think that it is something that we as a > Linux distribution should care about. AFAIK, conda uses RPATH and $ORIGIN variable for ELFs, which is similar to Gentoo Prefix-rpath, the more traditional approach than Prefix-libc inside Gentoo. On the other hand, the compatibility issues from libc versions is not negligible. I see several times conda fails to install a python module correctly, e.g. xgboost, because libc on the host being too old. I hope the two camps of people will merge their efforts someday. But I am not optimistic. > I have followed Johanness' link to > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap but frankly > do not yet know how to transform that into something computational > biologists would like to use and trust more than Conda. It would all > start with cross-distribution packages of dpkg, right? And the testing > infrastructure of Debian would need to care for such > off-root-experiences, too. I do not see that just around the corner. I am a long-term Debian user. It was Gentoo Prefix on HPC that introduced me to Gentoo. At the time of adopting Gentoo in my work flow aside Debian, I have long thought of giving dpkg the same power. I concluded that the most portable way to go is to reimplement the Prefix feature of Gentoo portage in Debian debhelper and treat Debian as a source-based distribution. As far as I can see, the easiest relocations are those modifiable in a plain text, like configuration file, script shebang, etc. The medium ones are ELF, with which we have tools like patchelf and chrpath, etc. The most difficult part includes quite a few hidden path assumptions that can only be specified at build time, like the default GCC include dir. I don't know whether distributing binary packages can alleviate the barrier. Chances are many patches are needed to be carried to be able to override paths at run time. (Considering Debian's existing patch base, that might not be a big drawback). For me with a lot of CPU cycles, compiling everything by Gentoo in my home directory is not a problem. At the same time, I really would like to see dpkg can do something similar. Sometimes binary packages are more robust and flexible. Cheers, Benda 1. https://www.bioconductor.org 2. https://wiki.gentoo.org/wiki/Project:Science/Overlay#Scientific_repositories
Re: User-installable Debian packages?
On Sun, 30 Jul 2017 at 19:34:42 +0900, Benda Xu wrote: > As far as I can see, the easiest relocations are those modifiable in a > plain text, like configuration file, script shebang, etc. The medium > ones are ELF, with which we have tools like patchelf and chrpath, etc. > The most difficult part includes quite a few hidden path assumptions > that can only be specified at build time, like the default GCC include > dir. I don't know whether distributing binary packages can alleviate > the barrier. Chances are many patches are needed to be carried to be > able to override paths at run time. Lots of packages hard-code paths at build time. This is often not considered to be a bug. Flatpak's approach to this is to use bind-mounts (in a new mount namespace set up by bubblewrap) so that the "app" (the leaf package, together with any libraries that are bundled with it) always appears to be installed in --prefix=/app, which can safely be hard-coded into binaries that are built as Flatpak apps. Flatpak apps are normally recompiled from source with --prefix=/app, but I don't think it's coincidental that /app is the same length as /usr and can therefore be patched in with a binary editor as a last resort :-) S
Re: User-installable Debian packages?
On 30.07.17 13:47, Simon McVittie wrote: > On Sun, 30 Jul 2017 at 19:34:42 +0900, Benda Xu wrote: >> As far as I can see, the easiest relocations are those modifiable in a >> plain text, like configuration file, script shebang, etc. The medium >> ones are ELF, with which we have tools like patchelf and chrpath, etc. >> The most difficult part includes quite a few hidden path assumptions >> that can only be specified at build time, like the default GCC include >> dir. I don't know whether distributing binary packages can alleviate >> the barrier. Chances are many patches are needed to be carried to be >> able to override paths at run time. > Lots of packages hard-code paths at build time. This is often not > considered to be a bug. > > Flatpak's approach to this is to use bind-mounts (in a new mount > namespace set up by bubblewrap) so that the "app" (the leaf package, > together with any libraries that are bundled with it) always appears > to be installed in --prefix=/app, which can safely be hard-coded into > binaries that are built as Flatpak apps. > > Flatpak apps are normally recompiled from source with --prefix=/app, > but I don't think it's coincidental that /app is the same length as > /usr and can therefore be patched in with a binary editor as a last > resort :-) > Users will not care if it is flatpak, singularity, conda or prefix - they want all the packages and the packages shall work. What I like about all of these efforts is that from what I grasped we will stop caring too much about backports. Also, what is today a bit clumsy to use because of all the dependencies, may become easier: snapshot.debian.org, once it decides to also monitor flatpaks, I mean. What it somewhat boils down to me is that we need to decide about the roles a Linux distribution shall have. And if we want problem-centric communities (like BioConda) to come up with a pan-distributional gentoo-prefix-like setup. The folks that are only after the immediate scientific findings will go for the community-effort. Those aiming for more, and here I see all the effort that goes into * package description + translation * man pages * hardened builds * reproducible builds * continuous integration tests (ok, BioConda does those too, now) * redundancy removal between packages I see our distribution's infrastructure as still relevant and also as an important momentum to drive the developments on upstream's side. To me, Singularity solves quite a bit as of today. But to win the contributors to BioConda of today, we would need to change a lot. Most drastically is the need for immediately relevant user feedback. The conda/brew folks solve that by github forks of their build instructions, which every user can immediately use. Debian falls behind on that front. And even when Debian decides to eventually surface more with its package build instructions, we would not have anything in place for users that want the binary update _now_ and not after an upload plus a likely manual review. So, for embracing our users more, we need to * allow them to install packages as users, not only as admins, which is what this thread is mainly about * allow them to seamlessly give feedback from which they can immediately benefit, which I do not see how to address without an autogenerated launchpad. Need to think about it all, many thanks for all your constructive feedback Steffen
Bug#870146: ITP: node-ip -- IP address utilities for node.js
Package: wnpp Severity: wishlist Owner: suman X-Debbugs-CC: debian-devel@lists.debian.org * Package name : node-ip Version : 1.1.5 Upstream Author : Fedor Indutny * URL : https://github.com/indutny/node-ip * License : Expat Programming Lang: JavaScript Description : IP address utilities for node.js ip utility in node.js helps one with ip related queries.Node-ip helps in quickly accesing ip address. . Ip utility helps fetch results on ip address, comapre ip address, validate ip address, range checking, subnet information etc. . Node.js is an event-based server-side JavaScript engine. An Internet Protocol address (IP address) is a numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication.Node-ip helps in quickly accesing ip address etc. Praveen agreed to sponser this package.
Re: User-installable Debian packages?
Hi Simon, Simon McVittie writes: > Flatpak's approach to this is to use bind-mounts (in a new mount > namespace set up by bubblewrap) so that the "app" (the leaf package, > together with any libraries that are bundled with it) always appears > to be installed in --prefix=/app, which can safely be hard-coded into > binaries that are built as Flatpak apps. I can see the use cases for desktop, but this is the restriction of Flatpak for shared HPC servers: not all administrators are willing to grant the users the seccomp and permission for creating new namespaces, and not all administrators will upgrade or recompile kernels to support namespaces. If /app is not available, it is difficult for a user to override the hardcoded /app of Flatpak into /home/user/app. In principle, we can create an _appdebian_ by hardcoding /app to every debian package, not unlike hardcoded /system in Android systems. Cheers, Benda
Bug#870160: ITP: golang-github-approvals-go-approval-tests -- ApprovalTests for go
Package: wnpp Severity: wishlist Owner: Shengjing Zhu * Package name: golang-github-approvals-go-approval-tests Version : 0.0~git20170712.0.c1e747e Upstream Author : approvals authors * URL : https://github.com/approvals/go-approval-tests * License : Apache-2.0 Programming Lang: Go Description : ApprovalTests for go Golden master Verification Library. . ApprovalTests allows for easy testing of larger objects, strings and anything else that can be saved to a file(images, sounds, csv, etc.) It's the new dependency for packer's[1] test suite. I will package this inside pkg-go team and need sponsor as well. [1] https://tracker.debian.org/pkg/packer Regards, Shengjing Zhu signature.asc Description: PGP signature
Bug#870161: ITP: stendhal -- Multiplayer online adventure game with an old school feel
Package: wnpp Severity: wishlist Owner: Carlos Donizete Froes * Package name: stendhal Version : 0.1 Upstream Author : Carlos Donizete Froes * URL : https://github.com/coringao/stendhal * License : GPL-2+ Programming Lang: Shell Description : Multiplayer online adventure game with an old school feel Features a new, rich and expanding world in which you can explore towns, buildings, plains, caves and dungeons. . You will meet NPCs and acquire tasks and quests for valuable experience. . Your character will develop and grow with each new level up become stronger and better. . With the money you acquire can buy new items and improve your armour and weapons.
Re: building a debian kernel package(s) for a foreign architecture
On Thu, Jul 6, 2017 at 2:53 AM, Karsten Merker wrote: > > IIRC both pbuilder and sbuild automatically set the "nocheck" > profile when crossbuilding. For crossbuilding the kernel IIRC > one needs to set a number of additional build-profiles: > > - cross > - pkg.linux.notools > - nopython > > So the following sbuild command should in theory (sorry, cannot > actually test that right now) crossbuild the kernel package for > arm64: > > sbuild -d unstable --host=arm64 --profiles=cross,pkg.linux.notools,nopython > .dsc Thanks so much for the info! Especially the "nopython" option, which was confusing at first glance. I don't see the profile setting in kernel handbook, but only in the source: - https://anonscm.debian.org/cgit/kernel/linux.git/tree/debian/README.source So I updated my wiki entry: - https://wiki.debian.org/HowToCrossBuildAnOfficialDebianKernelPackage Now I confirm it builds only image and header 2 debs when targeting "UNRELEASED". Cheers, -- Roger Shimizu, GMT +9 Tokyo PGP/GPG: 4096R/6C6ACD6417B3ACB1
Bug#870191: ITP: golang-github-opentracing-opentracing-go -- Go platform API for OpenTracing
Package: wnpp Severity: wishlist Owner: =?utf-8?q?Mart=C3=ADn_Ferrari?= * Package name: golang-github-opentracing-opentracing-go Version : 1.0.2 Upstream Author : The OpenTracing Authors * URL : http://opentracing.io/ * License : MIT Programming Lang: Golang Description : Go platform API for OpenTracing By offering consistent, expressive, vendor-neutral APIs for popular platforms, OpenTracing makes it easy for developers to add (or switch) tracing implementations with an O(1) configuration change. OpenTracing also offers a lingua franca for OSS instrumentation and platform-specific tracing helper libraries. This package is a Go platform API for OpenTracing.
Bug#870195: ITP: golang-github-opentracing-contring-go-stdlib -- OpenTracing instrumentation for packages in the Go stdlib
Package: wnpp Severity: wishlist Owner: =?utf-8?q?Mart=C3=ADn_Ferrari?= * Package name: golang-github-opentracing-contring-go-stdlib Version : 0.0+git201705028.48e4d76 Upstream Author : opentracing-contrib * URL : https://github.com/opentracing-contrib/go-stdlib * License : BSD-3 Programming Lang: Golang Description : OpenTracing instrumentation for packages in the Go stdlib This package contains OpenTracing (http://opentracing.io/) instrumentation for packages in the Go standard library. Instrumentation is provided for the following packages, with the following caveats: * net/http: Client and server instrumentation. Only supported with Go 1.7 and later.
Re: IMPORTANT: Do live Debian images have a future?
Hi, On Mon, 26 Jun 2017 15:08:21 +0100 Steve McIntyre wrote: > While the bugs are annoying, what worries me more is that they were > only spotted in release builds. There had been testing versions of > live images available for multiple weeks beforehand, presumably with > the same bugs included. (Almost) none of them reported. This shows > that we don't have enough people using these live images and/or caring > about filing bugs. It's hard, so hard. Because most of users don't use it until it is "Released" - Released, then complain about it (of course, there is an exception, but... ;) So, we should consider "automated testing", IMO. openSUSE people uses "openQA" suite, that seems to be useful. http://open.qa/ https://www.youtube.com/watch?v=2K6ZmvIOtPw If we can put it into Debian, CI for live image would be easy. Some of components are already in Debian (that I did), but not all of them. Still working on it, I'll try my best. -- Regards, Hideki Yamane henrich @ debian.or.jp/org http://wiki.debian.org/HidekiYamane
Bug#870218: (no subject)
Package: wnpp Severity: wishlist Owner: Yinhang Liu * Package name: libxcam Version : 1.0.0 Upstream Author : Wind Yuan * URL : https://github.com/01org/libxcam * License : Apache-2.0 Programming Lang: C, C++ Description : Extended camera features and cross platform computer vision project libXCam is a project for extended camera features and focus on image/video quality improvement. There are lots features supported in image pre-processing, image post-processing and smart analysis. This library makes GPU/CPU working together to improve image quality. OpenCL is used to improve performance in different platforms.
Bug#870219: ITP: libxcam -- Extended camera features and cross platform computer vision project
Package: wnpp Severity: wishlist Owner: Yinhang Liu * Package name: libxcam Version : 1.0.0 Upstream Author : Wind Yuan * URL : https://github.com/01org/libxcam * License : Apache-2.0 Programming Lang: C, C++ Description : Extended camera features and cross platform computer vision project libXCam is a project for extended camera features and focus on image/video quality improvement. There are lots features supported in image pre-processing, image post-processing and smart analysis. This library makes GPU/CPU working together to improve image quality. OpenCL is used to improve performance in different platforms.
Re: New package split of util-linux
On Thu, Jul 27, 2017 at 12:32:24PM +0200, Andreas Henriksson wrote: > On Wed, Jul 26, 2017 at 11:03:08AM +0200, Adam Borowski wrote: > > But why should mount be Essential? It's useless in containers and chroots. > > I'm very open to making things non-essential if possible, not limited to > only mount. (Why should bsdutils be essential for example? But how do we > make it non-essential? Even if policy didn't forbid depending on > essential packages, how would we even identify users that should add > a dependency? See also #474540 for another example of this practical > problem.) https://wiki.debian.org/BusterPriorityRequalification has some interesting discussion. Making things non-Essential is indeed hard, but making them Important[1] is easier as for most tools these two are considered synonymous. As we're very early in Buster's cycle, a somewhat cavalier approach could be acceptable: to degrade programs and see what breaks. It's safe for upgrades (tools really dislike removing Important) and for new regular whole-machine installs; only uses that might have issues are chroots and small containers. > I aware of but have no practical experience with the Important: yes > field. I can only guess and hope that if we use that for mount there > won't be any weird upgrade problems. (Help welcome to verify it!) I've done some research, and: * user-facing tools support Important well enough, as of Stretch: while dpkg itself doesn't protest, frontends like apt, dselect, aptitude, synaptic, gnome-packagekit, apper are all _too_ paranoid here. This is actually good for ordinary users -- converting a full machine to a container isn't exactly a common use case. The only frontend that doesn't stop you is cupt, but with popcon vote of 23 it's not much of a concern, especially that you tend to remove Buster packages using Buster tools. * on the other hand, current dpkg-gencontrol does not support this field * the Policy doesn't mention it either Thus: it's not legal to use Important yet but there are no blockers to have it for Buster. If you'd like to play some more, my test packages are in: deb http://angband.pl/debian essimp main (-src, https), key: wget -qO- https://angband.pl/deb/archive.html|apt-key add - "test-essential", "test-important" > > What about making the split at different levels of essentialness? With the > > new Important: yes (not be confused with priority: important), this makes > > three levels, and thus three packages: > > * stuff that's needed on every Debian system > > * stuff needed on every bare-metal box / "real" VM > > * things you can go without > > I would be very interested to see your resulting of this splitup! As Steve Langasek mentioned, not all containers are alike. Unlike old-style implementations like vserver or openvz (even they let you allow ccaps SECURE_MOUNT and BINARY_MOUNT), modern containers are a really fuzzy concept, with a mix-and-match of chroot, cgroups, namespaces, seccomp. But there's a pretty common set of things typical containers don't need. They don't need to manipulate partitions, fsck, etc. That's most of util-linux's size. > In theory I'm not really sure there's anything matching the "stuff > that's needed on every Debian system" criteria in src:util-linux > if considering less usual usecases.) Yeah but a container has a host which can rescue it, and then you can install anything you want into the container. So it's about minimal functionality only. Meow! [1]. A poorly chosen name, because of confusion with priority:important. But AFAIK it was repurposing an ancient hidden feature rather than a new design. -- ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs (the five fishes + two breads affair) ⠈⠳⣄ • use glitches to walk on water
