Hi Sean, Thanks for your feedback. Below are: * some responses to your questions * a description of some major changes recently incorporated in the package * some questions about the long-term plans of a fully featured (non-minimal) Chapel package.
Note that we have upgraded our package to Chapel 1.15, and changed the source package name to 'chapel' rather than 'chapel-minimal', which can be found here: https://mentors.debian.net/package/chapel Responses to questions ---------------------- > > 7. The package is not compliant with the FHS. Almost everything is > > installed into /usr/lib/chapel, plus a symlink from /usr/bin/chpl into > > /usr/lib/chapel (only to the 64-bit version?). The main executable > > needs to be installed into /usr/bin/chpl. Please take a look at Debian > > policy section 9.1.1 and install the files appropriately. > The basic distinction is that /usr/share is for arch-independent files, > /usr/lib for arch-dependent files or a mix of arch-independent and > arch-dependent if necessary. > The CHPL_HOME problem ought to be fixed by patching chapel to look in > different places for these different resources. For example, you can > have it look in /usr/share/chapel for the modules/ subdir. Is there > something about chapel's design that makes this difficult? We're investigating updating Chapel to work with an installation structure more normal for Debian. We're currently pursuing: runtime (.a and .o files) in /usr/lib/chapel/1.15/runtime/lib runtime includes (.h files) in /usr/lib/chapel/1.15/runtime/include compiler binary (chpl) in /usr/bin modules/, utils/, make/ in /usr/share/chapel/1.15 > You are installing lots of *.h, *.c and *.py files into > /usr/lib/chapel. What is the purpose of this? With the changes we are working on, we will no longer install .py files to /usr/lib/chapel. We'll install the .py files to /usr/share/chapel. We will however continue to install some .h files to /usr/lib/chapel or /usr/share/chapel. We will explain more in answer to your next questions. There is a single .c file we might need to also install to /usr/lib/chapel. See below. > How is the C source code necessary for compiling chapel programs? > *.h files are supposed to be in /usr/include/ ... again, could you > explain how they are necessary for compiling chapel programs? The Chapel compiler normally functions by translating Chapel source code to C. The headers in /usr/lib/chapel/runtime/include are necessary for this generated C code to build with the Chapel runtime that is already built (and stored in /usr/lib/chapel/runtime/lib). These headers could be architecture-specific, so we intend to store them in /usr/lib/chapel rather than /usr/share/chapel. These headers are not meant to be user-visible for general C program development. Instead, they are only useful internally to the Chapel compiler. For this reason, we think it would be a mistake to make them generally available in /usr/include. In some configurations (namely with LLVM support), the Chapel compiler needs a sigle .c file in order to function correctly, as it is building upon the work of a C compiler. For this reason, we may include a C file in /usr/lib/chapel or /usr/share/chapel. The other C files you observed have been removed from the upstream project. We can see in GCC a precedent for this sort of thing; for example there is a /usr/lib/gcc/x86_64-linux-gnu/6.3.0/include directory on my system. Based on discussion in #debian-mentors, we're including a version number as in /usr/lib/chapel/<vers> and /usr/share/chapel/<vers>. The reason for this is that since Chapel is a language, people often need to be able to install more than one version at a time. > > > 11. I'm not an expert on multi-arch issues but this package seems to be > > > targeted only at 32-bit and 64-bit Linux machines, just from looking at > > > the 'linux32' and 'linux64' directories you have a lot of. Debian > > > supports lots of other architectures, and the package should work on > > > those. Is that something that can be fixed? > > I have not made any changes in response to this feedback. > > This is a way in which Chapel distinguishes platforms, which are independent > > of the architectures Debian distinguishes. I would expect all > > Debian-supported > > architectures to be categorized as either 'linux32' or 'linux64' platforms > > for Chapel's purposes. > > For a list of our platforms, see our documentation: > > > > http://chapel.cray.com/docs/1.14/usingchapel/chplenv.html#chpl-host-platform > > It seems that things have changed since we last spoke. I built the > package and I now get paths like this: > > /usr/lib/chapel/runtime/src/threads/pthreads/gen/linux64.gnu.arch-native.loc-flat.comm-none.tasks-fifo.tmr-generic.unwind-none.mem-cstdlib.atomics-intrinsics.gmp-none.hwloc-none.re-none.wide-struct.fs-none/ > > According to the docs you linked to, a value of CHPL_HOST_PLATFORM is > meant to be much shorter than this ... or am I misunderstanding? The paths of built runtime libraries are encoded with the build-configuration information. That is what you are observing above. Note that, of all of that configuration information, 'linux64' is the only platform-distinguishing configuration. The other path components reflect the configuration of Chapel that was built. Do you feel that we have addressed your concerns about the FHS? Major changes in package ------------------------ As noted above, we've pursued a more traditional 'make install' approach for Chapel installation to conform to the FHS more closely. This patch is currently under review upstream as well. Additionally, we have reduced the amount of contents stripped out of the source package, and split our package up into several packages, such as 'chapel-minimal-compiler', 'chapel-minimal-runtime', and 'chapel-doc'. This is anticipating the desire to distribute multiple build configurations for Chapel that can all be built from a single source package, e.g. a non-minimal Chapel. The license files within copyright have been updated to more of the third-party licenses that we are bundling (see next section for more details). You can track the progress of these changes at: github.com/chapel-lang/chapel-packaging. Do these changes sound good to you? How to create a non-minimal Chapel package? ------------------------------------------- 1. Chapel uses the Sandia qthreads user-level tasking library by default. But, it is not enabled in the minimal package we are building so far. As a result, Chapel programs using the minimal package will have worse performance and will potentially deadlock if they use too many tasks. The process of building a Chapel source release normally compiles qthreads with non-default configurations for functionality and performance. Functionality configurations include: guard pages, scheduler, disable-spawn-cache, and with-hwloc, and numa-vs-flat. A qthreads Debian package does not exist, and would be very complex, given the matrix of configurations it would need to support as packages. For this reason, we believe it would be best for our source package to rebuild qthreads in the configuration that Chapel supports. We could potentially: * re-build qthreads from source in our package build process * make a chapel-qthreads package built with our configuration * ... something else you might know about ? Note that we've updated the Debian package to includes the qthreads source code and the appropriate entries in the copyright file. We're hoping for feedback on that change. 2. Chapel includes a fork of re2 that includes patches necessary for complete formatted I/O functionality. Since the minimal package does not include re2 or these patches, use of the Regexp library as well as some formatted I/O features will not function. We attempted to contribute these patches to the main re2 project, but the upstream maintainer doesn't appear to be willing to accept the changes. How should we handle this situation? Note that these changes add to the RE2 API but do not affect existing functionality. See https://groups.google.com/forum/#!topic/re2-dev/gyZ6ukLJm30 We could potentially: * re-build re2 from source in our package build process * make a chapel-re2 package built with our fork * ... something else you might know about ? We will need to answer similar questions for other dependencies besides qthreads and re2. To give a sense of the challenges in this area, we have created a table that shows how Chapel uses third-party code: patches = Non-upstream patches applied for functionality/performance configs = Configured in a non-default way for functionality/performance dpkg = Debian package exists sys-support = Chapel can support a system-installed version, e.g. CHPL_GMP=system. If not, a patch would be required to depend on an external Debian package. func = Necessary for functionality perf = Improves performance port = Expands portability (to ARM in the one case below) ======================================================= | 3-party | patches | configs | dpkg | sys-support| |-----------------------------------------------------| | qthreads | none | func, perf | N | N | | jemalloc | none | perf | Y | WIP | | hwloc | none | default | Y | WIP | | gasnet | port | port, perf | N | N | | gmp | perf | default | Y | Y | | re2 | func | default | Y | N | | llvm | none* | default | Y | Y | ======================================================= * the Chapel LLVM support currently uses an LLVM header file that is not installed. Note that there are some other minor caveats not covered in this table, e.g. jemalloc Debian package is 3.6 and Chapel relies on 4.5 for full functionality. How should we handle the third-party dependencies as we expand beyond just chapel-minimal? Look forward to hearing back from you. Thanks, Ben
