Hey Josh, I had a look into this/pex/subpar a bit more:
- external pex rules aren't used (e.g. benley/bazel_rules_pex <https://github.com/benley/bazel_rules_pex> - although it looks like the heron tools/rules/pex/pex_rules.bzl <https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl> was copied+pasted from the same source at some point - the earliest copy in heron doesn't appear in the remote repo so I guess with changes). Although the public repo hasn't been touched in months, there isn't an issue (yet?) for bazel 0.20.0 - content of pex files are imported from by the pex_loader module maintained in heron - I couldn't find the code via google, but maybe it's something that could live in pex <https://github.com/pantsbuild/pex/tree/master/pex> - the pex format appears stable (or inactive?) so hopefully we'll be ok for now - pex's design documentation <https://www.pantsbuild.org/pex_design.html> doesn't list subpar as an alternative - I made an issue for clarification <https://github.com/pantsbuild/pantsbuild.github.io/issues/10> I'm under the impression that switching out pex for par would be quite a bit of work (and not sure yet if it introduces more limitations), including a making a par_loader and also deciding on maintaining both for a while or dropping pex. Moving away from the loaders may be nice so we aren't tied to a language if possible, but I haven't looked into the how it is used, I think elsewhere the interface is a program argument/flag and protobuf so that might be an option. I don't know if there's any required customisation within the heron copy of pex_rules.bzl (I hope not), so it may be worth trying to drop in the public rules and using those, then at least the rules should be easy to update and there's a better shot at community support. Does using bazel_pex_rules repo sound passable? Thanks, Oliver On Sat, 8 Dec 2018 at 13:43, Josh Fischer <[email protected]> wrote: > I'm open to any suggestions.. I'm not much of a python developer, so my > knowledge is limited in this part of the project. I appreciate the > thoughts and ideas. Thanks Oliver. Let us know what you come up with. > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <[email protected]> > wrote: > > > Google's subpar may be an option as a replacement for .pex - I imagine > it'd > > have better support in Bazel/generally. Has anyone got any thoughts on > > pex/subpar? I can ponder more and look up stuff next week > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <[email protected] wrote: > > > > > Hey All, > > > > > > I've spent little bit of time over the past day or two working to allow > > > Heron to be built with newer versions of Bazel (at this time v-0.20.0). > > > I've managed to restructure certain files within the Heron to project > to > > > comply with what Bazel is expecting.. I think I have most of it worked > > out > > > with one big exception (I think) of the python pex rules that were > > written > > > by the community and seem to be several months old. I do not know > enough > > > about PEX to update this custom rule and verify it works as expected.. > I > > am > > > afraid that if we don't address this issue soon will be stuck with an > > older > > > version of Bazel moving forward.. Another thought I had: "is it > > required > > > that we use pex"? Is there a way we could get away from it and just > use > > > supported official Bazel rules if this issue is too hard to fix or > > > maintain? > > > > > > Here is my output when building > > > > > > bazel build --config=darwin heron/... > > > --incompatible_remove_native_http_archive=false > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded > > > > > > INFO: Build options have changed, discarding analysis cache. > > > > > > ERROR: > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > in eggs attribute of pex_binary rule > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > '@pytest_whl//file:file' does not produce any pex_binary eggs files > > > (expected .egg or .whl). Since this rule was created by the macro > > > 'pex_pytest', the error might have been caused by the macro > > implementation > > > in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > ERROR: > > > > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1: > > > in eggs attribute of pex_binary rule > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner: > > > '@py_whl//file:file' does not produce any pex_binary eggs files > (expected > > > .egg or .whl). Since this rule was created by the macro 'pex_pytest', > the > > > error might have been caused by the macro implementation in > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14 > > > > > > ERROR: Analysis of target > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed; > > > build aborted: Analysis of target > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed; > > > build aborted > > > > > > ignore the flag --incompatible_remove_native_http_archive=false I'm > > using > > > that ignore a warning with http_archive targets temporarily while I > work > > > through this python issue. I have a fork of heron at > > > *https://github.com/joshfischer1108/incubator-heron > > > <https://github.com/joshfischer1108/incubator-heron>. *The branch I > am > > > working off of within that repo is support-bazel-0.20.0 > > > < > > > > > > https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0 > > > >. > > > If anyone has any experience with python and could possibly look over > > this > > > and help me get Heron up to speed with the most recent version of > Bazel I > > > would greatly appreciate it.. Just a FYI, as this is still a work in > > > progress you'll notice I have some duplicate folder structures and > files > > in > > > the repo right now.. This on purpose.. Bazel has made some changes that > > > were breaking.. > > > > > > > > > Thanks, > > > > > > Josh > > > > > >
