On Wed, Aug 16, 2017 at 10:15 AM, Dustin Mitchell <dus...@mozilla.com>
wrote:
> This feels like a project that could benefit from a number of people
> coordinating a slow hacking-away at it, with the leaf nodes of that
> dependency tree being good bite-sized projects suitable for hacking on in
> spare cycles or for casual contributors. In my experience with other
> projects, one of the important pieces to making forward progress on Py3
> migrations is to use automated testing to prevent regressions. That
> requires chasing specific milestones like "all print statements removed and
> lint configured to reject them".
>
> Are there plans for some kind of coordination? I'd like to be involved
> hacking on those leaf nodes, but I don't have the breadth of understanding
> to figure out where to start.
>
We had a meeting a few weeks ago and we definitely want to establish some
kind of "lint" test that ensures basic Python 3 conformance once a
package/module has achieved it. Bug 1388013 and associated bugs are
starting to lay the groundwork for running Python unit tests with Python 3.
Once we're doing that, it shouldn't be too much effort to write a test for
Python 3 conformance.
FWIW, the general process Mercurial took for Python 3 compatibility looked
like:
1) Make all .py files tokenize/ast compile on Python 3
2) Make all .py files import cleanly on Python 3
3) Make all .py files run properly on Python 3
I think that worked out pretty well and recommend repeating the pattern at
Mozilla. Having a Python module import cleanly unblocks usage of just a
single function in that module from Python 3, which for many consumers is
all you need. In terms of tests, we'll want to establish "gates" where once
a module passes a tokenize/ast or import threshold, it stays that way.
> 2017-08-16 12:54 GMT-04:00 Gregory Szorc <g...@mozilla.com>:
>
>> In the past few weeks, a few independent projects have inquired about
>> using Python 3 for Firefox build/development tooling.
>>
>> tl;dr we now have bug 1388447 (aliased as "buildpython3") tracking
>> everything Python 3 related to Firefox development. Please use it.
>>
>> My thoughts on Python 3 and Firefox follow.
>>
>> Python 2 will stop being supported in 2020 (exact date TBD). In theory,
>> there will be a 2.7.X release in 2020 and no more official Python 2
>> releases will be made. It is therefore prudent for any actively-developed
>> Python code to transition away from Python 2 so it remains on a supported
>> programming language. Python 2 won't disappear over night and is likely to
>> be a fixture of operating system installs for years to come. But as time
>> goes by, it will be more and more annoying having to support legacy Python
>> 2 code. Any developer who spends a lot of time writing Python will likely
>> scoff at the thought of maintaining Python 2 code after they are accustomed
>> to Python 3. Python 2 code will tend to rot from neglect. And, running
>> Python 2 code will likely expose you to unpatched security vulnerabilities.
>> It's best to be off of Python 2 by its end of life in 2020.
>>
>> That being said, you don't need to transition to Python 3 and nobody is
>> forcing you to transition to Python 3. While there are gains from
>> transitioning, it is a lot of work. In many cases, porting is of the same
>> scope as a rewrite and therefore many of the reasons why rewrites are a bad
>> idea apply to porting to Python 3. So it would be prudent to conduct a
>> cost-benefit analysis on porting, paying special attention to what other
>> work is being neglected because a port is being done.
>>
>> IMO Python 3 is a superior programming language to Python 2. The run-time
>> performance of Python 3 is substantially better than Python 2 as of the 3.6
>> release (although start-up time is still an issue). There are compelling
>> Python 3 features that make it attractive. These include support for type
>> hinting, which combined with a type aware editor like PyCharm (which
>> Mozilla employees can get a license for), should facilitate easier hacking
>> on large code bases like the Firefox build system. Asyncio allows much
>> higher concurrency for I/O bound tasks (which there are several of in the
>> build system and tools). And there are a ton of standard library
>> improvements that reduce code complexity, attain perf wins, lower
>> development overhead, etc. In aggregate, these make large, actively
>> developed, performance sensitive code the most lucrative targets to port.
>>
>> We should be encouraging the use of Python 3 for Firefox tooling. I dare
>> say that new Python code should strive to be Python 3 only if possible, as
>> new Python 2 in 2017 is a form of technical debt. At the very least, I
>> encourage all newly-written Python code to be as Python 3 compatible as
>> possible to minimize future porting work.
>>
>> Requiring Python 3 to build Firefox incurs a non-trivial amount of work,
>> especially to downstream packagers. Bug 636155 tracks *requiring* Python 3
>> to build Firefox. A summary of my thoughts are captured in comment #9. That
>> milestone likely won't happen until 2018Q2 or later.
>>
>> However, we don't need to require Python 3 to build Firefox in order to
>> use Python 3 in Firefox tooling. Non-essential development activities can
>> adopt Python 3 sooner. Examples of non-essential development activities
>> include mach commands not used by the build system, such as `mach watch`.
>> Strictly speaking, I even consider testing harnesses non-essential, as they
>> aren't required to *build*. (Although I'm not sure how much testing
>> downstream packagers do and we need to be cognizant about introducing a
>> Python 3 requirement on them.)
>>
>> Bug 1385381 recently landed to detect Python 3 in the build system. The
>> PYTHON3 subst variable, if defined, will point to a Python 3.5+ executable.
>> The MozbuildObject base class now exposes a `python3` attribute. It returns
>> a 2-tuple of (path, version). So, from mach command classes inheriting from
>> MachCommandBase and various build system contexts, you can now do something
>> like:
>>
>> python3 = self.python3[0]
>> if not python3:
>> raise Exception('Python 3 not found')
>> subprocess.check_call([python3, ...])
>>
>> Mach itself is still Python 2 and likely will be for some time. There
>> have been some discussions around making it work with Python 3. We may even
>> facilitate light magic to allow mach @Command functions to transparently be
>> invoked as Python 3. But for now, you'll have to invoke a separate Python 3
>> process from mach commands to use Python 3.
>>
>> There is a group interested in porting various Mozilla packages (like
>> mozbase) so they are Python 3 compatible. Porting things like mozprocess
>> will be a prerequisite for making large amounts of the Python in the
>> Firefox repo Python 3 compatible.
>>
>> Managing a flag day transition to Python 3 across the entire Firefox code
>> base is not feasible: it is just too much work. The conversion to Python 3
>> will be gradual and take years. Expect Python 2 and 3 to run side-by-side
>> for years, well past 2020. Components will adopt different schedules and
>> priorities for the transition. Early adopters of Python 3 will likely be
>> hindered by various package dependencies not yet working on Python 3. For
>> this reason, early Python 3 efforts should be focused on shared packages
>> like mozbase and new code that doesn't need too much of the in-tree Python
>> (which is likely unported).
>>
>> Because Python 2 and 3 will run side-by-side for years, shared packages
>> should strive to run on both Python 2 and 3. This is achievable in many
>> cases. Packages like six do a great job of wallpapering over differences.
>> In cases where the same code can't run on both 2 and 3 simultaneously, we
>> can look at using source rewriting. I have some experience with this and
>> you can reach out if you have any questions.
>>
>> Again, nobody is forcing you to use or port to Python 3. But Python 3 is
>> the future and anyone writing or maintaining a significant amount of Python
>> should be cognizant of the technical debt being incurred by continuing to
>> write Python 2 in 2017 and beyond. We should start thinking "Python 3
>> first" and start planning for a world that is "Python 3 everywhere." For
>> most of you, it should be business as usual for the next several months.
>> But once more Python 3 "infrastructure" is in place in Firefox tooling,
>> expect the calls for Python 3 to grow louder. There's no stopping the
>> Python 3 train. So please be aware that it's coming.
>>
>> _______________________________________________
>> dev-builds mailing list
>> dev-builds@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-builds
>>
>>
>
_______________________________________________
dev-builds mailing list
dev-builds@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds
