This patchset makes it possible to enable profile-guided-optimization (PGO) for Python 2.7 - both native and target recipes.
Enabling PGO for python-native is simple, one patch, basically just changing the make target if PGO is enabled in build conf. However, the target version requires a lot more convoluted approach because of the cross-build environment. Normally, (python-native in our case) PGO is done in one go by just running "make profile-opt". It first builds python binaries with profile instrumentation enabled. Then, a specific profile task, using that python, is run to get the profile data. Last, a complete re-build of python is done, feeding the compiler with the profile data obtained earlier. Unfortunately, in our cross-build environment this is not possible as the binaries are (cross-)built on the host machine but profiling needs to be done on the target architecture. The approach used in this patchset splits the PGO build of target python into three steps: 1. build a target python binary with profile instrumentation enabled 2. build a special target image and run the profile task there, copying the profile data back to the build host 3. build an optimized target python binary, guided by the profile data The first step is done by adding a new "python-profile-opt" recipe. It builds a special version of python that has the profile instrumentation enabled. The idea behind this is to keep "normal" python clean and isolated - not do any intermediate builds or contaminate python with pgo instrumentation. The second step is implemented by adding a new image recipe. This is a minimal image that basically only contains the functionality for running the special profile-instrumented python and obtaining the profile data. It re-uses facilities from oeqa to communicate with the target instance over ssh. Also here, the idea behind a separate "profiling image" is to keep it isolated and not contaminate the main target build. The third step is relatively straightforward, enabling the usage of profile data in do_compile in the python recipe. Instructions for doing a PGO build for target python are: 1. enable profile-optimized build by specifying PYTHON_PROFILE_OPT = "1" in local.conf 2. get profile data by running bitbake python-pgo-image -c profile 3. (re-)build your distro image Benchmark results from my i7-3770K desktop machine show 8% to 20% improvement for python-native and 3.6% to 15% improvement for python (qemux86 target). python-native: pystone: 141883 -> 168168 pystones/s (+18%) pybench avg: 3367 -> 2795 ms (-20%) regrtest time: 270 -> 250 s (-8%) python (target): pystone: 2045 -> 2355 pystones/s (+15%) pybench avg: 181901 -> 175600 ms (-3.6%) regrtest time: 4210 -> 4050 s (-3.8%) I've not done extensive testing to get statistically reliable results, but, the numbers above should give a useful enough ballpark figure. New configuration variables, related to this patchset are: PYTHON_PROFILE_OPT - set to "1" to enable PGO build of python PYTHON_PROFILE_TASK - profile task to run for python PYTHON_NATIVE_PROFILE_OPT - set to "1" to enable PGO build of python-native PYTHON_NATIVE_PROFILE_TASK - profile task to run for python-native I have a corresponding patchset for Python3, too, and I will send that shortly. The following changes since commit fc97963bc61bf16112859fe1d7e460a13d34baca: oeqa/selftest/devtool: rewrite modify testcase (2017-01-19 22:45:46 +0000) are available in the git repository at: git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338-py2 http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338-py2 Markus Lehtonen (10): python-native: support profile optimized build python: add python-profile-opt recipe python: remove path hack from setup.py python-profile-opt: rename libpython devtools/images: add python-pgo-image python: make profile-optimized build possible python-pgo-image: exclude tests from the default profile target python: add python-tools subpackage python-pgo-image: switch python default profile task to pybench python-pgo-image: enable sstate for do_profile meta/recipes-devtools/images/python-pgo-image.bb | 75 +++++++++++++++++ .../python/python-2.7-manifest.inc | 8 +- ...x-.so-loading-when-when-running-profile-t.patch | 26 ++++++ .../python/python-native_2.7.12.bb | 10 +++ .../python/python-profile-opt_2.7.12.bb | 13 +++ .../01-use-proper-tools-for-cross-build.patch | 10 --- ...rename-libpython-to-libpython-profile-opt.patch | 94 ++++++++++++++++++++++ meta/recipes-devtools/python/python_2.7.12.bb | 54 +++++++++++-- 8 files changed, 269 insertions(+), 21 deletions(-) create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb create mode 100644 meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch create mode 100644 meta/recipes-devtools/python/python-profile-opt_2.7.12.bb create mode 100644 meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch -- 2.10.2 -- _______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
