[Python-Dev] first post introduction and question regarding lto

[Soldea, Octavian]

Hello

This is my first post after I just have subscribed to the Python-Dev mailing 
list. In this context, the welcome confirmation message suggested to introduce 
myself.

My name is Octavian Soldea and I am with the Python optimization team in DSLOT 
group in Intel, Santa Clara, California. In the past I have received a PhD in 
Computer Science from the Technion, Israel Institute of Technology in Computer 
Vision.

I would like to ask regarding the possibility of compiling Python with lto 
only. My question is if is the community interested to enable the possibility 
of compiling Python with lto only, i.e. without pgo? I have followed
https://bugs.python.org/issue29243
and
https://bugs.python.org/issue29641
however, it is not clear if the separate compilation is already implemented. In 
my opinion this is a good option since pgo is not always desired.

Best regards,
  Octavian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Gregory P. Smith]

I've personally never seen a situation where PGO is not desired yet some
other fancier optimization such as LTO is. When do you encounter people
wanting that?  PGO always produces a 10-20% faster CPython interpreter.

I have no problem with patches enabling an LTO only build for anyone who
wants one if they do not already exist.  At the moment I'm not sure which
LTO toolchains actually work properly.  We removed it from inclusion in
--enable-optimizations because it was clearly broken in some common
environments.

If LTO without PGO is useful, making it work is probably just a matter of
making sure @LTOFLAGS@ show up on the non-PGO related Makefile.pre.in
targets as well as updating the help text for --with-lto in configure.ac.
 [untested]

-gps


On Mon, Aug 7, 2017 at 12:00 PM Soldea, Octavian 
wrote:

>
>
> Hello
>
>
>
> This is my first post after I just have subscribed to the Python-Dev
> mailing list. In this context, the welcome confirmation message suggested
> to introduce myself.
>
>
>
> My name is Octavian Soldea and I am with the Python optimization team in
> DSLOT group in Intel, Santa Clara, California. In the past I have received
> a PhD in Computer Science from the Technion, Israel Institute of Technology
> in Computer Vision.
>
>
>
> I would like to ask regarding the possibility of compiling Python with lto
> only. My question is if is the community interested to enable the
> possibility of compiling Python with lto only, i.e. without pgo? I have
> followed
>
> https://bugs.python.org/issue29243
>
> and
>
> https://bugs.python.org/issue29641
>
> however, it is not clear if the separate compilation is already
> implemented. In my opinion this is a good option since pgo is not always
> desired.
>
>
>
> Best regards,
>
>   Octavian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Soldea, Octavian]

Hello

Gregory: Thank you for your message. I really appreciate it.

In my opinion the use of lto only compilation mode can be of benefit from two 
reasons at least:

1.   Lto only can provide a platform for experimentations. Of course, it 
seems to be very application specific.

2.   Lto compilation is faster than an entire pgo + lto mode. While pgo can 
take a lot of time, lto can provide significant optimizations, as compared to 
baseline, at the cost of less compilation time. Of course, lto is time 
consuming. To the best of my knowledge, compiling with lto only is shorter than 
combining pgo and lto.

In this context, I will be more than happy to receive more feedback and 
opinions.

Best regards,
  Octavian




From: Gregory P. Smith [mailto:g...@krypto.org]
Sent: Monday, August 7, 2017 3:12 PM
To: Soldea, Octavian ; Python-Dev@python.org
Subject: Re: [Python-Dev] first post introduction and question regarding lto

I've personally never seen a situation where PGO is not desired yet some other 
fancier optimization such as LTO is. When do you encounter people wanting that? 
 PGO always produces a 10-20% faster CPython interpreter.

I have no problem with patches enabling an LTO only build for anyone who wants 
one if they do not already exist.  At the moment I'm not sure which LTO 
toolchains actually work properly.  We removed it from inclusion in 
--enable-optimizations because it was clearly broken in some common 
environments.

If LTO without PGO is useful, making it work is probably just a matter of 
making sure @LTOFLAGS@ show up on the non-PGO related 
Makefile.pre.in targets as well as updating the help 
text for --with-lto in configure.ac.  [untested]

-gps


Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Victor Stinner]

I don't think that PGO compilation itself is slow. Basically, I expect
that it only doubles the compilation time, but compiling Python takes
less than 1 minute usually. The slow part is the profiling task: run
the full Python test suite, which takes at least 20 minutes. The tests
must be run sequentially.

It would help to reduce the number of tests run in the profiling task.
We should also maybe skip tests which depend too much on I/O to get
more reproductible PGO performances.

Victor

2017-08-08 1:01 GMT+02:00 Soldea, Octavian :
> Hello
>
>
>
> Gregory: Thank you for your message. I really appreciate it.
>
>
>
> In my opinion the use of lto only compilation mode can be of benefit from
> two reasons at least:
>
> 1.   Lto only can provide a platform for experimentations. Of course, it
> seems to be very application specific.
>
> 2.   Lto compilation is faster than an entire pgo + lto mode. While pgo
> can take a lot of time, lto can provide significant optimizations, as
> compared to baseline, at the cost of less compilation time. Of course, lto
> is time consuming. To the best of my knowledge, compiling with lto only is
> shorter than combining pgo and lto.
>
>
>
> In this context, I will be more than happy to receive more feedback and
> opinions.
>
>
>
> Best regards,
>
>   Octavian
>
>
>
>
>
>
>
>
>
> From: Gregory P. Smith [mailto:g...@krypto.org]
> Sent: Monday, August 7, 2017 3:12 PM
> To: Soldea, Octavian ; Python-Dev@python.org
> Subject: Re: [Python-Dev] first post introduction and question regarding lto
>
>
>
> I've personally never seen a situation where PGO is not desired yet some
> other fancier optimization such as LTO is. When do you encounter people
> wanting that?  PGO always produces a 10-20% faster CPython interpreter.
>
>
>
> I have no problem with patches enabling an LTO only build for anyone who
> wants one if they do not already exist.  At the moment I'm not sure which
> LTO toolchains actually work properly.  We removed it from inclusion in
> --enable-optimizations because it was clearly broken in some common
> environments.
>
>
>
> If LTO without PGO is useful, making it work is probably just a matter of
> making sure @LTOFLAGS@ show up on the non-PGO related Makefile.pre.in
> targets as well as updating the help text for --with-lto in configure.ac.
> [untested]
>
>
>
> -gps
>
>
>
>
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Wang, Peter Xihong]

We evaluated different tests before setting down and proposing regrtest suite 
for PGO training, including using OpenStack benchmarks for OpenStack 
applications.  The regrtest suite was found consistently giving the best in 
terms of performance across applications/workloads. 

So I would hesitate to remove any test from the training suite (for the purpose 
of reducing build time), unless we are absolute sure it will not hurt 
performance across the board.

Thanks,

Peter

-Original Message-
From: Python-Dev 
[mailto:python-dev-bounces+peter.xihong.wang=intel@python.org] On Behalf Of 
Victor Stinner
Sent: Monday, August 07, 2017 4:43 PM
To: Soldea, Octavian 
Cc: Python-Dev@python.org
Subject: Re: [Python-Dev] first post introduction and question regarding lto

I don't think that PGO compilation itself is slow. Basically, I expect that it 
only doubles the compilation time, but compiling Python takes less than 1 
minute usually. The slow part is the profiling task: run the full Python test 
suite, which takes at least 20 minutes. The tests must be run sequentially.

It would help to reduce the number of tests run in the profiling task.
We should also maybe skip tests which depend too much on I/O to get more 
reproductible PGO performances.

Victor

2017-08-08 1:01 GMT+02:00 Soldea, Octavian :
> Hello
>
>
>
> Gregory: Thank you for your message. I really appreciate it.
>
>
>
> In my opinion the use of lto only compilation mode can be of benefit 
> from two reasons at least:
>
> 1.   Lto only can provide a platform for experimentations. Of course, it
> seems to be very application specific.
>
> 2.   Lto compilation is faster than an entire pgo + lto mode. While pgo
> can take a lot of time, lto can provide significant optimizations, as 
> compared to baseline, at the cost of less compilation time. Of course, 
> lto is time consuming. To the best of my knowledge, compiling with lto 
> only is shorter than combining pgo and lto.
>
>
>
> In this context, I will be more than happy to receive more feedback 
> and opinions.
>
>
>
> Best regards,
>
>   Octavian
>
>
>
>
>
>
>
>
>
> From: Gregory P. Smith [mailto:g...@krypto.org]
> Sent: Monday, August 7, 2017 3:12 PM
> To: Soldea, Octavian ; 
> Python-Dev@python.org
> Subject: Re: [Python-Dev] first post introduction and question 
> regarding lto
>
>
>
> I've personally never seen a situation where PGO is not desired yet 
> some other fancier optimization such as LTO is. When do you encounter 
> people wanting that?  PGO always produces a 10-20% faster CPython interpreter.
>
>
>
> I have no problem with patches enabling an LTO only build for anyone 
> who wants one if they do not already exist.  At the moment I'm not 
> sure which LTO toolchains actually work properly.  We removed it from 
> inclusion in --enable-optimizations because it was clearly broken in 
> some common environments.
>
>
>
> If LTO without PGO is useful, making it work is probably just a matter 
> of making sure @LTOFLAGS@ show up on the non-PGO related 
> Makefile.pre.in targets as well as updating the help text for --with-lto in 
> configure.ac.
> [untested]
>
>
>
> -gps
>
>
>
>
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gm
> ail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/peter.xihong.wang%40intel.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Soldea, Octavian]

Hello

Thank you for your messages. I am not quite sure I understood, therefore, I 
would like to ask if you see beneficial to have the option of lto separately?

Best regards,
  Octavian






-Original Message-
From: Wang, Peter Xihong 
Sent: Monday, August 7, 2017 5:00 PM
To: Victor Stinner ; Soldea, Octavian 

Cc: Python-Dev@python.org
Subject: RE: [Python-Dev] first post introduction and question regarding lto

We evaluated different tests before setting down and proposing regrtest suite 
for PGO training, including using OpenStack benchmarks for OpenStack 
applications.  The regrtest suite was found consistently giving the best in 
terms of performance across applications/workloads. 

So I would hesitate to remove any test from the training suite (for the purpose 
of reducing build time), unless we are absolute sure it will not hurt 
performance across the board.

Thanks,

Peter

-Original Message-
From: Python-Dev 
[mailto:python-dev-bounces+peter.xihong.wang=intel@python.org] On Behalf Of 
Victor Stinner
Sent: Monday, August 07, 2017 4:43 PM
To: Soldea, Octavian 
Cc: Python-Dev@python.org
Subject: Re: [Python-Dev] first post introduction and question regarding lto

I don't think that PGO compilation itself is slow. Basically, I expect that it 
only doubles the compilation time, but compiling Python takes less than 1 
minute usually. The slow part is the profiling task: run the full Python test 
suite, which takes at least 20 minutes. The tests must be run sequentially.

It would help to reduce the number of tests run in the profiling task.
We should also maybe skip tests which depend too much on I/O to get more 
reproductible PGO performances.

Victor

2017-08-08 1:01 GMT+02:00 Soldea, Octavian :
> Hello
>
>
>
> Gregory: Thank you for your message. I really appreciate it.
>
>
>
> In my opinion the use of lto only compilation mode can be of benefit 
> from two reasons at least:
>
> 1.   Lto only can provide a platform for experimentations. Of course, it
> seems to be very application specific.
>
> 2.   Lto compilation is faster than an entire pgo + lto mode. While pgo
> can take a lot of time, lto can provide significant optimizations, as 
> compared to baseline, at the cost of less compilation time. Of course, 
> lto is time consuming. To the best of my knowledge, compiling with lto 
> only is shorter than combining pgo and lto.
>
>
>
> In this context, I will be more than happy to receive more feedback 
> and opinions.
>
>
>
> Best regards,
>
>   Octavian
>
>
>
>
>
>
>
>
>
> From: Gregory P. Smith [mailto:g...@krypto.org]
> Sent: Monday, August 7, 2017 3:12 PM
> To: Soldea, Octavian ; 
> Python-Dev@python.org
> Subject: Re: [Python-Dev] first post introduction and question 
> regarding lto
>
>
>
> I've personally never seen a situation where PGO is not desired yet 
> some other fancier optimization such as LTO is. When do you encounter 
> people wanting that?  PGO always produces a 10-20% faster CPython interpreter.
>
>
>
> I have no problem with patches enabling an LTO only build for anyone 
> who wants one if they do not already exist.  At the moment I'm not 
> sure which LTO toolchains actually work properly.  We removed it from 
> inclusion in --enable-optimizations because it was clearly broken in 
> some common environments.
>
>
>
> If LTO without PGO is useful, making it work is probably just a matter 
> of making sure @LTOFLAGS@ show up on the non-PGO related 
> Makefile.pre.in targets as well as updating the help text for --with-lto in 
> configure.ac.
> [untested]
>
>
>
> -gps
>
>
>
>
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gm
> ail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/peter.xihong.wang%40intel.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Gregory P. Smith]

I don't know whether it is beneficial or not - but having the capability to
build LTO without PGO seems reasonable. I can review any pull requests
altering configure.ac and Makefile.pre.in to make such a change.

-@gpshead

On Mon, Aug 7, 2017 at 5:08 PM Soldea, Octavian 
wrote:

> Hello
>
> Thank you for your messages. I am not quite sure I understood, therefore,
> I would like to ask if you see beneficial to have the option of lto
> separately?
>
> Best regards,
>   Octavian
>
>
>
>
>
>
> -Original Message-
> From: Wang, Peter Xihong
> Sent: Monday, August 7, 2017 5:00 PM
> To: Victor Stinner ; Soldea, Octavian <
> octavian.sol...@intel.com>
> Cc: Python-Dev@python.org
> Subject: RE: [Python-Dev] first post introduction and question regarding
> lto
>
> We evaluated different tests before setting down and proposing regrtest
> suite for PGO training, including using OpenStack benchmarks for OpenStack
> applications.  The regrtest suite was found consistently giving the best in
> terms of performance across applications/workloads.
>
> So I would hesitate to remove any test from the training suite (for the
> purpose of reducing build time), unless we are absolute sure it will not
> hurt performance across the board.
>
> Thanks,
>
> Peter
>
> -Original Message-
> From: Python-Dev [mailto:python-dev-bounces+peter.xihong.wang=
> intel@python.org] On Behalf Of Victor Stinner
> Sent: Monday, August 07, 2017 4:43 PM
> To: Soldea, Octavian 
> Cc: Python-Dev@python.org
> Subject: Re: [Python-Dev] first post introduction and question regarding
> lto
>
> I don't think that PGO compilation itself is slow. Basically, I expect
> that it only doubles the compilation time, but compiling Python takes less
> than 1 minute usually. The slow part is the profiling task: run the full
> Python test suite, which takes at least 20 minutes. The tests must be run
> sequentially.
>
> It would help to reduce the number of tests run in the profiling task.
> We should also maybe skip tests which depend too much on I/O to get more
> reproductible PGO performances.
>
> Victor
>
> 2017-08-08 1:01 GMT+02:00 Soldea, Octavian :
> > Hello
> >
> >
> >
> > Gregory: Thank you for your message. I really appreciate it.
> >
> >
> >
> > In my opinion the use of lto only compilation mode can be of benefit
> > from two reasons at least:
> >
> > 1.   Lto only can provide a platform for experimentations. Of
> course, it
> > seems to be very application specific.
> >
> > 2.   Lto compilation is faster than an entire pgo + lto mode. While
> pgo
> > can take a lot of time, lto can provide significant optimizations, as
> > compared to baseline, at the cost of less compilation time. Of course,
> > lto is time consuming. To the best of my knowledge, compiling with lto
> > only is shorter than combining pgo and lto.
> >
> >
> >
> > In this context, I will be more than happy to receive more feedback
> > and opinions.
> >
> >
> >
> > Best regards,
> >
> >   Octavian
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: Gregory P. Smith [mailto:g...@krypto.org]
> > Sent: Monday, August 7, 2017 3:12 PM
> > To: Soldea, Octavian ;
> > Python-Dev@python.org
> > Subject: Re: [Python-Dev] first post introduction and question
> > regarding lto
> >
> >
> >
> > I've personally never seen a situation where PGO is not desired yet
> > some other fancier optimization such as LTO is. When do you encounter
> > people wanting that?  PGO always produces a 10-20% faster CPython
> interpreter.
> >
> >
> >
> > I have no problem with patches enabling an LTO only build for anyone
> > who wants one if they do not already exist.  At the moment I'm not
> > sure which LTO toolchains actually work properly.  We removed it from
> > inclusion in --enable-optimizations because it was clearly broken in
> > some common environments.
> >
> >
> >
> > If LTO without PGO is useful, making it work is probably just a matter
> > of making sure @LTOFLAGS@ show up on the non-PGO related
> > Makefile.pre.in targets as well as updating the help text for
> --with-lto in configure.ac.
> > [untested]
> >
> >
> >
> > -gps
> >
> >
> >
> >
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gm
> > ail.com
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/peter.xihong.wang%40intel.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listin

[Python-Dev] PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()

[Patrick Rutkowski]

I'm working on Windows. I have the following dead simple embedding
code that I'm using to test out the python interpreter:



Py_SetProgramName(L"MyApp");

Py_SetPath(
L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64\\python36.zip;"
L"C:\\Users\\rutski\\Documents\\python\\DLLs;"
L"C:\\Users\\rutski\\Documents\\python\\lib;"
L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64;"
L"C:\\Users\\rutski\\Documents\\python;"
L"C:\\Users\\rutski\\Documents\\python\\lib\\site-packages");

Py_Initialize();

PyRun_SimpleString(
"from time import time,ctime\n"
"print('Today is', ctime(time()))\n");



This code crashes trying to access address 0x0010 from within
PyRun_SimpleString(). The sequence of event's is this:

1) PyRun_SimpleString() tries to do AddModule("__main__")
2) AddModule tries to do PyImport_GetModuleDict()
3) PyImport_GetModuleDict() tries to doPyThreadState_GET()->interp
4) PyThreadState_GET() returns NULL, so the ->interp part crashes.

The weird thing is that calling PyImport_GetModuleDict() from within
my application directly works just fine. Weirder still is that the
whole thing actually executes fine if I build a windows command line
application with the embed code in main(), and execute it from a
terminal. The crash only happens when building a Windows GUI
application and calling the embed code in WinMain().

This is a python interpreter that I built from source on windows using
PCbuild\build.bat, so that I could track the crash. However, the exact
same crash was happening with the stock interpreter provided by the
python windows installer.

Does anyone have any ideas here?
-Patrick
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] first post introduction and question regarding lto

[Soldea, Octavian]

Hi Gregory

Thank you for your message. I will conduct several experiments and try to come 
back as soon as possible with a conclusion.

Best regards,
  Octavian

From: Gregory P. Smith [mailto:g...@krypto.org]
Sent: Monday, August 7, 2017 5:13 PM
To: Soldea, Octavian ; Wang, Peter Xihong 
; Victor Stinner 
Cc: Python-Dev@python.org
Subject: Re: [Python-Dev] first post introduction and question regarding lto

I don't know whether it is beneficial or not - but having the capability to 
build LTO without PGO seems reasonable. I can review any pull requests altering 
configure.ac and Makefile.pre.in 
to make such a change.

-@gpshead

On Mon, Aug 7, 2017 at 5:08 PM Soldea, Octavian 
mailto:octavian.sol...@intel.com>> wrote:
Hello

Thank you for your messages. I am not quite sure I understood, therefore, I 
would like to ask if you see beneficial to have the option of lto separately?

Best regards,
  Octavian






-Original Message-
From: Wang, Peter Xihong
Sent: Monday, August 7, 2017 5:00 PM
To: Victor Stinner mailto:victor.stin...@gmail.com>>; 
Soldea, Octavian mailto:octavian.sol...@intel.com>>
Cc: Python-Dev@python.org
Subject: RE: [Python-Dev] first post introduction and question regarding lto

We evaluated different tests before setting down and proposing regrtest suite 
for PGO training, including using OpenStack benchmarks for OpenStack 
applications.  The regrtest suite was found consistently giving the best in 
terms of performance across applications/workloads.

So I would hesitate to remove any test from the training suite (for the purpose 
of reducing build time), unless we are absolute sure it will not hurt 
performance across the board.

Thanks,

Peter

-Original Message-
From: Python-Dev 
[mailto:python-dev-bounces+peter.xihong.wang=intel@python.org]
 On Behalf Of Victor Stinner
Sent: Monday, August 07, 2017 4:43 PM
To: Soldea, Octavian 
mailto:octavian.sol...@intel.com>>
Cc: Python-Dev@python.org
Subject: Re: [Python-Dev] first post introduction and question regarding lto

I don't think that PGO compilation itself is slow. Basically, I expect that it 
only doubles the compilation time, but compiling Python takes less than 1 
minute usually. The slow part is the profiling task: run the full Python test 
suite, which takes at least 20 minutes. The tests must be run sequentially.

It would help to reduce the number of tests run in the profiling task.
We should also maybe skip tests which depend too much on I/O to get more 
reproductible PGO performances.

Victor

2017-08-08 1:01 GMT+02:00 Soldea, Octavian 
mailto:octavian.sol...@intel.com>>:
> Hello
>
>
>
> Gregory: Thank you for your message. I really appreciate it.
>
>
>
> In my opinion the use of lto only compilation mode can be of benefit
> from two reasons at least:
>
> 1.   Lto only can provide a platform for experimentations. Of course, it
> seems to be very application specific.
>
> 2.   Lto compilation is faster than an entire pgo + lto mode. While pgo
> can take a lot of time, lto can provide significant optimizations, as
> compared to baseline, at the cost of less compilation time. Of course,
> lto is time consuming. To the best of my knowledge, compiling with lto
> only is shorter than combining pgo and lto.
>
>
>
> In this context, I will be more than happy to receive more feedback
> and opinions.
>
>
>
> Best regards,
>
>   Octavian
>
>
>
>
>
>
>
>
>
> From: Gregory P. Smith [mailto:g...@krypto.org]
> Sent: Monday, August 7, 2017 3:12 PM
> To: Soldea, Octavian 
> mailto:octavian.sol...@intel.com>>;
> Python-Dev@python.org
> Subject: Re: [Python-Dev] first post introduction and question
> regarding lto
>
>
>
> I've personally never seen a situation where PGO is not desired yet
> some other fancier optimization such as LTO is. When do you encounter
> people wanting that?  PGO always produces a 10-20% faster CPython interpreter.
>
>
>
> I have no problem with patches enabling an LTO only build for anyone
> who wants one if they do not already exist.  At the moment I'm not
> sure which LTO toolchains actually work properly.  We removed it from
> inclusion in --enable-optimizations because it was clearly broken in
> some common environments.
>
>
>
> If LTO without PGO is useful, making it work is probably just a matter
> of making sure @LTOFLAGS@ show up on the non-PGO related
> Makefile.pre.in targets as well as updating the help 
> text for --with-lto in configure.ac.
> [untested]
>
>
>
> -gps
>
>
>
>
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> 

Re: [Python-Dev] PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()

[Larry Hastings]

On 08/07/2017 05:11 PM, Patrick Rutkowski wrote:

Does anyone have any ideas here?


My one idea: the GIL isn't initialized until you create a new thread.


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()

[Patrick Rutkowski]

On Mon, Aug 7, 2017 at 8:44 PM, Larry Hastings  wrote:
>
> My one idea: the GIL isn't initialized until you create a new thread.
>

That didn't seem to be it. I put a CreateThread() call right after
Py_Initialize(), and that didn't fix it. I also moved it around to
before Py_Initialize() and various other places, and nothing helped.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()

[Patrick Rutkowski]

After much stumbling around in the dark I have the answer. The
following table shows the results of tests which I performed pairing
different build configurations of my WinMain() app with different
python libraries. The terms "Release" and "Debug" refer to the
configuration type in Visual Studio. All builds were done in x64 mode.

Debug   | python36.lib   | Works
Debug   | python36_d.lib | Works
Debug   | python3_d.lib  | Works
Debug   | python3.lib| !!! CRASHES !!!
Release | python36.lib   | Works
Release | python36_d.lib | Works
Release | python3_d.lib  | !!! CRASHES !!!
Release | python3.lib| Works

So, it seems to be the case that picking a mismatched python binary
causes the crash, __but only with python3, not with python36__. This
makes me wonder what the differences is between the two in the first
place. I was getting the crash to begin with because I was linking my
Debug build with the release build python3.lib, since I thought it
shouldn't matter.

My problem is fixed now, but if anyone could sheld light on the
details of why exactly it happened then I would certainy be
interested.

On Mon, Aug 7, 2017 at 8:11 PM, Patrick Rutkowski
 wrote:
> I'm working on Windows. I have the following dead simple embedding
> code that I'm using to test out the python interpreter:
>
> 
>
> Py_SetProgramName(L"MyApp");
>
> Py_SetPath(
> L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64\\python36.zip;"
> L"C:\\Users\\rutski\\Documents\\python\\DLLs;"
> L"C:\\Users\\rutski\\Documents\\python\\lib;"
> L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64;"
> L"C:\\Users\\rutski\\Documents\\python;"
> L"C:\\Users\\rutski\\Documents\\python\\lib\\site-packages");
>
> Py_Initialize();
>
> PyRun_SimpleString(
> "from time import time,ctime\n"
> "print('Today is', ctime(time()))\n");
>
> 
>
> This code crashes trying to access address 0x0010 from within
> PyRun_SimpleString(). The sequence of event's is this:
>
> 1) PyRun_SimpleString() tries to do AddModule("__main__")
> 2) AddModule tries to do PyImport_GetModuleDict()
> 3) PyImport_GetModuleDict() tries to doPyThreadState_GET()->interp
> 4) PyThreadState_GET() returns NULL, so the ->interp part crashes.
>
> The weird thing is that calling PyImport_GetModuleDict() from within
> my application directly works just fine. Weirder still is that the
> whole thing actually executes fine if I build a windows command line
> application with the embed code in main(), and execute it from a
> terminal. The crash only happens when building a Windows GUI
> application and calling the embed code in WinMain().
>
> This is a python interpreter that I built from source on windows using
> PCbuild\build.bat, so that I could track the crash. However, the exact
> same crash was happening with the stock interpreter provided by the
> python windows installer.
>
> Does anyone have any ideas here?
> -Patrick
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com