Re: New to Python - Career question

[Paul Rubin]

Larry Martell  writes:
> I can tell they think I am old and they dismiss me right away.

http://oldgeekjobs.com ?
-- 
https://mail.python.org/mailman/listinfo/python-list

httplib HTTP class for python3?

[Mohammad Ghasemi]

Hi.
I want to convert a python 2 code to python 3 code and I have a problem with 
that.
I've asked my question on stackoverflow and never received an answer.
Where can I get an answer?

https://stackoverflow.com/questions/44411439/httplib-http-class-for-python3
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: httplib HTTP class for python3?

[Thomas Jollans]

On 10/06/17 11:33, Mohammad Ghasemi wrote:
> Hi.
> I want to convert a python 2 code to python 3 code and I have a problem with 
> that.
> I've asked my question on stackoverflow and never received an answer.
> Where can I get an answer?
> 
> https://stackoverflow.com/questions/44411439/httplib-http-class-for-python3
> 

You asked this only two days ago. "never received an answer" is a strong
statement considering.



As you obviously know, httplib as has renamed http.client.

> [...] from http.client import HTTP will not work.

According to the documentation
, the HTTP class is
ancient and "should not be used in new code". It makes sense that it
would have been removed in Python 3.0 - looks like you should just use
HTTPConnection.

More generally:

If you say something does not work, please provide a complete minimal
working example with an error message. Otherwise there is no way anybody
can know what it going wrong.

You provided example code (great!) but it's not complete: you are using
things without importing them, and it doesn't look like your code would
actually do much of anything if the relevant modules *were* imported.


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: httplib HTTP class for python3?

[Mohammad Ghasemi]

On Saturday, June 10, 2017 at 3:23:03 PM UTC+4:30, Thomas Jollans wrote:
> On 10/06/17 11:33, Mohammad Ghasemi wrote:
> > Hi.
> > I want to convert a python 2 code to python 3 code and I have a problem 
> > with that.
> > I've asked my question on stackoverflow and never received an answer.
> > Where can I get an answer?
> > 
> > https://stackoverflow.com/questions/44411439/httplib-http-class-for-python3
> > 
> 
> You asked this only two days ago. "never received an answer" is a strong
> statement considering.
> 
> 
> 
> As you obviously know, httplib as has renamed http.client.
> 
> > [...] from http.client import HTTP will not work.
> 
> According to the documentation
> , the HTTP class is
> ancient and "should not be used in new code". It makes sense that it
> would have been removed in Python 3.0 - looks like you should just use
> HTTPConnection.
> 
> More generally:
> 
> If you say something does not work, please provide a complete minimal
> working example with an error message. Otherwise there is no way anybody
> can know what it going wrong.
> 
> You provided example code (great!) but it's not complete: you are using
> things without importing them, and it doesn't look like your code would
> actually do much of anything if the relevant modules *were* imported.
> 
> 
> -- Thomas

As you said, HTTP Class should no longer be used.
You're right.
>From now on, I'll try to ask my questions more readable and understandable.
Thank you, Thomas.
-- 
https://mail.python.org/mailman/listinfo/python-list

Python Language Summit: Keeping Python competitive

[Steve D'Aprano]

Python 3.7 (unstable) is now as fast as Python 2.7, but some of the core
developers are hoping to improve matters, aiming for a 2 x speed improvement to
keep Python competitive with other languages.

https://lwn.net/Articles/723949/



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Progress on the Gilectomy

[Steve D'Aprano]

Larry Hastings is working on removing the GIL from CPython:

https://lwn.net/Articles/723949/


For those who don't know the background:

- The GIL (Global Interpreter Lock) is used to ensure that only one piece of
code can update references to an object at a time.

- The downside of the GIL is that CPython cannot take advantage of multiple CPU
cores effectively. Hence multi-threaded code is not as fast as it could be.

- Past attempts to remove the GIL caused unacceptable slow-downs for
single-threaded programs and code run on single-core CPUs.

- And also failed to show the expected performance gains for multi-threaded
programs on multi-core CPUs. (There was some gain, but not much.)


Thanks Larry for your experiments on this!





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Irmen de Jong]

On 10-6-2017 14:54, Steve D'Aprano wrote:
> Larry Hastings is working on removing the GIL from CPython:
> 
> https://lwn.net/Articles/723949/


Here is Larry's "How's it going" presentation from Pycon 2017 on this subject
https://www.youtube.com/watch?v=pLqv11ScGsQ

-irmen
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Psycopg2 pool clarification

[israel]

On 2017-06-08 19:55, Ian Kelly wrote:
On Thu, Jun 8, 2017 at 10:47 AM, Israel Brewster 
 wrote:

On Jun 7, 2017, at 10:31 PM, dieter  wrote:

israel  writes:

On 2017-06-06 22:53, dieter wrote:
...
As such, using psycopg2's pool is essentially
worthless for me (plenty of use for it, i'm sure, just not for me/my
use case).


Could you not simply adjust the value for the "min" parameter?
If you want at least "n" open connections, then set "min" to "n".


Well, sure, if I didn't care about wasting resources (which, I guess 
many people don't). I could set "n" to some magic number that would 
always give "enough" connections, such that my application never has 
to open additional connections, then adjust that number every few 
months as usage changes. In fact, now that I know how the logic of the 
pool works, that's exactly what I'm doing until I am confident that my 
caching replacement is solid.


Of course, in order to avoid having to open/close a bunch of 
connections during the times when it is most critical - that is, when 
the server is under heavy load - I have to set that number arbitrarily 
high. Furthermore, that means that much of the time many, if not most, 
of those connections would be idle. Each connection uses a certain 
amount of RAM on the server, not to mention using up limited 
connection slots, so now I've got to think about if my server is sized 
properly to be able to handle that load not just occasionally, but 
constantly - when reducing server load by reducing the frequency of 
connections being opened/closed was the goal in the first place. So 
all I've done is trade dynamic load for static load - increasing 
performance at the cost of resources, rather than more intelligently 
using the available resources. All-in-all, not the best solution, 
though it does work. Maybe if load was fairly constant it would make 
more sense though. So like I said *my* use c

 ase, whi
 ch
  is a number of web apps with varying loads, loads that also vary 
from day-to-day and hour-to-hour.


On the other hand, a pool that caches connections using the logic I 
laid out in my original post would avoid the issue. Under heavy load, 
it could open additional connections as needed - a performance penalty 
for the first few users over the min threshold, but only the first 
few, rather than all the users over a certain threshold ("n"). Those 
connections would then remain available for the duration of the load, 
so it doesn't need to open/close numerous connections. Then, during 
periods of lighter load, the unused connections can drop off, freeing 
up server resources for other uses. A well-written pool could even do 
something like see that the available connection pool is running low, 
and open a few more connections in the background, thus completely 
avoiding the connection overhead on requests while never having more 
than a few "extra" connections at any given time. Even if you left of 
the expiration logic, it would still be an improvement, because while 
unused connections

 wouldn't
  d
 rop, the "n" open connections could scale up dynamically until you 
have "enough" connections, without having to figure out and hard-code 
that "magic number" of open connections.


Why wouldn't I want something like that? It's not like its hard to 
code - took me about an hour and a half to get to a working prototype 
yesterday. Still need to write tests and add some polish, but it 
works. Perhaps, though, the common thought is just "throw more 
hardware at it and keep a lot of connections open at all time?" Maybe 
I was raised to conservatively, or the company I work for is too 
poor :-D


Psycopg is first and foremost a database adapter. To quote from the
psycopg2.pool module documentation, "This module offers a few pure
Python classes implementing *simple* connection pooling directly in
the client application" (emphasis added). The advertised list of
features at http://initd.org/psycopg/features/ doesn't even mention
connection pooling. In short, you're getting what you paid for.

It sounds like your needs are beyond what the psycopg2.pool module
provides.


Quite possible. Thus the reason I was looking for clarification on how 
the module was intended to work - if it doesn't work in the way that I 
want it to, I need to look elsewhere for a solution. My main reason for 
posting this thread was that I was expecting it to work one way, but 
testing showed it working another way, so I was trying to find out if 
that was intentional or user error. Apparently it's intentional, so 
there we go - in it's current form at least, my needs are beyond what 
the psycopg2 pool provides. Fair enough.



I suggest looking into a dedicated connection pooler like
PgBouncer. You'll find that it's much more feature-rich and
configurable than psycopg2.pool. It's production-ready, unlike your
prototype. And since it's a proxy, it can take connections from
multiple client apps and tune the pool to your overall load rather
than on

Re: Progress on the Gilectomy

[Serhiy Storchaka]

10.06.17 15:54, Steve D'Aprano пише:

Larry Hastings is working on removing the GIL from CPython:

https://lwn.net/Articles/723949/


For those who don't know the background:

- The GIL (Global Interpreter Lock) is used to ensure that only one piece of
code can update references to an object at a time.

- The downside of the GIL is that CPython cannot take advantage of multiple CPU
cores effectively. Hence multi-threaded code is not as fast as it could be.

- Past attempts to remove the GIL caused unacceptable slow-downs for
single-threaded programs and code run on single-core CPUs.

- And also failed to show the expected performance gains for multi-threaded
programs on multi-core CPUs. (There was some gain, but not much.)


Thanks Larry for your experiments on this!


And also GIL is used for guaranteeing atomicity of many operations and 
consistencity of internal structures without using additional locks. 
Many parts of the core and the stdlib would just not work correctly in 
multithread environment without GIL.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Stefan Behnel]

Serhiy Storchaka schrieb am 11.06.2017 um 07:11:
> 10.06.17 15:54, Steve D'Aprano пише:
>> Larry Hastings is working on removing the GIL from CPython:
>>
>> https://lwn.net/Articles/723949/
>>
>> For those who don't know the background:
>>
>> - The GIL (Global Interpreter Lock) is used to ensure that only one piece of
>> code can update references to an object at a time.
>>
>> - The downside of the GIL is that CPython cannot take advantage of
>> multiple CPU
>> cores effectively. Hence multi-threaded code is not as fast as it could be.
>>
>> - Past attempts to remove the GIL caused unacceptable slow-downs for
>> single-threaded programs and code run on single-core CPUs.
>>
>> - And also failed to show the expected performance gains for multi-threaded
>> programs on multi-core CPUs. (There was some gain, but not much.)
>>
>>
>> Thanks Larry for your experiments on this!
> 
> And also GIL is used for guaranteeing atomicity of many operations and
> consistencity of internal structures without using additional locks. Many
> parts of the core and the stdlib would just not work correctly in
> multithread environment without GIL.

And the same applies to external extension modules. The GIL is really handy
when it comes to reasoning about safety and correctness of algorithms under
the threat of thread concurrency. Especially in native code, where the
result of an unanticipated race condition is usually a crash rather than an
exception.

Stefan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Steve D'Aprano]

On Sun, 11 Jun 2017 04:21 pm, Stefan Behnel wrote:

> Serhiy Storchaka schrieb am 11.06.2017 um 07:11:
 
>> And also GIL is used for guaranteeing atomicity of many operations and
>> consistencity of internal structures without using additional locks. Many
>> parts of the core and the stdlib would just not work correctly in
>> multithread environment without GIL.
> 
> And the same applies to external extension modules. The GIL is really handy
> when it comes to reasoning about safety and correctness of algorithms under
> the threat of thread concurrency. Especially in native code, where the
> result of an unanticipated race condition is usually a crash rather than an
> exception.


Thank you Stefan and Serhiy!

I'm tired of people complaining about the GIL as a "mistake" without
acknowledging that it exists for a reason.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list