Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Thu, 14 Feb 2013 01:21:40 +0100 Victor Stinner wrote: > > UnicodeWriter (using the "writer += str" API) is the fastest method in > most cases, except for data = ['a'*10**4] * 10**2 (in this case, it's > 8x slower!). I guess that the overhead comes for the overallocation > which then require to

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 14/02/13 01:18, Chris Withers wrote: On 13/02/2013 11:53, Steven D'Aprano wrote: I fixed a performance bug in httplib some years ago by doing the exact opposite; += -> ''.join(). In that case, it changed downloading a file from 20 minutes to 3 seconds. That was likely on Python 2.5. I reme

Re: [Python-Dev] Usage of += on strings in loops in stdlib

Hi, I wrote quick hack to expose _PyUnicodeWriter as _string.UnicodeWriter: http://www.haypocalc.com/tmp/string_unicode_writer.patch And I wrote a (micro-)benchmark: http://www.haypocalc.com/tmp/bench_join.py ( The benchmark uses only ASCII string, it would be interesting to test latin1, BMP and

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 4:02 PM, Amaury Forgeot d'Arc wrote: > 2013/2/13 Lennart Regebro >> >> On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc >> wrote: >> > Yes, it's jitted. >> >> Admittedly, I have no idea in which cases the JIT kicks in, and what I >> should do to make that happen to m

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 7:06 PM, Maciej Fijalkowski wrote: > I actually wonder. > > There seems to be the consensus to avoid += (to some extent). Can > someone commit the change to urrllib then? I'm talking about reverting > http://bugs.python.org/issue1285086 specifically That's unquoting of URL

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 20:40, Christian Tismer wrote: If += is anyway a bit slower than other ways, forget it. I would then maybe add a commend somewhere that says "avoiding '+=' because it is not reliable" or something. += is a fastest way (in any implementation) if you concatenates only two strings.

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 19:06, Maciej Fijalkowski wrote: On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: On 2013-02-13 13:23, Lennart Regebro wrote: On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 and some of them are lit

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 1:27 PM, Maciej Fijalkowski wrote: > On Wed, Feb 13, 2013 at 8:24 PM, Brett Cannon wrote: > > > > > > > > On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski > > wrote: > >> > >> On Wed, Feb 13, 2013 at 7:33 PM, MRAB > wrote: > >> > On 2013-02-13 13:23, Lennart Regebro w

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 8:24 PM, Brett Cannon wrote: > > > > On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski > wrote: >> >> On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: >> > On 2013-02-13 13:23, Lennart Regebro wrote: >> >> >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> >> wrote

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski wrote: > On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: > > On 2013-02-13 13:23, Lennart Regebro wrote: > >> > >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka > >> wrote: > >>> > >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's num

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: > On 2013-02-13 13:23, Lennart Regebro wrote: >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> wrote: >>> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >>> than 3 >>> and some of them are literal strings. >> >> >>

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2013-02-13 13:23, Lennart Regebro wrote: On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 and some of them are literal strings. This has the benefit of being slow both on CPython and PyPy. Although using .

Re: [Python-Dev] Usage of += on strings in loops in stdlib

2013/2/13 Lennart Regebro > On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc > wrote: > > Yes, it's jitted. > > Admittedly, I have no idea in which cases the JIT kicks in, and what I > should do to make that happen to make sure I have the best possible > real-life test cases. > PyPy JIT ki

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 15:17, Daniel Holth wrote: On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka mailto:storch...@gmail.com>> wrote: I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 and some of them are literal strings. Fixed: x = ('%s' * len(abcd)) % abcd No, you

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc wrote: > Yes, it's jitted. Admittedly, I have no idea in which cases the JIT kicks in, and what I should do to make that happen to make sure I have the best possible real-life test cases. //Lennart

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc wrote: > > 2013/2/13 Lennart Regebro >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> wrote: >> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >> > than 3 >> > and some of them are literal strings. >> >> Thi

Re: [Python-Dev] Usage of += on strings in loops in stdlib

2013/2/13 Christian Tismer > On 13.02.13 15:27, Amaury Forgeot d'Arc wrote: > > > 2013/2/13 Lennart Regebro > >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> wrote: >> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >> than 3 >> > and some of them are literal str

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 15:23, Lennart Regebro wrote: > On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: >> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 >> and some of them are literal strings. > > This has the benefit of being slow both on CPython and PyPy. Although >

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 15:27, Amaury Forgeot d'Arc wrote: 2013/2/13 Lennart Regebro mailto:rege...@gmail.com>> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka mailto:storch...@gmail.com>> wrote: > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 > and some

Re: [Python-Dev] Usage of += on strings in loops in stdlib

2013/2/13 Lennart Regebro > On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka > wrote: > > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more > than 3 > > and some of them are literal strings. > > This has the benefit of being slow both on CPython and PyPy. Although > using .f

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13/02/2013 11:53, Steven D'Aprano wrote: I fixed a performance bug in httplib some years ago by doing the exact opposite; += -> ''.join(). In that case, it changed downloading a file from 20 minutes to 3 seconds. That was likely on Python 2.5. I remember it well. http://mail.python.org/pip

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 14:17, Daniel Holth wrote: On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka > wrote: On 13.02.13 10:52, Larry Hastings wrote: I've always hated the "".join(array) idiom for "fast" string concatenation--it's ugly and it flies in the fa

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 > and some of them are literal strings. This has the benefit of being slow both on CPython and PyPy. Although using .format() is even slower. :-) //Lennart

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka wrote: > On 13.02.13 10:52, Larry Hastings wrote: > >> I've always hated the "".join(array) idiom for "fast" string >> concatenation--it's ugly and it flies in the face of TOOWTDI. I think >> everyone should use "x = a + b + c + d" for string conc

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 10:52, Larry Hastings wrote: I've always hated the "".join(array) idiom for "fast" string concatenation--it's ugly and it flies in the face of TOOWTDI. I think everyone should use "x = a + b + c + d" for string concatenation, and we should just make that fast. I prefer "x = '%s%s%s

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13/02/13 22:46, Xavier Morel wrote: On 2013-02-13, at 12:37 , Steven D'Aprano wrote: # even less obvious than sum map(operator.add, array) That one does not work, it'll try to call the binary `add` with each item of the array when the map iterator is reified, erroring out. fu

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13/02/13 20:09, Chris Withers wrote: On 12/02/2013 21:03, Maciej Fijalkowski wrote: We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. That's... interest

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2013-02-13, at 12:37 , Steven D'Aprano wrote: > ># even less obvious than sum >map(operator.add, array) That one does not work, it'll try to call the binary `add` with each item of the array when the map iterator is reified, erroring out. functools.reduce(operator.add, array, '')

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 09:52, Nick Coghlan wrote: On Wed, Feb 13, 2013 at 5:42 PM, Alexandre Vassalotti wrote: I don't think so. Ropes are really useful when you work with gigabytes of data, but unfortunately they don't make good general-purpose strings. Monolithic arrays are much more efficient and simpl

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13/02/13 19:52, Larry Hastings wrote: I've always hated the "".join(array) idiom for "fast" string concatenation --it's ugly and it flies in the face of TOOWTDI. I think everyone should use "x = a + b + c + d" for string concatenation, and we should just make that fast. "".join(array) is

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 12.02.13 23:03, Maciej Fijalkowski wrote: How people feel about generally not having += on long strings in stdlib (since the refcount = 1 thing is a hack)? Sometimes the use of += for strings or bytes is appropriate. For example, I deliberately used += for bytes instead b''.join() (note tha

Re: [Python-Dev] Usage of += on strings in loops in stdlib

Le Wed, 13 Feb 2013 09:02:07 +0100, Victor Stinner a écrit : > I added a _PyUnicodeWriter internal API to optimize str%args and > str.format(args). It uses a buffer which is overallocated, so it's > basically like CPython str += str optimization. I still don't know how > efficient it is on Windows

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 12/02/2013 21:03, Maciej Fijalkowski wrote: We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. That's... interesting. I fixed a performance bug in httpli

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 02/12/2013 05:25 PM, Christian Tismer wrote: Ropes have been implemented by Carl-Friedrich Bolz in 2007 as I remember. No idea what the impact was, if any at all. Would ropes be an answer (and a simple way to cope with string mutation patterns) as an alternative implementation, and therefore s

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, Feb 12, 2013 at 10:03 PM, Maciej Fijalkowski wrote: > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. Can someone show the actual diff? O

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 10:02 AM, Victor Stinner wrote: > I added a _PyUnicodeWriter internal API to optimize str%args and > str.format(args). It uses a buffer which is overallocated, so it's > basically like CPython str += str optimization. I still don't know how > efficient it is on Windows, sin

Re: [Python-Dev] Usage of += on strings in loops in stdlib

I added a _PyUnicodeWriter internal API to optimize str%args and str.format(args). It uses a buffer which is overallocated, so it's basically like CPython str += str optimization. I still don't know how efficient it is on Windows, since realloc() is slow on Windows (at least on old Windows versions

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 5:42 PM, Alexandre Vassalotti wrote: > On Tue, Feb 12, 2013 at 5:25 PM, Christian Tismer > wrote: >> >> Would ropes be an answer (and a simple way to cope with string mutation >> patterns) as an alternative implementation, and therefore still justify >> the usage of that p

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, Feb 12, 2013 at 5:25 PM, Christian Tismer wrote: > Would ropes be an answer (and a simple way to cope with string mutation > patterns) as an alternative implementation, and therefore still justify > the usage of that pattern? > I don't think so. Ropes are really useful when you work with

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, 13 Feb 2013 08:16:21 +0100 Antoine Pitrou wrote: > On Wed, 13 Feb 2013 09:39:23 +1000 > Nick Coghlan wrote: > > On 13 Feb 2013 07:08, "Maciej Fijalkowski" wrote: > > > > > > Hi > > > > > > We recently encountered a performance issue in stdlib for pypy. It > > > turned out that someone co

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, 13 Feb 2013 09:39:23 +1000 Nick Coghlan wrote: > On 13 Feb 2013 07:08, "Maciej Fijalkowski" wrote: > > > > Hi > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses += for > > strings instead of "".join

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, 13 Feb 2013 00:28:15 +0100 Christian Heimes wrote: > Am 12.02.2013 22:32, schrieb Antoine Pitrou: > > For the record, io.StringIO should be quite fast in 3.3. > > (except for the method call overhead that Guido is complaining > > about :-)) > > AFAIK it's not the actual *call* of the meth

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2/12/2013 4:03 PM, Maciej Fijalkowski wrote: Hi We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. Now this hurts pypy (we can mitigate it to some degree

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2/12/2013 6:20 PM, MRAB wrote: Tuples are much like immutable lists; sets were added, and then frozensets; should we be adding mutable strings too (a bit like C#'s StringBuilder)? (Just wondering...) StringIO is effectively a mutable string with a file interface. sio.write('abc') is the equ

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13.02.13 02:09, Alexandre Vassalotti wrote: On Tue, Feb 12, 2013 at 1:44 PM, Antoine Pitrou > wrote: It's idiomatic because strings are immutable (by design, not because of an optimization detail) and therefore concatenation *has* to imply building

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 12.02.13 22:03, Maciej Fijalkowski wrote: Hi We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. Now this hurts pypy (we can mitigate it to some degree tho

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, Feb 12, 2013 at 1:44 PM, Antoine Pitrou wrote: > It's idiomatic because strings are immutable (by design, not because of > an optimization detail) and therefore concatenation *has* to imply > building a new string from scratch. > Not necessarily. It is totally possible to implement strin

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 13 Feb 2013 07:08, "Maciej Fijalkowski" wrote: > > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 1:20 AM, MRAB wrote: > On 2013-02-12 21:44, Antoine Pitrou wrote: >> >> On Tue, 12 Feb 2013 16:40:38 -0500 >> Ned Batchelder wrote: >>> >>> >>> But the only reason "".join() is a Python idiom in the first place is >>> because it was "the fast way" to do what everyone initi

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Wed, Feb 13, 2013 at 1:28 AM, Christian Heimes wrote: > Am 12.02.2013 22:32, schrieb Antoine Pitrou: >> For the record, io.StringIO should be quite fast in 3.3. >> (except for the method call overhead that Guido is complaining >> about :-)) > > AFAIK it's not the actual *call* of the method tha

Re: [Python-Dev] Usage of += on strings in loops in stdlib

Am 12.02.2013 22:32, schrieb Antoine Pitrou: > For the record, io.StringIO should be quite fast in 3.3. > (except for the method call overhead that Guido is complaining > about :-)) AFAIK it's not the actual *call* of the method that is slow, but rather attribute lookup and creation of bound metho

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2013-02-12 21:44, Antoine Pitrou wrote: On Tue, 12 Feb 2013 16:40:38 -0500 Ned Batchelder wrote: But the only reason "".join() is a Python idiom in the first place is because it was "the fast way" to do what everyone initially coded as "s += ...". Just because we all learned a long time a

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2013-02-12, at 22:40 , Ned Batchelder wrote: > But the only reason "".join() is a Python idiom in the first place is because > it was "the fast way" to do what everyone initially coded as "s += ...". > Just because we all learned a long time ago that joining was the fast way to > build a st

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, 12 Feb 2013 16:40:38 -0500 Ned Batchelder wrote: > > But the only reason "".join() is a Python idiom in the first place is > because it was "the fast way" to do what everyone initially coded as "s > += ...". Just because we all learned a long time ago that joining was > the fast way

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, 12 Feb 2013 16:40:38 -0500, Ned Batchelder wrote: > On 2/12/2013 4:16 PM, Brett Cannon wrote: > > On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou > > wrote: > > On Tue, 12 Feb 2013 23:03:04 +0200 > > Maciej Fijalkowski mailto:fij...@gmail.com>> wrote:

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On 2/12/2013 4:16 PM, Brett Cannon wrote: On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou > wrote: Hi ! On Tue, 12 Feb 2013 23:03:04 +0200 Maciej Fijalkowski mailto:fij...@gmail.com>> wrote: > > We recently encountered a performance issue in

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, 12 Feb 2013 13:32:50 -0800 "fwierzbi...@gmail.com" wrote: > On Tue, Feb 12, 2013 at 1:03 PM, Maciej Fijalkowski wrote: > > Hi > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses += for > > strings in

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, Feb 12, 2013 at 1:03 PM, Maciej Fijalkowski wrote: > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitig

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, Feb 12, 2013 at 11:16 PM, Brett Cannon wrote: > > > > On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou wrote: >> >> >> Hi ! >> >> On Tue, 12 Feb 2013 23:03:04 +0200 >> Maciej Fijalkowski wrote: >> > >> > We recently encountered a performance issue in stdlib for pypy. It >> > turned out th

Re: [Python-Dev] Usage of += on strings in loops in stdlib

On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou wrote: > > Hi ! > > On Tue, 12 Feb 2013 23:03:04 +0200 > Maciej Fijalkowski wrote: > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses += for > > strings inst

Re: [Python-Dev] Usage of += on strings in loops in stdlib

Hi ! On Tue, 12 Feb 2013 23:03:04 +0200 Maciej Fijalkowski wrote: > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can

[Python-Dev] Usage of += on strings in loops in stdlib

Hi We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. Now this hurts pypy (we can mitigate it to some degree though) and possible Jython and IronPython too. H