RE: [Python-Dev] Windows Low Fragmentation Heap yields speedup of ~15%
> A well-known trick is applicable in that case, if Martin thinks it's > worth the bother: > grow the list to its final size once, at the start (overestimating if > you don't know for sure). Then instead of appending, keep an index to > the next free slot, same as you'd do in C. Then the list guts never > move, so if that doesn't yield the same kind of speedup without using > LFH, list copying wasn't actually the culprit to begin with. I actually did that in Py2.1, and removed it when we switched to Zope 2.7/Py 2.3, because it was no longer worth it, presumably due to obmalloc becoming enabled. Unfortunately, I have lost the speedup gained by my Fast-Append-List in Py 2.1, but I recall it saved about 5% in a similar test than the one I have today. > Yes. For example, a 300-character string could do it (that's not > small to obmalloc, but is to LFH). Strings produced by pickling are > very often that large, and especially in Zope (which uses pickles > extensively under the covers -- reading and writing persistent objects > in Zope all involve pickle strings). With my amateur (in C-stuff) knowledge, this sounds as a very likely point. >From Evan's 17 Feb mail, it sounds that cPickle would not use obmalloc - if this is the case, wouldn't that be an obvious (and relatively easy) speedup? > If someone were motivated enough, it would probably be easiest to > run Martin's app on a Linux box, and use the free Linux tools to analyze it. As it is often the case, real-life figures do not come from a benchmark, but from an application test with real data, so putting the whole thing up on Linux would need quite some time - which I don't have :-( Best regards, Martin -Original Message- From: Tim Peters [mailto:[EMAIL PROTECTED] Sent: Friday, 18 Feb 2005 23:52 To: Evan Jones Cc: Gfeller Martin; Martin v. Löwis; Python Dev Subject: Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15% [Tim Peters] ... >> Then you allocate a small object, marked 's': >> >> bbbsfff [Evan Jones] > Isn't the whole point of obmalloc No, because it doesn't matter what follows that introduction: obmalloc has several points, including exploiting the GIL, heuristics aiming at reusing memory while it's still high in the memory heirarchy, almost never touching a piece of memory until it's actually needed, and so on. > is that we don't want to allocate "s" on the heap, since it is small? That's one of obmalloc's goals, yes. But "small" is a relative adjective, not absolute. Because we're primarily talking about LFH here, the natural meaning for "small" in _this_ thread is < 16KB, which is much larger than "small" means to obmalloc. The memory-map example applies just well to LFH as to obmalloc, by changing which meaning for "small" you have in mind. > I guess "s" could be an object that might potentially grow. For example, list guts in Python are never handled by obmalloc, although the small fixed-size list _header_ object is always handled by obmalloc. >> One thing to take from that is that LFH can't be helping list-growing >> in a direct way either, if LFH (as seems likely) also needs to copy >> objects that grow in order to keep its internal memory segregated by >> size. The indirect benefit is still available, though: LFH may be >> helping simply by keeping smaller objects out of the general heap's >> hair. > So then wouldn't this mean that there would have to be some sort of > small object being allocated via the system malloc that is causing the > poor behaviour? Yes. For example, a 300-character string could do it (that's not small to obmalloc, but is to LFH). Strings produced by pickling are very often that large, and especially in Zope (which uses pickles extensively under the covers -- reading and writing persistent objects in Zope all involve pickle strings). > As you mention, I wouldn't think it would be list objects, since resizing > lists using LFH should be *worse*. Until they get to LFH's boundary for "small", and we have only the vaguest idea what Martin's app does here -- we know it grows lists containing 50K elements in the end, and ... well, that's all I really know about it . A well-known trick is applicable in that case, if Martin thinks it's worth the bother: grow the list to its final size once, at the start (overestimating if you don't know for sure). Then instead of appending, keep an index to the next free slot, same as you'd do in C. Then the list guts never move, so if that doesn't yield the same kind of speedup without using LFH, list copying wasn't actually the culprit to begin with. > That would actually be something that is worth verifying, however. Not worth the time to me -- Windows is closed-source, and I'm too old to enjoy staring at binary disassemblies any more. Besides, list guts can't stay in LFH after the list exceeds 4K elements. If list-copying costs are significant here, they're far more likely
[Python-Dev] Re: Some old patches
Reinhold Birkenfeld wrote: > Hello, > > this time working up some of the patches with beards: No one listening? I'm sorry when patch reviews are not welcome in python-devel; in this case I'll post individual comments to the patches on SF. Reinhold ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Some old patches
On Wed, Feb 23, 2005, Reinhold Birkenfeld wrote: > Reinhold Birkenfeld wrote: >> >> this time working up some of the patches with beards: > > No one listening? I'm sorry when patch reviews are not welcome in > python-devel; in this case I'll post individual comments to the patches > on SF. You should definitely post the patch reviews to SF no matter what; that way there's a historical record. Patch review summaries are welcome on python-dev, but it's the nature of the medium that they don't always get responded to. BTW, it's not clear whether your e-mail address is munged or not, which likely contributes to reluctance to respond. -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ "The joy of coding Python should be in seeing short, concise, readable classes that express a lot of action in a small amount of clear code -- not in reams of trivial code that bores the reader to death." --GvR ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Some old patches
Aahz wrote: > On Wed, Feb 23, 2005, Reinhold Birkenfeld wrote: >> Reinhold Birkenfeld wrote: >>> >>> this time working up some of the patches with beards: >> >> No one listening? I'm sorry when patch reviews are not welcome in >> python-devel; in this case I'll post individual comments to the patches >> on SF. > > You should definitely post the patch reviews to SF no matter what; that > way there's a historical record. Patch review summaries are welcome on > python-dev, but it's the nature of the medium that they don't always get > responded to. > > BTW, it's not clear whether your e-mail address is munged or not, which > likely contributes to reluctance to respond. Well, what about trying? [I see you tried, so don't bother] I insert this "nospam" deliberately, as it keeps spam away. Perhaps it would be best to mention this in a signature. Reinhold -- Mail address is perfectly valid! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What about CALL_ATTR?
While rummaging in the old patches, I found this: """ The result of the PyCore sprint of me and Brett: the CALL_ATTR opcode (LOAD_ATTR and CALL_FUNCTION combined) that skips the PyMethod creation and destruction for classic classes (but not newstyle classes, yet.) The code is somewhat rough yet, it needs commenting, some renaming, and most importantly testing. It seems to work, however, and provides between a 35% and 5% speedup. (5% in 'average' code, up to 35% in instance method calls and instance creation alone.) It also needs to be updated to include newstyle classes. I will likely work on this on the flight home. """ (patch #709744) How is the status of this? Sounds promising, I'd say... Reinhold -- Mail address is perfectly valid! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What about CALL_ATTR?
Reinhold Birkenfeld wrote: While rummaging in the old patches, I found this: """ The result of the PyCore sprint of me and Brett: the CALL_ATTR opcode (LOAD_ATTR and CALL_FUNCTION combined) that skips the PyMethod creation and destruction for classic classes (but not newstyle classes, yet.) The code is somewhat rough yet, it needs commenting, some renaming, and most importantly testing. It seems to work, however, and provides between a 35% and 5% speedup. (5% in 'average' code, up to 35% in instance method calls and instance creation alone.) It also needs to be updated to include newstyle classes. I will likely work on this on the flight home. """ (patch #709744) How is the status of this? Sounds promising, I'd say... See my reply in the "Store x Load x --> DupStore" thread at http://mail.python.org/pipermail/python-dev/2005-February/051725.html . Basically Thomas discovered that it was slower when used with new-style classes. But this was almost two years ago with Thomas having not done hacking on the core for two years IIRC and me having practically zero experience. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Comment regarding PEP 328
In a recent discussion in a SF patch, I noticed that PEP 328* only seems to support relative imports within packages, while bare import statements use the entirety of sys.path, not solving the shadowing of standard library module names. I have certainly forgotten bits of discussion from last spring, but I would offer that Python could offer standard library shadowing protection through the use of an extended PEP 328 semantic. More specifically; after a 'from __future__ import absolute_import' statement, any import in the module performing "import foo" will only check for foo in the standard library, and the use of the leading period, "from . import foo", the will signify relative to the current path. ** The lack of a 'from __future__ import absolute_import' statement in a module will not change the import semantic of that module. This allows current code to continue to work, and for those who want to choose names which shadow the standard library modules, a way of dealing with their choices. Further, in the case of PEP 328, the package relative imports were to become the default in 2.6 (with deprecation in 2.5, availability in 2.4), but with the lack of an implementation, perhaps those numbers should be incremented. If the behavior I describe is desireable, it would subsume PEP 328, and perhaps should also become the default behavior at some point in time (perhaps in the same adjusted timeline as PEP 328). Alternatively, PEP 328 could be implemented as-is, and a second future import could be defined which offers this functionality, being permanently optional (or on a different timeline) via the future import. Essentially, it would ignore the empty path "" in sys.path when the functionality has been enabled via the proper future import in the current module. - Josiah * PEP 328 first describes the use of parenthesis in import statements so that long import listings do not require backslash-escaping of newlines. It then describes a semantic for not checking sys.path when performing an import, as well as allowing parent, cousin, uncle, etc., imports via additional leading periods. "from . import foo" for sibling imports, "from .. import foo" for parent imports, etc. ** I describe the semantic as being a per-module option, as this is the only backwards-compatible mechanism in the near future (Python 2.5). An import implementation would merely check the existance of the proper name binding -> object pair in the importer's global namespace. The standard library would need to be modified if or when current behavior is deprecated (this would extend the modules needing to be modified due to PEP 328). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Some old patches
Reinhold Birkenfeld wrote: No one listening? I'm sorry when patch reviews are not welcome in python-devel; in this case I'll post individual comments to the patches on SF. I have seen them, and I appreciate them, but I had no time to respond, yet (likewise for the 20+ other reviews which I still need to look at). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Comment regarding PEP 328
> In a recent discussion in a SF patch, I noticed that PEP 328* only seems > to support relative imports within packages, while bare import > statements use the entirety of sys.path, not solving the shadowing of > standard library module names. Hm. I'm not convinced that there is a *problem* with shadowing of standard library module names. You shouldn't pick a module name that shadows a standard library module, or if you do you shouldn't want to be able to still use those modules that you're shadowing. Anything else is just asking for trouble. > I have certainly forgotten bits of discussion from last spring, but I > would offer that Python could offer standard library shadowing > protection through the use of an extended PEP 328 semantic. > > More specifically; after a 'from __future__ import absolute_import' > statement, any import in the module performing "import foo" will only > check for foo in the standard library, and the use of the leading period, > "from . import foo", the will signify relative to the current path. ** And how exactly do you define "the standard library"? Anything that's on sys.path? That would seem the only reasonable interpretation to me. So I take it that you want the "script directory" off that path. (Let's for the sake of argument call it ".".) > The lack of a 'from __future__ import absolute_import' statement in a > module will not change the import semantic of that module. It's hard to imagine how this would work. sys.path is global, so either "." is on it, or it isn't. So things in "." are either considered part of the standard library, or they are not; this can't be made dependent on the module's importation of something from __future__. > This allows current code to continue to work, and for those who want to > choose names which shadow the standard library modules, a way of dealing > with their choices. My suggested way of dealing with their choices is summarized in the first paragraph of my reply above. > Further, in the case of PEP 328, the package relative imports were to > become the default in 2.6 (with deprecation in 2.5, availability in 2.4), > but with the lack of an implementation, perhaps those numbers should be > incremented. If the behavior I describe is desireable, it would subsume > PEP 328, and perhaps should also become the default behavior at some > point in time (perhaps in the same adjusted timeline as PEP 328). That's a separate issue; the absolute/relative import part of PEP 328 didn't make it into 2.4 so I suppose we should ++ all those version numbers. > Alternatively, PEP 328 could be implemented as-is, and a second future > import could be defined which offers this functionality, being > permanently optional (or on a different timeline) via the future import. I don't like permanently optional language features; that causes too much confusion. I'd much rather settle on clear semantics that everyone can understand (even if they may disagree). But I certainly would prefer that the proposed feature becomes a separate PEP which can be discussed, accepted or rejected, and implemented separately from PEP 328, which is complete and accepted and just awaiting someone to implement it. > Essentially, it would ignore the empty path "" in sys.path when the > functionality has been enabled via the proper future import in the > current module. But it's not always "" -- it's the directory where the "main" script was found. Let me explain the biggest problem I see for your proposal: what would be the canonical name for a module imported using your "new relative semantics"? Remember, the canonical name of a module is its __name__ attribute, and the key that finds it in the sys.modules dict. Because there's only one sys.modules dict (barring restricted execution sandboxes), the canonical name must be unique. So if there's a standard library module string, its canonical name is "string". Now suppose you have your own non-standard-linrary module read from a file string.py. What should its canonical name be? It can't be "string" because that's already reserved for the standard library module name. The best solution I can think of for this off the top of my head is to somehow allow for the arrangement of a pseudo-package named __main__ and to make all these non-standard-library modules reside (logically) in that module. If you can write a PEP along those lines you may be on to something -- but I expect that the way to turn it on is not to import something from __future__ but perhaps from __main__. I'm not exactly sure how to get "." off sys.path, but maybe you can think about that for your PEP proposal. What do you say? > - Josiah > > * PEP 328 first describes the use of parenthesis in import statements so > that long import listings do not require backslash-escaping of newlines. > It then describes a semantic for not checking sys.path when performing > an import, as well as allowing parent, cousin, uncle, etc., imports via > additional leading peri
Re: [Python-Dev] __str__ vs. __unicode__
Walter Dörwald wrote: M.-A. Lemburg wrote: Walter Dörwald wrote: M.-A. Lemburg wrote: > [...] __str__ and __unicode__ as well as the other hooks were specifically added for the type constructors to use. However, these were added at a time where sub-classing of types was not possible, so it's time now to reconsider whether this functionality should be extended to sub-classes as well. So can we reach consensus on this, or do we need a BDFL pronouncement? I don't have a clear picture of what the consensus currently looks like :-) If we're going for for a solution that implements the hook awareness for all hooks, I'd be +1 on that. If we only touch the __unicode__ case, we'd only be created yet another special case. I'd vote -0 on that. > [...] Here's the patch that implements this for int/long/float/unicode: http://www.python.org/sf/1109424 Any movement on this? +1 for making things work like str; if a subclass overrides __str__ it should use that method. If correctness of what is returned is a worry then a check could be tossed in before the value is returned. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
