Re: [Python-Dev] PATCH submitted: Speed up + for string concatenation, now as fast as "".join(x) idiom
Nicko van Someren <[EMAIL PROTECTED]> wrote: > It's not like having this patch is going to force anyone to change > the way they write their code. As far as I can tell it simply offers > better performance if you choose to express your code in some common > ways. If it speeds up pystone by 5.5% with such minimal down side > I'm hard pressed to see a reason not to use it. This has to wait until Python 2.6 (which is anywhere from 14-24 months away, according to history); including it would destroy binary capatability with modules compiled for 2.5, nevermind that it is a nontrivial feature addition. I also think that the original author (or one of this patch's supporters) should write a PEP outlining the Python 2.5 and earlier drawbacks, what changes this implementation brings, its improvements, and any potential drawbacks. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PATCH submitted: Speed up + for string concatenation, now as fast as "".join(x) idiom
Nicko van Someren wrote: > If it speeds up pystone by 5.5% with such minimal down side > I'm hard pressed to see a reason not to use it. can you tell me where exactly "pystone" does string concatenations? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PATCH submitted: Speed up + for string concatenation, now as fast as "".join(x) idiom
Fredrik> Nicko van Someren wrote: >> If it speeds up pystone by 5.5% with such minimal down side I'm hard >> pressed to see a reason not to use it. Fredrik> can you tell me where exactly "pystone" does string Fredrik> concatenations? I wondered about that as well. While I'm not prepared to assert without a doubt that pystone does no simpleminded string concatenation, a couple minutes scanning the pystone source didn't turn up any. If the pystone speedup isn't an artifact, the absence of string concatention in pystone suggests it's happening somewhere in the interpreter. I applied the patch, ran the interpreter under gdb with a breakpoint set in string_concat where the PyStringConcatenationObject is created, then ran pystone. The first hit was in site.py -> distutils/util.py -> string.py All told, there were only 22 hits, none for very long strings, so that doesn't explain the performance improvement. BTW, on my Mac (OSX 10.4.8) max() is not defined. I had to add a macro definition to string_concat. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Security Advisory for unicode repr() bug?
[ Bug http://python.org/sf/1541585 ] This seems to be handled like a security issue by linux distributors, it's also a news item on security related pages. Should a security advisory be written and official patches be provided? Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security Advisory for unicode repr() bug?
Georg> [ Bug http://python.org/sf/1541585 ] Georg> This seems to be handled like a security issue by linux Georg> distributors, it's also a news item on security related pages. Georg> Should a security advisory be written and official patches be Georg> provided? I asked about this a few weeks ago. I got no direct response. Secunia sent mail to webmaster and the SF project admins asking about how this could be exploited. (Isn't figuring that stuff out their job?) This was corrected before 2.5 was released and the 2.4 source has (I think) already been patched, with 2.4.4 right around the corner. The bulk of the Python installations in the field are probably running on Windows (most of them provided by HP/Compaq), and it seems the Linux vendors are all over it. I don't know if Apple has picked up on it (or if the version they currently distribute is affected - 2.3.5 built Oct 5 2005). Would you provide a patch of some sort for Windows or just refer people to corrected installers? Given the apparently miserable results trying to get Windows users to install security fixes manually, I doubt a new 2.4.3 Windows installer would get much exercise. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security Advisory for unicode repr() bug?
[EMAIL PROTECTED] wrote: > Georg> [ Bug http://python.org/sf/1541585 ] > > Georg> This seems to be handled like a security issue by linux > Georg> distributors, it's also a news item on security related pages. > > Georg> Should a security advisory be written and official patches be > Georg> provided? > > I asked about this a few weeks ago. I got no direct response. Secunia sent > mail to webmaster and the SF project admins asking about how this could be > exploited. (Isn't figuring that stuff out their job?) Perhaps, judging from the name :) > This was corrected before 2.5 was released and the 2.4 source has (I think) > already been patched, with 2.4.4 right around the corner. The bulk of the > Python installations in the field are probably running on Windows (most of > them provided by HP/Compaq), and it seems the Linux vendors are all over it. > I don't know if Apple has picked up on it (or if the version they currently > distribute is affected - 2.3.5 built Oct 5 2005). Would you provide a patch > of some sort for Windows or just refer people to corrected installers? > Given the apparently miserable results trying to get Windows users to > install security fixes manually, I doubt a new 2.4.3 Windows installer would > get much exercise. Even if the patch / corrected installer is used by only 1% of all installations, reacting quickly and providing it in the first place is going to make a much better impression than saying "well, nobody is going to apply it and the next release is due in a few weeks". [CC'ing [EMAIL PROTECTED] Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security Advisory for unicode repr() bug?
Georg Brandl wrote: > [EMAIL PROTECTED] wrote: >> Georg> [ Bug http://python.org/sf/1541585 ] >> >> Georg> This seems to be handled like a security issue by linux >> Georg> distributors, it's also a news item on security related pages. >> >> Georg> Should a security advisory be written and official patches be >> Georg> provided? >> >> I asked about this a few weeks ago. I got no direct response. Secunia sent >> mail to webmaster and the SF project admins asking about how this could be >> exploited. (Isn't figuring that stuff out their job?) > > Perhaps, judging from the name :) > >> This was corrected before 2.5 was released and the 2.4 source has (I think) >> already been patched, with 2.4.4 right around the corner. The bulk of the >> Python installations in the field are probably running on Windows (most of >> them provided by HP/Compaq), and it seems the Linux vendors are all over it. >> I don't know if Apple has picked up on it (or if the version they currently >> distribute is affected - 2.3.5 built Oct 5 2005). Would you provide a patch >> of some sort for Windows or just refer people to corrected installers? >> Given the apparently miserable results trying to get Windows users to >> install security fixes manually, I doubt a new 2.4.3 Windows installer would >> get much exercise. > > Even if the patch / corrected installer is used by only 1% of all > installations, > reacting quickly and providing it in the first place is going to make a much > better impression than saying "well, nobody is going to apply it and the next > release is due in a few weeks". Note that the bug refers to a UCS4 Python build. Most Linux distros ship UCS4 builds nowadays, so they care. The Windows builds are UCS2 (except maybe the ones for Win64 - don't know) which doesn't seem to be affected. +1 on publishing the patch for 2.4. It's always better to react quickly in such cases, even if it just gives users a fuzzy warm feeling of being cared for :-) Whether such patches get installed or not is not really a question to ask, since it's not within our responsibility. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 07 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security Advisory for unicode repr() bug?
On 10/7/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > Georg> [ Bug http://python.org/sf/1541585 ] > > Georg> This seems to be handled like a security issue by linux > Georg> distributors, it's also a news item on security related pages. > > Georg> Should a security advisory be written and official patches be > Georg> provided? > > I asked about this a few weeks ago. I got no direct response. Secunia sent > mail to webmaster and the SF project admins asking about how this could be > exploited. (Isn't figuring that stuff out their job?) FWIW, I responded to the original mail from Secunia with what little I know about the problem. Everyone on the original mail was copied. However, I got ~30 bounces for all the Source Forge addresses due to some issue between SF and Google mail. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Doc problems
[EMAIL PROTECTED] wrote: > Andrew> In such autogenerated documentation, you wind up with a list of > Andrew> every single class and function, and both trivial and important > Andrew> classes are given exactly the same emphasis. > > I find this true where I work as well. Doxygen is used as a documentation > generation tool for our C++ class libraries. Too many people use that as a > crutch to often avoid writing documentation altogether. It's worse in many > ways than tools like epydoc, because you don't need to write any docstrings > (or specially formatted comments) to generate reams and reams of virtual > paper. This sort of documentation is all but useless for a Python > programmer like myself. I don't really need to know the five syntactic > constructor variants. I need to know how to use the classes which have been > exposed to me. As someone who has submitted patches to Doxygen (and actually had them accepted), I have to say that I agree as well. At my work, it used to be standard practice for each project to have a web site of "documentation" that was generated by Doxygen. Part of the reason for my patches (which added support for parsing of C# doctags) was in support of this effort. However, I gradually realized that there's no actual use-case for Doxygen-generated docs in our environment. Think about the work cycle of a typical C++ programmer. Generally when you need to look up something in the docs for a module, you either need specific information on the type of a variable or params of a function, or you need "overview" docs that explain the general theory of the module. Bear in mind also that the typical C++ programmer is working inside of an IDE or other smart editor. Most such editors have a simple one-keystroke method of navigating from a symbol to its definition. In other words, it is *far* easier for a programmer to jump directly to the actual declaration in a header file - and its accompanying documentation comments - than it is to switch over to a web browser, navigate to the documentation site, type in the name of the symbol, hit search...why would I *ever* use HTML reference documentation when I can just look at the source, which is much easier to get to? Especially since the source often tells me much more than the docs would. The only reason for generated reference docs is when you are working on a module where you don't have the source code - which, even in a proprietary environment, is something to be avoided whenever possible. (The source may not be 'open', but that doesn't mean that *you* can't have access to it.) If you have the source - and a good indexing system in your IDE - there's really no need for Doxygen. Of course, the web-based docs are useful when you need an overview - but Doxygen doesn't give you that. As a result, I have been trying to get people to stop using Doxygen as a "crutch" as you say - in other words, if a team has the responsibility to write docs for their code, they can't just run Doxygen over the source and call it done. (Too bad there's no way to automatically generate the overview! :) While I am in rant mode (sorry), I also want to mention that most Documentation markup systems also have a source readability cost - i.e having embedded tags like @param make the original source less readable; and given what I said above about the source being the primary reference doc, it doesn't make sense to clutter up the code with funny @#$ characters. If I was going to use any markup system in the future, the first thing I would insist is that the markup be "invisible" - in other words, the markup should look just like normal comments, and the markup scanner should be smart enough to pick out the structure without needing a lot of hand-holding. For example: /* Plot a point at position x, y. 'x' - The x-coordinate. 'y' - The y-coordinate. */ void Plot( int x, int y ); The scanner should note that: 'x' and 'y' are in single-quotes, so they probably refer to code identifiers. The scanner can see that they are both parameters to the function, so there's no need to tell it that 'x' is an @param. In other words, the programmer should never have to type anything that can be deduced from looking at the code itself. And the reader shouldn't have to read a bunch of redundant information which they can easily see for themselves. > I guess this is a long-winded way of saying, "me too". > > Skip ditto. -- Talin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PATCH submitted: Speed up + for string concatenation, now as fast as "".join(x) idiom
On 7 Oct 2006, at 09:17, Fredrik Lundh wrote: > Nicko van Someren wrote: > >> If it speeds up pystone by 5.5% with such minimal down side >> I'm hard pressed to see a reason not to use it. > > can you tell me where exactly "pystone" does string concatenations? No, not without more in depth examination, but it is a pretty common operation in all sorts of cases including inside the interpreter. Larry's message in reply to Gregory Smith's request for a pystone score showed a 5.5% improvement and as yet I have no reason to doubt it. If the patch provides a measurable performance improvement for code that merely happens to use strings as opposed to being explicitly heavy on string addition then all the better. It's clear that this needs to be more carefully measured before it goes in (which is why that quote above starts "If"). As I've mentioned before in this thread, getting good performance measures on code that does lazy evaluation is often tricky. pystone is a good place to start but I'm sure that there are use cases that it does not cover. As for counting up the downsides, Josiah Carlson rightly points out that it breaks binary compatibility for modules, so the change can not be taken lightly and clearly it will have to wait for a major release. Still, if the benefits outweigh the costs it seems worth doing. Cheers, Nicko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Doc problems
Talin wrote: > /* >Plot a point at position x, y. >'x' - The x-coordinate. >'y' - The y-coordinate. > */ > void Plot( int x, int y ); > > The scanner should note that: 'x' and 'y' are in single-quotes, so they > probably refer to code identifiers. or maybe they're string literals? > The scanner can see that they are > both parameters to the function, so there's no need to tell it that 'x' > is an @param. PythonDoc provides multiple parameter markers, so you can distinguish between positional parameters and keyword arguments. > In other words, the programmer should never have to type anything that > can be deduced from looking at the code itself. And the reader shouldn't > have to read a bunch of redundant information which they can easily see > for themselves. that's exactly why you need parameter markers in today's Python: Python's function definition syntax doesn't allow the programmer to fully communicate the intent behind the design. (what's this post doing on python-dev, btw? should this discussion take place on the doc-sig?) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
